Overview
How testing, scoring, and guardrails work together.
Agents don't fail like traditional software. A prompt tweak, a model upgrade, or a new tool can change behavior in ways unit tests never exercise — and because agents take actions, a bad trajectory isn't just a wrong answer. It's a refund issued, an email sent, a record deleted.
Most teams discover agent risk in production, one incident at a time. Harbor replaces that with a loop you run on every release.
Each stage feeds the next: what testing finds, scoring watches for; what scoring surfaces, guardrails contain.
How Harbor connects
Harbor ingests OpenTelemetry traces from your agent through one of two paths. Both produce the same traces and power the same risk assessment — they differ in what they can see and how much you have to change.
| SDK | Gateway | |
|---|---|---|
| What you change | A few lines in your agent | Your LLM base URL, nothing else |
| Sees tool calls | With their actual execution results | As the model proposes them |
| Enforces guardrails | Inline, in your process | Inline, on the wire |
| Covers many agents at once | One integration each | Yes, via one URL swap |
Running both is the highest-assurance setup, not a conflict: the SDK enforces at ground truth while the gateway guarantees capture. Spans are deduplicated by trace and span id, so a call captured twice appears once.
Test with Adversarial Testing
White-box maps every AI system in your repositories — agents, tools, system prompts, memory, retrieval, MCP connections — and builds a dependency graph for each one. When a change lands, Harbor compares the changed files against those graphs, works out which systems the change can actually reach, and scans only those against our vulnerability cookbook of AI-specific exploit patterns.
Connect your repository so every pull request is assessed.
Review white-box findings on the systems each change affects.
With those findings, black-box adversarial testing runs attack scenarios against a provisioned instance of your agent — inspect the resulting attack trajectories.
Outcome: every release ships with a documented assessment instead of an assumption.
Score with Risk Score
In production, Harbor runs its proprietary risk assessment on every trace and trajectory it ingests, producing a risk score you can baseline, trend, and alert on.
Explore scored traces and trajectories to see where risk concentrates.
Set thresholds and alerts so risky behavior surfaces immediately.
Outcome: risk becomes a number you can track per agent, per release.
Guard with Guardrails
Guardrails cap the inputs of your agent's tools — amounts, recipients, scopes, and anything else an agent passes along. Enforcement runs inline through the Gateway or the SDK, so a call that exceeds its cap never reaches the tool.
Define input caps per tool, informed by what testing and scoring found.
Enforce them inline via the Gateway or the SDK.
Review guardrail hits alongside risk scores.
Outcome: a confused or compromised agent is bounded by policy, not by luck.
The loop closes
Guardrail hits and high-risk trajectories become adversarial test cases for your next release.