Build an AI Agent Evaluation with JEV

Author(s): Quan Huynh

Originally published on Towards AI.

Build a small eval harness for a tool-using AI agent: code checks the work it did, and JEV judges the words it wrote.

One run of my incident agent told me a checkout slowdown was caused by a config deploy that shrank the database pool from 50 connections to 5. It was right. The explanation was clear; it cited four tools, and it even ruled out a payment-provider warning that showed up later in the logs.

Then I ran it with a shorter system prompt and got an answer that read almost the same. That one was wrong in two ways. It cited a tool it never called, and it never saw the warning it was supposed to rule out. If I had only read the paragraph, I would have shipped it.

That is the problem with grading an agent by reading its answer. A good paragraph and a good investigation are two different things. This post builds a small harness that checks both. Plain Python checks the work. Jev, a fast structured evaluation model, judges the explanation. You can clone it and run it in a few minutes.

Run it first, read about it later

I think the fastest way to understand an eval harness is to watch it grade something. So let’s start there.

You need Python 3.10 or newer, an OpenAI API key, and (for the judge step) a Cloudflare account. The code is in the devops-ai-guidelines repo:

git clone https://github.com/VersusControl/devops-ai-guidelines.gitcd devops-ai-guidelines/07-evaluating-ai-agents/code/chapter-08
python -m pip install -r requirements.txt
cp .env.example .env

Open .env and fill in OPENAI_API_KEY. Leave the Cloudflare fields empty for now. OPENAI_MODEL defaults to gpt-4.1-mini; I used gpt-5.4-mini.

Now run the agent once:

python run_agent.py

Before you look at the output, here is what that command does. It takes about ten seconds, and nothing in it touches a real system.

It loads a recorded incident and checks that the case hasn’t expired.
It hands the agent two things: the alert text and four read-only tools (get_metrics, get_logs, get_deploys, get_db_status). Each tool returns a response recorded from the incident, not live data.
The model picks one tool per turn. The runner calls it, writes down the call, and passes the result back. This repeats for up to five turns.
When the model thinks it knows the cause, it calls submit_diagnosis with a paragraph plus a few structured fields. The script prints that.

The answer key is in the same JSON file, but the agent never gets it. That matters later: it’s the only reason a grade means anything.

Here is what one run printed for me on September 24, 2026:

Scenario: checkout-latency-after-pool-change
Agent model: gpt-5.4-mini-2026-03-17
Alert: checkout-service p95 latency > 2s
Tool calls: get_metrics, get_deploys, get_db_status, get_logs
Root cause: A deploy at 14:02 changed DB_MAX_CONNECTIONS 50 -> 5, which immediately
reduced database pool capacity and caused checkout requests to queue and time out.
Category: deploy
Cause change: DB_MAX_CONNECTIONS 50 -> 5
Cause effect: pool_exhausted
Cited evidence: get_metrics, get_logs, get_deploys, get_db_status
Rejected signals: payment_provider_latency, checkout_image_v1_9_2, general_capacity_limit
Steps: 5

Your run will not match this word for word. The tool data is frozen, but the model can pick a different tool order or different wording every time. That’s normal, and it’s exactly why we need a grader instead of eyeballs.

What that output actually tells you

It’s worth slowing down here, because every line in that output is either something the agent did or something the agent said. The whole harness is built on keeping those two apart.

Read the figure from top to bottom:

Alert is the input. The agent gets only this and four read-only tools.
Tool calls are recorded by the runner, not reported by the agent. The agent can’t edit it after the fact. This is the most trustworthy line.
Root cause is the agent’s own paragraph. It’s free text, so no == check can grade it. This is the line Jev reads.
Category, Cause change, and Cause effect are short structured fields the agent must fill in. Code can compare them exactly with a hidden answer key.
Cited evidence is a claim. The agent says it relied on these tools. The grader checks that claim against the Tool calls line.
Rejected signals is also a claim: “I saw these and ruled them out.” The recording has a planted misleading signal (a payment-provider latency line that appears after the alert). The grader checks that the agent actually saw it and named it.
Steps are counted by the runner. This case allows five: four tool calls and one answer.

So in this run, the agent did the work: it called all four tools, including get_logs, so it really could have seen the payment line. The paragraph also sounds right. But “sounds right” is the part we can’t check with code, and that is where Jev comes in.

The incident behind the example

The example is an incident agent for a checkout service. The alert says p95 latency crossed two seconds. The real cause is a config deploy one minute earlier that cut DB_MAX_CONNECTIONS from 50 to 5. The pool fills up, requests wait for a connection, and checkout times out.

The order on that timeline is the whole puzzle. The deploy comes before the alert, so it can be the cause. The payment-provider line comes after, so it can’t be, even though “payment provider slow” sounds like a checkout problem. A good agent notices the order. A lucky one just picks the scariest line.

Every tool response comes from a recorded JSON file, not from production. The same file holds a hidden answer key: the true cause, the evidence that proves it, the planted distraction, and the step budget. The runner never gives the answer key to the agent. It loads it only after the agent has committed to an answer.

Incidents are just my example. The pattern works for any agent that uses tools and then explains itself: a support agent, a SQL agent, a code-review agent.

Two checks, one grade

An agent eval has to answer two separate questions:

Did the agent do the work? Which tools ran, what came back, how many steps.
Is what it wrote true and supported by what came back?

The first question has exact answers, so code should answer it. I call these hard gates. The second question is about meaning, so it needs a model.

The four hard gates live in grade.py:

cause: the category, change, and effect match the answer key, and the recorded tool data supports them (the deploy happened before the alert, the pool really is full).
evidence: the required tools were actually called, and every cited tool was actually called.
distraction: the misleading line was in the data the agent received, and the agent named it as rejected without blaming it.
steps: the agent finished within its budget and ended with an answer.

What Jev is, briefly

Jev is a model from TypeSafe AI. TypeSafe calls it a System One model: instead of generating text, it answers typed questions about a piece of content you give it. You send a state (the thing to judge) and a map of questions. Each question is one of three types: Noul, Choice, Score.

The answers come back as JSON your code can read directly. There is no paragraph to parse, and no chance of a missing field or a stray sentence before the JSON. That alone removes a whole class of bugs I’ve hit with LLM-as-judge setups.

Why Jev and not another LLM as the judge

You can use any chat model as a judge. I’ve done it. Here is why I picked Jev for this one.

It’s built for this shape of task. An eval judge doesn’t need to write anything. It needs to answer narrow questions like “does this explanation connect the change to the impact?” and give you a number. TypeSafe’s own docs list “score, judge, verify, guardrail” as core use cases.

It’s cheap enough to run on every commit. TypeSafe’s published price for jev-1.13.0 is $0.042 per million ($42 per billion) input tokens, and output tokens are free. Their launch post puts typical chat-model input prices at $0.20 to $10 per million tokens, with output around five times that.

To make that concrete, here is a small, made-up but realistic eval budget: 10 cases, 5 runs each, on 20 pull requests a day. That’s 1,000 judge calls a day. Say each call sends about 1,000 input tokens, and a chat judge writes about 100 tokens back.

The bars use a log scale, because a normal one would make Jev’s bar invisible. The chat numbers are ranges from TypeSafe’s post, not quotes for a specific model, so plug in your own. The point doesn’t change much: a judge that runs on every case, several times each, on every pull request adds up fast. With Jev it mostly doesn’t. (Cloudflare lists its own price for Jev in the dashboard, and it may not match TypeSafe’s direct price. Check before a large run.)

It’s fast. TypeSafe reports 70 to 500 ms end to end, because it produces all answers in one parallel pass instead of token by token. Their headline “193.6x faster, 444.6x cheaper” comes from their own workflow evals, and they say themselves that it’s on the high end of real-world gains. I’d plan for less, but even a tenth of that is a big deal inside CI.

Confidence comes with every answer. Jev is trained to return calibrated probabilities. That gives you a natural review_needed lane: when the judge isn’t sure, a person looks, instead of the pipeline guessing.

It isn’t magic, and TypeSafe is honest about that in its Jev 1.13 jaggedness page. It reads instructions literally. It’s weak at counting, arithmetic, and comparing dates. Accuracy drops when the state is full of irrelevant detail. And it doesn’t treat the state as hostile by default, so text inside it can try to steer the answer.

That list turned into my design rules. Anything numeric or time-based stays in Python: step counts, “did the deploy happen before the alert”, “was the payment line after 14:03”. Jev only gets the part code can’t do, which is reading the paragraph.

Why Cloudflare Workers AI

Workers AI is Cloudflare’s serverless inference service. You call models on Cloudflare’s GPUs with one account ID and one API token, over plain HTTPS. You don’t deploy a Worker for this; the REST API is enough.

Become a Medium member

Jev is listed in the Workers AI catalog as a third-party model. I used Cloudflare for one practical reason: we already had credit on our Cloudflare account, and it kept every model call on one bill. There is a free daily allocation (10,000 Neurons a day at the time of writing), but some models, including third-party ones, need a paid plan or prepaid AI Gateway credits.

You don’t have to use Cloudflare. TypeSafe offers Jev directly at with its own API key, plus Python and JavaScript SDKs. The request is the same idea: a state and a map of questions.

Pick whichever fits your billing. The grader only cares that it sends a state and questions and gets typed answers back.

Step by step: judging one run with Jev

First, a quick tour of the code

Before any code, here’s the map. The code folder has a handful of small files, and each one does one job:

Two objects travel through these files. If you understand them, the rest of the code is easy to follow.

The Conclusion is what the agent hands back. Some fields are written by the model, and some by the runner. For the run above, it looks roughly like this:

Conclusion(
# written by the model
root_cause="A deploy at 14:02 changed DB_MAX_CONNECTIONS 50 -> 5, which ...",
category="deploy",
cause_change="DB_MAX_CONNECTIONS 50 -> 5",
cause_effect="pool_exhausted",
evidence=["get_metrics", "get_logs", "get_deploys", "get_db_status"],
rejected_signals=["payment_provider_latency", "checkout_image_v1_9_2", ...],
# written by the runner, which the model can't edit
trajectory=[{"type": "call_tool", "tool": "get_metrics"}, ..., {"type": "conclude"}],
observations={"get_db_status": {"pool_size": 5, "in_use": 5, "waiting": 40}, ...},
)

trajectory is the list of actions in order. observations is what each tool actually returned. These are the same two groups from the output figure earlier: what the agent did and what it said.

The answer key is the truth, stored in the same JSON file but never given to the agent:

"answer_key": {
"true_category": "deploy",
"true_cause": "The 14:02 deploy reduced DB_MAX_CONNECTIONS from 50 to 5, exhausting the connection pool.",
"cause_change": "DB_MAX_CONNECTIONS 50 -> 5",
"cause_effect": "pool_exhausted",
"required_evidence": ["get_deploys", "get_db_status"],
"distraction_id": "payment_provider",
"max_steps": 5
}

(I trimmed a few fields that describe where the distraction appears.)

Now the whole judged run fits in a few lines. This is the core of run_judge.py, simplified:

agent_input, conclusion = run_agent(settings, case_path=case) # the agent investigates and commits
answer = load_answer_key(case) # only now load the truth
result = evaluate(conclusion, answer, send_to_jev, DEMO_POLICY)
print(result["status"]) # pass, fail, review_needed, or incomplete

And evaluate, in jev_judge.py, does three things:

hard = grade(conclusion, answer) # 1. the four hard gates, in plain Python
payload = build_request(conclusion, answer, ...) # 2. a short state plus two questions for Jev
response = send(payload) # 3. HTTPS to Cloudflare, typed answers back
# then check the reply and combine hard gates + Jev into one status

The four steps below walk through that flow: what goes into Jev, what we ask it, how the call works, and how to run the full grade.

1. Build the judge’s input from the grader side

Jev can’t read a Python object, and it shouldn’t see everything anyway. So build_request takes the Conclusion and the answer key and makes one small dictionary, called the state. It’s the only thing Jev reads about this run:

state = {
"case_id": answer.scenario_id,
"expected_cause": answer.true_cause, # from the hidden answer key
"agent_explanation": conclusion.root_cause, # the paragraph the agent wrote
"alert_started_at": conclusion.alert["started_at"],
"observed_tool_responses": observed, # short facts, only from tools that ran
}

Think of it as a note to a reviewer: “Here’s what really happened, here’s what the agent wrote, and here’s what its tools showed. Does the paragraph hold up?”

Two details matter here. expected_cause is fine to send to the judge, because the agent has already answered and can’t change its answer. And observed is a set of short facts computed from conclusion.observations, like {"pool_size": 5, "in_use": 5, "waiting": 40}, not raw log lines. Only tools the agent really called show up there. That keeps the state small (Jev likes that) and limits what leaves your machine.

Before anything goes out, an approval step checks the state against the recorded case and rejects explanations that look like they contain a token, URL, or email address. That’s a guard for these synthetic cases, not a real redaction policy. For production incidents, you’d need a reviewed allowlist.

2. Ask two narrow questions

The rubric is two questions in one request:

QUESTIONS = {
"explanation_quality": {
"type": "score",
"instructions": "How well does the agent explanation connect the expected change to the observed checkout impact?",
"criteria": [
"The explanation contradicts the observations or gives a wrong mechanism",
"The explanation names the change but leaves out how it caused checkout impact",
"The explanation connects the observed change, its supported mechanism, and checkout timeouts",
],
},
"unsupported_claim": {
"type": "noul",
"instructions": "Does the agent explanation make a material claim unsupported by the observed tool responses?",
"criteria": {
"true": "A material claim has no support in the returned tool responses",
"false": "The material claims follow from the returned tool responses",
},
},
}

The Score places the explanation on three described levels, 0 to 2. It comes back as a probability for each level, and the score is the weighted average. For example, probabilities of 0, 0.25, and 0.75 give a score of 0(0)+1(0.25)+2(0.75)=1.750(0)+1(0.25)+2(0.75)=1.75.

The Noul asks one specific thing: did the agent claim something no tool showed? Watch the direction. A high value is bad here, because “true” means unsupported.

I kept these as two questions on purpose. An explanation can describe the right mechanism and still invent a rollback that never happened. One blended “quality” number would hide that.

3. Call Jev through Cloudflare

Add three values to .env: CLOUDFLARE_ACCOUNT_ID, CLOUDFLARE_API_TOKEN (create it under Workers AI, “Use REST API”), and JEV_MODEL. The documented request looks like this:

url = f"https://api.cloudflare.com/client/v4/accounts/{account_id}/ai/run"
body = {"model": "typesafe/jev", "input": {"state": state, "questions": QUESTIONS}}
# POST it with "Authorization: Bearer ". The answer is under result.answers.

The response has model, answers, and usage. The grader checks every field before using it: the types, the ranges, that the Score probabilities add up to 1, and that model is the version you calibrated against (jev-1.13.0). If Cloudflare or TypeSafe moves the model behind the name, the grade becomes incomplete until you re-check your thresholds. A silent model upgrade should never shift your benchmark.

4. Run the full grade

python run_judge.py

This starts a new agent run, prints it in the same format as before, then prints the four gates, Jev’s model and two answers, and the final status. It exits nonzero for anything but pass, so it drops straight into CI.

The thresholds live in plain Python, in DEMO_POLICY: a minimum score of 1.5, a minimum confidence of 0.6, and an unsupported-claim value of at most 0.2 to pass (0.8 or more is a clear failure; in between goes to review). I picked those to show the wiring. They are not measured numbers. More on that below.

Use the grade to improve the agent

One grade for one run tells you very little. The real value of an eval is comparison: change one thing, run again, see which check moved.

The repo has ten small simulation files in scenarios/simulations. Each names a recorded case and the agent settings to use:

{
"id": "checkout-latency-01",
"case": "pool",
"change": "Baseline: one vague instruction",
"instructions": "Find out why checkout is slow. Call submit_diagnosis when you are done.",
"max_steps": 5
}

The tool data, the answer key, the gates, and the judge questions never change between files. Only the agent does. So when a result moves, you know what moved it.

python run_simulations.py # all ten, in parallel
python run_simulations.py --only checkout-latency-01
python run_simulations.py --no-judge # hard gates only, no Cloudflare needed

Here are the hard-gate results from my run on the pool case, one prompt change at a time:

What happened, in order:

01 found the right cause, but wrote the change as a sentence (“changed from 50 to 5 in the 14:02 config deploy”), so the exact cause check failed. Its “rejected signals” were full sentences, not names. Its paragraph, by the way, read perfectly well.
02 fixed the cause field. But this time the agent skipped get_logs and still listed it as evidence. The evidence gate caught a citation for a tool that never ran. That’s the run I described at the top of this post.
03 gave the rejections proper names, but the agent still skipped the logs. So it couldn’t have seen the payment line it was supposed to rule out. The instruction was right; the agent still didn’t look.
04 asked it to compare times against the alert. That pushed it to read the logs, where the timestamps are. All four gates passed.

Then I pushed on version 04. With only three steps allowed, it ran out before answering, and every gate failed. That’s the eval catching a budget that’s too tight. With an alert that wrongly blamed the payment provider, it still found the deploy. And when I ran the same prompt against the other two incidents, it failed the distraction gate on both: it named the rejected signals after tools (get_db_status) instead of the component in the log line. One more instruction (“name each one after the component in the log line”) fixed the capacity case, but on the dependency case the agent wrote db where the answer key expects db_pool.

That last one is a good test of the grader, not just the agent. Is db close enough? Maybe. But if you loosen the rule, write it down as a scoring change and rerun every case. Don’t edit the answer key until the table goes green.

Where does Jev fit into this story? Look at version 01 again. Two exact checks failed, while the paragraph itself was fine. A judge that only reads prose would have passed it. The reverse happens too: a run can pass all four gates while its paragraph adds a claim no tool showed, like “the team rolled the change back at 14:05” (an example I made up). No gate reads prose, so only the judge sees that. You need both checks, and neither one gets to overrule the other.

Before you trust the numbers

A few things I’d do before putting this in a real pipeline:

Calibrate the thresholds. Collect human-reviewed explanations: correct, a lucky guess, a wrong cause, an unsupported extra claim, and a borderline one. Compare Jev’s answers with the reviewers, look at every false pass and false failure, then pick thresholds and check them on held-out cases.
Run each version more than once. My table is one run per prompt. The model can take a different path next time. Call something an improvement only after it holds over several runs.
Pin the judge. Record the model ID and rubric version with every grade. Re-run your labeled examples when either one changes.
Watch what you send. Jev’s docs say English is its strongest language, and that state is not treated as hostile by default. Keep the state short, put a clear instruction in every question, and don’t send real incident data to any third party without a reviewed redaction policy.
Run the offline tests. python -m unittest discover -p 'test_*.py' checks the plumbing with fake responses and no API keys. It proves the code path, not the models.

In short

Keep what the agent did (recorded by the runner) separate from what it said (written by the model). Grade the first with code.
Use a judge only for what code can’t check, and ask it narrow typed questions instead of “is this good?”
Jev fits the judge role well: typed answers, calibrated confidence, low latency, and a price low enough to run on every commit. Keep numbers, dates, and counting in code, where it’s weak.
Treat judge failures as incomplete, never as a score.
Change one thing at a time and let the gates tell you what to fix next.

If you want to go deeper, this harness is the running example in my free book, Evaluating AI Agents. It builds everything here from scratch: recording a case, replaying it, keeping the answer away from the agent, the hard gates, the Jev judge, and then turning scores into a benchmark and a CI gate.

Published via Towards AI

Source link