One Jev request, three scores: a policy label, a rubric level, and the probability the reply asks for a password, all recorded in Logfire.
Modeled cost of 1M evaluated replies: $62, vs $4,542 with Braintrust before data charges.
Blogpost
Pydantic AI is now a supported agent in @ScaleAILabs AgentEnv Framework.
Put it in a realistic environment (an inbox, a help desk), give it a task, and a verifier grades the result.
Change one ID to run a different agent on the same task.
pydantic.dev/articles/agent…
Elumynt runs 100+ daily agentic routines. Their Pydantic AI agent now writes each deterministic step as a Python program and runs it in Pydantic Monty. LLM spend on routines is down by over 80%, and the same inputs give the same output every day.
Security teams have gotten back ~332,000 analyst hours from @SentinelOne’s agents, at 99.7% agreement with human experts.
They get scored one capability at a time, each with its own rubric and judges scoped to it, gated in CI/CD by Pydantic Evals.