Braintrust

Imagine doing your job without ever looking anything up on the internet. That's an agent without web search. Giving agents the web changed what they can do, especially on anything current. But agents don't search like people, so optimizing their performance is a new challenge. There's a lot of great research out there on search behavior, but we wanted to answer a more operational question: when should search be on, and how should you configure it? We evaluated 1,329 current events questions across 4 models and 14 conditions, comparing You.com, provider built-in search, and no search. We found that: - Search reduced the gap between models from 47.9 points to 5.6 - Retrieval gain declined with event age, from ~45 points for recent events to ~24 for the oldest - Runs with 5+ searches scored 19–49%. A fifth query was associated with lower performance Read the research → https://lnkd.in/g4SVhbja

The Braintrust eval library has a repo of skills your coding agent can read, so you can easily build and run evals on your data. Here's an example of using a skill in Claude Code to compare the Codex CLI and Pi, both running GPT-5.6 Sol, on a 30-task stratified SWE-bench Verified dataset. The skill saves those experiments and traces to Braintrust. Try it yourself → https://lnkd.in/euUiMUQw See more skills → https://lnkd.in/e855R8Px

If you want to reduce agent costs without trading away quality, the right unit of analysis is cost per resolved request, measured against the quality bar your product needs. We evaled different strategies for optimizing agent cost-efficiency, and found that the strongest approach is routing plus guarded escalation, not a single-model replacement. Read more → https://lnkd.in/gftcwRZX

Rex (YC S26) is an AI-native service for automating order-to-cash. From day one, their engineers made a deliberate investment to trace every agent's touchpoint using Braintrust. Then they built evals against production data to drive model selection, latency optimization, and auditing of customer-specific operating procedures. Read more → https://lnkd.in/eGT2qxnG

The entire internet spent the last 48 hours saying the same thing: I don't want your agent, I want my agent to use your product. The Braintrust MCP now lets the agent you already use act on what it finds in Braintrust. That means your agent can: - Turn traces that fail a scorer into a regression dataset - Build monitors for total cost, cost per trace, and cost by model - Create and test scorers against production traces - Compare prompt changes across quality, latency, and cost Bring your agent or use ours, whatever works best for your team. Read more → https://lnkd.in/eeigpeeB

For a long-horizon agent making hundreds of decisions per trajectory, scoring the final answer is not enough to know which step went wrong. Braintrust and Basis built behavior specs, an open standard for defining and evaluating how an agent should behave across a trajectory. Write specs in Markdown, judge them against production traces, and test continuously as you update prompts or tools. Everything is open source at agentbehavior.dev. Read more → https://lnkd.in/gjqVvvMX

Some agents need a sandbox where they can do work, liking editing files, installing dependencies, and launching builds. To score these agents you run the tests and check what changed, which takes a clean container per attempt. Harbor is a Python framework for specifying sandboxed agent tasks and running them in isolated Docker containers. Each task holds an instruction for the agent and a verifier that inspects the container once the agent stops. With the Harbor native Braintrust plugin, you can eval an agent in a sandbox, compare across runs, and share with anyone, without moving a folder. Read more → https://lnkd.in/e-b8gkcc

What's new: - Braintrust's Eval library, a resource of open source evals and 20+ skills, for running your own evals and inspecting our results - Kimi K3 and DeepSeek V4 join the Braintrust model lineup across playgrounds, prompts, and the Gateway, no AI provider setup required - Reduce request-path logging latency from AWS Lambda functions with the Braintrust Lambda Extension - Azure AI Gateway is now supported as an AI provider, letting you use one provider for all backends behind an Azure API Management endpoint Read more → https://lnkd.in/eGcXkubq

There are two ways to score a coding agent. By its output, or by its behavior. Output scoring tells you if the agent produced a desired result, but can miss cases where the agent does something it was instructed not to. So we ran an eval to see if scoring by behavior can catch things that output scoring misses. We told an agent to find code through vector search only and to never grep the filesystem, and enforced these rules four ways. Then we looked at both output and behavior scores. Output scoring couldn't tell the four agents apart, whereas behavior scoring caught the leaks that we expected to find. Read more → https://lnkd.in/eVdmEeiG