Skip to content
All articles
Engineering15 April 2026·3 min read

Ameya Bhatawdekar on building AI evaluations at Braintrust

Agents changed the shape of the problem.

Paycux engineering

Agents changed the shape of the problem. A human signing in is a single decision made once; an agent acting on someone's behalf is a decision that has to hold for every call it makes afterwards, often hours later, often without anyone watching.

This piece walks through how we think about it at Paycux, what we have changed our minds about, and where the sharp edges are.

The evaluation problem

Consider the evaluation problem. Consent is the part most implementations get wrong. Asking once at install time and then acting indefinitely is not consent; it is a standing grant with no expiry and no visibility.

The audit trail matters more here than in any human flow. When something goes wrong, the question is never "did the user intend this" in the abstract — it is which agent, acting under whose authority, made which call, and whether the record can prove it.

What braintrust is building

What braintrust is building is where this gets concrete. The audit trail matters more here than in any human flow. When something goes wrong, the question is never "did the user intend this" in the abstract — it is which agent, acting under whose authority, made which call, and whether the record can prove it.

Consent is the part most implementations get wrong. Asking once at install time and then acting indefinitely is not consent; it is a standing grant with no expiry and no visibility.

  • Scope every agent credential to one principal and one task
  • Give tokens minutes of life, not days
  • Record which agent acted, under whose authority, on what
  • Make revocation a single call that takes effect immediately

Prompt engineering is engineering

Prompt engineering is engineering deserves its own treatment. Consent is the part most implementations get wrong. Asking once at install time and then acting indefinitely is not consent; it is a standing grant with no expiry and no visibility.

Consent is the part most implementations get wrong. Asking once at install time and then acting indefinitely is not consent; it is a standing grant with no expiry and no visibility.

The audit trail matters more here than in any human flow.

Shipping AI with confidence

Shipping AI with confidence deserves its own treatment. An agent's credential should describe what it may do, not who owns it. Scope it to the narrowest set of operations that make the task possible, bind it to a single principal, and give it a lifetime measured in minutes rather than days.

The audit trail matters more here than in any human flow. When something goes wrong, the question is never "did the user intend this" in the abstract — it is which agent, acting under whose authority, made which call, and whether the record can prove it.

Where this leaves us

The pattern repeats across every system we have looked at: the hard part is not the mechanism, it is keeping the mechanism honest as the surrounding assumptions change.

If you are working through the same problem and want to compare notes, the docs cover the mechanics and the console shows the behaviour on your own data.

Everything here, already built

Sign-in, enterprise SSO, directory provisioning, roles and an audit trail behind one API. Start with the quickstart and have a working sign-in this afternoon.

Start selling to enterprise customers

Create an account, point sign-in at Paycux, and get back to the part of the product that is actually yours.