How to Evaluate AI Agent Action Controls

What should the evaluation cover?

Agree on one agent, one connector version, one destination and one independently observable result. The example below uses a document delivery. A platform team can apply the same method to a code merge or service restart by changing the fields and destination check.

Write down who can change the policy, whether another account can bypass the connector, and who has read access to the destination. Those boundaries determine what your result can establish.

Five sandbox checks

1. Establish a valid baseline

   Approve one document version for one recipient in an isolated workspace. Submit it once through the normal connector.

   Observe

   Look up the delivery at the destination. Record the recipient, document version and number of matching deliveries.

   Pass condition

   One matching delivery. A control that blocks every request also fails this case.

2. Change an approved field

   Keep the original approval and replace the recipient. Repeat with a different document version.

   Observe

   Inspect the decision record and query the destination for a delivery on the substituted terms.

   Pass condition

   The changed request requires a fresh decision. No delivery on the substituted terms appears under the original approval.

3. Expire the agent’s grant

   Run the same request just inside and just outside its access window.

   Observe

   Compare the timestamps, agent identity, decision and provider-observed permissions. Record whether this connector is the only route that can write.

   Pass condition

   The in-window request follows policy. The expired grant cannot authorise a new action through the protected route. Report any separate access the provider still allows.

4. Lose the reply and retry

   Allow the destination to create the delivery, interrupt the acknowledgement, then retry using the original operation identifier.

   Observe

   Count independent destination records across both attempts. Check what an operator sees while the outcome is uncertain.

   Pass condition

   There is one matching delivery. Until the destination can be checked, the operation stays unresolved.

5. Remove a source of risk evidence

   Repeat a request with its supporting record unavailable or stale.

   Observe

   Inspect the assessment for the missing input and its timestamp. Check the policy decision separately.

   Pass condition

   The assessment identifies the evidence gap. The resulting action follows the customer’s chosen policy for incomplete evidence.

What should the report contain?

For each case, save the exact input, the policy and connector versions, the decision, the operation identifier, and the destination observation. Use separate fields for what was authorised and what was observed. Mark a missing observation as unknown.

If the destination cannot be read independently, record that limitation. A connector reporting success is useful operational evidence, but it cannot establish the final recipient or count by itself. Re-run affected cases after a material change to the connector, destination or policy.

Where to inspect the technology

• Steelmatter Testing & Assurance describes connector qualification, replay and regression testing.
• The action lab lets you inspect four illustrative records and change their conditions.
• The public evidence specification documents the observer, signed export and verifier for the published implementation.
• The recovery procedures describe how the public reference tracks a request with an uncertain outcome.