Test before release. Test again when dependencies change.

A scan can flag security risks. Behavioral tests check that a skill does the work it is expected to do. Both belong in the evidence for an exact release.

Example test run · three-way-match@2.4.1
Static checks
Manifest reviewed
Sandbox
Observed access recorded
Golden set
Expected results compared
Regression
Compared with 2.4.0
Compatibility
Target agent tools recorded

Six layers of tests

  • Manifest and structure

    Confirm the bundle has valid instructions, typed inputs and outputs, and declarations for required tools.

  • Sandbox execution

    Run scripts in isolation and review whether observed access matches the manifest.

  • Golden-set tests

    Score known inputs against expected outputs and cover each declared failure mode.

  • Adversarial cases

    Check how the skill behaves when it receives misleading inputs or encounters edge cases.

  • Regression checks

    Compare the candidate with its predecessor to see where results improved or declined.

  • Compatibility

    Record which models and agent tools were exercised by the test suite.

Continuous re-testing

A result belongs to the model, tools, and cases that produced it. Re-test when they change.

ChangeReview
Model changeCheck the release with the new model
Tool or connector changeRe-run affected cases
New golden casesCompare coverage and expectations
Scheduled reviewSet a cadence that fits the workspace

Test on your own work

  1. Add representative cases

    Use examples drawn from the work your team already reviews.

  2. Set acceptance thresholds

    Decide the minimum score and regression tolerance for each trust tier.

  3. Compare before rollout

    Run an evaluation against expected outputs before granting production access.

What auditors receive

RecordWhat it shows
Suite versionCases and approver
Results by caseInputs, expected and actual outputs, and score
Regression comparisonChanges from the prior release
Sandbox traceObserved access
Release decisionApplied threshold and approval