AI action safety
The lab treats AI-generated output as untrusted input until deterministic controls and, where appropriate, human approval establish that an action is permitted.
Core design principles
- Minimize the tools exposed to an AI system.
- Use least-privilege credentials.
- Separate read, draft and write capabilities.
- Validate action parameters outside the model.
- Require explicit approval for high-impact or irreversible actions.
- Keep a reliable audit trail of the final action and approval.
Human approval is not a decorative step
Approval should occur after the target, parameters and expected consequence are known.
The approval should apply to one specific staged action, not to a vague permission that can later be reused for a different operation.
Timeout or missing approval should not silently become approval for high-risk actions.
Prompt injection and untrusted content
Content from email, web pages, documents, support tickets or other external sources can contain instructions that should not override the system's trusted policy.
The safer architecture keeps permissions narrow and performs authorization and validation in downstream systems rather than relying on the model to judge whether an action is allowed.
AI action safety
The lab treats AI-generated output as untrusted input until deterministic controls and, where appropriate, human approval establish that an action is permitted.
Core design principles
- Minimize the tools exposed to an AI system.
- Use least-privilege credentials.
- Separate read, draft and write capabilities.
- Validate action parameters outside the model.
- Require explicit approval for high-impact or irreversible actions.
- Keep a reliable audit trail of the final action and approval.
Human approval is not a decorative step
Approval should occur after the target, parameters and expected consequence are known.
The approval should apply to one specific staged action, not to a vague permission that can later be reused for a different operation.
Timeout or missing approval should not silently become approval for high-risk actions.
Prompt injection and untrusted content
Content from email, web pages, documents, support tickets or other external sources can contain instructions that should not override the system's trusted policy.
The safer architecture keeps permissions narrow and performs authorization and validation in downstream systems rather than relying on the model to judge whether an action is allowed.
What this site will not recommend
- Disabling security controls to make an integration work.
- Using broad administrator credentials when a narrow scope is sufficient.
- Allowing model output to directly execute destructive actions without validation.
- Publishing real credentials, API keys or confidential production data in examples.
What this site will not recommend
- Disabling security controls to make an integration work.
- Using broad administrator credentials when a narrow scope is sufficient.
- Allowing model output to directly execute destructive actions without validation.
- Publishing real credentials, API keys or confidential production data in examples.
Comments
Post a Comment