This document was first approved by the CoSAI Technical Steering Committee (TSC) on 14 July 2025
-
Building necessary trust for users and organizations to adopt Agentic Systems. Without trust, agents will never be widely adopted by users. Organizations will require proven security to trust agents to interact with their online services or access internal systems. Enterprise customers will only invest in agentic systems if they can deploy capabilities without taking on unacceptable risk.
-
An early, responsible approach to the security of agentic systems can help establish this trust. The rapid advancement of AI agent capabilities demands a proactive and collaborative approach to security. As these systems become more sophisticated, interconnected, ubiquitous and integrated into critical infrastructure, the potential for unintended consequences, malicious exploitation, and systemic failures grows dramatically.
To provide the foundational security properties required to:
- enable secure use of current and evolving agentic systems and frameworks
- minimize risks due to the potential for unpredictable behavior and emergent capabilities of LLMs
- ensure agentic actions have human accountability and oversight appropriate to the risk of the expected business outcome
- ensure foundational cybersecurity controls are in use prior to the rapid adoption of generative AI
Agentic System A framework that uses AI Agents, models and classical tools to autonomously perform tasks, make decisions, and take actions on behalf of users within defined parameters and oversight mechanisms.
Human Governance A management approach that ensures human beings maintain decision-making authority, oversight, and accountability over AI systems, rather than allowing unbounded autonomous operation.
Meaningful Control The ability for humans to understand, predict, and intervene in AI system decisions and actions in a substantive and effective manner.
Resource Any element of an human + computer system, including data, agent, identities, software, hardware, and communications.
Shared Model of Accountability A framework that clearly defines and distributes accountabilities among all stakeholders (technology producers, service implementers, and human principals) in an agentic system.
The following are foundational principles for creating Secure-by-Design Agentic Systems. These principles are intended to guide producers, practitioners, adopters, and policy makers in fostering a robust ecosystem of secure and trustworthy Agentic Systems.
- Human-governed and Accountable:
- …architected for meaningful control with clear, shared accountability throughout their lifecycle
- …subject to risk-based actionable controls and oversight ensuring alignment with expected business outcomes and failure recovery capabilities…
- …constrained by well-defined boundaries on authority and aligned with principals’ risk tolerance
- Bounded and Resilient:
- …designed with strict, purpose-specific entitlements on capabilities and resource access
- ...protected by robust, defensive measures including foundational cybersecurity controls, confidentiality controls, and AI-specific defenses against all threat actor
- …built for continuous validation of alignment with intended purposes and expected outcomes with predefined failure modes
- Transparent and Verifiable:
- …supported by secure AI supply chain controls covering provenance and selection, model integrity, and runtime validation
- …generating comprehensive telemetry of inputs, plans, decisions, communications, and outputs
- …enabling real-time monitoring, comprehension and forensic analysis for oversight and incident response
N.B. The Coalition for Secure AI focuses exclusively on the cybersecurity aspects of producing, implementing, and operating AI.