A standard vendor questionnaire tells you who the vendor is and how they store your data. It won't tell you what an AI agent executes on your servers once you authorize it—which is the only thing that decides whether it's safe.
Procurement won't let the agent tool through until security signs off. Security hands the requester—usually the SRE or platform lead who wants the tool—a vendor questionnaire. I've watched this exact handoff in a lot of enterprise deals: the security team parses the SOC 2 report line by line, procurement builds a binder thicker than the product's own docs, and everyone treats the exercise as thorough.
The problem is that the questionnaire was written for a SaaS that stores PDFs, not for software that autonomously runs commands on your infrastructure. It asks where your data lives and who the vendor's sub-processors are. It has almost nothing to say about what the agent runs once it's inside, what gets blocked, and who's accountable when it goes wrong. The questions that actually predict whether an AI agent tool is safe are all about execution—and that's the section most 2026 questionnaires are still missing. This post is that section.
What does a standard AI vendor risk assessment check in 2026?
An AI vendor risk assessment is the review a buyer runs before authorizing a third-party AI tool—and in 2026 it sits on top of a serious framework stack. NIST AI RMF and its Generative AI Profile, ISO/IEC 42001 for AI management systems, and the EU AI Act's risk classification are the three anchors most questionnaires map to. Regulated buyers layer on more: DORA Article 28 requires financial entities to keep a register of every ICT third-party arrangement and treat AI vendors as ICT providers, and SOC 2 vendor management sets the baseline for evidence. Newer entrants are converging fast—the US Treasury's Financial Services AI Risk Management Framework (February 2026) introduced 230 control objectives with an explicit third-party-risk domain, and Singapore's IMDA extended its Model AI Governance Framework toward agentic AI.
One calendar note worth getting right, because it's changed: the EU's Digital Omnibus package (in force from July 2026) moved the stand-alone high-risk (Annex III) obligations from August 2026 to December 2027. But August 2, 2026 is not cancelled—most Article 50 transparency obligations and GPAI enforcement still take effect on schedule.
All of this is good work. But look at what it measures: who the vendor is, how they handle your data, how they govern their models, and which controls they attest to. It's the lineage of a data-processor review. It stops at the edge of the thing that actually matters for an agent.
Why doesn't that questionnaire cover what an AI agent does?
TL;DR: Because it was built for software that stores your data, not software that runs commands on your infrastructure. Execution is a different axis of risk, and the questionnaire doesn't have an axis for it.
An AI agent tool's risk isn't where your data sits at rest. It's what the agent executes on your servers once it's authorized, and under whose identity. That's a different question than any data-residency clause can answer.
There's one version of it that sounds deceptively simple and that most deployments quietly fail: what did the agent do, why did it do it, and who was accountable? Answering that requires three things agentic systems usually lack—predictable execution paths, permissions scoped to the task instead of standing, and a clear line of ownership back to an authenticated human, not a shared service account. If a tool can't answer that question after the fact, no amount of SOC 2 attestation changes the fact that you can't govern it.
And you can't audit what you never captured. The Cloud Security Alliance found that 82% of enterprises have unknown AI agents in their environments. The visibility gap starts before the agent ever runs a command—and the questionnaire is where you're supposed to close it.
What runtime questions belong on the questionnaire?
Here's the section I'd add to any AI vendor assessment. Each question maps back to a framework the buyer already cares about:
- Which commands or actions can be blocked at runtime, and how? This is the execution-control question the data review never asks (NIST AI RMF, EU AI Act risk controls).
- When is human approval required, and how does the human intervene? Maps to the EU AI Act Article 14 human-oversight obligation.
- Can you terminate or intervene on a running agent—and how fast? Article 14's "stop" obligation is now getting numbers attached: some 2026 questionnaires are asking for termination under five minutes for production agents, and under one minute for agents with transaction authority. Treat the specific windows as an emerging buyer trend, not a standard—but the question is real.
- What did the agent do, why, and under whose identity? The audit trail has to name an authenticated human, not just a service account (EU AI Act Article 12 record-keeping, SOC 2, ISO 42001).
- Are the agent's permissions standing, or scoped to the task and auto-expiring? Standing privilege is the blast radius (least privilege; NIST AI RMF govern/manage).
- What's the inbound attack surface of the tool itself? The vendor's own software is now part of your attack surface.
These aren't hypothetical gaps. In Kiteworks' 2026 Data Security and Compliance Risk Forecast, 60% of teams said they couldn't be sure they could terminate a misbehaving agent at runtime, and 63% couldn't enforce agent purpose limitations. If most teams can't answer those questions about their own stack, they should be asking them of every vendor.
Why do runtime questions matter more in 2026?
TL;DR: The paper exercise is becoming theater—both sides now point AI at the questionnaire—and the year's real incidents were all about what agents executed, not where data was stored.
Something is quietly breaking the questionnaire as a control. Increasingly, vendors use AI agents to auto-fill security questionnaires and buyers use AI agents to auto-review the answers. When both sides automate the paperwork, the paperwork stops being evidence. The only questions that survive that arms race are the ones about runtime behavior you can't fake in a PDF: what the agent executed, what got blocked, and who approved it.
The incidents reported across 2026 point the same direction. Command-execution vulnerabilities in widely used MCP SDKs, an RCE-plus-credential-theft chain in an AI developer assistant, and a reported agent-orchestrated espionage campaign all had the same shape—the risk wasn't the vendor's data handling, it was what the agent could run once it was connected. We've written up how this plays out in our analysis of agent attack surface, and a companion piece walks through the first in-the-wild LLM-agent intrusion. Every one of them would have passed a data-processor questionnaire.
How do you actually score a vendor on execution control?
The runtime section maps almost one-to-one onto what an AI-native PAM built on execution control can actually do. Here's how Alpacon answers each question, with the shipped-versus-roadmap line drawn plainly:
- Block at runtime. Every command is scored in real time (file transfers are ACL-gated today, with intent-judged transfers on the roadmap), and Alpacon can hard-deny a high-risk operation or route it to admin approval rather than blocking the whole session. The default posture is monitor-and-record; enforcement is a deployment choice, so the accurate claim is that Alpacon can block or hold at the command, not that it always does by default.
- Human approval. The ambiguous action routes to a human approver who clears it before it proceeds, so the agent waits instead of guessing.
- Stop a running agent. Where enforcement is on, targeted in-flight intervention can hold or deny the specific action rather than killing the whole session, so intervention lands at the command instead of after the fact. Kill-chain and attack-pattern detection run today as post-session forensic analysis (MITRE-mapped, on the recorded session); real-time in-session detection is on the roadmap.
- What did it do, under whose identity. Every governed session is recorded, and execution audit puts humans and agents on one timeline, each action bound to an authenticated identity and the exact credential behind it—not a shared service account. (We go deeper on that in a separate audit write-up.)
- Standing versus scoped. Session-scoped control and just-in-time, auto-expiring access mean permissions exist for the task and revoke when the session ends.
- Inbound attack surface. An outbound-only architecture means zero inbound ports on the connected host, so the tool doesn't add an inbound door.
Be clear about which of these are table-stakes. Session recording, JIT expiry, and an outbound-only architecture are things any serious PAM should already clear—useful, but not what separates an AI-native vendor. The ones that decide it are the runtime controls: whether a command gets scored and can be held at execution, and whether intervention lands in-flight rather than as an after-the-fact report. That's the axis a data-processor questionnaire has no column for.
The prompt-injection question is worth calling out, because it's where vendors overpromise. The honest answer isn't "we detect every injection"—it's that the enforcement point is the command, not the prompt. Whatever talked the agent into running rm -rf, the command still gets scored and can be held at execution. Auto-generating framework-mapped SOC 2 or ISO evidence from those records is on our roadmap, not a shipped claim; the records that evidence is built from are captured today.
FAQ
What is an AI vendor risk assessment? An AI vendor risk assessment is the review a buyer runs before authorizing a third-party AI tool—vetting who the vendor is, how they handle data, and which frameworks (NIST AI RMF, ISO 42001, the EU AI Act) they map to. For an AI agent tool, it also has to cover what the agent executes on your infrastructure, not just where your data sits.
What should an AI vendor risk assessment include for AI agents? Beyond the standard data-processor review, add six execution-control questions: what commands can be blocked at runtime, when a human approves, whether you can stop a running agent and how fast, what it did and under whose identity, whether permissions are standing or scoped and auto-expiring, and the tool's own inbound attack surface.
How do you score an AI vendor on execution control? Map each runtime question to what the tool can actually do at execution—whether a command gets scored and can be held or denied in the moment rather than only logged, whether intervention lands in-flight, and whether every action ties back to an authenticated human. Be clear about which capabilities are shipped versus roadmap.
The takeaway
Your vendor questionnaire is doing the wrong job if it stops at data handling. An AI agent's risk lives one layer down, in what it executes on your infrastructure—so add the runtime section, and score every vendor on whether they can block, approve, stop, and account for what their agent runs. That's the layer we built Alpacon to govern.
