Credential Hygiene (Claude Managed Agent)

Overview

The Credential Hygiene (Claude Managed Agent) evaluation checks that a Claude managed agent deployment handles credentials securely. It covers three distinct properties: whether credentials are embedded inline in agent definitions, whether sessions connecting to MCP servers attach vault-managed credentials, and whether those vault credentials use short-lived OAuth tokens rather than static bearer tokens.

Metrics

Inline Credential Detection

Checks whether any agent in the system — main agent or sub-agents — has credentials embedded directly in the system prompt or in MCP server URLs. Inline credentials bypass vault-based rotation and revocation: if the agent definition is exfiltrated, those credentials are immediately usable.

MCP Vault Usage

Checks whether recent sessions for the agent are created with vault IDs when MCP servers are configured. Sessions without vault IDs indicate that credentials are not being managed through the vault, which means they must be embedded elsewhere or MCP connections are unauthenticated.

Vault Credential Type

Checks whether the credentials stored in the vault use short-lived MCP OAuth tokens rather than static bearer tokens. Static bearer tokens are long-lived and have no automatic rotation or expiry — if leaked, they remain valid until manually revoked.

Motivation

Credentials are a critical attack surface in agent deployments. Inline credentials embedded in a system prompt or URL are exposed to anyone who can read the agent definition — including other agents in a multi-agent system or an attacker who has exfiltrated the configuration. Even if credentials are correctly stored in a vault, sessions that are not created with vault IDs do not benefit from that protection. And even with vault usage, static bearer tokens provide no automatic rotation or time-limited exposure window, so a leaked token remains a liability indefinitely.

Together, these three checks verify that credentials are stored in the vault (not inline), that sessions actually use the vault at runtime, and that the vault itself contains appropriately scoped and time-limited credentials.

Methodology

1. Inline credential detection: The evaluation retrieves the main agent and all sub-agents recursively. Each agent's system prompt is scanned with regex patterns for common credential formats, and each MCP server URL is parsed to detect userinfo credentials. Note: the system prompt check uses regex rather than a judge model to avoid sending potentially sensitive content to an external API. It catches obvious accidental embedding but may miss credentials with non-standard formats.
2. MCP vault usage: The most recent sessions (up to sample_size) are sampled. For agents with MCP servers configured, each session's vault_ids field must be non-empty. This is a sample-based check — a passing result does not guarantee future sessions will include vault IDs.
3. Vault credential type: All credentials in the specified vault are listed and their auth.type is inspected. static_bearer credentials are flagged; mcp_oauth credentials pass, with the presence of a refresh token and expiry noted.

Scoring

Inline Credential Detection

MCP Vault Usage

Vault Credential Type

Examples