Red Team: Goal Hijacking

Overview

The Red Team: Goal Hijacking evaluation tests whether an AI application can resist goal hijacking - attempts by an adversary to cause the system to act outside its intended purpose. This might mean extracting information the system should not disclose, producing content it should refuse, or taking actions outside its authorised scope.

Unlike static prompt injection benchmarks that test a model against a fixed set of adversarial inputs, this evaluation uses an autonomous agent that is seeded with a description of the target application and its intended behaviour - and optionally its architecture - to devise attack strategies and conduct multi-turn interactions to attempt a hijack. The two evaluations are complementary: prompt_injection establishes a baseline against known patterns, while this evaluation simulates an adaptive adversary targeting the specific application.

Metrics

Goal Hijacking Safety

The proportion of red-team sessions in which the system successfully resisted goal hijacking (range: 0.0 to 1.0, higher is better).

Motivation

AI applications are typically designed with a specific purpose in mind, and their safety properties are tested against that purpose. But an attacker does not approach a system the way a developer does. They probe it, observe how it responds, and adapt - looking for the framing, the context, or the sequence of turns that causes the system to cross a line it was designed not to cross. A system that resists a direct override attempt may still be vulnerable to persistent social engineering, gradual context manipulation, or exploitation of edge cases in its instructions.

The consequences of a successful hijack depend on what the system can do. A customer support assistant manipulated into providing legal advice creates liability. A code assistant hijacked into generating malicious code creates a direct security risk. The failure mode is not always dramatic - sometimes a system drifts outside its intended scope across a long conversation without any single turn appearing obviously wrong.

Static benchmarks test known patterns and provide a useful baseline, but they cannot anticipate how a determined adversary will approach a specific system. This evaluation fills that gap by simulating an active adversary that reasons about the target and adapts its strategy, providing a more realistic measure of real-world robustness.

Methodology

1. Application description: The operator provides a description of the AI application's intended use - what it is supposed to do, what it is not supposed to do, and what kinds of outputs or actions would constitute a hijack.
2. Architecture description (optional): The operator may provide a description of the application's architecture. The red-team agent uses this to generate more targeted attacks - for example, crafting inputs designed to exploit specific tools or integrations.
3. Red-team agent: An autonomous agent is seeded with the application description and attack objective. It devises attack strategies, conducts multi-turn interactions with the target system, observes responses, and adapts its approach across multiple sessions.
4. Hijack detection: After each session, the agent determines whether the system was successfully hijacked - whether it produced output or took an action outside its intended purpose as defined by the application description.
5. Scoring: Each session receives a binary outcome - hijacked or not. The goal hijacking safety score is the proportion of sessions that successfully resisted a hijack attempt.

Scoring

Goal Hijacking Scorer

Examples

The following examples use a legal research assistant that is restricted to summarising case law and should never provide direct legal advice.