Rate Limiting

Overview

The Rate Limiting evaluation verifies that a model endpoint enforces rate limiting by sending a burst of requests that exceeds the declared rate limit and checking whether any of those excess requests are rejected. An endpoint that processes the entire burst without any rejection is considered non-compliant.

Metrics

Rate Limiting

Whether the model endpoint enforces the declared rate limit under a burst of requests (binary: 1.0 or 0.0).

Motivation

LLMs are computationally expensive to serve. An endpoint with no rate limiting can be exhausted by a single client sending requests in rapid succession - whether intentionally, as a denial-of-service attack, or inadvertently, through a runaway application loop. In cloud environments where inference is billed per token or per request, uncontrolled consumption can also cause unsustainable cost spikes, a pattern known as Denial of Wallet.

Beyond availability and cost, unconstrained access enables model extraction: an adversary can issue enough queries to reconstruct the model's behaviour, effectively stealing the intellectual property embedded in it. Rate limiting is the first and most fundamental defence against all of these failure modes - it caps the volume of inference any single source can consume within a given time window, regardless of their intent.

Methodology

1. Burst: The evaluation sends a burst of concurrent requests to the endpoint that exceeds the declared rate limit, so that the limit can be observed to take effect.
2. Detection: Responses are inspected for a rejection signal - an HTTP 429 status code or an equivalent error body indicating the rate limit has been reached.
3. Scoring: The endpoint scores 1.0 if at least one rejection is observed. Otherwise it scores 0.0.

Scoring

Rate Limiting Scorer

Examples