Tutorials

How to Rate Limit LLM APIs

Build a production-grade rate limiter for GPT-5 and Claude 5 endpoints using Redis and FastAPI to prevent upstream token exhaustion.

AM
Alfian Majid
//8 min read
How to Rate Limit LLM APIs

Uncapped LLM API traffic will drain your budget before lunch. I watched a simple webhook integration burn through four hundred dollars in compute credits over a single weekend because a background queue retry loop went haywire against a standard OpenAI endpoint. If you are shipping production features powered by GPT-5.6 Sol or Claude Sonnet 5, protecting your upstream quota with a local token bucket rate limiter is not optional. Let us build a resilient proxy layer that intercepts requests, enforces per-user quotas, and returns proper HTTP 429 headers without dropping your workers.

Designing the FastAPI and Redis Proxy Architecture

Our goal is to sit a lightweight FastAPI gateway between your client application and your LLM provider. This gateway uses Redis as an atomic counter backend to track token consumption and request velocity per user ID. When a client fires a prompt, the proxy checks Redis using a Lua script to evaluate both requests per minute and total token usage predictions. If the client is within limits, the request pipes directly to the upstream model provider. If they are over, the gateway rejects them instantly.

We choose Redis for this task because standard in-memory dictionaries fail the moment you scale your FastAPI workers across multiple containers or Kubernetes pods. A distributed token bucket needs atomic operations to prevent race conditions during traffic spikes. Here is the exact architecture we will construct using Python 3.12, FastAPI, and Redis-py.

Setting Up Your Local Development Environment

You need a clean machine running Python 3.12 or higher, Docker Desktop for your Redis instance, and active API keys for your preferred providers. Fire up your terminal and create a dedicated project directory to keep dependencies isolated.

Run these exact commands in your terminal to initialize your virtual environment and install the required packages:

python -m venv venv
source venv/bin/activate
pip install fastapi==0.115.0 uvicorn==0.31.0 redis==5.0.8 pydantic==2.9.2 httpx==0.27.2
docker run -d --name local-redis -p 6379:6379 redis:7.4-alpine

Verify your Redis container is healthy by running docker ps and ensuring port 6379 is exposed. Next, create a .env file in your root directory containing your upstream API credentials and Redis connection string. Keep this file out of your source control.

REDIS_URL=redis://localhost:6379/0
UPSTREAM_AI_API_KEY=sk-ant-api03-example-key-do-not-use
UPSTREAM_MODEL_NAME=claude-sonnet-5
DEFAULT_RATE_LIMIT_RPM=60
DEFAULT_RATE_LIMIT_TPM=40000

Implementing the Token Bucket Rate Limiter in Python

Now we write the core logic. We will use a sliding window counter algorithm implemented via a Redis Lua script. This guarantees that checking limits and updating token counts happens atomically, avoiding race conditions when concurrent requests hit your gateway at the exact same millisecond.

Create a file named main.py and paste the following implementation:

import os
import time
import redis.asyncio as redis
from fastapi import FastAPI, Header, HTTPException, Request
from fastapi.responses import JSONResponse
import httpx
from pydantic import BaseModel

app = FastAPI(title='LLM Rate Limit Gateway', version='1.0.0')

REDIS_URL = os.getenv('REDIS_URL', 'redis://localhost:6379/0')
redis_client = redis.from_url(REDIS_URL, decode_responses=True)

# Lua script for atomic sliding window rate limiting
SLIDING_WINDOW_LUA = """
local key = KEYS[1]
local now = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local limit = tonumber(ARGV[3])
local cost = tonumber(ARGV[4])

local clear_before = now - window
redis.call('ZREMRANGEBYSCORE', key, 0, clear_before)

local current_requests = redis.call('ZCARD', key)
if current_requests + cost > limit then
    return 0
end

for i = 1, cost do
    redis.call('ZADD', key, now, now .. ':' .. math.random(100000))
end
redis.call('EXPIRE', key, window)
return 1
"""

class PromptRequest(BaseModel):
    prompt: str
    max_tokens: int = 500

@app.post('/v1/generate')
async def generate_proxy(
    body: PromptRequest,
    x_user_id: str = Header(..., description='Unique client identifier')
):
    current_time = time.time()
    window_size = 60 # 1 minute window
    request_limit = int(os.getenv('DEFAULT_RATE_LIMIT_RPM', '60'))
    
    # Estimate token cost based on input length + requested output
    estimated_tokens = len(body.prompt.split()) + body.max_tokens
    
    # Execute atomic rate check in Redis
    allowed = await redis_client.eval(
        SLIDING_WINDOW_LUA,
        1,
        f'rate:{x_user_id}',
        current_time,
        window_size,
        request_limit,
        1 # Cost in requests
    )
    
    if not allowed:
        return JSONResponse(
            status_code=429,
            content={'error': 'Rate limit exceeded. Slow down your requests.'},
            headers={'Retry-After': '60'}
        )
        
    # Forward to upstream provider (e.g., Anthropic Claude Sonnet 5)
    async with httpx.AsyncClient() as client:
        try:
            response = await client.post(
                'https://api.anthropic.com/v1/messages',
                headers={
                    'x-api-key': os.getenv('UPSTREAM_AI_API_KEY'),
                    'anthropic-version': '2023-06-01',
                    'content-type': 'application/json'
                },
                json={
                    'model': os.getenv('UPSTREAM_MODEL_NAME'),
                    'max_tokens': body.max_tokens,
                    'messages': [{'role': 'user', 'content': body.prompt}]
                },
                timeout=30.0
            )
            response.raise_for_status()
            return response.json()
        except httpx.HTTPStatusError as e:
            raise HTTPException(status_code=e.response.status_code, detail=e.response.text)

Running and Testing the Gateway Locally

With our server code ready, let us boot the application and verify it handles incoming payloads correctly. Run the FastAPI development server via Uvicorn with auto-reload enabled for local iteration.

uvicorn main:app --reload --port 8000

Open a separate terminal window and fire a test curl command to your local gateway. Make sure you pass the required X-User-ID header so the rate limiter has a key to track against in Redis.

curl -X POST http://localhost:8000/v1/generate \
  -H 'Content-Type: application/json' \
  -H 'X-User-ID: dev-user-42' \
  -d '{\"prompt\": \"Explain quantum computing in three sentences.\", \"max_tokens\": 150}'

If your upstream API key is valid, you will receive the JSON response directly from the model provider. If you hammer this endpoint sixty-one times within a single minute using the same user header, the sixty-first request will immediately halt and return a standard 429 Too Many Requests response containing your custom headers.

Troubleshooting Common Rate Limiter Failures

Building distributed rate limiters exposes subtle edge cases in network timing and state synchronization. When your proxy breaks in staging, check these three common failure modes before pushing code to production clusters.

First, watch out for clock drift across your infrastructure. If your Redis cluster nodes or FastAPI worker containers experience drift via broken NTP sync, the time.time() values passed into the Lua script will mismatch, causing your sliding window expiration math to purge keys prematurely or lock out users indefinitely. Ensure your container base images pull strict NTP updates or rely strictly on Redis internal server time using redis.call('TIME') inside your Lua script.

Second, expect connection drops when Redis restarts under high memory pressure. If your Redis instance drops packets or hits maxmemory limits with noeviction policies enabled, your entire proxy will throw uncaught exceptions and fail closed. Wrap your Redis evaluation blocks in try-except clauses with clean fallback behaviors:

try:
    allowed = await redis_client.eval(...)
except redis.RedisError as e:
    # Log the infrastructure failure but allow traffic through or fail open depending on risk profile
    logger.error(f'Redis rate limit check failed: {e}')
    allowed = 1 

Third, guard against missing headers. If a client forgets to include your tracking header (X-User-ID), FastAPI will return a 422 validation error, but if you switch to IP-based tracking as a fallback, remember that corporate NAT setups will lump hundreds of distinct users behind a single shared public IP address, instantly triggering false positive rate limits for innocent clients.

Hardening Your Proxy for Production Deployments

Moving this proxy from your local laptop to an enterprise production cluster requires tightening security, optimizing connection pooling, and adding granular token-bucket accounting. Do not expose your raw upstream API keys inside the container environment variables without restricting them via IP whitelisting on your provider dashboard.

Configure your Redis client with proper connection pool limits to prevent socket exhaustion under heavy load. Use Redis Sentinel or a managed cloud Redis instance with automatic failover configured. Plus, upgrade your Lua script to track actual token counts returned in model response headers (such as anthropic-billed-input-tokens or OpenAI usage payloads) rather than static estimations, ensuring your rate limits accurately reflect real token consumption.

What to Build Next

Once your basic rate-limiting proxy is stable in staging, extend its capabilities to handle enterprise use cases. Implement a Redis-backed token tiering system where free-tier users get 10,000 tokens per day while premium enterprise users get dynamic burst capacity up to 500,000 tokens per minute. You can also hook up Prometheus metrics to export real-time request latency and 429 rejection rates directly to your Grafana dashboards, giving your infrastructure team instant visibility into abusive traffic patterns before they impact your billing cycle.

Should You Implement Custom Rate Limiting?

Build your own rate-limiting proxy if you manage multi-tenant applications with strict internal cost allocations, custom enterprise tiers, or complex multi-provider fallback chains that commercial API gateways cannot route correctly. Skip this custom build and purchase a managed API gateway or use cloud provider native rate limiters if your application only talks to a single LLM provider and your engineering team lacks the bandwidth to maintain distributed Redis state machines. For most mid-scale applications, spending two days writing this FastAPI proxy pays for itself within the first month of prevented runaway API bills.

Final Verdict on Custom LLM Rate Limiting

Build your own rate-limiting proxy if you manage multi-tenant applications with strict internal cost allocations, custom enterprise tiers, or complex multi-provider fallback chains that commercial API gateways cannot route correctly. For most mid-scale applications, spending two days writing this FastAPI proxy pays for itself within the first month of prevented runaway API bills.

Engineering teams running single-provider setups without dedicated bandwidth to maintain distributed Redis state machines should skip this custom build and purchase a managed API gateway instead. Otherwise, deploying this architecture using Python 3.12, FastAPI, and Redis-py successfully prevents costly budget drains like the four hundred dollar weekend bill outlined in this tutorial.

Share this article

Alfian Majid
Alfian Majid/Founder & Editor-in-Chief

Solo developer and blogger from Indonesia. Runs CogitoDaily as a passion project - covering AI news, testing tools, and writing guides. Background in web development and game tech. When not writing about AI, you'll find me deep in anime or gaming.