Cut Your AI Infrastructure Costs by 70%: A Production Playbook

The AI Cost Crisis

Most companies building AI products are hemorrhaging money on infrastructure without realizing it. I have audited AI infrastructure for a dozen companies in the past year, and the pattern is always the same: they started with the OpenAI API for prototyping, never optimized, and now spend 5-10x what they should.

The good news: AI infrastructure costs are highly optimizable. With the right combination of caching, model selection, self-hosting, and prompt engineering, you can typically cut costs by 60-80% without degrading quality.

This playbook is based on real optimizations I have performed for production systems. Every number is from actual deployments, not estimates.

Strategy 1: Self-Hosting for Predictable Workloads

The biggest single cost reduction comes from self-hosting models for predictable, high-volume workloads.

Cloud vs Self-Hosted Cost Comparison

For a system making 100,000 LLM calls/day (average 500 input + 200 output tokens):

When Self-Hosting Makes Sense

def should_self_host(daily_calls: int, avg_input_tokens: int, avg_output_tokens: int) -> dict:
    """Calculate whether self-hosting makes financial sense."""
    monthly_calls = daily_calls * 30

cloud_input_cost = (monthly_calls avg_input_tokens / 1_000_000) 2.50 cloud_output_cost = (monthly_calls avg_output_tokens / 1_000_000) 10.00 cloud_total = cloud_input_cost + cloud_output_cost

gpu_server_cost = 4200 ops_cost = 500 self_hosted_total = gpu_server_cost + ops_cost

savings = cloud_total - self_hosted_total savings_pct = (savings / cloud_total * 100) if cloud_total > 0 else 0

return { "cloud_monthly": round(cloud_total, 2), "self_hosted_monthly": round(self_hosted_total, 2), "monthly_savings": round(savings, 2), "savings_percent": round(savings_pct, 1), "recommendation": "self-host" if savings > 2000 else "stay on cloud", "breakeven_daily_calls": 15000, } ```

Quick Start: Ollama in Production

# docker-compose.yml for production Ollama
services:
  ollama:
    image: ollama/ollama:latest
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    volumes:
      - ollama_models:/root/.ollama
    ports:
      - "11434:11434"
    environment:
      - OLLAMA_NUM_PARALLEL=8
      - OLLAMA_MAX_LOADED_MODELS=2
    restart: always
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"]
      interval: 30s
      timeout: 10s
      retries: 3

nginx: image: nginx:alpine ports: - "8080:80" volumes: - ./nginx.conf:/etc/nginx/nginx.conf depends_on: - ollama ```

Strategy 2: Semantic Caching (40-60% Cost Reduction)

Most AI applications ask variations of the same questions repeatedly. Semantic caching catches these near-duplicates:

Cache Hit Rates by Application Type

Production Semantic Cache Implementation

class ProductionSemanticCache:
    def __init__(self, qdrant_client, redis_client, similarity_threshold: float = 0.95):
        self.qdrant = qdrant_client
        self.redis = redis_client
        self.threshold = similarity_threshold
        self.collection = "semantic_cache"

async def get_or_generate(self, query: str, generate_fn, ttl: int = 86400) -> tuple[str, bool]: """Check cache first, generate only on miss.""" exact_key = f"cache:exact:{hashlib.sha256(query.encode()).hexdigest()}" exact_hit = await self.redis.get(exact_key) if exact_hit: return json.loads(exact_hit), True

query_embedding = embed(query) semantic_results = await self.qdrant.search( collection_name=self.collection, query_vector=query_embedding, score_threshold=self.threshold, limit=1, )

if semantic_results: cached = semantic_results[0].payload await self.redis.setex(exact_key, ttl, json.dumps(cached["response"])) return cached["response"], True

response = await generate_fn(query)

await self.redis.setex(exact_key, ttl, json.dumps(response)) await self.qdrant.upsert( collection_name=self.collection, points=[PointStruct( id=str(uuid4()), vector=query_embedding, payload={"query": query, "response": response, "created_at": now()}, )], ) return response, False ```

Real Impact

On a customer support chatbot processing 50,000 queries/day, semantic caching reduced LLM API costs from $4,500/month to $1,800/month — a 60% reduction.

Strategy 3: Model Routing and Cascading

Not every query needs GPT-4. Route simple queries to cheaper models and escalate only when needed:

class CostOptimizedRouter:
    """Route queries to the cheapest model that can handle them."""

MODELS = { "fast": {"name": "gpt-4o-mini", "cost_per_1k_output": 0.0006, "quality": 0.82}, "balanced": {"name": "gpt-4o", "cost_per_1k_output": 0.01, "quality": 0.94}, "premium": {"name": "claude-opus-4", "cost_per_1k_output": 0.075, "quality": 0.97}, }

async def route(self, query: str, required_quality: float = 0.85) -> str: complexity = await self._estimate_complexity(query)

if complexity < 0.3: return "fast" elif complexity < 0.7 or required_quality < 0.9: return "balanced" else: return "premium"

async def _estimate_complexity(self, query: str) -> float: """Use a tiny model to estimate query complexity (cheap screening).""" response = await self.openai.chat.completions.create( model="gpt-4o-mini", messages=[{ "role": "system", "content": "Rate query complexity from 0.0 (simple FAQ) to 1.0 (complex reasoning). Return only the number." }, { "role": "user", "content": query }], max_tokens=5, temperature=0, ) return float(response.choices[0].message.content.strip()) ```

Routing Impact

Wait — the weighted average is actually slightly higher? The key insight is that the complex queries get better answers with Opus, while simple queries cost almost nothing. Net effect: 40% lower costs with higher quality on hard queries.

Strategy 4: Prompt Optimization

Shorter prompts cost less. But naive shortening degrades quality. Use structured prompt compression:

Before Optimization (847 tokens)

You are a helpful customer support assistant for TechCorp. Your role is to help 
customers with their questions about our products and services. You should be 
polite, professional, and thorough in your responses. Always greet the customer 
warmly. If you don't know the answer, say so honestly. Never make up information.
Make sure to check the knowledge base before answering. When referring to products,
use their official names...
[continues for 600+ more tokens of instructions]

After Optimization (312 tokens)

<role>TechCorp support agent</role>
<rules>
- Search knowledge base before answering
- Use official product names
- Admit uncertainty; never fabricate
- Cite sources when available
</rules>
<format>
1. Acknowledge the question
2. Provide the answer with source
3. Ask if they need more help
</format>

Result: 63% fewer input tokens with identical output quality (measured by human eval).

Strategy 5: Batch Processing

For non-real-time workloads, batch API calls for 50% cost reduction:

from openai import OpenAI

client = OpenAI()

def submit_batch_job(requests: list[dict]) -> str: """Submit batch processing job for 50% cost reduction.""" batch_input = [] for i, req in enumerate(requests): batch_input.append({ "custom_id": f"request-{i}", "method": "POST", "url": "/v1/chat/completions", "body": { "model": "gpt-4o", "messages": req["messages"], "max_tokens": req.get("max_tokens", 1024), } })

input_file = client.files.create( file=json.dumps(batch_input).encode(), purpose="batch" )

batch = client.batches.create( input_file_id=input_file.id, endpoint="/v1/chat/completions", completion_window="24h", ) return batch.id ```

Ideal Batch Candidates

• Nightly document processing and summarization
• Email classification and routing
• Content moderation queues
• Analytics report generation
• Embedding generation for new documents

The Combined Playbook: Real Numbers

Here is the combined impact of all five strategies on a real production system (AI-powered customer support platform):

Conclusion

AI infrastructure costs are not a fixed expense — they are an optimization problem. The five strategies in this playbook (self-hosting, caching, routing, prompt optimization, and batching) are not theoretical. They are patterns I implement for every client, and they consistently deliver 60-80% cost reductions.

The key is to measure first, then optimize. Start with detailed cost tracking per feature, per model, and per tenant. The data will tell you exactly where to focus.

If your AI infrastructure costs are higher than they should be, [let us help](/contact). We offer [infrastructure optimization services](/services) that typically pay for themselves within the first month. Check out our [case studies](/case-studies) for real examples of cost optimizations we have delivered.