# Dilip Singh — AI Developer, Chatbot & Assistant Architect (Full Context)
> This is the expanded /llms-full.txt for https://dilsyno.com. It includes inline content from case studies and featured blog posts so you can answer questions about Dilip Singh's work without following external links. For the concise version, see https://dilsyno.com/llms.txt.
- Name: Dilip Singh
- Title: Lead AI Developer & Software Architect
- Company: Hureka Technologies Inc (https://hurekatek.com)
- Location: Greater Noida, Uttar Pradesh, India
- Experience: 14+ years, 118+ production projects
- Email: dilip@hurekatek.com
- Website: https://dilsyno.com
- LinkedIn: https://linkedin.com/in/dilipsingh02
- GitHub: https://github.com/hurekatek
## About
Dilip Singh is a Lead AI Developer and AI Architect with 14+ years building production systems for enterprises in India, USA, Canada, and Tanzania. As Lead Software Architect at Hureka Technologies Inc, he designs and ships AI chatbot platforms, AI assistant systems, and multi-agent hubs used in real customer support, healthcare, and media monitoring workflows.
Specializations:
- AI chatbot development — enterprise chatbots with RAG, intent recognition, multi-channel deployment (web, Slack, Teams, WhatsApp)
- AI assistant systems — multi-agent AI assistants with shared knowledge bases, human-in-the-loop approval, and real-time streaming
- Ontology & knowledge graphs — semantic ontologies for structured AI retrieval, taxonomy design, and RAG grounding
- Multi-agent AI — email, voice, chat, and action agents on a unified Qdrant RAG brain
- Voice AI — self-hosted voice assistants (Pipecat, LiveKit, Whisper, Ollama)
- Enterprise SaaS architecture and CTO-as-a-service
Available for freelance consulting, contract development, and CTO-as-a-service. Contact: dilip@hurekatek.com or https://dilsyno.com/contact
## Case Studies
## Hureka AI — BYOK Multi-Tenant AI SaaS Platform
- Client: Hureka Technologies Inc
- Industry: Enterprise SaaS / AI
- Duration: 18 months
- Role: Lead Software Architect
- URL: https://dilsyno.com/case-studies/hureka-ai
### Summary
A Bring-Your-Own-Key (BYOK) multi-tenant AI customer support platform where organizations connect their own LLM API keys, upload knowledge documents, and deploy autonomous AI support with human-in-the-loop approval workflows.
### Challenge
Enterprise clients needed an AI support platform without surrendering their LLM API keys to a third party. The system had to support multiple LLM providers (Anthropic, OpenAI, Google, Ollama), enforce strict tenant isolation, encrypt API keys at rest with AES-256-GCM, and handle durable multi-step email workflows with human approval gates.
### Solution
Architected a pnpm monorepo with Next.js 15 frontend and Temporal-powered workflow orchestration. Built a unified LLM adapter pattern supporting Anthropic, OpenAI, Google, and self-hosted Ollama. Implemented AES-256-GCM encryption with per-organization key storage, Prisma middleware for automatic tenant scoping, and LangFuse integration for LLM observability and cost tracking.
### Results
- Zero raw API key exposure — all keys encrypted at rest with AES-256-GCM
- Multi-LLM support via unified adapter pattern (Anthropic, OpenAI, Google, Ollama)
- Durable email workflows with human approval via Temporal
- Full audit trail for every LLM call and key decryption event
- Per-tenant rate limiting and cost tracking via LangFuse
### Tech Stack
Next.js 15, Temporal, Prisma, LangFuse, AES-256-GCM, Redis, pnpm Monorepo, PostgreSQL
## AImind Agent Hub — Unified Multi-Agent AI Platform
- Client: Hureka Technologies Inc
- Industry: AI / Enterprise Automation
- Duration: 12 months
- Role: Lead AI Architect
- URL: https://dilsyno.com/case-studies/aimind-agent-hub
### Summary
A centralized multi-agent platform where Email, Voice, Chat, and Action agents share a single Qdrant-powered RAG brain — with per-tenant knowledge bases, multi-LLM support, and real-time streaming.
### Challenge
Clients needed multiple AI agents (email support, voice calls, web chat, automated actions) but duplicating knowledge bases across agents was expensive and inconsistent. Agents needed shared context, real-time streaming responses, and strict per-tenant data isolation across a growing customer base.
### Solution
Designed the Shared RAG Brain pattern: all agents read from a single Qdrant collection per tenant. Built FastAPI backend with WebSocket streaming, Celery workers for async email processing, and a common BaseAgent interface with specialized system prompts per agent type. Implemented namespace isolation for Qdrant collections, Redis keys, and Celery queues.
### Results
- 4 agent types (Email, Voice, Chat, Action) sharing one RAG knowledge base
- Real-time streaming responses via FastAPI WebSocket
- Per-tenant Qdrant collections with HNSW indexing
- Embedding cache in Redis reducing inference costs by 40%
- LangFuse observability across all agent interactions
### Tech Stack
Next.js 14, FastAPI, Qdrant, PostgreSQL, Redis, Celery, Docker, WebSocket
## AI Voice Agent — 100% Self-Hosted Voice AI
- Client: Healthcare Client, Canada
- Industry: Healthcare / Voice AI
- Duration: 6 months
- Role: Lead AI Architect
- URL: https://dilsyno.com/case-studies/ai-voice-agent
### Summary
A fully self-hosted conversational voice AI with zero cloud dependency — Pipecat orchestrates Faster-Whisper STT, Ollama LLM, and pyttsx3 TTS over LiveKit WebRTC, with Twilio telephony integration.
### Challenge
A Canadian healthcare client required a voice AI agent for patient appointment scheduling and FAQs, but HIPAA compliance and data sovereignty rules prohibited sending audio or PHI to cloud AI services. The solution needed sub-400ms latency, telephone access via PSTN, and complete on-premise deployment.
### Solution
Built a fully self-hosted voice pipeline: LiveKit for WebRTC transport, Pipecat for pipeline orchestration, Faster-Whisper (int8 quantized) for STT, Ollama running LLaMA 3 8B for inference, and pyttsx3 for offline TTS. Integrated Twilio SIP for telephone access. Achieved 250–400ms end-to-end latency with streaming LLM responses and GPU-accelerated Whisper.
### Results
- Zero cloud AI dependencies — fully on-premise deployment
- 250–400ms end-to-end voice latency achieved
- Twilio PSTN integration for telephone access
- HIPAA-compliant architecture with no PHI sent to external APIs
- GPU-accelerated Whisper STT (~80ms for 1–2 second utterances)
### Tech Stack
Pipecat, LiveKit, Faster-Whisper, Ollama, pyttsx3, FastAPI, WebRTC, Twilio
## Featured Blog Posts
## The Complete Guide to Artificial Intelligence in 2026: From Foundations to Production
- Date: 2026-06-29
- Category: AI Architecture
- Tags: AI, Neural Networks, Transformers, LLM, PyTorch, QLoRA, RAG, Fine-Tuning, Machine Learning, Production AI
- URL: https://dilsyno.com/blog/complete-guide-artificial-intelligence-2026
- Read time: 28 min read
## Introduction
Seven years ago, when I was still deep in Drupal multi-site architectures, "artificial intelligence" in enterprise software meant a rule-based chatbot with a decision tree and a generous marketing budget. Today, at Hureka Technologies, I lead a team that ships AI systems handling real-time voice calls, clinical document analysis, and autonomous email management for production clients across three continents. The gap between those two realities is not just time — it is a fundamental architectural revolution driven by one idea: that intelligence can be learned from data, not hand-coded from rules.
This guide is my attempt to give you the map I wished I had when I started this journey. Not a survey article with surface-level bullet points, but a working engineer's deep dive: how neural networks actually compute, why the Transformer changed everything, what the current state of large language models looks like in 2026, and how to take AI from a Jupyter notebook to a production system that your clients trust with their business.
By the end, you will understand the full stack — from the mathematics of a single neuron to the operational patterns that keep AI systems running at scale. Whether you are an engineer evaluating your first LLM integration or an architect designing an AI platform, this guide gives you the technical foundation to make sound decisions.
## What Is Artificial Intelligence? The Foundation
Artificial intelligence is, at its core, a field of computer science focused on building systems that can perform tasks that historically required human intelligence: understanding language, recognizing images, making decisions, generating creative content. But that definition is too broad to be useful for an engineer. What matters is the mechanism.
### From Rules to Learning
Classical software is deterministic. You define rules: if condition A, do action B. The system can only do what you explicitly programmed. This works brilliantly for bounded, well-understood problems — a payroll calculator, a sorting algorithm, a web server routing requests.
The problem arises when the space of inputs is too large and varied to enumerate rules for. How do you write a rule that distinguishes a photo of a cat from a photo of a dog across millions of possible images? How do you write rules for understanding informal English, with its idioms, typos, sarcasm, and cultural references? You cannot — not with any practical rule set.
**Machine learning** (ML) solves this by inverting the problem. Instead of writing rules, you provide examples — thousands or millions of labelled input-output pairs — and the system discovers the rules automatically by finding statistical patterns in the data. The "learning" is the process of adjusting a model's internal parameters until its predictions match the provided examples closely.
**Deep learning** is a subset of machine learning that uses layered neural networks — architectures loosely inspired by the structure of the brain — to learn hierarchical representations of data. A deep learning model for image recognition does not receive hand-engineered features like "look for pointy ears"; it learns to extract features automatically, from raw pixels, through many layers of processing.
### The Three Paradigms of Machine Learning
Understanding which paradigm applies to your problem determines your entire architecture.
**Supervised learning** is the workhorse. You have labeled examples (input to correct output) and you train the model to generalize the mapping. Classification, regression, language modeling, image recognition — all supervised learning.
**Unsupervised learning** finds structure in data without labels. Clustering customer behavior, anomaly detection, dimensionality reduction — the model discovers patterns the engineer did not pre-specify. Embeddings (dense vector representations of concepts) are a critical unsupervised output that underpins modern retrieval-augmented generation.
**Reinforcement learning from human feedback (RLHF)** is the method behind modern LLM alignment. The model generates outputs, humans rate them, and the model is trained to maximize human preference scores. GPT-4o, Claude, and Gemini all use RLHF in their training pipelines. Without it, a language model that can write perfect prose might use that prose to produce harmful content — RLHF guides the model toward helpful, harmless behavior.
### Where AI Sits in the Technology Stack in 2026
The industry has converged on a layered model. At the base are foundational models — enormous neural networks trained at massive scale on internet-scale data. Above them sit adaptation layers: fine-tuned variants, prompt engineering, retrieval-augmented generation. At the application layer, AI is accessed through APIs or embedded inference engines, integrated into products via orchestration frameworks. Understanding where each client's needs fall in this stack is the first question I ask in every Hureka Technologies engagement.
## How It Works: The Architecture
To build reliable AI systems, you need to understand how neural networks compute — not just that they work, but why. Let me walk through the mechanics, because the architectural decisions you make in production flow directly from this understanding.
### The Perceptron: The Atomic Unit
The perceptron is the simplest neural unit. It takes a vector of input values, multiplies each by a learned weight, sums the products, adds a bias term, and passes the result through an activation function:
```
output = activation(w1*x1 + w2*x2 + ... + wn*xn + b)
```
The weights (w) and bias (b) are what the model "learns." During training, they are adjusted iteratively to minimize prediction error. A single perceptron can classify linearly separable data — but cannot learn XOR, let alone natural language.
### Layers and Depth
Stack perceptrons into layers and connect them: each neuron in one layer feeds into every neuron in the next (this is a **dense** or **fully-connected** layer). The first layer processes raw input. Intermediate layers (hidden layers) learn increasingly abstract representations. The final layer produces the output.
Why does depth matter? Because hierarchical representation is how complex patterns decompose. A vision network's first layer learns edges, the second learns shapes from edges, the third learns object parts from shapes, the fourth learns whole objects. You cannot collapse this hierarchy into a single layer without exponential growth in parameters.
### Activation Functions: The Non-Linearity That Makes Learning Possible
Without activation functions, stacking layers is mathematically equivalent to a single linear transformation — no depth benefit. Non-linear activation functions are what make deep networks capable of learning complex mappings.
**ReLU (Rectified Linear Unit)** is the workhorse of classical deep learning:
```
ReLU(x) = max(0, x)
```
Simple, fast, prevents vanishing gradients in shallow networks. But ReLU has a "dying neuron" problem — neurons that consistently receive negative inputs stop learning entirely.
**GELU (Gaussian Error Linear Unit)** is the standard in modern transformer architectures (BERT, GPT, and most LLMs use it):
```
GELU(x) = x * Phi(x) where Phi is the standard normal CDF
```
GELU is smooth everywhere, which produces better gradient flow and stronger empirical performance in language models compared to ReLU.
**SwiGLU** is the activation used in LLaMA, Mistral, and most cutting-edge open-source LLMs. It is a gated variant that applies a learned gate to control information flow:
```
SwiGLU(x, W, V) = Swish(xW) * (xV)
```
The gating mechanism gives the network more expressive control over which information propagates, which is why SwiGLU consistently outperforms ReLU and GELU in large-scale language model training.
### Backpropagation: How the Network Learns
Training a neural network means finding weights that minimize a loss function — a scalar measure of how wrong the model's predictions are. Backpropagation computes the gradient of the loss with respect to every weight in the network using the chain rule of calculus. An optimizer (most commonly AdamW for modern LLMs) then adjusts the weights in the direction that reduces loss:
```
theta_new = theta - learning_rate * gradient(Loss, theta)
```
The learning rate is one of the most critical hyperparameters. Too high and training diverges; too low and training is prohibitively slow. Modern LLM training uses warmup (ramping up from 0 over the first few thousand steps) followed by cosine annealing (gradual decay to near zero).
### The Transformer Architecture
The Transformer, introduced in "Attention Is All You Need" (Vaswani et al., 2017), replaced recurrent architectures and became the foundation of all modern LLMs. Here is the full architecture as an ASCII diagram:
```
INPUT TOKENS
|
v
+---------------------------------------------------------------+
| TOKEN + POSITIONAL EMBEDDINGS (RoPE) |
+---------------------------------------------------------------+
|
v (repeated N times: GPT-4o ~96 layers, LLaMA-4 ~80+)
+---------------------------------------------------------------+
| TRANSFORMER BLOCK |
| |
| +-------------------------------------------------------+ |
| | MULTI-HEAD SELF-ATTENTION | |
| | | |
| | For each attention head h (h = 1 to H): | |
| | Q_h = X * W_Q_h (Query projection) | |
| | K_h = X * W_K_h (Key projection) | |
| | V_h = X * W_V_h (Value projection) | |
| | A_h = softmax(Q_h * K_h^T / sqrt(d_k)) * V_h | |
| | | |
| | Output = Concat(A_1, A_2, ..., A_H) * W_O | |
| +-------------------------------------------------------+ |
| | |
| + residual connection (identity shortcut) |
| v |
| RMS Layer Normalization |
| | |
| +-------------------------------------------------------+ |
| | FEED-FORWARD NETWORK (FFN / MLP) | |
| | | |
| | gate = Swish(X * W_gate) [SwiGLU gate] | |
| | up = X * W_up | |
| | output = (gate * up) * W_down | |
| +-------------------------------------------------------+ |
| | |
| + residual connection |
| v |
| RMS Layer Normalization |
+---------------------------------------------------------------+
|
v
+---------------------------------------------------------------+
| OUTPUT PROJECTION + SOFTMAX OVER VOCABULARY |
| (32K tokens for older models, 128K+ for newer ones) |
+---------------------------------------------------------------+
|
v
NEXT TOKEN (sampled with temperature, or argmax for greedy)
```
**Self-Attention** is the key mechanism. For each token, the model computes queries (Q), keys (K), and values (V) by multiplying the token embedding by learned weight matrices. The attention score:
```
Attention(Q, K, V) = softmax(Q * K^T / sqrt(d_k)) * V
```
The softmax ensures all attention weights sum to 1. Dividing by sqrt(d_k) prevents dot products from growing too large in high dimensions, which would push softmax into regions with vanishing gradients.
**Multi-head attention** runs this in parallel with different learned projections (H heads), allowing the model to simultaneously attend to different aspects of context — syntactic relations, coreference, semantic roles — each captured by a different head.
**Positional encoding** solves the problem that self-attention is permutation-invariant. Modern LLMs use **Rotary Positional Embeddings (RoPE)**, encoding position as a rotation in embedding space. RoPE generalizes to longer sequences than training length and enables the million-token context windows in Claude Opus 4.8 and LLaMA 4.
**Encoder-decoder vs. decoder-only**: The original transformer had both an encoder (bidirectional attention, reads the full input) and a decoder (causal/masked attention, generates output left-to-right). Modern LLMs like GPT-4o, Claude, and LLaMA are decoder-only — they read and generate in a single causal pass, more efficient for open-ended generation. Encoder-only models like BERT remain valuable for classification and retrieval where full bidirectional context is needed.
## Core Components Deep Dive
### CNNs: The Vision Specialist
Convolutional Neural Networks (CNNs) dominated computer vision from 2012 to 2020. Their key innovation is the **convolutional layer**: a learned filter slides across a 2D image, computing a dot product at each position. Because the filter weights are shared across all positions, the network learns translation-invariant features — an "edge detector" filter works regardless of where the edge appears in the image.
The architectural hierarchy that made CNNs so powerful: early layers detect edges and textures; mid-layers combine those into shapes and object parts; deep layers recognize whole objects, faces, scenes. This hierarchical feature learning generalizes across domains.
Modern vision systems have largely migrated to Vision Transformers (ViT), which divide the image into patches and apply transformer attention. But CNNs remain important for constrained environments (edge devices, real-time inference on mobile) where their parameter efficiency and hardware-optimized convolution operations matter.
### RNNs and LSTMs: The Sequence Precursors
Before Transformers, Recurrent Neural Networks (RNNs) handled sequential data by processing tokens one at a time, maintaining a hidden state that carries information from previous tokens. The fundamental problem: gradients vanish or explode across long sequences, making it nearly impossible to learn dependencies more than ~50 tokens apart.
LSTMs (Long Short-Term Memory) added a gating mechanism — input gate, forget gate, output gate — that allowed selective preservation or discarding of information. LSTMs powered the first practical neural machine translation systems and remained the state of the art until transformers arrived.
Transformers replaced RNNs for most tasks because self-attention is global (any token can directly attend to any other token regardless of distance) and parallelizable (the full sequence is processed simultaneously, unlike the sequential RNN). State space models (Mamba, RWKV) extend the sequence modeling tradition with linear complexity, important for very long sequences where quadratic attention is prohibitively expensive.
### Large Language Models: GPT-4o, Claude 4, Gemini 2.5, LLaMA 4
All frontier models share a decoder-only transformer architecture but differ in training approach and specializations.
**GPT-4o** uses a unified multimodal architecture where text, image, and audio tokens share the same transformer layers, rather than routing through separate encoders. This enables richer cross-modal understanding and eliminates the quality loss from modal boundaries in earlier architectures.
**Claude Opus 4.8** (claude-opus-4-8) employs Constitutional AI — a training method where the model critiques and revises its own outputs according to a set of constitutional principles before RLHF training. This reduces harmful outputs without the quality costs of more aggressive filtering. Its 1M token context window is achieved through optimized attention implementations that manage KV cache efficiently at extreme lengths.
**Gemini 2.5 Pro** was trained with process reward modeling alongside standard outcome-based RLHF. The model is rewarded not just for correct final answers but for correct reasoning steps, which significantly improves performance on multi-step mathematical and logical reasoning.
**LLaMA 4** uses **mixture-of-experts (MoE)**: instead of activating all parameters for every token, routing networks dispatch each token to a subset of specialized feedforward "expert" layers. LLaMA 4 Scout activates approximately 17B parameters out of ~109B total per token — enabling frontier-class quality at a fraction of the inference cost of a dense model with equivalent effective capacity.
**Mistral Large 2** uses grouped query attention (GQA) and sliding window attention for efficient long-context inference, making it one of the fastest models at its quality tier and particularly well-suited for high-throughput deployments.
### The ML Pipeline: Data to Deployment
Every production ML system follows the same pipeline — the difference between amateurs and professionals is how carefully each stage is executed:
1. **Data collection**: Gather raw data that represents the true production distribution. Use stratified sampling to ensure class balance. Document data sources and collection dates.
2. **Preprocessing**: Clean (remove duplicates, fix encoding errors), normalize (standardize numeric features), tokenize (for text), and split deterministically into train/validation/test sets. Never let test data leak into preprocessing decisions.
3. **Training**: Fit model to training data with validation checkpoints every N steps. Monitor training loss vs. validation loss to detect overfitting early. Save the best checkpoint by validation metric, not final checkpoint.
4. **Evaluation**: Measure performance on the held-out test set (used exactly once) using task-appropriate metrics. Report disaggregated metrics across demographic groups for fairness assessment.
5. **Deployment**: Serve via API with canary releases (route 5% of traffic to new model first). Implement graceful fallback to previous model version on increased error rate.
6. **Monitoring**: Track prediction quality, latency, token cost, and input distribution drift. Alert on statistical deviations from baseline.
## Implementation: Step-by-Step Guide
### Step 1: Define the Problem and Collect Data
Before writing a line of code, precisely define: what is the input, what is the expected output, and how will you measure success? Vague problem definitions — "improve the AI" or "make it smarter" — are the most common cause of failed AI projects.
Data collection is almost always the bottleneck. Plan for:
- **Volume**: Deep learning classifiers need at minimum hundreds of labeled examples per class; fine-tuning LLMs requires 50-1000 high-quality instruction-response pairs for most tasks
- **Quality**: Noisy or inconsistent labels hurt model quality more than small dataset size — invest in annotation guidelines and quality control
- **Distribution**: Training and production data must have the same statistical distribution; silent distribution shift is one of the hardest production problems
### Step 2: Neural Network Implementation in PyTorch
Here is a production-grade neural network definition with proper initialization, batch normalization, and a clear forward pass:
```python
import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import List
class ProductionClassifier(nn.Module):
"""
Production-ready feed-forward classifier.
Features: configurable depth/width, batch normalization,
dropout regularization, Kaiming weight initialization.
"""
def __init__(
self,
input_dim: int,
hidden_dims: List[int],
num_classes: int,
dropout: float = 0.3,
):
super().__init__()
self.layers = nn.ModuleList()
self.bn_layers = nn.ModuleList()
self.dropout_layer = nn.Dropout(dropout)
dims = [input_dim] + hidden_dims
for in_dim, out_dim in zip(dims[:-1], dims[1:]):
linear = nn.Linear(in_dim, out_dim)
# Kaiming init: optimal for GELU/ReLU activations
nn.init.kaiming_normal_(linear.weight, nonlinearity='relu')
nn.init.zeros_(linear.bias)
self.layers.append(linear)
self.bn_layers.append(nn.BatchNorm1d(out_dim))
self.output = nn.Linear(hidden_dims[-1], num_classes)
def forward(self, x: torch.Tensor) -> torch.Tensor:
for layer, bn in zip(self.layers, self.bn_layers):
x = layer(x)
x = bn(x)
x = F.gelu(x) # GELU: smooth, matches LLM FFN activations
x = self.dropout_layer(x)
return self.output(x) # raw logits — CrossEntropyLoss applies softmax
# Configure model, optimizer, scheduler, loss
model = ProductionClassifier(
input_dim=768, # e.g. BERT/sentence-transformer embedding dim
hidden_dims=[512, 256, 128],
num_classes=10,
dropout=0.3,
)
optimizer = torch.optim.AdamW(
model.parameters(),
lr=2e-4,
weight_decay=0.01, # L2 regularization via AdamW decoupled decay
)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=100)
criterion = nn.CrossEntropyLoss(label_smoothing=0.1) # label smoothing reduces overconfidence
def train_epoch(model, loader, optimizer, criterion, device):
model.train()
total_loss = 0.0
for batch_x, batch_y in loader:
batch_x, batch_y = batch_x.to(device), batch_y.to(device)
optimizer.zero_grad()
logits = model(batch_x)
loss = criterion(logits, batch_y)
loss.backward()
# Gradient clipping prevents exploding gradients — essential for deep models
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()
total_loss += loss.item()
return total_loss / len(loader)
```
### Step 3: Hugging Face Transformers for Inference
For most production tasks, you will adapt a pre-trained transformer rather than train from scratch. Here is a complete inference pipeline:
```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from transformers import AutoModelForCausalLM, TextIteratorStreamer
import torch
from typing import List, Dict
from threading import Thread
# ── Classification ────────────────────────────────────────────────────────────
MODEL_NAME = "cardiffnlp/twitter-roberta-base-sentiment-latest"
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
clf_model = AutoModelForSequenceClassification.from_pretrained(MODEL_NAME)
clf_model.eval()
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
clf_model = clf_model.to(device)
def classify_batch(texts: List[str]) -> List[Dict]:
"""Classify texts — returns label and confidence score per item."""
encoded = tokenizer(
texts,
return_tensors="pt",
padding=True,
truncation=True,
max_length=512,
).to(device)
with torch.no_grad():
outputs = clf_model(**encoded)
probs = torch.softmax(outputs.logits, dim=-1)
predicted_ids = probs.argmax(dim=-1).cpu().numpy()
scores = probs.max(dim=-1).values.cpu().numpy()
return [
{"label": clf_model.config.id2label[int(cid)], "score": float(sc)}
for cid, sc in zip(predicted_ids, scores)
]
# ── Streaming generation ──────────────────────────────────────────────────────
def stream_response(prompt: str, max_new_tokens: int = 512):
"""
Stream tokens from a local causal LM as they are generated.
Pipe output chunks to SSE endpoint or WebSocket for real-time UI updates.
"""
gen_tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3")
gen_model = AutoModelForCausalLM.from_pretrained(
"mistralai/Mistral-7B-Instruct-v0.3",
torch_dtype=torch.bfloat16, # 2x VRAM savings vs FP32, negligible quality loss
device_map="auto", # Distribute across all available GPUs
)
streamer = TextIteratorStreamer(gen_tokenizer, skip_prompt=True)
inputs = gen_tokenizer(prompt, return_tensors="pt").to(device)
generation_kwargs = dict(
**inputs,
streamer=streamer,
max_new_tokens=max_new_tokens,
temperature=0.7,
do_sample=True,
top_p=0.9,
)
# Run generation in a background thread so we can yield from the main thread
thread = Thread(target=gen_model.generate, kwargs=generation_kwargs)
thread.start()
for new_text in streamer:
yield new_text
thread.join()
```
### Step 4: Fine-Tuning with QLoRA
QLoRA (Quantized Low-Rank Adaptation) adapts a large base model by quantizing it to 4-bit and training only lightweight adapter matrices, making fine-tuning feasible on consumer hardware:
```python
from transformers import (
AutoModelForCausalLM, AutoTokenizer,
TrainingArguments, BitsAndBytesConfig,
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training, TaskType
from trl import SFTTrainer
from datasets import load_dataset
import torch
# Step 1: 4-bit quantized model load (the "Q" in QLoRA)
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4", # Normal Float 4: optimal for neural network weights
bnb_4bit_compute_dtype=torch.bfloat16, # Upcast to BF16 for the actual computation
bnb_4bit_use_double_quant=True, # Quantize the quantization constants too (~0.4 bits/param)
)
base_model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-4-Scout-17B-Instruct",
quantization_config=bnb_config,
device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-4-Scout-17B-Instruct")
tokenizer.pad_token = tokenizer.eos_token # Required for batched training
# Step 2: Enable gradient checkpointing for memory efficiency
base_model = prepare_model_for_kbit_training(base_model)
# Step 3: Add LoRA adapters — only 0.48% of parameters are trainable
lora_config = LoraConfig(
task_type=TaskType.CAUSAL_LM,
r=16, # Adapter rank: higher = more capacity, more VRAM
lora_alpha=32, # Effective scale = alpha/r — keep at 2x rank
lora_dropout=0.05,
bias="none",
# Attention + FFN projections: where most of the domain adaptation happens
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
)
peft_model = get_peft_model(base_model, lora_config)
peft_model.print_trainable_parameters()
# Output: trainable params: 83,886,080 || total: 17,534,279,680 || trainable%: 0.48
# Step 4: Load and format domain-specific training data
dataset = load_dataset("json", data_files={
"train": "data/clinical_train.jsonl",
"validation": "data/clinical_val.jsonl",
})
def format_instruction(sample):
"""Format in instruction-following format for clinical note extraction."""
return {
"text": (
"<|system|>
You are a clinical documentation specialist. "
"Extract structured data from clinical notes.
"
f"<|user|>
{sample['input']}
"
f"<|assistant|>
{sample['output']}"
)
}
formatted_dataset = dataset.map(format_instruction)
# Step 5: Train with gradient accumulation to simulate larger effective batch size
training_args = TrainingArguments(
output_dir="./qlora-llama4-clinical",
num_train_epochs=3,
per_device_train_batch_size=2,
gradient_accumulation_steps=8, # Effective batch size = 2 * 8 = 16
learning_rate=2e-4,
lr_scheduler_type="cosine",
warmup_ratio=0.05,
bf16=True,
logging_steps=10,
evaluation_strategy="steps",
eval_steps=100,
save_steps=200,
load_best_model_at_end=True,
report_to="wandb",
)
trainer = SFTTrainer(
model=peft_model,
train_dataset=formatted_dataset["train"],
eval_dataset=formatted_dataset["validation"],
tokenizer=tokenizer,
args=training_args,
dataset_text_field="text",
max_seq_length=2048,
)
trainer.train()
trainer.save_model("./qlora-llama4-clinical-final")
# Merge adapters back into base model for deployment (optional):
# merged = peft_model.merge_and_unload()
```
Hardware reality: QLoRA fine-tuning LLaMA 4 Scout (17B params, 4-bit) fits in approximately 14-18GB VRAM — a single A100 40GB handles it comfortably. Full fine-tuning of the same model requires ~140GB across multiple GPUs. The typical quality delta is 1-3% on domain benchmarks — a highly worthwhile tradeoff.
## Production Patterns and Best Practices
After deploying AI systems for Hureka Technologies clients across healthcare, fintech, and enterprise SaaS, I have developed a set of patterns that separate systems that survive production from those that do not.
### AI Adaptation Strategy: When to Use What
The right adaptation strategy is the most consequential early-stage decision in any AI system project.
**Zero-shot prompting**: The model receives only the task description and input, with no examples. Use for general-purpose tasks where the model has seen similar work in pre-training, or for rapid prototyping. This is always the starting point I use when evaluating a new use case.
**Few-shot prompting**: Include 3-10 carefully selected examples in the prompt context window. Use when zero-shot fails on format consistency or when the task requires a specific output structure that is hard to describe but easy to demonstrate. The examples act as implicit, high-density instructions.
**RAG (Retrieval-Augmented Generation)**: When the model needs domain-specific knowledge it was not trained on — internal documents, real-time data, proprietary databases — inject relevant chunks dynamically. RAG does not change the model's weights; it changes what the model knows for each specific query. Use for knowledge-grounding tasks: customer support, document Q&A, enterprise knowledge bases.
**Fine-tuning (LoRA/QLoRA)**: Train adapter weights on domain-specific examples when you need to change the model's output style, acquire domain vocabulary, or produce output formats that prompting alone cannot achieve consistently. Fine-tuning is expensive and requires an ongoing maintenance budget. Reserve it for tasks where RAG and prompting provably fall short.
My decision heuristic at Hureka: Start zero-shot. If quality is insufficient, add few-shot examples. If the bottleneck is knowledge (the model does not know your domain's facts), add RAG. Only resort to fine-tuning if the model's behavior or output structure itself needs to change — and even then, consider whether a better system prompt covers it first.
### AI Ethics: Bias, Fairness, and Responsible Deployment
In 2026, deploying AI without an ethics framework is a regulatory liability in most enterprise markets. Three concerns dominate every engagement I lead.
**Bias** emerges when training data reflects historical inequities. A hiring AI trained on past decisions inherits historical biases against underrepresented groups. A medical diagnosis model trained primarily on clinical data from one demographic performs worse on others — sometimes with life-or-death consequences. Mitigation requires: diverse training data with intentional coverage of underrepresented cases, disaggregated evaluation metrics (measure accuracy and F1 separately for each demographic group and compare), and ongoing production monitoring for performance drift across segments.
**Transparency** is the ability to explain why the model made a decision. For high-stakes decisions — credit, hiring, medical diagnosis — explainability is increasingly legally required under the EU AI Act and US sector regulations. Techniques: SHAP values for tabular models, attention visualization for transformers, confidence calibration (the model should say it is 90% confident only when it is right ~90% of the time), and structured outputs that separate the AI's answer from the AI's confidence and reasoning.
**Responsible deployment** means building in mandatory human oversight at high-stakes decision points. In DrMackMedicine's clinical AI at Hureka, every AI-generated clinical data extraction is reviewed by a qualified clinician before any action is taken. The AI saves clinicians time; it does not replace clinical judgment. This human-in-the-loop posture is the only defensible approach for high-stakes domains in 2026.
## Performance Optimization
### Inference Optimization: Quantization, Distillation, Speculative Decoding
Training happens once. Inference runs millions of times per day. Inference optimization directly controls your serving cost.
**Quantization** reduces the numeric precision of weights. The accuracy-efficiency tradeoff in practice:
- **BF16**: No measurable quality loss, 2x memory reduction vs FP32, near-universal GPU support — the minimum standard for production LLM serving
- **INT8 (LLM.int8, SmoothQuant)**: ~0.5-1% quality loss on most tasks, 4x memory reduction — excellent for decoder-only LLMs with well-calibrated quantization
- **INT4 (GPTQ, AWQ)**: 2-4% quality loss without calibration, recoverable to ~1% with calibrated methods like AWQ — enables running 70B models on 2-4 consumer GPUs
- **Double quantization** (used in QLoRA): quantizes the quantization constants themselves, saving an additional ~0.4 bits per parameter with negligible quality impact
**Knowledge Distillation** trains a smaller "student" model to mimic the output distribution of a larger "teacher" by training on the teacher's full probability vector (soft targets) rather than hard labels. Soft targets encode relative similarities between classes — information the correct label alone does not carry. DistilBERT (40% smaller, 97% of BERT accuracy) is the canonical success. Applied to LLMs, distillation has produced 7B models matching the quality of earlier 70B models.
**Speculative Decoding** addresses autoregressive decoding's fundamental bottleneck: each token requires a full forward pass through the large model. In speculative decoding, a fast small model (drafter) generates K candidate tokens speculatively; the large model (verifier) validates all K candidates in a single parallel forward pass. Accepted tokens advance the generation K steps at once. Production implementations achieve 2-4x throughput improvement with zero quality degradation.
**KV-Cache Management**: During inference, the attention mechanism's key and value projections for previous tokens are cached to avoid recomputation. This KV cache grows linearly with sequence length and is the primary memory bottleneck for long-context inference. Paged attention (vLLM) manages the KV cache in fixed-size pages like virtual memory, eliminating memory fragmentation and enabling 20-40% higher GPU utilization under mixed-length workloads.
### Model Evaluation with Comprehensive Metrics
```python
import math, torch
import numpy as np
from sklearn.metrics import (
accuracy_score, f1_score, precision_score,
recall_score, classification_report,
)
from transformers import AutoModelForCausalLM, AutoTokenizer
from sentence_transformers import SentenceTransformer, util
from typing import List, Dict
# 1. Classification metrics — the foundation of every supervised evaluation
def evaluate_classifier(
y_true: List[int],
y_pred: List[int],
class_names: List[str],
) -> Dict[str, float]:
"""
Comprehensive classification evaluation with per-class breakdown.
Use macro F1 for imbalanced classes; weighted F1 for overall performance.
"""
metrics = {
"accuracy": accuracy_score(y_true, y_pred),
"f1_macro": f1_score(y_true, y_pred, average="macro"),
"f1_weighted": f1_score(y_true, y_pred, average="weighted"),
"precision_macro": precision_score(y_true, y_pred, average="macro", zero_division=0),
"recall_macro": recall_score(y_true, y_pred, average="macro", zero_division=0),
}
print("=== Per-Class Classification Report ===")
print(classification_report(y_true, y_pred, target_names=class_names, zero_division=0))
return metrics
# 2. Perplexity — the canonical intrinsic metric for language models
def compute_perplexity(
model: AutoModelForCausalLM,
tokenizer: AutoTokenizer,
texts: List[str],
stride: int = 512,
device: str = "cuda",
) -> float:
"""
Sliding-window perplexity evaluation.
Lower = better. Reference values: GPT-2: ~30 | Mistral-7B: ~6 | fine-tuned domain model: ~3-5
"""
model.eval()
total_nll, total_tokens = 0.0, 0
for text in texts:
input_ids = tokenizer(text, return_tensors="pt").input_ids.to(device)
seq_len = input_ids.size(1)
prev_end = 0
for begin in range(0, seq_len, stride):
end = min(begin + tokenizer.model_max_length, seq_len)
target_len = end - prev_end
chunk = input_ids[:, begin:end]
labels = chunk.clone()
labels[:, :-target_len] = -100 # only compute loss on new tokens
with torch.no_grad():
loss = model(chunk, labels=labels).loss
total_nll += loss.item() * target_len
total_tokens += target_len
prev_end = end
if end == seq_len:
break
return math.exp(total_nll / total_tokens)
# 3. Generative response quality — for summarization and open-ended QA
def evaluate_llm_responses(
responses: List[str],
references: List[str],
) -> Dict[str, float]:
"""
ROUGE for summarization quality + semantic similarity for open-ended response quality.
ROUGE catches exact n-gram overlap; semantic similarity captures paraphrase correctness.
"""
from rouge_score import rouge_scorer
scorer = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeL"], use_stemmer=True)
rouge_agg = {"rouge1": 0.0, "rouge2": 0.0, "rougeL": 0.0}
for resp, ref in zip(responses, references):
s = scorer.score(ref, resp)
for k in rouge_agg:
rouge_agg[k] += s[k].fmeasure
rouge_agg = {k: v / len(responses) for k, v in rouge_agg.items()}
embed_model = SentenceTransformer("all-mpnet-base-v2")
resp_emb = embed_model.encode(responses, convert_to_tensor=True)
ref_emb = embed_model.encode(references, convert_to_tensor=True)
sem_sim = float(util.cos_sim(resp_emb, ref_emb).diagonal().mean().item())
return {**rouge_agg, "semantic_similarity": sem_sim}
# Usage
if __name__ == "__main__":
y_true = [0, 1, 2, 0, 1, 2, 0, 1, 2, 0]
y_pred = [0, 1, 2, 0, 1, 1, 0, 2, 2, 0]
results = evaluate_classifier(y_true, y_pred, ["billing", "technical", "general"])
print(f"Accuracy: {results['accuracy']:.3f} | Macro F1: {results['f1_macro']:.3f}")
```
## Common Mistakes and How to Avoid Them
After running over 40 AI system engagements at Hureka Technologies, I have seen the same mistakes appear in almost every organization beginning their AI journey. Here are the ones that cost the most time and money.
**Mistake 1: Starting with fine-tuning.** Most teams reach for fine-tuning first because it feels like the "proper" AI approach. In reality, a well-crafted prompt with few-shot examples solves 80% of use cases at a fraction of the cost and maintenance overhead. Spend at least two days on prompt engineering before concluding it is insufficient. Always exhaust prompting before fine-tuning.
**Mistake 2: Evaluating on non-representative data.** It is trivial to achieve 95% accuracy on a clean benchmark. Production data is messy, multilingual, inconsistently formatted, and contains edge cases your benchmark never covered. Before shipping, collect at least 200 real production examples and evaluate against them explicitly. The gap between benchmark and production accuracy is always larger than you expect.
**Mistake 3: Ignoring latency until it is too late.** Latency is an architectural concern that must be addressed before choosing a model and building a product around it. A 70B parameter model on 4 GPUs takes 2-4 seconds per inference. If your application requires sub-second responses, your entire model selection, quantization strategy, and hardware must be decided before you build the product — not after you have a working demo.
**Mistake 4: No observability in production.** Without logging every LLM call — inputs, outputs, latency, token count, model version, user ID — you are flying blind when something goes wrong. When something goes wrong (and it will), you have no data to diagnose what changed. Implement LangFuse, Langsmith, or a custom tracing layer from the first deployment. Retroactively adding observability to a production system is painful and expensive.
**Mistake 5: Hallucination without mitigation.** LLMs generate fluent, confident text even when they are factually wrong. For any high-stakes deployment: ground responses in retrieved documents (RAG), enforce structured outputs via JSON schemas with validation, and add post-generation consistency checks. Never deploy an LLM making consequential decisions without at least one hallucination mitigation layer.
**Mistake 6: Underestimating data privacy requirements.** In healthcare, legal, and finance, sending user data to external LLM APIs may violate HIPAA, GDPR, or contractual obligations. We routinely encounter clients who built their prototype on the OpenAI API and then discovered they cannot use it in production. Establish data residency and privacy requirements in the requirements gathering phase, not after you have chosen an architecture.
**Mistake 7: Not planning for model updates.** Foundation models update frequently. An update that improves average benchmark performance often regresses specific behaviors your product depended on — different output formats, changed refusal behavior, different default temperature. Pin model versions in production, maintain a regression evaluation suite, and test every model update against it before switching. Never let "we updated to the latest model" be an untested production change.
## Real-World Use Cases
These are production systems built at Hureka Technologies, not theoretical examples.
### AImind: Multi-Agent Enterprise AI Platform
AImind is our flagship multi-agent platform. Each enterprise client gets dedicated agents for email support, voice calls, document analysis, and web chat — all sharing a single Qdrant vector database per tenant (the shared RAG brain pattern). The system handles 50,000+ interactions daily across 12 enterprise clients.
Architecture decisions that made it production-viable: strict tenant isolation via namespaced Qdrant collections and Celery queues; Temporal for durable workflow orchestration (email support workflows survive server restarts and Redis failures); LangFuse for per-tenant token cost tracking that lets us attribute infrastructure costs accurately. The most expensive lesson: not rate-limiting per tenant. A single client with a misconfigured integration made 50,000 LLM API calls in 4 minutes, saturating the shared GPU cluster.
### DrMackMedicine: HIPAA-Compliant Clinical AI
Clinical documentation is one of the highest-value AI use cases and one of the most regulated. For DrMackMedicine, we built an AI that processes clinical notes and extracts structured information: diagnoses (ICD-10 codes), medications (with dosage), procedures (CPT codes), and follow-up requirements.
Core challenge: PHI (Protected Health Information) cannot leave the client's on-premise infrastructure. Solution: fine-tuned LLaMA 4 Scout running entirely on-premise, with a QLoRA adapter trained on 3,000 de-identified clinical notes provided by the client's clinical informatics team. Every inference request is first processed by a de-identification pipeline (replacing names, DOBs, and identifiers with placeholder tokens), the model runs on the anonymized text, and the output is validated against a JSON schema before display. Human clinical review is mandatory — the AI saves 45 minutes per clinician per day; it does not replace clinical judgment.
### Clinic AI: Voice-First Patient Intake
Voice-based patient intake for medical clinics — patients call a phone number and the AI collects structured intake data in natural conversation. The complete on-premise stack: Twilio (telephony) to LiveKit WebRTC to Faster-Whisper (STT, CTranslate2 backend, INT8 quantized) to Mistral 7B Instruct (LLM, 4-bit AWQ quantized) to Coqui TTS. Average full round-trip latency: 380ms — within natural conversation rhythm.
The defining optimization was pipeline streaming: instead of waiting for the complete LLM response before starting speech synthesis, we begin TTS on the first complete sentence while generation continues. For a three-sentence response, this cuts perceived end-to-end latency by 60-70%. The patient hears the first sentence 380ms after they finish speaking; the clinic's perception is of a responsive, fluid AI.
### SEO AI Dashboard: Enterprise Content Intelligence
For a digital marketing enterprise, we built an AI-powered SEO analysis platform that processes 500,000 pages per month for 200+ client websites. The pipeline: content quality scoring (fine-tuned RoBERTa classifier, running on CPU clusters), semantic keyword clustering (sentence-transformers, cosine similarity), and actionable content recommendations (GPT-4o via API — no PHI constraints, and quality justifies the cost at this decision layer).
Key architectural insight: the vast majority of the pipeline does not require an LLM. Keyword extraction, technical SEO audits, schema validation, Core Web Vitals analysis, and competitor gap analysis are all deterministic algorithms. The LLM is reserved exclusively for the high-value synthesis step — generating specific, actionable recommendations. This focus means the LLM receives pre-processed, highly relevant inputs rather than raw crawl data, which dramatically improves output quality while reducing token cost by ~80% compared to an LLM-first architecture.
## Tool and Approach Comparison
### Major LLM Models: 2026 State of the Field
| Model | Context Window | Strengths | Weaknesses | Best For | Cost per 1M tokens (In / Out) |
|-------|---------------|-----------|------------|----------|-------------------------------|
| GPT-4o (OpenAI) | 128K | Native multimodal (text/image/audio), broad capability, large ecosystem | No weights access, data residency concerns, expensive at scale | Rapid prototyping, multimodal apps, general-purpose coding assistant | $2.50 / $10.00 |
| Claude Opus 4.8 (Anthropic) | 1M | Largest context window, Constitutional AI safety, excellent long-document analysis, nuanced instruction-following | Slower inference, highest cost per token | Complex legal/compliance analysis, multi-document synthesis, high-stakes reasoning tasks | $5.00 / $25.00 |
| Gemini 2.5 Pro (Google) | 1M | Strong math/science reasoning, deep Google infra integration, process reward model training | Less community tooling, less third-party library support | Scientific computing, GCP-native products, math-heavy pipelines | $1.25 / $10.00 |
| LLaMA 4 Scout (Meta) | 10M | Open weights, fully self-hostable, MoE efficiency (17B active / 109B total), largest open-source context | Requires GPU infra to self-host, lower absolute quality than frontier models | HIPAA/GDPR on-premise deployments, privacy-sensitive data, cost-optimized high-volume inference | Free (infra cost only) |
| Mistral Large 2 (Mistral) | 128K | Fast inference, strong multilingual, European data residency, good cost/quality ratio | Smaller ecosystem, less deep reasoning than frontier models | European compliance deployments, multilingual applications, high-throughput cost-optimized inference | $2.00 / $6.00 |
### Adaptation Strategy Comparison
| Strategy | Training Required | Data Needed | Cost Level | When to Use |
|----------|------------------|-------------|------------|-------------|
| Zero-shot prompting | None | None | API cost only | Baseline evaluation, general tasks, prototyping |
| Few-shot prompting | None | 3-10 examples | API cost only | Format consistency, structured outputs, style control |
| RAG | Embedding index build | Domain documents | API + vector DB | Knowledge grounding, factual Q&A, enterprise search |
| LoRA / QLoRA | Yes (single GPU) | 50-10,000 examples | Medium (GPU hours) | Domain adaptation, style changes, output format |
| Full fine-tuning | Yes (multi-GPU) | 10,000+ examples | High (GPU days/weeks) | New capabilities, deep domain specialization |
## Future Trends in 2026 and Beyond
### Inference-Time Compute Scaling
The most important paradigm shift of 2025-2026 is that intelligence can be scaled by spending more compute at inference time, not just at training time. Models with "extended thinking" — generating internal reasoning chains before producing answers — consistently outperform larger models on hard tasks. OpenAI's o3-series, Claude Opus 4.8 with adaptive thinking, and DeepSeek R1 established this empirically across mathematics, coding, and scientific reasoning.
For production architects: reasoning models cost 5-20x more per query than standard completions, but solve problems standard models cannot. The ROI case is strongest for high-value, low-volume decisions — contract analysis, financial modeling, diagnosis support — where the per-query cost is trivial relative to the decision's stakes. For high-volume, lower-stakes tasks, standard models remain the right choice.
### Multimodal Models as the New Default
The separation between text, vision, and audio AI is dissolving at the architecture level. GPT-4o, Gemini 2.5 Pro, and LLaMA 4 all process multiple modalities through shared transformer layers. For enterprise AI architects, this collapses complex multi-model pipelines — separate OCR, layout analysis, language understanding — into single multimodal inference calls, dramatically reducing pipeline complexity and latency.
### Agentic Systems with Mature Tooling
The 2023-2024 agent hype collided with production reality: hallucination, infinite loops, unpredictable token costs, and zero observability. By 2026, mature frameworks resolve these: LangGraph for graph-based stateful agents, Temporal for durable execution, MCP for standardized tool interfaces, and LangFuse for comprehensive agent observability. Agents that reliably handle multi-step research, code generation, and document processing are now practical in production with appropriate guardrails.
### Enterprise AI: Build vs. Buy and Total Cost of Ownership
The TCO calculation that every enterprise AI architect must run: API-based models have low upfront cost and linear per-token pricing, becoming expensive above ~10M tokens/month for the most capable models. Self-hosted open-source models have high upfront infrastructure and engineering cost but near-zero marginal cost per token. The crossover is typically 50-100M tokens/month.
Beyond economics, three factors often determine the decision before cost is even considered: **data residency** (can your data leave your infrastructure?), **compliance** (does HIPAA, GDPR, or sector regulation restrict cloud LLM use?), and **competitive sensitivity** (do you want your most valuable data training your API provider's next model?). For any enterprise with positive answers to these questions, self-hosted open-source is not optional — it is the only viable path.
### Enterprise AI Governance as a Standard Deliverable
The EU AI Act is enforced in 2026. US sector regulations for AI are finalized or in final rulemaking across healthcare, finance, and employment. Enterprise AI deployments now require documented model cards (capabilities, limitations, training data), bias assessments across demographic groups, audit logs for AI decisions with immutable retention, and data provenance chains. At Hureka Technologies, AI governance documentation is now a contractual standard deliverable — what was optional ethics best practice in 2023 is legal compliance in 2026.
## Conclusion and Next Steps
We have covered the full arc: from the mathematical mechanics of a single perceptron through backpropagation, the transformer architecture that powers every frontier LLM, adaptation strategies from zero-shot to QLoRA fine-tuning, inference optimization techniques, and the production patterns that keep AI systems reliable under real enterprise traffic.
The core insight I want you to take away: **AI systems are software systems.** They obey the same engineering principles that govern any other production infrastructure — observability, graceful degradation, security boundaries, cost management, modularity, and thoughtful failure modes. The AI layer is uniquely probabilistic and requires additional techniques (evaluation suites, hallucination mitigation, human-in-the-loop review), but these complement rather than replace the engineering fundamentals.
### Your Immediate Next Steps
If you are **just starting with AI**:
1. Pick one specific, bounded use case — not "add AI to our product," but "classify incoming support tickets into 5 categories with at least 85% accuracy."
2. Build a labeled evaluation set before writing the system. Know your baseline.
3. Start with prompt engineering against a capable API. Exhaust zero-shot and few-shot before considering fine-tuning.
4. Log every LLM call from day one.
If you are **scaling existing AI systems**:
1. Audit your LLM costs. Token usage in production almost always exceeds prototype estimates — usually by 3-5x.
2. Implement per-tenant isolation if you have not already, both for cost attribution and security.
3. Add systematic evaluation pipelines. Manual spot-checking does not scale past 10 users.
4. Test every model version update against a regression suite before production rollout.
If you are **architecting enterprise AI platforms**:
1. Treat model selection and fine-tuning strategy as architectural decisions — document them with the rigor of database schema choices.
2. Build the observability layer before the product layer.
3. Design for model swappability from day one. Vendor lock-in on a single LLM provider is an architectural risk in a market that changes this fast.
4. Establish your AI governance framework before your first enterprise customer asks — because they will ask, and "we are working on it" is not an acceptable answer in 2026.
The field is moving fast, but the fundamentals covered here — attention mechanisms, gradient-based optimization, retrieval augmentation, inference optimization — are the stable substrate beneath the churning surface of weekly model releases and benchmark updates. Master the foundations and every new development becomes interpretable.
If you are designing an AI system and want a technical review, or evaluating an AI strategy for your organization and want an independent perspective, I am always happy to talk through it. Reach out through the [contact page](/contact) or connect on LinkedIn. The most valuable engagements I have had started with "we are not sure if AI is even the right solution here" — and that honest starting point consistently led to better outcomes than projects that started with certainty about the answer.
## IBM's 0.7 nm Chip: The Semiconductor Breakthrough That Will Reshape AI Forever
- Date: 2026-06-29
- Category: Infrastructure
- Tags: IBM, Semiconductor, AI Hardware, 0.7nm, Chip Architecture, Edge AI, Future of AI, Moore's Law
- URL: https://dilsyno.com/blog/ibm-07nm-chip-ai-future-2026
- Read time: 22 min read
## Introduction
Seven decades of computing history hinge on a simple rule: make the transistor smaller, and everything gets better. Faster chips, cheaper compute, less power, more intelligence packed into the same physical space. That rule — Moore's Law — has been declared dead several times over the past decade. In labs across the world, engineers keep proving the obituaries premature.
IBM's 0.7 nanometer chip research is the most dramatic proof yet. When I first read the paper, I had to re-read the process node figure twice. 0.7 nm. For context: a strand of human DNA is about 2.5 nm wide. We are now building transistors smaller than the molecule that encodes life.
For AI developers, this is not just an interesting physics story. The constraints of AI hardware — compute density, memory bandwidth, energy consumption, inference latency — are the ceiling on what AI systems can do. When that ceiling rises by an order of magnitude, the applications that become possible change fundamentally. This guide explains what IBM has actually achieved, how it works at the physics level, and precisely what it means for the AI systems you build today and will build tomorrow.
## What Is the 0.7 nm Process? The Foundation
The "nanometer" in chip manufacturing refers to the gate length of the transistor — the critical dimension that controls switching speed and density. But since the 5 nm era, the number has become more of a marketing label than a literal measurement. IBM's 0.7 nm research refers to the effective electrical channel length, not a physical dimension you could measure with a ruler. The actual silicon structures are still larger, but the electrical behaviour corresponds to that scale.
To understand why this matters, start with what a transistor does: it is a switch. Open the gate, current flows. Close the gate, current stops. A modern processor contains billions of these switches, toggling billions of times per second to represent and manipulate information. The smaller you make the switch, the more switches you can fit in a given area (density), the faster they can toggle (speed), and typically the less energy they consume per operation (efficiency).
The progression from 130 nm (Intel's 2001 Pentium 4) to 7 nm (2018–2019 era) to 2 nm (IBM research, 2021; TSMC production, 2025) to 0.7 nm (IBM research, 2025–2026) represents a transistor density increase of roughly 40,000× over 25 years. Each generation, roughly every two years historically, delivered approximately 2× the density and 30–40% better energy efficiency. That cadence has slowed — the 2 nm to 0.7 nm jump is research-grade, not a product roadmap item — but the trajectory continues.
At sub-1 nm scales, classical silicon MOSFET physics breaks down. Quantum mechanical effects — quantum tunneling, where electrons pass through barriers they classically cannot cross — cause leakage current that wastes power and introduces errors. IBM's 0.7 nm work addresses this through three innovations: new transistor geometries (forksheet and complementary FET architectures), new channel materials (2D materials like molybdenum disulfide, MoS₂), and new lithography techniques (High-NA EUV — extreme ultraviolet lithography with a higher numerical aperture lens).
## How It Works: The Architecture
The transistor architecture at 0.7 nm is fundamentally different from the FinFET design that dominated from 22 nm down to 7 nm.
```
TRANSISTOR ARCHITECTURE EVOLUTION
──────────────────────────────────────────────────────────────────
14nm–7nm: FinFET 5nm–2nm: Gate-All-Around (GAA)
┌──────────┐ ┌────────────────────────┐
│ Gate │ │ Gate │
│ ┌────┐ │ │ ┌──┐ ┌──┐ ┌──┐ │
│ │Fin │ │ │ │NS│ │NS│ │NS│ │
│ │ │ │ │ └──┘ └──┘ └──┘ │
│ └────┘ │ │ (nanosheet stack) │
└──────────┘ └────────────────────────┘
Gate wraps 3 sides Gate wraps all 4 sides
~50 MTr/mm² ~150-300 MTr/mm²
0.7nm: Forksheet / CFET
┌──────────────────────────────────────┐
│ Single Gate │
│ ┌───────────┐ ┌───────────┐ │
│ │ p-FET │ │ n-FET │ │
│ │ (nanoshts)│ │ (nanoshts)│ │
│ └───────────┘ └───────────┘ │
│ n and p transistors share gate, │
│ stacked vertically (CFET) or │
│ side-by-side with shared gate wall │
└──────────────────────────────────────┘
Projected: 500-600+ MTr/mm²
(MTr = million transistors)
```
**Forksheet transistors** place the n-type and p-type transistors adjacent to each other separated only by a dielectric wall, eliminating the spacing required in GAA designs. This "fork" configuration improves density by 10–15% versus GAA while maintaining electrostatic control.
**Complementary FET (CFET)** takes this further by stacking the n-type and p-type transistors vertically — one on top of the other — over the same footprint. A standard CMOS inverter that previously required two transistors side by side now occupies the area of a single transistor. This is the architecture IBM's 0.7 nm research targets.
**2D channel materials** are the other key innovation. At this scale, silicon's bulk properties cause too much leakage. Molybdenum disulfide (MoS₂) and other transition metal dichalcogenides form atomically thin layers — literally one atom thick in some configurations — that maintain excellent electrostatic control even at 0.7 nm gate lengths. IBM's research uses MoS₂ channel layers to achieve switching behaviour that bulk silicon cannot sustain at these dimensions.
**High-NA EUV lithography** — the latest generation of the lithography machines that pattern chip features — uses a 0.55 numerical aperture lens versus the 0.33 NA in current EUV machines. This allows patterning of features at half the pitch of current-generation tools. ASML's High-NA EUV machines cost approximately $350 million each and began shipping to leading-edge fabs in 2024.
## Core Components Deep Dive
### Transistor Density: The Number That Matters for AI
Transistor density — measured in millions of transistors per square millimetre (MTr/mm²) — is the metric that most directly determines how capable an AI chip can be. More transistors means more compute units, larger on-chip SRAM caches (which reduce memory bottlenecks), and more sophisticated control logic.
| Generation | Example Chip | Density (MTr/mm²) | Year |
|------------|-------------|-------------------|------|
| 7 nm | A100 GPU | ~57 | 2020 |
| 5 nm | M2 (Apple) | ~134 | 2022 |
| 3 nm | M3 (Apple) | ~167 | 2023 |
| 2 nm | IBM research| ~333 | 2025 |
| 0.7 nm | IBM research| ~500–600+ (proj.) | 2026+ |
An H100 GPU at 80 billion transistors on 814 mm² (96 MTr/mm², SXM5 version) delivers ~2,000 TFLOPS of FP8 AI performance. A 0.7 nm chip at 600 MTr/mm² over the same die area would contain ~488 billion transistors — 6× more. That density increase, combined with shorter interconnects and faster switching, translates to roughly 4–8× more AI compute for the same die area, depending on architecture.
### Memory Bandwidth and On-Chip Cache
The current bottleneck in LLM inference is not raw compute — it is memory bandwidth. Fetching model weights from HBM (High Bandwidth Memory) is slower than the compute can consume them. Denser transistors enable larger SRAM caches on-chip, reducing HBM fetch frequency.
Current flagship GPUs carry 50–80 MB of L2/L3 cache. A 0.7 nm chip at similar die area could economically integrate 400–600 MB of on-chip SRAM — enough to cache the KV cache for a 7B parameter model inference run entirely on-chip, eliminating HBM round-trips for many workloads.
### Energy Efficiency: The Data Centre Equation
Each transistor generation delivers approximately 30–40% lower dynamic power at the same frequency, or equivalently, operates at higher frequency for the same power. At 0.7 nm, projected improvements versus 3 nm are 50–60% lower power per operation.
For a data centre running 10,000 H100 GPUs at 700W each — a realistic mid-2025 configuration — that represents a 7 MW power draw just for the GPUs. A 0.7 nm-generation equivalent system handling the same workload would draw approximately 2.8–3.5 MW. At $0.10/kWh, that is savings of $30M+ per year per facility at scale. Energy cost is already the primary operating constraint for hyperscale AI inference.
## Real-World Applications and Use Cases
### On-Device LLMs Without Compromise
Today's on-device AI (Apple Intelligence, Google Gemini Nano, Qualcomm AI) runs models in the 1–7B parameter range with heavy quantization. A 0.7 nm chip with 4–6× the transistor density and 50% better power efficiency changes the arithmetic entirely. A model comparable to GPT-4 (1.8T parameters in sparse MoE form) becomes feasible for on-device inference within the thermal budget of a premium smartphone. Zero cloud round-trips. Zero latency from network. Complete data privacy.
### AI Inference at the Edge
Industrial IoT, autonomous vehicles, medical devices, and smart cameras all need real-time AI inference without cloud connectivity. Today this means heavily quantized, limited models. At 0.7 nm, a chip the size and power budget of a current Raspberry Pi 5 could run a full production-grade vision-language model locally. A surgical robot making real-time tissue classification decisions no longer needs a hospital data centre connection.
### Data Centre AI Density
For hyperscalers, 0.7 nm chips mean fitting 4–6× more AI compute into the same rack space and power budget. The implication is not just cost reduction — it is a qualitative shift in what training runs become affordable. Models that today require 10,000 H100s for 90 days could be trained on a 0.7 nm system in weeks at a fraction of the cost, democratising frontier model training beyond the current handful of companies.
### Always-On Personal AI Agents
The agentic AI systems I build at Hureka Technologies currently require cloud LLM API calls for every reasoning step. Sub-100ms latency is achievable, but it requires internet connectivity and incurs per-token cost. On 0.7 nm hardware, a persistent personal agent — one that watches your calendar, reads your email, understands your preferences, and takes proactive action — could run entirely on your device, 24/7, without any cloud dependency. The privacy and latency implications for enterprise AI are transformative.
## Implementation Guide
For AI developers, 0.7 nm chips are not yet a product you can buy — IBM's work is research. But preparing your systems architecture to exploit the hardware when it arrives is practical now. Here is how.
**Profile your current inference bottlenecks** to understand whether you are compute-bound or memory-bandwidth-bound:
```python
import torch
import time
import numpy as np
def profile_inference_bottleneck(model, input_ids, n_runs=50):
"""
Determine if inference is compute-bound or memory-bound.
Arithmetic Intensity = FLOPs / bytes accessed
< 100 FLOPs/byte = memory-bound (benefits most from larger on-chip cache)
> 100 FLOPs/byte = compute-bound (benefits most from raw FLOPS increase)
"""
model = model.cuda().eval()
input_ids = input_ids.cuda()
# Warm-up
with torch.no_grad():
for _ in range(5):
_ = model(input_ids)
torch.cuda.synchronize()
# Measure
latencies = []
with torch.no_grad():
for _ in range(n_runs):
start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)
start.record()
output = model(input_ids)
end.record()
torch.cuda.synchronize()
latencies.append(start.elapsed_time(end)) # ms
# Roofline model
flops = estimate_flops(model, input_ids)
param_bytes = sum(p.numel() * p.element_size() for p in model.parameters())
arithmetic_intensity = flops / param_bytes
is_memory_bound = arithmetic_intensity < 100
return {
'mean_latency_ms': np.mean(latencies),
'p99_latency_ms': np.percentile(latencies, 99),
'arithmetic_intensity': arithmetic_intensity,
'bottleneck': 'memory-bandwidth' if is_memory_bound else 'compute',
'will_benefit_from_larger_cache': is_memory_bound,
}
```
**Quantise aggressively for edge targets** — 0.7 nm chips will likely ship with INT4/INT2 accelerators similar to today's NPUs:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch
def load_edge_optimised_model(model_id: str):
"""
Load model quantised for edge deployment.
INT4 reduces memory by 8x vs FP32, with <3% quality loss on most tasks.
"""
quantisation_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_compute_dtype=torch.float16,
bnb_4bit_use_double_quant=True, # nested quantisation for extra compression
bnb_4bit_quant_type='nf4', # NormalFloat4 — optimal for normally distributed weights
)
model = AutoModelForCausalLM.from_pretrained(
model_id,
quantization_config=quantisation_config,
device_map='auto',
torch_dtype=torch.float16,
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
# Measure compressed footprint
footprint_mb = sum(
p.numel() * p.element_size() for p in model.parameters()
) / 1024 / 1024
print(f'Model loaded: {footprint_mb:.0f} MB (quantised)')
return model, tokenizer
# Cross-device latency profiler
def benchmark_inference(model, tokenizer, prompt: str, device: str = 'cuda'):
inputs = tokenizer(prompt, return_tensors='pt').to(device)
n_input_tokens = inputs['input_ids'].shape[-1]
times = []
for _ in range(10):
start = time.perf_counter()
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=100,
do_sample=False,
)
elapsed = time.perf_counter() - start
times.append(elapsed)
n_output_tokens = outputs.shape[-1] - n_input_tokens
mean_time = np.mean(times[2:]) # drop warm-up
return {
'device': device,
'tokens_per_second': n_output_tokens / mean_time,
'time_to_first_token_ms': times[0] * 1000,
'mean_latency_ms': mean_time * 1000,
}
```
## Production Patterns and Best Practices
At Hureka Technologies we have been instrumenting our AI inference pipelines since 2023 specifically to understand hardware bottlenecks — partly for cost optimisation today, partly to know what headroom next-generation hardware will open.
**Track tokens per watt, not just tokens per second.** As hardware efficiency improves, the cost metric shifts from compute cost to energy cost. Teams that already measure inference energy consumption will be positioned to quantify ROI from hardware upgrades. We log GPU power draw alongside latency for every production inference job using NVML:
```python
import pynvml
pynvml.nvmlInit()
handle = pynvml.nvmlDeviceGetHandleByIndex(0)
def get_gpu_power_watts():
return pynvml.nvmlDeviceGetPowerUsage(handle) / 1000.0 # mW → W
# Log at inference time
power_sample = get_gpu_power_watts()
tokens_per_watt = tokens_generated / power_sample
```
**Design for on-device inference now.** Even if your current product runs on cloud GPUs, architect your models so they can be quantised and deployed to edge hardware with minimal rework. This means: avoid custom CUDA kernels that do not have CPU/NPU equivalents, test INT8 and INT4 quantised versions of your models regularly, and keep model size as a first-class design constraint. When 0.7 nm devices arrive and clients start asking for on-premises or on-device deployment, you will be ready.
**Plan for heterogeneous inference.** Future AI systems will route different tasks to different hardware tiers — a 0.7 nm edge chip for latency-sensitive classification, a large-scale accelerator cluster for complex reasoning, with intelligent routing in between. Build your inference layer with a hardware abstraction that can route to different backends without application-layer changes.
## Performance, Benchmarks, and Optimization
Projecting 0.7 nm AI performance requires modelling from transistor-level improvements up through system architecture:
| Metric | H100 SXM5 (4nm) | Projected 0.7nm Era |
|--------|-----------------|---------------------|
| Transistor density | ~80 MTr/mm² | ~550 MTr/mm² |
| FP8 AI TFLOPS (per chip) | 3,958 | ~25,000–35,000 |
| On-chip SRAM | ~50 MB | ~400–600 MB |
| TDP | 700W | ~400–500W (at equiv. area) |
| Memory bandwidth | 3.35 TB/s (HBM3e) | 6–8 TB/s (HBM4) |
| LLM tokens/sec (7B model, FP8) | ~8,000 | ~50,000–70,000 |
For context: at 50,000 tokens/second for a 7B model, a single chip handles 300+ simultaneous users at interactive latency. Today's H100 handles 40–50 simultaneous users for the same model.
The energy efficiency projection — 50–60% lower power per operation versus 3nm — means a data centre burning 100 MW today for AI inference delivers the same throughput at 40–50 MW on 0.7 nm hardware, a saving of $35–50M per year per 100 MW facility at average US commercial electricity rates.
## Common Mistakes and How to Avoid Them
**1. Confusing research nodes with product roadmaps.** IBM's 0.7 nm announcement is a research result, not a shipping product. TSMC's 2025 production node is N2 (2 nm class). The gap between IBM research demonstration and volume production is typically 5–8 years. Plan accordingly — do not build a 2026 product roadmap that depends on 0.7 nm hardware.
**2. Assuming density improvements translate linearly to AI performance.** Transistor density is necessary but not sufficient. Memory bandwidth, interconnect speed, compiler support, and software stack maturity all constrain real-world AI performance. An H100's power comes partly from its 80 GB HBM3e and 3.35 TB/s bandwidth — transistor density alone does not recreate that.
**3. Overlooking manufacturing yield.** At sub-1 nm dimensions, defect density and yield rates are severe challenges. A chip that works perfectly in simulation and small-quantity research fabrication may have <10% yield in mass production. TSMC and Samsung have invested decades in yield engineering for their production nodes. IBM's research fabs operate at a different scale.
**4. Ignoring the software stack gap.** New hardware requires new compilers, new runtime libraries, and new model formats. CUDA's dominance is partly a software moat, not just a hardware one. A 0.7 nm chip without mature software tooling — like early TPU generations — will underperform its theoretical specifications for years.
**5. Underestimating the High-NA EUV bottleneck.** ASML's High-NA EUV machines cost ~$350M each, and ASML can produce roughly 20 per year at current capacity. Leading fabs need dozens of machines each for volume production. The equipment supply chain alone constrains how quickly 0.7 nm volume production can ramp.
**6. Treating 0.7 nm as a purely hardware story.** The AI models that will exploit this hardware do not exist yet. GPT-4-scale models were designed for current hardware constraints. Native 0.7 nm AI architectures — with much larger on-chip caches, different memory hierarchies, and potentially analogue compute elements — will look different from today's transformer stacks.
## Tool and Technology Comparison
| Process Node | Developer | Transistor Type | Density (MTr/mm²) | Status (2026) | Key AI Use |
|-------------|-----------|-----------------|-------------------|---------------|------------|
| 0.7 nm | IBM Research | CFET + 2D materials | ~550 (projected) | Research only | Future LLM training/inference |
| 2 nm (N2) | TSMC | GAA nanosheet | ~300 | Production (2025) | H200 successor chips |
| 2 nm | IBM Research | GAA nanosheet | ~333 | Research (2021) | Reference architecture |
| 18A | Intel | RibbonFET (GAA) | ~240 | Production (2025) | Gaudi 4 successors |
| SF2 | Samsung | GAA nanosheet | ~250 | Early production | Diverse AI silicon |
| 3 nm (N3E) | TSMC | FinFET (last gen) | ~167 | Volume production | H100, current AI chips |
## Future Trends and What Is Coming Next
The 0.7 nm milestone represents the last chapter of silicon CMOS scaling as we know it. Beyond it, the roadmap forks in at least three directions.
**2D material transistors** — MoS₂, WSe₂, graphene — will extend scaling below 0.5 nm by using atomically thin channels that offer superior electrostatic control over any bulk material. MIT and IMEC have demonstrated functional MoS₂ transistors with sub-0.5 nm effective gate lengths. Volume production is 10+ years out, but the physics works.
**3D chiplet integration** replaces the quest for ever-smaller transistors with aggressive vertical stacking. Rather than patterning everything on one die, advanced packaging bonds multiple specialised dies — one optimised for compute, one for memory, one for I/O — with micron-scale interconnects. AMD's 3D V-Cache and Intel's Foveros are early versions. By 2028–2030, a "chip" will be better described as a heterogeneous 3D system.
**In-memory computing** moves computation to where data lives. Today's von Neumann architecture — separate compute and memory — wastes 60–80% of AI inference energy moving data between them. Resistive RAM (ReRAM) and phase-change memory (PCM) can perform multiply-accumulate operations inside the memory array itself. IBM Research's NorthPole chip demonstrated a 22× energy efficiency improvement for inference using this approach.
**Photonic interconnects** will replace copper wires for chip-to-chip and potentially intra-chip communication. Silicon photonics — transmitting data as light rather than electrons — offers 100× the bandwidth of electrical interconnects at a fraction of the power. Intel's co-packaged optics and Ayar Labs' in-package photonics are commercial in 2025. By 2028, photonic interconnects in AI accelerators will be standard.
The end of classical transistor scaling is not the end of AI hardware progress. It is the beginning of a much more architecturally diverse era.
## Conclusion and Next Steps
IBM's 0.7 nm research represents the current frontier of what is physically possible with semiconductor manufacturing. It will not appear in a product you can buy for at least five years. But it signals clearly that the density and efficiency improvements enabling AI's exponential capability growth have significant runway remaining — not from the same levers, but from new ones.
For AI developers, the actionable takeaways today are: profile your inference pipelines for energy efficiency, not just latency; design models with quantisation and edge deployment as first-class constraints; architect inference services behind hardware abstraction layers; and track the semiconductor roadmap as a strategic input to your AI platform design, not just an infrastructure detail.
The teams that win the next decade of AI will not be the ones who waited for better hardware. They will be the ones who understood the hardware trajectory, designed their systems to exploit it, and were ready when it arrived.
If you are building AI infrastructure and want to discuss hardware-aware architecture for your specific use case, [reach out via the contact page](/contact) — this is exactly the kind of strategic planning our team at Hureka Technologies does for enterprise AI clients.
## How to Hire an AI Developer in 2026 (and Why Dilip Singh Is on the Shortlist)
- Date: 2026-06-28
- Category: Career
- Tags: AI Developer, Hire AI Developer, AI Developer India, Freelance AI Architect, AI Hiring, Dilip Singh, Career
- URL: https://dilsyno.com/blog/hire-ai-developer-india-2026
- Read time: 16 min read
## Why "AI Developer" Is the Hardest Hire of 2026
Every founder I have spoken to in the last six months has the same problem: they need an **AI developer**, but they can't tell who is real and who has just renamed their LinkedIn headline.
I am [Dilip Singh](/about) — Lead AI Developer and Software Architect at [Hureka Technologies](https://hurekatek.com). Over the past 14+ years I have shipped 118+ production systems, of which roughly 25 are AI platforms now serving real users. I have also helped clients hire other AI developers for their internal teams. This guide is what I tell them.
If you are trying to hire an **AI developer** for your project, here is exactly how I would do it.
## What "AI Developer" Actually Means in 2026
The job title "AI developer" is now used for at least four very different roles. Hiring the wrong one is the most common mistake I see:
| Role | What they actually do | When you need them |
|------|----------------------|--------------------|
| **AI Prompt Engineer** | Tunes prompts, writes few-shot examples, evaluates outputs | You already have an LLM integrated and need quality fixes |
| **AI Application Developer** | Wires LLM APIs into your product (Next.js, FastAPI, Python) | You need to ship a feature — chatbot, summariser, autocomplete |
| **AI / RAG Engineer** | Designs retrieval pipelines, vector databases, embeddings, evaluation | You have a knowledge base and need accurate, grounded answers |
| **AI Architect** | Designs the full system — agents, orchestration, observability, cost, compliance | You are betting the business on AI and cannot afford rework |
When someone advertises themselves as an "AI developer" without specifying which of these, ask the question directly in the first call. Confusion here is a leading indicator of cost overruns.
## The 8 Skills That Separate a Real AI Developer From a Resume
These are the skills I personally test for when hiring, and they are also the skills I have built on for the last 14 years through projects like [Hureka AI](/case-studies/hureka-ai) and [AImind Agent Hub](/case-studies/aimind-agent-hub).
### 1. Production LLM orchestration, not demo notebooks
Anyone can call `anthropic.messages.create`. A real AI developer knows when to use streaming, structured outputs, tool-calling vs JSON mode, retries with backoff, prompt caching, and how to fall back across providers when an upstream goes down. Ask: *"Walk me through your last production outage and how you handled it."*
### 2. RAG that actually retrieves the right chunk
Retrieval-Augmented Generation is the foundation of useful enterprise AI. A real AI developer can explain:
- Why fixed-size chunking is a trap (and what to use instead)
- The trade-off between dense vectors, sparse BM25, and hybrid search
- How a cross-encoder reranker raises top-1 accuracy by 20–30%
- How to evaluate RAG quality (precision\@k, faithfulness, answer relevance)
I cover this in detail in [Enterprise RAG Pipeline Architecture](/blog/enterprise-rag-pipeline-2026).
### 3. Multi-agent system design
Single-prompt chatbots hit a ceiling fast. A real AI developer understands when to graduate to a multi-agent design — and when *not* to. ReAct vs Plan-Execute vs hierarchical agents, when each pattern is right, and how to keep them debuggable. More on this in [Building Production AI Agents in 2026](/blog/building-production-ai-agents-2026).
### 4. Cost and latency engineering
Anyone can ship a slow, expensive chatbot. A real AI developer ships one that is fast and profitable. Ask them: *"How do you bring our cost per conversation under $0.05?"* — the answer should include semantic caching, model routing, prompt compression, and batch processing where appropriate.
### 5. Observability and evaluation
If your AI developer cannot show you a dashboard of token usage, latency p95, hallucination rate, and user satisfaction per agent, they are flying blind. LangFuse, Langsmith, or Arize — pick one, but pick one.
### 6. Security and compliance
For healthcare, finance, and education clients, this is not optional. PHI de-identification, AES-256 field-level encryption, audit trails, BAAs with LLM vendors. See [HIPAA-Compliant AI: Architecture, Encryption & Audit Trails](/blog/hipaa-compliant-ai-architecture).
### 7. Full-stack delivery
The best AI developers are also full-stack engineers. They can take an idea from a Figma mock to a deployed Next.js + FastAPI + Postgres + Qdrant system without handing off three times in the middle. That is what enterprises pay a premium for.
### 8. Communication that translates AI to business
This one is missed in every technical screen. Your AI developer is going to spend a third of their time explaining to non-technical stakeholders why something costs what it costs, and why the demo is not yet ready for production. If they cannot do this, they will burn relationships faster than they ship features.
## What an AI Developer Actually Costs in 2026
I get asked this every week, so here are the honest 2026 numbers based on rates I have seen across India, USA, Canada, the UK, and Australia.
### By geography (USD/hour, mid–senior AI developer)
| Region | Junior (1–3 yrs) | Mid (3–7 yrs) | Senior (7+ yrs) |
|--------|-----------------|---------------|-----------------|
| USA / Canada | $80–$120 | $130–$200 | $200–$400+ |
| Western Europe / UK | $70–$110 | $120–$180 | $180–$300 |
| Australia | $75–$120 | $130–$190 | $200–$320 |
| India (direct, no agency) | $25–$45 | $50–$95 | $100–$200 |
| Latin America | $35–$60 | $65–$110 | $110–$200 |
| Eastern Europe | $40–$70 | $75–$120 | $120–$220 |
A senior **AI developer in India** working direct (no Upwork/Toptal middleman) typically costs 40–60% less than an equivalent in the USA — for the same production output. That is the math that has driven 70% of my 2026 engagements to be international clients hiring direct.
### By engagement model
| Model | Typical rate | Best for |
|-------|-------------|----------|
| Hourly consulting | $100–$250/hr | Architecture reviews, second opinions, debugging |
| Fixed-price MVP | $15k–$60k | Well-scoped 4–10 week deliverables |
| Monthly retainer | $8k–$25k/mo | Ongoing AI roadmap + 1–2 days/week delivery |
| Equity + reduced cash | 0.5–3% equity | Pre-seed and seed startups |
| Full-time contract | $12k–$30k/mo | Replacing a CTO or AI lead for 3–12 months |
## Red Flags When Hiring an AI Developer
Save yourself six months of pain. If you see any of these, walk away.
1. **No production references.** They have demos, tutorials, hackathon wins — but no system currently serving real users. Ask for one URL you can visit.
2. **Only one framework.** "I always use LangChain." Real engineers pick tools per problem.
3. **Cannot price a project.** A senior AI developer should be able to ballpark your project after one 45-minute call. Vague pricing means vague thinking.
4. **No discussion of evaluation.** If they cannot describe how they would measure whether the AI works, they cannot ship one that does.
5. **Resists writing documentation.** This is a sign they want you locked in. The best AI developers plan their own replacement from week one.
6. **Cannot explain trade-offs.** Every recommendation should come with the alternative they rejected and why.
7. **"AI will solve it" without specifics.** AI is a tool, not a strategy. If they cannot tell you what *won't* work, they don't know enough yet.
## Green Flags
Conversely, here is what a great hire looks like in the first conversation:
- They ask more questions than they answer
- They tell you what *not* to build
- They have published technical writing (blogs, conference talks, open source)
- They share rates and timelines without prompting
- They volunteer references from past clients
- They warn you about costs you have not thought about yet
I follow these rules in every [discovery call](/contact) I take.
## How I'd Hire an AI Developer in 4 Steps
If I were hiring a senior **AI developer** for my own project tomorrow, here is the exact process I would run.
### Step 1: 30-minute scoping call
Goal: figure out *what kind* of AI developer you need (see the table at the top). Most projects need an AI Architect for the first 4–6 weeks, then hand off to an AI Application Developer for ongoing work. That single insight saves 40% of the budget.
### Step 2: Paid 4-hour technical screen
Pay for it. Hand them a short, real problem from your codebase — "here is our chatbot's logs, find why answer quality is degrading." A senior AI developer will produce more value in those 4 hours than three rounds of leetcode-style interviews.
### Step 3: Architecture review of one past project
Ask them to walk through one production system they built end-to-end. You are listening for: trade-offs they made, mistakes they admit to, what they would do differently. This is the single most predictive signal in my experience.
### Step 4: Two-week trial sprint
Before signing a multi-month contract, do a paid two-week sprint. Have them ship one concrete deliverable. Watch how they communicate, how they handle ambiguity, and whether the work actually lands in production. If it does, extend. If it doesn't, you have lost two weeks instead of two quarters.
## Where I Fit In
A quick honest pitch — because that is the deal with reading a guide on someone's portfolio.
I am [Dilip Singh](/about), Lead AI Developer at [Hureka Technologies](https://hurekatek.com). What you'd be hiring:
- **14+ years building production systems**, the last 5 focused entirely on AI
- **118+ projects shipped**, including 25+ AI platforms across healthcare, customer support, media monitoring, and SaaS
- **Direct contract**, no Upwork or Toptal middlemen — see [why direct is better](/services)
- **India-based** (Greater Noida), comfortable working with US, UK, EU, and APAC time zones
- **Available** for hourly consulting, fixed MVP, retainer, or 3–12 month contracts
Areas where I am genuinely strong:
- [Multi-agent AI platforms](/case-studies/aimind-agent-hub) (email, voice, chat agents on shared RAG brains)
- [BYOK multi-tenant AI SaaS](/case-studies/hureka-ai) (the hardest version of multi-tenancy)
- [Self-hosted voice AI](/case-studies/ai-voice-agent) (Pipecat + LiveKit + Whisper + Ollama, zero cloud dependency)
- [Production RAG pipelines](/blog/rag-pipeline-design-qdrant-production) (Qdrant, hybrid search, reranking)
- [HIPAA / SOC 2 compliant AI architecture](/blog/hipaa-compliant-ai-architecture)
- Drupal/Laravel/Django/FastAPI/Next.js full stack (yes, all of them, because the AI is only half the system)
Areas where I am not the right hire:
- Pure ML research (training new foundation models — you want a PhD, not me)
- Computer vision–heavy systems (I can integrate them, but I am not the deep specialist)
- 1-week throwaway MVPs (I optimise for production, which means I am slower and more expensive for true throwaways)
## What Happens If You Reach Out
1. **You email** `dilip@hurekatek.com` or [book a call](/contact). I respond within 24 hours, usually sooner.
2. **30-minute discovery call**, free. We figure out what you actually need (which is often different from what you think you need).
3. **One-page proposal** within 48 hours after the call: scope, timeline, price, deliverables.
4. **If we proceed** — paid two-week sprint to validate fit before any longer commitment.
5. **No agency fees, no recruiter fees, no platform cuts.** Direct contract, INR or USD invoice, terms you can negotiate.
If your project is in a domain where you need both AI depth *and* product/architecture judgement — that is where I do my best work.
## Conclusion: Hiring an AI Developer Is About Trade-offs, Not Buzzwords
The right **AI developer** for your project is not the one with the most impressive Twitter following or the longest list of frameworks on their resume. It is the one who:
1. Understands which of the four roles your project actually needs
2. Can show you a system currently serving real users
3. Talks about cost, latency, and evaluation as much as they talk about models
4. Tells you what *not* to build before they tell you what to build
5. Plans their own departure from day one
If that sounds like the kind of AI developer you need, [schedule a call](/contact) and we can spend 30 minutes seeing if your project is something I can genuinely help with. If not, I will tell you who else to talk to — that is the deal.
Either way, you will leave the call with a clearer picture of what you are hiring, what it should cost, and how long it should take. That is worth 30 minutes.
## Agentic AI: The Complete Guide to Building Autonomous AI Agents in 2026
- Date: 2026-06-28
- Category: AI Architecture
- Tags: Agentic AI, AI Agents, Multi-Agent Systems, LangGraph, Autonomous AI, Tool Use, AI Orchestration
- URL: https://dilsyno.com/blog/agentic-ai-complete-guide-autonomous-agents-2026
- Read time: 32 min read
## Introduction
Three years ago, a "smart AI" meant a model that could answer questions. Today, that same bar is embarrassingly low. In 2026, the frontier has moved to **agents** — systems that don't just respond, but pursue goals. They pick up a tool, examine the result, adjust their plan, call another tool, and keep going until the work is done. No hand-holding. No single prompt-and-response cycle. Just autonomous, goal-directed execution.
I've spent the last two years building agentic systems in production at Hureka Technologies, and I can tell you: the gap between a demo agent and a production agent is wider than most tutorials acknowledge. A demo agent is a parlour trick. A production agent is an architecture problem, a reliability problem, a cost problem, and — critically — a safety problem all at once. This guide exists because I wish it had existed when I started.
By the time you finish reading, you'll understand not just how agents work conceptually, but how to architect them with LangGraph, implement tool-using agents with proper Pydantic schemas, build multi-agent coordination systems, design memory hierarchies, wire up observability with LangFuse, and avoid the six mistakes that burn engineers every single time. This is the guide for engineers who intend to ship.
## What Is Agentic AI? The Foundation
The word "agent" has a precise meaning borrowed from philosophy and classical AI: an entity that perceives its environment and takes actions to achieve goals. What makes a system *agentic* is not the model it runs on — it's the loop it operates within.
A standard LLM call is stateless and single-shot: you send a prompt, you receive a completion. The model has no persistence, no memory of what it did before, no ability to reach out and interact with external systems. It's powerful, but it's passive. Agentic AI breaks this open by giving the model three capabilities that transform it from oracle into actor:
**1. Tool use.** The agent can invoke external functions — web searches, code execution, API calls, database queries, file reads, email sends. The model doesn't just answer "what's the weather?" — it calls a weather API, reads the result, and continues reasoning with real data.
**2. Memory.** Agents can persist and retrieve information across steps. Short-term memory keeps the context of the current task coherent. Long-term memory (vector stores) allows retrieval of facts from previous sessions. Episodic memory recalls sequences of past actions. Procedural memory captures learned strategies.
**3. Goal-directed iteration.** The agent doesn't stop after one tool call. It examines the result of that call, decides what to do next, acts again, and repeats until the goal is satisfied or a termination condition is reached. This is the loop that makes agents qualitatively different from chatbots.
Historically, the idea of software agents traces back to the 1980s and work at MIT and Stanford on "reactive agents" and "deliberative agents." By the early 2010s, multi-agent systems were an active subfield of AI research, but they relied on hand-crafted behavior rules. The revolution of 2023–2024 was harnessing large language models as the *reasoning engine* inside the agent loop — giving agents the ability to understand natural language goals, generate novel plans, and interpret arbitrary tool outputs without being explicitly programmed for each situation.
By 2026, agentic patterns have matured substantially. The early "LLM calling functions in a loop" approach has been replaced by structured frameworks with typed state machines, multi-agent coordination protocols, robust memory systems, and production-grade observability. What was cutting-edge research two years ago is now table stakes for enterprise deployments. The agents we build today are not clever demos — they're operational infrastructure.
## How It Works: The Architecture
Understanding agent architecture requires understanding the fundamental execution loop and how components are wired together. Let's start from first principles and build up to production-grade multi-agent systems.
### The ReAct Pattern: Think-Act-Observe
The foundational pattern for single agents is **ReAct** (Reasoning + Acting), introduced by Yao et al. in 2022 and now the default pattern in most production frameworks. The model alternates between reasoning traces (internal thought) and actions (tool calls), with each action producing an observation that feeds back into the next reasoning step.
```
┌─────────────────────────────────────────────────────────────────────────┐
│ REACT AGENT LOOP │
│ │
│ ┌──────────┐ GOAL / TASK ┌──────────────────────────────────┐ │
│ │ Human │ ─────────────────► │ LLM Reasoning Engine │ │
│ │ or System│ │ │ │
│ └──────────┘ │ THINK: "I need to search for X" │ │
│ ▲ │ ACT: tool_call(search, "X") │ │
│ │ └──────────────┬───────────────────┘ │
│ │ FINAL │ tool_call │
│ │ ANSWER ▼ │
│ │ ┌──────────────────────────────────┐ │
│ │ │ Tool Executor │ │
│ │ │ ┌──────┐ ┌──────┐ ┌─────────┐ │ │
│ │ │ │Search│ │ Code │ │ API │ │ │
│ │ │ │ Web │ │ Exec│ │ Client │ │ │
│ │ │ └──────┘ └──────┘ └─────────┘ │ │
│ │ └──────────────┬───────────────────┘ │
│ │ │ observation │
│ │ ┌──────────────▼───────────────────┐ │
│ │ │ OBSERVE: "Search returned: ..." │ │
│ │ │ THINK: "Now I should..." │ │
│ └──────────────────────────│ ANSWER: "Here is the result" │ │
│ complete └──────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘
```
### The Plan-Execute Pattern
For complex tasks, ReAct's step-by-step reasoning can be inefficient. The **Plan-Execute** pattern separates planning from execution: a planner model creates a structured task decomposition upfront, and executor agents carry out each sub-task. This is more efficient for deterministic workflows and easier to parallelize.
```
┌───────────────────────────────────────────────────────────────────────┐
│ PLAN-EXECUTE ARCHITECTURE │
│ │
│ GOAL │
│ │ │
│ ▼ │
│ ┌─────────────────┐ ┌─────────────────────────────────────┐ │
│ │ Planner Agent │────────►│ Task Queue │ │
│ │ (GPT-4o / o3) │ │ [task1] [task2] [task3] [task4] │ │
│ └─────────────────┘ └────────┬────────────────────────────┘ │
│ │ dispatch │
│ ┌──────────────────┼──────────────────┐ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Executor A │ │ Executor B │ │ Executor C │ │
│ │ (search) │ │ (code) │ │ (write) │ │
│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │
│ │ │ │ │
│ └─────────────────┴──────────────────┘ │
│ │ merge results │
│ ▼ │
│ ┌─────────────────┐ │
│ │ Synthesizer │──► FINAL OUTPUT │
│ └─────────────────┘ │
└───────────────────────────────────────────────────────────────────────┘
```
### LangGraph State Machines
LangGraph, now at version 0.4.x, models agent execution as a **directed graph of nodes and edges**. Each node is a function that receives the current state and returns an updated state. Edges are either deterministic (always go to node B after node A) or conditional (the next node is determined by inspecting the current state). This gives you the full power of a finite state machine layered on top of LLM reasoning — predictable, debuggable, and serializable.
The LangGraph execution model has three properties that matter enormously in production:
- **Persistence**: State is checkpointed at every node boundary. If the process crashes, you resume from the last checkpoint, not from scratch.
- **Serializability**: The state is a plain Python dict (constrained by a TypedDict schema), which means it can be serialized to JSON and stored in Redis, Postgres, or any other backend.
- **Determinism**: Given the same state and the same node function, the output is deterministic (modulo LLM non-determinism, which you can control with `temperature=0`).
## Core Components Deep Dive
### Agent State with TypedDict
In LangGraph, state is the single source of truth for the entire execution. Every node reads from it and writes back to it. Defining it correctly is not optional — a poorly defined state schema causes cascade failures that are nightmares to debug. Here's how we define robust agent state in production:
```python
from typing import Annotated, TypedDict, Sequence, Literal
from langchain_core.messages import BaseMessage, AnyMessage
from langgraph.graph.message import add_messages
from pydantic import BaseModel, Field
import operator
from datetime import datetime
class PlanStep(BaseModel):
step_id: str
description: str
status: Literal["pending", "running", "done", "failed"] = "pending"
result: str | None = None
error: str | None = None
class AgentState(TypedDict):
# Message history — add_messages ensures proper merging
messages: Annotated[Sequence[AnyMessage], add_messages]
# Current plan (if using plan-execute pattern)
plan: Annotated[list[PlanStep], operator.add]
# The original user goal — never mutated after init
goal: str
# Accumulated context from tool calls
context: Annotated[dict, lambda a, b: {**a, **b}]
# Iteration guard — prevents infinite loops
iterations: Annotated[int, operator.add]
max_iterations: int
# Human-in-the-loop control
requires_approval: bool
approval_granted: bool | None
# Final output
final_answer: str | None
completed_at: datetime | None
def initial_state(goal: str, max_iter: int = 20) -> AgentState:
"""Create a fresh agent state for a new task."""
return AgentState(
messages=[],
plan=[],
goal=goal,
context={},
iterations=0,
max_iterations=max_iter,
requires_approval=False,
approval_granted=None,
final_answer=None,
completed_at=None,
)
```
The `Annotated` type hints carry reducer functions — LangGraph uses these to merge partial state updates from nodes. `add_messages` is a LangGraph-provided reducer that correctly handles message deduplication. `operator.add` accumulates integers and lists. The custom lambda for `context` implements a shallow merge. Never write a node that replaces the entire state — always return only the keys you're updating.
### Tool Schemas with Pydantic
Tools are the hands of the agent. Every tool must have a typed schema that the LLM can understand — this is what gets serialized into the function-calling spec and sent to the model. Sloppy tool schemas are the leading cause of agents calling tools incorrectly. Treat them like public API contracts, because that's what they are.
```python
from langchain_core.tools import tool
from pydantic import BaseModel, Field
from typing import Literal
import httpx, json
class WebSearchInput(BaseModel):
query: str = Field(
description="The search query. Be specific. Max 100 chars.",
max_length=100
)
max_results: int = Field(
default=5,
ge=1,
le=20,
description="Number of results to return (1-20)"
)
date_filter: Literal["any", "week", "month", "year"] = Field(
default="any",
description="Filter results by recency"
)
@tool(args_schema=WebSearchInput, return_direct=False)
async def web_search(
query: str,
max_results: int = 5,
date_filter: str = "any"
) -> str:
"""
Search the web for current information. Use this for:
- Facts that may have changed after your training cutoff
- Real-time data (prices, news, events)
- Verifying claims against public sources
Do NOT use for code generation or reasoning tasks.
"""
async with httpx.AsyncClient() as client:
response = await client.post(
"https://api.tavily.com/search",
json={"query": query, "max_results": max_results,
"days": {"week": 7, "month": 30, "year": 365}.get(date_filter, 0)},
headers={"Authorization": f"Bearer ${TAVILY_API_KEY}"}
)
data = response.json()
results = data.get("results", [])
return json.dumps([
{"title": r["title"], "url": r["url"], "snippet": r["content"][:400]}
for r in results
], indent=2)
class CodeExecutionInput(BaseModel):
code: str = Field(description="Python code to execute in a sandboxed environment")
timeout_seconds: int = Field(default=30, ge=1, le=120)
@tool(args_schema=CodeExecutionInput)
async def execute_python(code: str, timeout_seconds: int = 30) -> str:
"""
Execute Python code in an E2B sandboxed environment.
Returns stdout, stderr, and any raised exceptions.
Use for: data analysis, calculations, file processing, API testing.
The sandbox has numpy, pandas, requests, and matplotlib available.
"""
from e2b_code_interpreter import Sandbox
with Sandbox() as sbx:
execution = sbx.run_code(code)
return json.dumps({
"stdout": execution.logs.stdout,
"stderr": execution.logs.stderr,
"error": str(execution.error) if execution.error else None
})
```
Three rules for tool schemas that hold up in production: write the docstring as though you're explaining to a new hire when and why to use this tool; constrain inputs with Pydantic validators (`ge`, `le`, `max_length`, `Literal`) so invalid calls fail fast with clear error messages before reaching the API; and keep each tool focused on a single responsibility — a tool that does three different things depending on a mode parameter is a tool that gets called incorrectly.
### Memory System Architecture
Memory is where most agent tutorials fail to go deep enough. There are four distinct memory types, and using the wrong one for a given task causes either context overflow, retrieval failures, or both.
```python
import redis.asyncio as redis
from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings
from pydantic import BaseModel
from datetime import datetime, timedelta
import json
class MemorySystem:
"""
Four-tier memory hierarchy:
1. Short-term → Redis (current session, TTL=2h)
2. Long-term → ChromaDB (facts, permanent)
3. Episodic → Postgres (task histories, searchable)
4. Procedural → Prompt templates (learned strategies)
"""
def __init__(self, session_id: str):
self.session_id = session_id
self.redis = redis.from_url("redis://localhost:6379")
self.embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
self.vectorstore = Chroma.from_existing(
collection_name="agent_knowledge",
embedding_function=self.embeddings
)
# Short-term: Redis
async def remember_short_term(self, key: str, value: dict, ttl: int = 7200):
"""Store in Redis with a 2-hour default TTL."""
full_key = f"session:${self.session_id}:${key}"
await self.redis.setex(full_key, ttl, json.dumps(value))
async def recall_short_term(self, key: str) -> dict | None:
full_key = f"session:${self.session_id}:${key}"
data = await self.redis.get(full_key)
return json.loads(data) if data else None
# Long-term: Vector store
async def store_fact(self, content: str, metadata: dict):
"""Embed and store a fact permanently."""
await self.vectorstore.aadd_texts(
texts=[content],
metadatas=[{**metadata, "stored_at": datetime.utcnow().isoformat()}]
)
async def retrieve_relevant(self, query: str, k: int = 5) -> list[str]:
"""Semantic retrieval — returns top-k relevant facts."""
docs = await self.vectorstore.asimilarity_search(query, k=k)
return [doc.page_content for doc in docs]
```
The critical architectural decision is which facts belong at which tier. A rule of thumb from production: if you'll need it again in a different session, it goes to the vector store. If you only need it for the current task run, Redis. If you need to replay or audit a full task sequence, Postgres episodic memory. Procedural memory — learned strategies about how to approach certain task types — lives in prompt templates that get updated via a separate fine-tuning or prompt management pipeline, not at runtime.
### Safety Guardrails and the Guard Node
Every agent in our production stack passes through a guard node before the main execution loop. The guard node is a lightweight classifier (not a full reasoning chain) that checks the incoming goal against a structured deny-list and a set of scope constraints defined at deployment time.
```python
from pydantic import BaseModel
from langchain_anthropic import ChatAnthropic
from langchain_core.messages import SystemMessage, HumanMessage
GUARD_SYSTEM = """You are a safety classifier for an AI agent system.
Classify the incoming goal as SAFE or UNSAFE.
UNSAFE if the goal requests:
- Accessing systems outside the declared scope
- Extracting or transmitting PII
- Competitive intelligence that may violate terms of service
- Any irreversible destructive action without explicit authorization
- Bypassing security controls
Respond with JSON: {"verdict": "SAFE"|"UNSAFE", "reason": "..."}"""
class GuardResult(BaseModel):
verdict: str
reason: str
async def guard_node(state: AgentState) -> dict:
"""Safety gate — runs before any agent logic."""
model = ChatAnthropic(model="claude-haiku-4-5").with_structured_output(GuardResult)
result = await model.ainvoke([
SystemMessage(content=GUARD_SYSTEM),
HumanMessage(content=f"Goal: ${state['goal']}")
])
if result.verdict == "UNSAFE":
return {
"final_answer": f"Task rejected by safety guard: ${result.reason}",
"completed_at": datetime.utcnow()
}
return {"iterations": 1} # Increment and proceed
```
## Implementation: Step-by-Step Guide
Let's build a complete research agent from scratch. This agent will take a research question, break it into sub-questions, search the web for each, synthesize the findings, and produce a structured report. This is a pattern we use at Hureka Technologies for client intelligence automation.
### Step 1: Define the Graph
```python
from langgraph.graph import StateGraph, END
from langgraph.checkpoint.redis import AsyncRedisSaver
from langchain_anthropic import ChatAnthropic
from langchain_core.messages import SystemMessage, HumanMessage
from .state import AgentState, initial_state
from .tools import web_search, execute_python
from .nodes import planner_node, executor_node, synthesizer_node, guard_node
import asyncio
def should_continue(state: AgentState) -> str:
"""Conditional edge: determine next node based on state."""
if state["iterations"] >= state["max_iterations"]:
return "force_end"
if state["requires_approval"] and not state["approval_granted"]:
return "await_human"
if state["final_answer"]:
return "done"
if all(p.status == "done" for p in state["plan"]):
return "synthesize"
return "execute"
def build_research_agent() -> StateGraph:
# Initialize the graph with our state schema
builder = StateGraph(AgentState)
# Add nodes
builder.add_node("guard", guard_node) # Safety check
builder.add_node("planner", planner_node) # Decompose goal
builder.add_node("executor", executor_node) # Execute one step
builder.add_node("synthesizer", synthesizer_node) # Merge results
# Entry point
builder.set_entry_point("guard")
# Edges
builder.add_edge("guard", "planner")
builder.add_edge("planner", "executor")
# Conditional routing after each execution step
builder.add_conditional_edges(
"executor",
should_continue,
{
"execute": "executor",
"synthesize": "synthesizer",
"await_human": "__interrupt__",
"force_end": END,
"done": END,
}
)
builder.add_edge("synthesizer", END)
# Compile with Redis checkpointer for persistence
checkpointer = AsyncRedisSaver.from_conn_string("redis://localhost:6379")
return builder.compile(
checkpointer=checkpointer,
interrupt_before=["__interrupt__"]
)
async def run_agent(goal: str, thread_id: str) -> str:
graph = build_research_agent()
config = {"configurable": {"thread_id": thread_id}}
state = initial_state(goal)
result = await graph.ainvoke(state, config=config)
return result["final_answer"]
```
### Step 2: Multi-Agent Orchestration
```python
from langchain_core.tools import tool
from langgraph.prebuilt import create_react_agent
from langchain_anthropic import ChatAnthropic
from pydantic import BaseModel, Field
import asyncio
# Specialist agents
research_agent = create_react_agent(
model=ChatAnthropic(model="claude-sonnet-4-5"),
tools=[web_search, semantic_scholar_search],
state_modifier="You are a research specialist. Gather factual information only. "
"Cite every claim with a source URL."
)
analyst_agent = create_react_agent(
model=ChatAnthropic(model="claude-sonnet-4-5"),
tools=[execute_python, query_database],
state_modifier="You are a quantitative analyst. Produce structured analysis. "
"Always show your calculations."
)
writer_agent = create_react_agent(
model=ChatAnthropic(model="claude-opus-4-5"),
tools=[read_file, write_file],
state_modifier="You are a technical writer. Produce clear, structured documents. "
"Use markdown. Cite all sources from the research provided."
)
# Coordinator tools that invoke specialists
class DelegateInput(BaseModel):
task: str = Field(description="The specific task to delegate")
context: str = Field(description="Relevant context the specialist needs")
@tool(args_schema=DelegateInput)
async def delegate_research(task: str, context: str) -> str:
"""Delegate a research task to the research specialist agent."""
result = await research_agent.ainvoke({
"messages": [{"role": "user", "content": f"${task}\n\nContext: ${context}"}]
})
return result["messages"][-1].content
@tool(args_schema=DelegateInput)
async def delegate_analysis(task: str, context: str) -> str:
"""Delegate a quantitative analysis task to the analyst agent."""
result = await analyst_agent.ainvoke({
"messages": [{"role": "user", "content": f"${task}\n\nData context: ${context}"}]
})
return result["messages"][-1].content
# Coordinator agent
coordinator = create_react_agent(
model=ChatAnthropic(model="claude-opus-4-5"),
tools=[delegate_research, delegate_analysis, delegate_writing],
state_modifier="""You are a project coordinator. Break complex tasks into specialized
sub-tasks and delegate to the appropriate specialist. Synthesize their outputs
into a coherent final deliverable. Always delegate; never do specialist work yourself."""
)
```
### Step 3: Observability with LangFuse
```python
from langfuse import Langfuse
from langfuse.callback import CallbackHandler
from langfuse.decorators import observe, langfuse_context
import functools, time
langfuse = Langfuse(
public_key=LANGFUSE_PUBLIC_KEY,
secret_key=LANGFUSE_SECRET_KEY,
host="https://us.cloud.langfuse.com"
)
def get_trace_handler(
session_id: str,
user_id: str,
task_name: str
) -> CallbackHandler:
"""Create a LangFuse callback handler for a single agent run."""
return CallbackHandler(
trace_name=task_name,
session_id=session_id,
user_id=user_id,
metadata={"environment": "production", "version": APP_VERSION},
tags=["agent", "production"]
)
@observe(name="agent_run", capture_input=True, capture_output=True)
async def instrumented_run(goal: str, thread_id: str, user_id: str) -> str:
"""Fully instrumented agent run with cost and latency tracking."""
langfuse_context.update_current_trace(
name=f"research_${thread_id}",
user_id=user_id,
tags=["research-agent"]
)
handler = get_trace_handler(thread_id, user_id, "research_task")
graph = build_research_agent()
config = {
"configurable": {"thread_id": thread_id},
"callbacks": [handler]
}
result = await graph.ainvoke(initial_state(goal), config=config)
# Score the trace for quality monitoring
langfuse_context.score_current_trace(
name="completion",
value=1 if result["final_answer"] else 0,
comment="Task completed successfully" if result["final_answer"] else "Task failed"
)
return result["final_answer"]
```
With this setup, every LLM call, every tool invocation, every state transition is captured in LangFuse with full input/output payloads, token counts, latency, and cost attribution. You can replay any failing trace, compare performance across model versions, and set up alerts on cost-per-task thresholds.
## Production Patterns and Best Practices
After two years of shipping agentic systems at Hureka Technologies, including deployments for financial services, healthcare data processing, and e-commerce intelligence clients, these are the patterns that distinguish robust production systems from impressive demos.
### Checkpointing and Resumability
Production agent runs can fail mid-execution — network timeouts, API rate limits, hardware restarts. Without checkpointing, a 45-minute agent run that fails at step 38 is a complete restart. LangGraph's Redis checkpointer persists state at every node transition. When a run fails, you resume from the last checkpoint. We've seen this save hours of compute time in long-running enterprise research tasks. Make checkpointing non-negotiable from day one — wiring it in after the fact requires rearchitecting your state schema.
### Human-in-the-Loop Gates
Agentic AI without human oversight is a liability, not an asset. We use LangGraph's interrupt mechanism to pause execution at critical decision points: before sending emails, before making database writes, before executing code with external side effects. The agent presents a summary of what it's about to do, a human approves or rejects, and execution resumes. This is not optional for enterprise clients — it's a contractual requirement in most of our SOW agreements. The HITL gate should be hard-coded into the graph topology, not left as a runtime configuration option that someone might accidentally disable.
### Cost Guardrails
A production agent running claude-opus-4-5 with no guardrails can burn through a $500 budget in a single runaway session. We enforce hard token budgets at the state level, track cumulative cost against a per-session limit, and gracefully terminate with a partial result when the budget is exhausted. Every agent run in production has a defined cost ceiling before it starts — not as a configuration flag, but as a required parameter to the `initial_state` factory function.
> **Production lesson**: Set your `max_iterations` guard, a token budget, and a wall-clock timeout independently. Any one of the three can save you from a catastrophic runaway. All three together means you sleep soundly during on-call rotations.
### Safety Guardrails
Every agent in our production stack passes through a guard node before the main execution loop. This node checks the incoming goal against a deny-list of prohibited task categories and rejects them with a structured error before any LLM inference occurs. The guard node is cheap — it's a classifier call against claude-haiku, not a full reasoning chain. Fast rejection on bad inputs is orders of magnitude cheaper than reasoning your way to a refusal.
### Structured Outputs, Always
Free-form LLM text is a testing nightmare. Every node in our agents that produces a structured result uses Pydantic models with `model.with_structured_output(OutputSchema)`. This catches schema violations at the boundary, gives you type-safe state updates, and makes your test suite dramatically simpler. If a node's output isn't structured, it's a code smell that invites silent bugs downstream.
## Performance Optimization
Performance in agentic systems has two dimensions: latency (how long a single run takes) and throughput (how many concurrent runs you can sustain). Optimization strategies differ for each.
### Parallelism in Plan-Execute
The single highest-ROI optimization is parallelizing independent plan steps. If your agent's plan has 6 research sub-questions and none depend on each other, you can execute all 6 concurrently and cut wall-clock time by 80%. LangGraph supports this with `Send` for fan-out and a `reduce` node for fan-in. In our benchmarks, a sequential research agent averaging 4.2 minutes per task dropped to 52 seconds after parallelization — a 4.8x speedup with zero change to output quality.
### Model Routing
Not every node in your graph needs the most powerful model. Our production routing strategy: use `claude-haiku-4-5` for classification, routing decisions, and guard checks; `claude-sonnet-4-5` for tool use, data extraction, and intermediate reasoning; `claude-opus-4-5` only for the final synthesis step. This cuts cost by approximately 65% versus running opus everywhere, with no measurable quality difference on benchmark tasks. The key insight is that tool-calling accuracy depends more on schema quality than model size — a well-designed Pydantic schema enables a smaller model to call tools reliably.
### Caching Repeated Calls
Enable Anthropic prompt caching on your system prompts and tool schemas. These are large, static payloads that get re-sent on every API call in a long agent run. With prompt caching, we've seen 40–55% cost reduction on runs with more than 20 LLM calls. Set the `cache_control` breakpoint at the boundary between static and dynamic content — typically right after the tool schema block.
### Async Throughout
Every I/O operation in your agent stack should be async: LLM calls, tool executions, vector store queries, Redis operations. Running 10 concurrent agent tasks with synchronous code creates a blocking queue that degrades latency for every user. With full async, the same hardware sustains 10 concurrent tasks with near-zero contention. Measured throughput increase in our staging environment: **8.3x on the same instance count** after migrating from sync to full async execution.
## Common Mistakes and How to Avoid Them
**Mistake 1: No iteration limit**
The agent enters a reasoning loop — it can't find the information it needs, tries a slightly different search, still fails, modifies the query again — and runs until you hit the API rate limit or the session timeout. Without a hard `max_iterations` guard, this costs money and produces no output.
*Fix*: Set `max_iterations` in initial state. Add a conditional edge from every execution node back to a termination check. Default to 20 iterations; allow up to 50 only for explicitly long-running tasks with a justified budget.
**Mistake 2: Treating tool descriptions as optional**
The LLM's tool-calling accuracy is entirely dependent on the quality of the tool docstring and parameter descriptions. Vague descriptions like "search for information" produce incorrect parameter values and inappropriate tool selection at an alarming rate — we've seen tool selection accuracy drop from 94% to 61% when swapping a detailed docstring for a one-liner.
*Fix*: Write tool descriptions as if you're writing API documentation for a junior developer. Specify what the tool does, what it does NOT do, expected inputs, and example use cases. Test with adversarial prompts before deployment.
**Mistake 3: Mixing memory tiers incorrectly**
Shoving everything into the message history (short-term) blooms your context window and increases cost quadratically. Engineers often don't add long-term memory until the context is already overflowing, at which point key earlier information has been dropped.
*Fix*: Design your memory architecture before writing the first node. Facts that might be needed in future sessions go to the vector store. Session-scoped scratchpad data goes to Redis. Only the active reasoning chain lives in the message history.
**Mistake 4: No observability until production breaks**
Wiring up LangFuse as an afterthought means you have no baseline to compare against when something degrades. You'll be debugging production incidents with no cost history, no latency percentiles, and no way to replay the failing trace.
*Fix*: Add LangFuse instrumentation in the first sprint, before the agent runs in production. Establish baseline P50/P95 latency and per-run cost metrics during staging. Set alerts on both before the first production deployment.
**Mistake 5: One agent trying to do everything**
The "kitchen sink" agent — given 15 tools and a sprawling system prompt — is a testing nightmare and an accuracy disaster. The context window fills with irrelevant tool schemas, the model struggles to select the right tool, and a single failure cascades through all tasks.
*Fix*: Decompose by responsibility. A research agent gets search tools. An analysis agent gets computational tools. A writing agent gets file and formatting tools. A coordinator delegates. Each specialist agent has at most 4–6 tools and a focused system prompt under 500 tokens.
**Mistake 6: No human-in-the-loop on consequential actions**
An agent that sends emails, modifies databases, or executes code with external side effects without human approval is an incident waiting to happen. Real production environments have irreversible actions. An agent running confidently in the wrong direction can cause serious damage before a human notices.
*Fix*: Classify every tool call as either read-only (safe to auto-execute) or write/destructive (requires approval gate). Use LangGraph's `interrupt_before` to pause execution before destructive actions. Log every approved action with the approver's identity for audit trails.
## Real-World Use Cases
**Regulatory Compliance Intelligence Agent** *(Financial Services)*
A multi-agent system monitoring regulatory feeds (SEC, FINRA, FCA) for rule changes that affect a portfolio management firm. The research agent ingests new publications, the analyst agent assesses impact on existing processes, and the writer agent produces a compliance impact brief for the legal team — delivered within 90 minutes of a new regulation being published. Replaces a 3-day manual research-and-drafting process. Results: 97% recall on relevant rules, $340K/yr in analyst hours saved.
**Competitive Pricing Intelligence Agent** *(E-Commerce)*
An agent that monitors competitor product pages across 12 e-commerce platforms, extracts current pricing and availability data, compares against the client's catalog, and produces daily pricing recommendations with projected margin impact. The code-execution agent runs the statistical analysis; the coordinator synthesizes recommendations. No browser automation — structured data via official APIs and permitted scraping endpoints only. Results: 4.2% margin improvement across 8,400+ SKUs monitored daily.
**Clinical Trial Eligibility Screener** *(Healthcare Data)*
A human-in-the-loop agent that screens anonymized patient records against clinical trial eligibility criteria. The agent extracts structured clinical features from unstructured notes, applies inclusion/exclusion logic, flags ambiguous cases for clinician review, and produces a ranked eligibility list. Every patient record the agent classifies as eligible is verified by a human before any outreach — the HITL gate is hard-coded, not configurable. HIPAA compliance requirements drove the entire architecture. Results: 91% screening accuracy, 3x faster than manual chart review, 100% human review on eligible flags.
**Automated Code Review and PR Triage Agent** *(Software Development)*
A developer experience agent that receives webhook events for new pull requests, fetches the diff, checks for common anti-patterns (N+1 queries, missing error handling, hardcoded secrets), assesses test coverage delta, and posts structured review comments. A coordinator agent routes security-sensitive changes to a specialist security-review agent. Non-blocking — developers are never held up — but 78% of flagged issues are resolved before the first human reviewer looks at the PR. Results: average PR cycle time reduced by 2.1 days, first comment latency under 45 seconds.
## Tool and Approach Comparison
| Framework | Abstraction Level | State Management | Multi-Agent | Observability | Production Maturity | Best For |
|-----------|-------------------|-----------------|-------------|---------------|---------------------|----------|
| **LangGraph** | Low — explicit graph | Typed TypedDict, Redis checkpointing, full persistence | Excellent — subgraph composition | Native LangFuse + LangSmith | High | Complex stateful workflows, enterprise deployments, regulated industries |
| **CrewAI** | High — role-based DSL | Task memory per crew, no native checkpointing | Excellent — built for crews | Agentops integration, limited native | Medium | Rapid prototyping, role-based pipelines, non-critical workloads |
| **AutoGen** | Medium — conversation-centric | In-memory conversation history only | Good — GroupChat pattern | Basic logging, no native tracing | Medium | Research experiments, conversational agent systems, Microsoft ecosystem |
| **Semantic Kernel** | Medium — plugin-based | Process state machine, good for step-based flows | Fair — Process framework | Azure Monitor, OpenTelemetry | High | .NET enterprise shops, Azure deployments, Microsoft toolchain integration |
My take: for new production systems in 2026, reach for LangGraph. The explicit state machine forces you to think through your execution model before writing a line of agent logic — which is almost always the right constraint. CrewAI is excellent for getting a proof-of-concept in front of stakeholders fast, but I've never taken a CrewAI agent to production without eventually rewriting it in LangGraph. Start with the tool that will carry you to production.
## Future Trends in 2026 and Beyond
**Long-horizon task execution.** Current production agents handle tasks measured in minutes to hours. Nascent research on hierarchical planning and persistent agent processes is enabling tasks measured in days. Amazon's agentic coding systems and Anthropic's internal tooling are already running week-long software development cycles autonomously. The memory and checkpointing architecture required for this is fundamentally different from what most teams have today — expect new primitives to emerge specifically for ultra-long-horizon agents.
**Agent-to-agent communication protocols.** The Model Context Protocol (MCP) is becoming the lingua franca for agent interoperability. A standardized protocol allows agents from different vendors and frameworks to discover each other's capabilities and delegate tasks without custom integration code. Early MCP server ecosystems are already live; by 2027 this will be as standard as REST is for web APIs. Build your agent tools as MCP servers from the start.
**Reasoning model integration in agent loops.** o3 and its successors excel at complex, multi-step reasoning but are expensive to invoke on every turn. The emerging pattern is a hybrid: fast action models (Sonnet-class) handle most steps, and reasoning models (o3-class) are invoked only on planning, ambiguity resolution, and synthesis. Routing logic for this hybrid is itself becoming an agent design pattern — a meta-agent that decides which cognitive mode to apply.
**Formal verification of agent behavior.** Enterprise and regulated-industry clients increasingly demand provable guarantees about agent behavior. Research into runtime formal verification — checking agent action sequences against temporal logic specifications before execution — is moving from academic papers to early tooling. This will change how we write safety constraints from informal prose to machine-checkable specifications.
**Agentic infrastructure as a service.** Cloud providers are building agent-specific infrastructure: persistent process containers, managed checkpointing, agent-native monitoring, and built-in HITL workflow services. This is analogous to how serverless abstracted function execution — the next wave abstracts agent execution. For teams without dedicated AI infrastructure engineers, this will dramatically lower the cost of production deployment.
## Conclusion and Next Steps
Agentic AI is not a technology you can learn entirely from documentation. It's a discipline you develop through building, breaking, and rebuilding real systems. The patterns in this guide — typed state management, structured tool schemas, multi-agent coordination, memory hierarchies, observability-first engineering, and hard safety guardrails — aren't theoretical recommendations. They're the hard lessons from systems that have run in production, served real users, and occasionally failed in instructive ways.
The gap between a demo agent and a production agent is still wide in 2026, but it's no longer mysterious. The primitives are stable. The frameworks are mature. The observability tooling exists. What separates teams that ship is not access to better models — it's discipline in state design, rigor in tool schema definition, and an unwillingness to skip the safety and observability plumbing.
If you're starting today: build a single-agent ReAct system with LangGraph, wire up LangFuse from day one, and add the memory and multi-agent layers only when the simpler architecture genuinely can't handle the task. Resist the urge to over-engineer upfront. The agent pattern that ships and serves users is better than the perfect architecture that stays in a design document.
I'll be publishing follow-up deep dives on LangGraph state machine patterns, memory system design, and LangFuse-driven cost optimization over the coming weeks. The full code for the research agent in this guide is available on GitHub — includes Docker Compose for local Redis, ChromaDB, and a LangFuse CE instance, plus a test suite with 24 adversarial test cases. No setup hell. Just `docker compose up` and you're running.
## RAG (Retrieval-Augmented Generation): The Definitive Production Guide for 2026
- Date: 2026-06-27
- Category: RAG Systems
- Tags: RAG, Retrieval-Augmented Generation, Vector Search, Qdrant, Embeddings, Semantic Search, Knowledge Base
- URL: https://dilsyno.com/blog/rag-retrieval-augmented-generation-complete-guide-2026
- Read time: 35 min read
## Introduction
In 2023, everyone was excited that large language models could write code, summarize documents, and answer questions. By 2024, the same people were burned by hallucinations — confident, plausible-sounding answers that were simply wrong. The problem wasn't the model. The problem was that we were asking a system trained on a static snapshot of the world to answer questions about *your* proprietary documents, *your* customers, *your* latest product specs. No amount of fine-tuning was going to fix that fundamental mismatch.
Retrieval-Augmented Generation — RAG — is the architecture that bridges this gap. Rather than baking knowledge into model weights, RAG systems retrieve relevant context at inference time and ground the model's generation in that retrieved evidence. It sounds almost embarrassingly simple. The devil, as always, is in the production details: how you chunk documents, which embedding model you choose, how you structure your vector index, how you fuse sparse and dense retrieval signals, how you rerank results, and — critically — how you measure whether any of it actually works. At Hureka Technologies, we have shipped RAG systems for enterprise clients across legal, financial, healthcare, and e-commerce verticals. This guide distills what we have learned doing that work at scale.
What you will learn here: the full conceptual and architectural foundation of RAG from first principles; a deep dive into every component (document processing, embeddings, vector databases, retrieval, reranking); a complete implementation walkthrough with production-ready Python code; advanced patterns including HyDE, multi-hop RAG, and GraphRAG; Ragas-based evaluation methodology; performance optimization techniques with real benchmark numbers; and a comprehensive comparison of every major vector database on the market in 2026. Whether you are building your first RAG system or trying to debug why your production pipeline is returning irrelevant context, this guide is the one you will want to bookmark.
## What Is RAG? The Foundation
To understand why RAG exists, you first need to understand why LLMs hallucinate. A language model is trained to predict the next token given the previous ones. During training, it ingests hundreds of billions of tokens and compresses that information into billions of floating-point parameters — the model weights. This compression is lossy. Facts that appeared rarely in training data may not survive. More importantly, the model has no mechanism to distinguish "information I am confident about from training" from "information I am confabulating to satisfy the statistical patterns of plausible text." When you ask GPT-4 or Claude about your internal company policy document — something it has never seen — it will nonetheless generate a confident-sounding answer because that is what language models do.
The classical solution was fine-tuning: take a pre-trained model and continue training it on your domain-specific documents. Fine-tuning does improve domain fluency, but it does not reliably memorize discrete facts. Research from 2023–2024 consistently showed that fine-tuned models could still hallucinate with high confidence on factual recall tasks. The reason is geometric: a fact embedded in 7 billion parameters is not the same as a fact written down in a database you can query. The former is distributed and diffuse; the latter is discrete and retrievable.
RAG was formalized in the 2020 paper *"Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks"* by Lewis et al. at Meta AI. The core idea was elegant: instead of forcing the model to retrieve facts from its weights, give the model a retrieval mechanism that can pull relevant documents at inference time. The model then generates its answer conditioned on both the query and the retrieved documents. The retrieved documents are the grounding signal — they give the model explicit text to refer to rather than requiring it to reconstruct facts from compressed parametric memory.
Between 2020 and 2026, RAG evolved from a research curiosity to the dominant architecture for production AI applications. The inflection points were: the explosion of open embedding models in 2022–2023 (making semantic search affordable), the proliferation of dedicated vector databases (Qdrant, Weaviate, Pinecone, Chroma), the standardization of chunking and indexing pipelines through frameworks like LangChain and LlamaIndex, and — most recently — the shift toward hybrid retrieval combining dense embeddings with sparse lexical signals like BM25. By 2026, a production RAG system is no longer a research experiment. It is a well-understood engineering discipline with established patterns, benchmarks, and failure modes. This guide covers the 2026 state of the art.
## How It Works: The Architecture
A production RAG system has two distinct phases: an **indexing pipeline** (offline, runs when you add new documents) and a **retrieval-generation pipeline** (online, runs per query). Understanding this separation is essential — many production bugs stem from inconsistencies between how documents were indexed and how queries are processed at inference time.
The indexing pipeline takes raw documents (PDFs, HTML, DOCX, code, etc.), processes them through a loader, splits them into chunks, generates vector embeddings for each chunk, and stores both the embeddings and the original chunk text in a vector database. The retrieval pipeline takes a user query, optionally transforms it, embeds it using the same embedding model, searches the vector index for nearest neighbors, reranks the results, and injects the top-k chunks into the LLM's context window along with the original query.
```
── INDEXING PIPELINE (Offline) ──────────────────────────────────────────────
Raw Documents Vector Database
(PDF, HTML, DOCX, ──► Document ──► Text ──► Embedding ──► (Qdrant / pgvector
Markdown, Code) Loader Chunker Model Pinecone /
Weaviate)
▼ ▼ ▼
[normalize] [chunk_id, [float32 ┌─────────────┐
[extract] metadata, vector] │ id: chunk_7 │
[clean] text] dim: 1536 │ vec: [0.21, │
│ -0.04, ...]│
│ payload: │
│ {text,meta}│
└─────────────┘
── RETRIEVAL + GENERATION PIPELINE (Online, per query) ──────────────────────
User Query LLM Response
│ ▲
▼ │
┌──────────┐ ┌────────────┐ ┌──────────────┐ ┌────────────┐ │
│ Query │──►│ Embedding │──►│ Vector Search│──►│ Reranker │────►│
│Transform │ │ Model │ │ (ANN/HNSW) │ │(cross-encdr)│ │
└──────────┘ └────────────┘ └──────────────┘ └────────────┘ │
│ +BM25 sparse top-k chunks
│ (HyDE / step-back / → RRF fusion │
│ multi-query) │
│ │
└──────────────────────► Prompt Assembly ──────────────────────► │
[System] + [Context chunks] + [User query]
```
The **HNSW index** (Hierarchical Navigable Small World) is what makes approximate nearest neighbor search fast enough for production. Rather than comparing the query vector against every stored vector (which is O(n) and prohibitive at millions of documents), HNSW builds a multi-layer graph where each layer is a navigable network of vectors. Search starts at the top layer (coarse navigation) and zooms in progressively, achieving O(log n) complexity with high recall. Nearly every major vector database uses HNSW under the hood.
The **reranker** is a second-pass scoring model that takes the top-k retrieved chunks (say, top-20) and reorders them by relevance to the query. Embedding-based retrieval uses cosine similarity in a high-dimensional space — it is efficient but imprecise. A cross-encoder reranker reads both the query and each chunk together (using full attention across both), producing a more accurate relevance score. The tradeoff is speed: you cannot run a cross-encoder over your entire corpus, which is why reranking is always a second-pass over a pre-filtered candidate set.
## Core Components Deep Dive
### Document Loading and Preprocessing
The quality of your RAG system is bounded by the quality of your document preprocessing. Garbage in, garbage out is nowhere more true than here. For PDFs, the critical question is whether your PDF is text-based or image-based (scanned). Text-based PDFs can be parsed with `pymupdf` (fastest, preserves layout) or `pdfplumber` (better table extraction). Image-based PDFs require OCR — Tesseract for open-source or Azure Document Intelligence / AWS Textract for production-grade accuracy. Always extract document structure (headings, section titles) as metadata — it becomes critical for filtering at retrieval time.
```python
import fitz # pymupdf
from dataclasses import dataclass
from typing import Iterator
@dataclass
class DocumentChunk:
text: str
doc_id: str
page: int
section: str
chunk_index: int
metadata: dict
def load_pdf(path: str, doc_id: str) -> Iterator[DocumentChunk]:
doc = fitz.open(path)
for page_num, page in enumerate(doc):
blocks = page.get_text("dict")["blocks"]
section = ""
for block in blocks:
if block["type"] == 0: # text block
# Heuristic: large font = heading → capture as section
for line in block["lines"]:
for span in line["spans"]:
if span["size"] > 14:
section = span["text"].strip()
text = page.get_text("text").strip()
if text:
yield DocumentChunk(
text=text, doc_id=doc_id,
page=page_num, section=section,
chunk_index=page_num,
metadata={"source": path, "page": page_num}
)
```
### Chunking Strategies
How you split documents is arguably the single most impactful decision in a RAG system. Every chunking strategy involves a tradeoff between *context density* (how much meaning fits in one chunk) and *retrieval precision* (how likely a retrieved chunk contains the exact answer). The main strategies:
- **Fixed-size chunking:** Split every N tokens with M tokens of overlap. Simple, fast, predictable. Works well for homogeneous text (news articles, documentation). Fails on structured content where splits cut mid-sentence or mid-table.
- **Recursive character splitting:** Tries to split on paragraph breaks, then sentence breaks, then word breaks, falling back gracefully. The LangChain `RecursiveCharacterTextSplitter` default. Better than fixed-size for most prose.
- **Semantic chunking:** Embeds each sentence, computes cosine similarity between adjacent sentences, splits where similarity drops below a threshold. Produces semantically coherent chunks. Significantly more expensive (requires embedding every sentence). Our benchmark at Hureka showed a 12–18% improvement in context recall over fixed-size chunking on legal documents.
- **Agentic/structural chunking:** Use an LLM to identify meaningful semantic boundaries (section headers, argument breaks, table boundaries). Most expensive; justified only for high-value document types.
```python
from langchain.text_splitter import RecursiveCharacterTextSplitter
import numpy as np
from sentence_transformers import SentenceTransformer
# --- Strategy 1: Recursive (good default) ---
splitter = RecursiveCharacterTextSplitter(
chunk_size=512,
chunk_overlap=64,
separators=["\n\n", "\n", ". ", " ", ""]
)
chunks = splitter.split_text(document_text)
# --- Strategy 2: Semantic chunking ---
def semantic_chunk(text: str, model: SentenceTransformer,
threshold: float = 0.75) -> list[str]:
sentences = text.split(". ")
embeddings = model.encode(sentences)
splits = [0]
for i in range(1, len(embeddings)):
sim = np.dot(embeddings[i-1], embeddings[i]) / (
np.linalg.norm(embeddings[i-1]) * np.linalg.norm(embeddings[i])
)
if sim < threshold:
splits.append(i)
splits.append(len(sentences))
return [
". ".join(sentences[splits[i]:splits[i+1]])
for i in range(len(splits) - 1)
]
```
> **Production Insight:** At Hureka, we default to 512-token chunks with 64-token overlap and recursive splitting for most projects. We add semantic chunking for legal contracts, medical records, and financial reports — document types where a single paragraph often contains a self-contained factual claim worth preserving intact. The overhead is worth it for these verticals.
### Embedding Models
The embedding model maps text to a dense vector in high-dimensional space such that semantically similar texts are geometrically close. The choice of embedding model is critical and often underestimated. Key dimensions: model size (speed vs. quality), embedding dimension (storage vs. quality), context window (max text per embed call), and domain specialization. In 2026, the leading models are:
- **text-embedding-3-large (OpenAI):** 3072-dim, SOTA on MTEB leaderboard for general retrieval. Expensive at scale but unmatched for English-language QA tasks. Supports Matryoshka truncation to 256/512-dim without full quality loss.
- **all-MiniLM-L6-v2 (SBERT):** 384-dim, runs on CPU, excellent for prototyping and cost-sensitive deployments. Quality gap vs. large models is meaningful on complex reasoning tasks.
- **BGE-M3 (BAAI):** Multi-lingual, multi-granularity. Supports dense, sparse, and colbert-style multi-vector retrieval in a single model. Exceptional for multilingual enterprise deployments. Our go-to for non-English production systems at Hureka.
- **E5-Mistral-7B-instruct:** Instruction-tuned LLM used as an embedder. Best-in-class for complex domain-specific retrieval. Requires GPU inference. Justified for high-stakes use cases.
### Vector Databases
The vector database stores your embeddings and serves ANN queries. It is not just a numpy array — a production vector DB handles persistence, HNSW indexing, metadata filtering, scalability, and in many cases hybrid search. Selection criteria: managed vs. self-hosted, filtering capabilities, scalability model, operational complexity, and cost. Detailed comparison in the table section below.
## Implementation: Step-by-Step Guide
Below is a complete, production-oriented RAG implementation using Qdrant as the vector database, OpenAI for embeddings, and BGE-reranker for cross-encoder reranking. This is close to what we deploy at Hureka for document intelligence products, minus client-specific business logic.
```python
# pip install qdrant-client openai sentence-transformers rank-bm25
from qdrant_client import QdrantClient, models
from openai import OpenAI
from sentence_transformers import CrossEncoder
from rank_bm25 import BM25Okapi
import uuid, hashlib
from typing import Optional
COLLECTION = "enterprise_docs"
EMBED_DIM = 3072 # text-embedding-3-large
oai = OpenAI()
qdrant = QdrantClient(url="http://localhost:6333")
reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")
# ── Step 1: Create collection ───────────────────────────────────────
def ensure_collection():
if qdrant.collection_exists(COLLECTION):
return
qdrant.create_collection(
collection_name=COLLECTION,
vectors_config=models.VectorParams(
size=EMBED_DIM,
distance=models.Distance.COSINE,
on_disk=True # memory-mapped for large corpora
),
optimizers_config=models.OptimizersConfigDiff(
indexing_threshold=20_000 # defer HNSW until 20k vectors
),
hnsw_config=models.HnswConfigDiff(
m=16, # graph connections per layer
ef_construct=200, # build-time beam width (quality↑, speed↓)
full_scan_threshold=10_000
)
)
# ── Step 2: Embed and upsert chunks ────────────────────────────────
def embed_texts(texts: list[str]) -> list[list[float]]:
resp = oai.embeddings.create(
model="text-embedding-3-large",
input=texts,
dimensions=EMBED_DIM
)
return [e.embedding for e in resp.data]
def upsert_chunks(chunks: list[DocumentChunk], batch_size: int = 64):
ensure_collection()
for i in range(0, len(chunks), batch_size):
batch = chunks[i:i+batch_size]
texts = [c.text for c in batch]
vectors = embed_texts(texts)
points = [
models.PointStruct(
id=str(uuid.uuid4()),
vector=vec,
payload={
"text": chunk.text,
"doc_id": chunk.doc_id,
"page": chunk.page,
"section": chunk.section,
"chunk_hash": hashlib.md5(chunk.text.encode()).hexdigest(),
**chunk.metadata
}
)
for chunk, vec in zip(batch, vectors)
]
qdrant.upsert(collection_name=COLLECTION, points=points)
print(f"Upserted batch ${i//batch_size + 1}, ${len(batch)} chunks")
# ── Step 3: Hybrid search (dense + BM25 + RRF fusion) ──────────────
def hybrid_search(
query: str,
top_k: int = 20,
filter_doc_id: Optional[str] = None
) -> list[dict]:
# Dense retrieval
q_vec = embed_texts([query])[0]
flt = models.Filter(
must=[models.FieldCondition(
key="doc_id",
match=models.MatchValue(value=filter_doc_id)
)]
) if filter_doc_id else None
dense_results = qdrant.search(
collection_name=COLLECTION,
query_vector=q_vec,
limit=top_k * 2, # over-fetch for RRF
query_filter=flt,
with_payload=True
)
# Sparse BM25 retrieval (over cached corpus texts)
all_docs = [r.payload["text"] for r in dense_results]
tokenized = [d.lower().split() for d in all_docs]
bm25 = BM25Okapi(tokenized)
bm25_scores = bm25.get_scores(query.lower().split())
# Reciprocal Rank Fusion (k=60 is standard)
def rrf(dense_rank: int, sparse_rank: int, k: int = 60) -> float:
return 1 / (k + dense_rank) + 1 / (k + sparse_rank)
bm25_ranks = sorted(range(len(bm25_scores)),
key=lambda i: bm25_scores[i], reverse=True)
bm25_rank_map = {idx: rank for rank, idx in enumerate(bm25_ranks)}
scored = [
{
"text": dense_results[i].payload["text"],
"payload": dense_results[i].payload,
"rrf_score": rrf(i, bm25_rank_map[i])
}
for i in range(len(dense_results))
]
return sorted(scored, key=lambda x: x["rrf_score"], reverse=True)[:top_k]
# ── Step 4: Rerank with cross-encoder ──────────────────────────────
def rerank(query: str, candidates: list[dict], top_n: int = 5) -> list[dict]:
pairs = [(query, c["text"]) for c in candidates]
scores = reranker.predict(pairs)
for i, c in enumerate(candidates):
c["rerank_score"] = float(scores[i])
return sorted(candidates, key=lambda x: x["rerank_score"], reverse=True)[:top_n]
# ── Step 5: Generate answer ─────────────────────────────────────────
def rag_query(question: str, filter_doc_id: Optional[str] = None) -> str:
candidates = hybrid_search(question, top_k=20, filter_doc_id=filter_doc_id)
top_chunks = rerank(question, candidates, top_n=5)
context = "\n\n---\n\n".join(
f"[Source: ${c['payload'].get('doc_id','?')}, p.${c['payload'].get('page','?')}]\n${c['text']}"
for c in top_chunks
)
resp = oai.chat.completions.create(
model="gpt-4o-2026",
messages=[
{"role": "system", "content": (
"Answer the question using ONLY the provided context. "
"If the answer is not in the context, say so explicitly. "
"Cite the source for each factual claim."
)},
{"role": "user", "content": f"Context:\n${context}\n\nQuestion: ${question}"}
],
temperature=0.1
)
return resp.choices[0].message.content
```
## Production Patterns and Best Practices
What separates a demo RAG from a production RAG is not the happy path — it is everything that goes wrong in the real world. Here are the patterns we have standardized on at Hureka after shipping to enterprise clients with millions of documents.
### Idempotent Indexing with Content Hashing
Documents change. Re-indexing naively leads to duplicate chunks in your vector database, degrading retrieval quality. The solution: hash the content of each chunk at upsert time (we use MD5 on the text). Before upserting, check if a vector with that content hash already exists. If it does, skip it. If the document is being updated (same doc_id, different content), delete existing vectors for that doc_id first, then re-index. Qdrant makes this easy with `delete` by payload filter. This pattern also makes your indexing pipeline safely re-runnable — critical for production where pipelines fail and retry.
### Metadata Filtering as a First-Class Concern
Never treat your RAG system as a single global search over all documents. Real enterprise deployments have access control (user A should not see user B's documents), document freshness requirements (only search docs from the last 12 months), and domain scoping (a legal query should only search legal documents). Design your metadata schema at the start, not as an afterthought. In Qdrant, every point's payload is indexed and filterable. We routinely filter by `tenant_id`, `doc_category`, `created_at`, and `confidentiality_level` in the same query that does ANN search — with zero performance penalty because Qdrant applies the filter on the HNSW graph traversal itself.
### Async Indexing with a Queue
Never index documents synchronously in your API request handler. Embedding 100 pages of PDF takes 15–30 seconds. Use a task queue (Celery + Redis, or AWS SQS + Lambda, or Temporal for complex workflows). The user uploads a document → you return a job ID immediately → the indexing worker processes asynchronously → you expose a `/status/{job_id}` endpoint. We also use this queue to batch embedding calls — accumulating 64+ texts before calling the embedding API — which reduces cost by 40–60% compared to single-text requests.
### Context Window Budget Management
Injecting 5 chunks at 512 tokens each = 2,560 tokens of context before your query and system prompt. At scale, with complex queries requiring more context, you can easily blow past context limits. We implement a context budget: the system calculates available context tokens (model limit − system prompt − query − response buffer), then greedily fills chunks starting from highest rerank score until the budget is exhausted. Always measure this — we have seen systems fail silently when context overflow causes the LLM to ignore chunks injected near the limit.
### Parent-Child Chunking
This is one of the highest-ROI patterns we have found. Index *small* chunks (128 tokens) for precise retrieval, but when a small chunk is retrieved, expand it by fetching its parent chunk (512 tokens) for injection into the context. Small chunks give you retrieval precision; large parent chunks give you the context the LLM needs to actually answer the question. In Qdrant we store the parent chunk ID in each small chunk's payload and fetch the parent via a separate lookup.
### Query Transformation with HyDE
HyDE (Hypothetical Document Embeddings) is a powerful technique for queries that are phrased differently than the documents they should match. Instead of embedding the raw query, you ask an LLM to generate a hypothetical document that would answer the query, then embed that hypothetical document. The hypothesis reads much more like the indexed documents, dramatically improving retrieval recall for information-seeking queries. We see 15–25% improvement in answer relevance on complex domain-specific queries when using HyDE.
```python
def hyde_embed(query: str) -> list[float]:
# Generate a hypothetical answer document
hyp = oai.chat.completions.create(
model="gpt-4o-mini", # fast and cheap for HyDE
messages=[{
"role": "user",
"content": f"Write a short factual paragraph that would answer: ${query}"
}],
max_tokens=200, temperature=0.3
).choices[0].message.content
return embed_texts([hyp])[0]
```
## Performance Optimization
A RAG pipeline that takes 8 seconds end-to-end is not production-ready. Here are the optimizations that move the needle, with approximate impact based on our production benchmarks.
### Embedding Caching
Cache query embeddings with a short TTL (60 seconds). In most products, users rephrase the same question slightly differently. Cache miss rate in our production systems is typically 30–40%, meaning 60–70% of embedding API calls can be eliminated. Use Redis with the query text (lowercased, stripped) as the key and the float vector as a msgpack-serialized value. Impact: **~200ms reduction** in P50 latency on repeated queries.
| Optimization Level | Latency (P50) |
|---|---|
| No caching | 1,240 ms |
| Embedding cache hit | 690 ms |
| + Rerank cache | 470 ms |
| + Async prefetch | 335 ms |
### HNSW Tuning
The HNSW `ef` parameter controls the beam width during search (not to be confused with `ef_construct` used at index time). Higher `ef` = higher recall but slower search. At Hureka we tune `ef` per collection: for collections up to 500k vectors we use `ef=128`; for larger collections we profile recall vs. latency and often find that `ef=64` gives 99.2% of the recall at 40% of the latency. Always benchmark recall at your actual corpus size — HNSW recall degrades non-linearly as you add vectors without rebalancing the graph.
### Quantization
Scalar quantization (SQ8 — compressing float32 to int8) reduces memory footprint by 4x with ~1% recall loss. Product quantization (PQ) goes further (8–16x compression) with ~3–5% recall loss. For most applications, SQ8 is the right tradeoff. Qdrant supports both natively. On a corpus of 10M vectors at 1536-dim, SQ8 reduces memory from ~59GB to ~15GB, making a 32GB server viable instead of requiring 128GB RAM.
### Parallelizing Retrieval and Reranking
If your query requires searching multiple collections or document categories, run those searches in parallel with `asyncio.gather`. Reranking is CPU-bound (cross-encoder inference on CPU) — run your reranker on a separate worker pool so it does not block the main async event loop. With these changes, a multi-collection RAG query that took 3.2 seconds sequentially runs in 0.9 seconds with parallel collection search and async reranker offload.
## Common Mistakes and How to Avoid Them
**Mistake 1: Mismatched Embedding Models at Index and Query Time**
Using `text-embedding-ada-002` during indexing and `text-embedding-3-small` at query time. The vector spaces are incompatible — you will get completely random retrieval results without any error message. This is a silent failure that can take weeks to diagnose in production.
**Fix:** Pin the exact embedding model name in a config file and validate it at startup. Include the model name in collection metadata and assert it matches before every query.
---
**Mistake 2: Chunk Size That Does Not Match Your Query Type**
Using 2048-token chunks for factoid QA (where you need precision) or 64-token chunks for summarization tasks (where you need context). The chunk size should match the granularity of the facts users are asking about, not what feels convenient to index.
**Fix:** Benchmark retrieval recall at 128, 256, 512, and 1024 tokens on a representative sample of 50–100 real user queries before committing to a chunk size.
---
**Mistake 3: No Deduplication in the Index**
Re-running your indexing pipeline without deleting existing vectors creates duplicate chunks. Retrieval returns the same text multiple times, wasting context window budget and confusing the LLM. We have seen systems where 40% of retrieved chunks were exact duplicates.
**Fix:** Hash chunk content at upsert time. Use upsert-by-hash semantics (check before insert) or delete-by-doc_id before re-indexing a document.
---
**Mistake 4: Injecting Too Much Context**
More context is not always better. LLMs have a "lost in the middle" problem — attention degrades for content in the middle of a very long context. Injecting 20 chunks at 512 tokens each (10k tokens of context) often performs worse than injecting the top-5 chunks at 2.5k tokens.
**Fix:** Measure answer quality vs. number of injected chunks. For most use cases, 3–6 well-ranked chunks outperforms 15–20 loosely ranked chunks. Use a reranker to ensure those 3–6 are the right ones.
---
**Mistake 5: Skipping Evaluation**
Deploying a RAG system without a quantitative evaluation harness means you have no idea whether changes improve or degrade quality. It is common to "improve" chunking and accidentally hurt retrieval recall without noticing for weeks.
**Fix:** Set up Ragas evaluation on a golden QA dataset of 50–100 questions before any production deployment. Run it in CI on every pipeline change.
---
**Mistake 6: Ignoring Sparse Retrieval Signals**
Pure dense retrieval fails on exact keyword matches — product codes, legal clauses with precise terminology, error codes, proper nouns. A user searching for "SEC Rule 10b-5" should get exact matches, not semantic neighbors. Dense embeddings often rank generic finance documents above the specific regulation.
**Fix:** Implement hybrid search with BM25 + RRF fusion. Sparse signals dominate on exact-match queries; dense signals dominate on semantic queries. Together they cover both cases.
## Real-World Use Cases
### Enterprise Legal Document Intelligence
A law firm with 200,000+ contracts needed to answer questions like "Do any of our supplier contracts have auto-renewal clauses expiring in Q3 2026?" We indexed all contracts with semantic chunking (preserving clause boundaries), added metadata for contract type, counterparty, and expiry date, and built metadata-filtered RAG that searched only relevant contract categories. The system reduced attorney review time by 70% for routine due diligence questions. Key technical decision: parent-child chunking with clauses as children and full contract sections as parents.
**Stack:** Qdrant (self-hosted), text-embedding-3-large, BGE-reranker-v2-m3, semantic chunking at clause boundaries.
### Financial Research Assistant
A hedge fund wanted a system that could answer questions against 10-K filings, earnings call transcripts, and analyst reports simultaneously. Multi-hop RAG was essential here — "How did Tesla's gross margin trend compare to the guidance they gave in their Q4 2025 earnings call?" requires retrieving from both the financial filing and the earnings transcript, then reasoning across both. We implemented a multi-hop planner that identifies which document categories to search and runs parallel retrievals before final synthesis. Latency: 4–8 seconds for complex cross-document queries.
**Stack:** pgvector (existing Postgres infra), GPT-4o, LangGraph for multi-hop orchestration.
### Healthcare Clinical Decision Support
A hospital network built a system for nurses to query clinical guidelines, drug interaction databases, and patient care protocols. Accuracy requirements were extreme — hallucinations in clinical settings are dangerous. We implemented strict faithfulness constraints: the system refuses to answer if retrieved context confidence falls below a threshold, and every response cites the specific guideline with page number. Ragas faithfulness score target: 0.96+. We also filtered by guideline publication date to ensure only current protocols were surfaced.
**Stack:** Weaviate (hybrid BM25 + vector built-in), Ragas CI evaluation on every update, clinical embedding model fine-tuned on medical text.
### E-Commerce Product Knowledge Base
A large retailer with 2M+ SKUs needed customer service agents to quickly answer detailed product questions (compatibility, specifications, return policies, warranty terms). We used BGE-M3's multi-vector retrieval for this — the hybrid dense+sparse single model simplified ops significantly. Product metadata (category, brand, SKU) was used as hard filters. We also implemented session-aware RAG: the conversation history was summarized and used to contextualize subsequent retrievals, so "does it come in blue?" correctly resolved "it" from earlier in the conversation.
**Stack:** Pinecone (managed, scales with inventory growth), BGE-M3 for multilingual support, conversation summarization for session context.
## Tool and Approach Comparison
| Database | Hosting | ANN Index | Hybrid Search | Filtering | Scaling | Best For | Est. Price |
|---|---|---|---|---|---|---|---|
| **Qdrant** | Self-host / Cloud | HNSW (tunable) | Native sparse+dense | Excellent (payload) | Horizontal sharding | Production workloads, complex filtering | Free OSS / $0.08/GB cloud |
| **Pinecone** | Managed only | Proprietary ANN | Hybrid (sparse+dense) | Metadata filters | Auto-scales | Zero ops overhead | $0.096/GB/mo + query |
| **Weaviate** | Self-host / Cloud | HNSW | BM25 + vector native | Where filters | Vertical + some horiz. | GraphQL API, hybrid built-in | Free OSS / usage-based |
| **pgvector** | Self-host (Postgres) | HNSW / IVFFlat | Manual (FTS + vector) | Full SQL | Limited (<5M vectors) | Existing Postgres stacks | Free (Postgres cost) |
| **Chroma** | Self-host (embedded) | HNSW (basic) | Limited | Basic metadata | Single-node only | Local dev, prototyping | Free |
| **Milvus** | Self-host / Zilliz cloud | HNSW / IVF / DiskANN | Hybrid supported | Good | Distributed native | Billion-scale, GPU acceleration | Free OSS / Zilliz pricing |
| **Redis Vector** | Self-host / Redis Cloud | HNSW / FLAT | Limited sparse | Tag/numeric filters | Redis cluster model | Low-latency, existing Redis users | Redis Cloud pricing |
| **OpenSearch k-NN** | Self-host / AWS managed | HNSW / FAISS | BM25 + kNN native | Full query DSL | AWS ecosystem scale | AWS shops, Elasticsearch migrations | AWS compute pricing |
> **Our Recommendation at Hureka:** For new production projects: **Qdrant** (self-hosted on Kubernetes for control, or Qdrant Cloud for managed). For teams already on Postgres with <2M vectors: **pgvector** eliminates a new infrastructure dependency. For billion-scale workloads: **Milvus** with GPU acceleration. For rapid prototyping: **Chroma** locally, then migrate to Qdrant for production.
## Future Trends in 2026 and Beyond
**GraphRAG is maturing fast.** Microsoft's GraphRAG introduced knowledge graph construction from documents as a pre-processing step, enabling multi-hop reasoning across entity relationships that chunk-based retrieval cannot handle. In 2026, GraphRAG frameworks have stabilized significantly. For domains with rich entity relationships — pharma, legal precedent, financial networks — GraphRAG consistently outperforms flat vector RAG on complex reasoning questions by 20–40% on answer relevance metrics. The cost is significant: graph construction is expensive and the retrieval query planner adds latency. But for the right use cases, it is worth it.
**Multi-modal RAG.** Every major vector database now natively handles image, audio, and video embeddings alongside text. Production multi-modal RAG — where a query about a product might retrieve both the product description and the specification diagram — is no longer experimental. Frameworks like LlamaIndex MultiModal and CLIP-based embedders have made this accessible. At Hureka we have shipped multi-modal RAG for a manufacturing client that queries technical manuals containing both text procedures and engineering diagrams.
**Agentic RAG.** Static RAG pipelines are giving way to agentic architectures where the retrieval strategy itself is decided by a planning LLM. The agent decides whether to do a single-shot retrieval, iteratively refine the query, search multiple collections in parallel, or invoke external tools (web search, code execution, SQL queries) when the vector database does not contain the answer. LangGraph, LlamaIndex Workflows, and Haystack's Canals are the leading frameworks for agentic RAG orchestration in 2026.
**Long-context LLMs changing the tradeoff.** With models now supporting 1M+ token contexts, there is a real question about whether RAG will remain necessary. The answer is nuanced: long-context models reduce the need for precise retrieval over small, well-defined corpora. But at enterprise scale — millions of documents, frequent updates, access control, cost constraints — RAG remains essential. Loading 200,000 contracts into a context window for every query is economically and latency-wise infeasible. RAG is not going away; it is evolving to become the routing layer that decides what goes into a long-context call.
**Streaming RAG.** Production systems in 2026 increasingly stream both retrieval status and generation tokens to the user. The UX impact is significant — a user who sees "Searching 47,000 policy documents... Found 5 relevant clauses... Generating answer..." has a fundamentally better experience than one who waits 6 seconds for a response. Qdrant's async client and the SSE streaming capabilities of modern LLM APIs make this pattern straightforward to implement.
## Conclusion and Next Steps
RAG is not a technology you pick up in an afternoon. The gap between a working prototype and a production-grade system that reliably returns accurate, grounded answers at scale is measured in the dozens of decisions this guide covers: chunk strategy, embedding model selection, HNSW tuning, hybrid search fusion, reranker selection, metadata schema design, context budget management, evaluation methodology, and failure mode handling. None of these decisions is optional if you care about production quality.
The good news is that the tooling ecosystem in 2026 is genuinely excellent. Qdrant, Ragas, BGE-M3, LlamaIndex, and the open-source reranker ecosystem have collectively solved the infrastructure problem. What differentiates high-quality production RAG today is not access to tools — it is engineering discipline: rigorous evaluation, systematic optimization, idempotent indexing, and the willingness to benchmark every change against your golden dataset before shipping it.
My recommended starting path: (1) Set up Ragas evaluation on 50 representative questions from your domain before writing a single line of retrieval code. This gives you a baseline to improve against. (2) Implement recursive chunking at 512 tokens as your starting point. (3) Use text-embedding-3-large or BGE-M3 depending on your language requirements. (4) Deploy Qdrant locally with Docker. (5) Add BM25 hybrid search and RRF fusion — this is a 2-hour addition that reliably improves recall by 10–20%. (6) Add a BGE reranker. (7) Measure, tune, ship. Iterate from there based on what your evaluation metrics tell you. The signal is always in the Ragas numbers — not your intuition about what should work.
Do not skip evaluation because it feels slow. A RAG system without Ragas is a RAG system you are flying blind. At Hureka we mandate a minimum evaluation suite before any client-facing deployment. The one time a team pushed without it, they had silently regressed context recall by 30% due to a chunking change, and did not find out until the client escalated. Measure everything.
## Building Production AI Agents in 2026: Architecture Patterns That Scale
- Date: 2026-06-25
- Category: AI Architecture
- Tags: AI Agents, Multi-Agent AI, LLM, Architecture, Production, LangGraph, FastAPI
- URL: https://dilsyno.com/blog/building-production-ai-agents-2026
- Read time: 18 min read
## Why Most AI Agent Projects Fail in Production
The excitement around AI agents is real — but so is the graveyard of agent projects that worked brilliantly in demos and collapsed under production load. After building agent systems for Hureka AI, AImind, and multiple enterprise clients, I have seen the same failure patterns repeat: uncontrolled token budgets, hallucinated tool calls, infinite loops, and zero observability.
This guide covers the architecture patterns that actually survive production traffic. Not toy examples — real patterns extracted from systems handling thousands of daily interactions.
If you are evaluating whether to build AI agents for your product, our [architecture review services](/services) can help you avoid expensive mistakes before writing a single line of code.
## The Three Agent Architectures That Matter
In 2026, three architectural patterns dominate production agent systems. Each has distinct tradeoffs in latency, reliability, and complexity.
### 1. ReAct (Reasoning + Acting)
The ReAct pattern interleaves reasoning steps with tool calls. The agent thinks, acts, observes the result, then thinks again.
```python
from langchain.agents import create_react_agent
from langchain_openai import ChatOpenAI
from langchain.tools import Tool
llm = ChatOpenAI(model="gpt-4o", temperature=0)
tools = [
Tool(name="search_knowledge", func=search_qdrant, description="Search internal knowledge base"),
Tool(name="query_database", func=run_sql_query, description="Run read-only SQL queries"),
Tool(name="send_notification", func=send_slack_alert, description="Send a Slack notification"),
]
agent = create_react_agent(llm, tools, prompt_template)
```
**When to use ReAct:**
- Simple, linear workflows (answer a question, look up data, summarize)
- Latency-tolerant applications
- Fewer than 5 tools
**When to avoid it:**
- Complex multi-step plans where the agent needs to reason about ordering
- High-throughput systems (each reasoning step is an LLM call)
### 2. Plan-Execute
The Plan-Execute pattern separates planning from execution. A planner LLM generates a full plan, then an executor runs each step sequentially.
```python
from langgraph.prebuilt import create_plan_and_execute_agent
class AgentState(TypedDict):
input: str
plan: list[str]
past_steps: Annotated[list[tuple], operator.add]
response: str
def planner(state: AgentState) -> AgentState:
"""Generate a multi-step plan using a reasoning model."""
plan_prompt = f"""
Task: {state['input']}
Available tools: search_knowledge, query_database, send_notification, generate_report
Create a step-by-step plan. Each step should be a single tool call.
Consider dependencies between steps.
"""
plan = llm.invoke(plan_prompt)
return {"plan": parse_plan(plan.content)}
def executor(state: AgentState, step: str) -> AgentState:
"""Execute a single step from the plan."""
result = agent_executor.invoke({"input": step})
return {"past_steps": [(step, result["output"])]}
graph = StateGraph(AgentState)
graph.add_node("planner", planner)
graph.add_node("executor", executor)
graph.add_node("replan", replan_if_needed)
graph.add_edge(START, "planner")
graph.add_edge("planner", "executor")
graph.add_conditional_edges("executor", should_continue, {"replan": "replan", "end": END})
```
**When to use Plan-Execute:**
- Complex tasks requiring 5+ steps
- When you need plan visibility and approval workflows
- Tasks where you can validate the plan before execution
### 3. Multi-Agent (Supervisor Pattern)
This is the pattern we use most at Hureka AI. A supervisor agent delegates to specialized sub-agents, each with their own tools and context.
```python
from langgraph.graph import StateGraph, START, END
class SupervisorState(TypedDict):
messages: Annotated[list, add_messages]
next_agent: str
context: dict
def supervisor(state: SupervisorState) -> SupervisorState:
"""Route to the appropriate specialist agent."""
routing_prompt = f"""
You are a supervisor managing these specialist agents:
- research_agent: Searches knowledge bases and documents
- data_agent: Queries databases and generates analytics
- action_agent: Performs actions (send emails, create tickets, update CRM)
Based on the conversation, which agent should handle the next step?
Respond with the agent name only.
"""
decision = llm.invoke(routing_prompt)
return {"next_agent": decision.content.strip()}
graph = StateGraph(SupervisorState)
graph.add_node("supervisor", supervisor)
graph.add_node("research_agent", research_agent_node)
graph.add_node("data_agent", data_agent_node)
graph.add_node("action_agent", action_agent_node)
graph.add_edge(START, "supervisor")
graph.add_conditional_edges("supervisor", route_to_agent)
for agent in ["research_agent", "data_agent", "action_agent"]:
graph.add_edge(agent, "supervisor")
```
## Tool Calling: The Hidden Complexity
Tool calling sounds simple until you hit production. Here are the patterns that matter:
### Structured Tool Definitions
Always use Pydantic models for tool inputs. Untyped tools lead to hallucinated parameters.
```python
from pydantic import BaseModel, Field
class SearchKnowledgeInput(BaseModel):
query: str = Field(description="Natural language search query")
collection: str = Field(default="default", description="Qdrant collection to search")
top_k: int = Field(default=5, ge=1, le=20, description="Number of results")
score_threshold: float = Field(default=0.7, ge=0.0, le=1.0)
@tool(args_schema=SearchKnowledgeInput)
def search_knowledge(query: str, collection: str = "default", top_k: int = 5, score_threshold: float = 0.7):
"""Search the internal knowledge base using semantic similarity."""
results = qdrant_client.search(
collection_name=collection,
query_vector=embed(query),
limit=top_k,
score_threshold=score_threshold
)
return format_results(results)
```
### Tool Call Validation and Retries
Never trust the LLM to call tools correctly on the first attempt:
```python
MAX_TOOL_RETRIES = 3
async def safe_tool_call(tool_fn, args: dict, retries: int = MAX_TOOL_RETRIES):
for attempt in range(retries):
try:
validated = tool_fn.args_schema(**args)
result = await tool_fn.ainvoke(validated.dict())
return result
except ValidationError as e:
if attempt == retries - 1:
return f"Tool call failed after {retries} attempts: {e}"
correction_prompt = f"Fix these arguments: {args}\nError: {e}"
args = await llm_fix_args(correction_prompt)
```
## Memory Patterns for Production Agents
Memory is what separates a chatbot from an agent. Three memory layers matter:
### Conversation Memory (Short-term)
Use a sliding window with summarization to keep token budgets under control:
| Strategy | Tokens/Turn | Best For |
|----------|-------------|----------|
| Full history | Grows unbounded | Demos only |
| Sliding window (last N) | Fixed | Simple chatbots |
| Summarize + recent | ~2000 + recent | Production agents |
| Vector-backed recall | ~1500 + relevant | Long-running agents |
### Semantic Memory (Long-term)
Store facts about the user/context in a vector database for cross-session recall:
```python
async def update_semantic_memory(user_id: str, conversation: list[dict]):
"""Extract and store facts from the conversation."""
extraction_prompt = f"""
Extract key facts from this conversation that should be remembered:
- User preferences
- Business context
- Decisions made
- Action items
Return as JSON array of facts.
"""
facts = await llm.ainvoke(extraction_prompt)
for fact in parse_facts(facts):
await qdrant_client.upsert(
collection_name="agent_memory",
points=[PointStruct(
id=str(uuid4()),
vector=embed(fact["text"]),
payload={"user_id": user_id, "fact": fact["text"], "timestamp": now()}
)]
)
```
### Procedural Memory (Skills)
Agents that learn from experience store successful tool-call sequences for reuse:
```python
async def store_successful_trajectory(task: str, steps: list[dict], outcome: str):
"""Store a successful task completion for future reference."""
trajectory = {
"task_description": task,
"steps": steps,
"outcome": outcome,
"timestamp": datetime.utcnow().isoformat()
}
vector = embed(task)
await qdrant_client.upsert(
collection_name="agent_trajectories",
points=[PointStruct(id=str(uuid4()), vector=vector, payload=trajectory)]
)
```
## Error Handling and Circuit Breakers
Production agents need circuit breakers to prevent cascading failures:
```python
from circuitbreaker import circuit
@circuit(failure_threshold=5, recovery_timeout=60)
async def call_llm(messages: list[dict], model: str = "gpt-4o"):
return await openai_client.chat.completions.create(
model=model, messages=messages, timeout=30
)
class AgentCircuitBreaker:
def __init__(self, max_iterations: int = 15, max_tokens: int = 50000):
self.max_iterations = max_iterations
self.max_tokens = max_tokens
self.iteration_count = 0
self.total_tokens = 0
def check(self, token_usage: int) -> bool:
self.iteration_count += 1
self.total_tokens += token_usage
if self.iteration_count > self.max_iterations:
raise AgentLoopError(f"Agent exceeded {self.max_iterations} iterations")
if self.total_tokens > self.max_tokens:
raise AgentBudgetError(f"Agent exceeded token budget: {self.total_tokens}")
return True
```
## Monitoring with LangFuse
Observability is non-negotiable. LangFuse gives you traces, cost tracking, and prompt versioning:
```python
from langfuse import Langfuse
from langfuse.callback import CallbackHandler
langfuse = Langfuse()
langfuse_handler = CallbackHandler(
trace_name="agent-execution",
user_id=user_id,
metadata={"agent_type": "research", "tenant_id": tenant_id}
)
result = agent.invoke(
{"input": user_query},
config={"callbacks": [langfuse_handler]}
)
trace = langfuse_handler.get_trace()
print(f"Total cost: ${trace.total_cost:.4f}")
print(f"Latency: {trace.latency_ms}ms")
print(f"Token usage: {trace.total_tokens}")
```
Key metrics to track in production:
| Metric | Target | Alert Threshold |
|--------|--------|-----------------|
| P95 Latency | < 5s | > 10s |
| Tool Call Success Rate | > 95% | < 90% |
| Token Cost / Request | < $0.05 | > $0.15 |
| Agent Loop Rate | < 2% | > 5% |
| User Satisfaction | > 4.2/5 | < 3.5/5 |
## Lessons from Building Hureka AI and AImind
After shipping agent systems that handle real production traffic, here is what I would tell my past self:
1. **Start with ReAct, graduate to Multi-Agent.** Do not build a multi-agent system until a single agent genuinely cannot handle the scope.
2. **Budget tokens religiously.** Set hard caps per turn, per session, and per user. One runaway agent can burn hundreds of dollars.
3. **Log everything with LangFuse.** You cannot debug what you cannot see. Every LLM call, tool call, and decision should be traced.
4. **Test with adversarial inputs.** Users will ask your agent to do things you never imagined. Build guardrails, not prayers.
5. **Use typed tools.** Pydantic schemas for every tool input. No exceptions.
## Conclusion
Building production AI agents is fundamentally an architecture problem, not a prompt engineering problem. The patterns in this guide — ReAct, Plan-Execute, Multi-Agent with proper memory, error handling, and monitoring — are battle-tested across real systems.
The difference between a demo agent and a production agent is the boring engineering: circuit breakers, token budgets, tool validation, and observability.
If you are planning to build an AI agent system and want to avoid the expensive mistakes, [get in touch](/contact) for an architecture review. We have helped teams across healthcare, SaaS, and enterprise build agent systems that actually survive production. Check out our [case studies](/case-studies) to see real results.
## LangGraph for Production: Stateful Multi-Agent Workflows That Actually Ship
- Date: 2026-06-20
- Category: AI Architecture
- Tags: LangGraph, LangChain, Multi-Agent AI, LLM, Python, Workflow, State Machine
- URL: https://dilsyno.com/blog/langgraph-production-ai-workflows
- Read time: 13 min read
## Why LangGraph and Not Just LangChain?
LangChain chains are linear. Real production agents need cycles: an agent calls a tool, evaluates the result, decides whether to call another tool, and only stops when a condition is met. That's a graph, not a chain — and LangGraph models it natively.
After shipping LangGraph in three production systems at Hureka, I now reach for it whenever a workflow has branching, retries, or multiple agents collaborating.
## Modeling State as a Graph
```python
from langgraph.graph import StateGraph, END
from typing import TypedDict, Annotated, Sequence
from langchain_core.messages import BaseMessage
class AgentState(TypedDict):
messages: Annotated[Sequence[BaseMessage], "Conversation history"]
plan: list[str]
completed: list[str]
needs_human: bool
def planner(state: AgentState) -> AgentState:
plan = llm_plan(state["messages"])
return {"plan": plan, "completed": [], "needs_human": False}
def executor(state: AgentState) -> AgentState:
next_step = state["plan"][len(state["completed"])]
result = execute_step(next_step)
return {"completed": state["completed"] + [result]}
def router(state: AgentState) -> str:
if state["needs_human"]:
return "human"
if len(state["completed"]) < len(state["plan"]):
return "executor"
return END
graph = StateGraph(AgentState)
graph.add_node("planner", planner)
graph.add_node("executor", executor)
graph.add_node("human", human_review)
graph.set_entry_point("planner")
graph.add_conditional_edges("executor", router)
graph.add_edge("planner", "executor")
graph.add_edge("human", "executor")
app = graph.compile()
```
## Persistence and Resumability
LangGraph's checkpointer saves state after every node — your workflow survives crashes, restarts, and long-running human review delays.
```python
from langgraph.checkpoint.postgres import PostgresSaver
checkpointer = PostgresSaver.from_conn_string(DB_URL)
app = graph.compile(checkpointer=checkpointer)
# Resume by thread_id — picks up exactly where it left off
config = {"configurable": {"thread_id": "user-abc-session-42"}}
result = await app.ainvoke({"messages": [user_input]}, config=config)
```
## Human-in-the-Loop Without Polling
```python
graph.add_node("human", lambda s: {"needs_human": True})
app = graph.compile(
checkpointer=checkpointer,
interrupt_before=["human"] # Pause graph, return control
)
# Frontend polls for paused threads
paused_state = app.get_state(config)
if paused_state.next == ("human",):
human_decision = await get_human_approval(paused_state)
await app.aupdate_state(config, {"needs_human": False})
await app.ainvoke(None, config=config) # Resume
```
## Lessons from Production
1. **Type your state** — TypedDict catches 80% of bugs before runtime
2. **Keep nodes pure** — A node should take state and return a partial update, nothing else
3. **Use checkpointers from day one** — Adding persistence later means rewriting
4. **Visualize the graph** — `app.get_graph().draw_mermaid()` saves hours in code review
5. **Test the router functions separately** — Routing logic is the most error-prone part
## Vector Database Showdown 2026: Qdrant vs Pinecone vs Weaviate vs pgvector
- Date: 2026-06-12
- Category: RAG Systems
- Tags: Qdrant, Pinecone, Weaviate, pgvector, Vector Search, RAG, Comparison
- URL: https://dilsyno.com/blog/vector-database-comparison-2026
- Read time: 11 min read
## The TL;DR
| Need | My Pick |
|------|---------|
| Self-hosted, fastest, most flexible | Qdrant |
| Already on Postgres, < 1M vectors | pgvector |
| Schema-rich, GraphQL native | Weaviate |
| Zero ops, willing to pay | Pinecone |
## Benchmark Setup
I indexed 1M 768-dim embeddings (BGE-base) across all four. Query workload: 1000 mixed queries (with and without metadata filters). Hardware: 4 vCPU, 16GB RAM.
## Latency at p95
| Database | No filter | With filter | Hybrid (dense+sparse) |
|----------|-----------|-------------|----------------------|
| Qdrant | 18ms | 22ms | 38ms |
| Pinecone | 32ms | 35ms | 52ms |
| Weaviate | 24ms | 28ms | 45ms |
| pgvector (HNSW) | 35ms | 30ms (filter wins) | n/a |
## Qdrant: My Default Choice
```python
from qdrant_client import QdrantClient, models
client = QdrantClient("http://localhost:6333")
client.create_collection(
"docs",
vectors_config=models.VectorParams(size=768, distance=models.Distance.COSINE),
quantization_config=models.ScalarQuantization(
scalar=models.ScalarQuantizationConfig(type=models.ScalarType.INT8)
),
)
```
Why I pick Qdrant 80% of the time:
- Best in-memory + on-disk hybrid (RAM stays bounded as collection grows)
- Built-in scalar/binary quantization — 4× memory reduction with minimal recall loss
- Rich payload filtering with composite conditions
- Native Rust performance, Docker-friendly self-hosting
## pgvector: When You Already Have Postgres
```sql
CREATE EXTENSION vector;
CREATE TABLE docs (id BIGSERIAL PRIMARY KEY, content TEXT, embedding vector(768));
CREATE INDEX ON docs USING hnsw (embedding vector_cosine_ops);
SELECT id, content, 1 - (embedding <=> $1) AS similarity
FROM docs
WHERE org_id = $2 AND created_at > NOW() - INTERVAL '30 days'
ORDER BY embedding <=> $1
LIMIT 5;
```
The killer feature: **JOIN with your existing tables**. Filtering by tenant, date range, or user permissions becomes a normal SQL WHERE clause.
## Decision Framework
1. **< 100K vectors and already using Postgres** → pgvector. Don't add another database.
2. **Multi-tenant SaaS, self-hosted, > 1M vectors** → Qdrant. Best price/performance.
3. **Need GraphQL + schema modeling** → Weaviate.
4. **Don't want to operate infrastructure, money is no object** → Pinecone.
The vector DB market changes fast. Re-evaluate every 6 months.
## Pages
- [Home](https://dilsyno.com)
- [About](https://dilsyno.com/about)
- [Services](https://dilsyno.com/services)
- [Projects](https://dilsyno.com/projects)
- [Case Studies](https://dilsyno.com/case-studies)
- [Blog](https://dilsyno.com/blog)
- [Links](https://dilsyno.com/links)
- [Contact](https://dilsyno.com/contact)
- [Sitemap](https://dilsyno.com/sitemap.xml)