AI

Less interested in the model itself than in the infrastructure that makes it reliable, observable, and cheap enough to run in production.

The Infrastructure Problem

Model training gets the press. Model serving is where engineering matters. A 7B parameter LLM running on commodity GPUs beats a 70B model that takes 30 seconds per token. Quantization, batching, KV-cache management—these are architectural decisions, not ML wizardry.

Cost Optimization for LLMs

Token-by-token cost compounds fast. The difference between:

Calling GPT-4 (expensive per token, high latency)
Running Mistral 7B locally (cheap amortized, fast)
Using a mixture of experts (small model + fallback to large)

…is a 100x swing in monthly spend. Most teams pick the first because it’s easy. The systems that survive economically pick the third.

Observability for AI

Traditional observability (latency, error rate) isn’t enough. You need:

Token efficiency: How many tokens did this request use? Was that acceptable?
Model version tracking: Which model version served which request? Can we trace quality degradation to a release?
Output quality metrics: Not just “did it return?” but “was the response correct?” Automated scoring, user feedback loops.

The systems that fail silently are the ones where the model degrades gradually and nobody notices until customer churn appears in the dashboards.