What "cloud-native" actually buys you for an AI platform
Every “AI platform” pitch starts with the model. The interesting engineering is almost never there — it’s in the layer underneath that most people never think about until it breaks at 2 a.m.
Containers get you portability, not efficiency
Packaging inference workloads as containers is table stakes at this point. What actually matters is what you do with that portability: GPU bin-packing across a shared node pool, right-sizing memory limits per model instead of copy-pasting one manifest everywhere, and treating cold-start latency as a first-class metric instead of an afterthought. A cloud-native AI platform that just means “Kubernetes plus a model server” hasn’t done the hard part yet.
Autoscaling has to understand the workload, not just CPU
Standard HPA metrics (CPU, memory) are close to useless for GPU-bound inference. The signal that actually predicts whether you need another replica is request queue depth or GPU utilization — and even then, GPU node provisioning is slow enough that naive reactive autoscaling means real users eating a cold-start penalty. The platforms that get this right pre-warm capacity against traffic patterns instead of purely reacting to load after it arrives.
Cost is an architecture decision, not a FinOps afterthought
The single biggest lever is usually the boring one: which requests actually need the expensive model, and which can be served by something smaller and cheaper. A tiered serving strategy — cheap model first, escalate to the expensive one only when confidence is low — routinely cuts inference spend more than any amount of infrastructure tuning. Treating this as a Day 2 FinOps problem instead of a Day 0 architecture decision is the most common expensive mistake I see.
Observability is different here than for a typical service
Latency and error rate still matter, but they’re not sufficient. You also need visibility into things a traditional APM was never built for: token throughput, batch efficiency, model version skew across replicas during a rollout, and drift in the shape of incoming requests. Most teams bolt this on after their first confusing incident. It’s cheaper to build it in from the start.
None of this is exotic. It’s the same platform-engineering discipline — capacity planning, autoscaling, cost visibility, observability — applied to a workload shape that’s newer than the discipline itself.