Small models are quietly winning the boring workloads

Frontier models get the headlines, but the workhorse deployments we set up this year tell a different story. Ticket routing, PII detection, invoice field extraction, sentiment tagging, log summarization: these tasks are increasingly served by small open-weight models running on infrastructure the client already owns.

The economics drive it. A fine-tuned 7B model on a single L4 GPU handles millions of classification requests per month at a fixed, predictable cost, with no per-token meter and no data leaving the VPC. For regulated industries, the data-residency argument alone closes the decision.

The catch is operational. Self-hosting a model means owning GPU provisioning, model updates, quantization choices, and latency SLOs. It is a platform-engineering commitment, not an API key. Teams that treat it casually end up with an unmonitored pet server that one engineer understands.

Our rule of thumb for clients: stay on hosted APIs while iterating on product-market fit for the feature. Move a workload in-house when three things are true: the task is stable, the volume is high enough that the meter hurts, and a platform team exists to own the runtime.

The likely end state for most mid-market stacks is hybrid: small local models for high-volume structured tasks, frontier APIs for open-ended reasoning, and a router that knows the difference.