Cloud platforms offer an extraordinary range of choices, but abundance can obscure the real design problem. The team must connect workload characteristics to architecture: batch or online inference, predictable or spiky traffic, latency sensitivity, data residency, operational expertise, and the consequences of an outage. A thoughtful platform decision is more useful than a generic preference for managed or self-managed services.
Managed services can reduce undifferentiated work, while self-managed systems can provide control where specialized requirements justify it. Serverless designs are attractive for irregular workloads, but cold starts, limits, and observability must be part of the decision. Dedicated capacity can provide predictable performance, but idle resources become a direct cost.
Design for the real bottleneck
Scale is not only request volume. Storage throughput, feature retrieval, network paths, model loading, queue depth, and human response time can all become the limiting factor. A billion-request system begins with a clear budget for latency, availability, and cost at each boundary. That budget lets teams compare architectures with evidence instead of slogans.
Infrastructure should also make the safe path easy: least-privilege identity, encrypted data, private connectivity where needed, repeatable provisioning, and a useful audit trail. The model cannot be reliable if the platform around it is opaque or impossible to rebuild.
Architecture questions
- What must be online, and what can be processed asynchronously?
- Which component owns the latency budget?
- How will capacity scale and what will scaling cost?
- Can the environment be recreated from versioned configuration?