System design for multi-tenant ML infrastructure: isolation, quotas and backpressure

Generated by Claude through GigAI 0.1.11.10 on 2026-10-12.

Concept: System design and architecture · also covers Distributed systems, ML platform · Step 34 of 35 in MLOps / ML platform engineer · version 1

Practice this

On this page

A shared ML platform runs training, batch and serving workloads from many teams on common clusters and accelerators. The design problem is to give each tenant isolation, a fair and predictable share of scarce capacity, and protection from other tenants under overload. This lesson covers the isolation layers, quota and fairness mechanisms, backpressure patterns and cost attribution that make sharing safe.

Read more

How it is done in practice

Teams usually begin with one namespace per team and environment, stamped out by automation with RBAC bindings, a default deny network policy, a ResourceQuota, a LimitRange and cost labels. GPU nodes sit in dedicated pools behind taints, and a batch queueing layer handles guaranteed shares, borrowing and preemption for training jobs. Shared services such as the inference gateway, the tracking server and the orchestrator enforce per-tenant rate and concurrency limits and return fast rejections when saturated, with clients configured for bounded, jittered retries. Utilization and cost per tenant are reviewed regularly, quotas are adjusted from data, and the design is load tested with one deliberately abusive tenant to confirm others stay within their service objectives.

Technologies

Kubernetes namespacesResourceQuotaLimitRangeNetworkPolicyPriorityClassKueueVolcanovClustergVisorKyvernoOPA GatekeeperEnvoy

Diagram

flowchart TB
  A[Tenant A requests] --> GW[Gateway with per-tenant rate limits]
  B[Tenant B requests] --> GW
  GW -->|over limit| REJ[Fast reject]
  GW --> Q[Bounded queue with criticality]
  Q -->|overload| SHED[Shed low priority]
  Q --> ADM[Admission and quota check]
  ADM --> NSA[Namespace A quota and policies]
  ADM --> NSB[Namespace B quota and policies]
  NSA --> GPU[Shared GPU node pool]
  NSB --> GPU
  GPU --> COST[Usage and cost by tenant label]
flowchart TB
  A[Tenant A requests] --> GW[Gateway with per-tenant rate limits]
  B[Tenant B requests] --> GW
  GW -->|over limit| REJ[Fast reject]
  GW --> Q[Bounded queue with criticality]
  Q -->|overload| SHED[Shed low priority]
  Q --> ADM[Admission and quota check]
  ADM --> NSA[Namespace A quota and policies]
  ADM --> NSB[Namespace B quota and policies]
  NSA --> GPU[Shared GPU node pool]
  NSB --> GPU
  GPU --> COST[Usage and cost by tenant label]

Code

yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-a-compute
  namespace: team-a
spec:
  hard:
    requests.cpu: "200"
    requests.memory: 800Gi
    limits.memory: 1000Gi
    requests.nvidia.com/gpu: "8"
    pods: "300"
    persistentvolumeclaims: "50"

Subtopics

Tenancy models and isolation layers

Control plane isolation

Kubernetes documentation distinguishes multi-team tenancy, where tenants are teams inside one organization, from multi-customer tenancy, where tenants are external and untrusted. The basic unit is the namespace: RBAC roles are bound per namespace, quotas attach to it and most policies are scoped by it. A common convention is one namespace per team and environment, created by automation with the same baseline every time. When tenants need their own cluster-scoped resources, such as custom resource definitions, namespaces are not enough and teams move to virtual control planes or a cluster per tenant.

Data plane isolation

Workloads share nodes, a network and a kernel unless told otherwise. NetworkPolicy with a default deny rule stops cross-namespace traffic. Pod Security Standards block privileged containers. Dedicated node pools with taints and tolerations keep one tenant's pods off another's nodes, which matters for GPU nodes and for sensitive data. Sandboxed runtimes add a stronger boundary for untrusted code such as user notebooks. Storage isolation comes from per-tenant buckets or prefixes with scoped identities.

Quotas, limits and fair sharing of compute

ResourceQuota and LimitRange

A ResourceQuota caps the aggregate consumption of one namespace: total CPU and memory requests and limits, storage requests, object counts such as pods or services, and extended resources such as GPUs through a requests-prefixed resource name. When a quota for CPU or memory is active, every new pod must declare that value or it is rejected, so a LimitRange is normally added to supply defaults. Quotas can be scoped, for example by PriorityClass, so a tenant gets a small allowance of high priority pods and a larger one of preemptible pods. Quota is enforced at admission and does not affect pods already running.

Beyond static quotas

Static quotas strand capacity: a team that is idle still holds its GPUs on paper. Batch queueing systems add borrowing, where a tenant can exceed its guaranteed share while others are idle and is preempted when the owner returns. Design decisions include guaranteed versus burst share, preemption order, and whether jobs are admitted all at once so distributed training does not start with half its workers.

Backpressure and overload handling on shared services

Per-tenant limits and load shedding

The SRE book treats overload as something every service must plan for. Capacity should be measured in real resource cost, not raw request count, since requests differ widely in cost; for model serving this means tokens or GPU seconds. Each tenant gets a per-customer limit, and requests over the limit are rejected early and cheaply instead of queuing. Requests carry a criticality label so that under pressure the service sheds low priority traffic, such as batch scoring, before interactive traffic. Rejecting is better than degrading everyone.

Client behavior and retries

Rejections still cost the server something, so clients throttle themselves: adaptive throttling tracks requests sent against requests accepted and starts dropping locally when the ratio grows. Retries need budgets, both per request and per client, or a small overload becomes a retry storm that spreads through layers. In ML platforms the same ideas apply to bounded queues in front of inference servers, admission limits on pipeline submissions and concurrency caps per tenant on the orchestrator.

Cost attribution and capacity trade-offs per tenant

Making cost visible per tenant

The AWS Well-Architected Machine Learning Lens applies the cost optimization pillar across the ML lifecycle: tag resources, right-size training and inference, and choose pricing models to match workload behavior. On a shared platform this starts with mandatory labels for team, project and environment, injected by the golden path and enforced by admission policy, so every pod, volume and endpoint maps to an owner. Showback reports spending per tenant; chargeback bills it. Shared costs such as the control plane, idle GPU headroom and the platform's own services need an explicit allocation rule. Typical levers are interruptible capacity for fault-tolerant training with checkpointing, scale to zero for idle endpoints, smaller or quantized models for cost-sensitive serving, and quota sized from measured utilization.

Sources

Related lessons

Practice this
Self-service platforms: golden paths, templates and developer toolingWriting and reviewing design docs and architecture decision records