Security, identity and access, and compliance for ML platforms

Generated by Claude through GigAI 0.1.11.10 on 2026-10-12.

Concept: Security · also covers Identity and access, DevSecOps, Compliance and governance, Regulated industry · Step 31 of 35 in MLOps / ML platform engineer · version 1

An ML platform concentrates valuable things: training data, model weights, expensive compute and credentials to many systems. Securing it means giving every human and workload an identity with least privilege, proving where code and model artifacts came from, and running a governance process that manages AI-specific risk. This lesson covers Kubernetes RBAC, cloud IAM, supply chain integrity under SLSA, and the NIST AI Risk Management Framework.

How it is done in practice

Security is built into the paved path so that the secure option is the default one. A new team gets a namespace, a group binding, a service account mapped to a scoped cloud role and a bucket prefix from one request. Pipelines build images on hosted runners, sign them and attach provenance, and an admission policy verifies signatures before anything runs. Secrets come from a secret manager, not from environment files. Model promotion requires an approval recorded in the registry, with lineage and evaluation attached. Access is reviewed on a schedule using last-used data, and audit logs from the cluster and the cloud account flow to a central store the platform team cannot edit.

Technologies

Diagram

flowchart TB
  U[Engineer] -->|federated sign-in| G[IdP group]
  G --> RB[RoleBinding in team namespace]
  RB --> NS[Namespace workloads]
  NS --> SA[Service account]
  SA -->|temporary credentials| R[Scoped cloud role]
  R --> D[Team data prefix and model store]
  CI[Hosted build] -->|signed image and provenance| REG[Registry]
  REG --> ADM[Admission policy verifies signature]
  ADM --> NS

flowchart TB
  U[Engineer] -->|federated sign-in| G[IdP group]
  G --> RB[RoleBinding in team namespace]
  RB --> NS[Namespace workloads]
  NS --> SA[Service account]
  SA -->|temporary credentials| R[Scoped cloud role]
  R --> D[Team data prefix and model store]
  CI[Hosted build] -->|signed image and provenance| REG[Registry]
  REG --> ADM[Admission policy verifies signature]
  ADM --> NS

Code

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: team-fraud
  name: training-submitter
rules:
- apiGroups: ["kubeflow.org"]
  resources: ["pytorchjobs"]
  verbs: ["create", "get", "list", "watch", "delete"]
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list", "watch"]

Subtopics

Kubernetes RBAC for ML tenants

The part specific to ML platforms

RBAC rules are additive only, with no deny rules, so tenant isolation rests on what is never granted. For ML tenants, bind namespaced Roles to groups from the identity provider, not to individual users, and give training operators and pipeline controllers their own service accounts. Watch the permissions that escalate: creating pods lets a subject mount any secret in the namespace, and the bind, escalate and impersonate verbs bypass normal limits. Custom resources for training jobs and inference services need explicit rules, since built-in roles do not cover them.

Cloud IAM and workload identity

Temporary credentials everywhere

IAM best practice is to remove long-lived keys. Humans sign in through federation with an identity provider and multi-factor authentication, and workloads assume roles that issue temporary credentials. On Kubernetes, a service account is mapped to a cloud role, so a training job receives short-lived credentials scoped to its own bucket prefix and nothing else. Start from the permissions actually used, then tighten: generate policies from access activity, validate them with a policy analyzer, add conditions such as required encryption or source network, and use permission boundaries and organization-wide guardrails so a team admin cannot grant beyond a ceiling. Review and remove unused roles and permissions regularly.

Supply chain integrity with SLSA

Provenance and build levels

SLSA is a specification for securing the path from source to artifact. Its build track defines levels: at level 1 the build produces provenance describing how the artifact was made; at level 2 the build runs on a hosted platform that signs the provenance; at level 3 the platform is hardened so builds are isolated and the signing material is out of reach of the build steps. Consumers verify provenance against expectations before deploying. For ML, apply the same idea to model artifacts: record the training code commit, data snapshot, base image digest and pipeline run, sign the result, and have admission control reject unsigned images or models.

AI risk governance and LLM security

NIST AI RMF

The NIST AI Risk Management Framework is a voluntary framework organized around four functions. Govern sets policies, roles and accountability. Map establishes the context of a system and identifies its risks. Measure analyzes and tracks those risks with quantitative and qualitative methods. Manage prioritizes and acts on them. The platform supplies the evidence: lineage from data to deployed model, model cards, evaluation results, approval records, access logs and monitoring history. Building these as automatic by-products of the pipeline is far cheaper than assembling them for an audit.

LLM-specific controls

LLM applications add an attack surface where untrusted text can steer the model. Treat retrieved documents, tool outputs and user input as data, not instructions. Limit what a model can do through narrowly scoped tool permissions and require human approval for side effects. Run code execution tools in sandboxes with no ambient credentials and restricted network egress. Filter outputs for sensitive data, log prompts and tool calls for audit with redaction, and red-team the system regularly with injection and data exfiltration scenarios, tracking the results as regression tests.

Sources

• docKubernetes: Using RBAC Authorization (kubernetes.io)
• docSecurity best practices in IAM (docs.aws.amazon.com)
• specSLSA Specification v1.0 (slsa.dev)
• guideNIST AI Risk Management Framework (www.nist.gov)

Related lessons

• AWS and EKS foundations for an ML platform
• Kubernetes for ML workloads: containers, deployments and jobs
• System design for multi-tenant ML infrastructure: isolation, quotas and backpressure
• Model registry, packaging and versioning
• CI/CD for models and services with GitHub Actions
• Reproducible environments: GitOps with Argo CD and infrastructure as code with Terraform
• Agent architectures, agent frameworks and the Model Context Protocol