Running AI Workloads on Azure Kubernetes Service: From Platform Control to Model Serving

Running AI Workloads on Azure Kubernetes Service: From Platform Control to Model Serving

Most teams begin their AI journey by using a managed model API. This approach works initially but can become problematic when costs become unpredictable at scale, data sovereignty or compliance issues prevent sending data to third-party endpoints, or the team needs a fine-tuned model or a custom inference runtime. At this stage, the focus shifts from simply consuming AI to actively operating it.

AKS enables platform teams to handle AI workloads just like any other production workloads: they can schedule, secure, monitor, scale, and govern them using the same cloud-native operating model they are already familiar with.

What Makes AI Workloads Different

AI workloads put pressure on a platform in ways that typical stateless web services do not.

GPU availability remains the primary constraint. GPUs are costly, regionally limited, and often have quotas. Poor scheduling leads to wasted spending or request starvation. AI workload container images are large, often multiple gigabytes for the runtime alone, excluding model weights. Model artifacts can reach tens of gigabytes, and loading them at startup causes cold-start latency not usually seen in standard API pods.

Inference traffic is often bursty. A batch job might use a GPU node intensively for hours, while a real-time inference endpoint may remain idle between requests. Distributed training requires coordination among nodes, adding overhead. Unlike web API inputs and outputs, AI workload inputs and outputs, such as prompts, embeddings, and completions, often contain sensitive data that requires controlled access and auditing.

These characteristics are not reasons to avoid Kubernetes; rather, they highlight why a platform with proper scheduling, autoscaling, security boundaries, and observability is essential.

AKS Capabilities That Matter for AI

The key advantage of AKS is not just its ability to run containers, but that it enables teams to manage high-value AI workloads with enterprise controls already integrated.

GPU node pools and autoscaling:AKS provides dedicated GPU-optimized VM families (NC, ND, NV series) as node pools. The cluster autoscaler can scale these pools down to zero when idle and up as necessary, reducing costs by eliminating idle GPU capacity. Furthermore, Node Auto-Provisioning automatically chooses node SKUs based on the needs of pending pods.

Workload Identity replaces static credentials in pods by federating Kubernetes service accounts with Azure AD. This allows a model-serving pod to access artifacts from Azure Blob Storage or retrieve secrets from Key Vault without relying on long-lived secrets within the cluster.

Networking in Azure CNI with network policies assigns each pod a routable IP address, enabling precise traffic rules between namespaces and workloads. Ingress options include solutions like Traefik and the managed Application Gateway Ingress Controller, which support TLS termination and path-based routing to multiple model endpoints.

Observabilitywith Azure Monitor, Managed Prometheus, and Grafana offers built-in metrics at both the cluster and workload levels. GPU utilization data from DCGM Exporter can be collected alongside typical pod metrics. Logs are sent to Log Analytics for querying and alerting.

Governance: Azure Policy for AKS imposes guardrails during admission, including mandatory resource limits, permitted image registries, and restrictions on privileged containers. Microsoft Defender for Containers enhances security by providing runtime threat detection and scanning for image vulnerabilities.

Model Serving with KAITO on AKS

KAITO (Kubernetes AI Toolchain Operator) is an open-source operator that makes it easier to self-host large language models on Kubernetes. The AKS AI toolchain operator add-on includes KAITO as a managed cluster component, eliminating the need for manual installation and maintenance.

Enable the add-on on an existing cluster:

az aks update \
  --name $CLUSTER_NAME \
  --resource-group $AZURE_RESOURCE_GROUP \
  --enable-ai-toolchain-operator \
  --enable-oidc-issuer

KAITO introduces a Workspace custom resource. When you apply a workspace manifest, the operator automatically provisions the appropriate GPU node pool without pre-creating it, then pulls the model and exposes an inference endpoint compatible with OpenAI. Deploy Phi-4 Mini.

kubectl apply -f https://raw.githubusercontent.com/kaito-project/kaito/refs/heads/main/examples/inference/kaito_workspace_phi_4_mini.yaml
kubectl get workspace workspace-phi-4-mini -w

Once the workspace reaches Ready, the model is available as an internal cluster service. Query it using a standard chat completions request:

export SERVICE_IP=$(kubectl get svc workspace-phi-4-mini \
  -o jsonpath='{.spec.clusterIP}')

curl http://$SERVICE_IP/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "phi-4-mini",
    "messages": [{"role": "user", "content": "What is AKS?"}]
  }'

KAITO uses vLLM as its backend, offering teams continuous batching, streaming responses, and efficient memory handling without needing manual setup. The model operates fully within the organization’s AKS environment, simplifying compliance with data residency and regulatory standards.

Before deploying to production, check the supported KAITO versions and model presets in the AKS add-on documentation, as the managed add-on and the upstream KAITO project may not always offer the same features at the same time.

Security and Governance Considerations

Model endpoints are application endpoints that require equivalent security measures as other production services, and sometimes even more, due to the sensitive data they handle and the potential threats like prompt injection, data extraction, and model abuse.

Use workload identity for all pod-to-Azure connections. Do not mount service account tokens to pods that don’t require them. Keep model access keys and API credentials in Azure Key Vault and expose them via the Secrets Store CSI Driver.

Implement network controls by applying Kubernetes Network Policies to limit pod access to model-serving endpoints. Use private ingress if external access isn’t needed. Terminate TLS at the ingress layer and enforce request authentication.

Specify container image digests for supply chain security in deployment manifests. Limit permissible registries via Azure Policy to ensure only images from trusted sources run in the cluster. Use Defender for Containers to scan images for vulnerabilities before deploying them to production.

Isolation and RBAC establish separate model-serving namespaces distinct from application namespaces. Implement the principle of least privilege with Kubernetes RBAC to ensure workloads access only the necessary resources. Enable audit logging on the control plane and direct logs to Log Analytics for retention and alerting.

Application-layer controls such as rate limiting, authentication, authorization, content safety filtering, and inference logging are not handled automatically by the platform. These need to be designed explicitly for each model endpoint, whether through an API gateway, a sidecar, or application code.

Operational Realities and the Decision Guide

GPU quota is limited and varies by region. Request it early, choose VM SKUs that match your model’s needs, and set resource limits on pods to prevent any workload from monopolizing a node. Cold starts for models, including image pulls and weight loading, can take several minutes on larger models. Use sufficient initialDelaySeconds in health probes to accommodate this. Since GPU autoscaling is slower and costlier than scaling web pods, plan capacity carefully and maintain monitoring dashboards and alerts before traffic increases.

Production fundamentals for AI workloads mirror those for other types: using PodDisruptionBudgets to ensure safe upgrades, topology spread constraints to maintain zone availability, separate node pools for various workload classes, and specific maintenance windows for node image updates.

Conclusion

AKS provides teams with a comprehensive platform for AI workloads, going beyond merely hosting containers. The most effective approach is focused and incremental: select a single workload, verify quotas, costs, latency, and security measures, automate deployment, and gradually expand once the operating model’s effectiveness is confirmed.

The true benefit of AKS for AI isn’t just running a model; it’s enabling teams to execute AI as a managed, observable, and scalable platform workload.