When Your Cluster Isn’t Sufficient: Introducing Virtual Pools in AKS
There’s a moment in every Kubernetes journey when the cluster begins to feel… heavy. Not broken, not misconfigured, but stretched in ways that weren’t obvious at first. You scale node pools, adjust autoscaling thresholds, maybe even add spot nodes. Yet, some workloads still don’t quite fit. They either need to scale instantly beyond your node capacity or represent spiky, unpredictable demand that makes your carefully tuned AKS cluster feel rigid.
This is where virtual node pools, also known as virtual pools in Azure Kubernetes Service, come into play.
The illusion of infinite capacity
A virtual pool in AKS is not a traditional node pool. There are no VMs backing it, no kubelet you manage, no OS patches to worry about. Instead, it is a projection of Azure Container Instances (ACI) into your Kubernetes cluster.
When you create a virtual node, AKS integrates ACI as a backend compute option. From the Kubernetes API viewpoint, it appears as just another node. You can schedule pods, add labels, and set taints on it. However, internally, these pods do not run on your cluster’s VM-based node pools; instead, they are deployed directly as ACI containers.
The key change is that you’re no longer limited by the capacity of your cluster’s node pools. Instead, you’re basically expanding your cluster into a serverless execution layer.
How it works under the hood
The virtual node is implemented using a virtual kubelet, which translates Kubernetes scheduling decisions into Azure Container Instance deployments. When a pod is assigned to this virtual node, the following occurs:
• The Kubernetes scheduler assigns the pod to the virtual node
• The virtual kubelet intercepts that scheduling decision
• It translates the pod spec into an ACI container group
• Azure spins up the container almost immediately
There is no node provisioning step, VM scale set expansion, or image pre-pulling on nodes. The latency profile differs significantly, especially during burst scenarios.
Enabling virtual pools in AKS
To enable a virtual node pool, your cluster must be deployed using Azure CNI networking, which is mandatory. ACI-backed pods need to connect to your virtual network, and this connection requires Azure CNI.
An Azure CLI flow looks like this:
#!/usr/bin/env bash
set -euo pipefail
RESOURCE_GROUP="rg-aks-virtualpool"
CLUSTER_NAME="aks-virtualpool-demo"
LOCATION="swedencentral"
VNET_NAME="aks-vnet"
SUBNET_NAME="aks-subnet"
ACI_SUBNET_NAME="aci-subnet"
# Create resource group
az group create \
--name $RESOURCE_GROUP \
--location $LOCATION
# Create VNet with two subnets
az network vnet create \
--resource-group $RESOURCE_GROUP \
--name $VNET_NAME \
--address-prefix 10.0.0.0/8 \
--subnet-name $SUBNET_NAME \
--subnet-prefix 10.240.0.0/16
# Create dedicated subnet for ACI
az network vnet subnet create \
--resource-group $RESOURCE_GROUP \
--vnet-name $VNET_NAME \
--name $ACI_SUBNET_NAME \
--address-prefix 10.241.0.0/16
# Register the ACI resource provider before enabling virtual nodes
az provider register \
--namespace Microsoft.ContainerInstance
while [[ "$(az provider show --namespace Microsoft.ContainerInstance --query registrationState -o tsv)" != "Registered" ]]; do
echo "Waiting for Microsoft.ContainerInstance registration..."
sleep 10
done
# Create AKS cluster with Azure CNI
az aks create \
--resource-group $RESOURCE_GROUP \
--name $CLUSTER_NAME \
--network-plugin azure \
--vnet-subnet-id $(az network vnet subnet show \
--resource-group $RESOURCE_GROUP \
--vnet-name $VNET_NAME \
--name $SUBNET_NAME \
--query id -o tsv) \
--enable-managed-identity \
--node-count 2 \
--node-vm-size Standard_D2s_v3 \
--generate-ssh-keys \
--ssh-access disabled
# Enable virtual node (ACI)
az aks enable-addons \
--resource-group $RESOURCE_GROUP \
--name $CLUSTER_NAME \
--addons virtual-node \
--subnet-name $ACI_SUBNET_NAME
At this point, your cluster contains a virtual node. If you inspect it:
kubectl get nodes -o wide
You’ll see something like:
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
aks-nodepool1-99106009-vmss000000 Ready <none> 11m v1.33.6 10.240.0.4 <none> Ubuntu 22.04.5 LTS 5.15.0-1102-azure containerd://1.7.30-2
aks-nodepool1-99106009-vmss000001 Ready <none> 11m v1.33.6 10.240.0.33 <none> Ubuntu 22.04.5 LTS 5.15.0-1102-azure containerd://1.7.30-2
virtual-node-aci-linux Ready agent 118s v1.25.0-vk-azure-aci-1.6.2 10.240.0.53 <none> <unknown> <unknown> <unknown>
Scheduling workloads onto the virtual pool
By default, nothing lands on the virtual node. You explicitly target it using node selectors or taints and tolerations.
A simple example pod:
apiVersion: v1
kind: Pod
metadata:
name: api-burst-sample
labels:
app: api
tier: burst
spec:
tolerations:
- key: virtual-kubelet.io/provider
operator: Equal
value: azure
effect: NoSchedule
containers:
- name: api
image: mcr.microsoft.com/azuredocs/aci-helloworld
resources:
requests:
cpu: 250m
memory: 256Mi
nodeSelector:
kubernetes.io/role: agent
type: virtual-kubelet
You can observe it:
kubectl get pod burst-workload -o wide
The node will show as the virtual node, but there is no backing VM.
A real scenario: burst scaling beyond cluster limits
Imagine a platform team managing an API infrastructure on AKS. Usually, the workload remains stable and comfortably fits within a few node pools. However, during peak times such as a marketing campaign or a product launch, traffic can surge significantly.
Traditional autoscaling with the Cluster Autoscaler causes delays because new nodes need to be provisioned, images must be pulled, and pods scheduled. Despite aggressive tuning, this process can still take several minutes. Now introduce a virtual pool.
You configure your deployment with a fallback:
• Primary scheduling on regular node pools
• Overflow scheduling on the virtual node
A pattern using multiple deployments can look like this:
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-primary
spec:
replicas: 2
selector:
matchLabels:
app: api
tier: primary
template:
metadata:
labels:
app: api
tier: primary
spec:
containers:
- name: api
image: mcr.microsoft.com/azuredocs/aci-helloworld
resources:
requests:
cpu: 250m
memory: 256Mi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-burst
spec:
replicas: 1
selector:
matchLabels:
app: api
tier: burst
template:
metadata:
labels:
app: api
tier: burst
spec:
tolerations:
- key: virtual-kubelet.io/provider
operator: Equal
value: azure
effect: NoSchedule
containers:
- name: api
image: mcr.microsoft.com/azuredocs/aci-helloworld
resources:
requests:
cpu: 250m
memory: 256Mi
nodeSelector:
kubernetes.io/role: agent
type: virtual-kubelet
Now you can scale the burst deployment dynamically:
kubectl scale deployment api-burst --replicas=5
Those additional replicas are instantly provisioned in ACI, without impacting your cluster capacity.
Output will look something like this:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
api-burst-649c89bcb7-kh9pv 1/1 Running 0 2m18s 10.241.0.6 virtual-node-aci-linux <none> <none>
api-burst-649c89bcb7-nfjbj 1/1 Running 0 2m18s 10.241.0.9 virtual-node-aci-linux <none> <none>
api-burst-649c89bcb7-sb4r8 1/1 Running 0 2m18s 10.241.0.7 virtual-node-aci-linux <none> <none>
api-burst-649c89bcb7-tlz7f 1/1 Running 0 2m18s 10.241.0.8 virtual-node-aci-linux <none> <none>
api-burst-649c89bcb7-vj9g9 1/1 Running 0 58m 10.241.0.5 virtual-node-aci-linux <none> <none>
Those additional replicas are immediately provisioned in ACI, without affecting your cluster capacity.
Where virtual pools shine
Virtual pools are not meant to replace node pools; rather, they serve as an extension mechanism. They are most effective when workload traits align with ACI’s strengths: quick startup times, minimal infrastructure management, and elastic scaling. Short-term jobs are ideal, such as CI/CD runners, batch processing, or event-triggered data transformation tasks, as they can execute immediately without waiting for node provisioning.
Event-driven architectures also benefit; integrating with KEDA enables scaling based on queue length or external signals, with overflow handled by the virtual pool. Additionally, they are useful for isolating untrusted or experimental workloads in ACI via the virtual node, helping contain potential issues within the core cluster. Where virtual pools break down
The abstraction is strong, but its trade-offs become quite apparent in real-world architectures.
Networking is a primary friction point. Although ACI integrates with your VNet, it lacks support for some Kubernetes networking features. Advanced CNI functions, network policies, and detailed observability tools such as Cilium or eBPF-based tracing are not provided to the same extent.
Storage presents another limitation; persistent volumes are not supported in the same way as with VM-backed nodes. If your workload requires stateful storage, virtual pools may not be suitable.
DaemonSets do not operate on virtual nodes. Consequently, your standard observability tools, security agents, or service mesh sidecars might not function as intended.
Latency-sensitive workloads can also be affected. Although startup is quick, networking paths and performance traits differ from those of VM-based nodes.
Cost is also a factor. ACI has a different billing model. For steady-state workloads, it can sometimes be more costly than using pods on reserved or spot-backed nodes.
Observability and operational reality
From a platform engineering point of view, virtual pools can lead to a split-brain operational situation.
Your cluster now runs across two execution environments:
• VM-based node pools with full Kubernetes control
• ACI-backed virtual nodes with limited control
Your observability setup should account for this boundary. Tools like Azure Monitor can collect logs and metrics, but for kernel-level details, such as what Inspektor Gadget or Cilium Hubble offer, ACI does not support these. This isn’t a flaw, but a limitation.
Understanding this boundary is crucial for architects. A mental model for architects
Think of virtual pools as an elasticity layer attached to your cluster, rather than as part of it. Your primary workloads run on node pools that you control for environment, networking, and security.
The virtual pool functions as a pressure relief valve, enabling you to reduce workloads during demand spikes or when dedicated nodes are unnecessary, effectively serving as a serverless extension.
This hybrid approach helps AKS appear less like a fixed cluster and more like an adaptable, evolving platform.
Closing reflection
Virtual pools in AKS may appear straightforward at first, but they fundamentally change how you approach capacity management. They combine Kubernetes with serverless ideas, allowing you to handle unpredictability without overprovisioning. However, they also expose some limitations of the Kubernetes abstraction, as not all behaviors remain consistent.
For cloud engineers and platform architects, the main advantage is understanding when to utilize virtual pools and, equally important, when to refrain from using them. Ultimately, architecture focuses on setting appropriate boundaries and understanding the consequences of exceeding them.