[{"content":"\nTransferring a Dapr-enabled application from local Kubernetes to AKS should be straightforward. The annotations, app ID, application port, sidecar API, and workload manifest stay familiar. The key difference is ownership: a temporary local setup transforms into a shared runtime that requires a version policy, availability plan, network access, identity strategy, and support structure.\nThis post moves the sample from Installing Dapr on Kubernetes Locally with OrbStack to an existing AKS cluster. We will compare the open-source installation path with the AKS Dapr extension, install the extension, verify the control plane, and deploy the same NGINX workload.\nOutcome: an existing AKS cluster will run the Dapr control plane through the managed cluster-extension mechanism, and the same hello-dapr application will start with an injected sidecar.\nThis is Part 3 of the Dapr on Kubernetes and AKS series. It assumes you are familiar with the sidecar model and have access to an AKS cluster. Setting up a production-ready cluster is a different architectural task; this article concentrates specifically on Dapr.\nWhat moves and what changes The application boundary moves cleanly. Your Deployment still requests injection with dapr.io/enabled, identifies itself with dapr.io/app-id, and tells Dapr where the application listens with dapr.io/app-port.\nThe environment around that Deployment changes substantially:\nConcern OrbStack development cluster AKS cluster Cluster Local, single node, disposable Remote, managed Kubernetes control plane with production node pools Dapr installation Dapr CLI with --dev Prefer the AKS Dapr extension for Azure-managed lifecycle integration Runtime dependencies Development Redis and Zipkin Deliberately selected state, pub/sub, secret, and telemetry services Images Local image store or public registry Normally Azure Container Registry or another reachable registry Identity Local credentials and development components Microsoft Entra Workload Identity and least-privilege Azure access Networking Host-integrated local networking Azure CNI, ingress, egress, private endpoints, DNS, and policy choices Availability One node and no meaningful failure-domain test Multiple nodes and zones where the workload requires them Upgrades Reinstall when convenient Planned control-plane and sidecar version lifecycle The lesson is subtle: you are not migrating a local Dapr installation. You are deploying the same application contract into a differently operated environment.\nChoose one Dapr owner AKS supports two broad installation approaches:\nInstall open-source Dapr with the Dapr CLI or Helm. Install the Microsoft Dapr cluster extension through Azure CLI or infrastructure as code. Both produce Dapr control-plane workloads inside Kubernetes. The difference is how those resources are installed, configured, upgraded, and supported.\nDecision Dapr CLI or Helm AKS Dapr extension Best fit Portable Kubernetes ownership, custom Helm lifecycle, non-Azure consistency AKS platforms that want Azure cluster-extension lifecycle and support integration Installation owner Your platform automation Azure cluster-extension resource Version management You select and upgrade charts/runtime Extension version and release-train settings Default HA posture Must be selected deliberately HA is enabled by default in current extension defaults Configuration CLI flags or Helm values Extension configuration settings or Bicep properties Support entry point Dapr open-source project and your platform process Azure for extension operations; upstream Dapr for runtime behavior where applicable For this series, I use the AKS Dapr extension. It fits the managed-cluster operating model and makes the installation visible as an Azure resource. That does not make every Dapr runtime concern Microsoft\u0026rsquo;s responsibility, nor does it remove the need to test upgrades.\nThe non-negotiable rule is to choose one lifecycle owner. Microsoft explicitly recommends continuing to manage an extension-installed Dapr runtime through the extension. Running dapr upgrade -k or an unrelated Helm upgrade against the same installation can create configuration drift and conflicting ownership.\nIf your AKS cluster already contains Dapr installed with Helm or the CLI, do not install a second control plane. Use Microsoft\u0026rsquo;s documented OSS-to-extension migration path, which can adopt an existing Helm release. Inventory the release name, namespace, values, Dapr resources, and version before changing ownership.\nPrerequisites and variables You need:\nan Azure subscription and permission to manage the target AKS cluster; an existing AKS cluster; a current Azure CLI and kubectl; the k8s-extension Azure CLI extension; and the Dapr CLI if you want to use dapr status -k for verification. Set explicit variables rather than scattering resource names through commands:\nexport SUBSCRIPTION_ID=\u0026#34;\u0026lt;subscription-id\u0026gt;\u0026#34; export AKS_RESOURCE_GROUP=\u0026#34;\u0026lt;aks-resource-group\u0026gt;\u0026#34; export AKS_CLUSTER_NAME=\u0026#34;\u0026lt;aks-cluster-name\u0026gt;\u0026#34; export DAPR_EXTENSION_NAME=\u0026#34;dapr\u0026#34; Select the subscription and confirm the active identity:\naz account set --subscription \u0026#34;$SUBSCRIPTION_ID\u0026#34; az account show \\ --query \u0026#39;{subscription:name, subscriptionId:id, tenantId:tenantId}\u0026#39; \\ --output table Install or update the cluster-extension CLI support:\naz extension add --name k8s-extension --upgrade Register the required resource provider and feature if they are not already registered:\naz provider register \\ --namespace Microsoft.KubernetesConfiguration \\ --wait az feature registration create \\ --namespace Microsoft.KubernetesConfiguration \\ --name ExtensionTypes Registration is subscription-scoped and may take time. Check both states before continuing:\naz provider show \\ --namespace Microsoft.KubernetesConfiguration \\ --query registrationState \\ --output tsv az feature show \\ --namespace Microsoft.KubernetesConfiguration \\ --name ExtensionTypes \\ --query properties.state \\ --output tsv Both should report Registered.\nPoint kubectl at the intended AKS cluster Retrieve credentials and verify the context before installing a cluster-wide runtime:\naz aks get-credentials \\ --resource-group \u0026#34;$AKS_RESOURCE_GROUP\u0026#34; \\ --name \u0026#34;$AKS_CLUSTER_NAME\u0026#34; \\ --overwrite-existing kubectl config current-context kubectl get nodes -o wide This check matters even more after the local tutorial: the same workstation may have OrbStack, test AKS, and production AKS contexts. Never infer the target from the terminal prompt.\nBefore installing, look for an existing Dapr owner:\naz k8s-extension list \\ --cluster-type managedClusters \\ --cluster-name \u0026#34;$AKS_CLUSTER_NAME\u0026#34; \\ --resource-group \u0026#34;$AKS_RESOURCE_GROUP\u0026#34; \\ --output table helm list --all-namespaces kubectl get namespace dapr-system --ignore-not-found If an extension or Helm release already owns Dapr, stop and understand it. The correct next step may be an update or migration, not creation.\nInstall the Dapr extension For a production-oriented baseline, use the stable release train and disable automatic minor-version upgrades. Microsoft currently warns that automatic control-plane minor upgrades are more appropriate for development and test environments; production upgrades should be deliberate.\naz k8s-extension create \\ --cluster-type managedClusters \\ --cluster-name \u0026#34;$AKS_CLUSTER_NAME\u0026#34; \\ --resource-group \u0026#34;$AKS_RESOURCE_GROUP\u0026#34; \\ --name \u0026#34;$DAPR_EXTENSION_NAME\u0026#34; \\ --extension-type Microsoft.Dapr \\ --release-train stable \\ --auto-upgrade-mode none No --dev flag appears here. The extension installs the Dapr runtime, not a tutorial Redis or Zipkin stack. That is intentional: production dependencies should have explicit topology, security, persistence, backup, and ownership.\nThe extension\u0026rsquo;s current defaults enable high availability, Prometheus metrics, and mTLS. Do not mistake a default for a complete design. Review placement and Scheduler storage, topology spread, disruption budgets, resource requests, certificate lifecycle, and monitoring against your cluster standards before calling the runtime production-ready.\nThe extension needs outbound HTTPS access to Microsoft\u0026rsquo;s Dapr artifact location under mcr.microsoft.com/daprio, in addition to the normal AKS outbound requirements. Restricted-egress clusters must account for that dependency.\nVerify Azure and Kubernetes state First verify the Azure resource:\naz k8s-extension show \\ --cluster-type managedClusters \\ --cluster-name \u0026#34;$AKS_CLUSTER_NAME\u0026#34; \\ --resource-group \u0026#34;$AKS_RESOURCE_GROUP\u0026#34; \\ --name \u0026#34;$DAPR_EXTENSION_NAME\u0026#34; \\ --query \u0026#39;{state:provisioningState, version:version, autoUpgradeMode:autoUpgradeMode}\u0026#39; \\ --output table The provisioning state should become Succeeded. Then inspect the Kubernetes side:\nkubectl get pods -n dapr-system -o wide kubectl get deployments,statefulsets,services -n dapr-system dapr status -k You should recognize the same responsibilities from the OrbStack installation: operator, sidecar injector, Sentry, placement, and Scheduler. Replica counts and workload shapes differ because the extension uses an HA-oriented baseline.\nRepresentative output. Extension versions, replica counts, pod suffixes, nodes, and timing vary.\nAlso inspect the effective extension settings rather than relying on memory:\naz k8s-extension show \\ --cluster-type managedClusters \\ --cluster-name \u0026#34;$AKS_CLUSTER_NAME\u0026#34; \\ --resource-group \u0026#34;$AKS_RESOURCE_GROUP\u0026#34; \\ --name \u0026#34;$DAPR_EXTENSION_NAME\u0026#34; \\ --output json Store the desired configuration in Bicep, Terraform, or another reviewed deployment definition. A successful interactive command is a useful experiment, not a long-term source of truth.\nRedeploy the local sample The local sample used the public nginx:1.27-alpine image, so AKS can pull it without ACR integration in an unrestricted cluster. Create a namespace and deploy the companion manifest:\nkubectl create namespace dapr-demo \\ --dry-run=client \\ --output yaml | kubectl apply --filename - kubectl apply \\ --namespace dapr-demo \\ --filename hello-dapr-aks.yaml kubectl rollout status \\ deployment/hello-dapr \\ --namespace dapr-demo Download the AKS sample manifest, or save this YAML:\napiVersion: apps/v1 kind: Deployment metadata: name: hello-dapr spec: replicas: 2 selector: matchLabels: app: hello-dapr template: metadata: labels: app: hello-dapr annotations: dapr.io/enabled: \u0026#34;true\u0026#34; dapr.io/app-id: \u0026#34;hello-dapr\u0026#34; dapr.io/app-port: \u0026#34;80\u0026#34; dapr.io/sidecar-cpu-request: \u0026#34;100m\u0026#34; dapr.io/sidecar-memory-request: \u0026#34;128Mi\u0026#34; dapr.io/sidecar-cpu-limit: \u0026#34;500m\u0026#34; dapr.io/sidecar-memory-limit: \u0026#34;256Mi\u0026#34; spec: containers: - name: web image: nginx:1.27-alpine ports: - name: http containerPort: 80 resources: requests: cpu: 50m memory: 64Mi limits: cpu: 250m memory: 128Mi readinessProbe: httpGet: path: / port: http initialDelaySeconds: 2 periodSeconds: 5 --- apiVersion: v1 kind: Service metadata: name: hello-dapr spec: selector: app: hello-dapr ports: - name: http port: 80 targetPort: http The essential Dapr annotations are unchanged. The AKS version adds a second replica and explicit resource requests and limits for both the application and sidecar. These values are safe starting points for a tiny sample, not sizing recommendations for real workloads.\nVerify that both pods contain web and daprd:\nkubectl get pods \\ --namespace dapr-demo \\ --selector app=hello-dapr kubectl get pods \\ --namespace dapr-demo \\ --selector app=hello-dapr \\ --output jsonpath=\u0026#39;{range .items[*]}{.metadata.name}{\u0026#34; \u0026#34;}{.spec.containers[*].name}{\u0026#34;\\n\u0026#34;}{end}\u0026#39; Each pod should report 2/2 ready. Invoke the service through one Dapr sidecar using a temporary port forward:\nkubectl port-forward \\ --namespace dapr-demo \\ deployment/hello-dapr \\ 3500:3500 In another terminal:\ncurl --fail --show-error \\ http://localhost:3500/v1.0/invoke/hello-dapr/method/ The NGINX welcome page proves that the same Dapr invocation contract works on AKS.\nWhere ACR enters the path The public sample image hides an important production difference. Your own application image normally lives in Azure Container Registry. OrbStack can use a local build directly; AKS nodes need registry access.\nIf you want to exercise the ACR path without building a custom application yet, import the tutorial\u0026rsquo;s public NGINX image into your registry. ACR performs this copy directly, so Docker does not need to be running on your workstation:\nexport ACR_NAME=\u0026#34;\u0026lt;acr-name\u0026gt;\u0026#34; az acr import \\ --name \u0026#34;$ACR_NAME\u0026#34; \\ --source docker.io/library/nginx:1.27-alpine \\ --image hello-dapr:1.0.0 Confirm that the repository and tag are present:\naz acr manifest list-metadata \\ --registry \u0026#34;$ACR_NAME\u0026#34; \\ --name hello-dapr \\ --output table Resolve the tag to its immutable manifest digest and construct the complete image reference:\nDIGEST=\u0026#34;$( az acr manifest show-metadata \\ --registry \u0026#34;$ACR_NAME\u0026#34; \\ --name \u0026#34;hello-dapr:1.0.0\u0026#34; \\ --query digest \\ --output tsv )\u0026#34; ACR_LOGIN_SERVER=\u0026#34;$( az acr show \\ --name \u0026#34;$ACR_NAME\u0026#34; \\ --query loginServer \\ --output tsv )\u0026#34; IMAGE=\u0026#34;${ACR_LOGIN_SERVER}/hello-dapr@${DIGEST}\u0026#34; printf \u0026#39;%s\\n\u0026#39; \u0026#34;$IMAGE\u0026#34; The result has this form:\n\u0026lt;acr-name\u0026gt;.azurecr.io/hello-dapr@sha256:\u0026lt;digest\u0026gt; In hello-dapr-aks.yaml, replace the image in the web container:\ncontainers: - name: web image: \u0026lt;acr-name\u0026gt;.azurecr.io/hello-dapr@sha256:\u0026lt;digest\u0026gt; Use the actual value printed by the preceding command; do not include the angle brackets. Keeping the digest in the manifest makes the deployed image immutable, while the 1.0.0 tag remains a convenient name for finding that digest.\nFor a registry in the same tenant, attach ACR to AKS to grant the cluster identity pull access, and then verify that access:\naz aks update \\ --resource-group \u0026#34;$AKS_RESOURCE_GROUP\u0026#34; \\ --name \u0026#34;$AKS_CLUSTER_NAME\u0026#34; \\ --attach-acr \u0026#34;$ACR_NAME\u0026#34; az aks check-acr \\ --resource-group \u0026#34;$AKS_RESOURCE_GROUP\u0026#34; \\ --name \u0026#34;$AKS_CLUSTER_NAME\u0026#34; \\ --acr \u0026#34;${ACR_NAME}.azurecr.io\u0026#34; Apply the updated manifest and wait for the new image to roll out:\nkubectl apply \\ --namespace dapr-demo \\ --filename hello-dapr-aks.yaml kubectl rollout status \\ deployment/hello-dapr \\ --namespace dapr-demo Registry access belongs to the AKS node or kubelet identity. Dapr does not pull the application image and does not replace Kubernetes image authentication.\nWhen you have your own application and a Dockerfile, replace the import step with an ACR build. The command builds the image in Azure and pushes it to the same repository:\naz acr build \\ --registry \u0026#34;$ACR_NAME\u0026#34; \\ --image hello-dapr:1.0.0 \\ --file Dockerfile \\ . After the build, repeat the digest lookup and update the image reference in the manifest.\nComponents should change even when the app does not The OrbStack tutorial installed Redis and Zipkin for convenience. Do not export those development components and call the result production-ready.\nOn AKS, decide separately:\nwhich state store and pub/sub broker meet durability and delivery requirements; whether components authenticate with Microsoft Entra Workload Identity instead of keys; how secrets are stored and scoped; where traces, metrics, and logs are collected; which namespaces and app IDs may use each Dapr component; and who owns backup, recovery, upgrades, and incident response for every backing service. The application API may stay stable while the components become Azure Service Bus, Azure Cosmos DB, Azure Managed Redis, Azure Key Vault, and an OpenTelemetry pipeline. That separation is one of Dapr\u0026rsquo;s advantages, but it does not remove the operational characteristics of those services.\nVersioning is a two-step operation The extension manages the Dapr control plane. Existing application pods keep the sidecar image injected when they were created. After a tested control-plane update, restart Dapr-enabled workloads so newly created pods receive the intended sidecar version:\nkubectl rollout restart \\ deployment/hello-dapr \\ --namespace dapr-demo That control-plane/sidecar distinction is why production upgrades need a rollout plan, compatibility validation, and non-production rehearsal. Installing through a managed mechanism does not make application restarts disappear.\nThe AKS extension currently supports a rolling window that includes the current and previous Dapr versions. Check the versions available to your cluster rather than copying a version number from an article. The preview command under az k8s-extension extension-types can fail when its embedded API version is retired. The newer az aks extension type command does not currently allow Dapr, so query the documented cluster-extension API directly with az rest:\nif AKS_RESOURCE_ID=\u0026#34;$( az aks show \\ --resource-group \u0026#34;$AKS_RESOURCE_GROUP\u0026#34; \\ --name \u0026#34;$AKS_CLUSTER_NAME\u0026#34; \\ --query id \\ --output tsv )\u0026#34; \u0026amp;\u0026amp; [ -n \u0026#34;$AKS_RESOURCE_ID\u0026#34; ]; then az rest \\ --method get \\ --url \u0026#34;${AKS_RESOURCE_ID}/providers/Microsoft.KubernetesConfiguration/extensionTypes/Microsoft.Dapr/versions\u0026#34; \\ --url-parameters \\ api-version=2024-11-01-preview \\ releaseTrain=stable \\ showLatest=true \\ --query \u0026#39;value[].properties.version\u0026#39; \\ --output table else echo \u0026#34;Could not resolve the AKS resource ID.\u0026#34; \u0026gt;\u0026amp;2 fi If az aks show warns that its behavior has been altered by aks-preview and then reports an unsupported API version, remove that optional extension before retrying. It is not required for this article:\naz extension remove \\ --name aks-preview The clean handoff The move from OrbStack to AKS preserves the part developers depend on: the Dapr application contract. It replaces the disposable platform around that contract with an explicit operating model.\nThe practical rules are:\nChoose the extension or open-source lifecycle and keep one owner. Verify the Azure extension resource and the Kubernetes control plane. Keep application annotations portable. Replace development components with production services deliberately. Treat control-plane upgrades and sidecar rollouts as related but separate steps. Next in the series, we will use two services to explore Dapr service invocation, named discovery, namespaces, HTTP and gRPC choices, and resiliency policies.\nReturn to the Dapr on Kubernetes and AKS series index.\nSources Microsoft Learn — Install the Dapr extension for AKS Microsoft Learn — Configure the Dapr extension Microsoft Learn — Dapr extension overview Microsoft Learn — Migrate Dapr OSS to the AKS extension Microsoft Learn — Integrate Azure Container Registry with AKS Dapr Docs — Production guidelines on Kubernetes ","permalink":"https://wolkwacht.nl/posts/from-orbstack-to-aks-installing-dapr-on-a-managed-cluster/","summary":"\u003cp\u003e\u003cimg alt=\"A Dapr-enabled application moving from a local Kubernetes workstation to a managed AKS cluster\" loading=\"lazy\" src=\"/images/2026/dapr-orbstack-to-aks-feature.png\"\u003e\u003c/p\u003e\n\u003cp\u003eTransferring a Dapr-enabled application from local Kubernetes to AKS should be straightforward. The annotations, app ID, application port, sidecar API, and workload manifest stay familiar. The key difference is ownership: a temporary local setup transforms into a shared runtime that requires a version policy, availability plan, network access, identity strategy, and support structure.\u003c/p\u003e\n\u003cp\u003eThis post moves the sample from \u003ca href=\"/posts/installing-dapr-on-kubernetes-locally-with-orbstack/\"\u003eInstalling Dapr on Kubernetes Locally with OrbStack\u003c/a\u003e to an existing AKS cluster. We will compare the open-source installation path with the AKS Dapr extension, install the extension, verify the control plane, and deploy the same NGINX workload.\u003c/p\u003e","title":"From OrbStack to AKS: Installing Dapr on a Managed Cluster"},{"content":"\nTo quickly grasp Dapr on Kubernetes, it’s best to start locally rather than with an AKS cluster. Local setup allows for rapid pod creation in seconds, minimizes the consequences of mistakes, and keeps all control-plane components easily accessible for inspection.\nIn this post, we\u0026rsquo;ll demonstrate setting up OrbStack\u0026rsquo;s single-node Kubernetes cluster on macOS, installing Dapr in development mode, examining the creation process, deploying a simple web app with a sidecar, and accessing it via Dapr\u0026rsquo;s HTTP API. The same annotations and runtime model apply to AKS later, with only differences in the platform and installation approach.\nThis is Part 2 of the Dapr on Kubernetes and AKS series. If the distinction between Dapr, Kubernetes, and a service mesh is still unclear, begin with What Is Dapr, and Why Run It on Kubernetes?.\nOutcome: by the end, one OrbStack pod will contain an NGINX application container and a Dapr sidecar. A request sent to the sidecar\u0026rsquo;s service-invocation API will reach NGINX and return its welcome page.\nWhy OrbStack for this exercise OrbStack offers a lightweight, single-node Kubernetes cluster integrated with macOS, including kubectl. It shares its container image engine with the Docker environment and ensures Kubernetes service addresses are accessible from the Mac. This creates a streamlined process for building images, applying manifests, and testing services.\nUsing another local Kubernetes environment: this article uses OrbStack, but Dapr does not depend on it. You can follow the same tutorial with Docker Desktop, Rancher Desktop, Minikube, kind, k3d, or another conformant local Kubernetes installation. Use that platform\u0026rsquo;s instructions to start the cluster and select its kubectl context; from dapr init -k --dev --wait onward, the Dapr installation, annotations, sample manifest, and invocation steps are the same. Image loading, service exposure, ingress, and host networking can differ between local Kubernetes products.\nDon\u0026rsquo;t confuse this convenience with production equivalence. OrbStack\u0026rsquo;s managed cluster has one node, uses Flannel by default, and cannot reproduce AKS availability zones, Azure CNI behavior, managed identities, Azure load balancers, or a realistic node-pool failure. It is the right place to learn Dapr injection and APIs, not to prove production topology.\nOrbStack also behaves differently from remote clusters in a few useful ways:\nlocally built images are immediately available to Kubernetes because Docker and Kubernetes use the same image store; images tagged latest are pulled by default, so use a non-latest tag or set imagePullPolicy: IfNotPresent for a local image; ClusterIP, NodePort, and LoadBalancer services are reachable from macOS; and wildcard names under *.k8s.orb.local can be used for LoadBalancer and Ingress endpoints, although no ingress controller is installed by default. Our sample uses a public, versioned NGINX image and kubectl port-forward, which keeps the first exercise independent of ingress and local-image behavior.\nPrerequisites You need:\nmacOS with a current OrbStack installation; the Dapr CLI; curl; and enough local memory and disk for the Kubernetes node, Dapr control plane, Redis, Zipkin, and the sample pod. OrbStack supplies kubectl, but an existing standalone kubectl works as well. Check the tools before changing the cluster:\norb version kubectl version --client dapr version If the Dapr CLI is missing and you use Homebrew, install the official formula from the Dapr tap:\nbrew install dapr/tap/dapr-cli The commands below act on the current Kubernetes context. This matters if you also operate AKS or another cluster from the same workstation.\nStart Kubernetes and verify the context Start OrbStack\u0026rsquo;s managed Kubernetes cluster:\norb start k8s Verify the active context before installing anything:\nkubectl config current-context kubectl cluster-info kubectl get nodes -o wide The context should point to OrbStack, and one node should reach Ready. If it does not, stop here. Running dapr init -k against the wrong context can install Dapr into a real remote cluster.\nA useful habit is to make the intended context explicit for consequential commands:\nkubectl config get-contexts If necessary, switch to the OrbStack context shown by that command:\nkubectl config use-context orbstack Context names can differ between versions or installations, so use the value displayed on your machine rather than assuming it.\nRepresentative output. Context names, versions, and node details vary by installation.\nInstall Dapr in development mode Install Dapr into the current cluster and wait for the rollout:\ndapr init -k --dev --wait The flags are worth separating:\n-k selects Kubernetes mode; --dev adds Redis and Zipkin plus development components and tracing configuration; and --wait waits for the Kubernetes installation to become ready, with a default timeout of five minutes. Development mode is convenient because later examples can use a state store, pub/sub broker, and trace backend without separate provisioning. It is not a production profile. Redis and Zipkin are learning dependencies here, not an availability or data-durability design.\nIf your connection is slow, increase the wait timeout:\ndapr init -k --dev --wait --timeout 600 The command pulls the Dapr runtime and development images. On a first installation, image downloads often account for most of the wait.\nRepresentative output. Generated workload names and installation messages can vary by Dapr version.\nInspect what Dapr installed Start with the supported Dapr status command:\ndapr status -k Then inspect the namespace directly:\nkubectl get pods -n dapr-system kubectl get deployments,statefulsets,services -n dapr-system A current Kubernetes installation includes these control-plane responsibilities:\nService Responsibility dapr-operator Watches Dapr resources and manages component updates and Dapr service endpoints dapr-sidecar-injector Mutates annotated pods and adds the daprd container dapr-sentry Acts as Dapr\u0026rsquo;s certificate authority and manages workload certificates for mTLS dapr-placement Maintains placement tables for actors dapr-scheduler Schedules jobs and supports actor reminders and workflows The exact pod names include generated suffixes, and the Scheduler is a StatefulSet rather than a conventional stateless deployment. Judge the installation by readiness and workload type, not by copying pod names from an article.\nDevelopment mode also creates Redis and Zipkin workloads and installs resources in the default namespace. Inspect them with:\nkubectl get pods dapr components -k dapr configurations -k What happened to dapr dashboard -k? Older Dapr tutorials use dapr dashboard -k. The Dashboard command was deprecated and removed from the current Dapr CLI documentation in May 2026. Do not treat its absence as a broken installation. For this series, dapr status, dapr components, dapr configurations, dapr logs, and ordinary kubectl inspection provide the supported workflow.\nIf you are following this article with an older CLI that still contains the command, it may work, but building a new workflow around a removed command only creates future cleanup.\nSidecar injection before we deploy The Dapr sidecar injector is a Kubernetes admission webhook. When Kubernetes creates a pod whose template contains dapr.io/enabled: \u0026quot;true\u0026quot;, the webhook modifies that pod specification and adds a daprd container.\nThree annotations establish the essential contract:\nannotations: dapr.io/enabled: \u0026#34;true\u0026#34; dapr.io/app-id: \u0026#34;hello-dapr\u0026#34; dapr.io/app-port: \u0026#34;80\u0026#34; dapr.io/enabled requests injection. dapr.io/app-id gives the application a Dapr identity used for discovery and several building-block behaviors. dapr.io/app-port tells Dapr where the application listens inside the shared pod network. The annotation belongs under spec.template.metadata.annotations in a Deployment. Putting it on the Deployment\u0026rsquo;s top-level metadata does not annotate the pods and therefore does not trigger injection.\nThe sidecar and application share the pod network namespace. NGINX listens on port 80; daprd exposes its HTTP API on port 3500 and gRPC API on port 50001 by default. The application image does not contain Dapr.\nDeploy the hello service Save the following as hello-dapr.yaml, or download the companion manifest:\napiVersion: apps/v1 kind: Deployment metadata: name: hello-dapr spec: replicas: 1 selector: matchLabels: app: hello-dapr template: metadata: labels: app: hello-dapr annotations: dapr.io/enabled: \u0026#34;true\u0026#34; dapr.io/app-id: \u0026#34;hello-dapr\u0026#34; dapr.io/app-port: \u0026#34;80\u0026#34; spec: containers: - name: web image: nginx:1.27-alpine imagePullPolicy: IfNotPresent ports: - name: http containerPort: 80 readinessProbe: httpGet: path: / port: http initialDelaySeconds: 2 periodSeconds: 5 --- apiVersion: v1 kind: Service metadata: name: hello-dapr spec: selector: app: hello-dapr ports: - name: http port: 80 targetPort: http Apply it and wait:\nkubectl apply -f hello-dapr.yaml kubectl rollout status deployment/hello-dapr kubectl get pods -l app=hello-dapr The pod should show 2/2 ready containers: web and daprd. Confirm their names:\nkubectl get pods -l app=hello-dapr \\ -o jsonpath=\u0026#39;{.items[0].spec.containers[*].name}{\u0026#34;\\n\u0026#34;}\u0026#39; Expected output resembles:\nweb daprd Kubernetes created the application pod from your Deployment, while the admission webhook added daprd during pod creation. Editing annotations on a running pod doesn\u0026rsquo;t retrofit a sidecar; changing a Deployment template triggers a new rollout.\nInvoke the application through Dapr Forward the Dapr HTTP API from the Deployment to your Mac:\nkubectl port-forward deployment/hello-dapr 3500:3500 Leave that terminal running. In another terminal, call NGINX through Dapr\u0026rsquo;s service-invocation API:\ncurl --fail --show-error \\ http://localhost:3500/v1.0/invoke/hello-dapr/method/ The URL contains two separate identities:\nlocalhost:3500 is the Dapr sidecar API you forwarded; hello-dapr is the target Dapr app ID; and / after method is the path Dapr sends to NGINX on port 80. The returned NGINX welcome HTML proves that the request entered the Dapr sidecar and reached the application. For comparison, the Kubernetes Service can be tested directly:\nRepresentative output. Kubernetes generates the pod suffix and timing values.\nkubectl port-forward service/hello-dapr 8080:80 curl --fail --show-error http://localhost:8080/ Both requests reach the same application. The second uses Kubernetes networking directly; the first enters through the Dapr application API. Later posts will make the distinction meaningful by calling between applications and applying resiliency, tracing, and component behavior.\nInspect the sidecar log if you want to see the runtime initialize:\ndapr logs -k --app-id hello-dapr The equivalent kubectl command is useful when diagnosing container-specific failures:\nkubectl logs deployment/hello-dapr -c daprd Common first-run failures The pod shows 1/1, not 2/2 The sidecar was not injected. Check that all three Dapr annotations are under spec.template.metadata.annotations, that dapr.io/enabled is the string \u0026quot;true\u0026quot;, and that the injector is ready:\nkubectl get deployment dapr-sidecar-injector -n dapr-system kubectl describe pod -l app=hello-dapr After correcting the Deployment, restart its pods:\nkubectl rollout restart deployment/hello-dapr The sidecar is present but invocation returns an error Confirm that dapr.io/app-port matches the port on which the application actually listens. An annotation of 8080 cannot reach NGINX listening on 80. Read both containers\u0026rsquo; logs:\nkubectl logs deployment/hello-dapr -c daprd kubectl logs deployment/hello-dapr -c web Also wait until the pod is ready. Dapr\u0026rsquo;s readiness is tied to its ability to reach the configured application port.\nA local image cannot be pulled OrbStack shares locally built images with Kubernetes, but Kubernetes normally tries to pull images tagged latest. Give the image a stable local tag such as hello-dapr:dev and use imagePullPolicy: IfNotPresent, or use Never when you deliberately want the deployment to fail if the local image is absent.\nFor AKS, this convenience disappears. You need to push images to a registry such as Azure Container Registry and make them accessible to the cluster. That difference will matter when we move the sample to Azure.\nDapr installation pods are stuck in ImagePullBackOff Check the detailed event message before retrying:\nkubectl describe pod -n dapr-system \u0026lt;pod-name\u0026gt; Corporate proxies, registry restrictions, DNS failures, and rate limits can all look like an image problem. By default, the Dapr CLI pulls control-plane and sidecar images from its configured public registry. Fix access rather than repeatedly reinstalling.\nYou installed into the wrong cluster Check the current context:\nkubectl config current-context Don\u0026rsquo;t uninstall immediately until you know what else uses Dapr in that cluster. On a disposable OrbStack cluster created only for this tutorial, uninstalling is straightforward; on a shared cluster, removal is an operational change.\nClean up without deleting OrbStack Remove the sample:\nkubectl delete -f hello-dapr.yaml Keep Dapr installed if you plan to follow the next building-block exercises. To remove the Dapr installation and its development dependencies from this local cluster:\ndapr uninstall -k If you only want to pause local resource use, stop the cluster:\norb stop k8s Deleting the OrbStack Kubernetes cluster is more destructive than stopping it and removes its workloads. Use orb delete k8s only when you intend to.\nWhat transfers to AKS The important pieces are already portable: pod annotations, the Dapr app ID, the application port, the sidecar API, and most of the workload manifest. On AKS, the meaningful changes are around them:\nthe cluster has remote nodes and a production networking model; application images come from a registry such as ACR; the Dapr extension is an installation and lifecycle option; development Redis and Zipkin are replaced by deliberate managed or self-operated dependencies; identity, secret access, ingress, resource limits, availability, and observability require production choices. The local exercise was therefore not a toy version of a different architecture. It was the same Dapr application boundary in a deliberately smaller environment.\nContinue with From OrbStack to AKS: Installing Dapr on a Managed Cluster. We compare the Dapr CLI and AKS extension paths and redeploy this sample with an Azure-appropriate operating model.\nReturn to the Dapr on Kubernetes and AKS series index.\nSources OrbStack Docs — Kubernetes Dapr Docs — Deploy Dapr on a Kubernetes cluster Dapr Docs — Overview of Dapr on Kubernetes Dapr Docs — Dapr sidecar overview Dapr Docs — Dapr arguments and Kubernetes annotations Dapr Docs — Dapr CLI reference ","permalink":"https://wolkwacht.nl/posts/installing-dapr-on-kubernetes-locally-with-orbstack/","summary":"\u003cp\u003e\u003cimg alt=\"A developer workstation running a local Kubernetes cluster with an application and Dapr sidecar\" loading=\"lazy\" src=\"/images/2026/dapr-orbstack-feature.png\"\u003e\u003c/p\u003e\n\u003cp\u003eTo quickly grasp Dapr on Kubernetes, it’s best to start locally rather than with an AKS cluster. Local setup allows for rapid pod creation in seconds, minimizes the consequences of mistakes, and keeps all control-plane components easily accessible for inspection.\u003c/p\u003e\n\u003cp\u003eIn this post, we\u0026rsquo;ll demonstrate setting up OrbStack\u0026rsquo;s single-node Kubernetes cluster on macOS, installing Dapr in development mode, examining the creation process, deploying a simple web app with a sidecar, and accessing it via Dapr\u0026rsquo;s HTTP API. The same annotations and runtime model apply to AKS later, with only differences in the platform and installation approach.\u003c/p\u003e","title":"Installing Dapr on Kubernetes Locally with OrbStack"},{"content":"Dapr on Kubernetes and AKS Dapr gives applications a consistent set of APIs for common distributed-systems concerns while leaving the backing infrastructure replaceable. This series starts with the mental model, builds a fast local development loop, moves the same workload to Azure Kubernetes Service, and then works through the building blocks and production concerns.\nThe series is written for developers and platform engineers who already know the basics of Kubernetes but are new to Dapr. Each post stands on its own; reading in order gives you the smoothest path from first principles to production.\nSeries index What Is Dapr, and Why Run It on Kubernetes? — the sidecar model, building blocks, Kubernetes architecture, and where AKS fits Installing Dapr on Kubernetes Locally with OrbStack — a fast local cluster, control-plane verification, sidecar injection, and a first service From OrbStack to AKS: Installing Dapr on a Managed Cluster — the Dapr CLI and AKS extension paths compared Service Invocation and the Dapr Building Blocks You Will Use Daily — named service calls, resiliency, namespaces, HTTP, and gRPC (coming next) State Management: Pluggable State Stores on AKS — Cosmos DB, Redis, concurrency, consistency, and queries (planned) Pub/Sub Messaging with Azure Service Bus and Event Hubs — subscriptions, routing, dead letters, and CloudEvents (planned) Bindings, Secrets, and Actors — integrations, Azure Key Vault, virtual actors, and a first look at workflows (planned) Observability: Tracing, Metrics, and Logging for Dapr on AKS — OpenTelemetry, Prometheus, logs, dashboards, and diagnosis (planned) Securing and Scaling Dapr for Production on AKS — mTLS, access control, workload identity, tenancy, resources, and autoscaling (planned) Production Readiness: Upgrades, Multi-Cluster DR, and Lessons Learned — versioning, GitOps, recovery, cost, and operating guidance (planned) The index will be updated as each post is published.\n","permalink":"https://wolkwacht.nl/dapr-on-aks-series/","summary":"\u003ch1 id=\"dapr-on-kubernetes-and-aks\"\u003eDapr on Kubernetes and AKS\u003c/h1\u003e\n\u003cp\u003eDapr gives applications a consistent set of APIs for common distributed-systems concerns while leaving the backing infrastructure replaceable. This series starts with the mental model, builds a fast local development loop, moves the same workload to Azure Kubernetes Service, and then works through the building blocks and production concerns.\u003c/p\u003e\n\u003cp\u003eThe series is written for developers and platform engineers who already know the basics of Kubernetes but are new to Dapr. Each post stands on its own; reading in order gives you the smoothest path from first principles to production.\u003c/p\u003e","title":"Dapr on Kubernetes and AKS Series"},{"content":"\nEvery distributed application gathers various plumbing components. One service requires retries and timeouts, another needs a message broker client, and a third must store some state, retrieve secrets, or discover other services. Over time, each team ends up with a unique set of client libraries, connection handling methods, telemetry, and failure behaviors.\nDapr, the Distributed Application Runtime, provides a uniform API for the underlying infrastructure. Your application communicates with a local Dapr process via HTTP or gRPC. Dapr then interacts with services, the state store, the broker, the secret store, or other components on your application\u0026rsquo;s behalf. While the application retains control of its business logic, Dapr offers reusable capabilities for building distributed applications.\nThis separation explains why Dapr is valuable on Kubernetes and AKS. While Kubernetes handles container scheduling and management, it lacks a standard API for application-level functions such as state, messaging, workflows, or secrets. Dapr addresses this gap, enabling services to interact seamlessly without needing to use the same language or SDK.\nThe short version: use Dapr when you want common distributed-systems behavior behind portable application APIs. Don\u0026rsquo;t use it just because an application runs on Kubernetes.\nThis is the first post in the Dapr on Kubernetes and AKS series. It establishes the mental model; the next posts turn it into a working local environment and then an AKS deployment.\nThe problem is repeated integration, not missing code Consider an order service built in .NET, a stock service implemented in Go, and a notification worker created in Python. These components must communicate, publish events, save data, retrieve credentials, and generate useful traces. Without a common runtime, each team selects and configures its own libraries for these tasks.\nThe first implementation usually works. The cost appears later:\nretry policies differ between services and sometimes amplify an outage; broker-specific code leaks into business logic; tracing stops at asynchronous boundaries; secrets are loaded and refreshed in several different ways; replacing a state store or message broker becomes an application rewrite; and platform standards have to be implemented once per language stack. Dapr does not make a distributed system simple. Network calls can still fail, messages can still arrive more than once, and data still needs a deliberate consistency model. Dapr can make the plumbing consistent and move much of the integration-specific code behind stable APIs.\nAn application can publish via Dapr\u0026rsquo;s pub/sub API without importing a specific broker\u0026rsquo;s SDK. A platform team can configure the pub/sub component independently. Although switching components doesn\u0026rsquo;t guarantee a completely effortless migration since brokers vary in semantics and capabilities, it minimizes how much a technology choice affects application code.\nThe sidecar is the boundary Dapr typically runs a daprd process alongside each enabled application instance. In Kubernetes, this process is executed as a separate container within the application pod. Since both containers share the same pod network, the application communicates with Dapr through a local HTTP or gRPC endpoint.\nThe request path is deliberately local at the start:\nThe application calls a Dapr API on localhost. The sidecar applies the relevant building-block behavior and configuration. The sidecar talks to another Dapr-enabled service or to a configured component. Telemetry, resiliency, security, and component behavior are applied at that boundary where supported. This gives polyglot teams a useful contract. A Java service and a Python worker can use the same HTTP or gRPC APIs even when their language ecosystems differ. Dapr SDKs can make those APIs more convenient, but they are optional rather than the architectural boundary.\nThe trade-off is genuine: each enabled application replica requires additional resources for scheduling, monitoring, upgrading, and processing. Calls also incur an extra step through the local sidecar. Whether the benefits of consistency and portability justify this cost depends on the workload. A small monolith communicating directly with a single database may see little advantage. In contrast, a polyglot system with recurring state management, messaging, service invocation, and workflow requirements is a much stronger candidate for Dapr.\nDapr is not a service mesh The shared use of sidecars creates the most common Dapr misunderstanding. Dapr and a service mesh can both provide mTLS, retries, metrics, and tracing, but they operate from different starting points.\nDapr Service mesh Exposes application APIs that developers call deliberately Primarily manages network traffic between workloads Adds building blocks such as state, pub/sub, actors, secrets, and workflows Adds traffic policy, routing, encryption, and network-level observability Uses an application identity such as a Dapr app ID for service invocation Commonly reasons about services, workloads, addresses, and network routes Changes application integration code to call Dapr APIs Can often be introduced without changing application code The two complement each other when a platform needs both application building blocks and advanced network traffic management. They also overlap. If you deploy Dapr and a mesh together, decide explicitly which layer owns mTLS, retries, and tracing; duplicating those behaviors can produce confusing telemetry and unsafe retry multiplication.\nDapr should therefore not be sold as a lighter Istio or Linkerd. A mesh is infrastructure-centric. Dapr is developer-centric. The choice is not automatically one or the other.\nThe building blocks at a glance Dapr is incrementally adoptable. An application can use one building block without committing to all of them.\nService invocation calls another application by its Dapr app ID through HTTP or gRPC. Dapr can add discovery, security, tracing, and configured resiliency to the call path. State management stores and retrieves key-value state through a common API. Capabilities such as transactions, queries, concurrency, and consistency depend on the selected state component. Publish and subscribe separates publishers from subscribers through topics and a configured message broker. Dapr carries events using CloudEvents by default and supports routing and delivery controls. Bindings connect an application to external systems through input triggers and output operations. They are useful when the integration fits a component\u0026rsquo;s contract and does not justify custom client code. Actors implement virtual, addressable entities with encapsulated state and turn-based concurrency. They fit some stateful domain models well and are unnecessary for ordinary stateless services. Secrets retrieve secret values through a consistent API or let other Dapr components reference them, keeping secret material out of component definitions. Workflow coordinates durable, multi-step processes whose execution must survive restarts and failures. Configuration lets applications subscribe to changes in supported configuration stores without embedding a store-specific client. These APIs are backed by components. A state API might use Redis in development and Azure Cosmos DB in another environment. Pub/sub might use Redis Streams, Apache Kafka, or Azure Service Bus. The API is the application contract; the component is the adapter and its configuration.\nPortability still has edges. Not every component implements every optional feature, and operational characteristics don\u0026rsquo;t become identical just because the API is shared. Treat component capability tables and backing-service behavior as part of your design.\nWhy Kubernetes is a natural host Kubernetes gives Dapr a consistent way to attach the runtime to applications. When a workload is annotated for Dapr, the sidecar injector modifies newly created pods and adds the daprd container. This keeps the application image independent from the runtime and makes the pattern repeatable across deployments.\nA Kubernetes-mode Dapr installation also adds control-plane services, normally in the dapr-system namespace:\nthe sidecar injector adds Dapr to annotated pods; the operator watches Dapr resources and communicates component updates; Sentry issues and rotates workload certificates used for Dapr mTLS; Placement maintains actor placement information; and the Scheduler supports scheduling and durable execution capabilities used by jobs, actors, and workflows in current Dapr releases. Kubernetes custom resources provide platform teams with a declarative way to manage Dapr components, configurations, subscriptions, resiliency policies, and related resources. Kubernetes also offers the necessary operational infrastructure, including namespaces, service accounts, deployments, health checks, resource controls, and rolling updates.\nNone of this implies that Kubernetes is necessary. Dapr also enables self-hosted development and supports various hosting options. Kubernetes is especially suitable when you already need a cluster, offering consistent injection, declarative setup, and lifecycle control.\nWhat AKS adds AKS does not alter Dapr\u0026rsquo;s application APIs; instead, it modifies the surrounding platform. Initially, AKS offers a managed Kubernetes control plane along with Azure integrations for identity, networking, policy, monitoring, and container registries. Dapr components can then interface with Azure services like Azure Service Bus, Azure Cosmos DB, Azure Cache for Redis, and Azure Key Vault. When a component supports Microsoft Entra authentication, AKS Workload Identity eliminates the requirement for long-lived connection strings.\nSecond, Microsoft offers the Dapr extension for AKS. This extension deploys the Dapr control plane via Azure\u0026rsquo;s cluster-extension system and manages versioning and configuration. Microsoft advises keeping clusters installed with this extension managed through it, rather than mixing extension management with Dapr CLI operations.\nThe extension reduces installation effort but does not make the Dapr runtime an Azure-managed service. You remain responsible for application annotations, component design, identity and access management, sidecar resources, observability, upgrade policies, and backing-service behavior. Additionally, understanding the difference between Microsoft support for extension operations and upstream Dapr runtime issues is important for setting correct production expectations.\nInstalling open-source Dapr into a small Kubernetes cluster is quick and straightforward for local learning. When using an AKS platform, the extension becomes the clear choice since it aligns with Azure\u0026rsquo;s lifecycle management and offers high availability out of the box. We will explicitly compare both approaches later in this series.\nWhen Dapr is a good fit Dapr deserves a proof of concept when several of these are true:\nservices are written in multiple languages; the same integration concerns recur across teams; applications need service invocation, messaging, state, secrets, or durable workflows; the platform team wants a shared contract without prescribing one language framework; portability between supported component implementations has real value; or developers need consistent local and Kubernetes-hosted behavior. Pause before adding it when the architecture is simple, direct client libraries are effective, latency is highly critical, the organization can\u0026rsquo;t support another runtime, or only a limited feature set will be used. A platform abstraction is valuable when it reduces more complexity than it adds.\nThe most effective adoption approach is incremental. Start with a single bounded problem such as service invocation between two services or publishing a single domain event and evaluate its impact on code quality, latency, operability, and failure behavior. Avoid migrating everything at once just to test a hypothesis.\nWhat comes next Continue with Installing Dapr on Kubernetes Locally with OrbStack, where we build a local cluster, inspect the current control plane, and deploy a minimal Dapr-enabled service. After that, we will move the same mental model and sample workload to AKS.\nContinue with the Dapr on Kubernetes and AKS series index.\nSources Dapr Docs — Dapr concepts Dapr Docs — Dapr sidecar overview Dapr Docs — Dapr and service meshes Dapr Docs — Dapr building blocks Dapr Docs — Dapr on Kubernetes Microsoft Learn — Dapr extension for AKS and Arc-enabled Kubernetes Microsoft Learn — Install the Dapr extension for AKS ","permalink":"https://wolkwacht.nl/posts/what-is-dapr-and-why-run-it-on-kubernetes/","summary":"\u003cp\u003e\u003cimg alt=\"A cloud-native application and its Dapr sidecar connect to state, messaging, secrets, and services\" loading=\"lazy\" src=\"/images/2026/dapr-on-aks-part-1-feature.png\"\u003e\u003c/p\u003e\n\u003cp\u003eEvery distributed application gathers various plumbing components. One service requires retries and timeouts, another needs a message broker client, and a third must store some state, retrieve secrets, or discover other services. Over time, each team ends up with a unique set of client libraries, connection handling methods, telemetry, and failure behaviors.\u003c/p\u003e\n\u003cp\u003eDapr, the Distributed Application Runtime, provides a uniform API for the underlying infrastructure. Your application communicates with a local Dapr process via HTTP or gRPC. Dapr then interacts with services, the state store, the broker, the secret store, or other components on your application\u0026rsquo;s behalf. While the application retains control of its business logic, Dapr offers reusable capabilities for building distributed applications.\u003c/p\u003e","title":"What Is Dapr, and Why Run It on Kubernetes?"},{"content":"\nThe real question is not whether to use the Azure Key Vault provider for the Secrets Store CSI Driver or the External Secrets Operator (ESO). Instead, it\u0026rsquo;s whether your workload should access secret data as mounted files or via native Kubernetes Secret objects.\nThat choice influences application design, rotation, incident handling, RBAC, observability, Helm compatibility, and GitOps. In short: CSI offers a more secure default for applications that consume files; ESO provides a more practical platform default when Kubernetes-native compatibility is a priority. This decision involves both GitOps and security considerations.\nHow each one actually works The AKS-managed Azure Key Vault provider runs with the Secrets Store CSI Driver. A pod references a namespaced SecretProviderClass and mounts an inline CSI volume. During mount, the provider authenticates to Key Vault, retrieves the requested objects, and materializes them in the pod filesystem. It creates no Kubernetes Secret unless secretObjects is configured. Microsoft documents support for secrets, keys, certificates, auto-rotation, and optional Kubernetes Secret synchronization.\nESO is a controller. An ExternalSecret references a namespaced SecretStore or cluster-scoped ClusterSecretStore. The controller authenticates to Key Vault, reads the remote values, and creates or updates a normal Kubernetes Secret. Pods then consume that object through environment variables or projected volumes. Key Vault remains the source of truth, but secret material now also exists in the Kubernetes API data path.\nThat extra object is neither automatically wrong nor free. It is the central tradeoff.\nThe core tradeoff: where secrets live CSI\u0026rsquo;s default file-only pattern avoids storing secret values as Kubernetes Secret objects. This reduces the number of entries in etcd and keeps secrets hidden from standard Secret-listing processes, limiting the entities that can access them via the API. However, it does not prevent access if a malicious application, privileged pod, node, or identity with read access to the mount is present. CSI reduces the attack surface but doesn\u0026rsquo;t eliminate it.\nThe price is compatibility. Kubernetes cannot directly populate an environment variable from a CSI-mounted file. Many standard Helm charts require `a secretKeyRef, and some operators accept only a Secret name. While CSI secret sync is possible, it removes the architectural benefit of having “no Kubernetes Secret” for that value.\nESO supports native Secrets, so existing charts, operators, admission policies, backup exclusions, and developer workflows work without modifications. Git only keeps references and templates, not actual values, while the controller generates the runtime objects. This approach increases the impact of the Kubernetes API and RBAC: anyone with read access to the generated Secret can obtain its value.\nAvoid simplifying to “etcd is encrypted” or “etcd is plaintext.\u0026quot; Remember, Kubernetes Secret data is merely base64-encoded, not encrypted directly by the Secret resource. AKS safeguards the underlying service storage and provides Kubernetes KMS-based encryption for Secrets, using either platform-managed or customer-managed keys. In regulated settings, explicitly verify the KMS configuration, API access, key ownership, backups, and support processes.\nRotation and freshness are consumer problems CSI autorotation periodically polls Key Vault; the AKS add-on documents a default rotation poll interval of two minutes when autorotation is enabled. It updates mounted content and, if configured, the synced Kubernetes Secret. An application reading the CSI filesystem must watch for file changes or reopen the file. A pod restart is not inherently required for a normal file mount. A subPath mount does not receive automatic updates, and an environment variable sourced from a synced Secret requires a restart or rollout.\nESO\u0026rsquo;s default Periodic policy reads the provider on spec.refreshInterval and updates the target Secret. It also supports CreatedOnce and OnChange. Updating the Secret does not guarantee that the process has adopted the value: projected Secret volumes eventually update, but the application must reread them; environment variables remain fixed for the life of the container.\nThe honest rotation SLO is therefore:\nprovider polling delay + controller or driver reconciliation + Kubernetes propagation + application reload delay\nMonitor the full chain. A green Key Vault rotation event says nothing about the credential a long-running process currently holds.\nIdentity and authorization in production Both approaches can use Microsoft Entra Workload ID on AKS, avoiding static client secrets. I covered the broader identity principle in Identity Is the Perimeter You Forgot to Guard; the key point is to give each trust boundary its own identity and the minimum Key Vault role.\nFor CSI, SecretProviderClass is namespaced, and the consuming pod drives the mount. With Workload Identity, its service account determines the Azure identity. Permission to create a SecretProviderClass does not itself grant Key Vault access, but arbitrary service-account and provider pairings still deserve policy control.\nESO adds a controller-mediated boundary. A SecretStore is namespaced; a ClusterSecretStore is reusable across namespaces and can be restricted with namespace conditions. Decide whether ESO uses one central identity, referenced service accounts, or per-namespace identities. Restrict both who may reference a store and what its Azure identity may read. A cluster-wide store backed by a vault-wide reader becomes a high-value broker.\nGitOps fit ESO\u0026rsquo;s API maps cleanly to GitOps: commit ExternalSecret and SecretStore, let Flux reconcile them, and let ESO materialize the Secret. Status conditions expose whether the runtime object is current. This fits the reconciliation model in GitOps on Azure Kubernetes Service with Flux without placing secret values in Git.\nCSI\u0026rsquo;s SecretProviderClass, service account, and pod volume declaration are equally valid GitOps resources. The friction appears at the application contract: a chart that insists on an existing Secret cannot declare the file-only state you want. Patching charts, adding init logic, or enabling CSI sync creates platform exceptions that Flux can reconcile but your team must still own.\nNeither controller makes Git a secret store. Commit remote keys, vault URIs, templates, and identity bindings; never commit the values or a rendered Secret manifest.\nMulti-cloud and multi-backend reality ESO supports providers including Azure Key Vault, AWS Secrets Manager, Google Secret Manager, and HashiCorp Vault. That helps when one platform spans clouds or must migrate backends, although provider semantics and maturity vary.\nThe Secrets Store CSI Driver itself has multiple providers, but the AKS-managed add-on and operational path discussed here use the Azure Key Vault provider. That Azure focus is an advantage for a single-cloud AKS estate: Microsoft manages the add-on lifecycle and integration. It is a constraint if a single platform contract must span clouds and backends.\nProduction comparison Dimension Secrets Store CSI Driver External Secrets Operator Secret object created? No by default; optional secretObjects sync creates one Yes; ExternalSecret creates or updates a native Secret Rotation mechanism Driver/provider poll; updates mount and optional synced Secret Controller reconciliation using refreshPolicy and refreshInterval GitOps compatibility CRDs and pod specs reconcile cleanly; file consumption can conflict with chart assumptions Strong fit for native Secret consumers and declarative CRDs Multi-backend support Provider ecosystem exists; AKS-managed path is Azure Key Vault-specific Broad provider catalog across clouds and secret stores Operational overhead AKS manages the add-on, but every workload needs volume wiring and reload behavior You operate and upgrade the controller, CRDs, RBAC, stores, and controller capacity Failure if Key Vault is unreachable Existing pods retain their last mounted content; rotation fails, and new mounts or pod starts can fail Existing generated Secrets remain usable; refresh reports errors, and pods can still start while the Secret exists The outage row matters in production. ESO deliberately creates a local cache in Kubernetes, which can improve workload startup during a Key Vault outage but extends the lifetime and reach of secret material. CSI keeps fewer copies but creates a new pod mount that depends on Key Vault, identity, network, provider, and node-plugin health. Test both stale-secret behavior and scale-out during a denied-vault or blocked-egress exercise.\nDecision guide Choose CSI when minimizing Kubernetes Secret objects is a real control objective, workloads can read mounted files, your estate is Azure-centric, and you can implement file reload plus startup-failure handling. It is especially compelling for certificates and applications already designed around filesystem credentials.\nChoose ESO when existing tooling requires native Secrets, teams rely heavily on Helm conventions, one platform contract must cover several backends, or GitOps consistency matters more than avoiding the Secret object. Pair it with tight Secret RBAC, explicit KMS requirements, short-lived credentials, namespace-scoped stores where practical, and alerting on stale reconciliation.\nDo not choose CSI and then enable Secret sync everywhere without acknowledging that you have effectively selected both trade-offs. Do not choose ESO merely because the manifest is shorter while ignoring controller identity and cluster-wide store scope.\nMy take For a smaller AKS workload within a single team, I default to the AKS-managed CSI add-on if the application reads files natively. It removes a controller from the platform and, by default, avoids creating Kubernetes Secrets. If the application or chart expects secretKeyRef, I would use ESO rather than wrap the workload in brittle glue.\nFor a platform team standardizing across clusters, namespaces, and secret backends, ESO is usually the better paved road. Its CRDs align with how Flux-based platforms already express the desired state, and native Secrets minimize exceptions for product teams. That recommendation comes with a non-negotiable set of controls: scoped stores, separated workload identities, restricted Secret reads, KMS posture, stale-sync alerts, and tested rotation reloads.\nRunning both is not an architectural failure. Use CSI for workloads whose security model benefits from file-only delivery, and ESO for workloads that need the Kubernetes Secret contract. The mistake is not having two tools; it is letting teams choose without defining which secret classes, identities, and failure modes belong to each path.\nSources and validation notes Microsoft: Azure Key Vault provider for Secrets Store CSI Driver on AKS Microsoft: CSI autorotation and Kubernetes Secret sync Microsoft: CSI Driver identity access with Workload Identity Microsoft: AKS KMS data-encryption concepts External Secrets Operator: Azure Key Vault provider External Secrets Operator: ExternalSecret API External Secrets Operator: SecretStore scoping Kubernetes: Secrets Validated against Microsoft, Kubernetes, and External Secrets Operator documentation in September 2026. Be aware that versions, preview status, defaults, and provider behavior may change; verify the target AKS release and ESO version before deploying in production.\n","permalink":"https://wolkwacht.nl/posts/2026-09-05_secrets-management-showdown--azure-key-vault-csi-driver-vs-external-secrets-operator/","summary":"\u003cp\u003e\u003cimg alt=\"Two secrets-management paths lead from a secure vault into Kubernetes workloads\" loading=\"lazy\" src=\"/images/2026/feature.svg\"\u003e\u003c/p\u003e\n\u003cp\u003eThe real question is not whether to use the Azure Key Vault provider for the Secrets Store CSI Driver or the External Secrets Operator (ESO). Instead, it\u0026rsquo;s whether your workload should access secret data as \u003cstrong\u003emounted files\u003c/strong\u003e or via \u003cstrong\u003enative Kubernetes Secret objects\u003c/strong\u003e.\u003c/p\u003e\n\u003cp\u003eThat choice influences application design, rotation, incident handling, RBAC, observability, Helm compatibility, and GitOps. In short: \u003cstrong\u003eCSI offers a more secure default for applications that consume files; ESO provides a more practical platform default when Kubernetes-native compatibility is a priority.\u003c/strong\u003e This decision involves both GitOps and security considerations.\u003c/p\u003e","title":"Secrets Management Showdown: Azure Key Vault CSI Driver vs. External Secrets Operator"},{"content":"\nKubernetes is very good at reporting desired state. A Deployment can be available while its pods restart every few hours. A cluster can show green nodes while workloads have no disruption budgets, probes, resource limits, or network isolation. Dashboards show signals; they do not necessarily connect them into an operational judgment.\nThat is the gap KubeBuddy tries to fill. The open-source KubeDeck project is a Go-based command-line scanner that connects through an existing Kubernetes context and evaluates cluster health, workloads, security, RBAC, networking, storage, and configuration. It can add provider-specific checks for AKS and GKE and emit terminal, HTML, JSON, and CSV output. There is no controller, Helm release, or long-running agent to install in the cluster.\nThis is a review of the KubeBuddy at kubebuddy.io and KubeDeckio/KubeBuddy. An unrelated dashboard project uses the same name at kubebuddy.org; do not mix their documentation or security assumptions.\nHow KubeBuddy fits into a platform KubeBuddy is best understood as a snapshot scanner. It queries the API server, evaluates a catalog of more than 100 checks, and produces findings with severity, status, recommendations, and references. Optional integrations enrich that snapshot with Prometheus metrics or cloud-provider configuration. The official overview covers node and pod health, risky RBAC, networking, storage, and cloud best practices.\nThat architecture has real advantages. There is no in-cluster component to patch, no DaemonSet consuming resources, and no admission webhook in the deployment path. A platform team can run the same binary from a workstation, hardened jump host, container, or CI worker. HTML is useful for a human review; JSON or CSV can feed a pipeline, evidence store, or reporting system.\nIt also defines the boundary of the product. KubeBuddy does not replace Prometheus, alerting, runtime threat detection, policy admission, or a managed Kubernetes security service. A periodic snapshot can miss a short-lived failure. A finding identifies a condition; it does not prove exploitability, ownership, or business impact. Treat KubeBuddy as a second opinion and control validation layer, not as the source of truth for real-time health.\nA five-minute first scan The installation guide recommends Homebrew on macOS and Linux. Homebrew 6 requires explicit trust for non-official taps; installing the fully qualified formula limits trust to that formula:\nbrew install KubeDeckio/homebrew-kubebuddy/kubebuddy kubebuddy version Before scanning, make the target explicit. Accidentally assessing the wrong context is surprisingly easy:\nkubectl config current-context kubebuddy probe kubebuddy summary mkdir -p reports kubebuddy run \\ --html-report \\ --json-report \\ --yes \\ --output-path ./reports probe verifies access, summary provides a quick inventory, and run writes shareable reports. For direct terminal output, use kubebuddy scan --output text. These commands follow the project’s getting-started workflow.\nNever publish a raw report blindly. Findings can expose namespace names, workload topology, image references, RBAC relationships, and configuration weaknesses. Store reports as security-sensitive build artifacts, apply retention controls, and redact before sharing outside the operations boundary.\nThe permission question matters “Agentless” describes deployment, not privilege. KubeBuddy does not require cluster-admin, but the documented full scan needs broad cluster-wide reads across workloads, nodes, events, networking, storage, RBAC, CRDs, metrics, ConfigMaps, and Secrets. The project’s permissions reference includes a sample ClusterRole and access tests.\nValidate the effective identity before a scan:\nkubectl auth can-i list secrets --all-namespaces kubectl auth can-i list clusterroles kubectl auth can-i list customresourcedefinitions.apiextensions.k8s.io kubectl auth can-i get nodes.metrics.k8s.io From a security architecture perspective, broad read access is still powerful. Secret get or list access may reveal decoded credentials to a compromised scanner process, while RBAC and CRD visibility provides a detailed map of the platform. Use a dedicated identity, short-lived credentials, an isolated runner, pinned KubeBuddy versions, and restricted report storage. Review the supplied role rather than applying it unchanged: its wildcard read rule is convenient for broad CRD coverage but is wider than strict least privilege. Missing permissions should produce an understood coverage gap, not an automatic grant of cluster-admin.\nWhere it earns a place 1. Pre-change and post-change evidence Run a scan before and after a Kubernetes upgrade, CNI change, ingress migration, or node-pool replacement. Comparing JSON output helps distinguish pre-existing debt from change-induced regression. This is particularly useful during a maintenance window when operators need a repeatable checklist, not another dashboard to interpret.\n2. Cluster onboarding and inherited environments When a platform team inherits a cluster, KubeBuddy provides a fast first pass across workloads, events, RBAC, storage, networking, and resilience settings. It will not replace discovery interviews or threat modelling, but it can turn an unfamiliar cluster into a prioritized investigation backlog.\n3. CI and scheduled hygiene scans The JSON report makes recurring scans practical. Run KubeBuddy from a network-restricted runner with a read-only service account, archive the output, and let a small policy step decide whether new critical findings should fail the job. Do not gate production on an unversioned latest image or on every warning: baselines, suppressions, and an exception process are essential to prevent alert fatigue.\n4. AKS and GKE platform reviews Generic Kubernetes checks cannot see every managed-service decision. KubeBuddy adds AKS and GKE coverage; AKS scans can receive subscription, resource-group, and cluster identifiers. The documented AKS form is:\nkubebuddy run \\ --aks \\ --subscription-id \u0026#34;$AZURE_SUBSCRIPTION_ID\u0026#34; \\ --resource-group \u0026#34;$AKS_RESOURCE_GROUP\u0026#34; \\ --cluster-name \u0026#34;$AKS_CLUSTER_NAME\u0026#34; \\ --html-report \\ --yes \\ --output-path ./reports Cloud API access creates a second authorization plane. Separate Kubernetes RBAC from Azure or Google Cloud IAM, scope both identities deliberately, and capture which checks were skipped when credentials are unavailable.\n5. Security triage and risk-path discussion An isolated warning often looks harmless. Several connected weaknesses, an exposed service, permissive workload identity, sensitive Secret access, and broad RBAC, may form a credible attack path. KubeBuddy’s checks and risk documentation is useful for moving the conversation from a flat finding count to chains of conditions. The tool accelerates triage; a security engineer still validates reachability, compensating controls, and impact.\nWhat I like—and what I would challenge The strongest design choice is operational simplicity. A native binary, external execution, multiple report formats, and optional cloud or Prometheus enrichment make adoption easy. The Headlamp plugin offers another interface, while optional KubeBuddy Radar adds history and comparison. Keeping the CLI useful without Radar avoids making the hosted workflow mandatory.\nThe interfaces are not equivalent. The browser-side Headlamp plugin uses resources the current Headlamp session can read, but it does not run provider API, Prometheus, PowerShell, kubectl, or native Go engine checks. Radar is a separate authenticated control plane for saved profiles, private report history, trends, and comparisons; the scan still executes locally. That separation is sensible, but it means an architecture review must document where execution occurs, which data leaves the runner, and whether a paid Radar workflow is in scope. Compare results only when the runtime, check catalog, exclusions, and credentials are equivalent.\nThe main limitation is the nature of snapshot analysis. Resource configuration and current state provide evidence, but not complete causality. A restart may have happened outside the scan window; a permissive role may be intentionally constrained by identity governance; a missing NetworkPolicy may be offset by another enforcement layer. Teams must validate findings and tune exclusions rather than chase a perfect score.\nI would also scrutinize release maturity and supply-chain controls before enterprise rollout. Pin a tested version, verify provenance and checksums where available, review the open-source changelog, and test new check catalogs against a non-production cluster. “100+ checks” is not the same as complete coverage of CIS benchmarks, Kubernetes policy, cloud posture, or runtime behavior. Map checks to your own controls and document what remains uncovered.\nAdoption recommendation Start with one non-production cluster and one named owner. Run the scan manually, classify findings as actionable, accepted, false positive, or needs context, and record scan duration plus permission gaps. Then repeat after remediation. If the signal is useful, put the exact version and configuration in source control and schedule scans from a dedicated runner.\nFor a fleet, define a minimum profile per cluster type, centralize reports securely, and measure new or regressed findings rather than total findings alone. Use HTML for operational review and JSON/CSV for automation. Keep monitoring, admission control, vulnerability scanning, and runtime detection in place.\nMy conclusion: KubeBuddy is a useful, low-friction diagnostic lens for platform teams, especially for inherited clusters, change validation, managed Kubernetes reviews, and periodic hygiene. Its agentless model lowers the cost of trying it, but its broad read access and snapshot nature deserve explicit controls. Used as a repeatable second opinion, it can expose the uncomfortable space between “Kubernetes says healthy” and “the platform is operated safely.”\nFurther reading KubeBuddy product and architecture overview Official CLI documentation Installation and bundled-check behavior Getting started and report commands Kubernetes permission requirements GitHub source repository and MIT license Project changelog ","permalink":"https://wolkwacht.nl/posts/2026-08-25_kubebuddy--an-agentless-second-opinion-for-kubernetes/","summary":"\u003cp\u003e\u003cimg alt=\"An external diagnostic drone scans a Kubernetes cluster without entering its boundary\" loading=\"lazy\" src=\"/images/2026/kubebuddy-feature.png\"\u003e\u003c/p\u003e\n\u003cp\u003eKubernetes is very good at reporting desired state. A Deployment can be available while its pods restart every few hours. A cluster can show green nodes while workloads have no disruption budgets, probes, resource limits, or network isolation. Dashboards show signals; they do not necessarily connect them into an operational judgment.\u003c/p\u003e\n\u003cp\u003eThat is the gap \u003ca href=\"https://kubebuddy.io/\"\u003eKubeBuddy\u003c/a\u003e tries to fill. The open-source KubeDeck project is a Go-based command-line scanner that connects through an existing Kubernetes context and evaluates cluster health, workloads, security, RBAC, networking, storage, and configuration. It can add provider-specific checks for AKS and GKE and emit terminal, HTML, JSON, and CSV output. There is no controller, Helm release, or long-running agent to install in the cluster.\u003c/p\u003e","title":"KubeBuddy: An Agentless Second Opinion for Kubernetes"},{"content":"\nKubernetes has become so familiar that many teams don\u0026rsquo;t choose to adopt it anymore; they just start using it. When a new internal API appears, someone sets up an AKS cluster, and within weeks, the team manages node pools, upgrade channels, ingress, identity, policies, observability, and a backlog of Helm charts. None of these decisions are incorrect, but the issue is that the workload might never have required a Kubernetes platform in the first place.\nAzure Container Apps (ACA) has closed enough of the gap that “AKS by default” is no longer the safe choice for smaller workloads. ACA will not replace Kubernetes, but it removes substantial platform work when an application only needs containers, autoscaling, secure connectivity, revisions, and a deployment path.\nBy “smaller,” I refer to a workload managed by a single team, consisting of around three to ten services, with moderate or sporadic traffic and no strict reliance on Kubernetes-specific extensions. If your upcoming service environment matches this description, then AKS needs to demonstrate its value.\nWhat counts as a smaller workload? This comparison targets applications with:\na single team responsible for both delivery and operations; a small number of APIs, workers, and scheduled jobs; no custom resource definitions, operators, or controllers; no need for sophisticated multi-tenant scheduling; and traffic that is modest, bursty, or concentrated during business hours. This is not a judgment on a large internal developer platform, a regulated multi-cluster system, or a shared cluster supporting multiple teams. Similarly, ACA isn\u0026rsquo;t \u0026ldquo;Kubernetes without Kubernetes.” It provides an application platform rather than the Kubernetes API itself. This abstraction is both the product and its limitation.\nWhere Azure Container Apps genuinely wins The first advantage is elasticity. ACA Consumption can scale an app to zero when no replicas are needed, incurring no resource charges at zero. It employs KEDA-based declarative scaling for HTTP, TCP, and event sources like queues. Since CPU and memory rules cannot start a workload from zero, choose a trigger that is active even when no replica runs. Microsoft’s scaling documentation clearly explains both the capabilities and limitations.\nThat makes ACA particularly attractive for queue workers, internal tools, development environments, and APIs that can tolerate a cold start. I covered the billing mechanics and optimization controls in Optimizing Azure Container Apps Costs, so I will not repeat the full FinOps model here.\nThe second advantage is the operational surface area outside your direct control, including customer-managed node pools, Kubernetes upgrades, node-image patch cycles, CNI selection, and ingress-controller lifecycle. You remain responsible for application security, images, sizing, scaling, identity, observability, and network design. Managed services reduce exposed layers but do not lessen your responsibilities.\nACA also provides capabilities that would otherwise require platform assembly. Dapr integration is available per app. KEDA-based scaling is part of the service. Revisions provide immutable deployment snapshots, and multiple active revisions can receive weighted traffic for blue-green or A/B releases. These are platform features, not add-ons you must install and reconcile.\nThe route to production highlights key differences. ACA requires an environment, registry access, app definitions, ingress, identity, scaling, and observability. AKS additionally demands a cluster operating model, node pools, networking, ingress, namespaces, RBAC, policies, upgrades, and autoscaling. While AKS Automatic simplifies this list with managed defaults, it still leaves Kubernetes platform choices intact..\nWhere AKS still wins AKS performs strongly when the Kubernetes API is integrated into the product. However, if your requirements include operators, CRDs, admission controllers, specialized scheduling, or infrastructure-reconciling controllers, ACA’s abstraction can become a barrier instead of an advantage.\nThe same applies to deep service networking. ACA supports virtual-network integration, internal environments, user-defined routes in supported configurations, network security groups, and environment-level ingress controls. However, it does not provide the same pod-level canvas as AKS. AKS gives teams Kubernetes NetworkPolicy, Cilium capabilities, custom gateways, and deeper service-mesh control. That distinction matters for the patterns explored in my Azure Kubernetes Application Network series.\nAKS is more appropriate for large-scale multi-tenant setups. Features such as namespaces, quotas, admission policies, RBAC, network policies, and placement controls establish clear tenant boundaries. However, a Container Apps environment does not serve as a direct substitute for a managed shared Kubernetes platform.\nFor AI and GPU tasks, ACA now provides both serverless and dedicated GPU options, so the idea that “GPU means AKS” is no longer always accurate. AKS remains the preferred choice when requiring a wider selection of GPU node sizes, node labels and taints, topology-aware placement, device plugins, custom schedulers, or precise control over model-serving infrastructure. Refer to Running AI Workloads on AKS for insights into the control plane.\nExisting investment changes the answer. Trusted Helm charts, Flux reconciliation, policy libraries, golden clusters, and an experienced on-call team can make AKS the lower-risk path. Leaving a mature platform has a cost.\nHead-to-head Dimension Azure Container Apps AKS Scale to zero Native for suitable HTTP, TCP, and event-driven rules; zero replicas incur no resource-usage charge Pods can reach zero with KEDA, but worker-node cost remains unless node capacity also scales down; system capacity remains Pricing model Consumption is billed per replica resource allocation and external requests; Dedicated is instance-based Primarily node/VM-based, plus selected cluster tier and supporting services; control-plane terms depend on tier Operational overhead Microsoft operates the underlying orchestration, nodes, and platform upgrades AKS Automatic manages more lifecycle work; AKS Standard exposes more choices and shared responsibility Networking control VNet integration, internal environments, NSGs, private endpoints, and supported UDR patterns; less app-level programmability More CNI, ingress, egress, NetworkPolicy, Cilium, and service-mesh choice Extensibility No direct Kubernetes API, CRDs, or arbitrary operators; managed Dapr integration Full Kubernetes API and ecosystem, including operators, CRDs, admission, and mesh tooling Multi-region / DR Deploy separate environments per region and place Front Door or Traffic Manager above them; application and data failover remain yours Deploy separate clusters per region; Fleet can coordinate parts of fleet operations, but application and data failover remain yours Learning curve Smaller for teams that understand containers and Azure application primitives Higher because teams must understand Kubernetes and the selected AKS operating model CI/CD and GitOps Strong CI/CD and revision-based releases; GitOps is possible through IaC workflows but is not the native Kubernetes reconciliation model Mature Helm, Flux, Argo CD, policy-as-code, and Kubernetes-native promotion patterns The important row is not “features.” It is required control. Paying extra for unused control is wasteful. Losing control that you truly need creates architectural debt.\nCost reality check Consider three services, each allocated 0.25 vCPU and 0.5 GiB. They are active for ten hours on each of 22 business days and scale to zero outside that window. Assume three million external requests per month. Using illustrative public ACA Consumption rates of $0.000024 per active vCPU-second, $0.000003 per GiB-second, and $0.40 per million requests after the monthly free grants, the compute-and-request estimate is about $13 per month.\nIf all three replicas stay active throughout the month, the simplified ACA estimate increases to approximately $54. For AKS, two worker nodes costing about $0.10 per node-hour result in a $146 monthly node baseline, excluding disks, load balancers, logging, egress, support, or a paid control-plane tier. Using one node would be cheaper, but it would not be a realistic comparison for production availability.\nThis model is not an Azure quote. Prices differ based on region, agreement, savings plan, reservation, VM family, and duration. ACA’s free grant applies to the entire subscription, not on a per-app basis, so other apps might already be using it. The example assumes active billing during the specified hours; actual costs may vary due to idle times, cold starts, and changes in replica concurrency. Always verify with the Azure pricing calculator and actual utilization.\nThe pattern is more significant than the exact figure. ACA performs best when services are idle for extended periods and can scale down to zero. As replicas stay active continuously, the difference lessens. During sustained use, compare ACA Dedicated, Consumption savings plans, and a well-optimized AKS node pool. If an existing AKS cluster has spare capacity, the extra cost of running a small service is likely minimal. Cost is a factor, but not the sole consideration; aspects like control, isolation, reliability, and team skills are more important.\nAsk these five questions before defaulting to AKS Do we need the Kubernetes API? If a CRD, operator, custom controller, admission webhook, or Kubernetes-native platform contract is required, choose AKS. Do we need deep tenant or network control? Namespace-scale isolation, custom scheduling, pod-level policy, or a sophisticated mesh points to AKS. What does demand look like? Bursty and business-hours workloads favor ACA Consumption. Steady, high utilization deserves a measured comparison. What have we already built? A mature AKS platform can make another workload cheap and safe. A team with no Kubernetes operating model should count the platform build honestly. Where will this workload be in 18 months? Choose for the credible growth path, not an imagined hyperscale future. Moving later has a cost; over-platforming from day one has one too. Start with ACA when the answers reveal an application problem. Start with AKS when they reveal a platform problem.\nMy take Yes: Azure Container Apps is taking over smaller workloads from AKS, which is a positive development. It does not replace AKS; instead, it handles the APIs, workers, scheduled jobs, and small service setups that never required a full Kubernetes platform in the first place.\nAKS remains the better foundation when extensibility, scheduling, isolation, networking depth, or ecosystem compatibility is required. But choosing it out of habit is no longer conservative. For a small team, unnecessary Kubernetes can be riskier because each additional layer requires ownership during upgrades, incidents, and staff changes.\nI would reconsider this balance if AKS Automatic delivered true workload-level scale-to-zero economics without a persistent cluster baseline, or if ACA exposed substantially deeper policy, networking, and GitOps primitives without losing its application-platform simplicity. Until then, the default should be deliberate: ACA for the application-shaped problem; AKS for the Kubernetes-shaped one.\nRelated reading Optimizing Azure Container Apps Costs: Scaling to Zero, Workload Profiles, and FinOps Reducing AKS Costs: Autoscaling, Spot Nodes, Rightsizing, and FinOps Practices Azure Kubernetes Application Network, Part 1 Azure Kubernetes Application Network, Part 2 Azure Kubernetes Application Network, Part 3 Sources Azure Container Apps billing Scaling in Azure Container Apps Azure Container Apps networking Traffic splitting between revisions Dapr integration with Azure Container Apps AKS Automatic AKS network policies ","permalink":"https://wolkwacht.nl/posts/2026-09-04_is-azure-container-apps-eating-akss-lunch-for-smaller-workloads/","summary":"\u003cp\u003e\u003cimg alt=\"A small set of containerized services chooses between a lightweight managed platform and a more sophisticated Kubernetes platform\" loading=\"lazy\" src=\"/images/2026/aca-vs-aks-feature.png\"\u003e\u003c/p\u003e\n\u003cp\u003eKubernetes has become so familiar that many teams don\u0026rsquo;t choose to adopt it anymore; they just start using it. When a new internal API appears, someone sets up an AKS cluster, and within weeks, the team manages node pools, upgrade channels, ingress, identity, policies, observability, and a backlog of Helm charts. None of these decisions are incorrect, but the issue is that the workload might never have required a Kubernetes platform in the first place.\u003c/p\u003e","title":"Is Azure Container Apps Eating AKS's Lunch for Smaller Workloads?"},{"content":"\nAzure Kubernetes Service (AKS) removes most of the undifferentiated heavy lifting of running Kubernetes, but it does not remove your responsibility for securing what runs on top of it. Every week, new clusters go into production with permissive RBAC, public API servers, root-privileged containers, and no plan to stay current with CVEs. None of that is an AKS problem; it is an operating-model problem.\nThis practical, opinionated guide walks through securing an AKS cluster from start to finish: identity, network, workload, supply chain, runtime security, and the often-overlooked process of maintaining security over time.\nImportance of the shared responsibility line AKS handles the control plane components, such as the API server, etcd, scheduler, and controller-manager, while Microsoft is responsible for patching and security. The rest of the node OS setup, network segmentation, RBAC, workload security context, secrets management, image provenance, and regular patching is managed by you. AKS now offers two operating modes that shift the location of this boundary.\nAKS Automatic offers a secure baseline with numerous preconfigured controls, including auto-upgrade, Azure CNI Overlay with network policy, Workload Identity, Azure Policy, and Defender-ready defaults, all largely managed for you. AKS Standard provides full control over every knob, placing the responsibility on you to set each one correctly. Whichever mode you run, the hardening domains below are the same. What differs is how much of it AKS has already done for you (e.g., Microsoft Learn and Best practices for cluster security in AKS).\nAKS shared responsibility model: Microsoft manages the control plane, you manage identity, network policy, workload security, image provenance and upgrade cadence\nRely on established external frameworks instead of creating your own Before writing a single policy, identify the frameworks you will be assessed against. Reinventing a hardening checklist from memory is how gaps occur. Four references matter most for AKS:\nCIS Azure Kubernetes Service (AKS) Benchmark: a CIS benchmark designed specifically for AKS, created through a collaboration between the Center for Internet Security and the Azure team to ensure alignment with the Microsoft security baseline for AKS. (CIS blog). CIS Kubernetes Benchmark: the upstream, distribution-agnostic benchmark provides AKS with information on which of its node images meet the requirements of each CIS Kubernetes Benchmark version. (Microsoft Learn, CIS Kubernetes benchmark). NSA/CISA Kubernetes Hardening Guidance: a joint technical report by NSA and CISA targeting security teams, DevOps, and system administrators. It covers pod security, network separation, authentication, logging, and upgrade practices, and is currently at version 1.2. (CISA advisory, full PDF). Microsoft Cloud Security Benchmark for AKS: Microsoft\u0026rsquo;s control mapping for the AKS resource type covers identity, network, logging, and posture, supporting the compliance recommendations in Microsoft Defender for Cloud. However, the published baseline document is based on benchmark v1.0, which Microsoft indicates may be outdated. Therefore, consider it a control map and verify details against the current AKS security documentation (Microsoft Learn, Azure security baseline for AKS). Use the CIS AKS Benchmark and the NSA/CISA guide as your reference controls, with Azure Policy and Defender for Cloud to ensure enforcement and provide evidence. This setup gives CISOs a verifiable connection from stating \u0026ldquo;we are hardened\u0026rdquo; to a recognized external standard, making it significantly more persuasive in compliance reviews than merely referencing an internal wiki page.\n1. Identity and access: kill standing access first The API server is the most valuable target within the cluster. Two modifications are more important than anything else in this context:\nIntegrate Kubernetes RBAC with Microsoft Entra ID so that cluster access is managed through your current identity governance, conditional access, and MFA policies, rather than using a separate kubeconfig credential that no one rotates. Disable all local accounts completely. Even if Entra integration is active, the clusterAdmin local credential acts as non-auditable backdoor unless you explicitly deactivate it. az aks create \\ --resource-group rg-aks-prod \\ --name aks-prod-weu \\ --location westeurope \\ --enable-aad \\ --enable-azure-rbac \\ --aad-admin-group-object-ids `\u0026lt;entra-group-object-id\u0026gt;` \\ --disable-local-accounts \\ --generate-ssh-keys For workloads, avoid distributing long-lived Azure credentials as Kubernetes Secrets. Instead, use Microsoft Entra Workload ID, which federates a Kubernetes service account with an Entra app registration via OIDC. This setup allows pods to exchange a short-lived Kubernetes token for an Azure AD token, eliminating the need to store any secret in the cluster.\naz aks create \\ --resource-group rg-aks-prod \\ --name aks-prod-weu \\ --enable-oidc-issuer \\ --enable-workload-identity \\ --generate-ssh-keys References: Microsoft Entra Workload ID for AKS, Manage local accounts with Microsoft Entra integration\n2. Network: shrink the attack surface before you filter it Hardening the network starts with reducing exposure, then adding filtering on top.\nMake the API server private. A private cluster eliminates the public endpoint entirely; the API server can only be accessed via your VNet, a peered network, or a private DNS zone.\naz aks create \\ --resource-group rg-aks-prod \\ --name aks-prod-weu \\ --network-plugin azure \\ --network-policy azure \\ --enable-private-cluster \\ --private-dns-zone system \\ --generate-ssh-keys Enforce NetworkPolicy. Using network policy in Azure (or Cilium on newer Azure CNI Overlay clusters) allows you to create standard Kubernetes NetworkPolicy objects that are actively enforced rather than ignored. For example, you can block pod egress to the Azure Instance Metadata Service (IMDS), which is often exploited as a pivot point for credential theft in cloud-hosted clusters.\napiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: restrict-instance-metadata namespace: default spec: podSelector: matchLabels: {} policyTypes: - Egress egress: - to: - ipBlock: cidr: 10.10.0.0/16 except: - 169.254.169.254/32 Use this pattern across all namespaces lacking an explicit, reviewed requirement to call IMDS. Reference: Best practices for cluster security in AKS: restrict access to the Instance Metadata API\n3. Workloads: least privilege at the pod level Cluster-level controls are ineffective if all pods run as root, use writable filesystems, and retain full capabilities. Establish a security baseline that every workload is required to adhere to:\napiVersion: v1 kind: Pod metadata: name: hardened-example spec: securityContext: runAsNonRoot: true runAsUser: 10001 seccompProfile: type: RuntimeDefault containers: - name: app image: myregistry.azurecr.io/app:1.4.2 securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: [\u0026#34;ALL\u0026#34;] resources: limits: cpu: \u0026#34;500m\u0026#34; memory: \u0026#34;256Mi\u0026#34; Don\u0026rsquo;t depend on developers remembering this. Use the Azure Policy add-on for AKS to enforce it centrally; it integrates Gatekeeper/OPA and provides preconfigured initiatives aligned with the Kubernetes Pod Security Standards.\naz aks create \\ --resource-group rg-aks-prod \\ --name aks-prod-weu \\ --enable-addons azure-policy \\ --generate-ssh-keys Set the \u0026ldquo;Kubernetes cluster pod security restricted standards for Linux-based workloads\u0026rdquo; initiative to deny mode for production namespaces, and audit mode during the cleanup of existing violations. Ensure enforcement is active:\nkubectl get constrainttemplates Then confirm a privileged pod is rejected:\ncat \u0026lt;\u0026lt;EOF | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: nginx-privileged spec: containers: - name: nginx-privileged image: mcr.microsoft.com/oss/nginx/nginx:1.15.5-alpine securityContext: privileged: true EOF Expected result: the API server denies the pod with denied by azurepolicy-container-no-privilege…, and the pod is not scheduled. Reference: Use Azure Policy to secure your AKS clusters.\n4. Secrets and the software supply chain The primary failure modes here are secrets stored in environment variables or configuration maps, and unscanned, unpatched images running in production.\nMount secrets from Azure Key Vault via the Secrets Store CSI Driver instead of native Kubernetes Secrets, ensuring values are not stored unencrypted in etcd at the application layer and enabling centralized rotation. Limit image pulls to your Azure Container Registry (ACR) using admission-controlled trusted registries, and activate ACR content trust/image signing for all production deployments. Enable Image Cleaner to automatically detect and remove stale, vulnerable images (using Trivy), preventing them from being targets for lateral movement. az aks create \\ --resource-group rg-aks-prod \\ --name aks-prod-weu \\ --enable-image-cleaner \\ --image-cleaner-interval-hours 24 \\ --generate-ssh-keys Reference: Use Image Cleaner on AKS.\n5. Runtime protection and visibility Static hardening prevents an attacker from succeeding initially, while runtime protection alerts you to attempts later. Enable Microsoft Defender for Containers to assess image vulnerabilities in ACR, detect threats via Kubernetes audit logs, and receive runtime anomaly alerts on nodes.\naz aks create \\ --resource-group rg-aks-prod \\ --name aks-prod-weu \\ --enable-defender \\ --generate-ssh-keys Combine this with Container Insights (Azure Monitor) to collect logs and metrics, and direct Kubernetes audit logs to a Log Analytics workspace to create an audit trail, not just generate alerts. Reference: Introduction to Microsoft Defender for Containers.\nThe five AKS defense-in-depth layers: identity and access, network, workloads, supply chain, and runtime and monitoring, stacked from outer to inner\nPutting it together: a hardened baseline in one command Combine all the above into one cluster-create call (adjust names, IDs, and region for your environment):\naz aks create \\ --resource-group rg-aks-prod \\ --name aks-prod-weu \\ --location westeurope \\ --enable-aad \\ --enable-azure-rbac \\ --aad-admin-group-object-ids `\u0026lt;entra-group-object-id\u0026gt;` \\ --disable-local-accounts \\ --enable-oidc-issuer \\ --enable-workload-identity \\ --network-plugin azure \\ --network-policy azure \\ --enable-private-cluster \\ --private-dns-zone system \\ --enable-defender \\ --enable-image-cleaner \\ --image-cleaner-interval-hours 24 \\ --enable-addons azure-policy \\ --auto-upgrade-channel stable \\ --node-os-upgrade-channel NodeImage \\ --generate-ssh-keys This isn\u0026rsquo;t a copy-and-paste-to-production command; it\u0026rsquo;s a checklist encoded as CLI flags. Review each flag against your network topology (hub-spoke peering for the private DNS zone, existing Entra groups, and existing Log Analytics workspace) before running it.\nKeeping the hardening current: hardening is not a one-time event This aspect is often overlooked in blog posts and deployments, but it\u0026rsquo;s crucial because hardened clusters can slowly become vulnerable over time. A cluster that is CIS-compliant when deployed isn\u0026rsquo;t automatically compliant several months later unless it undergoes active maintenance. The support status for Kubernetes versions, node OS packages, and the policy and CVE landscape are all in constant flux.\nA continuous five-step loop for keeping AKS hardening current: set upgrade channels, scan with kube-bench, watch Azure Policy state, track Defender secure score, reconcile via GitOps\nAutomate version and node currency AKS supports only a limited range of Kubernetes minor versions, usually from N to N-2. If you don\u0026rsquo;t update, the cluster will eventually fall out of support, stopping security patches for the control plane. To avoid this, specify an upgrade channel instead of leaving it on none.\naz aks update \\ --resource-group rg-aks-prod \\ --name aks-prod-weu \\ --auto-upgrade-channel stable \\ --node-os-upgrade-channel NodeImage stable maintains the cluster on the most recent patch of minor version N-1, balancing up-to-date software with the risk of deploying a new minor release into production. Combine it with a Planned Maintenance window of at least four hours to ensure upgrades occur at a predictable time rather than during peak traffic. Reference: Automatically upgrade an AKS cluster.\nNode OS patches are independent of Kubernetes version upgrades. Linux nodes automatically download security patches nightly, but the node image itself updates only during an upgrade. This means a newly scaled node might run on an outdated, unpatched image. The NodeImage auto-upgrade channel addresses this by refreshing node images automatically as Microsoft releases new ones, approximately weekly. Reference: Upgrade AKS node images.\nCheck for drift on a schedule, not by accident Configuration drift happens when an engineer adds a hostNetwork: true pod, a NetworkPolicy gets silently removed during troubleshooting, or a namespace is left out of Azure Policy and not re-added. This is the leading cause of a hardened baseline weakening. To avoid this, three complementary checks are performed regularly:\n1. CIS benchmark scans with kube-bench. Run kube-bench as a scheduled Kubernetes Job on your worker nodes. Since managed control planes like AKS restrict access to master nodes, focus the runs on node-level and policy checks for which you are responsible. Consider the remaining checks as covered under Microsoft\u0026rsquo;s control-plane SLA.\napiVersion: batch/v1 kind: CronJob metadata: name: kube-bench-node namespace: security spec: schedule: \u0026#34;0 3 * * 1\u0026#34; jobTemplate: spec: template: spec: hostPID: true containers: - name: kube-bench image: docker.io/aquasec/kube-bench:v0.9.5 command: [\u0026#34;kube-bench\u0026#34;, \u0026#34;node\u0026#34;, \u0026#34;--benchmark\u0026#34;, \u0026#34;cis-1.9\u0026#34;] volumeMounts: - name: var-lib-kubelet mountPath: /var/lib/kubelet readOnly: true - name: etc-systemd mountPath: /etc/systemd readOnly: true restartPolicy: Never volumes: - name: var-lib-kubelet hostPath: path: /var/lib/kubelet - name: etc-systemd hostPath: path: /etc/systemd backoffLimit: 1 Send results to Log Analytics and trigger alerts for new failures; not only rely on the absolute score, but also on a drop from 96% to 91%, as that is more actionable than a static \u0026ldquo;91% compliant\u0026rdquo; tile on the dashboard. Reference: kube-bench, aquasecurity/kube-bench on GitHub.\n2. Azure Policy compliance state as continuous evidence. Since the Azure Policy add-on assesses each admission request and regularly rechecks existing objects, its compliance status acts as a real-time drift indicator rather than a snapshot in time. Query it regularly and provide the results to whatever the CISO\u0026rsquo;s team reviews.\naz policy state list \\ --resource-group rg-aks-prod \\ --filter \u0026#34;complianceState eq \u0026#39;NonCompliant\u0026#39;\u0026#34; \\ --query \u0026#34;[].{policy:policyDefinitionName, resource:resourceId}\u0026#34; \\ -o table 3. Microsoft Defender for Cloud secure score, tracked over time. Defender regularly reviews the AKS security baseline, including coverage for identity, network, and image scanning, and generates severity-ranked recommendations. Monitor the secure score for the AKS resource group weekly; if the score remains flat or declines after a hardening effort, it indicates that drift has begun.\nClose the loop with change management, not just tooling Tools detect drift while the process prevents it. In practice, two habits make the difference:\nStore all cluster-hardening configurations, such as Azure Policy assignments, NetworkPolicy manifests, and admission baselines, in Git via pull requests. Reconcile these with GitOps tools like Flux or Argo CD instead of applying changes manually with kubectl. Any manual kubectl edit to a hardened resource will be detected as drift during the next reconciliation. Re-run the CIS AKS Benchmark and NSA/CISA guidance checklist with each minor Kubernetes upgrade instead of annually. New Kubernetes minor versions sometimes alter default behaviors, such as API deprecations and PodSecurity admission defaults, which can silently reopen previously closed controls. A closing checklist If there\u0026rsquo;s one key takeaway from this post, it\u0026rsquo;s that hardening AKS isn\u0026rsquo;t achieved with a single az aks create command. Instead, it involves five interconnected layers: identity, network, workload, supply chain, and runtime, each aligned with an external framework for auditability. Additionally, ongoing practices such as weekly kube-bench scans, continuous Azure Policy checks, maintaining Defender secure scores, and ensuring upgrade channels remain active are part of a continuous effort. Consider the first a project and the second a permanent operational commitment; doing so ensures the \u0026ldquo;hardened\u0026rdquo; status remains valid even after a year.\nReferences Best practices for cluster security and upgrades in AKS — Microsoft Learn Concepts: Security in AKS — Microsoft Learn Azure security baseline for AKS — Microsoft Learn Manage local accounts with Microsoft Entra integration — Microsoft Learn Deploy and configure an AKS cluster with Microsoft Entra Workload ID — Microsoft Learn Use Azure Policy to secure your AKS clusters — Microsoft Learn Use Image Cleaner on AKS — Microsoft Learn Introduction to Microsoft Defender for Containers — Microsoft Learn Automatically upgrade an AKS cluster — Microsoft Learn Upgrade AKS node images — Microsoft Learn Center for Internet Security (CIS) Kubernetes benchmark — Microsoft Learn New Release: CIS Azure Kubernetes Service (AKS) Benchmark — CIS CIS Kubernetes Benchmarks — CIS Updated Kubernetes Hardening Guide — CISA Kubernetes Hardening Guide v1.2 (PDF) — NSA/CISA kube-bench — aquasecurity/kube-bench, GitHub ","permalink":"https://wolkwacht.nl/posts/secure-azure-kubernetes-service-aks/","summary":"\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://miro.medium.com/v2/resize:fit:1050/1*Fr-4dUIA6Y7uo_cOKAnxcQ.jpeg\"\u003e\u003c/p\u003e\n\u003cp\u003eAzure Kubernetes Service (AKS) removes most of the undifferentiated heavy lifting of running Kubernetes, but it does not remove your responsibility for securing what runs on top of it. Every week, new clusters go into production with permissive RBAC, public API servers, root-privileged containers, and no plan to stay current with CVEs. None of that is an AKS problem; it is an operating-model problem.\u003c/p\u003e\n\u003cp\u003eThis practical, opinionated guide walks through securing an AKS cluster from start to finish: identity, network, workload, supply chain, runtime security, and the often-overlooked process of maintaining security over time.\u003c/p\u003e","title":"Secure Azure Kubernetes Service (AKS)"},{"content":"\nA cloud architect’s field notes on apple/container v1.2.0, for the devops engineers and application specialists who have to make the “what runs on our Macs” decision.\nA year ago I published Running Containers the Apple Way: A First Look into Apple Container on macOS. At the time, container was a WWDC 2025 curiosity: a Swift CLI with no stability guarantees and a limited list of things it could actually do. I was cautiously optimistic and noted, in effect, “watch this space.”\nI watched the space. It moved.\ncontainer turned one year old on June 9, 2026, shipping its first stable 1.0.0 release, with CLI and XPC APIs frozen and patch-level compatibility guaranteed. Since then, it has continued to progress steadily: from 1.1.0 to 1.2.0 as of this writing, with the 1.2.0 release on July 29, 2026. This blog provides a validated, practical overview of the project\u0026rsquo;s current state: what\u0026rsquo;s changed, what can be built with it now, and how an architect can justify tooling choices to a platform team, highlighting where it truly integrates into a DevOps workflow and where it still falls short.\nThe short version, for the skim readers\ncontainer is a native macOS CLI written in Swift, designed specifically for Apple silicon. Unlike Docker Desktop, Colima, or OrbStack, which run containers inside a shared Linux VM, container operates in a lightweight, dedicated VM for each container. It handles OCI-compliant images, meaning you can run any image you pull with Docker and deploy any container built with it anywhere OCI images are supported.\nA year in, the headline changes are:\n1.0.0 stability: The CLI and XPC API are now fixed at the minor-version boundary, allowing you to build tools on top without concern for breaking changes with each release. Container machine: A new feature offering persistent, systemd-capable Linux environments with your Mac’s home directory and user account directly mapped. Includes a TOML config file replacing the old UserDefaults settings, a proper container copy command, enhanced JSON/YAML output, container stats, capability management, custom init images, nested virtualization, and IPv6 networking. Designed for Apple Silicon only, best used with macOS 26 (macOS 15 has limitations), and still lacking a built-in Compose feature after a year. That gap remains, and I plan to address it. Details on requirements and how to install or upgrade. container requires a Mac with Apple silicon. It is supported on macOS 26, where the Virtualization and networking improvements it depends on are available; macOS 15 is usable but has documented network limitations (containers can’t reach each other, and multiple isolated networks aren’t available).\nIf you’re new to it, grab the signed installer from the releases page and start the system service:\ncontainer system start If you\u0026rsquo;ve already installed it from last year’s earlier versions, the upgrade can be done with a single script call; no need to uninstall beforehand.\ncontainer system stop /usr/local/bin/update-container.sh container system start Downgrading remains straightforward, as Apple maintains an open escape route:\ncontainer system stop /usr/local/bin/uninstall-container.sh -k # -k keeps your data /usr/local/bin/update-container.sh -v 0.3.0 container system start Check what you’re running with:\ncontainer system version --format json Under the hood: still one VM per container, and that’s the whole point\nApple\u0026rsquo;s architectural gamble from a year ago remains the same but has matured. Instead of booting a single large Linux VM to run all containers, as most Docker-on-Mac solutions do, it leverages the open-source Containerization Swift package to launch a minimal, dedicated lightweight VM for each individual container.\nPractically, that buys you three things Apple is explicit about in its own technical overview:\nSecurity: each container operates with complete VM isolation, using only essential utilities and libraries and without sharing a kernel namespace. Privacy: when you attach host data to a container, you only mount what it requires. There’s no shared VM that requires pre-mounting everything \u0026ldquo;just in case.” Performance: even with full VMs, boot times match those of containers in shared VMs, and the memory usage remains close to what the containerized process requires. The CLI communicates with a launchd agent named container-apiserver, which oversees three groups of XPC helpers: container-core-images for handling images and content storage, container-network-vmnet for virtual networking, and a separate container-runtime-linux instance for each running container. This entire setup utilizes macOS’s Virtualization framework, vmnet, XPC, launchd, Keychain, and unified logging, giving it a seamless integration without any feelings of being added as an afterthought.\nA note for capacity planning: the macOS Virtualization framework currently offers only limited support for memory ballooning. For example, if you launch a container with memory 16g, Activity Monitor might display significantly less memory usage because pages freed by the Linux guest aren’t always returned to the host. On a Mac running numerous memory-intensive containers, you might need to restart them periodically. This is a known and documented limitation, not a bug that requires extensive troubleshooting.\nAn example: build, run, and publish a web server\nStart the system service and configure a local DNS domain (optional but recommended), which assigns a .test hostname to each named container.\ncontainer system start sudo container system dns create test Write a Dockerfile:\nFROM docker.io/python:alpine WORKDIR /content RUN apk add curl RUN echo \u0026#39;\u0026lt;!DOCTYPE html\u0026gt;\u0026lt;html\u0026gt;\u0026lt;head\u0026gt;\u0026lt;title\u0026gt;Hello\u0026lt;/title\u0026gt;\u0026lt;/head\u0026gt;\u0026lt;body\u0026gt;\u0026lt;h1\u0026gt;Hello, world!\u0026lt;/h1\u0026gt;\u0026lt;/body\u0026gt;\u0026lt;/html\u0026gt;\u0026#39; \u0026gt; index.html CMD [\u0026#34;python3\u0026#34;, \u0026#34;-m\u0026#34;, \u0026#34;http.server\u0026#34;, \u0026#34;80\u0026#34;, \u0026#34;--bind\u0026#34;, \u0026#34;0.0.0.0\u0026#34;] Build it:\ncontainer build --tag web-test --file Dockerfile . Run it, detached, self-cleaning on exit:\ncontainer run --name my-web-server --detach --rm web-test container ls confirms it’s up and shows its IP on the isolated vmnet subnet:\nID IMAGE OS ARCH STATE IP CPUS MEMORY STARTED\nmy-web-server web-test:latest linux arm64 running 192.168.64.3/24 4 1024 MB 2026–08–06T14:42:07Z\nbuildkit ghcr.io/apple/container-builder-shim/builder:0.13.0 linux arm64 running 192.168.64.2/24 2 2048 MB 2026–08–06T13:33:10Z\nHit it directly by IP, or by the .test hostname you set up earlier:\ncurl http://my-web-server.test Watch it live with the resource-monitoring command that shipped this year:\ncontainer stats --no-stream my-web-server Container ID Cpu % Memory Usage Net Rx/Tx Block I/O Pids\nmy-web-server 0.07% 37.30 MiB / 1.00 GiB 3.68 KiB / 0.59 KiB 19.86 MiB / 2.30 MiB 1\nAnd publish it to any OCI-compliant registry; Docker Hub is the default, but you can point [registry].domain in ~/.config/container/config.toml anywhere:\ncontainer registry login some-registry.example.com container image tag web-test some-registry.example.com/fido/web-test:latest container image push some-registry.example.com/fido/web-test:latest None of this needed Docker Desktop, a paid license, or an unwanted background VM.\nWhat’s genuinely new since last year’s “first look” container machine — the headline 1.0 feature This feature distinguishes the container\u0026rsquo;s purpose. While \u0026lsquo;container run\u0026rsquo; models a single application process, \u0026lsquo;container machine\u0026rsquo; provides a persistent Linux environment booted from a standard OCI image, running its real init system, with your macOS username and home directory automatically mapped.\ncontainer machine create ubuntu:24.04 --name dev container machine run -n dev whoami # your host username, not root container machine run -n dev pwd # /home/\u0026lt;you\u0026gt; — your Mac home dir, mounted in container machine run -n dev # interactive shell Set a default so you can drop -n, resize resources on the fly, and run real background services with systemd:\ncontainer machine set-default dev container machine set -n dev cpus=4 memory=8G container machine stop dev \u0026amp;\u0026amp; container machine run -n dev -- nproc container machine run -n dev -- systemctl start postgresql For a DevOps audience, this straightforward solution addresses the need for a genuine Ubuntu environment to test deployment scripts while still accessing the same repository used in VS Code on a Mac. You edit directly on macOS, compile, run, and test on Linux without copying, rsyncing, or complex bind-mount configurations. Additionally, you can quickly create separate machines for each target distribution, such as Alpine, Ubuntu, or Debian, each sharing the same $HOME and dotfiles.\nA TOML configuration file The previous get/set subcommands for the UserDefaults-backed container system property have been removed, marking a documented breaking change in CLI version 1.0. Configuration is now stored in ~/.config/container/config.toml, offering greater transparency and simplifying templating across multiple engineering laptops.\ncontainer system property ls [build] cpus = 2 memory = \u0026#34;2048mb\u0026#34; rosetta = true [container] cpus = 4 memory = \u0026#34;1gb\u0026#34; [registry] domain = \u0026#34;docker.io\u0026#34; If you’re deploying this to a team, such a file is exactly the type you\u0026rsquo;d add to a dotfiles repository or distribute through MDM.\ncontainer cp, richer inspection, and container stats container cp (host ↔ container file transfer) shipped in 1.0 after sitting open as a feature request since the very first weeks of the project. JSON, YAML, and TOML output for list/inspect across containers, images, networks, and volumes was normalized in the same release, which matters the moment you start scripting against this tool rather than typing commands by hand:\ncontainer ls --format json --all | jq \u0026#39;.[] | select(.status == \u0026#34;running\u0026#34;) | [.configuration.id, .networks[0].address]\u0026#39; Multiplatform builds and Rosetta-backed amd64 You can build for both architectures simultaneously and run the x86–64 version seamlessly using Rosetta translation:\ncontainer build --arch arm64 --arch amd64 --tag registry.example.com/fido/web-test:latest --file Dockerfile . container run --arch amd64 --rm registry.example.com/fido/web-test:latest uname -a Fine-grained Linux capabilities, custom init, and nested virtualization Containers initially have a limited, documented ability set by default. You can explicitly add, remove, or reset these capabilities, which is especially helpful during compliance reviews to clearly define what a container is permitted to do.\ncontainer run --cap-drop ALL --cap-add SETUID --cap-add SETGID alpine id — init provides a lightweight PID-1 that handles zombie processes and forwards signals for applications not originally designed to be PID 1. — init-image extends this functionality by allowing you to wrap vminitd with custom boot-time logic, such as an eBPF filter, a logging sidecar, or custom instrumentation, before the container’s main entrypoint executes. On M3-and-later hardware, virtualization offers nested virtualization within the guest, enabling teams to run hypervisor workloads inside their VM-based containers.\nIsolated networks and IPv6 macOS 26 introduces the container network create feature, enabling the creation of multiple isolated vmnet subnets. This is useful for replicating a segmented network topology locally instead of relying on a single flat subnet:\ncontainer network create foo --subnet 192.168.100.0/24 --subnet-v6 fd00:1234::/64 container run -d --name my-web-server --network foo --rm web-test If your understanding of container is limited to what the WWDC 2025 build could do, here\u0026rsquo;s a comparison in one table.\nUse cases: where this fits in a real DevOps workflow Local development on a compliance-conscious Mac fleet. Docker Desktop’s licensing terms require a paid subscription for larger companies. The container is Apache-2.0 licensed, built by the OS vendor, and has no license restrictions, making it an easy choice for platform teams to standardize tooling across engineering.\nReproducibility for CI related tasks on Apple silicon runners. As more CI fleets adopt Apple silicon for cost savings and performance benefits, a native, scriptable, OCI-compliant runtime that operates without a licensed daemon simplifies the process. The JSON-first output introduced in version 1.0 significantly eases integration with pipeline tools.\nMulti-distro compatibility testing simplifies the process by transforming the need to manually set up three VMs to check whether the install script works on Debian, Ubuntu, and Alpine into just three straightforward container creation commands. Each command already includes your dotfiles and repository, making setup faster and more efficient.\nSecurity-sensitive workloads require true isolation rather than just namespace separation. Using per-container VMs with a minimal attack surface presents a significantly different security risk compared to shared-kernel containers. This distinction is especially important for anyone running third-party or less-trusted images locally.\nCross-architecture builds without needing a second machine — combining arm64 and amd64 architectures in one build command. Powered by Rosetta-backed emulation, which is significantly faster than QEMU translation, this setup simplifies testing for compatibility with our x86 datacenter without the need to provision an Intel machine.\nPrototyping systemd-dependent services. If your production environment uses real systemd units, container machine allows you to develop in an environment that closely matches it, rather than simulating it with a foreground process inside a standard container.\nOnboarding and standardized development environments can be effectively managed with a Dockerfile documented in container-machine.md and a one-line command to create a container machine. This approach serves as a practical alternative to a golden AMI or a Vagrantfile when onboarding new engineers on a Mac. It provides a reliable Linux environment within minutes, complete with preconfigured dotfiles, at minimal cost other than disk space.\nAuditable and scriptable tools designed for platform teams. The transition to standardized JSON, YAML, and TOML output covering list and inspect, along with a templateable and diffable config file, provides exactly what a platform engineering team requires. It enables the development of internal tools on a container runtime without the need to reverse-engineer custom text outputs.\nReasons to choose it and areas where I’d exercise caution Choose this option if you’re using Apple Silicon, need a native, license-free OCI runtime with genuine VM-level isolation, appreciate a stable CLI and API that Apple now guarantees, and don’t require multi-container orchestration by default.\nHold off on using it or combine it with another tool if your workflow relies heavily on Docker Compose. Even after a year, there\u0026rsquo;s still no official container-compose command. Apple has recognized this gap, and the community has filled it with tools like container-compose, but these are not as polished as the native experience users expect from Compose. If your team mainly works with Compose files, plan to spend time assessing these community solutions before making a decision. You’re also limited to Apple silicon devices and, for full features, macOS 26 or later—there’s intentionally no cross-platform support. Additionally, the memory ballooning issue mentioned earlier should be noted in your onboarding documentation so that team members don\u0026rsquo;t report it as a bug, since it\u0026rsquo;s an expected behavior.\nWhere it sits next to Docker Desktop and OrbStack I won’t claim to have conducted a comprehensive head-to-head comparison in this article, so I will concentrate on presenting only verified facts rather than referencing others\u0026rsquo; data. Both Docker Desktop and OrbStack operate containers within a shared Linux VM, whereas containers generally run in isolated environments. This fundamental structural difference largely accounts for other variations: containers provide enhanced per-workload isolation and reduce the potential impact of issues, but they also come with a less developed ecosystem, lacking features like Compose, having fewer plugins, and supporting fewer third-party integrations. If your main concerns are “native, free, isolated, and you\u0026rsquo;re open to early adoption,\u0026rdquo; then container is a practical choice now, compared to a year ago. However, if you\u0026rsquo;re seeking a seamless, \u0026ldquo;drop-in\u0026rdquo; Compose replacement with no changes to your workflow, that solution isn’t quite ready yet.\nThe verdict, one year in Last year’s piece ended with “watch this space.” After a year, a major stable release, and a minor 0.2 update, the honest update is: it was worth watching. The container evolved from a WWDC demo with no stability guarantees to version 1.2.0, featuring a frozen CLI and API, a new persistent environment model with a container machine, and numerous operational improvements, including TOML config, cp, stats, capability control, and multiplatform builds. I would now consider it for teams evaluating container tools on Apple silicon. Although it’s not yet a full replacement for Docker Desktop, mainly because of the Compose gap for DevOps engineers and application specialists working natively on Apple silicon, it’s no longer just a curiosity. It has become part of the infrastructure.\n","permalink":"https://wolkwacht.nl/posts/apple-container-one-year-on-from-tech-demo-to-a-real-docker-desktop-alternative/","summary":"\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*3ujuYjD-4EBlsmyTGA0ntg.jpeg\"\u003e\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eA cloud architect’s field notes on apple/container v1.2.0, for the devops engineers and application specialists who have to make the “what runs on our Macs” decision\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003eA year ago I published \u003ca href=\"https://medium.com/@jurgenallewijn/running-containers-the-apple-way-a-first-look-into-apple-container-on-macos-5740445f47e5\"\u003eRunning Containers the Apple Way: A First Look into Apple Container on macOS\u003c/a\u003e. At the time, container was a WWDC 2025 curiosity: a Swift CLI with no stability guarantees and a limited list of things it could actually do. I was cautiously optimistic and noted, in effect, “watch this space.”\u003c/p\u003e","title":"Apple container, one year on: from tech demo to a real Docker Desktop alternative"},{"content":"\nIntroduction: Cost Is an Outcome of Scheduling Decisions Your AKS bill is calculated before Azure sends an invoice. Kubernetes translates pod specifications into resource requirements, including resource requests, affinity rules, topology constraints, DaemonSet overhead, and scaling limits, all of which impact the efficiency of pod placement on nodes.\nThis post\u0026rsquo;s main point is clear: durable optimization involves enhancing the entire chain, from workload demand to pod replicas, schedulable capacity, VM allocation, and cost attribution. Reducing node count without understanding this chain does not save money; it simply shifts costs into latency, evictions, throttling, and extended recovery periods.\nBuild a Cost and Capacity Model There are four key numbers, seldom the same: Provisioned capacity is what the VM SKU shows; allocatable capacity is what’s available after the OS, kubelet, system pods, eviction thresholds, and DaemonSets take their part; requested capacity is what pods have reserved; and consumed capacity is what pods actually use. The scheduler assigns pods based on allocatable capacity and declared requests, not on average utilization.\nTo analyze costs effectively, monitor metrics such as node-pool hourly costs, allocatable CPU and memory, total pod requests, actual P50/P95 resource consumption, unschedulable pod durations, and workload throughput. Of all dashboard metrics, two ratios are most critical: request efficiency (usage ÷ requested resources) and allocation efficiency (requested ÷ allocatable resources). Both are essential because high request efficiency alone can mask inefficient bin-packing, especially when pod shapes do not align with the chosen VM SKU, resulting in wasted fractional CPU or memory on each node.\nTreat Autoscaling as Interacting Control Loops AKS typically operates four autoscalers simultaneously, each at a different layer and interval, without awareness of the others’ goals.\nHorizontal Pod Autoscaler (HPA) changes replica count based on CPU, memory, or custom metrics. KEDA (available as a managed AKS add-on) converts external event signals queue depth, Prometheus metrics, GPU queue wait time into replica demand, including scale-to-zero. Vertical Pod Autoscaler (VPA) recommends, or on AKS 1.34+ applies in-place, per-pod resource requests. Cluster Autoscaler (or Node Auto Provisioning) reacts only when pods can’t be scheduled, or nodes become removable. Poor coordination leads to common failure modes: inflated requests cause early scale-out; missing requests skew HPA, risking unsafe pod density; an unstable HPA without proper Cluster Autoscaler stabilization causes node churn; and running VPA in Auto mode with HPA on the same signal results in conflicting decisions.\n# HPA scales on request-based CPU utilization apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: checkout-api namespace: payments spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: checkout-api minReplicas: 3 maxReplicas: 30 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 65 behavior: scaleUp: stabilizationWindowSeconds: 30 scaleDown: stabilizationWindowSeconds: 300 --- # VPA in recommendation-only mode: safe to run alongside HPA apiVersion: autoscaling.k8s.io/v1 kind: VerticalPodAutoscaler metadata: name: checkout-api-vpa namespace: payments spec: targetRef: apiVersion: apps/v1 kind: Deployment name: checkout-api updatePolicy: updateMode: \u0026#34;Off\u0026#34; # observe recommendations before enabling Auto Scale-up should aim for meeting application latency goals; scale-down must consider startup times, pod disruption budgets, termination grace periods, and rescheduling headroom. Provide production pools with a resilient minimum, while letting asynchronous pools scale down to zero. Adjust Cluster Autoscaler profiles based on workload classes, as a single, aggressive profile across the cluster can ultimately negatively impact a workload that wasn’t properly tuned.\nImprove Scheduler Bin-Packing Through Rightsizing Rightsizing isn’t just about \u0026ldquo;lowering requests.” CPU resources are compressible and throttled when there\u0026rsquo;s contention, but memory isn’t. Memory pressure leads to eviction or OOM kills. It\u0026rsquo;s important to set requests independently for each resource and container, including service-mesh sidecars and log shippers, which are often left at default settings and can silently dominate a pod’s resource footprint.\nBase your measurements on historical data from Azure Managed Prometheus or Container Insights, rather than relying on a single kubectl top sample. Include data on peaks, deployments, cold starts, GC pauses, scheduled jobs, and seasonal traffic variations. Set VPA to updateMode: “Off” initially, allowing you to review its lower bound, target, and upper bound before applying changes to production manifests.\nThe shape of a pod is as important as overall utilization. A workload may seem efficient overall but still leave resources unused because CPU-heavy and memory-heavy pods can\u0026rsquo;t co-locate on the same node. Consider requested pod sizes against candidate VM CPU:memory ratios, maximum pods per node, DaemonSet overhead, and zone placement. Rightsizing pods and selecting suitable node SKUs should be viewed as a single, integrated task.\nDesign Node Pools Around Workload Constraints Cluster Autoscaler assesses each node pool separately, considering its specific labels, taints, zones, and VM types. Having too many small pools fragments capacity and reduces utilization, while a single heterogeneous pool can weaken isolation and lead to unpredictable scaling.\nA practical default configuration involves a small dedicated system pool for essential AKS components, along with a limited number of workload pools based on valid scheduling needs such as taints and tolerations, node affinity, topology spread constraints, persistent-volume zone affinity, GPU requirements, or CPU architecture. Setting minimum and maximum counts for each pool isn\u0026rsquo;t only a cost control measure; it also influences resilience.\nNode Auto Provisioning (NAP), AKS’s managed Karpenter integration, became generally available in 2025 and is now the preferred default for automatically matching node configurations to pending pod demands. It selects VM size and family based on current pending pods rather than a pre-established pool. However, AKS Standard with manually configured node pools still offers the best option when teams require precise, auditable control over node-pool behavior.\nEngineer Spot Capacity as an Interruption Domain Treat Spot nodes as unreliable capacity that is intentionally isolated and consumed, not as direct substitutes for regular nodes. Typically, a design includes a standard system pool, a reliable baseline pool scaled for steady workload, and an autoscaled Spot pool handling interruptible demand.\n# AKS automatically applies this label and taint to every node in a Spot pool # shown here for reference, not something you create by hand. apiVersion: v1 kind: Node metadata: labels: kubernetes.azure.com/priority: spot spec: taints: - key: kubernetes.azure.com/scalesetpriority value: spot effect: NoSchedule --- # Workload tolerates Spot but prefers it, falls back to regular nodes on eviction apiVersion: apps/v1 kind: Deployment metadata: name: batch-worker labels: app: batch-worker spec: replicas: 6 selector: matchLabels: app: batch-worker template: metadata: labels: app: batch-worker spec: tolerations: - key: kubernetes.azure.com/scalesetpriority operator: Equal value: spot effect: NoSchedule affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 preference: matchExpressions: - key: kubernetes.azure.com/priority operator: In values: [\u0026#34;spot\u0026#34;] terminationGracePeriodSeconds: 30 containers: - name: worker image: myregistry.azurecr.io/batch-worker:1.4.0 resources: requests: cpu: \u0026#34;500m\u0026#34; memory: \u0026#34;512Mi\u0026#34; limits: memory: \u0026#34;512Mi\u0026#34; The key difference is whether you tolerate Spot or require it. Using a strict nodeSelector for Spot causes evicted pods to stay pending when Spot capacity disappears. In contrast, a preferred affinity allows the scheduler to fall back to regular nodes at a higher cost. Make sure to include this fallback expense in your budget model. It’s actual spending, not just a rounding error.\nDesign the application contract explicitly: multiple replicas, idempotent processing, checkpoints, a short shutdown path, correct SIGTERM handling, queue visibility timeouts, and bounded retries. Pod disruption budgets help with voluntary disruptions but do not protect against Spot’s 30-second eviction notice. Monitor eviction frequency, recovery time, pending-pod time, retry volume, and fallback-to-regular consumption; that data is the only proof that discounted compute is actually producing net savings.\nTune Scale-Down Without Breaking Availability Low utilization doesn’t necessarily justify removing a node. Factors such as local storage limitations, strict disruption budgets, singleton replicas, system pods, and limited capacity at other destinations in the cluster can prevent consolidation.\nCluster Autoscaler offers settings for scale-down delay, unneeded-time threshold, utilization threshold, and graceful termination. Using aggressive values can reduce idle Node minutes but may lead to more pod movements, cold starts, and operational disturbances. Before tuning, review the autoscaler’s events, history of unschedulable pods, disruption budgets, and node utilization trends. Always validate adjustments in a development or batch environment first, especially before applying changes to latency-sensitive production.\nAutomate FinOps with Kubernetes-Native Metadata Azure billing shows the infrastructure costs, while Kubernetes telemetry identifies the workload responsible. Combine dimensions such as subscription, cluster, node-pool, namespace, controller, and labels to create actionable allocation reports for engineers.\nImplement technical controls as standards rather than informal practices: mandate resource requests, assign ownership labels, track environment metadata, specify approved VM types, and define explicit Spot eligibility. Also, identify idle namespaces, abandoned load balancers, unattached disks, excessive log ingestion, and requests that significantly deviate from actual usage.\nReport the cost per request, job, tenant, or business transaction, not merely the cost per cluster. Utilize anomaly detection to link spending spikes with deployments, replica adjustments, node-pool expansions, or telemetry configuration modifications. Use reservations or savings plans to cover baseline measured usage, allowing autoscaling and Spot instances to handle the variable expenses.\nConclusion: Optimize the Feedback System Durable AKS savings are achieved through precise requests, compatible pod and node configurations, coordinated autoscalers, interruption-aware applications, and integrated cost telemetry into engineering choices. The aim isn’t to maximize utilization at all times but to maintain a controlled system that fulfills service levels with the least necessary paid capacity.\n","permalink":"https://wolkwacht.nl/posts/reducing-aks-costs-autoscaling-spot-nodes-rightsizing-and-finops-practices/","summary":"\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*2mXnRH0e5lM4mDSPdQmDDw.jpeg\"\u003e\u003c/p\u003e\n\u003ch3 id=\"introduction-cost-is-an-outcome-of-scheduling-decisions\"\u003eIntroduction: Cost Is an Outcome of Scheduling Decisions\u003c/h3\u003e\n\u003cp\u003eYour AKS bill is calculated before Azure sends an invoice. Kubernetes translates pod specifications into resource requirements, including resource requests, affinity rules, topology constraints, DaemonSet overhead, and scaling limits, all of which impact the efficiency of pod placement on nodes.\u003c/p\u003e\n\u003cp\u003eThis post\u0026rsquo;s main point is clear: durable optimization involves enhancing the entire chain, from workload demand to pod replicas, schedulable capacity, VM allocation, and cost attribution. Reducing node count without understanding this chain does not save money; it simply shifts costs into latency, evictions, throttling, and extended recovery periods.\u003c/p\u003e","title":"Reducing AKS Costs: Autoscaling, Spot Nodes, Rightsizing, and FinOps Practices"},{"content":"Running AI Workloads on Azure Kubernetes Service: From Platform Control to Model Serving Most teams begin their AI journey by using a managed model API. This approach works initially but can become problematic when costs become unpredictable at scale, data sovereignty or compliance issues prevent sending data to third-party endpoints, or the team needs a fine-tuned model or a custom inference runtime. At this stage, the focus shifts from simply consuming AI to actively operating it.\nAKS enables platform teams to handle AI workloads just like any other production workloads: they can schedule, secure, monitor, scale, and govern them using the same cloud-native operating model they are already familiar with.\nWhat Makes AI Workloads Different AI workloads put pressure on a platform in ways that typical stateless web services do not.\nGPU availability remains the primary constraint. GPUs are costly, regionally limited, and often have quotas. Poor scheduling leads to wasted spending or request starvation. AI workload container images are large, often multiple gigabytes for the runtime alone, excluding model weights. Model artifacts can reach tens of gigabytes, and loading them at startup causes cold-start latency not usually seen in standard API pods.\nInference traffic is often bursty. A batch job might use a GPU node intensively for hours, while a real-time inference endpoint may remain idle between requests. Distributed training requires coordination among nodes, adding overhead. Unlike web API inputs and outputs, AI workload inputs and outputs, such as prompts, embeddings, and completions, often contain sensitive data that requires controlled access and auditing.\nThese characteristics are not reasons to avoid Kubernetes; rather, they highlight why a platform with proper scheduling, autoscaling, security boundaries, and observability is essential.\nAKS Capabilities That Matter for AI The key advantage of AKS is not just its ability to run containers, but that it enables teams to manage high-value AI workloads with enterprise controls already integrated.\nGPU node pools and autoscaling:AKS provides dedicated GPU-optimized VM families (NC, ND, NV series) as node pools. The cluster autoscaler can scale these pools down to zero when idle and up as necessary, reducing costs by eliminating idle GPU capacity. Furthermore, Node Auto-Provisioning automatically chooses node SKUs based on the needs of pending pods.\nWorkload Identity replaces static credentials in pods by federating Kubernetes service accounts with Azure AD. This allows a model-serving pod to access artifacts from Azure Blob Storage or retrieve secrets from Key Vault without relying on long-lived secrets within the cluster.\nNetworking in Azure CNI with network policies assigns each pod a routable IP address, enabling precise traffic rules between namespaces and workloads. Ingress options include solutions like Traefik and the managed Application Gateway Ingress Controller, which support TLS termination and path-based routing to multiple model endpoints.\nObservabilitywith Azure Monitor, Managed Prometheus, and Grafana offers built-in metrics at both the cluster and workload levels. GPU utilization data from DCGM Exporter can be collected alongside typical pod metrics. Logs are sent to Log Analytics for querying and alerting.\nGovernance: Azure Policy for AKS imposes guardrails during admission, including mandatory resource limits, permitted image registries, and restrictions on privileged containers. Microsoft Defender for Containers enhances security by providing runtime threat detection and scanning for image vulnerabilities.\nModel Serving with KAITO on AKS KAITO (Kubernetes AI Toolchain Operator) is an open-source operator that makes it easier to self-host large language models on Kubernetes. The AKS AI toolchain operator add-on includes KAITO as a managed cluster component, eliminating the need for manual installation and maintenance.\nEnable the add-on on an existing cluster:\naz aks update \\ --name $CLUSTER_NAME \\ --resource-group $AZURE_RESOURCE_GROUP \\ --enable-ai-toolchain-operator \\ --enable-oidc-issuer KAITO introduces a Workspace custom resource. When you apply a workspace manifest, the operator automatically provisions the appropriate GPU node pool without pre-creating it, then pulls the model and exposes an inference endpoint compatible with OpenAI. Deploy Phi-4 Mini.\nkubectl apply -f https://raw.githubusercontent.com/kaito-project/kaito/refs/heads/main/examples/inference/kaito_workspace_phi_4_mini.yaml kubectl get workspace workspace-phi-4-mini -w Once the workspace reaches Ready, the model is available as an internal cluster service. Query it using a standard chat completions request:\nexport SERVICE_IP=$(kubectl get svc workspace-phi-4-mini \\ -o jsonpath=\u0026#39;{.spec.clusterIP}\u0026#39;) curl http://$SERVICE_IP/v1/chat/completions \\ -H \u0026#34;Content-Type: application/json\u0026#34; \\ -d \u0026#39;{ \u0026#34;model\u0026#34;: \u0026#34;phi-4-mini\u0026#34;, \u0026#34;messages\u0026#34;: [{\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: \u0026#34;What is AKS?\u0026#34;}] }\u0026#39; KAITO uses vLLM as its backend, offering teams continuous batching, streaming responses, and efficient memory handling without needing manual setup. The model operates fully within the organization’s AKS environment, simplifying compliance with data residency and regulatory standards.\nBefore deploying to production, check the supported KAITO versions and model presets in the AKS add-on documentation, as the managed add-on and the upstream KAITO project may not always offer the same features at the same time.\nSecurity and Governance Considerations Model endpoints are application endpoints that require equivalent security measures as other production services, and sometimes even more, due to the sensitive data they handle and the potential threats like prompt injection, data extraction, and model abuse.\nUse workload identity for all pod-to-Azure connections. Do not mount service account tokens to pods that don\u0026rsquo;t require them. Keep model access keys and API credentials in Azure Key Vault and expose them via the Secrets Store CSI Driver.\nImplement network controls by applying Kubernetes Network Policies to limit pod access to model-serving endpoints. Use private ingress if external access isn\u0026rsquo;t needed. Terminate TLS at the ingress layer and enforce request authentication.\nSpecify container image digests for supply chain security in deployment manifests. Limit permissible registries via Azure Policy to ensure only images from trusted sources run in the cluster. Use Defender for Containers to scan images for vulnerabilities before deploying them to production.\nIsolation and RBAC establish separate model-serving namespaces distinct from application namespaces. Implement the principle of least privilege with Kubernetes RBAC to ensure workloads access only the necessary resources. Enable audit logging on the control plane and direct logs to Log Analytics for retention and alerting.\nApplication-layer controls such as rate limiting, authentication, authorization, content safety filtering, and inference logging are not handled automatically by the platform. These need to be designed explicitly for each model endpoint, whether through an API gateway, a sidecar, or application code.\nOperational Realities and the Decision Guide GPU quota is limited and varies by region. Request it early, choose VM SKUs that match your model\u0026rsquo;s needs, and set resource limits on pods to prevent any workload from monopolizing a node. Cold starts for models, including image pulls and weight loading, can take several minutes on larger models. Use sufficient initialDelaySeconds in health probes to accommodate this. Since GPU autoscaling is slower and costlier than scaling web pods, plan capacity carefully and maintain monitoring dashboards and alerts before traffic increases.\nProduction fundamentals for AI workloads mirror those for other types: using PodDisruptionBudgets to ensure safe upgrades, topology spread constraints to maintain zone availability, separate node pools for various workload classes, and specific maintenance windows for node image updates.\nConclusion AKS provides teams with a comprehensive platform for AI workloads, going beyond merely hosting containers. The most effective approach is focused and incremental: select a single workload, verify quotas, costs, latency, and security measures, automate deployment, and gradually expand once the operating model\u0026rsquo;s effectiveness is confirmed.\nThe true benefit of AKS for AI isn\u0026rsquo;t just running a model; it\u0026rsquo;s enabling teams to execute AI as a managed, observable, and scalable platform workload.\n","permalink":"https://wolkwacht.nl/posts/running-ai-workloads-on-azure-kubernetes-service-from-platform-control-to-model-serving/","summary":"\u003ch2 id=\"running-ai-workloads-on-azure-kubernetes-service-from-platform-control-to-modelserving\"\u003e\u003cstrong\u003eRunning AI Workloads on Azure Kubernetes Service: From Platform Control to Model Serving\u003c/strong\u003e\u003c/h2\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*lIZJHiR5c6hjfT5SbnQ8Dw.jpeg\"\u003e\u003c/p\u003e\n\u003cp\u003eMost teams begin their AI journey by using a managed model API. This approach works initially but can become problematic when costs become unpredictable at scale, data sovereignty or compliance issues prevent sending data to third-party endpoints, or the team needs a fine-tuned model or a custom inference runtime. At this stage, the focus shifts from simply \u003cem\u003econsuming\u003c/em\u003e AI to actively \u003cem\u003eoperating\u003c/em\u003e it.\u003c/p\u003e","title":"Running AI Workloads on Azure Kubernetes Service: From Platform Control to Model Serving"},{"content":"Optimizing Azure Container Apps Costs: Scaling to Zero, Workload Profiles, and FinOps Azure Container Apps enables teams to deploy APIs, background processors, scheduled jobs, and event-driven services without directly managing Kubernetes infrastructure. However, this convenience does not automatically ensure cost optimization.\nContainer Apps costs depend on application behavior: the number of replicas, their active duration, the workload profile hosting them, and the telemetry or network traffic they produce. An efficient setup for an unpredictable queue worker might be wasteful for a constantly busy API, and vice versa.\nThree practices make the largest difference:\nScale workloads to zero when idle. Match each workload with the appropriate profile and resources. Implement FinOps practices to make cost optimization an ongoing engineering activity. This article describes how these practices integrate without compromising reliability or user experience.\nKey takeaway: Optimizing costs for Azure Container Apps begins with understanding workload behavior rather than relying on a single platform setting.\nUnderstand the Azure Container Apps Cost Model The default Azure Container Apps environment offers both Consumption and Dedicated plans via workload profiles. Multiple applications within the same environment can select different profiles, enabling teams to assign workloads based on their technical needs and cost considerations.\nThe Consumption plan meters compute resources for each replica based on allocated vCPU-seconds and GiB-seconds. External HTTP requests are also billed accordingly. Replicas that scale down to zero incur no resource charges. A replica kept at its minimum configuration might receive a lower idle rate, but only under certain conditions. When an app scales above its minimum, the extra replicas are charged at the active rate.\nThe Dedicated plan charges based on the instances assigned to a workload profile, not per application. Multiple apps can share these instances, which can be cost-effective for continuous workloads. However, this is only true if the capacity is used efficiently, as underutilized instances still count as allocated capacity.\nLog Analytics ingestion, outbound data transfer, private networking, databases, message brokers, and registries can also materially affect the solution’s total cost. Compare architectures using the whole service, not only container compute.\nKey takeaway: Consumption rewards elasticity, whereas Dedicated rewards high and predictable utilization. The total cost also encompasses the services associated with the container.\nScale to Zero Where It Makes Sense Azure Container Apps employs Kubernetes Event-driven Autoscaling (KEDA) to manage replicas, adjusting based on HTTP or TCP concurrency and custom signals like CPU, memory, queue depth, Azure Service Bus, Azure Event Hubs, Kafka, and Redis.\nSetting minReplicas to 0 allows an app revision to have no running replicas when its scaling trigger is inactive. For a Consumption workload, this removes resource consumption charges during the zero-replica period.\nThe following excerpt configures an HTTP application to scale from zero to a maximum of ten replicas:\nproperties: template: scale: minReplicas: 0 maxReplicas: 10 rules: - name: http-demand http: metadata: concurrentRequests: \u0026#34;50\u0026#34; For event-driven workloads, using a queue-based KEDA scaler is more common. The example below shows how to scale a worker according to the depth of an Azure Service Bus queue:\n# ARM/Bicep: Microsoft.App/containerApps properties.template.scale properties: template: scale: minReplicas: 0 maxReplicas: 20 rules: - name: queue-depth custom: type: azure-servicebus metadata: queueName: orders messageCount: \u0026#34;20\u0026#34; auth: - secretRef: sb-connection triggerParameter: connection These values serve as initial guidelines. The appropriate concurrency target should be determined based on response time, resource consumption, dependencies, and the traffic capacity of a single replica. Confirm this through load testing and monitoring in production.\nScaling to zero is usually a strong fit for:\nQueue workers that only run when messages arrive\nScheduled or manually triggered processing\nDevelopment and test services used during limited hours\nInternal tools with intermittent demand\nAPIs where occasional startup latency is acceptable\nThe tradeoff involves cold-start latency, as the platform needs to create a replica and initialize the container. Using large images and slow application setups can make this delay more noticeable.\nAvoid scaling down to zero unless necessary. For customer-facing APIs with strict latency goals, workloads that need persistent connections, or services that require instant responses, maintaining at least one replica is advisable. You can also manage costs effectively by adjusting replica sizes and setting a practical maximum.\nAlso differentiate long-running applications from finite tasks. Azure Container Apps jobs are typically better suited to scheduled or event-triggered tasks, as their processes cease consuming compute resources once the work is complete.\nKey takeaway: Use zero as the minimum only if the workload can tolerate a startup delay and has a dependable trigger to reactivate it.\nChoose the Right Workload Profile A workload profile specifies the compute resources, memory, isolation model, scaling behavior, and billing structure for an application.\nUse Consumption for workloads with significant idle times or sharp demand changes. Use Dedicated when consistent utilization can occupy a large portion of the chosen instances, when larger resource sizes are needed, or when a specific hardware profile is required.\nDon\u0026rsquo;t assume an always-on workload should automatically use Dedicated. First, measure CPU, memory, replica count, and growth. Dedicated becomes advantageous only when applications sharing a profile regularly utilize its capacity.\nSeparating workloads can enhance performance and make cost tracking clearer. For instance, a latency-sensitive API might operate under one profile, while bursty workers utilize Consumption. This prevents background spikes from skewing the API’s capacity planning and allows each part to have its own scaling policy.\nMicrosoft also details a Flexible workload profile in preview. This option uses per-replica billing with single-tenant compute, includes a dedicated management fee, and does not support scaling to zero. Check its current availability before adoption.\nKey takeaway: Apply bursty demand to elastic capacity and steady demand to shared dedicated capacity only when measurements justify it.\nA Practical Optimization Scenario Think of an order-processing system consisting of three parts:\nA public API receives customer orders.\nA worker processes messages from a queue.\nA scheduled job generates a daily reconciliation report.\nThe API experiences traffic throughout the day and has a strict response-time goal. Start with a Consumption profile and at least one minimum replica, then adjust the HTTP concurrency rule based on load-test results. If traffic remains consistently high, compare its measured Consumption cost with that of a Dedicated profile used by other steady services.\nThe worker operates based on events and remains inactive for extended periods. It is well-suited for scale-to-zero solutions. Set up a queue rule compatible with KEDA, define a maximum number of replicas to prevent excessive scaling, and monitor both queue age and length to determine if scaling is adequate. Limiting the maximum prevents overwhelming downstream databases and APIs with sudden spikes.\nThe reconciliation task should be scheduled to run as a Container Apps job rather than remaining active all day. Allocate sufficient CPU and memory for efficient completion, but assess whether increasing resources truly shortens runtime enough to lower overall costs.\nBefore changing anything, capture a baseline:\nDaily vCPU and memory consumption Replica-hours by component Request or message volume Queue latency and processing duration Application latency and error rate Log ingestion and network charges Cost per order processed After the change, compare a representative business cycle. Savings only count if service-level objectives remain healthy.\nKey takeaway: Optimize each component separately, then assess the outcomes based on cloud expenses and service performance.\nMake FinOps Part of Operations Autoscaling minimizes runtime waste, while FinOps helps prevent it from happening again.\nStart with ownership. Use tags and names that identify the application, environment, team, cost center, and business service. Align resource groups and subscriptions with the organization’s reporting scopes.\nCreate Azure Cost Management budgets with actual-cost and forecasted-cost alerts. Budgets notify teams; they do not stop resources. Cost data is delayed, so budgets are not real-time controls.\nTrack unit economics rather than focusing only on the monthly bill. Useful measures include:\nCost per API request Cost per order or business transaction Cost per thousand queue messages Cost per active customer Non-production cost as a percentage of total cost Logging cost per application Review these metrics together with latency, throughput, errors, and availability. A rising bill can be healthy if transaction volume increases at a faster pace.\nSet up regular reviews, like monthly for production, and more often during significant growth or architecture shifts. Check for apps pinned to unnecessary minimum replicas, dedicated profiles with low usage, abandoned revisions or environments, oversized CPU and memory allocations, and verbose logs that add little operational value.\nKey takeaway: FinOps links scaling decisions with ownership, budgets, business volume, and service results.\nCommon Cost Mistakes Keeping minimum replicas “just in case.” A warm replica should match a latency or availability requirement; otherwise, it constitutes permanent baseline consumption.\nUsing a dedicated resource for irregular demand. Dedicated capacity can be economical; however, idle profile instances weaken the model. Combine compatible steady workloads and keep an eye on utilization.\nIgnoring observability costs. High-volume informational logs, duplicated telemetry, and extended retention periods can lead to unexpected costs. Establish appropriate log levels and retention policies tailored for each environment.\nScaling too aggressively. Setting low trigger thresholds and high maximums can lead to replica churn or overload dependencies. Adjust scaling based on overall end-to-end performance instead of relying solely on individual container metrics.\nTreating every environment like production. Development, test, and demonstration environments frequently require reduced limits, scale-to-zero configurations, shorter log retention periods, or scheduled shutdown procedures.\nKey takeaway: Most waste comes from defaults that were never revisited after the workload was better understood.\nOptimization Checklist Conclusion Azure Container Apps provides various methods to adjust compute costs based on actual demand, working most effectively when integrated as a system.\nScaling to zero halts consumption during true idle times. Workload profiles allow teams to distinguish between bursty services and consistent or specialized workloads. FinOps practices ensure these technical choices are monitored, assigned responsibility, and revisited as the application evolves.\nStart with a low-risk component, such as an intermittent worker or a non-production API. Record its current costs and performance, modify its scaling settings, and assess the outcome. This small feedback loop offers more value than a wide-ranging optimization based on assumptions.\nUseful links Billing in Azure Container Apps Set scaling rules in Azure Container Apps Workload profiles in Azure Container Apps Azure Container Apps jobs Create and manage Azure Cost Management budgets Identify anomalies and unexpected cost changes ","permalink":"https://wolkwacht.nl/posts/optimizing-azure-container-apps-costs-scaling-to-zero-workload-profiles-and-finops/","summary":"\u003ch2 id=\"optimizing-azure-container-apps-costs-scaling-to-zero-workload-profiles-andfinops\"\u003e\u003cstrong\u003eOptimizing Azure Container Apps Costs: Scaling to Zero, Workload Profiles, and FinOps\u003c/strong\u003e\u003c/h2\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*dGnS8EvsEJGb-z5FUGpasw.png\"\u003e\u003c/p\u003e\n\u003cp\u003eAzure Container Apps enables teams to deploy APIs, background processors, scheduled jobs, and event-driven services without directly managing Kubernetes infrastructure. However, this convenience does not automatically ensure cost optimization.\u003c/p\u003e\n\u003cp\u003eContainer Apps costs depend on application behavior: the number of replicas, their active duration, the workload profile hosting them, and the telemetry or network traffic they produce. An efficient setup for an unpredictable queue worker might be wasteful for a constantly busy API, and vice versa.\u003c/p\u003e","title":"Optimizing Azure Container Apps Costs: Scaling to Zero, Workload Profiles, and FinOps"},{"content":"\nPart 1 introduced AppNet, while Part 2 demonstrated its use in zero-trust and multi-cluster patterns. This concluding section explores a shared AI gateway, the operational signals derived from the mesh, and the architectural tradeoffs that influence whether AppNet should be integrated into your platform.\nSeries navigation Part 1: What AppNet is and how to get started Part 2: Zero-trust and multi-cluster patterns Part 3: AI gateway, observability and production fit Example three: a shared AI gateway that rate-limits by application AI Gateway with Per-Application Token Rate Limiting]\nThe third example has made AppNet interesting to platform teams that previously had no interest in service mesh, and Microsoft characterizes it as a first-party pattern. The scenario involves a platform team deploying a single agentgateway within the cluster that interfaces with Azure OpenAI and several other model providers. Numerous internal applications access this gateway from various namespaces. The finance team requires per-application token budgets to prevent a runaway LangChain agent from exhausting the entire OpenAI budget in a weekend.\nWithout AppNet, this process becomes cumbersome. The typical solution involves issuing an API key for each application, with the gateway imposing rate limits on those keys. Managing key rotation, storing keys securely in Key Vault, injecting them into the correct namespace, and revoking access during incidents all add complexity. This approach only succeeds if every application consistently uses the keys properly.\nAppNet modifies the model by leveraging ztunnel-enforced workload identities in all enrolled namespace pods. This allows agentgateway to verify caller identities using SPIFFE IDs from the mTLS handshake, eliminating the need for API keys for identity verification. Rate limiting is also applied on this basis.\nThe configuration consists of three parts. The first is an AgentgatewayPolicy linked to the Gateway that manages model API requests by enforcing rate limits and using the source namespace as the key.\napiVersion: agentgateway.io/v1 kind: AgentgatewayPolicy metadata: name: openai-token-budget namespace: ai-platform spec: targetRefs: - group: gateway.networking.k8s.io kind: Gateway name: agentgateway rateLimit: descriptors: - entries: - key: source_namespace value: \u0026#34;{{ source.identity.namespace }}\u0026#34; - key: model value: \u0026#34;{{ request.headers[\u0026#39;x-model\u0026#39;] }}\u0026#34; service: ratelimit.ai-platform.svc.cluster.local The second part is the rate-limit server itself, configured as a ConfigMap that specifies the tokens-per-minute budget for each application. For example, a team with the namespace fraud might have a larger quota than a team in the marketing-experiments namespace.\napiVersion: v1 kind: ConfigMap metadata: name: ratelimit-config namespace: ai-platform data: config.yaml: | domain: openai_tokens descriptors: - key: source_namespace value: fraud rate_limit: unit: minute requests_per_unit: 100000 - key: source_namespace value: marketing-experiments rate_limit: unit: minute requests_per_unit: 10000 - key: source_namespace rate_limit: unit: minute requests_per_unit: 1000 The third part is the AuthorizationPolicy on the gateway, which blocks requests from any entity not explicitly onboarded by the platform team. This prevents overlooked tenants from quietly consuming tokens. According to Istio security guidance, it is better to use explicit SPIFFE principals rather than namespace-only matching, because principals link rules to specific service accounts, not just to entities within a namespace (see the source-matching reference). Including both fields creates a layered defense that fails closed if either component is altered.\napiVersion: security.istio.io/v1 kind: AuthorizationPolicy metadata: name: agentgateway-allowed-tenants namespace: ai-platform spec: targetRefs: - kind: Gateway group: gateway.networking.k8s.io name: agentgateway action: ALLOW rules: - from: - source: principals: - cluster.local/ns/fraud/sa/fraud-agent - cluster.local/ns/checkout/sa/checkout-agent - cluster.local/ns/marketing-experiments/sa/marketing-agent - cluster.local/ns/reports/sa/reports-agent namespaces: [\u0026#34;fraud\u0026#34;, \u0026#34;checkout\u0026#34;, \u0026#34;marketing-experiments\u0026#34;, \u0026#34;reports\u0026#34;] As a result, finance benefits from predictable per-application spending limits enforced at the network level, while applications benefit from a seamless “just call the gateway” experience. Additionally, the platform team no longer needs to manage API keys. When a new team joins, onboarding is simplified to a pull request that adds their service account principal and namespace to the AuthorizationPolicy and a budget entry to the ConfigMap.\nObservability: metrics and logs that are already there One of the practical yet often overlooked features of AppNet is its ability to provide valuable signals immediately upon enrolling in a namespace (metrics reference, logs reference). Ztunnel offers per-connection telemetry showing mTLS success and failure, while the waypoint provides standard Istio access logs and Prometheus metrics. When you connect AppNet to an AKS cluster with Azure Managed Prometheus and Azure Managed Grafana enabled, these metrics are conveniently available in the same workspace your cluster uses. This means you don’t need extra Prometheus instances or additional scrape configurations, making your monitoring setup simpler and more efficient.\nA common alert signal is the rate of L4 mTLS failures per source and destination, indicating when a workload attempts to connect to an unauthorized namespace. Another useful signal is the waypoint’s 4xx rate broken down by target service, which reveals when an application starts sending unexpected traffic, such as a new endpoint or a changed method that your AuthorizationPolicy now blocks. Both metrics usually increase during deployments, and displaying them alongside application metrics in Grafana helps confirm whether everything is functioning correctly, even if the deploy appears successful.\nLogs such as control plane events and waypoint access logs can be sent to Azure Monitor Logs via the standard AKS log collection. CloudOps teams already using Container Insights won\u0026rsquo;t need to make any changes; the logs will appear in the same Log Analytics workspace with only minor label differences.\nWhere Azure Kubernetes Application Network is a good fit The honest sweet spot is a platform team that manages multiple AKS clusters, prefers default mTLS on east‑west traffic, and has Layer 7 policy needs that go beyond what network policies can express. If your environment hosts workloads from different tenants within the same cluster or has compliance needs requiring identity‑based authorization between services, then AppNet is perfectly suited for your needs.\nMulti-cluster topologies work especially well. If you\u0026rsquo;re operating active-active setups across regions or dividing workloads across clusters to minimize blast radius, AppNet eliminates the need for the federated Istio project, which has historically drained resources from many platform teams. Its shared trust domain and managed control plane eliminate the two common problem areas in federated meshes.\nAI platforms offering a shared gateway for multiple tenants are a natural fit, as demonstrated by the third example. Treating workload identity as a first‑class concept and enforcing it at the network layer eliminates a category of API‑key rotation issues that are challenging to resolve cleanly through other means.\nRegulated workloads see disproportionate benefits. Automated mTLS, identity-based policies, and managed upgrades simplify compliance documentation into a few Kubernetes resources that can be easily presented to an auditor. When asked, “Show me how you prevent service A from talking to service B”, the answer is just two lines.\nWhere Azure Kubernetes Application Network is the wrong tool AppNet functions as a service network. It is not an ingress controller, WAF, or a substitute for Application Gateway for Containers or Azure Front Door. If your challenge is to expose a public API with authentication, WAF rules, and TLS termination, AppNet alone is not sufficient; it provides the underlying layer that secures east‑west traffic within the cluster after the request arrives.\nIf you\u0026rsquo;re managing a single AKS cluster with just a few workloads and no multi-tenant needs, AppNet might be unnecessary. For most small teams, Kubernetes NetworkPolicy combined with a suitable ingress covers their main requirements. Incorporating a managed mesh adds complex concepts that your team will need to learn and stay updated on. While the data plane costs are minimal and the control plane is handled by Microsoft, explaining SPIFFE identities, waypoints, ambient labels, and Gateway API semantics to a small on-call team can be a significant cognitive challenge.\nLatency-critical workloads with ultra-low overhead, where every hundred microseconds matters, deserve careful benchmarking. Ambient mode is simpler than sidecar mode, but crossing a waypoint still introduces an extra hop, and HBONE tunnelling on ztunnel adds some overhead per connection. While this is negligible for most services, it may be significant for market-data fanouts or high-frequency trading bridges.\nIf you already have a mature Istio or Linkerd deployment and are satisfied with it, consider the migration to AppNet carefully. You should compare the migration cost with the upgrade and operational expenses needed to maintain your current setup. While the migration process is smooth, ambient mode coexists with sidecars during the transition, and it is not without costs.\nCurrently, AppNet assumes your clusters are on AKS. If you have a hybrid environment with some clusters in other clouds or on-premises OpenShift, AppNet won\u0026rsquo;t extend to cover them. In those cases, using a cross-cloud mesh based on open-source Istio or Cilium’s cluster mesh remains the suitable solution.\nSecurity and operational considerations Security architects will ask three key questions (AppNet security overview). First, how is the trust domain established? In AppNet, it is linked to your AppNet resource and uses certificates issued by the managed control plane, which are automatically rotated as SPIFFE‑compliant workload certificates. The root material is not exposed to you, which security teams appreciate (no custom CA to manage), but they must understand that this means trusting Microsoft’s key management and the associated audit trail.\nSecond, what is the blast radius? Ztunnel operates as a DaemonSet, so if a node is compromised, only the pods on that node are affected. AppNet does not extend this impact further. A waypoint, which is a namespaced Deployment, has a scope limited to the services that use it, smaller than a cluster‑wide Istio ingress.\nThird, what happens during a break‑glass scenario? If the control plane becomes unavailable, ztunnel continues enforcing the last known configuration and does not fail open. This behavior is expected and should be recorded in runbooks.\nThe cost model comprises three components: the AppNet resource charged per region, the data plane footprint (with ztunnel using a few hundred megabytes of memory per node and waypoints scaling with traffic), and the observability expenses you already incur for your workloads. Operationally, the biggest change is that you no longer need to run Istiod. Platform teams that previously spent a week each quarter on mesh upgrades often find that time reclaimed, while the remaining tasks mainly involve policy work, creating and reviewing AuthorizationPolicy, PeerAuthentication, Gateway, VirtualService, and Gateway API routing resources as part of regular PR reviews. This work can now be shared with your application teams, a welcome departure from the typical pattern in which only the platform team understands how the mesh operates.\nA note on migrating from sidecar Istio\nTeams already using Sidecar Istio can gradually switch to AppNet. The recommended approach is to deploy AppNet alongside the current Sidecar setup, enroll a non-critical namespace in ambient mode, verify that policies translate smoothly, and then migrate additional namespaces one by one. Ambient mode and sidecars can operate simultaneously; a pod with a sidecar and an ambient-enabled pod can communicate via mTLS if they share the same trust domain, and the waypoint uses the same protocols as the sidecar. The main challenge in migration is usually updating VirtualService resources that contain sidecar-specific routing tricks. These become Gateway API HTTPRoute resources on the waypoint, and while the translation process is straightforward, it can be tedious at times.\nOnce all namespaces have been migrated, the sidecar Istiod can be removed, returning the cluster from two coexisting control planes to a single managed control plane. Usually, this is when the platform team realizes the upgrade process has become easier.\nClosing thoughts Azure Kubernetes Application Network is not a revolution. It is a thoughtful packaging of ambient‑mode Istio into the Azure resource model, with the parts nobody enjoys operating, such as the control plane, the root CA, the upgrade cadence, and the multi‑cluster federation, lifted out of your responsibility. The pieces that are fundamentally your job, because they encode your security posture and your application topology, remain your responsibility, expressed as Kubernetes resources your team already reads every day.\nFor platform teams managing multiple AKS clusters and seeking an alternative to running unfettered on NetworkPolicy or incurring the sidecar costs, AppNet stands out in the Azure portfolio as a solution designed specifically for this purpose. For single-cluster setups, it might be more than necessary, and for hybrid or multi-cloud environments, it may not yet be the ideal choice. If you’re already considering adopting ambient-mode Istio, AppNet offers a managed approach. If service mesh adoption has been delayed due to operational concerns outweighing its benefits, AppNet deserves a reconsideration.\nBegin with a single cluster, one namespace, and an authorization policy you\u0026rsquo;ve always wanted to enforce but lacked the mesh support. This initial step is smaller than any previous managed mesh on Azure, making it the ideal way to determine whether AppNet suits your platform.\nThe three parts collectively provide the practical AppNet overview: grasp the managed ambient architecture, implement it in real platform patterns, and determine where it integrates into your AKS estate.\nSources Validated against the following sources in May 2026.\nMicrosoft Learn — Overview of Azure Kubernetes Application Network for AKS (Preview) Microsoft Learn — Get Started Microsoft Learn — Architecture Microsoft Learn — Traffic Management Use Cases Microsoft Learn — Supported Versions AKS Engineering Blog — Control AI spend with per-application token rate limiting using Application Network and agentgateway Istio — Configure waypoint proxies Istio — Add workloads to the mesh Istio — ztunnel architecture referenc ","permalink":"https://wolkwacht.nl/posts/azure-kubernetes-application-network-part-3-ai-gateway-observability-and-production-fit/","summary":"\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*nViN0g9SAYOnbvriKedPww.png\"\u003e\u003c/p\u003e\n\u003cp\u003ePart 1 introduced AppNet, while Part 2 demonstrated its use in zero-trust and multi-cluster patterns. This concluding section explores a shared AI gateway, the operational signals derived from the mesh, and the architectural tradeoffs that influence whether AppNet should be integrated into your platform.\u003c/p\u003e\n\u003ch2 id=\"series-navigation\"\u003eSeries navigation\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"https://medium.com/@jurgenallewijn/azure-kubernetes-application-network-part-1-what-appnet-is-and-how-to-get-started-ceb39df26624\"\u003ePart 1: What AppNet is and how to get started\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://medium.com/@jurgenallewijn/azure-kubernetes-application-network-part-2-zero-trust-and-multi-cluster-patterns-014a8384fc8d\"\u003ePart 2: Zero-trust and multi-cluster patterns\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003ePart 3: AI gateway, observability and production fit\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"example-three-a-shared-ai-gateway-that-rate-limits-by-application\"\u003eExample three: a shared AI gateway that rate-limits by application\u003c/h2\u003e\n\u003cp\u003e\u003cimg alt=\"AI Gateway with Per-Application Token Rate Limiting\" loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*a_PaJXKV8Hy4__OadD6Apw.jpeg\"\u003e\n\u003cem\u003eAI Gateway with Per-Application Token Rate Limiting]\u003c/em\u003e\u003c/p\u003e","title":"Azure Kubernetes Application Network, Part 3: AI Gateway, Observability and Production Fit"},{"content":"\nIn Part 1, we discussed the AppNet architecture, setup process, and waypoint model. This second part shifts focus from mechanics to platform patterns: it covers enforcing service-to-service authorization for a payments platform, and then using AppNet across AKS clusters for active-active regional architectures.\nSeries navigation Part 1: What AppNet is and how to get started Part 2: Zero-trust and multi-cluster patterns Part 3: AI gateway, observability and production fit Example one: Zero-trust authorization for a payments platform Zero-Trust Authorization — Payments Namespace\nImagine a team managing a payments platform on AKS. Their main services include checkout, ledger, fraud detection, notifications, and reports. They follow a straightforward rule that their compliance auditor emphasizes: only checkout and fraud services are allowed to call the ledger, and only via POST requests to /v1/transactions. All other calls must be blocked at the network level, not within the application code, and this restriction should be verifiable through tests without running the application.\nBefore AppNet, this team faced limited options: running a full Istio with sidecars, which they found too resource-intensive for about a hundred pods, or using Kubernetes NetworkPolicy, which allows restrictions based on pod labels and ports but cannot handle HTTP methods, paths, or workload identities from service accounts. Neither approach was ideal.\nWith AppNet enrolled in the payments namespace and a waypoint deployed, the authoring process uses only the Istio API. First, they set mTLS to STRICT mode to block any plaintext traffic to the ledger.\napiVersion: security.istio.io/v1 kind: PeerAuthentication metadata: name: payments-mtls-strict namespace: payments spec: mtls: mode: STRICT They then set a default‑deny baseline. Istio 1.25 added support for binding an AuthorizationPolicy to the istio-waypoint GatewayClass from the root namespace (such as istio-system), allowing you to establish a default authorization stance for all waypoints in the cluster. It’s crucial that this policy is an empty ALLOW rather than a blanket DENY: since Istio evaluates DENY policies before ALLOW policies, a DENY rule that covers everything cannot be overridden by a subsequent ALLOW. An empty ALLOW policy maintains the default-deny stance while permitting more specific ALLOW policies to grant access.\napiVersion: security.istio.io/v1 kind: AuthorizationPolicy metadata: name: deny-all-waypoints namespace: istio-system spec: targetRefs: - kind: GatewayClass group: gateway.networking.k8s.io name: istio-waypoint action: ALLOW Then, they specify the allow rule. The policy targets the ledger service via targetRefs, the ambient-mode method for attaching a policy to a waypoint. It permits traffic only from the two specified service accounts on the designated method and path.\napiVersion: security.istio.io/v1 kind: AuthorizationPolicy metadata: name: ledger-allow-checkout-and-fraud namespace: payments spec: targetRefs: - kind: Service group: \u0026#34;\u0026#34; name: ledger action: ALLOW rules: - from: - source: principals: - cluster.local/ns/payments/sa/checkout - cluster.local/ns/payments/sa/fraud to: - operation: methods: [\u0026#34;POST\u0026#34;] paths: [\u0026#34;/v1/transactions\u0026#34;] Since the principal is an SPIFFE identity derived from the Kubernetes service account and enforced by ztunnel and the waypoint, there is no bearer token to steal, no header to forge, and no way for another pod, such as a debugging sidecar or a misbehaving cron job, to communicate with Ledger. If the auditor requests proof, the team can run a pod with a different service account and verify that the call is rejected at the waypoint. Note that the kubectl run \u0026ndash;serviceaccount shortcut was removed in Kubernetes 1.24, so the current way to specify a service account for a one-time test pod is to use --overrides.\n# Pre-create the test service account (it lives only for the test) kubectl -n payments create serviceaccount probe kubectl -n payments run curl-test \\ --image=curlimages/curl:8.7.1 \\ --rm -it --restart=Never \\ --overrides=\u0026#39;{\u0026#34;spec\u0026#34;:{\u0026#34;serviceAccountName\u0026#34;:\u0026#34;probe\u0026#34;}}\u0026#39; \\ -- curl -sS -o /dev/null -w \u0026#34;%{http_code}\\n\u0026#34; \\ http://ledger.payments.svc.cluster.local/v1/transactions \\ -X POST -d \u0026#39;{}\u0026#39; The expected response is 403, originating from the waypoint rather than the ledger. The use of the plain http:// URL is intentional: the local ztunnel transparently upgrades outbound connections to HBONE-over-mTLS before they leave the node. As a result, the application code never handles TLS, and the data in transit remains unencrypted in the ambient data plane. This encapsulates the zero-trust approach in a single line: workload identity, transport encryption, and policy enforcement are all managed outside the application.\nA practical point often overlooked is that without the sidecar, on-call engineers don\u0026rsquo;t need to match application logs with sidecar logs within the same pod. Since all Layer 7 policy enforcement occurs on the waypoint, a normal Deployment with consistent logs and metrics, identifying issues at 3 am becomes easier. The scope of potential problematic pods is reduced and more straightforward to observe.\nExample two: multi‑cluster active‑active for a retailer Multi-Cluster Active-Active Topology\nA retailer manages two AKS clusters, one in East US 2 and another in Central US, with an Azure Front Door that routes traffic based on latency. They aim for an active‑active setup where the catalog service in either region can communicate with the inventory service in either region, ensuring all traffic is end‑to‑end mTLS‑encrypted regardless of the cluster involved. Additionally, they want the system to handle regional failures transparently. They prefer not to operate their own federated Istio, as their previous attempt two years ago led to a quarter of troubleshooting certificate rotation issues across clusters.\nIn an Azure regional-pair setup, East US 2 and Central US make a clearer example pair than combining a US primary region with a European secondary region. However, you can select a different region if your needs for latency, compliance, or user location suggest otherwise.\nAppNet natively supports multiple clusters, as detailed here. Both clusters are part of a single AppNet resource, and the managed control plane considers them a unified service network within a shared trust domain, with mTLS ensuring end-to-end security across cluster boundaries.\n# Attach the East US 2 cluster az appnet member join \\ --resource-group \u0026#34;$APPNET_RG\u0026#34; \\ --appnet-name \u0026#34;$APPNET_NAME\u0026#34; \\ --member-name \u0026#34;aks-prod-eus2-01\u0026#34; \\ --member-resource-id \u0026#34;/subscriptions/$SUBSCRIPTION_ID/resourceGroups/rg-aks-prod-eus2/providers/Microsoft.ContainerService/managedClusters/aks-prod-eus2-01\u0026#34; \\ --upgrade-mode FullyManaged # Attach the Central US cluster az appnet member join \\ --resource-group \u0026#34;$APPNET_RG\u0026#34; \\ --appnet-name \u0026#34;$APPNET_NAME\u0026#34; \\ --member-name \u0026#34;aks-prod-cus-01\u0026#34; \\ --member-resource-id \u0026#34;/subscriptions/$SUBSCRIPTION_ID/resourceGroups/rg-aks-prod-cus/providers/Microsoft.ContainerService/managedClusters/aks-prod-cus-01\u0026#34; \\ --upgrade-mode FullyManaged Both clusters require network reachability to AppNet’s east–west gateways, involving VNet peering, Azure Virtual WAN, route tables, and NSGs; these are concerns beyond what the service mesh can handle. AppNet does not eliminate the need for proper underlying networking; instead, it adds features like transparent mTLS, identity propagation, and service routing across cluster boundaries.\nSetting up multi-cluster calls requires minimal effort from the application team. The key step is making the participating services global. They deploy inventory in both clusters with identical Service names in the same namespace, label the application Service and waypoint Service with istio.io/global=”true”, and the AppNet control plane connects them. This allows a catalog pod request in East US 2 to inventory.catalog.svc.cluster.local to access endpoints across all AppNet clusters, while still using the same in-cluster DNS name.\nThe team designates the service and waypoint as global by applying the AppNet labels specified in the traffic management guide.\nkubectl -n catalog label service inventory \\ istio.io/global=\u0026#34;true\u0026#34; --overwrite kubectl -n catalog label service catalog-waypoint \\ istio.io/global=\u0026#34;true\u0026#34; --overwrite For explicit traffic shifting, the documented AppNet pattern involves creating waypoint-enrolled global Services that represent local and remote backend pools. An Istio VirtualService is then used to allocate traffic between them through weighting. For an active-active production setup, I recommend beginning with local-first routing at the application or service-discovery layer, followed by weighted shifting for controlled failovers, evacuation tests, and regional maintenance windows.\napiVersion: v1 kind: Service metadata: name: inventory-local namespace: catalog labels: app: inventory service: inventory-local istio.io/global: \u0026#34;true\u0026#34; istio.io/use-waypoint: catalog-waypoint spec: ports: - name: http port: 8080 targetPort: 8080 selector: app: inventory region: local --- apiVersion: v1 kind: Service metadata: name: inventory-remote namespace: catalog labels: app: inventory service: inventory-remote istio.io/global: \u0026#34;true\u0026#34; istio.io/use-waypoint: catalog-waypoint spec: ports: - name: http port: 8080 targetPort: 8080 selector: app: inventory region: remote --- apiVersion: networking.istio.io/v1 kind: VirtualService metadata: name: inventory namespace: catalog spec: hosts: - inventory http: - route: - destination: host: inventory-local.catalog.svc.cluster.local port: number: 8080 weight: 100 - destination: host: inventory-remote.catalog.svc.cluster.local port: number: 8080 weight: 0 Under normal conditions, weights keep traffic localized. In a regional drain or incident, the platform team can redirect some or all traffic to inventory-remote without modifying the calling code. With PeerAuthentication set to STRICT at the namespace level and an AuthorizationPolicy that permits only the catalog service account to access inventory, the topology can shift between regions without requiring certificate rotation. Because the mTLS handshake relies on the same trust domain across both clusters, a catalog call from East US 2 to Central US\u0026rsquo;s inventory appears, in identity verification, identical to a local call.\nThe cost discipline enabled by this is significant. Previously, the team operated a dedicated ingress for each cluster to handle inter-cluster calls with mTLS. Now, with AppNet, these ingress points are eliminated, reducing the public surface area. Additionally, east-west traffic costs are now contained within VNet peering, rather than routing through a load balancer and back.\nPart 3 concludes the series by discussing the shared AI gateway pattern, observability, security considerations, and identifying when AppNet is suitable or not.\nSources Validated against the following sources in May 2026.\nMicrosoft Learn — Overview of Azure Kubernetes Application Network for AKS (Preview) Microsoft Learn — Get Started Microsoft Learn — Architecture Microsoft Learn — Traffic Management Use Cases Microsoft Learn — Supported Versions AKS Engineering Blog — Control AI spend with per-application token rate limiting using Application Network and agentgateway Istio — Configure waypoint proxies Istio — Add workloads to the mesh Istio — ztunnel architecture reference ","permalink":"https://wolkwacht.nl/posts/azure-kubernetes-application-network-part-2-zero-trust-and-multi-cluster-patterns/","summary":"\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*8S6LwgmX_4rH_reHJddXvw.png\"\u003e\u003c/p\u003e\n\u003cp\u003eIn Part 1, we discussed the AppNet architecture, setup process, and waypoint model. This second part shifts focus from mechanics to platform patterns: it covers enforcing service-to-service authorization for a payments platform, and then using AppNet across AKS clusters for active-active regional architectures.\u003c/p\u003e\n\u003ch3 id=\"series-navigation\"\u003eSeries navigation\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"https://medium.com/@jurgenallewijn/azure-kubernetes-application-network-part-1-what-appnet-is-and-how-to-get-started-ceb39df26624\"\u003ePart 1: What AppNet is and how to get started\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003ePart 2: Zero-trust and multi-cluster patterns\u003c/li\u003e\n\u003cli\u003ePart 3: AI gateway, observability and production fit\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"example-one-zero-trust-authorization-for-a-paymentsplatform\"\u003eExample one: Zero-trust authorization for a payments platform\u003c/h2\u003e\n\u003cp\u003e\u003cimg alt=\"Zero-Trust Authorization\\u200a—\\u200aPayments Namespace\" loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*JO-dNjxdSS6v_qsk_kOxaQ.jpeg\"\u003e\n\u003cem\u003eZero-Trust Authorization — Payments Namespace\u003c/em\u003e\u003c/p\u003e","title":"Azure Kubernetes Application Network, Part 2: Zero-Trust and Multi-Cluster Patterns"},{"content":"Azure Kubernetes Application Network, Part 1: What AppNet Is and How to Get Started The space between \u0026lsquo;we have ingress\u0026rsquo; and \u0026lsquo;we run a full service mesh with sidecars on every pod\u0026rsquo; has always been a challenging middle ground. Platform teams want encrypted east-west traffic, identity-based authorization, and good observability, but aim to avoid the CPU, memory, startup delay, and on-call burden associated with a sidecar on each workload. Security architects look for a zero-trust framework they can verify during audits, not just visual diagrams. Application developers need routing, retries, and traffic shifting without needing to understand Envoy\u0026rsquo;s internals. Meanwhile, platform engineers prefer these features as first-class Azure resources rather than managing complex Helm charts that require constant oversight.\nAzure Kubernetes Application Network, or \u0026ldquo;AppNet,\u0026rdquo; is Microsoft’s solution for a balanced networking layer. It provides a fully managed Layer 7 network for Azure Kubernetes Service, built on top of Istio’s ambient mode. In this setup, ztunnel manages Layer 4 mutual TLS on each node, while waypoint proxies handle Layer 7 routing and policies, but only when enabled. Microsoft manages the control plane and updates, allowing users to set policies via Kubernetes-native APIs. The data plane remains separate from your pods. The product features a dedicated Azure resource type, created with az appnet create, with clusters added as members. Policies are defined using Istio and Gateway API resources familiar to existing teams.\nThis three-part series explains what AppNet is, guides you through setting it up on AKS, and explores how it transforms common platform patterns such as zero-trust authorization, multi-cluster active-active setups, and shared AI gateways.\nValidation note: The commands, prerequisites, and limitations discussed in this article were verified against Microsoft Learn and Istio documentation in May 2026. Since AppNet is still in preview, it’s advisable to confirm the latest CLI extension, feature registration, and supported version matrix before applying these examples in a production environment. Before deploying, first check if it is already available in the selected region.\nIn this first part, we cover the basics: what AppNet is, how the ambient architecture functions, the responsibilities of the Azure control plane, and the process of enrolling a namespace with a waypoint.\nSeries navigation Part 1: What AppNet is and how to get started Part 2: Zero-trust and multi-cluster patterns Part 3: AI gateway, observability and production fit What the Azure Kubernetes Application Network actually is Beyond marketing, Azure Kubernetes Application Network (AKAN) consists of three components. Firstly, it is an Azure resource type visible in the portal, ARM, Bicep, and the az appnet group in Azure CLI, with its lifecycle independent of any AKS cluster. Secondly, it functions as a managed control plane operated by Microsoft, based on Istio’s ambient architecture. Thirdly, it serves as a data plane within your AKS clusters, featuring a per-node ztunnel DaemonSet that manages Layer 4 mTLS and optional waypoint proxy Deployments that handle Layer 7.\nThe importance of the “ambient” component is often overlooked. In traditional Istio, each pod added to the mesh receives an Envoy sidecar. This sidecar handles mTLS termination and origination, runs Layer 7 filters, and controls the pod’s network namespace. While effective, this approach incurs high costs: a sidecar typically consumes 50 to several hundred megabytes of memory, increases pod startup times, requires careful management of injection and restart procedures when updating mesh configurations, and necessitates rolling updates for all pods whenever Istio is upgraded.\nAmbient mode separates these responsibilities. Secure transport and basic identity are managed by ztunnel, a Rust-based proxy running as a DaemonSet that handles Layer 4 functions: transparent mTLS with SPIFFE identities from Kubernetes service accounts, Layer 4 authorization, and telemetry. Ztunnel does not access Layer 7 data unless explicitly permitted. For Layer 7 tasks, such as HTTP routing, header modification, retries, timeouts, L7 authorization, traffic shifting, and fault injection, ambient mode uses waypoint proxies. These are standard Kubernetes Deployments of Envoy, attached to a namespace, service, or workload. If you do not deploy a waypoint in a namespace, that namespace does not incur L7 processing costs. Deploying a waypoint in a namespace ensures traffic is transparently routed through it.\nAzure Kubernetes Application Network takes this architecture, wraps it in an Azure resource, and hides the parts you shouldn’t have to worry about. You don’t install ztunnel, run Istiod, manage root certificates, own the upgrade cadence, or write Helm values for the mesh control plane. You create an AppNet, attach your AKS clusters as members, label the namespaces you want in the mesh, and from that point onward, you write policy in Kubernetes YAML.\nThe architecture at a level that matters for operations Azure Kubernetes Application Network — Three-Layer Architecture\nMicrosoft outlines AppNet as consisting of three layers. The management plane is the Azure resource surface: the [Microsoft.AppLink/appLinks](https://learn.microsoft.com/en-us/azure/templates/microsoft.applink/change-log/summary) resource type, the az appnet` CLI, the portal blade, and the RBAC that governs who can modify the AppNet and attach clusters to it. This layer is where platform teams determine, for example, “these five AKS clusters in Europe are part of our shared AppNet, while those three in the US belong to a different one,” and manage permissions to add or remove members.\nThe control plane is the part Microsoft runs for you. It is an Azure‑hosted Istiod that manages workload identities and pushes configuration to ztunnel and the waypoints. You never SSH into it, scale it, or handle on-call shifts for it. When a new version is released, Microsoft updates it in the control plane; the supported‑versions page lists compatible data plane versions. This means your mesh upgrade process is simply to follow the supported matrix during AKS updates, rather than orchestrating an Istiod deployment across all environments.\nThe data plane is what actually sits in your clusters. The ztunnel DaemonSet runs one pod per node in the applink-system namespace, alongside the istio-cni-node DaemonSet, which handles traffic interception. Ztunnel intercepts traffic from any pod whose namespace is labeled withistio.io/dataplane-mode=ambient. It does this at the network level rather than by injecting itself into the pod, which is why there is no sidecar and why the pod spec you write for your application is identical to what you’d write without the mesh. Waypoints are separate Deployments, usually one per namespace, scheduled through the Gateway API Istio waypoint reference. A waypoint scales like any other deployment; you can assign it node affinity, HPA rules, PodDisruptionBudgets, and resource requests, and it participates in cluster autoscaling decisions as usual.\nIdentity is the piece that ties it all together and should be the top priority for security architects. Every workload in an enrolled namespace gets a SPIFFE identity of the form spiffe://\u0026lt;trust-domain\u0026gt;/ns/\u0026lt;namespace\u0026gt;/sa/\u0026lt;serviceaccount\u0026gt;. That identity is bound to the Kubernetes service account the pod runs as. It’s presented during the mTLS handshake handled by ztunnel, and it’s what your Istio AuthorizationPolicy and PeerAuthentication resources reason about. If you’ve ever written a policy that says “workload X in namespace Y may call workload Z”, AppNet is the mechanism that lets you enforce that at the network level rather than hoping the application remembers to check a bearer token.\nPrerequisites and a clean setup AppNet has a few hard prerequisites that catch teams off guard. Read the get-started reference and supported versions before you start. At the time of writing, AppNet requires AKS 1.30 or later, AKS‑managed Microsoft Entra integration enabled (—enable-aad), the managed Kubernetes Gateway API enabled (—enable-gateway-api), Azure CLI 2.84 or later, and the appnet-preview extension installed. The AKS Istio service mesh add-on must not be enabled on the target cluster. Because this is still a preview service, also check the current limitations page for private cluster, Windows node pool, and regional support before you commit to a production design.\nThe shape of a clean bootstrap looks like this. You start with environment variables, so the rest of the commands are copy‑pasteable.\n# Environment export LOCATION=\u0026#34;eastus\u0026#34; export APPNET_RG=\u0026#34;rg-appnet-prod-eus\u0026#34; export APPNET_NAME=\u0026#34;appnet-prod-eus\u0026#34; export AKS_RG=\u0026#34;rg-aks-prod-eus\u0026#34; export CLUSTER_NAME=\u0026#34;aks-prod-eus-01\u0026#34; export MEMBER_NAME=\u0026#34;aks-prod-eus-01\u0026#34; # Tooling az extension add --name appnet-preview --upgrade az extension add --name aks-preview --upgrade Because AppNet is in public preview, register the Microsoft.AppLink Because AppNet is in public preview, register the Microsoft.AppLink provider and the AppNet public preview feature once per subscription. After registration completes, refresh the provider so the resource type is available in ARM and the CLI.provider and the AppNet public preview feature once per subscription. After registration completes, refresh the provider so the resource type is available to ARM and the CLI.\naz provider register --namespace Microsoft.AppLink az feature register --namespace Microsoft.AppLink \\ --name PublicPreview # Wait until the feature shows \u0026#34;Registered\u0026#34;, then refresh the provider az provider register --namespace Microsoft.AppLink Once the tooling is up to date and the feature is registered, create the resource group and the AppNet. The AppNet is cluster‑agnostic at this point; it is an Azure resource (type Microsoft.AppLink/appLinks) that will later own the member bindings.\naz group create \\ --name \u0026#34;$APPNET_RG\u0026#34; \\ --location \u0026#34;$LOCATION\u0026#34; az group create \\ --name \u0026#34;$AKS_RG\u0026#34; \\ --location \u0026#34;$LOCATION\u0026#34; az appnet create \\ --resource-group \u0026#34;$APPNET_RG\u0026#34; \\ --name \u0026#34;$APPNET_NAME\u0026#34; \\ --location \u0026#34;$LOCATION\u0026#34; \\ --identity-type SystemAssigned For demo work, — identity-type SystemAssigned is the path of least resistance. For production, a user‑assigned managed identity is the cleaner choice because its lifecycle is decoupled from the AppNet resource, which matters when you need to recreate the AppNet without losing role assignments. Replace the flag with — identity-type UserAssigned — user-assigned-identities \u0026lt;resourceId\u0026gt; and pre‑create the identity using Bicep or Terraform.\nThe provisioning step typically takes a few minutes. When you run az appnet show \u0026ndash;resource-group “$APPNET_RG\u0026quot; \u0026ndash;name “$APPNET_NAME\u0026quot;, it will display properties.provisioningState: Succeeded once the control plane is ready. At this stage, there is no activity inside any cluster yet because no cluster has been attached.\nTo attach an existing AKS cluster, first verify it meets the prerequisites, then use az appnet member join. The cluster needs AKS-managed Microsoft Entra integration, and the managed Kubernetes Gateway API enabled, but it should not have the AKS Istio service mesh add-on activated. In a new environment, it appears as follows.\n# Create the target cluster with AKS-managed Entra and Gateway API az aks create \\ --resource-group \u0026#34;$AKS_RG\u0026#34; \\ --name \u0026#34;$CLUSTER_NAME\u0026#34; \\ --location \u0026#34;$LOCATION\u0026#34; \\ --node-vm-size Standard_D2s_v3 \\ --enable-aad \\ --enable-gateway-api \\ --generate-ssh-keys SUBSCRIPTION_ID=$(az account show --query id -o tsv) az appnet member join \\ --resource-group \u0026#34;$APPNET_RG\u0026#34; \\ --appnet-name \u0026#34;$APPNET_NAME\u0026#34; \\ --member-name \u0026#34;$MEMBER_NAME\u0026#34; \\ --member-resource-id \u0026#34;/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$AKS_RG/providers/Microsoft.ContainerService/managedClusters/$CLUSTER_NAME\u0026#34; \\ --upgrade-mode FullyManaged The -upgrade-mode FullyManaged value is ideal for most platform teams: Microsoft automatically applies compatible data-plane updates to the cluster as the control plane progresses. SelfManaged is an alternative if you need to lock in a specific ztunnel version during a change window; it is helpful for strict testing requirements, but requires you to manage the upgrade timeline independently.\nAfter the member joins, the AppNet data plane is installed in the cluster, and the managed control plane starts monitoring it. You can sanity‑check from kubectl by looking at the applink-system namespace, which is where the ztunnel and CNI DaemonSets live for AppNet (this is distinct from the aks-istio-system namespace used by the unrelated AKS Istio service mesh add-on.\naz aks get-credentials \\ --resource-group \u0026#34;$AKS_RG\u0026#34; \\ --name \u0026#34;$CLUSTER_NAME\u0026#34; \\ --overwrite-existing \\ --admin kubectl get pods -n applink-system kubectl get daemonset -n applink-system You should see a ztunnel DaemonSet and an istio-cni-node DaemonSet, each running one pod per node. At this point, the mesh is technically there, but no application namespace is using it.\nEnrolling a namespace and deploying a waypoint Enrolling a namespace in the ambient data plane involves applying a single label. This label is detected by both the managed control plane and ztunnel on each node, and changing it within a namespace causes traffic in that namespace to join AppNet.\nkubectl create namespace payments kubectl label namespace payments \\ istio.io/dataplane-mode=ambient Currently, each pod in the payments namespace has its L4 traffic routed through ztunnel. Service-to-service calls between pods in AppNet-enrolled namespaces are now automatically encrypted with mTLS. The pod specifications remain unchanged, with no restarts or sidecars involved. Additionally, existing NetworkPolicies continue to function as before, since ztunnel respects them.\nTo access Layer 7 features such as route rules, L7 authorization, traffic splitting, retries, or any functionality based on HTTP headers and methods, you deploy a waypoint. The Gateway API resource for a waypoint uses gatewayClassName: istio-waypoint and includes a single HBONE listener on port 15008, Istio’s inbound HBONE port, allowing the waypoint to terminate the tunnel.\napiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: payments-waypoint namespace: payments labels: istio.io/waypoint-for: service spec: gatewayClassName: istio-waypoint listeners: - name: mesh port: 15008 protocol: HBONE kubectl apply -f payments-waypoint.yaml kubectl label namespace payments \\ istio.io/use-waypoint=payments-waypoint --overwrite The istio.io/waypoint-for: service label on the Gateway determines which types of traffic Istiod should redirect; service indicates all traffic within the cluster targeting services in this namespace. The istio.io/use-waypoint label on the namespace marks it as the default waypoint for all its services. If a specific service needs a different waypoint, such as when it has a higher request volume and requires dedicated proxy resources, the label can be applied directly to that Service resource. In such cases, the service-level label overrides the namespace-level label.\nAmbient Mode Traffic Flow — L4 vs L7\nUnderstanding what changes after this is important. Traffic entering the payments namespace from another AppNet-enrolled pod is first captured by that pod’s local ztunnel, then tunneled over HBONE to the waypoint, inspected at Layer 7 there, and finally forwarded to the destination pod’s ztunnel before reaching the workload. Traffic from outside the mesh, such as requests from an external ingress controller on a non-enrolled namespace, follows the routing configured for that ingress. AppNet does not modify your Ingress or external Gateway resources; instead, it alters the east‑west communication within the cluster fleet.\nPart 2 builds on this foundation with two platform patterns: identity-based authorization for a payments namespace and active-active traffic across AKS clusters.\nSources Validated against the following sources in May 2026.\nMicrosoft Learn — Overview of Azure Kubernetes Application Network for AKS (Preview) Microsoft Learn — Get Started Microsoft Learn — Architecture Microsoft Learn — Traffic Management Use Cases Microsoft Learn — Supported Versions AKS Engineering Blog —Control AI spend with per-application token rate limiting using Application Network and agentgateway Istio — Configure waypoint proxies Istio — Add workloads to the mesh Istio — ztunnel architecture reference ","permalink":"https://wolkwacht.nl/posts/azure-kubernetes-application-network-part-1-what-appnet-is-and-how-to-get-started/","summary":"\u003ch2 id=\"azure-kubernetes-application-network-part-1-what-appnet-is-and-how-to-getstarted\"\u003eAzure Kubernetes Application Network, Part 1: What AppNet Is and How to Get Started\u003c/h2\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*dItE5wyaGCr5QdI_9GdEbQ.png\"\u003e\u003c/p\u003e\n\u003cp\u003eThe space between \u0026lsquo;we have ingress\u0026rsquo; and \u0026lsquo;we run a full service mesh with sidecars on every pod\u0026rsquo; has always been a challenging middle ground. Platform teams want encrypted east-west traffic, identity-based authorization, and good observability, but aim to avoid the CPU, memory, startup delay, and on-call burden associated with a sidecar on each workload. Security architects look for a zero-trust framework they can verify during audits, not just visual diagrams. Application developers need routing, retries, and traffic shifting without needing to understand Envoy\u0026rsquo;s internals. Meanwhile, platform engineers prefer these features as first-class Azure resources rather than managing complex Helm charts that require constant oversight.\u003c/p\u003e","title":"Azure Kubernetes Application Network, Part 1: What AppNet Is and How to Get Started"},{"content":"When Your Cluster Isn’t Sufficient: Introducing Virtual Pools in AKS There’s a moment in every Kubernetes journey when the cluster begins to feel… heavy. Not broken, not misconfigured, but stretched in ways that weren’t obvious at first. You scale node pools, adjust autoscaling thresholds, maybe even add spot nodes. Yet, some workloads still don’t quite fit. They either need to scale instantly beyond your node capacity or represent spiky, unpredictable demand that makes your carefully tuned AKS cluster feel rigid.\nThis is where virtual node pools, also known as virtual pools in Azure Kubernetes Service, come into play.\nThe illusion of infinite capacity A virtual pool in AKS is not a traditional node pool. There are no VMs backing it, no kubelet you manage, no OS patches to worry about. Instead, it is a projection of Azure Container Instances (ACI) into your Kubernetes cluster.\nWhen you create a virtual node, AKS integrates ACI as a backend compute option. From the Kubernetes API viewpoint, it appears as just another node. You can schedule pods, add labels, and set taints on it. However, internally, these pods do not run on your cluster’s VM-based node pools; instead, they are deployed directly as ACI containers.\nThe key change is that you\u0026rsquo;re no longer limited by the capacity of your cluster’s node pools. Instead, you\u0026rsquo;re basically expanding your cluster into a serverless execution layer.\nHow it works under the hood The virtual node is implemented using a virtual kubelet, which translates Kubernetes scheduling decisions into Azure Container Instance deployments. When a pod is assigned to this virtual node, the following occurs:\n• The Kubernetes scheduler assigns the pod to the virtual node\n• The virtual kubelet intercepts that scheduling decision\n• It translates the pod spec into an ACI container group\n• Azure spins up the container almost immediately\nThere is no node provisioning step, VM scale set expansion, or image pre-pulling on nodes. The latency profile differs significantly, especially during burst scenarios.\nEnabling virtual pools in AKS To enable a virtual node pool, your cluster must be deployed using Azure CNI networking, which is mandatory. ACI-backed pods need to connect to your virtual network, and this connection requires Azure CNI.\nAn Azure CLI flow looks like this:\n#!/usr/bin/env bash set -euo pipefail RESOURCE_GROUP=\u0026#34;rg-aks-virtualpool\u0026#34; CLUSTER_NAME=\u0026#34;aks-virtualpool-demo\u0026#34; LOCATION=\u0026#34;swedencentral\u0026#34; VNET_NAME=\u0026#34;aks-vnet\u0026#34; SUBNET_NAME=\u0026#34;aks-subnet\u0026#34; ACI_SUBNET_NAME=\u0026#34;aci-subnet\u0026#34; # Create resource group az group create \\ --name $RESOURCE_GROUP \\ --location $LOCATION # Create VNet with two subnets az network vnet create \\ --resource-group $RESOURCE_GROUP \\ --name $VNET_NAME \\ --address-prefix 10.0.0.0/8 \\ --subnet-name $SUBNET_NAME \\ --subnet-prefix 10.240.0.0/16 # Create dedicated subnet for ACI az network vnet subnet create \\ --resource-group $RESOURCE_GROUP \\ --vnet-name $VNET_NAME \\ --name $ACI_SUBNET_NAME \\ --address-prefix 10.241.0.0/16 # Register the ACI resource provider before enabling virtual nodes az provider register \\ --namespace Microsoft.ContainerInstance while [[ \u0026#34;$(az provider show --namespace Microsoft.ContainerInstance --query registrationState -o tsv)\u0026#34; != \u0026#34;Registered\u0026#34; ]]; do echo \u0026#34;Waiting for Microsoft.ContainerInstance registration...\u0026#34; sleep 10 done # Create AKS cluster with Azure CNI az aks create \\ --resource-group $RESOURCE_GROUP \\ --name $CLUSTER_NAME \\ --network-plugin azure \\ --vnet-subnet-id $(az network vnet subnet show \\ --resource-group $RESOURCE_GROUP \\ --vnet-name $VNET_NAME \\ --name $SUBNET_NAME \\ --query id -o tsv) \\ --enable-managed-identity \\ --node-count 2 \\ --node-vm-size Standard_D2s_v3 \\ --generate-ssh-keys \\ --ssh-access disabled # Enable virtual node (ACI) az aks enable-addons \\ --resource-group $RESOURCE_GROUP \\ --name $CLUSTER_NAME \\ --addons virtual-node \\ --subnet-name $ACI_SUBNET_NAME At this point, your cluster contains a virtual node. If you inspect it:\nkubectl get nodes -o wide You’ll see something like:\nNAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME aks-nodepool1-99106009-vmss000000 Ready \u0026lt;none\u0026gt; 11m v1.33.6 10.240.0.4 \u0026lt;none\u0026gt; Ubuntu 22.04.5 LTS 5.15.0-1102-azure containerd://1.7.30-2 aks-nodepool1-99106009-vmss000001 Ready \u0026lt;none\u0026gt; 11m v1.33.6 10.240.0.33 \u0026lt;none\u0026gt; Ubuntu 22.04.5 LTS 5.15.0-1102-azure containerd://1.7.30-2 virtual-node-aci-linux Ready agent 118s v1.25.0-vk-azure-aci-1.6.2 10.240.0.53 \u0026lt;none\u0026gt; \u0026lt;unknown\u0026gt; \u0026lt;unknown\u0026gt; \u0026lt;unknown\u0026gt; Scheduling workloads onto the virtual pool By default, nothing lands on the virtual node. You explicitly target it using node selectors or taints and tolerations.\nA simple example pod:\napiVersion: v1 kind: Pod metadata: name: api-burst-sample labels: app: api tier: burst spec: tolerations: - key: virtual-kubelet.io/provider operator: Equal value: azure effect: NoSchedule containers: - name: api image: mcr.microsoft.com/azuredocs/aci-helloworld resources: requests: cpu: 250m memory: 256Mi nodeSelector: kubernetes.io/role: agent type: virtual-kubelet You can observe it:\nkubectl get pod burst-workload -o wide The node will show as the virtual node, but there is no backing VM.\nA real scenario: burst scaling beyond cluster limits Imagine a platform team managing an API infrastructure on AKS. Usually, the workload remains stable and comfortably fits within a few node pools. However, during peak times such as a marketing campaign or a product launch, traffic can surge significantly.\nTraditional autoscaling with the Cluster Autoscaler causes delays because new nodes need to be provisioned, images must be pulled, and pods scheduled. Despite aggressive tuning, this process can still take several minutes. Now introduce a virtual pool.\nYou configure your deployment with a fallback:\n• Primary scheduling on regular node pools\n• Overflow scheduling on the virtual node\nA pattern using multiple deployments can look like this:\napiVersion: apps/v1 kind: Deployment metadata: name: api-primary spec: replicas: 2 selector: matchLabels: app: api tier: primary template: metadata: labels: app: api tier: primary spec: containers: - name: api image: mcr.microsoft.com/azuredocs/aci-helloworld resources: requests: cpu: 250m memory: 256Mi --- apiVersion: apps/v1 kind: Deployment metadata: name: api-burst spec: replicas: 1 selector: matchLabels: app: api tier: burst template: metadata: labels: app: api tier: burst spec: tolerations: - key: virtual-kubelet.io/provider operator: Equal value: azure effect: NoSchedule containers: - name: api image: mcr.microsoft.com/azuredocs/aci-helloworld resources: requests: cpu: 250m memory: 256Mi nodeSelector: kubernetes.io/role: agent type: virtual-kubelet Now you can scale the burst deployment dynamically:\nkubectl scale deployment api-burst --replicas=5 Those additional replicas are instantly provisioned in ACI, without impacting your cluster capacity.\nOutput will look something like this:\nNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES api-burst-649c89bcb7-kh9pv 1/1 Running 0 2m18s 10.241.0.6 virtual-node-aci-linux \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; api-burst-649c89bcb7-nfjbj 1/1 Running 0 2m18s 10.241.0.9 virtual-node-aci-linux \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; api-burst-649c89bcb7-sb4r8 1/1 Running 0 2m18s 10.241.0.7 virtual-node-aci-linux \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; api-burst-649c89bcb7-tlz7f 1/1 Running 0 2m18s 10.241.0.8 virtual-node-aci-linux \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; api-burst-649c89bcb7-vj9g9 1/1 Running 0 58m 10.241.0.5 virtual-node-aci-linux \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; Those additional replicas are immediately provisioned in ACI, without affecting your cluster capacity.\nWhere virtual pools shine Virtual pools are not meant to replace node pools; rather, they serve as an extension mechanism. They are most effective when workload traits align with ACI’s strengths: quick startup times, minimal infrastructure management, and elastic scaling. Short-term jobs are ideal, such as CI/CD runners, batch processing, or event-triggered data transformation tasks, as they can execute immediately without waiting for node provisioning.\nEvent-driven architectures also benefit; integrating with KEDA enables scaling based on queue length or external signals, with overflow handled by the virtual pool. Additionally, they are useful for isolating untrusted or experimental workloads in ACI via the virtual node, helping contain potential issues within the core cluster. Where virtual pools break down\nThe abstraction is strong, but its trade-offs become quite apparent in real-world architectures.\nNetworking is a primary friction point. Although ACI integrates with your VNet, it lacks support for some Kubernetes networking features. Advanced CNI functions, network policies, and detailed observability tools such as Cilium or eBPF-based tracing are not provided to the same extent.\nStorage presents another limitation; persistent volumes are not supported in the same way as with VM-backed nodes. If your workload requires stateful storage, virtual pools may not be suitable.\nDaemonSets do not operate on virtual nodes. Consequently, your standard observability tools, security agents, or service mesh sidecars might not function as intended.\nLatency-sensitive workloads can also be affected. Although startup is quick, networking paths and performance traits differ from those of VM-based nodes.\nCost is also a factor. ACI has a different billing model. For steady-state workloads, it can sometimes be more costly than using pods on reserved or spot-backed nodes.\nObservability and operational reality From a platform engineering point of view, virtual pools can lead to a split-brain operational situation.\nYour cluster now runs across two execution environments:\n• VM-based node pools with full Kubernetes control\n• ACI-backed virtual nodes with limited control\nYour observability setup should account for this boundary. Tools like Azure Monitor can collect logs and metrics, but for kernel-level details, such as what Inspektor Gadget or Cilium Hubble offer, ACI does not support these. This isn\u0026rsquo;t a flaw, but a limitation.\nUnderstanding this boundary is crucial for architects. A mental model for architects\nThink of virtual pools as an elasticity layer attached to your cluster, rather than as part of it. Your primary workloads run on node pools that you control for environment, networking, and security.\nThe virtual pool functions as a pressure relief valve, enabling you to reduce workloads during demand spikes or when dedicated nodes are unnecessary, effectively serving as a serverless extension.\nThis hybrid approach helps AKS appear less like a fixed cluster and more like an adaptable, evolving platform.\nClosing reflection Virtual pools in AKS may appear straightforward at first, but they fundamentally change how you approach capacity management. They combine Kubernetes with serverless ideas, allowing you to handle unpredictability without overprovisioning. However, they also expose some limitations of the Kubernetes abstraction, as not all behaviors remain consistent.\nFor cloud engineers and platform architects, the main advantage is understanding when to utilize virtual pools and, equally important, when to refrain from using them. Ultimately, architecture focuses on setting appropriate boundaries and understanding the consequences of exceeding them.\n","permalink":"https://wolkwacht.nl/posts/when-your-cluster-isnt-sufficient-introducing-virtual-pools-in-aks/","summary":"\u003ch2 id=\"when-your-cluster-isnt-sufficient-introducing-virtual-pools-inaks\"\u003eWhen Your Cluster Isn’t Sufficient: Introducing Virtual Pools in AKS\u003c/h2\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*APZXx5B08vLSViA-2pLXCA.png\"\u003e\u003c/p\u003e\n\u003cp\u003eThere’s a moment in every Kubernetes journey when the cluster begins to feel… heavy. Not broken, not misconfigured, but stretched in ways that weren’t obvious at first. You scale node pools, adjust autoscaling thresholds, maybe even add spot nodes. Yet, some workloads still don’t quite fit. They either need to scale instantly beyond your node capacity or represent spiky, unpredictable demand that makes your carefully tuned AKS cluster feel rigid.\u003c/p\u003e","title":"When Your Cluster Isn’t Sufficient: Introducing Virtual Pools in AKS"},{"content":"GitOps on Azure Kubernetes Service: Building a Production-Ready Platform with Flux Kubernetes platforms seldom fail due to the technology itself; instead, failures are usually caused by operational issues.\nBetween the initial successful ‘kubectl apply’ and the deployment of the twentieth production cluster, platform teams realize a harsh reality: Kubernetes isn’t hard to run, but maintaining consistent operations is very challenging. Configuration drifts occur. Hotfixes are often applied directly, skipping pipelines. Infrastructure ends up as a patchwork of Git repositories, YAML files, CI/CD pipelines, and manual tweaks performed from someone’s laptop at midnight.\nThis is precisely the issue that GitOps was created to address.\nIn the Kubernetes ecosystem, multiple tools support GitOps workflows, with Flux being one of the most popular. Flux is an open-source continuous delivery toolkit maintained by the Cloud Native Computing Foundation. It constantly reconciles the cluster\u0026rsquo;s actual state with the desired state specified in Git, ensuring the cluster always stays aligned with the configuration stored in version control.\nIn this blog, we will walk through a complete implementation of Flux with Azure Kubernetes Service (AKS). The aim is to go beyond a simple installation and develop a deployment model that meets platform-grade standards, designed for platform engineers and cloud architects.\nWe will set up an AKS cluster with Azure Linux, bootstrap Flux using a GitHub repository, and show how Git commits automatically trigger Kubernetes deployments.\nBy the end, the cluster itself will become almost secondary. Git becomes the platform.\nUnderstanding Flux and the GitOps Model GitOps is a straightforward yet powerful concept. The desired state of infrastructure and applications is specified in Git, and automated controllers ensure this state is maintained within the cluster.\nFlux is a controller that implements this concept. It constantly watches a Git repository and ensures Kubernetes resources are aligned with what is specified in that repository.\nFlux uses a pull-based reconciliation model instead of pushing deployments via pipelines. The cluster retrieves its desired configuration from Git and applies it internally. This setup eliminates the need for external credentials for CI systems to access the cluster.\nThe Flux architecture includes multiple controllers operating within Kubernetes. These controllers observe sources like Git repositories, OCI artifacts, or Helm charts, and then reconcile Kubernetes resources according to the definitions stored within them.\nWhen a change is committed to Git, Flux detects it and updates the cluster automatically. If someone manually alters resources within the cluster, Flux detects the drift and reverts the configuration to match the Git state.\nThis reconciliation loop is the key to GitOps reliability. It ensures deployments are predictable, infrastructure changes are auditable, and rollbacks are simply Git reverts.\nWhy Platform Teams Choose Flux Flux has gained popularity among platform engineering teams because it functions similarly to Kubernetes: it is controller-based, declarative, and composable.\nA key strength of Flux is its Kubernetes-native design. Instead of functioning as a separate control plane, Flux uses controllers that integrate with the Kubernetes API. This enables teams to manage Git repositories, Helm releases, or image automation directly via Kubernetes resources.\nThis architecture seamlessly fits platform engineering models, positioning Kubernetes as the primary API for infrastructure delivery.\nFlux also supports a wide ecosystem, integrating with GitHub, GitLab, Bitbucket, and other Git providers. It also supports OCI artifacts, Helm charts, and managing multiple repositories simultaneously.\nSecurity is also a key benefit. Flux uses a pull-based approach, where the cluster fetches configuration from Git rather than having external systems push deployments into it. This minimizes the number of credentials needed for cluster access.\nFor organizations managing multiple clusters, Flux also enables multi-tenancy and fleet-level management across multiple repositories or environments.\nHowever, Flux has its limitations.\nA common criticism is its user experience when compared to tools like Argo CD. Traditionally, Flux emphasized Git workflows and command-line use over providing rich graphical dashboards.\nAnother challenge is the complexity of concepts. Since Flux offers multiple specialized controllers, such as GitRepository, Kustomization, HelmRelease, and ImageRepository, the initial learning curve may seem quite steep.\nThe positive news is that the ecosystem has progressed greatly, and adding graphical interfaces has made Flux much easier to access.\nThe Emergence of the Flux GUI Flux traditionally focused on Git workflows over graphical management tools, with platform engineers primarily interacting via Git and CLI commands.\nThis has changed recently.\nProjects like Weave GitOps UI and Kubernetes dashboard integrations have created graphical interfaces to visualize GitOps states, detect reconciliation loops, and display deployment statuses.\nThe Flux UI enables engineers to see connected Git repositories, deployed kustomizations, and the health status of reconciliation.\nFor platform teams overseeing numerous services, this visibility is highly beneficial. Engineers can swiftly determine if a Git commit caused a deployment, confirm whether resources were properly reconciled, and locate any failures.\nThe GUI does not substitute GitOps workflows; instead, it enhances them by providing operational visibility into the controllers’ activities within the cluster.\nArchitecture: AKS + Flux GitOps Platform Below is the conceptual architecture of a typical AKS GitOps platform.\nThe Git repository serves as the definitive source of truth. Flux constantly syncs the cluster with this repository. Any modifications committed to Git are automatically reflected in the cluster.\nDeploying a Secure AKS Cluster for GitOps Before installing Flux, we need a Kubernetes cluster configured for platform-level operations.\nIn this example, we will deploy AKS with the following characteristics:\n• Azure Linux node OS\n• Managed identity\n• Azure CNI networking\n• SSH access disabled on nodes\n• RBAC enabled\nDisabling SSH is a crucial security measure. AKS enables you to disable SSH access during cluster creation with the — ssh-access disabled argument, which prevents direct login to cluster nodes. First, configure environment variables.\n#!/usr/bin/env bash set -euo pipefail LOCATION=\u0026#34;swedencentral\u0026#34; RESOURCE_GROUP=\u0026#34;rg-aks-flux-blog\u0026#34; CLUSTER_NAME=\u0026#34;aks-flux-platform\u0026#34; NODE_COUNT=3 NODE_SIZE=\u0026#34;Standard_D4s_v5\u0026#34; Create the resoure group\naz group create \\ --name $RESOURCE_GROUP \\ --location $LOCATION Now create the Azure Kubernetes Cluster\naz aks create \\ --resource-group $RESOURCE_GROUP \\ --name $CLUSTER_NAME \\ --node-count $NODE_COUNT \\ --node-vm-size $NODE_SIZE \\ --network-plugin azure \\ --enable-managed-identity \\ --generate-ssh-keys \\ --node-osdisk-type Managed \\ --os-sku AzureLinux \\ --ssh-access disabled \\ --enable-oidc-issuer \\ --enable-workload-identity Retrieve cluster credentials\naz aks get-credentials \\ --resource-group $RESOURCE_GROUP \\ --name $CLUSTER_NAME Verify the cluster\nkubectl get nodes -o wide Currently, we have a production-ready AKS cluster featuring secured node access and Azure Linux as its foundation.\nInstalling Flux CLI Flux provides a CLI used for bootstrapping GitOps repositories.\nInstall the CLI.\nbrew install fluxcd/tap/flux Verify the installation\nflux --version Check cluster compatibility\nflux check --pre Output will look like\n► checking prerequisites ✔ Kubernetes 1.33.6 \u0026gt;=1.33.0-0 ✔ prerequisites checks passed This command checks the Kubernetes version compatibility and necessary permissions prior to installation.\nBootstrapping Flux with GitHub Flux employs a bootstrap process to install controllers and link the cluster with a Git repository. The bootstrap command sets up the controllers and commits the Flux configuration into the repository. Begin by creating a GitHub Personal Access Token with repository permissions, then export the token.\nexport GITHUB_TOKEN=\u0026lt;your-token\u0026gt; export GITHUB_USER=\u0026lt;github-username\u0026gt; Bootstrap Flux\nflux bootstrap github \\ --token-auth \\ --owner=$GITHUB_USER \\ --repository=aks-flux-platform \\ --branch=main \\ --path=clusters/production \\ --personal During bootstrap, Flux executes multiple actions at once. It installs Flux controllers into the Kubernetes cluster, creates a Git repository if one doesn\u0026rsquo;t already exist, commits the initial GitOps manifests to it, and configures the controllers to manage the cluster state based on the repository. After completing these steps, verify that the controllers are functioning correctly.\nkubectl get pods -n flux-system You should see controllers such as\nNAME READY STATUS RESTARTS AGE helm-controller-58b456999d-ghtgb 1/1 Running 0 100s kustomize-controller-76c5d4c759-jhb99 1/1 Running 0 100s notification-controller-6b5b7c8647-npwrd 1/1 Running 0 100s source-controller-584b6f58bd-hmlmv 1/1 Running 0 100s The cluster is now managed through Git.\nRecommended Git Repository Structure A GitOps repository should follow a structure that separates environments from workloads.\nExample:\naks-flux-platform │ ├── clusters │ └── production │ └── flux-system │ └── apps └── demo ├── deployment.yaml └── kustomization.yaml The clusters directory holds Flux configuration files, while the apps directory contains Kubernetes workloads. This separation enables platform engineers to manage infrastructure separately from application deployments.\nDeploying an Application with Flux Let\u0026rsquo;s deploy a basic NGINX application through GitOps.\nCreate the deployment manifest.\napiVersion: apps/v1 kind: Deployment metadata: name: demo-nginx namespace: default spec: replicas: 2 selector: matchLabels: app: demo-nginx template: metadata: labels: app: demo-nginx spec: containers: - name: nginx image: nginx:1.25 ports: - containerPort: 80 Create a service\napiVersion: v1 kind: Service metadata: name: demo-nginx spec: selector: app: demo-nginx ports: - port: 80 targetPort: 80 type: ClusterIP Create a kustomization.yaml\nresources: - deployment.yaml - service.yaml Now instruct Flux to deploy the application by creating a Flux Kustomization.\napiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: demo-app namespace: flux-system spec: interval: 5m path: ./apps/demo prune: true sourceRef: kind: GitRepository name: flux-system Commit the files to Git.\ngit add . git commit -m \u0026#34;deploy demo nginx app\u0026#34; git push Flux quickly identifies the commit and proceeds with the deployment.\nVerify the deployment.\nGitOps Reconciliation in Action Flux functions via reconciliation loops. It regularly checks the repository every few minutes to confirm that the cluster state aligns with the desired configuration.\nConsider the following scenario.\nAn engineer manually scales the deployment.\nkubectl scale deployment demo-nginx --replicas=5 The cluster now diverges from the Git configuration. During the next reconciliation interval, Flux detects the drift and resets the replica count to match the Git declaration.\nThis behaviour is what makes GitOps powerful. Git becomes the only source of truth.\nVisualizing Flux Operations Below is a simplified reconciliation flow.\nGit Commit | v +---------------+ | Git Repository| +-------+-------+ | v +---------------+ | Flux Source | | Controller | +-------+-------+ | v +---------------+ | Flux Kustomize| | Controller | +-------+-------+ | v +---------------+ | Kubernetes API| +---------------+ The controllers continuously observe Git and reconcile cluster state accordingly.\nObservability and Operations Flux integrates with Kubernetes observability tools like Prometheus and Grafana. Each controller provides metrics and events that describe the reconciliation status. These metrics help platform teams monitor GitOps health across multiple clusters. Operational dashboards usually visualize:\nrepository synchronization status reconciliation success rate deployment latency controller health. The Flux GUI offers similar insights through a graphical interface. Security Considerations A production GitOps platform demands meticulous security planning. In this example, we\u0026rsquo;ve adopted several best practices:\nSSH access to nodes is turned off to minimize attack points GitHub tokens are limited to specific repository permissions Flux controllers run using Kubernetes RBAC instead of external credentials. Since Flux relies on a pull-based approach, the cluster doesn\u0026rsquo;t need inbound connections from CI pipelines. These design choices greatly lower the risk compared to conventional deployment methods.\nLimitations and Trade-Offs Flux performs well in platform engineering but might not suit all teams. Organizations relying on visual deployment processes may lean toward Argo CD because of its established UI. Teams new to GitOps could find Flux\u0026rsquo;s concepts initially complex. Its modular controller structure also adds more Kubernetes objects that engineers need to grasp. Nonetheless, for platform teams building large-scale infrastructure, Flux provides a highly powerful and adaptable option.\nWhen Flux Becomes a Platform Primitive As Flux scales, it shifts from being just a deployment tool to becoming an integral part of the platform itself. Clusters are no longer set up manually; instead, everything is managed through Git. New clusters are automatically created and configured via GitOps repositories. Platform engineers can specify baseline infrastructure, networking policies, observability stacks, and security measures using declarative configurations. When a new environment is launched, Flux ensures the cluster aligns with the desired state set by the platform team. This approach is increasingly popular among modern platform engineering groups. In this model, Git acts as the API, Flux as the reconciling agent, and Kubernetes as the execution platform.\nFinal Thoughts Integrating Azure Kubernetes Service with Flux forms a robust GitOps-based platform architecture. AKS offers a managed Kubernetes control plane and scalable infrastructure, while Flux acts as the continuous reconciliation engine that aligns clusters with Git. This configuration ensures infrastructure and applications are versioned, auditable, and reproducible. For platform engineers and cloud architects, this method addresses many operational issues that arise as Kubernetes environments expand beyond a single cluster. Infrastructure stability improves, deployments become traceable, clusters can self-heal, and most significantly, the platform can be completely rebuilt from Git.\n","permalink":"https://wolkwacht.nl/posts/gitops-on-azure-kubernetes-service-building-a-production-ready-platform-with-flux/","summary":"\u003ch2 id=\"gitops-on-azure-kubernetes-service-building-a-production-ready-platform-withflux\"\u003eGitOps on Azure Kubernetes Service: Building a Production-Ready Platform with Flux\u003c/h2\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*Qi1MI875g92Eoz0jeeH1AQ.png\"\u003e\u003c/p\u003e\n\u003cp\u003eKubernetes platforms seldom fail due to the technology itself; instead, failures are usually caused by operational issues.\u003c/p\u003e\n\u003cp\u003eBetween the initial successful ‘kubectl apply’ and the deployment of the twentieth production cluster, platform teams realize a harsh reality: Kubernetes isn’t hard to run, but maintaining consistent operations is very challenging. Configuration drifts occur. Hotfixes are often applied directly, skipping pipelines. Infrastructure ends up as a patchwork of Git repositories, YAML files, CI/CD pipelines, and manual tweaks performed from someone’s laptop at midnight.\u003c/p\u003e","title":"GitOps on Azure Kubernetes Service: Building a Production-Ready Platform with Flux"},{"content":"When Kubernetes Lies: Seeing the Truth Inside AKS with Inspektor Gadget Cloud platforms are built on abstractions.\nWe tell developers that services talk to services. We tell security teams that network policies enforce isolation. We tell architects that identity is the new perimeter and that zero trust ensures nothing moves unless explicitly allowed. In Azure Kubernetes Service, these abstractions become even more powerful. Managed control planes, private clusters, Cilium networking, workload identity, Azure Firewall, private endpoints, and policy enforcement create the feeling of a highly controlled, highly predictable platform.\nAnd most of the time, that feeling is justified.\nUntil the day it isn’t.\nBeneath every abstraction, whether it\u0026rsquo;s a YAML file, Azure policy, or other layer, there\u0026rsquo;s always a Linux kernel executing system calls. Containers function as processes; pods are simply cgroups; and network policies boil down to packet-filtering decisions. When issues arise unexpectedly in production, it\u0026rsquo;s the kernel that holds the definitive truth.\nThe problem is that most cloud tooling never shows you that truth.\nMetrics reveal symptoms, logs provide interpretations, network tools display flows between pods, and security tools show alerts. However, when it comes to answering the crucial question, what actually happened?, these layers of abstraction often become obstacles rather than helpful solutions. This is where kernel-level observability changes how you operate Azure Kubernetes Service.\nThis is where Inspektor Gadget becomes one of the most valuable tools you can add to an AKS platform.\nThe gap between architecture and reality Modern AKS environments consist of layered controls making them complex systems. A single production cluster might incorporate Azure CNI or overlay networking, Cilium or kube-proxy, private DNS zones, outbound routing via NAT Gateway or Azure Firewall, workload identity for Azure resources, admission control policies, and various observability pipelines feeding into Azure Monitor and Log Analytics.\nFrom a design perspective, this aligns perfectly with our goals. We achieve defense-in-depth, least privilege, segmentation, auditability, and centralized monitoring. The architecture diagrams are clear, policies appear to be enforced, and dashboards indicate a healthy system.\nHowever, runtime behavior is distinct from architecture; it refers to what processes actually perform.\nWhen a compromised container opens a shell and begins probing internal services, this activity might not be clearly visible in application logs. Similarly, if a third-party library makes silent calls to external endpoints, metrics alone may not reveal the reason. Additionally, a misconfiguration that causes a workload to resolve public DNS instead of private DNS can introduce issues several layers removed from the initial cause.\nAt larger scales, these blind spots become more significant. In regulated settings, their importance grows even further. Regulations such as NIS2, DORA, and GDPR now require organizations to show operational control, not merely their intended configuration. It\u0026rsquo;s no longer sufficient to understand what your cluster is designed to do; you must also provide proof of its actual behavior.\nThat evidence lives in the kernel.\nWhat Inspektor Gadget really is Inspektor Gadget is a sandbox project under the Cloud Native Computing Foundation that aims to introduce kernel-level observability into Kubernetes. The official project website can be found at: https://www.inspektor-gadget.io/\nThe source code and releases are maintained on GitHub: https://github.com/inspektor-gadget/inspektor-gadget\nInspektor Gadget leverages eBPF, a Linux technology that enables small programs to run safely within the kernel. These programs are linked to specific kernel events, such as process execution, network connections, DNS resolution, file access, or system calls. Since the observation occurs directly within the kernel, visibility is precise and comprehensive, unaffected by layers above.\nWhat sets Inspektor Gadget apart from standard eBPF tools is its Kubernetes awareness. The kernel-collected events are enhanced with Kubernetes-specific details, including namespace, pod name, container name, and labels. This allows you to identify which workload created processes and sockets, rather than just seeing anonymous data.\nOn AKS, this is especially important. The best practice in Azure is to avoid allowing SSH access to nodes, and changing the host is not recommended. Inspektor Gadget runs as a privileged DaemonSet, functioning within the supported Kubernetes security framework while still exposing kernel-level events.\nThis provides an uncommon level of insight into cloud environments: direct visibility into container activity. To install Inspektor Gadget on Azure Kubernetes Service, you need a Linux-based AKS cluster and the necessary permissions to deploy resources and run cluster-wide tools. Once your local setup is connected to the cluster, install Inspektor Gadget using Helm.\nExample script for deploying Azure Kubernetes with Azure CNI Overlay and Cilium:\n#!/usr/bin/env bash set -euo pipefail # ----------------------------------------------------------------------------- # AKS + Azure CNI Overlay + Cilium dataplane + ACNS (AKS-managed Hubble Relay) # + Azure Linux node OS # # References: # - Create with ACNS + Cilium dataplane: https://learn.microsoft.com/azure/aks/how-to-configure-container-network-logs # - ACNS overview: https://learn.microsoft.com/azure/aks/advanced-container-networking-services-overview # - Azure Linux 3 guidance: https://learn.microsoft.com/azure/azure-linux/how-to-enable-azure-linux-3 # ----------------------------------------------------------------------------- # ====== EDIT THESE ====== LOCATION=\u0026#34;${LOCATION:-swedencentral}\u0026#34; # Choose an AKS-supported region (e.g. swedensouth, westeurope, northeurope) RESOURCE_GROUP=\u0026#34;${RESOURCE_GROUP:-rg-aks-acns-cilium-se}\u0026#34; CLUSTER_NAME=\u0026#34;${CLUSTER_NAME:-aks-acns-cilium-se}\u0026#34; NODE_COUNT=\u0026#34;${NODE_COUNT:-2}\u0026#34; NODE_VM_SIZE=\u0026#34;${NODE_VM_SIZE:-Standard_D4s_v5}\u0026#34; POD_CIDR=\u0026#34;${POD_CIDR:-192.168.0.0/16}\u0026#34; DISABLE_SSH=\u0026#34;${DISABLE_SSH:-true}\u0026#34; # Set to \u0026#39;true\u0026#39; to disable SSH access to nodes SKIP_KUBECONFIG=\u0026#34;${SKIP_KUBECONFIG:-false}\u0026#34; # Set to \u0026#39;true\u0026#39; to skip downloading kubeconfig KUBECONFIG_PATH=\u0026#34;${KUBECONFIG_PATH:-${HOME}/.kube/config}\u0026#34; # Path to save kubeconfig # Kubernetes version: # ACNS+Cilium features are supported on newer versions (docs show 1.33+ examples). # Set to an available version in your region if you want to pin. K8S_VERSION=\u0026#34;${K8S_VERSION:-}\u0026#34; # Azure Linux: # --os-sku AzureLinux will default to Azure Linux 3.0 on sufficiently new AKS versions. # If you MUST pin AzureLinux3 explicitly (without changing k8s version), check your CLI supports it. OS_SKU=\u0026#34;${OS_SKU:-AzureLinux}\u0026#34; # ======================= require() { command -v \u0026#34;$1\u0026#34; \u0026gt;/dev/null 2\u0026gt;\u0026amp;1 || { echo \u0026#34;Missing dependency: $1\u0026#34; \u0026gt;\u0026amp;2; exit 1; }; } require az require kubectl require curl require tar require sha256sum echo \u0026#34;==\u0026gt; Azure login check\u0026#34; az account show \u0026gt;/dev/null 2\u0026gt;\u0026amp;1 || az login \u0026gt;/dev/null echo \u0026#34;==\u0026gt; Preflight: validate AKS availability in ${LOCATION}\u0026#34; if ! az aks get-versions --location \u0026#34;${LOCATION}\u0026#34; -o none 2\u0026gt;/dev/null; then echo \u0026#34;AKS is not available or the location is invalid: ${LOCATION}\u0026#34; \u0026gt;\u0026amp;2 echo \u0026#34;Try another region, e.g. swedensouth, or check your subscription permissions.\u0026#34; \u0026gt;\u0026amp;2 exit 1 fi echo \u0026#34;==\u0026gt; Create resource group: ${RESOURCE_GROUP} (${LOCATION})\u0026#34; az group create --name \u0026#34;${RESOURCE_GROUP}\u0026#34; --location \u0026#34;${LOCATION}\u0026#34; -o none echo \u0026#34;==\u0026gt; Create AKS cluster (Azure CNI overlay + Cilium dataplane + ACNS + Azure Linux)\u0026#34; AKS_CREATE_ARGS=( --name \u0026#34;${CLUSTER_NAME}\u0026#34; --resource-group \u0026#34;${RESOURCE_GROUP}\u0026#34; --location \u0026#34;${LOCATION}\u0026#34; --node-count \u0026#34;${NODE_COUNT}\u0026#34; --node-vm-size \u0026#34;${NODE_VM_SIZE}\u0026#34; --network-plugin azure --network-plugin-mode overlay --pod-cidr \u0026#34;${POD_CIDR}\u0026#34; --network-dataplane cilium --enable-acns --os-sku \u0026#34;${OS_SKU}\u0026#34; ) if [[ \u0026#34;${DISABLE_SSH}\u0026#34; != \u0026#34;true\u0026#34; ]]; then AKS_CREATE_ARGS+=( --generate-ssh-keys ) fi if [[ -n \u0026#34;${K8S_VERSION}\u0026#34; ]]; then AKS_CREATE_ARGS+=( --kubernetes-version \u0026#34;${K8S_VERSION}\u0026#34; ) fi az aks create \u0026#34;${AKS_CREATE_ARGS[@]}\u0026#34; -o none echo \u0026#34;==\u0026gt; Get kubeconfig\u0026#34; if [[ \u0026#34;${SKIP_KUBECONFIG}\u0026#34; == \u0026#34;true\u0026#34; ]]; then echo \u0026#34; (Skipping kubeconfig download)\u0026#34; else mkdir -p \u0026#34;$(dirname \u0026#34;${KUBECONFIG_PATH}\u0026#34;)\u0026#34; az aks get-credentials \\ --resource-group \u0026#34;${RESOURCE_GROUP}\u0026#34; \\ --name \u0026#34;${CLUSTER_NAME}\u0026#34; \\ --file \u0026#34;${KUBECONFIG_PATH}\u0026#34; \\ --overwrite-existing -o none echo \u0026#34; Kubeconfig saved to: ${KUBECONFIG_PATH}\u0026#34; export KUBECONFIG=\u0026#34;${KUBECONFIG_PATH}\u0026#34; fi echo \u0026#34;==\u0026gt; Verify nodes (ensure OS is Azure Linux)\u0026#34; kubectl get nodes -o wide echo \u0026#34;==\u0026gt; Verify Cilium pods exist (AKS-managed Cilium runs in kube-system)\u0026#34; kubectl -n kube-system get pods | grep -E \u0026#34;^cilium-\u0026#34; \u0026gt;/dev/null echo \u0026#34;==\u0026gt; Verify ACNS-managed Hubble Relay is running\u0026#34; # Microsoft docs use label k8s-app=hubble-relay kubectl get pods -n kube-system -l k8s-app=hubble-relay -o wide echo \u0026#34;==\u0026gt; Install Hubble CLI (as per Microsoft docs)\u0026#34; # Pin the version shown in docs; update if your cluster components require a newer one. HUBBLE_VERSION=\u0026#34;${HUBBLE_VERSION:-v1.16.3}\u0026#34; HUBBLE_ARCH=\u0026#34;amd64\u0026#34; if [[ \u0026#34;$(uname -m)\u0026#34; == \u0026#34;aarch64\u0026#34; || \u0026#34;$(uname -m)\u0026#34; == \u0026#34;arm64\u0026#34; ]]; then HUBBLE_ARCH=\u0026#34;arm64\u0026#34;; fi tmpdir=\u0026#34;$(mktemp -d)\u0026#34; trap \u0026#39;rm -rf \u0026#34;${tmpdir}\u0026#34;\u0026#39; EXIT ( cd \u0026#34;${tmpdir}\u0026#34; curl -L --fail --remote-name-all \\ \u0026#34;https://github.com/cilium/hubble/releases/download/${HUBBLE_VERSION}/hubble-linux-${HUBBLE_ARCH}.tar.gz\u0026#34; \\ \u0026#34;https://github.com/cilium/hubble/releases/download/${HUBBLE_VERSION}/hubble-linux-${HUBBLE_ARCH}.tar.gz.sha256sum\u0026#34; sha256sum --check \u0026#34;hubble-linux-${HUBBLE_ARCH}.tar.gz.sha256sum\u0026#34; # Install to /usr/local/bin if writable, else ~/.local/bin install_dir=\u0026#34;/usr/local/bin\u0026#34; if [[ ! -w \u0026#34;${install_dir}\u0026#34; ]]; then install_dir=\u0026#34;${HOME}/.local/bin\u0026#34; mkdir -p \u0026#34;${install_dir}\u0026#34; export PATH=\u0026#34;${install_dir}:${PATH}\u0026#34; fi tar xzvfC \u0026#34;hubble-linux-${HUBBLE_ARCH}.tar.gz\u0026#34; \u0026#34;${install_dir}\u0026#34; ) echo \u0026#34;==\u0026gt; Port-forward Hubble Relay (leave this running in a separate terminal when observing flows)\u0026#34; echo \u0026#34; Command:\u0026#34; echo \u0026#34; kubectl port-forward -n kube-system svc/hubble-relay --address 127.0.0.1 4245:443\u0026#34; echo echo \u0026#34;==\u0026gt; After port-forward is running, you can observe flows with:\u0026#34; echo \u0026#34; hubble status\u0026#34; echo \u0026#34; hubble observe --follow\u0026#34; echo # ----------------------------------------------------------------------------- # Optional: Deploy Hubble UI (Microsoft manifest pattern) + port-forward # This expects the secret \u0026#39;hubble-relay-client-certs\u0026#39; to exist (ACNS-managed Hubble creates it). # ----------------------------------------------------------------------------- echo \u0026#34;==\u0026gt; Deploy Hubble UI (optional but recommended)\u0026#34; cat \u0026lt;\u0026lt;\u0026#39;YAML\u0026#39; | kubectl apply -f - apiVersion: v1 kind: ServiceAccount metadata: name: hubble-ui namespace: kube-system labels: app.kubernetes.io/part-of: retina --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: hubble-ui labels: app.kubernetes.io/part-of: retina rules: - apiGroups: [\u0026#34;\u0026#34;] resources: [\u0026#34;namespaces\u0026#34;, \u0026#34;pods\u0026#34;, \u0026#34;services\u0026#34;] verbs: [\u0026#34;get\u0026#34;, \u0026#34;list\u0026#34;, \u0026#34;watch\u0026#34;] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: hubble-ui labels: app.kubernetes.io/part-of: retina roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: hubble-ui subjects: - kind: ServiceAccount name: hubble-ui namespace: kube-system --- apiVersion: v1 kind: ConfigMap metadata: name: hubble-ui-nginx namespace: kube-system data: nginx.conf: | server { listen 8081; server_name _; gzip on; gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript; root /usr/share/nginx/html; location / { try_files $uri $uri/ /index.html; } location /healthz { add_header Content-Type text/plain; return 200 \u0026#39;ok\u0026#39;; } } --- apiVersion: apps/v1 kind: Deployment metadata: name: hubble-ui namespace: kube-system labels: k8s-app: hubble-ui app.kubernetes.io/name: hubble-ui app.kubernetes.io/part-of: retina spec: replicas: 1 selector: matchLabels: k8s-app: hubble-ui template: metadata: labels: k8s-app: hubble-ui app.kubernetes.io/name: hubble-ui app.kubernetes.io/part-of: retina spec: serviceAccountName: hubble-ui automountServiceAccountToken: true nodeSelector: kubernetes.io/os: linux containers: - name: frontend image: mcr.microsoft.com/oss/cilium/hubble-ui:v0.12.2 imagePullPolicy: Always ports: - name: http containerPort: 8081 livenessProbe: httpGet: path: /healthz port: 8081 readinessProbe: httpGet: path: / port: 8081 volumeMounts: - name: hubble-ui-nginx-conf mountPath: /etc/nginx/conf.d/default.conf subPath: nginx.conf - name: tmp-dir mountPath: /tmp - name: backend image: mcr.microsoft.com/oss/cilium/hubble-ui-backend:v0.12.2 imagePullPolicy: Always env: - name: EVENTS_SERVER_PORT value: \u0026#34;8090\u0026#34; - name: FLOWS_API_ADDR value: \u0026#34;hubble-relay:443\u0026#34; - name: TLS_TO_RELAY_ENABLED value: \u0026#34;true\u0026#34; - name: TLS_RELAY_SERVER_NAME value: ui.hubble-relay.cilium.io - name: TLS_RELAY_CA_CERT_FILES value: /var/lib/hubble-ui/certs/hubble-relay-ca.crt - name: TLS_RELAY_CLIENT_CERT_FILE value: /var/lib/hubble-ui/certs/client.crt - name: TLS_RELAY_CLIENT_KEY_FILE value: /var/lib/hubble-ui/certs/client.key livenessProbe: httpGet: path: /healthz port: 8090 readinessProbe: httpGet: path: /healthz port: 8090 ports: - name: grpc containerPort: 8090 volumeMounts: - name: hubble-ui-client-certs mountPath: /var/lib/hubble-ui/certs readOnly: true volumes: - name: hubble-ui-nginx-conf configMap: name: hubble-ui-nginx - name: tmp-dir emptyDir: {} - name: hubble-ui-client-certs projected: defaultMode: 0400 sources: - secret: name: hubble-relay-client-certs items: - key: tls.crt path: client.crt - key: tls.key path: client.key - key: ca.crt path: hubble-relay-ca.crt --- apiVersion: v1 kind: Service metadata: name: hubble-ui namespace: kube-system labels: k8s-app: hubble-ui app.kubernetes.io/name: hubble-ui app.kubernetes.io/part-of: retina spec: type: ClusterIP selector: k8s-app: hubble-ui ports: - name: http port: 80 targetPort: 8081 YAML echo echo \u0026#34;==\u0026gt; Hubble UI deployed. To access it:\u0026#34; echo \u0026#34; kubectl -n kube-system port-forward svc/hubble-ui 12000:80\u0026#34; echo \u0026#34; then open: http://localhost:12000/\u0026#34; echo echo \u0026#34;DONE.\u0026#34; First, retrieve cluster credentials.\naz aks get-credentials \\ --resource-group rg-aks-acns-cilium-se \\ --name aks-acns-cilium-se Add the Helm repository and install the DaemonSet.\nhelm repo add inspektor-gadget https://inspektor-gadget.github.io/charts helm repo update helm install ig inspektor-gadget/gadget \\ --namespace gadget \\ --create-namespace After a short moment, verify that the pods are running on all nodes.\nkubectl get pods -n gadget At this point, kernel probes are active across the cluster. Nothing has changed in your applications. No sidecars were injected. No code was modified. Yet the cluster is now capable of exposing runtime behavior that previously required node access.\nTo query this data, install the kubectl plugin via Krew.\nkubectl krew install gadget You can confirm the installation with:\nkubectl gadget version The cluster is now ready for kernel-level observation.\nSeeing processes instead of containers One of the simplest but most powerful demonstrations is tracing process execution.\nRun:\nkubectl gadget run trace_exec Then start a test workload in another terminal.\nkubectl run demo --image=busybox -- sleep 3600 kubectl exec -it demo -- sh Inside the shell, run a simple command like ls. The trace will immediately display the process execution, including the namespace, pod, and container details. Many engineers realize what Inspektor Gadget reveals at this moment. Typically, Kubernetes hides processes behind container boundaries, but with kernel tracing, you can see exactly what runs within those boundaries. Unexpected shells, debugging tools, package installers, or custom binaries become instantly visible.\nThis level of clarity is especially transformative for incident response.\nUnderstanding runtime behavior through DNS and TCP Most production issues in distributed systems eventually involve networking. But networking problems often start with name resolution. A service might resolve a public endpoint instead of a private one. A library might call an undocumented telemetry endpoint. A compromised container might attempt to communicate with a command-and-control server.\nTo observe DNS activity across the cluster:\nkubectl gadget run trace_dns Similarly, TCP connections can be monitored with:\nkubectl gadget run trace_tcp These traces show which workloads resolve specific domains and the connections they make. Since the data originates from the kernel, it accurately represents real runtime behavior rather than just the declared configuration.\nIn AKS environments with private endpoints, outbound filtering, and strict egress controls, this visibility enables platform teams to verify that traffic flows align with the architectural design.\nWhere Cilium Hubble fits into the picture Many AKS clusters already utilize Cilium and Hubble for network observability. Hubble offers valuable insights into network traffic between pods and services, including policy verdicts and, in some cases, application-layer information.\nHowever, Hubble operates at the network layer, whereas Inspektor Gadget operates at the kernel layer.\nHubble can identify when a pod is connected to an external IP address. Inspektor Gadget determines which process opened the socket. Hubble indicates whether the traffic was allowed or denied, while Inspektor Gadget shows the binary and command responsible for generating the traffic.\nIn practice, the two tools work together. Hubble determines whether the network policy functioned correctly, while Inspektor Gadget evaluates the workload\u0026rsquo;s behavior.\nIn zero-trust architectures, both perspectives are necessary.\nMore information about Hubble can be found at:\nhttps://docs.cilium.io/en/stable/gettingstarted/hubble/\nA real production scenario Consider a situation that many platform teams eventually face.\nA firewall administrator observes outbound traffic from an AKS subnet to public IP addresses that are not listed in the approved dependency list. The applications seem to be functioning normally. There are no clear errors in the logs, metrics are normal, and no security alerts have been triggered.\nThe platform team receives the question: which workload is responsible for this traffic?\nInstead of searching through logs or reviewing application code, the team begins with runtime observation.\nCreating a realistic workload in the payments namespace\nStart by creating the namespace\nkubectl create namespace payments Now deploy a small worker pod that continuously performs DNS lookups and outbound HTTPS calls. This simulates the behavior of a service reaching external dependencies.\ncat \u0026lt;\u0026lt;\u0026#39;EOF\u0026#39; | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: payments-worker namespace: payments labels: app: payments-worker spec: restartPolicy: Always containers: - name: worker image: wbitt/network-multitool command: [\u0026#34;/bin/sh\u0026#34;,\u0026#34;-c\u0026#34;] args: - | echo \u0026#34;Starting payments worker simulation...\u0026#34; while true; do # DNS lookups nslookup example.com \u0026gt;/dev/null 2\u0026gt;\u0026amp;1 || true nslookup kubernetes.default.svc.cluster.local \u0026gt;/dev/null 2\u0026gt;\u0026amp;1 || true # Outbound HTTPS call curl -s https://example.com \u0026gt;/dev/null 2\u0026gt;\u0026amp;1 || true # Simulate periodic command execution /bin/sh -c \u0026#34;date \u0026gt; /tmp/heartbeat\u0026#34; sleep 10 done EOF Observing DNS activity\nIn one terminal, start the DNS trace.\nkubectl gadget run trace_dns -n payments Within seconds, repeated queries appear for an unfamiliar domain.\nObserving outbound connections\nIn another terminal, observe TCP activity.\nkubectl gadget run trace_tcp:latest -n payments You will see outbound connections to public IP addresses on port 443. In a real incident, this is where platform teams start asking important questions:\nWhy is this workload talking to the internet?\nIs this expected behavior?\nDoes it match the architecture?\nObserving process execution\nFinally, observe process activity.\nkubectl gadget run trace_exec:latest -n payments The trace reveals a shell process executing a command periodically inside a container that was supposed to run only a Java application.\nAt this point, the investigation moves from speculation to evidence. The workload image can be inspected, the deployment history reviewed, and containment actions taken with confidence.\nThis is the difference between observing symptoms and observing reality.\nWhen finished, clean up:\nkubectl delete namespace payments Cleanup script for removing the AKS Cluster\n#!/usr/bin/env bash set -euo pipefail # ----------------------------------------------------------------------------- # Cleanup for the AKS + ACNS + Cilium + Hubble UI deployment script # # What it does: # 1) Deletes the optional Hubble UI resources created in kube-system (if present) # 2) Deletes the AKS cluster (optional; default true) # 3) Deletes the resource group (optional; default true) ✅ recommended # 4) Optionally removes the kubeconfig entry/context for the cluster # # Usage: # chmod +x cleanup-aks-acns.sh # ./cleanup-aks-acns.sh # # Optional flags: # ./cleanup-aks-acns.sh --force # ./cleanup-aks-acns.sh --no-wait # DELETE_RG=false ./cleanup-aks-acns.sh # DELETE_AKS=false ./cleanup-aks-acns.sh # REMOVE_KUBECONFIG=true ./cleanup-aks-acns.sh # # Docs: # - az group delete: https://learn.microsoft.com/cli/azure/group#az-group-delete # - az aks delete: https://learn.microsoft.com/cli/azure/aks#az-aks-delete # ----------------------------------------------------------------------------- # Must match your deploy defaults (override via env vars) LOCATION=\u0026#34;${LOCATION:-swedencentral}\u0026#34; RESOURCE_GROUP=\u0026#34;${RESOURCE_GROUP:-rg-aks-acns-cilium-se}\u0026#34; CLUSTER_NAME=\u0026#34;${CLUSTER_NAME:-aks-acns-cilium-se}\u0026#34; KUBECONFIG_PATH=\u0026#34;${KUBECONFIG_PATH:-${HOME}/.kube/config}\u0026#34; # Behavior toggles DELETE_AKS=\u0026#34;${DELETE_AKS:-true}\u0026#34; # delete cluster explicitly (not needed if deleting RG, but useful if DELETE_RG=false) DELETE_RG=\u0026#34;${DELETE_RG:-true}\u0026#34; # recommended: deletes everything created by deployment DELETE_HUBBLE_UI=\u0026#34;${DELETE_HUBBLE_UI:-true}\u0026#34; REMOVE_KUBECONFIG=\u0026#34;${REMOVE_KUBECONFIG:-false}\u0026#34; # remove kubeconfig context/user/cluster entries FORCE=false NO_WAIT=false for arg in \u0026#34;$@\u0026#34;; do case \u0026#34;$arg\u0026#34; in --force) FORCE=true ;; --no-wait) NO_WAIT=true ;; esac done require() { command -v \u0026#34;$1\u0026#34; \u0026gt;/dev/null 2\u0026gt;\u0026amp;1 || { echo \u0026#34;Missing dependency: $1\u0026#34; \u0026gt;\u0026amp;2; exit 1; }; } require az echo \u0026#34;========================================\u0026#34; echo \u0026#34; Cleanup: AKS + ACNS + Cilium + Hubble UI\u0026#34; echo \u0026#34; Location : ${LOCATION}\u0026#34; echo \u0026#34; Resource Group: ${RESOURCE_GROUP}\u0026#34; echo \u0026#34; Cluster : ${CLUSTER_NAME}\u0026#34; echo \u0026#34;========================================\u0026#34; echo # Azure login az account show \u0026gt;/dev/null 2\u0026gt;\u0026amp;1 || az login \u0026gt;/dev/null # Confirm deletion (unless forced) if [[ \u0026#34;${FORCE}\u0026#34; != \u0026#34;true\u0026#34; ]]; then echo \u0026#34;This cleanup may delete:\u0026#34; [[ \u0026#34;${DELETE_HUBBLE_UI}\u0026#34; == \u0026#34;true\u0026#34; ]] \u0026amp;\u0026amp; echo \u0026#34; - Hubble UI resources in kube-system (Deployment/Service/ConfigMap/RBAC)\u0026#34; [[ \u0026#34;${DELETE_AKS}\u0026#34; == \u0026#34;true\u0026#34; ]] \u0026amp;\u0026amp; echo \u0026#34; - AKS cluster: ${CLUSTER_NAME} (if it exists)\u0026#34; [[ \u0026#34;${DELETE_RG}\u0026#34; == \u0026#34;true\u0026#34; ]] \u0026amp;\u0026amp; echo \u0026#34; - Resource group: ${RESOURCE_GROUP} (and ALL contained resources) [recommended]\u0026#34; [[ \u0026#34;${REMOVE_KUBECONFIG}\u0026#34; == \u0026#34;true\u0026#34; ]] \u0026amp;\u0026amp; echo \u0026#34; - kubeconfig entries for ${CLUSTER_NAME} in ${KUBECONFIG_PATH}\u0026#34; echo read -r -p \u0026#34;Type the resource group name to confirm: \u0026#34; CONFIRM if [[ \u0026#34;${CONFIRM}\u0026#34; != \u0026#34;${RESOURCE_GROUP}\u0026#34; ]]; then echo \u0026#34;Confirmation failed. Aborting.\u0026#34; exit 1 fi fi # If we plan to touch Kubernetes objects, try to get credentials (best-effort) if [[ \u0026#34;${DELETE_HUBBLE_UI}\u0026#34; == \u0026#34;true\u0026#34; ]]; then if command -v kubectl \u0026gt;/dev/null 2\u0026gt;\u0026amp;1; then echo \u0026#34;==\u0026gt; Attempting to delete Hubble UI resources from kube-system (best-effort)\u0026#34; # best-effort kubeconfig usage export KUBECONFIG=\u0026#34;${KUBECONFIG_PATH}\u0026#34; # Try to get credentials (won\u0026#39;t fail the script if cluster is already gone) az aks get-credentials -g \u0026#34;${RESOURCE_GROUP}\u0026#34; -n \u0026#34;${CLUSTER_NAME}\u0026#34; --file \u0026#34;${KUBECONFIG_PATH}\u0026#34; --overwrite-existing -o none 2\u0026gt;/dev/null || true # Delete resources created by the deploy script (ignore if not found) kubectl -n kube-system delete svc hubble-ui --ignore-not-found || true kubectl -n kube-system delete deploy hubble-ui --ignore-not-found || true kubectl -n kube-system delete cm hubble-ui-nginx --ignore-not-found || true kubectl delete clusterrolebinding hubble-ui --ignore-not-found || true kubectl delete clusterrole hubble-ui --ignore-not-found || true kubectl -n kube-system delete sa hubble-ui --ignore-not-found || true else echo \u0026#34;==\u0026gt; kubectl not found; skipping Hubble UI Kubernetes resource cleanup.\u0026#34; echo \u0026#34; (If you delete the resource group, these will be removed anyway.)\u0026#34; fi fi # Delete AKS cluster (optional) if [[ \u0026#34;${DELETE_AKS}\u0026#34; == \u0026#34;true\u0026#34; \u0026amp;\u0026amp; \u0026#34;${DELETE_RG}\u0026#34; != \u0026#34;true\u0026#34; ]]; then echo \u0026#34;==\u0026gt; Deleting AKS cluster \u0026#39;${CLUSTER_NAME}\u0026#39; (resource group \u0026#39;${RESOURCE_GROUP}\u0026#39;)\u0026#34; if [[ \u0026#34;${NO_WAIT}\u0026#34; == \u0026#34;true\u0026#34; ]]; then az aks delete -g \u0026#34;${RESOURCE_GROUP}\u0026#34; -n \u0026#34;${CLUSTER_NAME}\u0026#34; --yes --no-wait || true echo \u0026#34; AKS deletion started (no-wait).\u0026#34; else az aks delete -g \u0026#34;${RESOURCE_GROUP}\u0026#34; -n \u0026#34;${CLUSTER_NAME}\u0026#34; --yes || true echo \u0026#34; AKS deletion completed.\u0026#34; fi else if [[ \u0026#34;${DELETE_RG}\u0026#34; == \u0026#34;true\u0026#34; ]]; then echo \u0026#34;==\u0026gt; Skipping explicit AKS delete because resource group deletion will remove it.\u0026#34; fi fi # Delete resource group (recommended) if [[ \u0026#34;${DELETE_RG}\u0026#34; == \u0026#34;true\u0026#34; ]]; then echo \u0026#34;==\u0026gt; Deleting resource group \u0026#39;${RESOURCE_GROUP}\u0026#39; (this removes ALL resources in it)\u0026#34; if [[ \u0026#34;${NO_WAIT}\u0026#34; == \u0026#34;true\u0026#34; ]]; then az group delete --name \u0026#34;${RESOURCE_GROUP}\u0026#34; --yes --no-wait echo \u0026#34; Resource group deletion started (no-wait).\u0026#34; else az group delete --name \u0026#34;${RESOURCE_GROUP}\u0026#34; --yes echo \u0026#34; Resource group deletion completed.\u0026#34; fi fi # Optionally remove kubeconfig entries if [[ \u0026#34;${REMOVE_KUBECONFIG}\u0026#34; == \u0026#34;true\u0026#34; ]]; then if command -v kubectl \u0026gt;/dev/null 2\u0026gt;\u0026amp;1; then echo \u0026#34;==\u0026gt; Removing kubeconfig entries for \u0026#39;${CLUSTER_NAME}\u0026#39; from ${KUBECONFIG_PATH}\u0026#34; export KUBECONFIG=\u0026#34;${KUBECONFIG_PATH}\u0026#34; # AKS usually uses contexts like \u0026#34;\u0026lt;cluster-name\u0026gt;\u0026#34; or \u0026#34;\u0026lt;cluster-name\u0026gt;-admin\u0026#34; # We\u0026#39;ll remove any context that contains the cluster name. mapfile -t contexts \u0026lt; \u0026lt;(kubectl config get-contexts -o name 2\u0026gt;/dev/null | grep -F \u0026#34;${CLUSTER_NAME}\u0026#34; || true) for ctx in \u0026#34;${contexts[@]}\u0026#34;; do kubectl config delete-context \u0026#34;${ctx}\u0026#34; \u0026gt;/dev/null 2\u0026gt;\u0026amp;1 || true done # Try to remove cluster entry and user entry that commonly match the cluster name # These names vary, so we do best-effort matching. mapfile -t clusters \u0026lt; \u0026lt;(kubectl config get-clusters 2\u0026gt;/dev/null | tail -n +2 | grep -F \u0026#34;${CLUSTER_NAME}\u0026#34; || true) for c in \u0026#34;${clusters[@]}\u0026#34;; do kubectl config delete-cluster \u0026#34;${c}\u0026#34; \u0026gt;/dev/null 2\u0026gt;\u0026amp;1 || true done mapfile -t users \u0026lt; \u0026lt;(kubectl config get-users 2\u0026gt;/dev/null | tail -n +2 | grep -F \u0026#34;${CLUSTER_NAME}\u0026#34; || true) for u in \u0026#34;${users[@]}\u0026#34;; do kubectl config unset \u0026#34;users.${u}\u0026#34; \u0026gt;/dev/null 2\u0026gt;\u0026amp;1 || true done echo \u0026#34; kubeconfig cleanup complete.\u0026#34; else echo \u0026#34;==\u0026gt; kubectl not found; skipping kubeconfig cleanup.\u0026#34; fi fi echo echo \u0026#34;Cleanup finished.\u0026#34; Operational and governance considerations Kernel-level observability is a powerful capability and must be handled with care. Access to Inspektor Gadget should be restricted using Kubernetes RBAC, and its deployment should adhere to operational procedures similar to those for other privileged diagnostic tools.\nIn most organizations, this responsibility typically falls to platform engineering, security operations, or incident response teams instead of general developers.\nFrom a compliance perspective, the value is considerable. Inspektor Gadget offers proof of runtime behavior, which can aid forensic investigations, fulfill audit requirements, and support zero-trust validation.\nIt enables teams to show not just policy configurations but also that workloads operate within expected limits.\nConclusion: closing the gap between design and execution Cloud architecture has grown highly complex. We develop identity-focused access systems, segmented networks, encrypted communication channels, and policy-based deployment pipelines. Azure Kubernetes Service offers managed infrastructure that ensures these designs are scalable and dependable.\nHowever, architecture reflects intent; security, reliability, and compliance ultimately rely on execution.\nExecution happens in the kernel.\nInspektor Gadget offers more than just another observability tool for your platform. It reestablishes visibility into the layer where containers transition into processes, where policies turn into packet decisions, and where abstractions fade away.\nFor platform teams managing AKS at scale, this alters the approach to incident investigation, validation of zero-trust assumptions, and demonstration of operational control.\nWhen integrated with tools such as Azure Monitor, Microsoft Defender for Containers, and Cilium Hubble, kernel-level visibility completes the observability stack. Metrics reveal performance, logs clarify behavior, network flows illustrate communication, and the kernel provides the underlying reality.\nOnce you’ve viewed your cluster from that angle, it becomes hard to depend solely on abstractions.\nUltimately, Kubernetes may describe the system you aimed to create, but the kernel reveals the system that is actually running.\n","permalink":"https://wolkwacht.nl/posts/when-kubernetes-lies/","summary":"\u003ch2 id=\"when-kubernetes-lies-seeing-the-truth-inside-aks-with-inspektor-gadget\"\u003eWhen Kubernetes Lies: Seeing the Truth Inside AKS with Inspektor Gadget\u003c/h2\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*wEP6tWQfmhbapBB1BR1H4w.png\"\u003e\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eCloud platforms are built on abstractions.\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eWe tell developers that services talk to services. We tell security teams that network policies enforce isolation. We tell architects that identity is the new perimeter and that zero trust ensures nothing moves unless explicitly allowed. In Azure Kubernetes Service, these abstractions become even more powerful. Managed control planes, private clusters, Cilium networking, workload identity, Azure Firewall, private endpoints, and policy enforcement create the feeling of a highly controlled, highly predictable platform.\u003c/p\u003e","title":"When Kubernetes Lies:"},{"content":"Cloud Security Is a System, Not a Stack Part 6 of the Cloud Security series\nThis article concludes a series about cloud security in existing Azure environments. In the earlier parts, we explored how meaningful security improvements often start small, why identity defines the real perimeter, how logging without intent creates blindness, how architecture determines blast radius, and why security quietly fails when ownership is unclear. Individually, these topics are familiar. Together, they reveal something more important: cloud security does not work as a collection of controls. It only works as a system.\nOne of the most persistent myths in cloud security is that maturity comes from accumulation. More tools, more policies, more alerts, more diagrams. Over time, many Azure environments become dense with security components, yet remain fragile. When incidents happen, they are not stopped by the number of controls in place, but by whether those controls reinforce each other.\nA system operates differently than a stack. In a stack, each layer functions independently, with failures in one layer often compensated by another. Conversely, in a system, components are interconnected and interdependent. Decisions about identity affect detection, while architecture influences the usefulness of logging. Governance plays a role in whether controls are updated or allowed to weaken over time. Changes in one element lead to responses from others, and this interconnected response fosters resilience.\nLooking back at the series, a clear pattern emerges: identity without oversight introduces unseen risk. Monitoring without accountability generates unnecessary noise. An architecture lacking identity discipline magnifies errors. Governance without operational input turns into mere spectacle. Each of these issues is subtle on its own. They are quiet. Gradual. Easy to ignore until they unexpectedly converge.\nThis is why cloud security incidents often seem unexpected in retrospect. It\u0026rsquo;s not because the signals weren\u0026rsquo;t present, but because they were disconnected. The platform was providing clues, but no one was interpreting the entire story.\nSecurity as a system begins with recognizing that no single control is enough. MFA lowers risk but doesn’t eliminate it entirely. Segmentation confines the impact but cannot prevent breaches. Logging uncovers behavior but doesn’t dictate responses. Governance sets expectations but can’t enforce intentions. Each component is incomplete, and its true value arises only when combined.\nWhat sets more mature environments apart is not perfection but coherence. Access decisions are consistent with architectural boundaries. Alerts are valuable because they mirror familiar patterns. Ownership is sufficiently clear so that responses occur without chaos from escalation. Exceptions are noticeable and uncomfortable, not hidden and permanent. The platform acts in expected ways, making surprises infrequent.\nThis coherence isn\u0026rsquo;t accidental; it\u0026rsquo;s built through consistent, sometimes unglamorous decisions such as prioritizing reliability over cleverness, narrowing scope instead of adding complex controls, and reexamining old assumptions rather than layering new abstractions. These choices are rarely highlighted in reference architectures but are crucial in shaping actual security results.\nAnother important realization is that cloud security systems are never static. Azure environments regularly evolve, with teams and workloads shifting, and threats adapting. A static system will ultimately fail, regardless of its initial strength. This highlights why security should be viewed as an ongoing operating model rather than just a fixed design. The capacity to observe, ask questions, and make adjustments is more important than following a single best practice.\nIn this perspective, cloud security shifts from merely enforcing rules to shaping user behavior. Platforms guide workflows, with clear boundaries promoting safer designs. Visible ownership fosters accountability, while meaningful signals encourage timely response. Over time, the system nudges teams toward improved decisions without requiring ongoing enforcement.\nThis viewpoint also reshapes how we see the roles of the cloud architect and security professional. Their task isn\u0026rsquo;t to eradicate risk, which is impossible, but to create systems where risk is transparent, controlled, and manageable. It involves replacing implicit trust with explicit decision-making and transforming security from a sporadic topic into a constant aspect of the platform.\nThis blog series aims to outline a cloud security mindset that remains practical in real-world conditions. It recognizes constraints, legacy systems, and human factors without succumbing to them. Instead of viewing security as an add-on, it considers it an intrinsic outcome of the platform\u0026rsquo;s operational approach.\nIf there is a single takeaway, it is this: cloud security improves fastest when you stop asking “what control are we missing?” and start asking “how does this system behave when something goes wrong?” The answers to that question are rarely found in documentation. They are found in access paths, alert responses, architectural boundaries, and ownership gaps.\nCloud security is not a destination you reach. It is a system you continuously shape.\nAnd that work is never finished, but it can become sustainable.\n","permalink":"https://wolkwacht.nl/posts/cloud-security-is-a-system-not-a-stack/","summary":"\u003ch2 id=\"cloud-security-is-a-system-not-astack\"\u003eCloud Security Is a System, Not a Stack\u003c/h2\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*ypjZieYT8-H9gfhbJdlP3A.png\"\u003e\u003c/p\u003e\n\u003cp\u003e\u003cem\u003ePart 6 of the Cloud Security series\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eThis article concludes a series about cloud security in existing Azure environments. In the earlier parts, we explored how meaningful security improvements often start small, why identity defines the real perimeter, how logging without intent creates blindness, how architecture determines blast radius, and why security quietly fails when ownership is unclear. Individually, these topics are familiar. Together, they reveal something more important: cloud security does not work as a collection of controls. It only works as a system.\u003c/p\u003e","title":"Cloud Security Is a System, Not a Stack"},{"content":"Security Fails Quietly When Nobody Owns It Part 5 of the Cloud Security series\nThis article is the fifth part in a series about cloud security within existing Azure environments. Previously, we discussed how foundational changes build momentum, why identity forms your true perimeter, how logging without purpose results in blind spots, and how architecture influences the scope of failure. This section moves away from technical details and addresses a more challenging topic: ownership. Most cloud security issues arise not from absent controls but from unclear responsibilities.\nIn many organizations, security exists everywhere and nowhere at the same time.\nSecurity controls exist. Ownership often doesn’t.\nControls are deployed. Policies are written. Tools are licensed. Yet when something goes wrong, the first question is rarely, \u0026ldquo;How did this happen?\u0026rdquo; but rather*, \u0026ldquo;Who was supposed to be watching this?\u0026rdquo;* The silence that follows that question is often the most revealing signal of all.\nCloud platforms dissolve traditional boundaries, meaning infrastructure teams no longer have sole ownership. Instead, application teams deploy directly into production, while security teams provide advice, monitor, and escalate issues without controlling the entire process. Although this shift offers benefits, it challenges older governance models that depended on centralized approval and enforcement. If these models are not replaced, security responsibilities become fragmented, ultimately vanishing.\nA common pattern in Azure environments is that security often becomes a secondary concern, emerging only in response to events. Identity reviews are conducted when auditors request them, logging is modified after incidents, and network rules are tightened when issues arise. None of these actions are malicious; they are purely reactive. Without designated ownership, security tends to be proactive only when compelled.\nOwnership is about accountability, not control. It requires someone to be responsible for the results, beyond just setting up the configuration. When a Conditional Access policy is established, who is accountable for maintaining its effectiveness? When a subscription is launched, who oversees its security status to prevent unnoticed decline? When alerts go off, who assesses their significance? Without definitive answers, security risks becoming a matter of routine procedures instead of active, operational management.\nThis is where many organizations struggle because cloud ownership spans roles. Platform teams own shared foundations but not individual workloads. Application teams own their services but not the underlying identity or network layers. Security teams understand risk but often lack the authority to enforce change. The gaps between these roles are where incidents are born.\nEffective cloud security governance does not re-centralize everything. It clarifies expectations. It defines which decisions are local, which are shared, and which are non-negotiable.\nGovernance explains expectations — it doesn’t replace responsibility\nIt makes security visible without creating unnecessary bureaucracy. Most importantly, it connects responsibility with the authority to act. Having security ownership without the tools to influence the results leads to frustration and failure.\nCulture influences security more than many strategies recognize. When security is seen as an external limit, teams tend to find exceptions, workarounds, and delays. Conversely, when security is integrated into their professional skills, teams adopt it internally. The key difference is seldom the tools used; instead, it depends on how leadership discusses security during normal times, outside of incidents.\nA key sign of security maturity is how organizations manage exceptions. In less mature settings, exceptions tend to build up unnoticed. Conversely, in more mature environments, exceptions are transparent, limited by time, and can be somewhat uncomfortable. This discomfort is deliberate, encouraging organizations to address underlying issues rather than accept ongoing risk.\nGovernance plays a key role in ensuring security improvements are maintained. Without feedback loops, these enhancements tend to fade over time, permissions may revert, alerts can be silenced, and architectural boundaries weaken. Achieving sustainable security depends on having mechanisms that regularly prompt reflection, not just audits for compliance, but also occasions when teams evaluate if the platform continues to function as intended.\nThis is why adopting security as an operating model is important. When security is integrated into the deployment, review, and evolution of environments, it becomes more adaptable to change. Conversely, when security is handled as a one-time project, it quickly loses relevance once focus moves elsewhere. Cloud environments evolve rapidly, making static security approaches unsustainable.\nThis operating model acknowledges that security work is an ongoing process. While this may seem unsatisfactory to organizations that prefer completion, in the cloud, stability is achieved through continuous adjustments rather than finality. Teams embracing this mindset shift from striving for perfection to developing adaptable systems.\nThis part of the series integrates the earlier themes. Identity, logging, and architecture only function effectively when individuals feel accountable for their continuous well-being. Without a sense of ownership, these elements become static snapshots frozen in time. When ownership is present, they develop and adapt alongside the platform.\nIn the concluding part of this series, we\u0026rsquo;ll take a step back to view the bigger picture. We\u0026rsquo;ll explore how the elements of foundations, identity, detection, architecture, and governance interconnect to form a unified security posture. This isn\u0026rsquo;t a simple checklist or reference model, but a mindset for cloud security that remains effective under real-world challenges.\n","permalink":"https://wolkwacht.nl/posts/security-fails-quietly-when-nobody-owns-it/","summary":"\u003ch2 id=\"security-fails-quietly-when-nobody-ownsit\"\u003eSecurity Fails Quietly When Nobody Owns It\u003c/h2\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*D7DDc6D_t7F-WaZHwne_ag.png\"\u003e\u003c/p\u003e\n\u003cp\u003e\u003cem\u003ePart 5 of the Cloud Security series\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eThis article is the fifth part in a series about cloud security within existing Azure environments. Previously, we discussed how foundational changes build momentum, why identity forms your true perimeter, how logging without purpose results in blind spots, and how architecture influences the scope of failure. This section moves away from technical details and addresses a more challenging topic: ownership. Most cloud security issues arise not from absent controls but from unclear responsibilities.\u003c/p\u003e","title":"Security Fails Quietly When Nobody Owns It"},{"content":"Architecture Decides the Blast Radius Part 4 of the Cloud Security series\nThis article is the fourth part of a series on cloud security in real-world Azure environments. Earlier, we explored how foundational improvements can create momentum, highlighted why identity is the most vital security control in the cloud, and explained how aimless logging can lead to visibility without comprehension. In this part, we move from signals and access to a more structural topic: architecture. Because some security controls will inevitably fail, the platform\u0026rsquo;s architecture determines whether such failures are contained or lead to disaster.\nA harsh truth in cloud security is that breaches are sometimes unavoidable. Credentials can be stolen through phishing, tokens may be leaked, and misconfigurations may go unnoticed during reviews. Often, the difference between a minor issue and a major breach is not the initial mistake itself, but rather the extent to which the environment enables the error to have an impact.\nArchitecture is commonly considered in terms of scalability, availability, and cost. Security is acknowledged but often viewed as an additional layer. In reality, architecture acts as a security control by establishing trust boundaries, restricting movement, and influencing failure propagation. A flat architecture allows security failures to spread quickly, whereas a well-designed, intentional architecture contains failures.\nMany current Azure configurations reflect organizational history rather than security needs. Subscriptions were created to give teams independence, shared networks to simplify management, and workloads were colocated for convenience. Although not necessarily wrong, these choices can increase risks over time, creating an environment where interconnectedness grows beyond full understanding.\nThis is where the blast radius model proves helpful. Instead of questioning if something is secure, it\u0026rsquo;s more useful to consider what happens if it fails.\nWhen everything trusts everything, failure travels fast.\nIf a workload is compromised, what else can it see? If an identity is compromised, how far can it be exploited? If a deployment pipeline is misused, which environments does it touch? These questions cut through abstract security debates and force architectural clarity.\nNetwork segmentation is often the first topic that comes up in this context, and it is frequently misunderstood. Segmentation is not about drawing as many lines as possible. It is about making trust explicit.\nGood architecture limits how far mistakes can travel\nIn flat networks, everything implicitly trusts everything else. In segmented designs, trust must be intentional. That shift alone dramatically reduces lateral movement, even when credentials are compromised.\nAzure makes segmentation deceptively easy to configure and deceptively hard to design well. Virtual networks, subnets, private endpoints, and service endpoints are powerful tools, but without a clear model, they often recreate the same flatness at a different layer. True segmentation aligns with workloads and risk, not with technical convenience. Production should not implicitly trust non-production. Shared services should be tightly scoped. Management planes should be isolated from workloads.\nWorkload boundaries matter just as much as network ones. When multiple applications share the same infrastructure, security failures blend together. Logs lose context. Alerts lose ownership. Responsibility becomes ambiguous. Separating workloads by subscription, environment, or lifecycle is not just a governance practice; it is a security one. It creates natural containment zones that isolate and understand incidents.\nThis is also where identity and architecture intersect. An identity with broad permissions in a flat environment is dangerous. The same identity in a segmented environment remains risky but far less destructive. Architecture compensates for human and process failures by limiting their impact. That is not a sign of mistrust; it is an acknowledgment of reality.\nOne of the most overlooked architectural security decisions is how management access is handled. Many environments expose management endpoints broadly because “only admins know about them.” That assumption rarely holds over time. Treating management planes as high-value targets and isolating them accordingly changes the platform\u0026rsquo;s overall risk profile. When management access is constrained, attackers are forced into noisier, more detectable paths.\nArchitectural security also influences response. In well-segmented environments, incidents are easier to contain. A compromised workload can be isolated without taking unrelated systems offline. Teams can respond surgically instead of resorting to blanket shutdowns. This reduces business impact and increases confidence in security controls, which in turn makes teams more willing to enforce them.\nWhat makes architecture challenging is that it is difficult to retrofit perfectly. Existing environments carry legacy decisions that cannot be undone overnight. But architecture does not need to be perfect to be effective. Even incremental improvements, introducing clearer boundaries, reducing unnecessary trust, and separating critical paths can dramatically reduce blast radius. The goal is not to redesign everything, but to stop making it worse and start making it intentional.\nArchitecture also sends a cultural signal. When teams see that environments are deliberately separated, that access paths are constrained, and that not everything can talk to everything else, security becomes visible in a constructive way. It shapes how people design new workloads. It nudges behavior without constant enforcement.\nThis part of the series ties together many of the earlier themes. Identity controls define who can act. Logging shows when something unusual happens. Architecture decides how far that action or anomaly can spread. When these elements align, the platform becomes resilient, not because it is impenetrable, but because it absorbs failure without collapsing.\nIn the next part of this series, we move away from controls and designs and focus on something more human: governance, ownership, and culture. Because even the best architecture and tooling fail when nobody feels responsible for security outcomes. Cloud security is sustainable only when it is embedded in how teams work, not just in how platforms are built.\n","permalink":"https://wolkwacht.nl/posts/architecture-decides-the-blast-radius/","summary":"\u003ch2 id=\"architecture-decides-the-blastradius\"\u003eArchitecture Decides the Blast Radius\u003c/h2\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*YmYE3Y54p5hTQIeAcJilvg.png\"\u003e\u003c/p\u003e\n\u003cp\u003e\u003cem\u003ePart 4 of the Cloud Security series\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eThis article is the fourth part of a series on cloud security in real-world Azure environments. Earlier, we explored how foundational improvements can create momentum, highlighted why identity is the most vital security control in the cloud, and explained how aimless logging can lead to visibility without comprehension. In this part, we move from signals and access to a more structural topic: architecture. Because some security controls will inevitably fail, the platform\u0026rsquo;s architecture determines whether such failures are contained or lead to disaster.\u003c/p\u003e","title":"Architecture Decides the Blast Radius"},{"content":"You Are Logging Everything — And Seeing Almost Nothing Part 3 of the Cloud Security series\nThis article is the third part in a series on cloud security, based on real Azure environments rather than idealized models. The first part emphasized establishing momentum through basic improvements. The second highlighted how identity is the most underestimated yet powerful security control in the cloud. This section shifts focus to telemetry, which most platforms already possess in large quantities: logs, metrics, alerts, and signals are ubiquitous in Azure. Despite this abundance, many organizations remain largely unaware of significant security risks.\nNearly every Azure environment generates more logs than operators can reasonably analyze. Diagnostic settings are enabled, activity logs are sent to Log Analytics, Defender generates alerts, and applications emit metrics. While visibility appears assured on paper, in reality, little of it translates into genuine awareness. When incidents occur, teams often realize that the necessary information was available but never linked to actionable responses.\nThis is the point at which cloud security subtly begins to fail.\nLogging is often viewed as merely a compliance task rather than a vital operational skill. Data collection happens because someone mandated it, not because there\u0026rsquo;s a clear purpose. This can lead to a false sense of security over time. Although the platform appears monitored, actual oversight is lacking. Security often becomes reactive, investigated only after an incident occurs, rather than a proactive measure that prevents or manages issues.\nOne reason is that cloud observability grows faster than human attention can keep up. While Azure simplifies enabling monitoring, it complicates prioritizing what truly matters. Without clear intent, logging becomes noise, and alerts are triggered without context. Dashboards may be left unmanaged, leading teams to lose trust in the signals altogether. When everything seems important, nothing stands out.\nWhen entering an existing environment, it’s common to see security telemetry reflect the organizational hierarchy. Logs are stored centrally, but accountability is unclear. Operations teams expect security to monitor, while security teams rely on operations to respond. Development teams are seldom engaged. Alerts often circulate between teams or go unnoticed because responding isn\u0026rsquo;t explicitly assigned to anyone. This isn’t a failure of tools; it’s a failure of clarity.\nEffective detection begins not with SIEM rules or threat intelligence feeds, but with defining what should never occur in your environment. This involves concrete, context-specific criteria, such as a privileged role assigned outside everyday workflows, a production resource changed from an unusual location, or a service identity accessing unfamiliar data. These are not rare threats; they are deviations from regular activity and are often detectable well before an attacker achieves their goal.\nThe issue is that many platforms don\u0026rsquo;t specify what constitutes \u0026rsquo;normal.\u0026rsquo; Without this baseline, alerts become meaningless. For example, a sign-in failure appears identical whether it\u0026rsquo;s a user mistyping a password or a brute-force attack. Similarly, a role assignment looks the same whether it occurs during a deployment pipeline or a manual emergency update. Context is essential to transforming telemetry into actionable security insights, and it is rarely provided by technology alone.\nAnother challenge is that detection is often implemented as a centralized function within inherently decentralized environments. Azure setups span teams, subscriptions, and workloads, each with its own cadence. A universal alerting strategy cannot accommodate this diversity, resulting in either overly broad alerts that are frequently triggered or overly narrow alerts that overlook issues. In either case, trust diminishes.\nOwnership transforms this dynamic. When alerts are connected explicitly to teams familiar with the workload, response times improve significantly. This isn’t because individuals work harder, but because the signals become meaningful. A development team is much more likely to respond to an alert about their own application\u0026rsquo;s unexpected behavior than to a vague “suspicious activity detected” message lacking context.\nSecurity logging is often based on the mistaken belief that more data leads to better detection. However, too much data can actually slow down responses, as analysts spend more time filtering than acting. Critical signals get lost in the noise. Experienced environments tend to log less but more intentionally, focusing on identity events, privilege changes, control-plane activities, and notable workload deviations. All other data aims to support these key signals rather than compete with them.\nDetection also has an emotional component that is seldom addressed. Frequent alerts lead to fatigue, causing teams to stop responding, not due to indifference, but because they are accustomed to false positives. This creates a vicious cycle in which real incidents are mistaken for noise because noise has become normalized. To break this pattern, restraint is necessary rather than more rules.\nEffective cloud security monitoring is integrated into platform operations, rather than acting as an external safeguard. When teams anticipate visibility into changes, detect unusual activity, and respond with measured responses rather than reactively, behaviors improve. Risky shortcuts lose appeal, and emergency access is handled more cautiously. Security transitions from a potential threat to a confidence-building support.\nThis section is closely linked to identity, which often shows signs of compromise before workloads are affected. Indicators such as privilege escalation, unusual sign-ins, and changes in access patterns are more significant than generic threat alerts. When identity management, logging, and response coordination are integrated, the platform achieves a level of situational awareness that no individual product can deliver.\nNone of these demands a perfect SOC or advanced threat hunting from the start. It needs purpose, determining what merits focus, assigning accountability, and acknowledging that visibility alone, without action, is merely data storage.\nIn the next part of this series, we shift from detection to design. We will examine how architectural choices related to networking, segmentation, and workload boundaries can either enhance or restrict the effect of unavoidable failures. In cloud security, breaches can\u0026rsquo;t always be prevented, but their outcomes are primarily determined by design.\n","permalink":"https://wolkwacht.nl/posts/you-are-logging-everything-and-seeing-almost-nothing/","summary":"\u003ch2 id=\"you-are-logging-everythingand-seeing-almostnothing\"\u003eYou Are Logging Everything — And Seeing Almost Nothing\u003c/h2\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*RVe8zaIeOjIEKx93Gtp7eg.png\"\u003e\u003c/p\u003e\n\u003cp\u003e\u003cem\u003ePart 3 of the Cloud Security series\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eThis article is the third part in a series on cloud security, based on real Azure environments rather than idealized models. The first part emphasized establishing momentum through basic improvements. The second highlighted how identity is the most underestimated yet powerful security control in the cloud. This section shifts focus to telemetry, which most platforms already possess in large quantities: logs, metrics, alerts, and signals are ubiquitous in Azure. Despite this abundance, many organizations remain largely unaware of significant security risks.\u003c/p\u003e","title":"You Are Logging Everything — And Seeing Almost Nothing"},{"content":"Identity Is the Perimeter You Forgot to Guard Part 2 of the Cloud Security series\nCloud security often fails not due to advanced attacks, but because of unnoticed access and unrevoked privileges. This is the second part of a series on cloud security in real-world Azure environments, which evolve over time under pressure and good intentions. The first part discussed how small, fundamental improvements can significantly improve security. In this segment, we focus on identity, as it is the critical factor determining success or failure in cloud security.\nMost Azure environments don\u0026rsquo;t initially prioritize identity in their design. Instead, they develop it gradually. When a subscription is created, someone needs quick access, so a role is assigned, and the system continues without much thought. Over time, new teams, workloads, and exceptions emerge, making identity an unnoticed part of the infrastructure. It functions adequately but silently accumulates risks because no one actively manages it.\nWhen security incidents occur, identity is frequently blamed afterward. Credentials get compromised, tokens are misused, or accounts have excessive permissions. Though these may seem technical, the root problem is usually organizational. Access is often managed as a fixed configuration rather than a temporary, contextual, and revocable resource.\nA common and risky misconception in cloud security is believing that identity management is fully addressed once MFA is activated and roles are set. In reality, identity is a dynamic and ongoing process, not simply secure or insecure. It shifts whenever someone joins, leaves, changes teams, or uses automation. Without active management, this system can become overexposed and vulnerable to breaches.\nWhen entering an existing Azure environment, identity risk often goes unnoticed because there are no obvious issues. There are no broken apps or failing pipelines; everything appears to function normally. This is the core problem. Privileged access tends to stay unnoticed when everything operates smoothly. When no problems arise, it’s easy for users to overlook questions such as why so many users can modify production resources or why service principals are granted broad permissions “just in case.”\nOnce you begin mapping permissions, patterns become evident rapidly. Privileged roles tend to concentrate among a few individuals labeled as “always helpful.” Emergency access accounts are in place but have never actually been tested. Automation identities often outlast the projects they were initially created for. None of these practices seem malicious; they appear to be practical. However, each of these patterns increases the potential impact of a single mistake or security breach.\nThis is why identity is often referred to as the new perimeter, though that phrase is often misunderstood. It\u0026rsquo;s not a buzzword about modern security measures; rather, it emphasizes that in the cloud, access control defines your security boundary. Unlike traditional security, there\u0026rsquo;s no network device that can correct over-permissions. Firewalls can\u0026rsquo;t reverse a role assignment once trust is established. When an identity is trusted, the platform presumes intent.\nEnhancing identity security within an existing environment begins not with sophisticated concepts but with difficult questions. Who truly requires this level of access today? Which accounts could cause the most harm if compromised? Which permissions were granted out of urgency rather than necessity? These questions are more challenging than implementing new technology because they demand discussion, not just configuration.\nOne of the most transformative changes an organization can undertake is rethinking its approach to privilege. Privileged access shouldn’t merely be something you possess; it should be something you actively utilize. Although this distinction appears subtle, it has significant implications. When access is perpetually available, it tends to go unnoticed. Conversely, when access is granted with clear boundaries, purpose, and visibility, it becomes a deliberate, conscious action. Many identity programs succeed or fail not because of the tools they use, but because of the discipline to treat privilege as an exceptional measure.\nMFA is a common but often undervalued control. While most environments have it enabled in some form, its application is inconsistent. Over time, exceptions accumulate: service accounts that can’t use MFA, users excluded to prevent disruption, and outdated protocols that haven’t been retired. Though each exception makes sense on its own, together they quietly expand the attack surface. Enforcing MFA everywhere isn’t revolutionary, but doing so consistently is transformative. It prevents many types of attacks without modifying any workloads.\nService identities require careful attention because they are often overlooked. Human access is reviewed periodically, typically in response to HR requirements, but automation is not part of this cycle. Many service principals, managed identities, and API permissions linger because removing them seems risky. The concern is, \u0026ldquo;What if something breaks?\u0026rdquo; While this fear is understandable, unmanaged automation identities frequently grant more power than individual users. Recognizing them as critical security assets and defining their ownership, scope, and expiration is a subtle yet highly effective enhancement you can implement.\nIdentity security exposes a challenging truth about cloud maturity. While many organizations allocate substantial resources to platform controls, they often underfund identity governance. Managing identity is a slow process that demands cross-team collaboration. It raises complex issues around trust and responsibility, with no straightforward solution that can be resolved quickly or in a single sprint. Despite these challenges, neglecting this aspect jeopardizes the effectiveness of all other security efforts.\nThe cultural effect of tightening identity controls is frequently underestimated. When individuals are aware that access is monitored, that privileges are temporary, and that exceptions are visible, their behavior shifts. Teams adapt their design approaches accordingly. Automation is implemented more purposefully, and security transitions from an external imposition to an integrated part of platform management.\nNone of this implies that identity security should act as an obstacle. The aim isn\u0026rsquo;t to hinder teams, but to eliminate the hidden risks that accumulate when access is treated as permanent. Properly designed identity controls facilitate agility by establishing trust. Understanding who has access, what they can do, and why makes changes safer rather than slower.\nThis is where identity security intersects with the series\u0026rsquo; central theme. Cloud security isn\u0026rsquo;t about achieving a final goal; it\u0026rsquo;s about continuously evolving systems that adapt faster than any architecture diagram can depict. Identity remains at the heart of this ongoing change. Every new workload, integration, and team affects it. Overlooking this fact doesn\u0026rsquo;t make security easier; it ensures future incidents.\nIn the next part of this series, the emphasis moves from identifying who can act to understanding what the platform reveals about itself. Logging, monitoring, and detection are typically considered operational matters, but they are equally vital for security. We will examine why many Azure environments capture the right signals yet fail to recognize significant risks, and how turning telemetry into actionable insights depends more on purpose than on tools.\n","permalink":"https://wolkwacht.nl/posts/identity-is-the-perimeter-you-forgot-to-guard/","summary":"\u003ch2 id=\"identity-is-the-perimeter-you-forgot-toguard\"\u003eIdentity Is the Perimeter You Forgot to Guard\u003c/h2\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*gX_R-vKzwKJ5LVyGljnUGw.png\"\u003e\u003c/p\u003e\n\u003cp\u003e\u003cem\u003ePart 2 of the Cloud Security series\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eCloud security often fails not due to advanced attacks, but because of unnoticed access and unrevoked privileges. This is the second part of a series on cloud security in real-world Azure environments, which evolve over time under pressure and good intentions. The first part discussed how small, fundamental improvements can significantly improve security. In this segment, we focus on identity, as it is the critical factor determining success or failure in cloud security.\u003c/p\u003e","title":"Identity Is the Perimeter You Forgot to Guard"},{"content":"\nMost cloud security stories begin in the wrong place.\nThis article launches a series on cloud security, focusing on real Azure environments rather than idealized designs. It examines platforms that have been operational for years, where identities developed naturally, workloads were deployed quickly, and security measures were gradually implemented. The series aims not to present new tools or frameworks but to analyze how security evolves in practice, identify where genuine risks arise, and highlight the most effective improvements when working with existing systems.\nPicture stepping into an established Azure environment where production workloads are active and essential applications rely on them. The platform is operational, particularly in terms of availability. However, a pressing concern persists: how truly secure is this environment? Not just in theory or according to a standard architecture, but in real-world practice today.\nThe first lesson from these scenarios is that security is rarely absent; it is often fragmented. For example, a Conditional Access policy requiring MFA might be in place, but it’s unclear who it covers. Defender for Cloud could be enabled in one subscription but not others. Logs are available, yet they are generally ignored unless an issue arises. This isn\u0026rsquo;t due to negligence but rather entropy. Azure environments naturally tend to grow more complex over time unless security measures are deliberately planned and managed.\nWhen engaging with an existing environment, the main mistake is setting overly ambitious goals too fast. Large security initiatives often falter not because of flawed controls but because they overload the organization. The most successful improvements are frequently surprisingly straightforward. They don’t need new tools, additional licenses, or a lengthy six-month plan. Instead, they demand clarity, consistency, and the confidence to declare “this is the baseline now.”\nIdentity is often the initial area where security improvements can be achieved. In many settings, identities serve as both the strongest control and the weakest link. Privileged roles have accumulated over time, service accounts often lack owners, and Global Administrator access is frequently a convenience rather than an exception. Improving this does not require a complete redesign. It begins with gaining visibility: understanding who has access, why they have it, and if that reason remains valid is a significant security upgrade. While enforcing MFA everywhere is not groundbreaking, applying it consistently can drastically reduce the attack surface.\nThe story often shifts to posture management, with Defender for Cloud revealing uncomfortable truths: the platform already knows your exposures. While its recommendations are imperfect and should not be followed blindly, they serve as a mirror many organizations prefer to avoid. The goal is not to \u0026ldquo;fix everything\u0026rdquo; but to identify which recommendations truly impact your workloads and make them non-negotiable priorities. When security guidance transforms into a shared agreement rather than an endless list, real progress begins.\nLogging and monitoring offer significant impact with relatively low effort. Although many Azure setups log data comprehensively, they often lack effective observation. Security logs are stored in Log Analytics workspaces that are rarely accessed. Alerts tend to be either too frequent or missing altogether. Transforming logs into actionable insights doesn\u0026rsquo;t necessitate a full SOC from the start. Instead, it involves defining what constitutes \u0026ldquo;abnormal\u0026rdquo; in your environment and assigning responsibility for responding to these events. Security without designated ownership is just showmanship.\nWhat makes these changes impactful is not their technical difficulty but their cultural impact. Every improvement conveys a message: security isn\u0026rsquo;t a future project; it\u0026rsquo;s integral to how we run this platform. That message has greater significance than any individual control. Teams change their approach when they understand that access is monitored, alerts are addressed, and exceptions are transparent.\nThere is a temptation to believe that absolute security begins with advanced topics such as zero trust architectures, confidential computing, or threat hunting. Those things absolutely matter, and they will be part of this series. But they only work when the foundation is solid. In existing Azure environments, the most significant risks are rarely exotic attacks. They are predictable failures of hygiene, consistency, and governance.\nThe paradox of cloud security is that the most significant improvements often seem dull. They lack the excitement of detailed diagrams or engaging conference presentations. Nonetheless, these improvements distinguish a merely operational environment from a truly trustworthy one.\nThis initial step isn’t about achieving perfection but about gaining momentum. It demonstrates that security improvements can be implemented without halting operations, requiring endless redesigns, or waiting for a distant future that may never come. Once this momentum is established, more advanced discussions can take shape with confidence.\nAnd that is where the real security work begins.\nThis initial section concentrates on the basics: prioritizing clarity over accessibility, consistency over complexity, and steady progress over perfection. These subtle adjustments help reduce risk quietly and open up space for more advanced security measures. In the next part of this series, the emphasis will move to identity, as it remains both the strongest control and the most overlooked vulnerability in nearly every Azure environment. We will examine how access and privileges can gradually deviate over time and demonstrate how reclaiming control of identity is the most impactful security improvement you can achieve.\n","permalink":"https://wolkwacht.nl/posts/the-first-security-conversation-you-have-too-late-hardening-an-existing-azure-environment/","summary":"\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*wkFXyYnv6jgx6grsYubf6g.png\"\u003e\u003c/p\u003e\n\u003cp\u003eMost cloud security stories begin in the wrong place.\u003c/p\u003e\n\u003cp\u003eThis article launches a series on cloud security, focusing on real Azure environments rather than idealized designs. It examines platforms that have been operational for years, where identities developed naturally, workloads were deployed quickly, and security measures were gradually implemented. The series aims not to present new tools or frameworks but to analyze how security evolves in practice, identify where genuine risks arise, and highlight the most effective improvements when working with existing systems.\u003c/p\u003e","title":"The First Security Conversation You Have Too Late: Hardening an Existing Azure Environment"},{"content":"The Unseen Work of a Cloud Architect: A Story About Building Azure the Hard (and Right) Way The story of implementing an Azure cloud environment rarely starts with technology. It begins with a seemingly simple conversation: someone in the business has a goal, a team has a new initiative, or an executive has read an article promising faster innovation, better resilience, or lower operational costs. The request sounds straightforward: “We want to move to Azure.” But it\u0026rsquo;s never that simple. Not because Azure is inherently complex, though it can be, but because cloud architecture is more about managing people, expectations, culture, pressure, and an ever-changing regulatory landscape.\nI’ve learned that every Azure journey starts with a promise of possibility. But possibility quickly clashes with reality, turning the cloud architect into a translator, negotiator, strategist, firefighter, therapist, and sometimes the villain who says “No, we can’t do that… yet.”\nThis story reveals what it genuinely means to implement an Azure cloud environment, not from polished diagrams or landing zone blueprints, but from the trenches where decisions matter, missteps have impacts, and architecture involves negotiation rather than simple design.\nThe First Meeting: Hope, Pressure, and the Myth of “Just Azure” It often begins with a room full of people nodding eagerly. Azure is seen as the future, strategic, and aligned with what competitors are adopting. There is a unanimous consensus that the cloud will deliver faster speeds and greater agility, two terms frequently mentioned together in meetings, often alongside a third: innovation.\nBut underneath the enthusiasm sits a quiet expectation: the architect will take this energy and turn it into something real. Something reliable. Something secure. Something compliant. Something scalable. And something that works on day one.\nWhat many overlook is that even the initial step of defining what “move to Azure” entails unlocks a multitude of decisions. Is it a migration, modernization, or starting fresh with a new platform? What regulations must be considered? How mature is the organization? What is the current security stance? How should landing zone governance integrate with existing processes? And perhaps most critically: what are the real objectives of the business beyond the marketing slogans?\nI’ve often observed that the most challenging aspect of cloud adoption is the translation layer. Business units focus on outcomes, finance on budgets, cybersecurity on risks, DevOps on pipelines, engineers on YAML, and leadership on the idea that “the cloud should be cheaper.\u0026quot; This last belief particularly haunts cloud architects more than they typically admit.\nEveryone agrees on migrating to Azure, but there is no consensus on what Azure precisely entails.\nThe Blueprint Nobody Sees One of the things that still surprises people is how much architecture is invisible. For every visible deployment of an AKS cluster, a virtual network, and a storage account, there are dozens of invisible decisions shaping it. Naming standards, tagging models, identity governance, enterprise app registrations, conditional access policies, region selection strategies, service principal lifecycle, cost ownership, FinOps guardrails, and compliance controls that exist precisely so nobody notices when they work.\nA cloud architect typically focuses on these key areas, but many people assume Azure automatically handles them. People are often surprised to learn that Azure doesn\u0026rsquo;t enforce naming conventions, establish network baselines, segment privileged access, or prevent developers from accidentally deploying a public IP from their laptops.\nAzure gives you the tools. The architect builds the system around them.\nDesigning such a system becomes more complex because architecture does not develop in isolation. Instead, it operates within an ecosystem of legacy systems, existing governance, entrenched processes, siloed teams, cautious security departments, underfunded operations, and developers who usually want the cloud but resist the guardrails.\nThe irony is that the most well-designed cloud environments seem simple and effortless. However, this simplicity results from making countless difficult decisions, most of which are settled long before the first workload is deployed in Azure.\nIdentity: The First Real Battle If you ever want to test a cloud architect’s patience, ask them about identity. Every cloud journey inevitably hits the wall called Entra ID, and the crash is rarely graceful.\nIdentity bridges the gap between abstract cloud goals and real business needs. It’s no longer just about elegant architecture; it’s about controlling who can do what, why, and whether the organization truly manages its directory to handle enterprise apps, privileged roles, or guest accounts. You encounter deeply nested groups, forgotten service principals with full permissions, orphaned accounts of former employees, and admins clinging to global admin rights “just in case.”\nThe conversation shifts from architecture to sociology. Identity reflects an organization\u0026rsquo;s history more than its engineering principles. Cleaning it up feels like excavating a digital archaeological site.\nHowever, this is also the point where the cloud architect must begin to establish a structured framework. This includes defining Administrative Units, implementing privileged access workflows, creating custom roles, enforcing separation of duties, conducting access reviews, and developing a governance model that prioritizes identity as the primary security boundary instead of an afterthought.\nEvery Azure environment must start with identity, as everything else relies on it. Despite its importance, identity is often underestimated, making it the initial challenge and a frequent point of political contention.\nNetworking: Where Every Decision is Permanent You can update policies, reassign roles, or restructure resource hierarchies. However, networks are fixed once established. Selecting address spaces, region topology, and isolation boundaries ties your organization to long-term architectural choices that are difficult and costly to change later.\nThe cloud architect knows this. Leadership often doesn’t.\nNetworking often involves balancing future flexibility with current limitations. You might advocate for a hub-and-spoke or virtual WAN setup. Someone may suggest, “Just do peering for now,” and you’ll need to explain what that entails. They’ll argue it’s only temporary. A year later, what was supposed to be temporary becomes permanent, and the architecture turns into a negotiation with the ghosts of past choices.\nHybrid connectivity serves as the great equalizer of ambition. It includes ExpressRoute, VPNs, firewalls, BGP, overlapping IP ranges, legacy environments with undocumented address spaces, and the persistent belief that connecting the cloud to on-premises systems simply requires purchasing the right SKU.\nAzure networking is powerful, flexible, and comprehensive. However, implementing it demands a persistent focus on long-term maintainability. Every shortcut becomes a cost that the cloud architect must pay later.\nSecurity: The Part Everyone Wants Until It Slows Them Down Cloud security is a topic everyone agrees is important until it slows things down. The struggle between innovation and safety drives cloud architecture, and that\u0026rsquo;s most evident in security design.\nYou propose Zero Trust. People agree. You suggest segmenting workloads. They agree. You recommend managed identities, proper certification rotation, and service principals with minimal privileges. They agree again. Then a developer requests Owner permissions on the subscription “just for a few days,” and suddenly the agreement ceases.\nSecurity involves balancing boundaries. If it\u0026rsquo;s too lenient, you’re setting up for future incidents. If it\u0026rsquo;s too strict, you become a bottleneck that everyone criticizes.\nThreat modeling, least privilege, securing service boundaries, policies, conditional access, Privileged Identity Management, Defender for Cloud, network controls, and key vault governance are not visible in the final application diagram. However, all of these elements influence the environment’s security.\nThe cloud architect is responsible for saying “no,” even if it’s unpopular. They need to remind teams that control isn’t the enemy of innovation, chaos is. However, that message often doesn’t resonate on the first try.\nCompliance: The Invisible Hand of European Cloud Architecture Being a cloud architect in Europe involves more than just designing systems; it requires navigating regulations such as GDPR, DORA, NIS2, and sector-specific rules that shape the environment before deploying an Azure resource. You must consider data sovereignty, residency, legal liabilities, audit trails, and ensure every service complies with increasingly strict standards.\nIt encourages a new perspective on boundaries. Regions now matter beyond just latency, affecting sovereignty controls, encryption models, and service classifications that shape your landing zone. Operational processes now become integral to architecture.\nAzure offers sovereign controls, customer-managed keys, regional restrictions, private endpoints, and policy-driven governance, but using these features effectively requires discipline. Explaining them to non-technical stakeholders can also be a challenging art.\nCompliance is rarely the hero of the story, but it quietly shapes every chapter.\nFinOps: Where Architecture Meets Reality Someone always inquires about the cost. Sometimes it’s too early, but mostly it’s late, often right after the first bill arrives.\nFinOps presents a distinct challenge for cloud architects because cloud economics differ significantly from traditional IT budgeting. While the cloud offers elasticity, this flexibility needs proper governance. Without governance, Azure risks turning into a space filled with abandoned POCs, neglected managed disks, oversized databases, and environments lacking ownership.\nYou can design an ideal architecture, but without accountability for spending, it becomes unsustainable. Thus, the architect’s role shifts to that of an educator, guiding product owners to see cost as a feature rather than an afterthought. They introduce tagging, enforce budgets, set up alerts, define ownership, and incorporate cost insights into operational workflows. They also advocate for automation to eliminate unused resources.\nFinOps is not a discipline that creates barriers. Instead, it helps prevent cloud adoption from failing due to its complexity. As with security, its benefits are usually acknowledged only after it prevents the organization from making potential errors.\nLanding Zones: The Promise and the Pain Landing zones are frequently viewed as the ultimate solution. They address governance, security, consistency, compliance, and operational readiness.\nBut they don’t solve the most challenging part: getting consensus.\nA landing zone isn’t just a template; it’s a binding agreement among teams. It specifies ownership, workload behaviors, and environment interactions. It formalizes decisions that many teams might prefer to keep ambiguous. Because it brings clarity, it often faces resistance.\nChoosing management groups becomes a debate over organizational responsibility. Policies are scrutinized in debates over autonomy. Network architecture shifts into a proxy battle between teams advocating for centralization and those seeking independence. Identity governance reveals hidden legacy structures that people hadn’t noticed before. As a result, the architect\u0026rsquo;s role shifts from designing infrastructure to enabling change.\nImplementing a landing zone is fulfilling not because of the technology itself, but because it creates coherence within an organization that has outgrown its previous methods.\nDevelopers: The Unofficial Customers of the Cloud No matter what leadership believes, developers are the primary users of the cloud. If the platform doesn\u0026rsquo;t support them, if pipelines are sluggish, permissions are confusing, environments vary, or guardrails are overly strict, they\u0026rsquo;ll find ways to bypass it. Developers don’t do this out of malice; they do it out of necessity. Their main goal is to deliver value.\nThe cloud architect’s responsibility is to create a platform that allows developers to move quickly without causing issues. This involves well-crafted pipelines, self-service deployment options, clear boundaries, predictable governance, and an experience that feels supportive rather than restrictive.\nThe most effective Azure environments are guided by something many organizations initially overlook: empathy for developers, who use the platform daily. If they don’t enjoy it, no one will.\nOperations: The Forgotten Pillar of Cloud Adoption Eventually, every cloud environment faces real-world operations, including monitoring, logging, alerts, incident response, disaster recovery, backup strategies, automation, and documentation. These factors decide whether the environment endures long enough to generate value.\nOperational maturity may not be glamorous; nobody cheers for the alert systems that avert outages or the runbooks used for midnight recoveries. However, these are the foundational elements of cloud reliability.\nAzure offers excellent tools such as Monitor, Log Analytics, Managed Prometheus, Automation, Backup, and Site Recovery. However, tools alone are not sufficient. You also need clear processes, ownership, and transparency. Without these, the cloud risks becoming a showcase of great architectural ideas that fail in everyday practice.\nPeople, Culture, and the True Challenge Most Architects Never Mention Among all the technical challenges, the most difficult part of implementing Azure is the people.\nDifferent teams have varying concerns: some fear losing control, others worry about increased accountability. Some desire total freedom, while others prefer strict governance. Some teams don’t fully grasp the cloud, others misunderstand it, and many resist change.\nThe cloud architect acts as a link between traditional practices and innovative opportunities. They need to champion guiding principles that may not yield immediate benefits. Their role involves persuading others smoothly, without pressure, while safeguarding the environment from shortcuts, all the while supporting business progress.\nThis is the emotional labor of cloud architecture, the aspect that rarely appears in reference architectures or diagrams. It demands patience, resilience, and the ability to remain calm amidst conflicting demands. This is what elevates architecture to a form of leadership.\nThe Moment Everything Finally Works Every cloud journey has a moment when everything comes together. FinOps dashboards provide clear cost insights. Developers deploy confidently via pipelines. Security teams trust the guardrails. Operations run smoothly without needing constant escalation. Landing zones stop deteriorating into chaos and begin maturing into stability. The business perceives value, not complexity.\nThis moment often happens quietly without celebration. A team deploys a new service in Azure smoothly, with no issues. It deploys seamlessly, adheres to governance, logs are sent to the proper workspace, costs are stable, permissions are accurate, and there\u0026rsquo;s no need for emergency access.\nThis absence of drama signifies architectural success. The cloud appears ordinary, predictable, and dull.\nAnd that’s the goal!\nReflections From the Architect’s Chair Reflecting on this, the most common misconception about cloud implementation is that it\u0026rsquo;s mainly a technical task. In reality, while the technical aspects are complex, they are generally predictable. Azure offers established patterns, best practices, landing zone frameworks, and extensive community knowledge. The true challenge of cloud architecture lies in everything surrounding the technology itself.\nThe issues stem from team misalignment, unclear goals, organizational inertia, conflicting priorities, and governance gaps that existed long before Azure was introduced. Additionally, political resistance to change and human fears of losing control or being exposed contribute to the problem.\nImplementing Azure isn\u0026rsquo;t just about provisioning resources; it involves creating a technical, cultural, and organizational system that can evolve more rapidly than its surrounding environment.\nA good architect understands Azure.\nA great architect knows how to lead people through change.\nAn exceptional architect creates a platform so seamless that the organization doesn’t notice the complexity because everything functions effortlessly.\nThe Ending That Isn’t an Ending What makes cloud architecture fascinating is that it never really ends. The platform evolves, services change. Regulations shift. New features introduce new possibilities. Workloads move. Teams grow. The architectural story is always in motion.\nThe role of the cloud architect consistently involves clarifying complexity, guiding chaos, and establishing structure for ambitions. They convert business requirements into technical solutions, safeguard the organization from external threats and internal shortcuts, and create enduring systems that outlast projects, roadmaps, and even the architects themselves.\nAzure is a powerful platform. However, the process of implementing it uncovers much more about the organization than about the technology itself. It is in this journey, often messy, political, challenging, and deeply human, that the true art of cloud architecture is discovered.\n","permalink":"https://wolkwacht.nl/posts/the-unseen-work-of-a-cloud-architect-a-story-about-building-azure-the-hard-and-right-way/","summary":"\u003ch2 id=\"the-unseen-work-of-a-cloud-architect-a-story-about-building-azure-the-hard-and-rightway\"\u003eThe Unseen Work of a Cloud Architect: A Story About Building Azure the Hard (and Right) Way\u003c/h2\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*VR5tGmYaHd6FvM9ac29fPg.png\"\u003e\u003c/p\u003e\n\u003cp\u003eThe story of implementing an Azure cloud environment rarely starts with technology. It begins with a seemingly simple conversation: someone in the business has a goal, a team has a new initiative, or an executive has read an article promising faster innovation, better resilience, or lower operational costs. The request sounds straightforward: “We want to move to Azure.” But it\u0026rsquo;s never that simple. Not because Azure is inherently complex, though it can be, but because cloud architecture is more about managing people, expectations, culture, pressure, and an ever-changing regulatory landscape.\u003c/p\u003e","title":"The Unseen Work of a Cloud Architect: A Story About Building Azure the Hard (and Right) Way"},{"content":"Operating AKS from your workstation: Inside Microsoft’s New AKS Desktop There has long been a peculiar duality in managing Azure Kubernetes Service. The Azure portal offers abundant information but often feels disconnected from the cluster’s core functions. Meanwhile, Kubernetes is accessible through kubectl, YAML files, and dashboards like Headlamp. One exists within the Azure control plane, and the other entirely within the Kubernetes API. Operators frequently switch between these two realms multiple times daily, yet they never quite close the gap.\nFor years, we accepted this fragmentation as normal. The portal shows you the Azure environment. kubectl shows you the cluster state. Dashboards provide a limited view of the cluster. But none of these are operator workspaces. They are interfaces for infrastructure, configuration, or introspection, not for the day-to-day tasks of someone working in Kubernetes.\nMicrosoft finally recognized this gap and developed a dedicated tool for frequent AKS cluster operators. It’s not just another dashboard, portal pane, or kubectl wrapper. It’s a purpose-built local UI designed specifically for operators: AKS Desktop, built on Headlamp.\nIt immediately seems like an essential tool that should have been there from the start, a fast local client connecting the Kubernetes interface with the Azure environment. It acts as an operator cockpit tailored for AKS rather than a generic CNCF dashboard. This is where logs, events, node pools, networking, workload identities, and Azure integration coexist seamlessly. Most importantly, it eliminates the need to switch between browser tabs, CLI sessions, and external tools to grasp what your cluster is actually doing.\nBefore exploring its features, it\u0026rsquo;s important to clarify that AKS Desktop is currently in Public Preview. The UI might undergo changes, and features may evolve or be added. Expect some rough edges. Nonetheless, the core vision is clear: Microsoft aims to develop a local-first operator experience for Kubernetes on Azure. Even in its preview stage, it is already transforming how you work with AKS daily.\nWhere AKS Desktop Fits in an Operator’s World Anyone managing multiple AKS clusters knows the operational surface gradually becomes a patchwork of various tools. The Azure portal provides views for upgrades, node pool management, and policy details, but hides the cluster\u0026rsquo;s internal workings. On the other hand, kubectl provides access to everything in the Kubernetes API but does not reveal Azure-specific aspects governing the cluster’s lifecycle, networking, and security. Tools like Lens and K9s help bridge some of these gaps, but they operate solely within the Kubernetes API. They lack understanding of Azure’s node pools, VMSS behavior, workload identity mappings, or the process of transitioning a cluster between API versions.\nThis fragmentation becomes exhausting quickly. You need logs, events, node pool status, workload identity bindings, ConfigMaps, deployment rollout updates, and Azure upgrade info, preferably all in one place without changing tools. Until now, that wasn’t feasible.\nAKS Desktop fills that gap by positioning itself as a local desktop app that integrates closely with both Azure’s control plane and Kubernetes runtime. It uses your Azure CLI authentication, retrieves cluster details via ARM, and connects directly to the Kubernetes API with kubeconfig, combining all these elements into a unified visual map.\nAlmost immediately, it’s evident that AKS Desktop isn\u0026rsquo;t just another generic Kubernetes dashboard. It does not aim to compete with CNCF tools or replicate the Kubernetes dashboard. Instead, it emphasizes Azure-native features and a vendor-specific approach. When exploring an AKS cluster in AKS Desktop, users see Azure node pools, network overlays, workload identity resources, upgrade paths, and Azure Policy evaluations alongside Pods, Deployments, Services, and Ingress.\nIt offers a well-balanced combination of Azure Resource Manager, Kubernetes API, and operational workflows. This creates a user interface that truly resembles an operator cockpit for AKS, rather than just a generic dashboard repurposed for the purpose.\nInstalling AKS Desktop: The Start of a Local-First Workflow The great thing is that installation is straightforward. It doesn’t involve backend systems or extensions, nor does it deploy controllers or operate inside the cluster. AKS Desktop is completely local, with authentication managed through your Azure CLI identity.\nBefore opening it for the first time, consider preparing your Azure environment.\naz login az account set --subscription \u0026#34;\u0026lt;your-subscription-id\u0026gt;\u0026#34; All your security measures, whether Conditional Access, MFA, PIM, or strict governance, are seamlessly integrated into AKS Desktop since it just uses your Azure CLI tokens. You log in once, and everything else proceeds automatically.\nOn macOS, installation is straightforward: either download the DMG or use Homebrew.\nbrew tap azure/aks-desktop brew install aks-desktop Windows users can use Winget:\nwinget install Microsoft.AKSDesktop Linux distributions can be installed using a .deb package or an archive. After installation, the application automatically detects all AKS clusters accessible to your identity, allowing you to generate or refresh the kubeconfig for any of them with a single click.\nIt\u0026rsquo;s refreshing that no components are deployed into the cluster. There are no agents, CRDs, DaemonSets, or any extensions to the Kubernetes API. The tool maintains a clear boundary between the local operator experience and the cluster runtime. The only interaction occurs via kubeconfig and the Azure control plane.\nWhen you select a cluster, the application automatically activates. The cluster topology appears, node pools materialize, workloads load quickly, and events start streaming in. Logs become accessible. It resembles a visual extension of kubectl but is deeply integrated with Azure’s architecture.\nThe Difference Between AKS Desktop and Headlamp This is the common question: “If I already use Headlamp, is AKS Desktop still necessary?” The answer is both straightforward and nuanced. Headlamp is entirely vendor-neutral, representing the Kubernetes environment exactly as the API describes it. This is especially beneficial if you manage multiple clusters across different clouds or operate hybrid platforms, as it provides a consistent and predictable interface.\nHowever, it fails to recognize Azure’s specific features. Headlamp does not detect node pool upgrade options, workload identity bindings, or VNet profiles. It also cannot display Azure Policy statuses or properties of the AKS-managed control plane. Instead, it interprets everything solely as Kubernetes.\nOne day, when troubleshooting a failed deployment, Headlamp will display the deployment events and pods. AKS Desktop will show the same information and also indicate the health of the node pool, whether the cluster is in the middle of an upgrade, whether Azure Policy is blocking resources, or whether workload identity issues are preventing token retrieval.\nBoth tools are valuable: one is a Kubernetes viewer, and the other serves as an Azure-integrated control panel. Together, they provide a comprehensive view.\nThe Azure Reality Layer: what AKS Desktop Adds When you explore your first cluster in AKS Desktop, you’ll immediately notice the distinct Azure-native layers. Node pools are not merely displayed as groups of nodes but as actual VMSS-backed pools, with details like size, type, version, and health clearly visible.\nNetworking now shows Services, Ingress, VNet integration, and the network profile of the cluster. Workload identity is prioritized as a first-class feature. Azure Policies provide enforcement and compliance updates. Upgrade options are now positioned beside the cluster identity rather than being tucked away in a separate portal page.\nThis is the real strength of AKS Desktop. It goes beyond simply querying the Kubernetes API by seamlessly merging ARM realities with cluster realities. It removes the need for operators to mentally assemble the environment from portal blades and CLI outputs. Instead, it presents the Azure architecture directly beside the Kubernetes context. Only when you see this fragmentation cleared up do you recognize how unsettling it was.\nThe Kubernetes Surface: Still Clear and Familiar Even with the Azure integration, AKS Desktop maintains clear visibility into Kubernetes workloads. It displays pods, logs, and events in real time without requiring complex terminal commands. ConfigMaps and Secrets are shown clearly, while Deployments, StatefulSets, Jobs, CronJobs, and DaemonSets are easily accessible.\nTroubleshooting is simplified: a quick visual scan of a pod in CrashLoopBackOff immediately shows its logs; a failing service becomes easy to triage; and a stuck rollout instantly reveals related ReplicaSet and pod events. While similar results can be achieved with kubectl, logs, and describe commands, AKS Desktop consolidates everything into user-friendly views that eliminate the need for mental juggling. It restores operator downtime, something Kubernetes seldom offers.\n# A look at the architecture ┌──────────────────────────────────────────────┐ │ AKS Desktop App │ │ (Local operator UI for Azure Kubernetes) │ └──────────────────────────────────────────────┘ │ │ Azure CLI Credentials ▼ ┌──────────────────────────────────────────────────────────────┐ │ kubeconfig Contexts │ │ (Generated \u0026amp; managed based on your selected AKS cluster) │ └──────────────────────────────────────────────────────────────┘ │ │ Direct Kubernetes API Requests ▼ ┌──────────────────────────────────────────────────────────────┐ │ AKS Control Plane │ │ ARM + AKS Managed Identity + Network + NodePools │ └──────────────────────────────────────────────────────────────┘ │ │ ARM \u0026amp; Azure-native metadata ▼ ┌──────────────────────────────────────────────────────────────┐ │ Azure Resource Manager │ │ Node Pools, Network Profile, Workload Identity, Policy │ └──────────────────────────────────────────────────────────────┘ Daily Workflow with AKS Desktop: A Natural Operator Flow Kubernetes operations follow a consistent rhythm: authenticate, select the correct context, inspect the cluster topology, check workloads, validate logs, scan events, triage node issues, monitor rollouts, troubleshoot identity bindings, track autoscaler behavior, and respond to network concerns. Typically, these tasks are performed across various tools. AKS Desktop streamlines the rhythm into a more seamless flow.\naz login az account set --subscription \u0026lt;id\u0026gt; Select your cluster.\nWatch the topology appear as if the cluster is sitting inside your workstation.\nAfter a few minutes, you may switch to a different cluster, changing contexts without the kubectl equivalent of stack traces in your terminal. Everything stays local and quick. Logs show up instantly. Events stream in real time. Node pools show their health status clearly, like a debugging console.\nFor teams managing multiple AKS clusters, this is a significant relief. It now feels like AKS is a local development environment rather than just a remote system behind the Azure portal’s interface.\nValidating Cluster Connectivity: The Familiar CLI Layer Although AKS Desktop handles kubeconfig for you, the CLI remains a constant companion. For operators who want to double-check cluster accessibility:\nkubectl cluster-info kubectl get nodes -o wide kubectl get pods --all-namespaces These commands should immediately reflect what you see in the application. If anything diverges, the troubleshooting flow becomes much more intuitive because the UI helps surface what the CLI hides in noise.\nIf you want to create kubeconfig manually:\naz aks get-credentials \\ --resource-group rg-mycluster \\ --name aks-mycluster \\ --overwrite-existing The thinking becomes: use the CLI for intent, AKS Desktop for navigation, and kubectl for precision. The three together create a near-perfect operator workflow.\nThe Workload Reality: A Sample Deployment for Demonstration To familiarize yourself with the interface, deploy a small workload and observe its appearance. This NGINX example is a classic and functions well in AKS Desktop.\napiVersion: apps/v1 kind: Deployment metadata: name: aksdesktop-sample labels: app: aksdesktop-sample spec: replicas: 2 selector: matchLabels: app: aksdesktop-sample template: metadata: labels: app: aksdesktop-sample spec: containers: - name: sample image: nginx:stable ports: - containerPort: 80 The moment it rolls out, AKS Desktop shows the pod startup sequence, container states, events, logs, and the overall deployment health.\nRollouts go from abstract kubectl lines to smooth, observable transitions.\nWorkload Identity, Troubleshooting, and the Azure Connection AKS Desktop greatly improves the operator experience, especially with workload identity. If you’ve ever debugged a misconfigured AzureIdentityBinding or seen pods fail because they couldn’t get tokens from the federated identity endpoint, you know how confusing the documentation can be compared to real-world issues.\nkubectl gives you the raw truth:\nkubectl get azureidentity -A kubectl get azureidentitybinding -A kubectl get serviceaccount -n myapp However, the application visually connects these components. It shows if the service account is mapped, if the identity exists, and whether the cluster is enforcing the correct node identity assignment. If any issues arise, the errors are visible in context rather than hidden in the describe output.\nNetworking and Node Pools: Visualizing the Azure Infrastructure Networking in AKS is notoriously confusing to newcomers because much of it lives in ARM rather than Kubernetes. AKS Desktop shows the Kubernetes network constructs and then directly ties them back to Azure’s network profile.\nNode pools appear not just as node groups but as actual VMSS-backed pools with their upgrade state and version alignment. When an upgrade is available, you see it clearly. When something is drifting from expected version levels, the tool shows it without requiring portal navigation.\nThis is the first time Azure-specific context appears alongside Kubernetes information.\n# AKS vs Headlamp ┌─────────────────────────┐ ┌─────────────────────────┐ │ AKS Desktop │ │ Headlamp │ │ Azure-aware operator UI │ │ Kubernetes-native UI │ └─────────────────────────┘ └─────────────────────────┘ │ │ │ Azure ARM + kube-API │ kube-API only ▼ ▼ ┌───────────────────────────┐ ┌──────────────────────────┐ │ Nodepools, VMSS, VNet, │ │ Deployments, Nodes, Logs │ │ Workload Identity, Update │ │ ConfigMaps, Secrets │ │ Status, Azure Policies │ └──────────────────────────┘ └───────────────────────────┘ The diagram captures the difference precisely. Headlamp sees Kubernetes. AKS Desktop sees Azure Kubernetes Service.\nThe Public Preview Reality It is important to acknowledge that AKS Desktop is currently in Public Preview, and preview experiences always carry a sense of incompleteness. Some views might feel early. Some features will evolve. Some integrations will deepen. Microsoft’s cadence for AKS improvements is fast, so the application will likely transform over the coming months.\nBut this is also the best moment for operators to adopt it. Early tools shape workflows, influence product direction, and reveal gaps that matter to people who run production clusters.\nSuppose you look at where the industry is heading. In that case, local-first operational tools, cloud-aware dashboards, and multi-cluster management workflows make AKS Desktop clearly part of Azure’s next-generation Kubernetes operator experience.\nRecommended Plugins and Tools That Enhance AKS Desktop AKS Desktop does not have a plugin system yet, but certain tools pair perfectly with it.\nkubelogin is essential because AKS Desktop relies on Azure CLI identity flows. Installing it is simple:\naz aks install-cli For Kubernetes practitioners who love CLI flexibility, krew adds the right amount of power:\nkubectl krew install ctx kubectl krew install ns kubectl krew install sniff kubectl krew install graph kubectl krew install view-allocations kubectl krew install neat kubectl krew install modify-secret These plugins complement AKS Desktop rather than replace it.\nK9s remains invaluable for low-latency terminal triage.\nVS Code’s Kubernetes plugins remain excellent for YAML work.\nThe trio of AKS Desktop, kubectl/k9s, and VS Code forms the most complete operator workflow currently available for AKS.\n# Day-to-Day Operations Flow ┌────────────────────────┐ │ az login \u0026amp; select sub │ └────────────────────────┘ │ ▼ ┌────────────────────────┐ │ Open AKS Desktop │ │ auto-detects clusters │ └────────────────────────┘ │ ▼ ┌────────────────────────┐ │ Select AKS cluster │ │ auto-generates context │ └────────────────────────┘ │ ▼ ┌───────────────────────────────────┐ │ Logs • Events • Upgrades │ │ Nodepools • Networking │ │ Workload Identity • Policy │ │ Everything visible in one place │ └───────────────────────────────────┘ Closing Reflections: Why This Matters For years, the Kubernetes ecosystem has focused on cluster tools such as CRDs, operators, dashboards, and automation. However, the operator’s daily experience often goes unnoticed, as managing AKS involves integrating Azure resource graphs, node pools, network configs, workload identities, policies, and cluster health across multiple clusters, regions, and subscriptions.\nAKS Desktop is Microsoft\u0026rsquo;s first effort to offer a high-quality operator experience on the workstation, creating an environment where Azure and Kubernetes realities coexist seamlessly. It doesn’t replace kubectl, the portal, or Headlamp, but ties them into a cohesive story, offering platform teams an anchor, consistency, speed, and a new mental model of AKS that\u0026rsquo;s more like a single-operator cockpit than a scattered set of tools.\nIt respects fundamentals: no agents, CRDs, cluster installs, or new APIs, just your machine, Azure identity, kubeconfig, and a well-designed UI understanding AKS. Over time, AKS Desktop may become a standard tool for operators. Even as a public preview, it already improves the daily experience by reducing friction, enhancing clarity, and making Azure Kubernetes feel more connected, understandable, and personal.\n","permalink":"https://wolkwacht.nl/posts/operating-aks-from-your-workstation/","summary":"\u003ch2 id=\"operating-aks-from-your-workstation-inside-microsofts-new-aksdesktop\"\u003eOperating AKS from your workstation: \u003cstrong\u003eInside Microsoft’s New AKS Desktop\u003c/strong\u003e\u003c/h2\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*VjDgDHRUsUzVYXwatUtuuA.png\"\u003e\u003c/p\u003e\n\u003cp\u003eThere has long been a peculiar duality in managing Azure Kubernetes Service. The Azure portal offers abundant information but often feels disconnected from the cluster’s core functions. Meanwhile, Kubernetes is accessible through kubectl, YAML files, and dashboards like Headlamp. One exists within the Azure control plane, and the other entirely within the Kubernetes API. Operators frequently switch between these two realms multiple times daily, yet they never quite close the gap.\u003c/p\u003e","title":"Operating AKS from your workstation"},{"content":"Beyond the Cloud: Running Kubernetes with Talos: Comparing Talos, k3s, AKS, and EKS “If Kubernetes is the engine, Talos is the chassis built specifically for it.”\nIntroduction: Rethinking the Kubernetes Operating System When Kubernetes first appeared, it was celebrated as the “Linux of the cloud.” But beneath every Kubernetes cluster, there’s always a traditional operating system like Ubuntu, CentOS, Flatcar, or another Linux distribution. These weren’t originally built specifically for the orchestration layer that overlays them.\nThis historical setup has quietly influenced how we manage, patch, and secure our clusters. Even as control planes have shifted to managed services such as Azure Kubernetes Service (AKS) and Amazon Elastic Kubernetes Service (EKS), the nodes themselves still rely on an operating system that wasn’t initially designed for immutability or declarative control.\nIntroducing Talos, an innovative operating system designed specifically for Kubernetes enthusiasts! Developed by Sidero Labs, Talos marks an exciting evolution from simply running Kubernetes on Linux to becoming an OS centered around Kubernetes itself. It enhances security by replacing SSH with a straightforward API, eliminating package managers, and tightly securing its surface to prevent attacks or drift.\nIn this blog, we will examine Talos, compare it with k3s, AKS, and EKS, and explore its role within the changing landscape of modern Kubernetes operations, from the cloud to the edge.\nThe Operating System for Kubernetes Traditional Linux distributions are developed for general-purpose computing, while Talos is a specialized, immutable OS that directly integrates with Kubernetes control plane components.\nFundamentally, Talos eliminates nearly everything you would typically find in a standard Linux system:\n• No bash\n• No SSH access\n• No mutable filesystem\n• No package manager\n• No direct systemd control\nInstead, all administrative tasks are performed through an API called talosd. This setup prevents configuration drift and ensures operations are declarative and version-controlled.\nThe entire lifecycle of a Talos node, from bootstrap to upgrade, is outlined in YAML and managed via the Talos API using talosctl.\nFor example, a minimal machine configuration might look like this:\n# talosconfig.yaml machine: type: controlplane install: image: ghcr.io/siderolabs/installer:v1.8.0 disk: /dev/sda network: hostname: talos-master-01 cluster: controlPlane: endpoint: https://10.0.0.10:6443 clusterName: talos-lab Every node, whether master or worker, is bootstrapped declaratively, similar to Infrastructure-as-Code. Unlike cloud-managed distributions, Talos does not abstract Kubernetes; instead, it exposes it more transparently.\nTalos vs. k3s: Secure Minimalism vs. Edge Minimalism Both Talos and k3s share a minimal philosophy, but they target different goals.\nk3s, developed by Rancher (now part of SUSE), is a compact Kubernetes distribution designed for edge and IoT use cases. It streamlines cluster deployment by packaging components into a single binary. In contrast, Talos is not a Kubernetes distribution; instead, it is the operating system that Kubernetes operates on.\nWhile k3s makes Kubernetes installation easier, Talos strengthens the underlying infrastructure.\nLet’s compare their philosophies:\nWhile k3s appeals to developers seeking lightweight Kubernetes at the edge, Talos caters to those requiring secure, auditable infrastructure where every state change is logged and no human has shell access.\nA Talos cluster doesn’t just run Kubernetes; it becomes part of it.\nTalos vs. Managed Service: AKS and EKS Managed Kubernetes services like AKS and EKS remove control plane management from your responsibilities. They handle Kubernetes version upgrades, API availability, and integration with cloud IAM, networking, and monitoring.\nHowever, in both services, the node operating system remains mutable. Even if you use Azure Linux, Ubuntu, or Bottlerocket (AWS), those systems still expose SSH, file systems, and package layers that can drift from a baseline over time.\nTalos eliminates those layers.\nFor organizations with strong regulatory or security postures — think NIS2, DORA, or ISO 27001 — that’s a powerful differentiator.\nHere’s how they differ conceptually:\nTalos doesn’t replace AKS or EKS; it complements them.\nYou can run AKS in Azure for managed workloads and deploy Talos clusters on the edge, in colocation sites, or even on bare metal for environments that require strict operational control.\nOperating Talos — Everything Is Declarative Every Talos node is configured and controlled through a single declarative interface.\nThat starts with the machine configuration file, a YAML document that defines networking, storage, certificates, and system extensions.\nExample configuration for a worker node:\n# worker.yaml machine: type: worker network: interfaces: - interface: eth0 dhcp: true install: image: ghcr.io/siderolabs/installer:v1.8.0 kubelet: nodeLabels: node-type: edge cluster: clusterName: talos-lab controlPlane: endpoint: https://10.0.0.10:6443 After generating the configuration with:\ntalosctl gen config talos-lab https://10.0.0.10:6443 You apply it directly to each node:\ntalosctl apply-config --insecure --nodes 10.0.0.21 --file worker.yaml From that point, node configuration and upgrades happen declaratively.\nUpgrading all nodes in a cluster to a new version is as simple as:\ntalosctl upgrade --nodes 10.0.0.21,10.0.0.22,10.0.0.23 --image ghcr.io/siderolabs/installer:v1.9.0 No SSH sessions, no imperative patching — only versioned, API-controlled changes.\nObservability and Integration Since Talos lacks a native shell or package management system, integrations with observability tools are handled externally via Kubernetes or sidecar agents.\nMetrics are collected with Prometheus, logs are streamed to Grafana Loki, and system events are accessible via the Talos API. Cloud services such as AKS or EKS offer built-in integrations with Azure Monitor or CloudWatch, whereas Talos follows a bring-your-own-observability strategy.\nSince Talos is immutable, you can always see exactly what’s running, with no hidden agents or leftover daemons. Observability pipelines are set up through Kubernetes manifests or external exporters, giving you complete control.\nExample Bicep for connecting a Talos edge cluster to Azure Monitor via Arc:\n#bicep resource arcConnection \u0026#39;Microsoft.HybridCompute/machines@2023-03-01\u0026#39; = { name: \u0026#39;talos-edge-node\u0026#39; location: \u0026#39;westeurope\u0026#39; properties: { osName: \u0026#39;Talos\u0026#39; osVersion: \u0026#39;1.8.0\u0026#39; connectivityStatus: \u0026#39;Connected\u0026#39; } } Security: Immutability as Policy In Talos, the OS itself enforces a security policy by design:\n• No interactive logins\n• No filesystem writes beyond /var and /etc\n• No package installations\n• All administrative actions go through an audited API\nEvery API call is logged and can be replayed or versioned. That’s a fundamental shift from “trust the admin” to “trust the configuration.”\nCompared with managed services like AKS or EKS, Talos pushes the security boundary lower. You can still apply Azure Policy, PodSecurityAdmission, or OPA Gatekeeper at the Kubernetes layer — but Talos ensures the node itself can’t be compromised through local access.\nThis makes Talos appealing for industries with stringent controls — financial services, defense, or healthcare — where you can’t allow operator drift or arbitrary SSH access.\nUse Cases in the Real World Imagine a national retail chain with hundreds of stores, each running a small edge cluster to manage local POS data and inventory caches. Running full AKS or EKS on-site would be impractical, and k3s might lack the necessary compliance controls. Talos fits this niche by providing secure, immutable edge clusters synchronized with cloud workloads. Consider an industrial manufacturer deploying Kubernetes within restricted OT zones disconnected from the internet. Talos supports air-gapped deployments using signed container images and declarative configurations, without the need for SSH or remote shells.\nHybrid architectures are also well-suited:\n• AKS or EKS manages centralized CI/CD, monitoring, and security policies.\n• Talos clusters expand compute capabilities to edges, labs, or regulated environments.\nAll layers remain Kubernetes-native and managed with the same tooling, yet each has its own control plane.\nExample Setup: Running Talos on Three Nodes Let’s walk through a compact Talos lab cluster, suitable for edge or homelab testing.\nWe’ll deploy three nodes — one control plane and two workers — on x86 or Raspberry Pi.\nGenerate configurations:\ntalosctl gen config talos-lab https://192.168.1.100:6443 Apply configs:\ntalosctl apply-config --nodes 192.168.1.101 --file controlplane.yaml talosctl apply-config --nodes 192.168.1.102,192.168.1.103 --file worker.yaml Bootstrap the control plane\ntalosctl bootstrap --nodes 192.168.1.101 Acces Kubernetes\ntalosctl kubeconfig ./ kubectl get nodes Once complete, you have a fully functional Kubernetes cluster — without SSH, without mutable OS layers, and entirely declaratively managed.\nFor validated documentation and hardware compatibility:\n👉 Talos Documentation — Getting Started\nComparative Summary Let’s consolidate everything into a single overview:\nConclusion: The OS Boundary Is Shifting Kubernetes used to abstract away the infrastructure layer. But as clusters span data centers, clouds, and edge devices, the operating system itself has become part of the control-plane conversation.\nTalos embodies the shift from flexible Linux distributions to API-based, immutable systems built for Kubernetes. It does not compete with managed services such as AKS or EKS; instead, it enhances them by extending the Kubernetes operating model to environments where control, auditability, and security are essential.\nFor platform and security architects, Talos introduces a new mental model:\n“If you can declare it, you can enforce it. If you can’t shell into it, you can trust it.”\nAs Kubernetes expands beyond the cloud into edge and regulated infrastructures, operating systems like Talos are leading the way for what comes next.\nFurther Reading: • Talos Systems Documentation\n• Sidero Labs GitHub\n• k3s Documentation\n• Azure Kubernetes Service (AKS)\n• Amazon EKS User Guide\n• CIS Kubernetes Benchmark\n","permalink":"https://wolkwacht.nl/posts/beyond-the-cloud-running-kubernetes-with-talos-comparing-talos-k3s-aks-and-eks/","summary":"\u003ch2 id=\"beyond-the-cloud-running-kubernetes-with-talos-comparing-talos-k3s-aks-andeks\"\u003eBeyond the Cloud: Running Kubernetes with Talos: Comparing Talos, k3s, AKS, and EKS\u003c/h2\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*iW_3TlxtWtboH8G39AoiEg.jpeg\"\u003e\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003e“If Kubernetes is the engine, Talos is the chassis built specifically for it.”\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch2 id=\"introduction-rethinking-the-kubernetes-operating-system\"\u003eIntroduction: Rethinking the Kubernetes Operating System\u003c/h2\u003e\n\u003cp\u003eWhen Kubernetes first appeared, it was celebrated as the “Linux of the cloud.” But beneath every Kubernetes cluster, there’s always a traditional operating system like Ubuntu, CentOS, Flatcar, or another Linux distribution. These weren’t originally built specifically for the orchestration layer that overlays them.\u003cbr\u003e\nThis historical setup has quietly influenced how we manage, patch, and secure our clusters. Even as control planes have shifted to managed services such as Azure Kubernetes Service (AKS) and Amazon Elastic Kubernetes Service (EKS), the nodes themselves still rely on an operating system that wasn’t initially designed for immutability or declarative control.\u003c/p\u003e","title":"Beyond the Cloud: Running Kubernetes with Talos: Comparing Talos, k3s, AKS, and EKS"},{"content":"Exploring the concept behind Azure’s next-generation ingress platform\nWhen Ingress Becomes Strategic There’s a moment in every Kubernetes journey when ingress stops being just a configuration detail and becomes an architectural constraint. You start with a small cluster, maybe one or two microservices, and the standard NGINX ingress controller does exactly what you expect: simple routes, basic certificates, nothing fancy. Then the platform expands. You integrate workloads across multiple regions, extend environments with Azure Container Apps, and add a service mesh, and suddenly that “temporary ingress setup” becomes a bottleneck.\nIngress represents the intersection of your cloud environment, governance, and users. It’s the point where security policies align with developer freedom, and where latency, observability, and compliance converge. In essence, this is where complexity emerges.\nAzure’s Application Gateway for Containers (AGC) serves as a new managed ingress category, rather than replacing NGINX or Traefik. It is not merely another controller for Kubernetes, but Azure’s effort to redefine ingress within a comprehensive, cloud-native delivery system that extends across clusters, workloads, and networks.\nFrom Controllers to Cloud Services To grasp AGC’s conceptual significance, let\u0026rsquo;s revisit how we arrived here. Kubernetes ingress has been both powerful and limited: it simplifies complex Layer 7 routing declaratively, yet it also expects you to handle the data plane on your own.\nFor most organizations, this involves deploying open-source controllers like NGINX, HAProxy, or Traefik as pods within the cluster. These controllers are flexible and extensible, but they also bring back many operational challenges you aimed to avoid with managed Kubernetes, such as patching, scaling, certificate renewal, configuration drift, and the ongoing question of “who owns this part of the puzzle.”\nAzure’s initial effort to address this issue was the Application Gateway Ingress Controller (AGIC). AGIC linked AKS with Azure Application Gateway v2, enabling ingress rules to be specified via Kubernetes objects that configured the Application Gateway. While functional, it depended heavily on maintaining synchronization between the cluster state and external infrastructure, involving a controller, a sync loop, and a translation layer.\nApplication Gateway for Containers eliminates the middle step. It’s not a controller communicating with Azure; instead, it’s Azure built from the ground up to natively support Kubernetes. Rather than configuring an external gateway, AGC interprets Kubernetes Gateway API resources directly. This creates a managed, multi-tenant ingress layer that is aware of the cluster topology, namespaces, and policies, without requiring you to run any pods.\nThe Broader Shift: From Ingress to Gateway API AGC’s core design centers on the Gateway API, a modern Kubernetes networking standard intended to replace the traditional Ingress API. More than just an update, the Gateway API represents a fundamental shift in approach. Unlike Ingress, which was deliberately minimalistic, the Gateway API is designed to be expressive. It distinctly divides responsibilities among infrastructure owners, application developers, and service operators, clarifying each group\u0026rsquo;s control areas.\nThe concept is straightforward yet impactful: • Developers specify HTTPRoutes that link paths to services. • Operators set up Gateways to control traffic entry points, exposed listeners, and policies. • Platform teams create GatewayClasses that specify the underlying implementation, such as a cloud service like AGC, a mesh component, or an in-cluster controller.\nAGC aligns perfectly with this model, serving as Azure’s managed implementation of a GatewayClass. It offers a declarative, Kubernetes-native interface to Azure networking. Practically, you can specify routing and policies using YAML, while the Azure control plane handles the complex tasks like TLS termination, scaling, and observability.\nBy integrating directly with the Gateway API, AGC is not just a proprietary ingress layer but a standards-based, cloud-ready gateway. It uses the same protocols as Kubernetes, ensuring compatibility across environments and remaining future-proof as the ecosystem shifts away from traditional Ingress.\nHow Application Gateway for Containers Fits In Conceptually, AGC sits at a very specific intersection:\nBetween Azure’s Layer 7 networking fabric and Kubernetes’ declarative configuration model.\nIn a traditional setup, the ingress controller for your cluster resides within the Kubernetes network boundary. It monitors Ingress or Gateway resources, sets up routes, and terminates connections directly on the cluster nodes. While this is effective, it also couples your control plane and data plane. As a result, when you scale nodes, replace instances, or rotate certificates, the ingress layer is affected.\nAGC separates these concerns by functioning as a managed Azure resource, not as a workload. It is aware of your clusters but does not depend on them for ensuring availability. Traffic first reaches the Application Gateway for Containers, where routing, WAF policies, and TLS termination happen, then it is securely directed to your backend pods.\nThis separation has significant implications for scalability, resilience, and security:\n• Scaling your cluster no longer impacts your ingress throughput.\n• Security policies, such as WAF rules or Private Link settings, are managed in Azure Policy rather than as Kubernetes ConfigMaps.\n• Network isolation is simplified because the ingress tier sits outside your compute plane but remains within your virtual network perimeter.\nIt’s ingress-as-a-service, but one that still behaves like a native Kubernetes component.\nDesign Principles and Conceptual Architecture At its core, Application Gateway for Containers is shaped by three architectural principles:\n1. Kubernetes-Native by Design\nAGC directly consumes Kubernetes resources. Gateways, Routes, and Backends are configured in YAML, managed through GitOps pipelines, and validated with standard Kubernetes admission workflows. There’s no need for external synchronization or manual setup. You define your intent; Azure guarantees execution.\n2. Separation of Control and Data Plane\nThe cluster no longer handles the ingress data path. The Application Gateway for Containers runs in Azure’s managed environment, handling routing, inspection, TLS, and WAF, while your AKS cluster concentrates only on workloads. This separation makes upgrades easier and enables independent scaling of compute and ingress components.\n3. Shared, Multi-Cluster Ready Infrastructure\nAGC is fundamentally designed for multi-tenancy. One gateway instance can serve multiple clusters, each within its own namespace and policy boundaries. This enables new topology options such as hub-and-spoke ingress, shared governance models, or centralized security enforcement across numerous clusters.\nTogether, these design principles create a pattern that resembles Azure Front Door combined with Kubernetes Gateway rather than “NGINX in disguise.” It’s a true dual citizen of both worlds: Kubernetes and Azure networking.\nWhy This Matters for Platform Teams For platform teams, ingress has traditionally been a balancing act between flexibility and control. Developers desire autonomy: they want to deploy microservices that work smoothly without needing firewall change requests. Conversely, security teams seek consistent enforcement, centralized visibility, and defined ownership.\nAGC serves as a link connecting those two worlds.\nSince it’s declarative and aware of Kubernetes, developers can continue to define routes, annotations, and service dependencies. However, enforcement is handled by Azure’s managed ingress layer, which is governed by enterprise-grade policies. This allows for standardization of WAF rules, TLS policies, and logging configurations across the organization without reducing flexibility.\nIt’s an architecture that supports federated governance: local autonomy within global guardrails.\nFor operations teams, this greatly simplifies their work. There’s no need to scale or patch an ingress controller, monitor a reverse proxy, or manually rotate SSL certificates. All components are Azure-native resources, observable via Azure Monitor, and managed by Azure Policy.\nAnd for security teams, the advantage is equally clear: ingress now operates within the same trust boundary as the rest of Azure networking. Traffic can be inspected, audited, and traced using familiar tools without losing the context of the Kubernetes workloads behind it.\nObservability and Policy Integration Observability has long been a hidden challenge in ingress management. When the ingress controller operates within the cluster, its metrics are simply additional Prometheus targets. This setup works until you need to correlate those metrics with Azure networking logs, Application Insights traces, or WAF alerts.\nSince AGC is part of the Application Gateway family, it benefits from Azure’s comprehensive observability tools. Metrics like request counts, latency, and response codes are automatically integrated into Azure Monitor. Security insights, including WAF events, blocked requests, and signature updates, are also accessible through the same portal and APIs used for traditional Application Gateways.\nFrom a design standpoint, ingress observability now seamlessly integrates with your overall Azure monitoring approach. You can trace an end-user request from the public IP to the backend pod, crossing both Azure and Kubernetes boundaries, all within the same tool.\nOn the governance side, AGC’s integration with Azure Policy enables administrators to maintain uniform configuration across all environments. For instance, it ensures that all gateways use HTTPS listeners, that WAF is set to prevention mode, and that deployments exposing plain HTTP endpoints are blocked. Previously, such policies were enforced via admission webhooks or custom controllers, but now they are managed at the platform level, making them consistent, auditable, and organization-wide.\nConceptual Alignment with Modern Platform Engineering One of the most intriguing aspects of AGC is not its technical details but its cultural implications. It showcases the increasing overlap between platform engineering and cloud networking.\nTraditionally, in Kubernetes operations, platform teams concentrate on internal cluster functions such as service meshes, observability, and developer self-service. Conversely, network and security teams manage external components such as gateways, firewalls, DDoS mitigation, and policy enforcement. These two domains often encounter challenges when their responsibilities intersect, especially during incidents.\nAGC dissolves traditional boundaries by offering a network-native service that communicates with Kubernetes. Practically, this allows platform engineers to declare their networking goals, while network teams maintain control over enforcement. Each team operates within its expertise, either Kubernetes or Azure, yet they access a unified control interface. This isn’t merely a technical upgrade; it fosters organizational harmony. Specifically, it transforms ingress into a shared responsibility instead of a source of conflict.\nMulti-Cluster, Multi-Region, and Hybrid Thinking The future of Kubernetes involves multiple clusters rather than a single large one. As platforms mature, they tend to split into regional, isolated environments, or hybrid setups that integrate AKS with edge or on-prem workloads. Traditional ingress controllers weren’t built for this complexity, as they assumed a single cluster boundary.\nAGC, in contrast, considers multi-cluster ingress as a primary feature. Since it operates within Azure’s control plane, a single Application Gateway for Containers can handle traffic for multiple clusters. Each cluster registers its namespaces and routing rules, but all traffic passes through a single, centralized managed ingress layer.\nFor enterprises implementing hub-and-spoke architectures, this approach is a game-changer. Instead of setting up separate ingress controllers for each environment, you can deploy one or two AGC instances within a shared network hub and link multiple AKS clusters for development, testing, and production across regions via Private Link or VNet peering. This method not only enhances operational efficiency but also reinforces governance by maintaining a single ingress policy uniformly across all environments.\nHybrid environments also gain advantages. AGC can function as the cloud-side ingress for clusters connected through Azure Arc or hybrid links, providing consistent governance for edge or on-premises Kubernetes deployments. The same Gateway API manifests are used regardless of the cluster\u0026rsquo;s location. This multi-cluster capability makes AGC more than just an ingress controller; it’s a comprehensive network service platform for large-scale Kubernetes connectivity.\nRelationship to Other Azure Networking Services While it may be tempting to view AGC as an extension of the existing Application Gateway or as a competitor to Front Door or API Management, it actually serves a different purpose. Azure Front Door primarily handles global, Layer 7 routing at the edge, making it ideal for distributing traffic across multiple regions and delivering content. API Management is centered on managing developer-facing APIs, overseeing their lifecycle, and handling authentication. Application Gateway for Containers operates nearer to the cluster boundary, where internal services adhere to enterprise policies.\nThink of it as the “inner ingress” that complements Front Door’s “outer ingress.” In contemporary Azure setups, traffic typically moves from Front Door to AGC to AKS, with each layer managing a specific role: global routing, secure ingress, and workload processing. This layered structure follows the defense-in-depth principle. Each gateway provides additional visibility and control without repeating functions.\nSecurity as a Core Capability Security is integral to AGC’s design. It uses Application Gateway’s WAF engine to enable Layer 7 inspection at the managed ingress level. This allows organization-wide WAF policies to be enforced on containerized applications without requiring extra proxies or filters within the cluster.\nTLS termination at the gateway separates certificate management from workload lifecycles. Integration with Private Link and Virtual Network ensures traffic remains within your Azure boundary, supporting compliance with strict regulations such as NIS2 or DORA for European workloads.\nFrom a Zero Trust approach, AGC functions as a policy enforcement point, authenticating and inspecting traffic before it reaches your pods. When combined with workload identity and network policies in AKS, it contributes to a comprehensive end-to-end security strategy.\nThe Developer Experience: Declarative and Transparent One of AGC\u0026rsquo;s most elegant features is what developers don’t need to do. They aren\u0026rsquo;t required to learn new Azure-specific APIs or deploy agents. Instead, they work with standard Kubernetes resources like Gateway, HTTPRoute, and Backend; AGC simply interprets these. To developers, it remains Kubernetes; to operators, it’s Azure networking with built-in compliance and observability.\nThis transparency is essential because it supports the GitOps approach. Developers specify their desired state, store it in Git, and the platform ensures implementation with no manual work or requests to external load balancer teams.\nFor platform engineers, AGC completes the GitOps cycle by allowing network ingress to be expressed declaratively, versioned with application code, and managed through the cloud infrastructure.\nConceptual Example: From Policy to Flow Imagine a scenario: Your organization manages multiple AKS clusters, one for development in West Europe, one for staging in North Europe, and one for production in Central US. Previously, each cluster had its own ingress controller, TLS certificates, and slightly different configurations handled by different teams. Managing ingress inconsistencies took as much effort as deploying workloads.\nWith AGC, this changes: a single managed gateway in your hub VNet provides listeners for each environment. Each AKS cluster only defines its Gateway and Route resources, covering its namespace and services. Azure enforces organization-wide WAF policies, unified logging, and consistent TLS settings.\nDevelopment teams deploy YAML, while the platform ensures compliance. This results in a centrally managed ingress service that is locally consumed, a principle that made managed Kubernetes successful, now extended to Layer 7 networking.\nOperational Benefits in Perspective Operationally, AGC’s influence is subtle yet significantly impactful. It elevates ingress from a cluster-level issue to a platform-wide capability, leading to three key outcomes:\nEasier lifecycle management with no need for controller upgrades or image patching. Lower resource use, as ingress traffic no longer competes with the cluster for CPU or memory. A predictable cost model, where you pay for a managed service instead of facing unpredictable in-cluster overhead. It also simplifies incident response. When traffic anomalies happen, begin your investigation with Azure Monitor, correlate WAF logs, and trace requests back to backend pods. Troubleshooting ingress becomes a straightforward process rather than a multi-tool puzzle.\nThis isn’t about replacing every controller; it’s about understanding when ingress maturity requires a managed, policy-based approach.\nThe Future of Cloud-Native Ingress Looking ahead, Application Gateway for Containers (AGC) is more than just a new Azure service; it signifies a broader shift in cloud-native architecture. The trend is clear: shift operational complexity from the cluster to the managed layer, while keeping configurations declarative and portable.\nSimilar to how Azure Container Storage handles storage management and Azure Monitor manages observability pipelines, AGC manages the network edge. This reflects the same principle: enabling engineers to focus on intent rather than infrastructure.\nAs Kubernetes adopts the Gateway API as a standard, this approach is expected to become the default across cloud providers. Azure’s early focus on AGC gives it an advantage, blending open standards with platform stability.\nUltimately, AGC might serve as a unifying element connecting AKS, Container Apps, and Arc-enabled clusters, providing a consistent ingress experience across all Azure platforms.\nClosing Reflection Ingress has long been the quiet backbone of the Kubernetes ecosystem—imperceptible when functioning smoothly but glaringly obvious when issues arise. With Application Gateway for Containers, Azure redefines ingress as a strategic platform service rather than just an implementation detail.\nIt is Kubernetes-native, managed by Azure, declarative, policy-driven, flexible, and secure. This approach symbolizes the natural merging of networking, operations, and platform engineering, serving as a link between developer independence and enterprise oversight.\nIn many respects, AGC is more than just ingress; it reflects the future of how workloads are connected, secured, and monitored in a cloud-native environment, a world where developers specify intent and the platform.\n","permalink":"https://wolkwacht.nl/posts/application-gateway-for-containers-the-future-of-cloud-native-ingress/","summary":"\u003cp\u003e\u003cem\u003eExploring the concept behind Azure’s next-generation ingress platform\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*R82lqawI32GXuYnN2V7NQg.jpeg\"\u003e\u003c/p\u003e\n\u003ch2 id=\"when-ingress-becomes-strategic\"\u003eWhen Ingress Becomes Strategic\u003c/h2\u003e\n\u003cp\u003eThere’s a moment in every Kubernetes journey when ingress stops being just a configuration detail and becomes an architectural constraint. You start with a small cluster, maybe one or two microservices, and the standard NGINX ingress controller does exactly what you expect: simple routes, basic certificates, nothing fancy. Then the platform expands. You integrate workloads across multiple regions, extend environments with Azure Container Apps, and add a service mesh, and suddenly that “temporary ingress setup” becomes a bottleneck.\u003c/p\u003e","title":"Application Gateway for Containers: The Future of Cloud-Native Ingress"},{"content":"Fleet, Arc, sovereign clouds, and the edge-first world of 2030\nThis article is the last part of the Azure Kubernetes Chronicles: Multi-Cluster Edition.\nRead the series introduction here.\nExamining today’s multi-cluster landscape, it is tempting to focus solely on the pain points: identity fragmentation, policy drift, networking complexity, and monitoring silos. But the story does not end there. Kubernetes itself has evolved from a single-cluster dream to a multi-cluster necessity, and the governance ecosystem around it is growing just as quickly. To understand where things are headed, we have to look beyond today’s tools and into the future that is already taking shape.\nKubernetes was first released by Google in 2014 as an open-source system for orchestrating containers within a single cluster, providing scheduling, scaling, and workload management across a unified infrastructure. By around 2016–2018, as enterprise adoption accelerated, organizations began running multiple clusters to achieve regional availability, meet compliance needs, and support scalability, which introduced new challenges in networking, identity, and governance across environments. From 2019 onward, the landscape shifted further with the rise of hybrid and multi-cloud strategies, where clusters were deployed not only in private data centers but also across public clouds such as Azure Kubernetes Service, Amazon EKS, and Google GKE, as well as at the edge with lightweight distributions like k3s. Today, in 2025, Kubernetes has matured into a multi-cluster, multi-platform ecosystem, where interoperability and governance layers, such as Azure Arc, Anthos, and Fleet, enable enterprises to manage fragmented environments as a cohesive and unified platform.\nThe first and most obvious trajectory is the convergence of Fleet and Arc. Today, Fleet provides strong governance inside Azure, while Arc extends Azure’s control plane to clusters outside of it. But enterprises want more than two separate tools. They want a single experience that spans both. It is not hard to imagine Microsoft moving in this direction, Fleet handling workload placement and resource propagation across AKS, Arc extending governance and security across EKS, GKE, and the edge. Together, they could form the unified multi-cluster platform that enterprises have been waiting for.\nAt the same time, the European conversation around digital sovereignty is gaining momentum. Initiatives like GAIA-X, along with sovereign providers such as OVHcloud and Deutsche Telekom, are becoming part of enterprise strategies. The goal is not to abandon hyperscalers, but to prove independence from them. In this landscape, enterprises may find themselves running AKS in Amsterdam, EKS in Frankfurt, and a GAIA-X-aligned cluster in Paris, all of which will need to be governed as a whole. The pressure will grow for platforms like Arc or Anthos to integrate sovereign clouds alongside global ones, ensuring that sovereignty does not lead to fragmentation.\nIs OVHcloud a viable option because of its presence in the United States?\nOVHcloud is headquartered in France and positions itself as a European sovereign cloud provider, emphasizing EU data residency and certifications such as SecNumCloud. While the U.S. CLOUD Act allows American authorities to compel U.S.-based providers to hand over data stored abroad, OVHcloud argues that only its U.S. subsidiary, OVH US, falls under that law. The European entity (OVHcloud in the EU) operates separately and is not subject to U.S. jurisdiction, meaning that data stored and processed entirely within its European infrastructure remains governed solely by European law. In practice, OVHcloud is a viable option for a European sovereign cloud, provided customers contract with and use only its European services and avoid any ties to its U.S. operations.\nThe edge explosion is the third force reshaping the future. Analysts predict that by 2030, the majority of enterprise data will be processed outside of traditional data centers. That means retail stores with their own Kubernetes clusters, factories running predictive maintenance models locally, hospitals analyzing imaging on-site, and telecom towers hosting thousands of micro data centers. Edge will not replace cloud, but it will outnumber it. Instead of managing a handful of hyperscale clusters, enterprises will be managing hundreds or thousands of small, distributed ones. Governance platforms will need to scale in a new way, not just deeper into Azure and AWS, but wider across thousands of edge locations.\nFinally, compliance itself will evolve. Instead of being an afterthought, regulations such as GDPR, NIS2, and DORA will be baked into platforms from the start. Enterprises will no longer bolt on compliance tools; instead, they will adopt predefined blueprints, hardened configurations, and automated audits. The future of Kubernetes governance will shift from proving compliance after the fact to operating within compliance by design.\nTaken together, these trends indicate a world where multi-cluster Kubernetes is not just the norm, but a foundational aspect. Fleet and Arc will merge into something greater. Sovereign clouds will become part of everyday enterprise architecture. Edge will redefine the scale and distribution of clusters. And compliance will shift from an obstacle to an embedded feature of the platform.\nMulti-cluster is no longer the messy truth of today. It is the strategic architecture of tomorrow. Enterprises that embrace this reality will not only survive regulatory and operational pressure but also position themselves for innovation in a world where cloud and edge are inseparable.\nPrevious: Azure Kubernetes Chronicles 11: Governing the Chaos\n","permalink":"https://wolkwacht.nl/posts/azure-kubernetes-chronicles-12-the-future-of-multi-cluster/","summary":"\u003cp\u003e\u003cem\u003eFleet, Arc, sovereign clouds, and the edge-first world of 2030\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*GBpQfjLqc4oxkCeqsO96SA.png\"\u003e\u003c/p\u003e\n\u003cp\u003eThis article is the last part of the Azure Kubernetes Chronicles: Multi-Cluster Edition.\u003cbr\u003e\nRead the series introduction \u003ca href=\"https://medium.com/@jurgenallewijn/azure-kubernetes-chronicles-multi-cluster-edition-0dab3518d297\"\u003ehere\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eExamining today’s multi-cluster landscape, it is tempting to focus solely on the pain points: identity fragmentation, policy drift, networking complexity, and monitoring silos. But the story does not end there. Kubernetes itself has evolved from a single-cluster dream to a multi-cluster necessity, and the governance ecosystem around it is growing just as quickly. To understand where things are headed, we have to look beyond today’s tools and into the future that is already taking shape.\u003c/p\u003e","title":"Azure Kubernetes Chronicles 12: The Future of Multi-Cluster"},{"content":"\nAzure Kubernetes Service (AKS) has long been the leading platform for containerized workloads on Microsoft Azure. While AKS offers scalability and resilience, managing it day-to-day still involves switching between various tools: Azure CLI, kubectl, ARM templates, and portal dashboards.\nThis is where the Microsoft AKS-MCP Server, part of the Model Context Protocol (MCP) ecosystem, makes a difference. By providing a structured, secure interface for agents and tools to perform Azure and Kubernetes actions, AKS-MCP changes how engineers interact with clusters.\nIt acts as a bridge between AI agents and operational activities, enabling conversational, secure, and auditable control over AKS resources.\nThis article covers installing and configuring the Microsoft AKS-MCP Server, validating it using live code examples, and demonstrating real operational use cases. We focus on practical engineering insights rather than marketing hype, providing an expert, reflective perspective on how it integrates with your existing DevOps and platform workflows.\nThe Case for MCP in Kubernetes Operations If you’ve ever wished to ask an intelligent assistant to “show me all failing pods in production” or “scale node pool np1 to five nodes,” you’re describing a perfect scenario for the Model Context Protocol.\nMCP provides a standardized method for external agents, such as AI models or ChatOps bots, to access functions and tools with structured, permissioned access. Microsoft’s implementation extends this protocol to Azure services, specifically now to AKS. Instead of giving an AI system shell access or open API tokens, you can set up an MCP Server that only exposes the specific capabilities you choose, such as listing clusters, applying manifests, retrieving pod logs, or scaling node pools.\nFor DevOps or platform engineers, this allows automation of insights, empowerment of internal bots, or enhancement of developer platforms, all while maintaining control and compliance. The AKS-MCP Architecture\nAKS-MCP is essentially a middleware that connects your intelligent clients to your Azure environment. It leverages the Azure SDK’s DefaultAzureCredential, enabling authentication through existing credentials or Managed Identities, and securely interacts with both Azure Resource Manager and the Kubernetes API.\nArchitectural view of that flow\nWhen a user or agent issues a natural-language command like “List node pools in my production cluster,” the process unfolds as follows:\nThe agent interprets the intent and constructs a call based on the MCP specification. This call is sent through supported transports such as stdio or HTTP. The AKS-MCP Server processes the call and uses the Azure and Kubernetes SDKs to execute the request. The results are returned as structured JSON to the agent for visualization or additional analysis. This architectural separation establishes clear boundaries: the agent never directly accesses your cluster, credentials stay within Azure’s identity system, and all operations can be logged and audited.\nDeploying the Microsoft AKS-MCP Server The AKS-MCP Server can be deployed in various ways: locally for testing, containerized within a management cluster, or hosted through Azure Container Apps. Below is an example of an installation pattern suitable for production deployments.\nPrerequisites You’ll need:\n• An active Azure subscription\n• One or more AKS clusters\n• Azure CLI, Helm, and kubectl installed\n• A service principal or Managed Identity with Contributor or AKS Cluster Admin roles\n• Network egress from the MCP pod to Azure APIs and your AKS API server\nStep 1: Obtain the Source and Container Image Clone the official repository:\ngit clone https://github.com/Azure/aks-mcp.git cd aks-mcp You can either build your own image or use the one published by Microsoft:\ndocker build -t myregistry.azurecr.io/aks-mcp:latest . docker push myregistry.azurecr.io/aks-mcp:latest Alternatively, deploy direct from GitHub Container Registry:\nhelm repo add aks-mcp https://raw.githubusercontent.com/Azure/aks-mcp/main/chart/ Step 2: Create an Azure Service Principal (optional) If you prefer explicit credentials over Managed Identity, create a service principal.\naz ad app create --display-name aks-mcp-server az ad app credential reset --id \u0026lt;appId\u0026gt; --credential-description \u0026#34;mcp\u0026#34; --years 2 az role assignment create --assignee \u0026lt;appId\u0026gt; --role Contributor --scope /subscriptions/\u0026lt;subscriptionId\u0026gt; Storing the generated credentials as a Kubernetes Secret:\napiVersion: v1 kind: Secret metadata: name: azure-credentials namespace: mcp type: Opaque stringData: tenantId: \u0026#34;\u0026lt;tenantId\u0026gt;\u0026#34; clientId: \u0026#34;\u0026lt;appId\u0026gt;\u0026#34; clientSecret: \u0026#34;\u0026lt;password\u0026gt;\u0026#34; subscriptionId: \u0026#34;\u0026lt;subscriptionId\u0026gt;\u0026#34; Step 3: Deploy via Helm helm install aks-mcp aks-mcp/aks-mcp --namespace mcp --create-namespace \\ --set image.repository=myregistry.azurecr.io/aks-mcp \\ --set image.tag=latest \\ --set env.azureTenantIdSecretName=azure-credentials \\ --set env.azureClientIdSecretName=azure-credentials \\ --set env.azureClientSecretSecretName=azure-credentials \\ --set env.azureSubscriptionIdSecretName=azure-credentials Once deployed, verify:\nkubectl get pods -n mcp kubectl logs -n mcp deployment/aks-mcp You should see log output confirming startup and readiness.\nStep 4: Connecting a Client Clients such as GitHub Copilot Agents or custom applications connect through MCP configuration files.\nExample .mcp.json:\n{ \u0026#34;mcpServers\u0026#34;: { \u0026#34;aks-mcp\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;stdio\u0026#34;, \u0026#34;command\u0026#34;: \u0026#34;/usr/local/bin/aks-mcp\u0026#34;, \u0026#34;args\u0026#34;: [\u0026#34;--transport=stdio\u0026#34;] } } } For HTTP exposure:\n{ \u0026#34;mcpServers\u0026#34;: { \u0026#34;aks-mcp\u0026#34;: { \u0026#34;url\u0026#34;: \u0026#34;https://aks-mcp.mycompany.com\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;http\u0026#34; } } } Test Connectivity:\nkubectl port-forward svc/aks-mcp 8080:8080 -n mcp Then run a local MCP client to send test requests.\nUsing AKS-MCP in Real Operations Once installed, the AKS-MCP Server seamlessly integrates into your daily operations. It’s more than just experimental technology; it acts as a productivity layer that enhances observability, remediation, and governance right where engineers work and make decisions.\nDiagnostics and Cluster Insight Imagine a ChatOps scenario:\n“Show me all AKS clusters in my subscription, then list pods in namespace payments that are not ready.”\nThe agent issues a sequence of MCP calls:\nfrom azure.mcp.client import MCPClient client = MCPClient(server_address=\u0026#34;http://localhost:8080\u0026#34;) clusters = client.invoke(\u0026#34;aks.getClusters\u0026#34;) pods = client.invoke(\u0026#34;aks.getPods\u0026#34;, {\u0026#34;cluster\u0026#34;: \u0026#34;prod-aks\u0026#34;, \u0026#34;namespace\u0026#34;: \u0026#34;payments\u0026#34;}) print(pods) The response is provided in JSON format, which your bot or console can then display as human-readable text.\nScaling and Resource Management For autoscaling or temporary scaling, instead of manually executing az aks nodepool scale, the agent can use the following command:\nres = client.invoke(\u0026#34;aks.scaleNodePool\u0026#34;, { \u0026#34;cluster\u0026#34;: \u0026#34;prod-aks\u0026#34;, \u0026#34;resourceGroup\u0026#34;: \u0026#34;rg-prod\u0026#34;, \u0026#34;nodePool\u0026#34;: \u0026#34;np1\u0026#34;, \u0026#34;newCount\u0026#34;: 5 }) The AKS-MCP Server translates this into ARM API calls and returns confirmation.\nRolling Deployments via Conversational Commands A natural conversation with an internal assistant might look like:\n“Deploy version v2.1.0 of orders-api to staging and verify rollout success.”\nThe agent decomposes this into steps:\n1. Apply the manifest\n2. Monitor rollout status\n3. Fetch logs for failed pods\nclient.invoke(\u0026#34;aks.applyManifest\u0026#34;, { \u0026#34;cluster\u0026#34;: \u0026#34;stg-aks\u0026#34;, \u0026#34;namespace\u0026#34;: \u0026#34;orders\u0026#34;, \u0026#34;manifest\u0026#34;: open(\u0026#34;orders-api-v2.1.0.yaml\u0026#34;).read() }) status = client.invoke(\u0026#34;aks.getRolloutStatus\u0026#34;, { \u0026#34;cluster\u0026#34;: \u0026#34;stg-aks\u0026#34;, \u0026#34;namespace\u0026#34;: \u0026#34;orders\u0026#34;, \u0026#34;deployment\u0026#34;: \u0026#34;orders-api\u0026#34; }) If the rollout fails, it calls getPodLogs automatically.\nThe engineer gets a narrative response instead of raw log output, such as:\n“Two pods in orders-api failed to start. Container api shows CrashLoopBackOff due to image pull error.”\nThat’s actionable intelligence instead of static output.\nGovernance and Security Reviews Since AKS-MCP exposes controlled functions, it can be securely integrated with internal governance bots.\nA compliance bot could ask:\n“List all cluster roles that allow delete privileges.”\nThe server queries the Kubernetes API and returns results, enabling the bot to suggest solutions. In hybrid environments, combine AKS-MCP with the general Azure MCP server to cross-check resource roles at both the Azure RBAC and Kubernetes RBAC levels.\nIntegrating into Platform Engineering Workflows For platform teams creating self-service portals or internal developer platforms (IDPs), AKS-MCP can serve as the core of conversational automation. Instead of revealing raw pipelines, developers can request infrastructure changes in simple English, with the MCP layer ensuring policy compliance and identity verification.\nThis approach integrates well with GitOps workflows, as MCP can initiate pull requests or run kubectl apply commands under supervision, connecting human intent with automated deployment.\nOperational Security Considerations AKS-MCP functions within your trusted Azure identity boundary, but it can still execute commands that change resources. To ensure security: • Always use Managed Identity for authentication.\n• Reduce network exposure by placing AKS-MCP in a private subnet or using internal ingress only.\n• Track API activity and MCP requests in Log Analytics.\n• Apply request throttling and validate inputs.\n• Enforce role segregation with readonly, readwrite, and admin tiers.\n• Keep comprehensive logs of all actions, detailing who or what performed them.\nThese measures align with Zero Trust principles and preserve traceability.\nExample: End-to-End Workflow in Action Here’s an example conversation an internal AI assistant might manage:\nEngineer: “Increase the payments node pool in production to six nodes.” MCP Client: Interprets the intent and constructs a scaleNodePool request. AKS-MCP Server: Checks permissions, then calls Azure ARM to execute the scaling. Result: Receives structured JSON indicating “Scale operation succeeded.” Shortly after, the same assistant can verify:\n“List node counts and running pods in production.”\nMCP calls getNodePools → getPods → aggregates → responds with counts.\nThis combination of agent reasoning and MCP enforcement provides teams with conversational control while maintaining governance.\nWhere AKS-MCP Fits in the Broader Azure Ecosystem AKS-MCP does not replace your CI/CD pipelines, Infrastructure as Code (IaC) templates, or monitoring systems; instead, it complements them. Think of it as a new operational interface that enables intent-based, just-in-time actions without bypassing existing policies.\nFor example: • Terraform or Bicep remain your declarative configuration tools.\n• Pipelines continue to perform standard deployments.\n• MCP servers manage ad-hoc operations, diagnostics, or self-service requests, all logged, identity-aware, and reversible.\nThe integration of MCP and AI assistants is poised to transform DevOps tools. By 2026, most enterprise clouds will likely have internal copilots relying on MCP connectors like this one.\nObservability and Auditing Each call handled by AKS-MCP can generate telemetry data. By sending logs to Azure Monitor or a centralized Log Analytics workspace, you obtain:\n• Complete audit records of AI-driven actions\n• Metrics on invocation frequency\n• Insights into latency and error rates\n• Custom dashboards that connect AI agent activity with cluster operations\nA recommended method is to tag all AKS-MCP pods with env=operations and export logs using ContainerInsights. You can then analyze usage patterns through Kusto queries, for example:\n# Kusto query ContainerLog | where Name == \u0026#34;aks-mcp\u0026#34; | project TimeGenerated, LogEntry, Computer | sort by TimeGenerated desc That provides immediate visibility into how your AI agents interact with your clusters.\nFuture Directions The current AKS-MCP Server already supports most essential AKS and Kubernetes functions, but upcoming versions are expected to feature: • Integration with Azure Policy for real-time enforcement\n• Built-in workload validation (similar to OPA or Gatekeeper)\n• Event-driven triggers for scaling and failover\n• A secure plugin architecture for custom tools\nAs it develops, broader adoption is likely in regulated industries requiring conversational automation with strict controls.\nReferences • Microsoft Learn: Get Started with Azure MCP Server\n• GitHub: Azure/aks-mcp\n• Blog: Announcing the AKS-MCP Server\n• TechCommunity: Deploying MCP Server Using Azure Container Apps\n• Medium: Automating Troubleshooting with Azure MCP\nConclusion The Microsoft AKS-MCP Server marks a subtle yet significant shift in managing Kubernetes on Azure. It doesn\u0026rsquo;t aim to replace engineers but to enhance their intentions by ensuring every command passes through an intelligent, policy-aware layer.\nFrom cluster scaling to rollout diagnostics, MCP makes the cloud more conversational, contextual, and compliant. In daily operations, it’s the difference between just finding the correct command and achieving the desired outcome.\nFor platform engineers and architects, this signals the start of a new phase: orchestrating not only containers but also context.\n","permalink":"https://wolkwacht.nl/posts/using-the-aks-mcp-server-in-day-to-day-operations/","summary":"\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*mnhzpOigE5A3FQamHVT-kA.png\"\u003e\u003c/p\u003e\n\u003cp\u003eAzure Kubernetes Service (AKS) has long been the leading platform for containerized workloads on Microsoft Azure. While AKS offers scalability and resilience, managing it day-to-day still involves switching between various tools: Azure CLI, kubectl, ARM templates, and portal dashboards.\u003c/p\u003e\n\u003cp\u003eThis is where the Microsoft AKS-MCP Server, part of the Model Context Protocol (MCP) ecosystem, makes a difference. By providing a structured, secure interface for agents and tools to perform Azure and Kubernetes actions, AKS-MCP changes how engineers interact with clusters.\u003c/p\u003e","title":"Using the AKS-MCP Server in Day-to-Day Operations"},{"content":"Azure Kubernetes Chronicles 11: Governing the Chaos Unifying identity, policy, networking, and observability across clouds and edge\nThis article is part of the Azure Kubernetes Chronicles: Multi-Cluster Edition.\nRead the series introduction here.\nGovernance pain points By the time enterprises have stretched Kubernetes across Azure, AWS, and the edge, the real problem is no longer the technology itself. It is governance. What began as a strategy to increase resilience and satisfy compliance quickly turns into a patchwork of identity systems, policy frameworks, networking models, and monitoring stacks. The result is fragmentation. Platform teams are left trying to stitch together three or more worlds, often with tools that were never designed to coexist.\nIdentity is often the very first thing to be challenged or questioned. In Azure, AKS integrates tightly with Entra ID, providing enterprises with a direct mapping from organizational roles to Kubernetes permissions. In AWS, EKS relies on IAM, with its own syntax, semantics, and mapping mechanisms. At the edge, lightweight distributions like k3s often default to local certificates and kubeconfig files. For developers moving between clusters, the experience is inconsistent. For security teams, enforcing least privilege becomes a nightmare. What looks unified on a whiteboard quickly splinters in reality.\nPolicy follows a similar trajectory. Within Azure, administrators can use Azure Policy for Kubernetes to apply rules at scale across clusters managed by Fleet or Arc. In EKS, there is no AWS-native equivalent. Teams must rely on open-source tools such as OPA or Kyverno to enforce the same standards. At the edge, policy enforcement is often manually bolted on, if it is present at all. Compliance auditors inevitably notice the gaps. A Pod Security restriction applied in one place but absent in another is enough to trigger uncomfortable questions.\nNetworking is the most visible source of complexity. Azure relies on VNets, with AKS clusters often configured with Azure CNI or Cilium. AWS EKS uses VPC CNI and a different model for assigning IP addresses. At the edge, k3s defaults to flannel, though it can be reconfigured to run Calico or Cilium. The differences are not cosmetic; they affect service discovery, IP address management, and cross-cluster communication. Some enterprises attempt to standardize with service meshes, while others try global DNS overlays. Few are fully satisfied.\nObservability completes the picture. In Azure, Container Insights and Azure Monitor provide a polished experience, but only for Azure-native clusters. In AWS, CloudWatch reigns, with its own dashboards and metrics model. At the edge, Prometheus and Grafana are the common choices, but they require self-management and rarely integrate cleanly with hyperscaler tools. The result is silos. Operations teams bounce between three or four monitoring stacks, none of which provide a true single pane of glass. Troubleshooting a cross-cloud outage becomes less a matter of engineering and more a matter of detective work.\nThese fractures are not theoretical. They show up in board reports, audit findings, and late-night incidents. They create operational drag and erode confidence in platform teams. And they push enterprises toward solutions that promise unification.\nPossible directions for solutions One option is Azure Arc, which extends Azure governance across AKS, EKS, GKE, and even edge clusters, such as k3s. With Arc, Entra ID can become the identity plane across all clusters. Azure Policy can apply consistently, regardless of where the cluster runs. Logs and metrics can flow back into Azure Monitor, giving a central view. Arc does not eliminate the complexity of multi-cloud, but it does normalize it into a governance model that auditors can understand.\nAnother option is Google Anthos, which was among the first to enter the cross-cloud management market. Anthos offers config sync, service mesh integration, and policy enforcement across GKE, AKS, and EKS. It is powerful, but it often feels Google-centric and comes with a cost structure that enterprises weigh carefully.\nGoogle Anthos is a platform for managing applications across hybrid and multi-cloud environments, helping organizations modernize existing apps, develop new ones, and run them securely anywhere — whether on Google Cloud, on-premises, or other clouds like AWS and Azure. Built on Kubernetes and open-source tools such as Istio and Knative, Anthos provides a unified framework for managing containerized applications, policies, and security across various environments. Its main benefits include preventing vendor lock-in, enhancing operational efficiency through centralized management, and facilitating digital transformation by standardizing application deployment across different infrastructure types.\nThe third option is to go DIY (Do-It-Yourself) with open-source tooling. Service meshes, such as Istio or Linkerd, and policy engines, like OPA, can be combined with observability stacks like Prometheus and Grafana to form a cross-cloud framework. This approach offers the most independence, but it also requires the most expertise. Few enterprises are willing to carry that burden without a strong internal engineering culture.\nIn practice, enterprises often combine these approaches. Fleet may govern AKS clusters inside Azure. Arc may extend that governance to AWS and the edge. Service meshes may abstract away networking differences. Open-source observability may provide a neutral layer on top of it. The goal is not perfection, but coherence. A governance framework that is strong enough to withstand audits, resilient enough to survive outages, and simple enough for platform teams to operate.\nGovernance is not glamorous. It does not make headlines or attract the same enthusiasm as AI or serverless platforms. But in the world of multi-cluster Kubernetes, governance is what makes the difference between a fragile patchwork and a robust enterprise strategy. Without it, identity fragments, policies drift, networks split, and observability fails. With it, the chaos becomes manageable.\nIn the next episode of the Azure Kubernetes Chronicles, we will look beyond today’s fractures and into the future. We’ll explore how Microsoft might merge Fleet and Arc, how sovereign cloud initiatives in Europe could reshape strategies, and how the edge explosion will redefine what “multi-cluster” means by 2030.\nPrevious: Azure Kubernetes Chronicles 10: Kubernetes at the Edge with k3s Next up: Azure Kubernetes Chronicles 12: The Future of Multi-Cluster\n","permalink":"https://wolkwacht.nl/posts/azure-kubernetes-chronicles-11-governing-the-chaos/","summary":"\u003ch2 id=\"azure-kubernetes-chronicles-11-governing-thechaos\"\u003eAzure Kubernetes Chronicles 11: Governing the Chaos\u003c/h2\u003e\n\u003cp\u003e\u003cem\u003eUnifying identity, policy, networking, and observability across clouds and edge\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*mTw2gBn5HroLM9-b5amaMQ.png\"\u003e\u003c/p\u003e\n\u003cp\u003eThis article is part of the Azure Kubernetes Chronicles: Multi-Cluster Edition.\u003cbr\u003e\nRead the series introduction \u003ca href=\"https://medium.com/@jurgenallewijn/azure-kubernetes-chronicles-multi-cluster-edition-0dab3518d297\"\u003ehere\u003c/a\u003e.\u003c/p\u003e\n\u003ch2 id=\"governance-painpoints\"\u003eGovernance pain points\u003c/h2\u003e\n\u003cp\u003eBy the time enterprises have stretched Kubernetes across Azure, AWS, and the edge, the real problem is no longer the technology itself. It is governance. What began as a strategy to increase resilience and satisfy compliance quickly turns into a patchwork of identity systems, policy frameworks, networking models, and monitoring stacks. The result is fragmentation. Platform teams are left trying to stitch together three or more worlds, often with tools that were never designed to coexist.\u003c/p\u003e","title":"Azure Kubernetes Chronicles 11: Governing the Chaos"},{"content":"Azure Kubernetes Chronicles 10: Kubernetes at the Edge with k3s From Raspberry Pi retail stores to 5G towers — the hidden force reshaping enterprise IT\nThis article is part of the Azure Kubernetes Chronicles: Multi-Cluster Edition.\nRead the series introduction here.\nWhen most people think of Kubernetes, they imagine large clusters humming inside hyperscale cloud regions, backed by the weight of Azure, AWS, or Google. However, the future of Kubernetes is increasingly not limited to vast data centers, but also extends to much smaller, more distributed locations, such as retail stores, factory floors, hospital wards, or 5G towers. At the core of this movement is k3s, a lightweight Kubernetes distribution that is small enough to run on a Raspberry Pi but robust enough to support enterprise workloads.\nThe rise of the edge is not a gimmick. A combination of latency, resilience, privacy, and operational efficiency is driving it. In retail, point-of-sale systems and video analytics must continue to function even when the internet connection is disrupted. In manufacturing, robotic arms and predictive maintenance models demand millisecond-level responses that a round-trip to the cloud cannot guarantee. In healthcare, privacy laws such as the GDPR require sensitive data, including medical imaging, to be processed locally rather than uploaded to a public cloud. In telecommunications, thousands of micro data centers are being deployed at cell towers to power 5G networks and AR/VR applications. These use cases all converge on the same conclusion: cloud alone is not enough. Enterprises need Kubernetes at the edge.\nk3s is uniquely suited to this role. It is lightweight, requiring only minimal resources; a single binary that can run on a device with as little as 512 MB of memory. It is simple to install, update, and manage, making it accessible even in environments with limited IT support. At the same time, it remains a fully CNCF-certified distribution, which means manifests written for AKS or EKS can often run on k3s without modification. For enterprises, this compatibility matters. It allows developers to build once and deploy anywhere, from the largest Azure cluster to the smallest Raspberry Pi.\nk3s is a compact, certified Kubernetes variant created by Rancher (now part of SUSE). It emphasizes simplicity and efficiency, particularly in edge computing, IoT, and resource-constrained environments. The distribution removes unnecessary components, consolidates the control plane into a single binary, and reduces memory and CPU usage, making it straightforward to deploy on devices such as Raspberry Pi, small servers, or virtual machines. Despite its streamlined design, k3s fully aligns with upstream Kubernetes, ensuring workloads and configurations are compatible across k3s and standard Kubernetes clusters. This provides a practical solution for testing, edge applications, and small-scale production without the complexity of full Kubernetes distributions.\nThe use cases are already everywhere. Retail chains are experimenting with in-store k3s clusters to keep sales terminals running independently of central systems, while also hosting AI models that analyze customer behavior or monitor queues. Factories are deploying k3s nodes next to machines, where they run telemetry aggregation, predictive maintenance workloads, and robotics controllers. Hospitals are building small-scale edge clusters to process imaging locally and monitor patients in real time, ensuring that sensitive data never leaves the building. Telecom providers are deploying Kubernetes at thousands of 5G sites, creating a standardized layer that runs close to customers and supports ultra-low-latency applications.\nBut as with multi-cloud, the benefits of edge Kubernetes come with challenges. Running a single Raspberry Pi cluster is easy. Running one thousand across a network of retail stores is not. Governance quickly becomes a headache. How do you update and patch so many distributed nodes? How do you ensure consistent role-based access controls when devices may be physically exposed to unauthorized access? How do you maintain observability when half your fleet is offline at any given moment? How do you keep policies consistent across hundreds or thousands of tiny clusters?\nThis is where platforms like Azure Arc begin to show their value. Arc allows organizations to onboard edge clusters into the same governance framework as their cloud clusters. Policies can be applied uniformly. Observability can be sent back to Azure Monitor. Security controls can extend all the way down to a Raspberry Pi in a store. In essence, Arc provides the missing control plane for edge Kubernetes, making the edge an extension of the enterprise rather than a disconnected experiment.\nFor CIOs and architects, k3s represents more than a toy for enthusiasts. It is proof of the edge-cloud convergence. Kubernetes is no longer just a cloud technology; it has become a unifying operational model that extends down to the smallest devices. It is a compliance enabler that reduces risk by keeping sensitive data local. It is an innovation sandbox, allowing enterprises to test new ideas at a low cost before scaling them globally.\nThe rise of edge Kubernetes shows that the future of enterprise IT is not only multi-cloud but also multi-location. Managing clusters across regions and providers will not be enough; platform teams will also need to master governance at the edge. The question is no longer whether enterprises will adopt edge clusters, but how they will integrate them into a consistent strategy.\nIn the next episode of the Azure Kubernetes Chronicles, we’ll turn to the central challenge that unites all of these threads: governance. From identity fragmentation to policy drift, and from networking complexity to observability silos, we’ll examine the complicated problems of running Kubernetes across clouds and edge, as well as the solutions that are beginning to emerge.\nPrevious: Azure Kubernetes Chronicles 9: EKS in the Enterprise\nNext up: Azure Kubernetes Chronicles 11: Governing the Chaos\n","permalink":"https://wolkwacht.nl/posts/azure-kubernetes-chronicles-10-kubernetes-at-the-edge-with-k3s/","summary":"\u003ch2 id=\"azure-kubernetes-chronicles-10-kubernetes-at-the-edge-withk3s\"\u003eAzure Kubernetes Chronicles 10: Kubernetes at the Edge with k3s\u003c/h2\u003e\n\u003cp\u003e\u003cem\u003eFrom Raspberry Pi retail stores to 5G towers — the hidden force reshaping enterprise IT\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*-QcIHUQsPkC2G-u32HnlNQ.png\"\u003e\u003c/p\u003e\n\u003cp\u003eThis article is part of the Azure Kubernetes Chronicles: Multi-Cluster Edition.\u003cbr\u003e\nRead the series introduction \u003ca href=\"https://medium.com/@jurgenallewijn/azure-kubernetes-chronicles-multi-cluster-edition-0dab3518d297\"\u003ehere\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eWhen most people think of Kubernetes, they imagine large clusters humming inside hyperscale cloud regions, backed by the weight of Azure, AWS, or Google. However, the future of Kubernetes is increasingly not limited to vast data centers, but also extends to much smaller, more distributed locations, such as retail stores, factory floors, hospital wards, or 5G towers. At the core of this movement is \u003ca href=\"https://k3s.io\"\u003ek3s\u003c/a\u003e, a lightweight Kubernetes distribution that is small enough to run on a Raspberry Pi but robust enough to support enterprise workloads.\u003c/p\u003e","title":"Azure Kubernetes Chronicles 10: Kubernetes at the Edge with k3s"},{"content":"Running Containers the Apple Way: A First Look into Apple Container on macOS For years, running containers on macOS has involved compromises. Docker Desktop used HyperKit to virtualize Linux, which was simple to use but resource-intensive. Alternatives like Rancher Desktop and OrbStack improved on this by being lighter, faster, and more developer-friendly, yet they all depended on the same core approach: emulating or virtualizing Linux within macOS.\nApple has now entered this space with a native solution: Apple Container, an OCI-compliant container runtime explicitly designed for Apple Silicon and macOS 15/16 and later. It is not simply a re-skin of Docker or a clone of Lima or Colima. Instead, it represents Apple’s unique approach to how containers should operate on macOS, using the Virtualization and Network frameworks that power iOS simulators and macOS sandboxing.\nFrom Virtual Machines to Containers on macOS On macOS, every container runtime faces a common challenge: containers are Linux processes that depend on features like cgroups, namespaces, and kernel functionalities, which macOS lacks.\nThe typical solution involves using a lightweight Linux virtual machine (VM).\nDocker Desktop employs HyperKit. Rancher Desktop incorporates Lima and QEMU. OrbStack features an optimized virtual machine layer directly integrated with macOS networking. Apple Container employs a more streamlined approach by leveraging Apple’s Virtualization Framework, which provides hypervisor and networking APIs natively supported on macOS. This results in fewer translation layers, quicker boot times, and no reliance on third-party hypervisors. When you run “container run alpine,” it’s not a large VM. It’s a small, purpose-built micro-VM managed by macOS specifically for container workloads. The virtualization operates natively, with virtio-based I/O and macOS’s networking stack handling the interfaces. Installing Apple Container\nThe installation process is surprisingly clean, although it currently only supports Apple Silicon hardware. You can install it manually using the signed package from Apple’s GitHub repository.\n# Check macOS version sw_vers # Download and install curl -L -o container.pkg https://github.com/apple/container/releases/latest/download/container.pkg sudo installer -pkg container.pkg -target / # Start the container system container system start # Verify container version The installation places binaries in /usr/local/bin and sets up a background service called containersd, which is Apple’s minimal equivalent to Docker’s dockerd. You can list or inspect it using launchctl list | grep containerd.\nRunning Containers After installation, the workflow resembles that of Docker or Podman. Since Apple Container adheres to the OCI standard, you can pull and run existing images without any changes.\n# Pull a standard image and run it container run alpine echo \u0026#34;Hello from Apple Container\u0026#34; Each invocation spins up a native Linux environment running in a micro-VM, with process isolation managed through Apple’s Virtualization API. Networking behaves predictably, too. Apple Container integrates with the host’s native interface stack, rather than bridging through HyperKit.\nThe logs, filesystem overlays, and image caches live under ~/Library/Containers/com.apple.containerd/, and you can inspect them using container image list or container ps.\nComparing the Ecosystem For DevOps engineers, the question isn’t “does it run containers?” but “how does it fit into my workflow?”\nDocker Desktop remains the default choice for enterprise teams. It offers a mature ecosystem, graphical dashboards, Kubernetes integration, and secure credential management. However, it’s resource-intensive and requires a license for business use.\nRancher Desktop provides greater openness. It utilizes Lima and containerd internally, supports both nerdctl and Docker CLI modes, and offers built-in Kubernetes clusters through k3s. It’s reliable for developers seeking control, but it may feel slower due to its multi-layer virtualization.\nOrbStack has claimed the developer experience crown. It’s lightweight, fast, and deeply integrated with macOS file systems and networking. OrbStack also supports Linux VMs and Docker images within a single UI, making local Kubernetes or Compose workloads feel seamless and intuitive.\nApple Container, on the other hand, feels minimalistic. It lacks a GUI, Kubernetes support, Compose, or swarm mode. It’s simply the container runtime. Yet, this simplicity is also its strength—it’s lean, native, and fast. It provides a clean starting point for developers who want to embed containers directly into macOS, rather than tacking them on.\nWhat’s Missing The initial release of Apple Container is primarily a proof of concept rather than a full production tool. It currently lacks volume mounting options aside from basic bind mounts and does not include a Kubernetes layer. Users cannot orchestrate multi-container applications or manage networks beyond the default bridge.\nMost DevOps engineers rely on Docker Compose or Helm for setup, but these are not available here. To run multi-container workloads, users must manually script container run commands or use buildctl and nerdctl to integrate the runtime once the API matures.\nAdditionally, there is no image build subsystem yet; Docker’s BuildKit and Podman’s build engine are not yet available. Although Apple’s roadmap mentions OCI build support, it is still in the early stages. Use Cases for Apple Container\nDespite its simplicity, Apple Container presents clear use cases where it already proves useful. It’s perfect for local development testing when you need to verify OCI-compliant images without Docker’s overhead. It’s also valuable for CI/CD agents on Apple hardware, especially where Docker Desktop licensing is restrictive.\nWith Apple Container, you can execute temporary build tasks or validate images directly, minimizing virtualization. Security-conscious developers will value its integration with macOS sandboxing and operation under Apple’s entitlement system, without requiring extra kernel extensions or privileged daemons. For some regulated development scenarios, this offers a significant advantage.\nExample: Running a Web App in Apple Container Here’s a minimal test you can try to see how it behaves under load.\n# Run a small HTTP server container run --publish 8080:8080 python:3.12-slim \\ python -m http.server 8080 Then, open http://localhost:8080 and you’ll see the directory listing of the working directory inside the container. You can inspect active sessions with \u0026lsquo;container ps\u0026rsquo; and stop them with \u0026lsquo;container kill \u0026lt;id\u0026gt;\u0026rsquo;.\nTo verify network stack behavior, try pinging the host from within the container — you’ll see that Apple Container uses its own isolated subnet managed by the Virtualization framework rather than a Docker bridge.\nIntegration with Kubernetes (Future) Currently, Apple Container does not offer a CRI (Container Runtime Interface) endpoint, so it cannot serve as a backend for Kubernetes. However, from an architectural perspective, nothing prevents it from supporting this functionality in future releases.\nIf Apple chooses to include CRI support, Kubernetes distributions such as k3s or kind could operate directly on macOS, eliminating the need for Lima or QEMU. This would enable Apple Container to become the most lightweight local Kubernetes runtime on Apple Silicon.\nCurrently, developers who require Kubernetes must use OrbStack, Rancher Desktop, or Docker Desktop. However, if Apple keeps advancing their technology, this could change, particularly considering the maturity of their virtualization APIs.\nClosing Thoughts Apple Container marks an interesting evolution in how macOS fits into the cloud-native developer ecosystem. It’s not trying to replace Docker Desktop or Rancher Desktop yet; instead, it’s redefining what “native” container execution means on Apple hardware.\nFor DevOps engineers, this is significant. A native, lightweight, secure runtime supported by Apple could eventually eliminate the licensing hurdles and performance overhead that have historically impacted container development on macOS.\nIt’s early, but promising. For those of us who spend our days between Kubernetes clusters, it’s another sign that containers are no longer limited to Linux. macOS is finally becoming a first-class citizen in the developer’s container ecosystem on its own terms.\n","permalink":"https://wolkwacht.nl/posts/running-containers-the-apple-way-a-first-look-into-apple-container-on-macos/","summary":"\u003ch2 id=\"running-containers-the-apple-way-a-first-look-into-apple-container-onmacos\"\u003e\u003cstrong\u003eRunning Containers the Apple Way: A First Look into Apple Container on macOS\u003c/strong\u003e\u003c/h2\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*nKy2CAjsTt__9GKK9ZfQlw.png\"\u003e\u003c/p\u003e\n\u003cp\u003eFor years, running containers on macOS has involved compromises. Docker Desktop used HyperKit to virtualize Linux, which was simple to use but resource-intensive. Alternatives like Rancher Desktop and OrbStack improved on this by being lighter, faster, and more developer-friendly, yet they all depended on the same core approach: emulating or virtualizing Linux within macOS.\u003c/p\u003e\n\u003cp\u003eApple has now entered this space with a native solution: \u003ca href=\"https://github.com/apple/container?tab=readme-ov-file\"\u003eApple Container\u003c/a\u003e, an OCI-compliant container runtime explicitly designed for Apple Silicon and macOS 15/16 and later. It is not simply a re-skin of Docker or a clone of Lima or Colima. Instead, it represents Apple’s unique approach to how containers should operate on macOS, using the Virtualization and Network frameworks that power iOS simulators and macOS sandboxing.\u003c/p\u003e","title":"Running Containers the Apple Way: A First Look into Apple Container on macOS"},{"content":"Azure Kubernetes Security Demystified: From Nodes to Pods Introduction Kubernetes has become the standard for hosting containerized applications, with Azure Kubernetes Service (AKS) being one of the most popular managed options. AKS simplifies management by handling the control plane, but securing the environment remains the customer’s responsibility. The shared responsibility model requires you to focus on strengthening worker nodes, controlling cluster access, and ensuring that workloads and pods operate with the least privileges.\nSecurity in Kubernetes involves multiple layers. The base layer is the host security of the virtual machines in the node pools. Next is the cluster layer, where identity, networking, and governance must be carefully established and maintained. At the top, securing pods and containers that run your business logic is crucial to prevent privilege escalation or unauthorized communication. This approach, known as “defense in depth,” emphasizes addressing security across various boundaries instead of relying on a single tool or configuration for full protection.\nWe will now take a closer look at the different layers that make up the ‘defense in depth’.\nHost Security in AKS Although AKS provides a managed service, the worker nodes are virtual machines within your subscription. This means that patching, upgrading, and monitoring are your responsibilities. The first decision you need to make is which operating system your nodes should run. AKS supports both Ubuntu and Azure Linux, formerly known as CBL-Mariner. Ubuntu is a more familiar choice for many teams, but Azure Linux has been specifically designed for Azure infrastructure. It has a smaller footprint, a more streamlined kernel, and faster patch delivery, which helps reduce the attack surface.\nKeeping the hosts updated is a vital operational task. AKS enables automatic node pool upgrades, ensuring nodes are regularly cycled and patched with the latest security fixes. You can also set maintenance windows so that upgrades happen outside business-critical hours, reducing the impact of restarts. For teams that want maximum control, upgrading can also be done manually by recreating node pools, draining pods, and reattaching them after patching.\nStorage and networking on the nodes need careful attention. All disks should be encrypted, using either platform-managed keys or customer-managed keys from Azure Key Vault. On the network side, placing nodes in a dedicated subnet with a network security group adds an extra layer of security. Inbound access should be highly restricted, and, whenever possible, the cluster API should be private to ensure only requests from the Azure backbone are accepted.\nMonitoring host security is an ongoing responsibility. Azure Monitor for containers offers telemetry on CPU, memory, and disk usage, and also detects daemon processes and abnormal node behavior. Combining this with Microsoft Defender for Containers provides runtime protection that can identify suspicious binaries, cryptomining attempts, or privilege escalations at the node level.\nCluster Security The cluster layer is where governance and identity meet. Access to the Kubernetes API is often the most critical control point. A public AKS cluster makes its API accessible to the internet, but in most cases, a private cluster is the better choice, ensuring traffic only flows through the Azure backbone. If a public endpoint is necessary, IP allowlists should be used to limit access to known corporate networks.\nAuthentication in AKS should always be linked to Azure Active Directory (now Entra ID). This enables you to connect groups from your corporate directory to Kubernetes roles. For example, a developer group can be granted permissions to list and create pods in a development namespace, while a platform engineering group may have higher rights for cluster setup. The essential principle here is the principle of least privilege. The cluster-admin role should only be used in emergencies, and audit trails should be kept for all actions.\nWhat “Least Privilege” Really Means\nThe principle of least privilege is fundamental to security in Kubernetes and beyond. It means that each user, service account, and workload should only have the permissions necessary to perform their tasks — no more, no less.\nFor example, a developer who only needs to read logs from a single namespace should not be granted full cluster-admin rights to simplify the process. Such broad access could allow them to delete pods, create secrets, or even shut down the cluster, and if compromised, could cause severe damage.\nIn Kubernetes, enforcing least privilege involves precisely mapping Entra ID groups to Kubernetes RBAC roles, setting permissions at the namespace level when possible, and regularly reviewing bindings. By limiting privileges, you reduce the impact of potential breaches and hinder attackers from moving laterally within the cluster. Think of least privilege as creating secure corridors: users and workloads can only access designated areas, keeping everything else protected.\nRole-Based Access Control (RBAC) provides the mechanism for defining these permissions. For instance, the following role grants developers limited access inside a namespace:\nkind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: namespace: dev name: developer-role rules: - apiGroups: [\u0026#34;\u0026#34;] resources: [\u0026#34;pods\u0026#34;] verbs: [\u0026#34;get\u0026#34;, \u0026#34;list\u0026#34;, \u0026#34;create\u0026#34;, \u0026#34;delete\u0026#34;] Regular auditing of roles and bindings is essential, since lingering permissions are a common source of escalation in compromised clusters.\nNetworking within the cluster should follow a zero-trust approach. AKS offers support for both Azure CNI Overlay and Cilium for pod networking. Enabling network policies ensures that pods only communicate with necessary services. Starting with a deny-all policy, you can add specific allow rules as needed. When combined with Azure Firewall or a Web Application Firewall at ingress points, this setup provides multiple security layers between the workload and external access.\nSecrets and configuration management require careful handling. Storing passwords or connection strings in ConfigMaps poses a security risk. Instead, Azure Key Vault should serve as the central repository for secrets, mounted into pods via the Secrets Store CSI Driver. For instance, a pod can access a database connection string directly from Key Vault by mounting it into its file system, thereby removing the necessity to embed secrets in YAML files.\nAnother key concern is securing the supply chain. Container images must be sourced from trusted registries, such as Azure Container Registry (ACR). Defender for Containers can automatically scan images during push to ACR, identifying vulnerabilities before deployment. To enforce this policy within the cluster, admission controllers such as Gatekeeper with Open Policy Agent (OPA) or Kyverno can be set up to reject images that come from untrusted sources or lack proper signatures. Signing images with Notary v2 or Cosign guarantees their integrity, and policies can require signature verification during runtime.\nNotary vs. Cosign: What You Need to Know\nWhen discussing supply chain security in Kubernetes, the issue of image signing naturally arises. The two main tools you\u0026rsquo;ll often see are Notary and Cosign. Although both aim to verify that container images are authentic and unchanged, they handle the task in different ways.\nNotary has a longstanding history and is closely linked with Docker and OCI registries. Its second version, Notary v2, is tailored to function within the OCI artifacts ecosystem, embedding signatures and metadata directly into registries. With Notary, the trust is anchored in the registry, allowing policies to ensure that only signed and verified images are admitted into your cluster. In Azure, this naturally integrates with Azure Container Registry, which utilizes Notary to support content trust.\nCosign, part of the Sigstore project, adopts a modern approach by not relying solely on registry-based trust. It can attach signatures as OCI artifacts and introduces ‘keyless signing,’ allowing images to be signed with ephemeral keys associated with your identity via an OpenID Connect (OIDC) provider like GitHub Actions or Azure AD. This facilitates smooth integration into CI/CD pipelines without the hassle of key management, making it highly appealing for contemporary DevSecOps teams.\nEssentially, Notary offers a conventional, registry-based model suitable for organizations that prefer deterministic trust rooted in their container registry. Cosign provides a flexible, cloud-native developer experience, especially when integrated with automated pipelines and keyless workflows. Many enterprises choose to adopt both: using Notary for registry-level enforcement and Cosign for signing within developer pipelines.\nThink of Notary as the vault-based approach, while Cosign is the pipeline-native method. Both aim to verify authenticity, but the choice depends on where in your workflow you want to establish trust.\nCluster-level security relies on robust logging and auditing. Kubernetes audit logs can be sent to Azure Monitor and stored in a Log Analytics Workspace. From there, they can be integrated with Microsoft Sentinel for correlation and threat detection across your broader environment.\nPod Security At the top of the stack are the workloads themselves. Pods are frequently the initial entry point for attackers, particularly when applications have exploitable vulnerabilities. In Kubernetes, Pod Security Admission (PSA) has replaced PodSecurityPolicy, offering a native way to enforce basic or restricted security policies across namespaces. Implementing restricted policies helps prevent pods from running as root or mounting sensitive host paths.\nAn example restricted policy might include the following configuration within a pod specification:\nsecurityContext: runAsUser: 1000 allowPrivilegeEscalation: false capabilities: drop: [\u0026#34;ALL\u0026#34;] This guarantees that the container runs as a non-root user, cannot escalate privileges, and does not inherit unnecessary Linux capabilities. Network segmentation should also be implemented at the pod level. For example, you could use a frontend service to communicate only with a backend service, while preventing lateral movement with unrelated workloads.\nThis is enforced through network policies, which effectively implement microsegmentation within the cluster. Resource requests and limits are another aspect of pod security. By setting appropriate CPU and memory requests, you prevent noisy neighbors from consuming all available resources and ensure that denial-of-service attempts cannot starve other workloads. A basic specification might look like this:\nresources: requests: cpu: \u0026#34;500m\u0026#34; memory: \u0026#34;256Mi\u0026#34; limits: cpu: \u0026#34;1\u0026#34; memory: \u0026#34;512Mi\u0026#34; Finally, runtime protection is crucial. Defender for Containers works with AKS to identify anomalies, such as reverse shells or crypto-mining activities, within pods. Open-source options like Falco provide similar capabilities by monitoring system calls in real-time and triggering alerts for unusual behavior.\nSecurity as a Continuous Process Securing AKS isn\u0026rsquo;t a one-time task. Azure Policy for AKS helps enforce rules across your clusters, such as blocking privileged pods or requiring integration with Key Vault. Ongoing security management with Defender CSPM detects and fixes misconfigurations quickly. Using tools like Azure Chaos Studio or open-source kube-hunter to simulate attacks offers a proactive way to assess resilience before actual incidents happen.\nIncorporating security into the DevOps pipeline, commonly referred to as DevSecOps, is essential. Images need to be scanned during build processes using tools like Trivy or Defender, policies should be enforced at the admission stage, and runtime environments should be under continuous surveillance. This approach helps identify vulnerabilities early and ensures a layered security strategy from the build stage through to production workloads.\nConclusion AKS offers a robust enterprise platform for Kubernetes, but security responsibilities span multiple layers. At the host level, selecting an appropriate OS, performing regular updates, encrypting disks, and monitoring runtime activity form the basis of security. On the cluster level, controlling access via Entra ID and RBAC, isolating networks with policies, securing supply chains, and auditing logs enhance resilience. At the pod level, implementing non-root execution, dropping capabilities, segmenting traffic, and monitoring runtime behavior completes the security framework.\nSecuring AKS is an ongoing process. With regulations like NIS2 and DORA influencing the European scene, organizations must view security as more than just a technical issue; it’s a core business concern. By integrating these practices into daily operations and adopting a defense-in-depth approach, enterprises can safely run critical workloads in AKS and stay ahead of emerging threats.\nReferences \u0026amp; Further Reading • Azure Kubernetes Service Security Baseline\n• Defender for Containers\n• Pod Security Standards\n• NIS2 Directive\n• DORA Regulation\n","permalink":"https://wolkwacht.nl/posts/azure-kubernetes-security-demystified-from-nodes-to-pods/","summary":"\u003ch2 id=\"azure-kubernetes-security-demystified-from-nodes-topods\"\u003eAzure Kubernetes Security Demystified: From Nodes to Pods\u003c/h2\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*TaWt7QA5dk2vpOJm78Oi9w.png\"\u003e\u003c/p\u003e\n\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003eKubernetes has become the standard for hosting containerized applications, with Azure Kubernetes Service (AKS) being one of the most popular managed options. AKS simplifies management by handling the control plane, but securing the environment remains the customer’s responsibility. The shared responsibility model requires you to focus on strengthening worker nodes, controlling cluster access, and ensuring that workloads and pods operate with the least privileges.\u003cbr\u003e\nSecurity in Kubernetes involves multiple layers. The base layer is the host security of the virtual machines in the node pools. Next is the cluster layer, where identity, networking, and governance must be carefully established and maintained. At the top, securing pods and containers that run your business logic is crucial to prevent privilege escalation or unauthorized communication. This approach, known as “defense in depth,” emphasizes addressing security across various boundaries instead of relying on a single tool or configuration for full protection.\u003c/p\u003e","title":"Azure Kubernetes Security Demystified: From Nodes to Pods"},{"content":"Why Amazon’s Kubernetes service is a compliance lever, not just duplication\nThis article is part of the Azure Kubernetes Chronicles: Multi-Cluster Edition.\nRead the series introduction here.\nWhenever companies discuss Kubernetes in Azure, AKS naturally comes up because it\u0026rsquo;s the go-to managed service for those within Microsoft’s ecosystem. However, across Europe, another name is gaining attention: Amazon Elastic Kubernetes Service, or EKS. For many organizations, EKS isn\u0026rsquo;t just another option; it\u0026rsquo;s become a vital part of their strategic planning.\nAt first glance, operating both AKS and EKS may seem redundant. Why would an enterprise support two distinct Kubernetes services, each with its own costs, governance, and operational challenges? The reasons are regulation, resilience, and strategic advantage. In key industries such as banking and telecommunications, European regulators are increasingly discouraging reliance on a single cloud provider. Frameworks like DORA require organizations to demonstrate they can recover from third-party service failures, which typically means being able to switch to an alternative provider. For a financial institution running core workloads on AKS, this often involves maintaining a backup cluster on EKS.\nGeography introduces an additional layer. Although Azure and AWS both have strong footprints across Europe, their presence isn\u0026rsquo;t identical. AWS’s well-established operations in Frankfurt, a crucial regulatory hub for Germany, make it an appealing option for companies that need to meet national standards. Additionally, AWS provides specific services such as SageMaker and Bedrock for AI/ML, and Kinesis for streaming, that can be seamlessly integrated with containerized applications. Opting for EKS in these situations is not about redundancy but about optimization: it ensures workloads are matched with the regions and features that best support them.\nBut adopting EKS does not come without challenges. The most immediate issue is identity. AKS connects directly with Entra ID, while EKS depends on IAM. Aligning these models is seldom straightforward. Enterprises often end up duplicating roles and policies, which creates inefficiency and increases risk. Policy enforcement also varies. In AKS, teams can rely on Azure Policy for Kubernetes. In EKS, the equivalent must be built with OPA or Kyverno. Keeping consistency across these two systems requires discipline and often leads to drift.\nNetworking remains a weak point. Azure’s VNets and AWS’s VPCs do not naturally align, making hybrid network design a complex process. Service discovery across providers is rarely seamless, and while service meshes provide one solution, they also increase overhead and skills requirements. Observability follows a similar pattern. Azure Monitor cannot understand CloudWatch, and CloudWatch cannot view resources in AKS. Some companies use neutral platforms like Prometheus and Grafana, but this adds another operational layer.\nThen there is the cost. On paper, AKS appears to be cheaper because its control plane is free, whereas EKS charges a flat monthly fee per cluster. At more minor scales, this difference is noticeable. At an enterprise level, the cost of the control plane is less significant than something less obvious: the need for duplicate skills. Running both AKS and EKS requires training teams on two identity models, two policy frameworks, and two monitoring stacks. The financial impact extends beyond infrastructure to include people, time, and processes.\nDespite these obstacles, EKS holds a crucial strategic position. It serves as a compliance facilitator, enabling enterprises to showcase multi-cloud resilience to regulators. It also acts as a legacy anchor for organizations that developed within AWS and are hesitant to leave. Additionally, it offers a competitive advantage for workloads that leverage AWS’s AI and data ecosystem. Most importantly, it provides CIOs with a bargaining chip: the power to negotiate with Microsoft confidently by demonstrating the ability to shift workloads to other platforms if necessary.\nImagine a large European logistics company. Its central platform operates on AKS in Amsterdam, but due to German regulations, it also needs a secondary cluster in Frankfurt, which EKS handles. Meanwhile, their distribution centers use k3s clusters for local IoT processing. This creates a diverse setup: AKS for essential tasks, EKS for compliance needs, and k3s for local edge processing. Without a unifying solution like Azure Arc, this setup can feel scattered; policies need to be duplicated, observability is isolated, and identity management is fragmented. However, choosing EKS isn’t just a preference; it\u0026rsquo;s a requirement for compliance and is made more appealing by AWS’s robust ecosystem.\nEKS is not mere duplication; it is a strategic choice reflecting the reality that enterprises often cannot rely on a single cloud. This is driven by factors such as regulatory compliance, resilience, or competitive pressures. For architects, the real challenge isn\u0026rsquo;t whether to include EKS; it\u0026rsquo;s already part of the landscape. Instead, the challenge is how to incorporate it into a governance model that also manages AKS and edge clusters.\nIn the next episode of the Azure Kubernetes Chronicles, we’ll pause from hyperscale providers and focus on the edge. We will examine how lightweight Kubernetes distributions, such as k3s, are revolutionizing sectors like retail, manufacturing, healthcare, and telecommunications by extending cloud-native capabilities to the most remote parts of the enterprise.\nPrevious: Azure Kubernetes Chronicles 8: Fleet in Focus\nNext up: Azure Kubernetes Chronicles 10: Kubernetes at the Edge with k3s\n","permalink":"https://wolkwacht.nl/posts/azure-kubernetes-chronicles-9-eks-in-the-enterprise/","summary":"\u003cp\u003e\u003cem\u003eWhy Amazon’s Kubernetes service is a compliance lever, not just duplication\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*4tQVG--n6ohJgnqR4Gr73g.png\"\u003e\u003c/p\u003e\n\u003cp\u003eThis article is part of the Azure Kubernetes Chronicles: Multi-Cluster Edition.\u003cbr\u003e\nRead the series introduction \u003ca href=\"https://medium.com/@jurgenallewijn/azure-kubernetes-chronicles-multi-cluster-edition-0dab3518d297\"\u003ehere\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eWhenever companies discuss Kubernetes in Azure, AKS naturally comes up because it\u0026rsquo;s the go-to managed service for those within Microsoft’s ecosystem. However, across Europe, another name is gaining attention: Amazon Elastic Kubernetes Service, or EKS. For many organizations, EKS isn\u0026rsquo;t just another option; it\u0026rsquo;s become a vital part of their strategic planning.\u003c/p\u003e","title":"Azure Kubernetes Chronicles 9: EKS in the Enterprise"},{"content":"Azure Kubernetes Chronicles 8: Fleet in Focus Azure’s multi-cluster powerhouse — and where it falls short\nThis article is part of the Azure Kubernetes Chronicles: Multi-Cluster Edition.\nRead the series introduction here.\nWhen Microsoft introduced Azure Kubernetes Fleet Manager, it was met with both excitement and curiosity. For years, platform teams running multiple AKS clusters had been piecing together governance with scripts, policies, and half-baked federation approaches. Fleet arrived with the promise of simplification: a central way to govern clusters, propagate policies, and even distribute workloads across Azure regions.\nFor enterprises already committed to Azure, it is the missing piece. Suddenly, a native tool emerged that could provide consistency, reduce duplication, and offer a global perspective on cluster operations. The value proposition was straightforward: configure once, apply everywhere, and finally gain the kind of multi-region resilience that regulators and boards increasingly expect.\nFleet does much of this well. It enables enterprises to manage multiple AKS clusters as if they were part of a single fleet, rather than isolated islands. Governance policies can be defined once and automatically applied across clusters, eliminating the manual drift that so often plagues operations. Workloads can be scheduled with awareness of geography and health, ensuring that users are directed to the nearest or most reliable cluster. Shared resources such as ConfigMaps or Secrets can be pushed fleet-wide, bringing a level of cohesion that previously required heavy lifting from operations teams. And because Fleet integrates tightly with Azure-native services such as Entra ID, Azure Policy, and Azure Traffic Manager, it feels less like a separate product and more like an organic extension of Azure itself.\nFrom a strategic lens, Fleet strengthens Azure’s story. For the CIO of a company running exclusively on Azure, it provides three key benefits: consistency, efficiency, and resilience. Consistency comes from the ability to unify governance across all AKS clusters. Efficiency stems from eliminating the need to configure each cluster independently. Resilience comes from native support for multi-region failover, turning complex architecture patterns into first-class capabilities. Within Azure’s walls, Fleet feels like the perfect answer to the multi-cluster challenge.\nBut here is where the picture becomes more complicated. Fleet today is an Azure-first, AKS-only solution. It assumes that governance occurs through Entra ID, that policies are defined through Azure Policy, and that observability is enabled through Azure Monitor. For organizations that live entirely in Azure, this alignment is powerful. For those that also run workloads on AWS EKS, GCP GKE, or at the edge with lightweight distributions like k3s, Fleet cannot extend its reach.\nConsider a European retailer that operates its online platform on AKS in Amsterdam, while regulators require a backup cluster in AWS Frankfurt. At the same time, every store in its chain runs local K3S clusters on Raspberry Pi devices to keep point-of-sale systems functioning. Inside Azure, Fleet delivers order and control. Outside of Azure, it offers little visibility. The AWS cluster and the dozens of store-level clusters remain disconnected, leaving governance teams with a fragmented picture.\nThis limitation is not a surprise. Fleet is still a relatively young service, and Microsoft initially built it to address Azure’s internal sprawl, rather than the broader multi-cloud landscape. Compared with more established platforms such as Google Anthos, it lacks features like unified cross-cloud policy enforcement or native hybrid observability. It is less a universal tool and more a competent Azure-native solution.\nFor enterprises deciding whether to adopt Fleet, the dividing line is clear. If the business is fully committed to Azure, Fleet makes perfect sense. It brings simplicity, removes operational noise, and builds resilience into the heart of AKS operations. However, if the business must operate across multiple clouds, sovereign environments, or edge scenarios, Fleet is insufficient on its own. In those contexts, it becomes a valuable component in a larger strategy, complemented by tools such as Azure Arc, service meshes, or open-source governance frameworks.\nFleet, in other words, is both powerful and limited. It shines in the world it was designed for, and within that world, it may become indispensable. But as soon as the enterprise strategy expands beyond Azure, Fleet alone cannot tell the whole story. That does not make it irrelevant; instead, it cements its role as a cornerstone for Azure-first enterprises, even as Microsoft signals plans to integrate it more closely with Arc.\nThe story of Fleet is the story of Azure’s strength and its boundaries. It serves as a reminder that in multi-cluster Kubernetes, no single tool can solve every challenge. Fleet is a step forward, but only part of the journey.\nFor the more technical details, check my post Azure Kubernetes Chronicles 6: Managing Multiple AKS Clusters with Azure Fleet.\nPrevious: Azure Kubernetes Chronicles 7: The Multi-Cluster Reality\n️Next up: Azure Kubernetes Chronicles 9: EKS in the Enterprise\n","permalink":"https://wolkwacht.nl/posts/azure-kubernetes-chronicles-8-fleet-in-focus/","summary":"\u003ch2 id=\"azure-kubernetes-chronicles-8-fleet-infocus\"\u003eAzure Kubernetes Chronicles 8: Fleet in Focus\u003c/h2\u003e\n\u003cp\u003e\u003cem\u003eAzure’s multi-cluster powerhouse — and where it falls short\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*hLzyJCm2aM0kDN376GjXhA.png\"\u003e\u003c/p\u003e\n\u003cp\u003eThis article is part of the Azure Kubernetes Chronicles: Multi-Cluster Edition.\u003cbr\u003e\nRead the series introduction \u003ca href=\"https://medium.com/@jurgenallewijn/azure-kubernetes-chronicles-multi-cluster-edition-0dab3518d297\"\u003ehere\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eWhen Microsoft introduced Azure Kubernetes Fleet Manager, it was met with both excitement and curiosity. For years, platform teams running multiple AKS clusters had been piecing together governance with scripts, policies, and half-baked federation approaches. Fleet arrived with the promise of simplification: a central way to govern clusters, propagate policies, and even distribute workloads across Azure regions.\u003c/p\u003e","title":"Azure Kubernetes Chronicles 8: Fleet in Focus"},{"content":"From the dream of one perfect cluster to the messy truth of enterprise sprawl\nThis episode is part of the Azure Kubernetes Chronicles: Multi-Cluster Edition.\nRead the series introduction here.\nWhen Kubernetes first entered the enterprise, many platform teams envisioned a single, perfect cluster; a hub where all workloads could reside, scale smoothly, and be governed consistently. The reality turned out very different.\nToday, almost no enterprise runs just one cluster. Instead, organizations find themselves managing multiple clusters scattered across regions, cloud providers, and even edge sites. This isn’t random chaos; it’s the result of regulatory demands, business continuity strategies, performance requirements, and the rise of edge computing.\nIn Europe, in particular, directives such as NIS2 and DORA are reshaping IT strategy. Compliance frameworks now demand resilience, redundancy, and independence from single providers. At the same time, performance expectations, mergers and acquisitions, and operational realities pull enterprises toward a multi-cluster future.\nWhy Multi-Cluster Kubernetes Matters At first glance, a single, enterprise-wide Kubernetes cluster appears more straightforward: one control plane, one set of policies, and one monitoring stack. But at scale, that ideal breaks down. Instead, organizations discover that multi-cluster is not a luxury; it is a necessity.\nRegulatory compliance is the first and most obvious driver. In the EU, financial institutions and critical infrastructure providers are being forced to demonstrate that their workloads can survive a regional or provider-level outage. A European bank might run its primary workloads on Azure Kubernetes Service in West Europe but operate a secondary cluster in AWS Frankfurt to prove resilience and regulatory independence. Compliance demands have transformed what once looked like an architectural choice into a non-negotiable requirement.\nBusiness continuity and disaster recovery provide an additional boost. Outages in hyperscaler regions are rare, but they do happen. When they do, relying on a single cluster within a single geography can leave an organization exposed to costly downtime. Distributing clusters across regions and providers turns resilience into a built-in feature rather than a bolt-on solution.\nPerformance considerations also drive multi-cluster adoption. Enterprises serving global users know that latency is critical. Workloads serving customers in Asia-Pacific often perform better when hosted in Singapore on AWS, while European workloads typically run more efficiently from Azure regions in Amsterdam or Dublin. By hosting clusters closer to users, organizations improve responsiveness and provide a smoother customer experience.\nMergers and acquisitions further complicate the picture. When one company buys another, its Kubernetes clusters do not merge overnight. Enterprises may operate with a patchwork of AKS, EKS, and even on-prem clusters for years. Consolidation is rarely simple, and in the meantime, the multi-cluster state becomes the de facto reality.\nSome organizations also adopt multi-cluster strategies by design, utilizing them as a means to separate concerns and responsibilities. Development, testing, staging, and production environments often run in distinct clusters to reduce risk and maintain control. Others separate clusters by business unit or geography, ensuring that failures or compliance exceptions remain isolated within their respective areas. What may appear to be duplication is often a deliberate risk management strategy.\nFinally, there is the edge. Not all workloads can run in the cloud. Retail stores, hospitals, and manufacturing sites often require systems that continue to function even if the internet connection is lost. Lightweight Kubernetes distributions such as k3s make it possible to run local clusters on devices as small as a Raspberry Pi or as compact as an Intel NUC. In practice, a retail chain may have hundreds of small but critical clusters that keep point-of-sale systems online or run AI models for queue management. Factories use them for machine telemetry and predictive maintenance, while hospitals process sensitive imaging data locally to comply with privacy rules. These clusters are small but strategically vital, and they significantly contribute to the overall cluster count.\nThe Risks of Cluster Sprawl The benefits of multi-cluster Kubernetes are apparent, but the risks are just as real. Identity is often the first fracture point. Azure clusters integrate seamlessly with Entra ID, AWS clusters with IAM, and edge clusters with local certificates. The result is a fragmented identity model that makes consistent access control challenging. Policies begin to drift as different clusters enforce different pod security standards, admission controllers, or OPA rules. Compliance teams eventually notice the gaps. Networking complexity soon follows, with each environment relying on different CNIs and service discovery methods, resulting in unreliable cross-cluster communication. Observability suffers too, as Azure Monitor, AWS CloudWatch, and self-managed Prometheus stacks rarely share a common language. To make matters worse, costs escalate quickly because every cluster brings its own control plane, node pools, ingress controllers, and monitoring agents. Without disciplined FinOps practices, a multi-cluster environment can spiral into multiple costs.\nCluster sprawl\nGovernance Pressure For enterprise architects, the key question is no longer whether a multi-cluster approach is needed, but how it can be governed effectively. Boards and regulators expect organizations to demonstrate consistent security and compliance across all environments, to produce disaster recovery playbooks that prove workloads can fail over successfully, to provide audit trails for NIS2, DORA, and GDPR, and to show that costs are under control. The pressure to deliver governance at scale is why solutions such as Azure Fleet, Azure Arc, Google Anthos, and open-source service meshes are becoming essential elements of enterprise Kubernetes strategies.\nClosing Thought: Multi-Cluster is the Enterprise Reality\nNo CIO begins a cloud journey by declaring an intention to run three types of Kubernetes clusters across two providers and hundreds of edge sites. Yet that is precisely where most enterprises arrive. They don’t get there by choice, but by necessity. Multi-cluster Kubernetes is the messy truth of enterprise sprawl — and also the foundation of resilience, compliance, and edge innovation.\nNext up: Azure Kubernetes Chronicles 8: Fleet in Focus\n","permalink":"https://wolkwacht.nl/posts/azure-kubernetes-chronicles-7-the-multi-cluster-reality/","summary":"\u003cp\u003e\u003cem\u003eFrom the dream of one perfect cluster to the messy truth of enterprise sprawl\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*lZBCyFZfoZ94hzuiLKjdqA.png\"\u003e\u003c/p\u003e\n\u003cp\u003eThis episode is part of the Azure Kubernetes Chronicles: Multi-Cluster Edition.\u003cbr\u003e\nRead the series introduction \u003ca href=\"https://jurgenallewijn.nl/azure-kubernetes-chronicles-multi-cluster-edition-0dab3518d297\"\u003ehere\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eWhen Kubernetes first entered the enterprise, many platform teams envisioned a single, perfect cluster; a hub where all workloads could reside, scale smoothly, and be governed consistently. The reality turned out very different.\u003c/p\u003e\n\u003cp\u003eToday, almost no enterprise runs just one cluster. Instead, organizations find themselves managing multiple clusters scattered across regions, cloud providers, and even edge sites. This isn’t random chaos; it’s the result of regulatory demands, business continuity strategies, performance requirements, and the rise of edge computing.\u003c/p\u003e","title":"Azure Kubernetes Chronicles 7: The Multi-Cluster Reality"},{"content":"From Fleet to Edge — navigating Kubernetes across clouds and continents\nWhen Kubernetes first entered the enterprise, many architects imagined a single, perfect cluster: one hub for all workloads, scalable, resilient, and secure. Reality had other plans.\nEnterprises today run multiple clusters across regions, providers, and even at the edge:\n• AKS in Azure for mainline production.\n• EKS in AWS to meet regulatory or resilience mandates.\n• k3s clusters on Raspberry Pi or NUCs powering retail stores, factories, and hospitals.\nThis isn’t just technical sprawl, it’s strategy. Regulations such as NIS2 and DORA, as well as sovereignty requirements and business continuity, all demand multi-cluster architectures. The challenge for platform teams is no longer whether a multi-cluster approach is needed, but how to govern it effectively.\nThat’s where this special Multi-Cluster Edition of Azure Kubernetes Chronicles comes in. Across six episodes, we’ll explore the business drivers, technical realities, and strategic responses shaping enterprise Kubernetes today.\nThe Episodes Azure Kubernetes Chronicles 7: The Multi-Cluster Reality\nFrom the dream of one perfect cluster to the messy truth of enterprise sprawl\nWhy enterprises end up with multiple clusters — and the regulatory, resilience, and edge drivers that make it inevitable.\nAzure Kubernetes Chronicles 8: Fleet in Focus\nAzure’s multi-cluster powerhouse — and where it falls short\nA deep dive into Azure Kubernetes Fleet Manager: what it solves brilliantly inside Azure, and why it struggles beyond.\nAzure Kubernetes Chronicles 9: EKS in the Enterprise\nWhy Amazon’s Kubernetes service is a compliance lever, not just duplication\nThe role of AWS EKS in dual-cloud strategies, compliance requirements, and its implications for governance.\nAzure Kubernetes Chronicles 10: Kubernetes at the Edge with k3s\nFrom Raspberry Pi retail stores to 5G towers — the hidden force reshaping enterprise IT\nHow lightweight Kubernetes (k3s) brings edge workloads to life, and the governance challenges at scale.\nAzure Kubernetes Chronicles 11: Governing the Chaos\nUnifying identity, policy, networking, and observability across clouds and edge\nIdentity fragmentation, policy drift, network sprawl, observability silos — and how Arc, Anthos, and service mesh attempt to fix them.\nAzure Kubernetes Chronicles 12: The Future of Multi-Cluster\nFleet, Arc, sovereign clouds, and the edge-first world of 2030\nLooking ahead: sovereign cloud adoption, edge explosion, and Microsoft’s Arc + Fleet convergence strategy.\nExciting news! This series of episodes will be arriving soon, with new parts posted online gradually over the next few weeks. Be sure to stay tuned and catch every episode as we share the content bit by bit!\nWhy This Series? For CloudOps engineers, architects, and platform teams, working with multi-cluster Kubernetes has become a regular part of your daily routine. It’s just the way things are in the enterprise. This series offers a friendly blend of practical tips, such as tools, best practices, and governance, along with strategic insights on compliance, sovereignty, and procurement leverage. It\u0026rsquo;s all here to support you as you embark on this exciting new chapter with Kubernetes.\n","permalink":"https://wolkwacht.nl/posts/azure-kubernetes-chronicles-multi-cluster-edition/","summary":"\u003cp\u003e\u003cem\u003eFrom Fleet to Edge — navigating Kubernetes across clouds and continents\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*wQpHdP8GGsNpAmHF1qgLSA.png\"\u003e\u003c/p\u003e\n\u003cp\u003eWhen Kubernetes first entered the enterprise, many architects imagined a single, perfect cluster: one hub for all workloads, scalable, resilient, and secure. Reality had other plans.\u003c/p\u003e\n\u003cp\u003eEnterprises today run multiple clusters across regions, providers, and even at the edge:\u003cbr\u003e\n • AKS in Azure for mainline production.\u003cbr\u003e\n • EKS in AWS to meet regulatory or resilience mandates.\u003cbr\u003e\n • k3s clusters on Raspberry Pi or NUCs powering retail stores, factories, and hospitals.\u003c/p\u003e","title":"Azure Kubernetes Chronicles: Multi-Cluster Edition"},{"content":"\nIntroduction: Why Centralized Management of AKS Clusters Matters Kubernetes adoption in enterprises has skyrocketed. What begins as a single proof-of-concept cluster often grows into dozens across various teams, environments, and regions. For organizations using Azure Kubernetes Service (AKS), this growth is even more common: different business units create clusters for their workloads, data residency rules require data to stay in specific locations, and production systems need failover capabilities across regions.\nInitially, managing a few clusters might seem straightforward. But as the number increases, complexity multiplies:\nHow do we ensure consistent governance? How do we enforce security and compliance uniformly? How can we manage workload placement without duplicating configurations? How do we achieve multi-region availability without relying on fragile, custom scripts? This is the key part of the multi-cluster challenge, highlighting its complexity and emphasizing the importance of effective management strategies to address various issues.\nIn today’s cloud-native environment, multi-cluster setups are often built-in, driven by geo-distribution, team separation, and scale. However, without centralized management, this setup can lead to cluster sprawl, policy inconsistencies, duplicated pipelines, and complicated upgrades.\nTo solve this, Microsoft introduced Azure Kubernetes Fleet Manager (Fleet). It’s a service designed to provide centralized governance, workload management, and lifecycle control across multiple AKS clusters — think of it as a control plane for your multiple control planes, helping coordinate Kubernetes environments at an enterprise level. This is the multi-cluster challenge.\nIn this blog, we’ll explore:\n• The challenges of managing multiple AKS clusters.\n• How Fleet Manager helps solve these issues.\n• The architecture of Fleet and its role in Azure-native environments.\n• A step-by-step guide with validated CLI and YAML samples.\n• Managing multi-region AKS deployments for high availability and compliance.\n• Integrating monitoring, security, and governance into Fleet.\nBy the end, you’ll discover how to smoothly shift from cluster sprawl to enterprise-grade multi-cluster management with Azure Fleet. It\u0026rsquo;s a journey that will empower you to manage your resources more effectively and confidently.\nChallenges of Multi-Cluster AKS Management Running multiple Kubernetes clusters is often a necessity for organizations, but managing them can pose several challenges:\nManaging multiple AKS clusters involves various complex tasks, including regular upgrades, patching, RBAC setup, and networking configurations, which, if done manually, can lead to errors and increased effort. Ensuring policy consistency across clusters such as Azure Policy, Pod Security Standards, or custom RBAC is vital to prevent drift and maintain compliance. Workload distribution also plays a critical role, with some workloads requiring regional deployment for compliance, others necessitating multi-cluster setups for high availability, or being cluster-agnostic, which requires sophisticated placement tools. The complexity extends to networking and failover management, involving synchronization of ingress controllers, DNS, TLS certificates, and integration with Azure Front Door or Traffic Manager for cross-region traffic routing. Observability across clusters is challenging due to the dispersed nature of logs and metrics, making centralized monitoring essential for troubleshooting and tracking service levels. Additionally, adhering to regulations such as GDPR, NIS2, and DORA requires maintaining data residency within specific jurisdictions, often necessitating clusters in designated regions (such as the EU or US), while managing a unified control plane.\nIntroducing Azure Kubernetes Fleet Manager Azure Kubernetes Fleet Manager (Fleet) offers a comprehensive solution for managing multiple Kubernetes clusters. It features a Centralized Management Hub that allows for easy registration and control of AKS clusters, enabling logical grouping by region, environment, or workload. Additionally, it sets workload placement policies based on location, affinity, or labels for optimal deployment. Additionally, it ensures consistent governance with unified RBAC and Azure Policy enforcement, and facilitates multi-region high availability by distributing workloads across clusters for fault tolerance and active-active deployment.\nThink of Fleet as the control plane for multiple control planes.\nFleet Hub-and-Member Model\nThe architecture follows a hub-and-spoke pattern:\nThe Fleet hub acts as the central management point, orchestrating policies and workload deployment. Member clusters are existing AKS clusters that join the Fleet for centralized management. Workloads are deployed centrally using Fleet placement policies for efficient operation. Implementing Fleet — Step by Step Let’s move from theory to practice.\nPrerequisites\nAzure CLI ≥ 2.61 Logged in with az login Fleet CLI extension az extension add --name fleet || az extension update --name fleet At least two existing AKS clusters, in this example, we use three. All clusters in the same Entra tenant (can be different subs/RGs/regions) Create a Fleet Hub\nTo perform workload placement, you must create a Fleet with a hub ( — enable-hub).\nRG=rg-fleet LOCATION=westeurope FLEET=aks-fleet-hub az group create -n $RG -l $LOCATION az fleet create \\ --resource-group $RG \\ --name $FLEET \\ --location $LOCATION \\ --enable-hub \\ --enable-managed-identity Get hub kubeconfig so kubectl points at it:\naz fleet get-credentials -g $RG -n $FLEET Add Member Clusters\nWESTEU_ID=\u0026#34;/subscriptions/\u0026lt;sub-id\u0026gt;/resourcegroups/aks-weu/providers/Microsoft.ContainerService/managedClusters/aks-weu\u0026#34; NORTHEU_ID=\u0026#34;/subscriptions/sub-id/resourcegroups/aks-neu/providers/Microsoft.ContainerService/managedClusters/aks-neu\u0026#34; EASTUS_ID=\u0026#34;/subscriptions/sub-id/resourcegroups/aks-eus/providers/Microsoft.ContainerService/managedClusters/aks-eus\u0026#34; # West Europe az fleet member create \\ --resource-group $RG \\ --fleet-name $FLEET \\ --name aks-weu \\ --member-cluster-id $WESTEU_ID \\ --member-labels env=prod \\ --member-labels region=weu # North Europe az fleet member create \\ --resource-group $RG \\ --fleet-name $FLEET \\ --name aks-neu \\ --member-cluster-id $NORTHEU_ID \\ --member-labels env=prod \\ --member-labels region=neu # East US az fleet member create \\ --resource-group $RG \\ --fleet-name $FLEET \\ --name aks-eus \\ --member-cluster-id $EASTUS_ID \\ --member-labels env=prod \\ --member-labels region=eus Label members for placement\n# From the hub cluster context kubectl get memberclusters kubectl label membercluster aks-weu tier=gold --overwrite kubectl label membercluster aks-neu tier=silver --overwrite kubectl label membercluster aks-eus tier=silver --overwrite Fleet auto-labels fleet.azure.com/location with the cluster’s Azure region, which is very useful for region-based placement.\nAuthorisation \u0026amp; RBAC Troubleshooting for Fleet Hubs\nWhen you run kubectl get memberclusters against a Fleet hub, you might see errors like:\nError from server (Forbidden): memberclusters.cluster.kubernetes-fleet.io is forbidden: User \u0026ldquo;\u0026lt;guid\u0026gt;\u0026rdquo; cannot list resource \u0026ldquo;memberclusters\u0026rdquo; in API group \u0026ldquo;cluster.kubernetes-fleet.io\u0026rdquo;\nThis means you’ve authenticated (AAD works via kubelogin), but your user doesn’t have RBAC permissions on the hub.\nMake sure kubelogin is installed (for instance, with Homebrew)\nbrew tap Azure/kubelogin\nbrew install kubelogin\nconvert kubeconfig for AAD auth kubelogin convert-kubeconfig -l azurecli\nChoose how to grant permissions\nOption 1: Azure RBAC for Kubernetes (preferred)\nAssign an Azure RBAC role to your AAD user at the hub AKS scope:\nGet hub AKS resource ID az resource list \\\n\u0026ndash;resource-type Microsoft.ContainerService/managedClusters \\\n\u0026ndash;query \u0026ldquo;[].{name:name,id:id}\u0026rdquo; -o table\nAssign RBAC Reader (or RBAC Admin/Cluster Admin if needed) HUB_AKS_ID=\u0026quot;\u0026lt;paste the id\u0026gt;\u0026quot;\nME=$(az ad signed-in-user show \u0026ndash;query id -o tsv)\naz role assignment create \\\n\u0026ndash;assignee $ME \\\n\u0026ndash;role \u0026ldquo;Azure Kubernetes Service RBAC Reader\u0026rdquo; \\\n\u0026ndash;scope $HUB_AKS_ID\nOption 2: Native Kubernetes RBAC (works everywhere)\nUse admin kubeconfig once, then bind your AAD user to a ClusterRole:\nGrab admin creds az fleet get-credentials -g \u0026lt;rg\u0026gt; -n \u0026lt;fleet\u0026gt; \u0026ndash;admin\nGet your AAD object ID ME=$(az ad signed-in-user show \u0026ndash;query id -o tsv)\nBind as viewer (read-only) kubectl create clusterrolebinding viewer-$ME \\\n\u0026ndash;clusterrole=view \\\n\u0026ndash;user=$ME\nOr bind as full admin (only if needed) kubectl create clusterrolebinding admin-$ME \\\n\u0026ndash;clusterrole=cluster-admin \\\n\u0026ndash;user=$ME\nRe-test\naz fleet get-credentials -g \u0026lt;rg\u0026gt; -n \u0026lt;fleet\u0026gt;\nkubelogin convert-kubeconfig -l azurecli\nkubectl get memberclusters\nkubectl get nodes\nStage Your Workload in the Hub\nWith a hubful Fleet, placement works by selecting resources from the hub and distributing them to member clusters through ClusterResourcePlacement (CRP). This gives you a single source of truth, but it also means you need to be thoughtful about what you add to the hub and how you structure it.\nStructure your staging layout.\nUse a clear repo and namespace structure so you can place selectively and avoid collisions.\nRecommended repo layout (example)\n├─ staging/ # what you apply to the hub │ ├─ base/ # shared manifests (workloads/policies) │ │ ├─ prod-eu/ │ │ │ ├─ deploy.yaml │ │ │ └─ service.yaml │ │ └─ prod/ │ │ ├─ web-frontend.yaml │ │ └─ networkpolicy.yaml │ └─ overlays/ # per-cluster/per-region overrides (optional) │ ├─ weu/ # West Europe overlay (prod or prod-eu) │ └─ neu/ # North Europe overlay (e.g., replicas=0 for DR) │ └─ eus/ # East US overlay (e.g., replicas=0 for DR) └─ crp/ # ClusterResourcePlacement definitions ├─ eu-only.yaml └─ prod-dr-fixed.yaml Create namespaces on the hub\nkubectl create namespace prod --dry-run=client -o yaml | kubectl apply -f - kubectl create namespace prod-eu --dry-run=client -o yaml | kubectl apply -f - Keep “hub side effects” out (envelopes \u0026amp; overlays)\nSome resources shouldn’t be active on the hub, such as cluster-scoped admission webhooks and certain RBAC or quotas. Here are two options:\n• Envelope objects and overlays: keep the “effectful” parts as overrides, so they only apply to member clusters through CRP.\n• Separate namespaces: keep the “overlay” manifests in a dedicated namespace (e.g., overrides-prod) and use a CRP that targets specific members (e.g., only the DR cluster) to apply those overlays there.\nExample: base workload (active everywhere you place it)\n# staging/base/prod-eu/deploy.yaml apiVersion: apps/v1 kind: Deployment metadata: name: orders-api namespace: prod-eu spec: replicas: 3 selector: matchLabels: { app: orders-api } template: metadata: { labels: { app: orders-api } } spec: containers: - name: orders-api image: ghcr.io/contoso/orders:1.4.2 ports: [{ containerPort: 8080 }] --- apiVersion: v1 kind: Service metadata: name: orders-api namespace: prod-eu spec: type: ClusterIP selector: { app: orders-api } ports: [{ port: 80, targetPort: 8080 }] Example: overlay for DR (replicas=0) — not active on hub\n# staging/overlays/neu/orders-api-zero-replicas.yaml apiVersion: apps/v1 kind: Deployment metadata: name: orders-api namespace: prod-eu spec: replicas: 0 selector: matchLabels: { app: orders-api } template: metadata: { labels: { app: orders-api } } spec: containers: - name: orders-api image: ghcr.io/contoso/orders:1.4.2 ports: [{ containerPort: 8080 }] Place the overlay only to NEU via CRP (PickFixed):\n# crp/prod-eu-dr-override.yaml apiVersion: placement.kubernetes-fleet.io/v1 kind: ClusterResourcePlacement metadata: name: prod-eu-dr-override spec: resourceSelectors: - group: apps version: v1 kind: Deployment name: orders-api namespace: prod-eu policy: placementType: PickFixed clusterNames: - aks-neu # Only DR cluster Net result: WEU gets replicas=3 (from base), NEU stays at replicas=0 (overlay).\nWhat is a ClusterResourcePlacement (CRP)?\nA ClusterResourcePlacement (CRP) is the core object that tells Azure Kubernetes Fleet what resources to place and where to place them across your AKS member clusters.\n• What → defined with resourceSelectors (e.g., a namespace, deployment, or network policy).\n• Where → controlled with a policy (e.g., all clusters, fixed clusters, or top N).\n• How → rollout strategies like RollingUpdate define update speed and disruption limits.\nExample: EU-only placement\napiVersion: placement.kubernetes-fleet.io/v1\nkind: ClusterResourcePlacement\nmetadata:\nname: eu-only\nspec:\nresourceSelectors:\n- group: \u0026quot;\u0026quot;\nversion: v1\nkind: Namespace\nname: prod-eu\npolicy:\nplacementType: PickAll\naffinity:\nclusterAffinity:\nrequiredDuringSchedulingIgnoredDuringExecution:\nclusterSelectorTerms:\n- labelSelector:\nmatchLabels:\ncompliance: eu\nTip:\n• Use labels on MemberClusters (e.g., compliance=eu, region=weu) for flexible placement.\n• Start with PickAll (all eligible clusters), then experiment with PickFixed (specific clusters) or PickN (top N clusters by label).\n• CRPs are hub-scoped: always kubectl apply them to your Fleet hub context, not member clusters.\nMonitoring, Security \u0026amp; Governance at Scale When you operate a single Kubernetes cluster, it’s already a challenge to keep observability, compliance, and security in sync. Managing a single Kubernetes cluster is tough enough to keep observability, compliance, and security in sync. But when you add various AKS clusters across regions, the challenge gets even tougher: you’ve got different telemetry pipelines, policies that start to drift apart, and a security team overwhelmed by disjointed alerts.\nFleet doesn’t replace your existing monitoring and security stack — instead, it gives you a central hub where those capabilities can be applied consistently.\nMonitoring Multi-Cluster Environments Azure Monitor for Containers, as a core component of Azure Monitor, serves as the primary tool for telemetry in container environments. It provides comprehensive metrics such as node and pod CPU and memory usage, kubelet health status, and API server responsiveness, enabling detailed performance monitoring. Additionally, it collects logs from containers and streams them into Log Analytics for centralized analysis and troubleshooting. The platform also offers prebuilt workbooks through Insights, which facilitate in-depth analysis of Kubernetes performance metrics and enable cost management, making it a vital resource for maintaining optimized and reliable containerized applications.\nWhen integrated with Fleet, all member clusters are onboarded to the same Log Analytics workspace, which provides a unified, single-pane view of telemetry data across the entire environment. This setup facilitates the creation of multi-cluster dashboards that enable the correlation of data across different regions or environments, such as assessing whether a workload is healthy in West Europe (WEU) versus North Europe (NEU). Additionally, alerts can be scoped at the Fleet level, allowing for comprehensive monitoring of services. For instance, an alert can be configured to trigger if the \u0026lsquo;prod-eu\u0026rsquo; namespace CRP fails in any cluster within the fleet, ensuring quick detection and response to issues regardless of the specific cluster affected.\nTip: Use Azure Managed Prometheus + Azure Managed Grafana for deep PromQL queries and custom dashboards. Grafana can be configured to query multiple Prometheus endpoints (one per cluster) and present them as a unified dashboard.\nSecurity Across the Fleet Securing multiple clusters requires a mix of preventive controls and detective measures.\nPreventive Security\nAzure Policy for AKS should be applied at the Fleet level to ensure that all new member clusters inherit consistent guardrails. This approach simplifies management and enforces standardized policies across the environment. For example, policies can be set to deny privileged containers, enforce HTTPS ingress exclusively, and restrict images to approved registries such as your Azure Container Registry. Additionally, maintaining RBAC consistency is vital; utilizing the Fleet to standardize role assignments helps ensure uniform access controls. It is advisable to keep cluster-admin roles minimal, delegating permissions through Azure RBAC integrated with Kubernetes to maintain a secure and manageable cluster ecosystem.\nDetective Security\nMicrosoft Defender for Containers, a component of Microsoft Defender for Cloud, offers comprehensive container security features. It actively scans container images stored within Azure Container Registry (ACR) to identify vulnerabilities before deployment. Additionally, it monitors runtime behavior within Azure Kubernetes Service (AKS) clusters to detect potential threats, such as crypto-mining activities and suspicious system calls, thereby enhancing the real-time security posture. Alerts generated by this service are primarily at the subscription level, providing a broad overview. Still, they can also be mapped back to specific fleet clusters for more granular incident response and management. This integrated approach helps organizations maintain a robust security environment across their containerized applications.\nCentralized Audit Logging: Stream Kubernetes audit logs to Log Analytics for correlation across all member clusters.\nTip: Pair Defender signals with Fleet labels, such as \u0026ldquo;show me vulnerabilities in compliance=eu clusters only”, so security teams can focus on prioritizing regulatory workloads.\nGovernance at Enterprise Scale Fleet shines in governance by letting you define policies once and apply them everywhere:\nClusterResourcePlacement as Governance\nWorkload placement is not the only aspect that can be managed using Custom Resource Policies (CRPs); baseline resources such as network policies, PodDisruptionBudgets, and RBAC roles can also be distributed in this manner. This approach ensures consistency across multiple clusters and simplifies the configuration management process. For example, you could deploy the same NetworkPolicy to all clusters labeled with env=prod, ensuring uniform network security policies are enforced across your production environment. This method provides a scalable and efficient way to manage common configurations in multi-cluster Kubernetes deployments.\nAzure Policy + Fleet Labels\nAzure Policy definitions can be scoped to a Fleet resource group or an entire subscription. Leveraging Fleet’s member labeling feature enables the enforcement of different policies based on environment-specific requirements. For instance, clusters labeled with env=dev might be permitted to use ephemeral storage, providing flexibility during development. In contrast, clusters designated with env=prod are subject to stricter controls, such as enforcing encryption at rest to ensure data security in production environments.\nCost Governance (FinOps)\nFleet does not directly manage costs; instead, it plays an indirect role by enforcing placement and scaling policies that influence overall financial outcomes. These policies help optimize resource utilization and operational efficiency, thereby impacting costs over time. To gain better visibility into these expenses, platform teams can utilize Azure Cost Management to group costs by cluster or fleet. This approach allows for a comprehensive view of the actual costs associated with multi-cluster operations, enabling more informed financial planning and resource allocation decisions.\nEnd-to-End Example: Governance in Action\nTo ensure robust security and proper network segmentation across all production clusters, it is essential to implement a comprehensive set of policies. This involves applying an Azure Policy Initiative that enforces strict governance. Specifically, privileged pods should be blocked to prevent unauthorized access or elevated permissions that could compromise the environment. Additionally, requiring namespaces to have designated labels helps improve resource organization, management, and policy enforcement. Furthermore, enforcing encryption on persistent volumes is critical to protecting sensitive data both at rest and in transit. Together, these measures form a layered security approach, aligning with best practices for managing production workloads in a secure, compliant manner.\nDistribute a baseline NetworkPolicy via CRP:\napiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny namespace: prod spec: podSelector: {} policyTypes: - Ingress - Egress CRP example:\napiVersion: placement.kubernetes-fleet.io/v1 kind: ClusterResourcePlacement metadata: name: prod-network-baseline spec: resourceSelectors: - group: networking.k8s.io version: v1 kind: NetworkPolicy name: default-deny namespace: prod policy: placementType: PickAll affinity: clusterAffinity: requiredDuringSchedulingIgnoredDuringExecution: clusterSelectorTerms: - labelSelector: matchLabels: env: prod Monitor with Azure Monitor and Defender for Containers\nAlerts are systematically collected and displayed on a centralized dashboard, allowing the security team to monitor and assess the compliance posture across all production clusters in real-time. This setup facilitates quicker identification of issues, more efficient responses, and overall improved security management.\nCase Study 1: EU Compliance (GDPR \u0026amp; NIS2) A European financial services company is subject to strict regulatory compliance requirements, notably the General Data Protection Regulation (GDPR) and the Network and Information Systems (NIS2) Directive. These regulations require that all customer data be stored and processed within the European Union\u0026rsquo;s territorial boundaries to ensure data sovereignty and enhance security.\nThe company operates on a global scale, offering customer-facing applications in both the United States and the European Union. While this international presence is vital for business growth and customer engagement, it introduces complex technical challenges. Specifically, EU regulations require that all workloads related to EU customers, including databases, processing services, and user data, must be hosted exclusively within EU data centers. This means the infrastructure must be segmented and managed in a way that keeps EU data isolated from non-EU data, ensuring compliance while still enabling seamless service across borders.\nImplementing such a setup requires robust data residency solutions, which may involve region-specific cloud regions, strict access controls, and effective data encryption strategies. Additionally, the company must continuously monitor and audit its infrastructure to maintain compliance with evolving regulations and to safeguard customer data across different jurisdictions.\nSolution with Fleet:\nCreate a Fleet hub in West Europe. Onboard West Europe and North Europe clusters. Create a PlacementPolicy to ensure that workloads tagged with \u0026lsquo;compliance=eu\u0026rsquo; only run in EU clusters. Assign US workloads to a separate placement in the East US cluster. Use Azure Policy to ensure only EU images and data are pulled. Architecture:\nAzure Front Door routes EU traffic to EU Fleet members. US traffic goes to East US cluster. Policies are enforced through Fleet and Azure Policy. Compliance guaranteed: no EU data ever leaves EU AKS clusters.\nEU-Only Workloads (GDPR/NIS2)\napiVersion: placement.kubernetes-fleet.io/v1 kind: ClusterResourcePlacement metadata: name: eu-only spec: resourceSelectors: - group: \u0026#34;\u0026#34; version: v1 kind: Namespace name: prod-eu policy: placementType: PickAll affinity: clusterAffinity: requiredDuringSchedulingIgnoredDuringExecution: clusterSelectorTerms: - labelSelector: matchLabels: compliance: eu Workloads restricted to EU clusters only — regulatory compliance enforced by placement.\nCase Study 2: Disaster Recovery (DR) Failover A global e-commerce platform runs its primary AKS cluster in West, and an international e-commerce platform operates its primary Azure Kubernetes Service (AKS) cluster in West Europe. To ensure business continuity and minimize risk of outages, they have set up a Disaster Recovery (DR) cluster in North Europe as a backup.\nIn the past, managing this setup involved manually running scripts to replicate workloads and changing DNS entries to redirect traffic during potential issues. This manual process was often slow and could lead to mistakes, making it less reliable for such critical infrastructure.\nSolution with Fleet:\nBoth clusters onboarded to Fleet. Workloads deployed with PlacementPolicy prioritizing West Europe. North Europe defined as passive cluster with zero replicas until failover. Azure Front Door automatically detects regional health and initiates failover. During DR testing, workloads automatically scaled in North Europe. RTO reduced from 2 hours to 15 minutes.\nDR tests became automated instead of manual.\nDisaster Recovery (Active/Passive)\napiVersion: placement.kubernetes-fleet.io/v1 kind: ClusterResourcePlacement metadata: name: prod-dr spec: resourceSelectors: - group: \u0026#34;\u0026#34; version: v1 kind: Namespace name: prod policy: placementType: PickFixed clusterNames: - aks-weu # primary - aks-neu # dr strategy: type: RollingUpdate rollingUpdate: maxSurge: 25% maxUnavailable: 25% unavailablePeriodSeconds: 60 To keep the DR cluster at replicas: 0 until failover:\n• Use overrides (CRP applied only to DR cluster).\n• Or manage skew via GitOps overlays.\nActive cluster in West Europe, standby cluster in North Europe. Azure Front Door handles failover.\nSidebar: Hub vs. Hubless Fleet\nHubless Fleet: Used for upgrading clusters only.\nHubful Fleet: Needed for workload placement, CRPs, and DNS load balancing.\nYou can upgrade from hubless to hubful, but not the other way around. The Hub API’s exposure (public or private) is fixed.\nWrapping Up Managing a single Kubernetes cluster is already complex, but overseeing multiple AKS clusters, whether ten, twenty, or fifty, can quickly become overwhelming. Without proper management, this cluster sprawl can lead to inconsistent upgrades, security gaps, and workloads that drift from compliance standards.\nAzure Kubernetes Fleet Manager simplifies this challenge. Its hub-and-member model provides a centralized control point that doesn’t require consolidating all clusters into a single large one. This approach maintains the flexibility of multiple AKS clusters while enabling centralized governance and easier management.\nBased on the case studies, here are some encouraging lessons to keep in mind:\nCompliance is straightforward: You don’t need complicated scripts to ensure EU workloads stay in EU regions. CRPs and cluster labels make everything clear and easy to audit. Resilience becomes simpler: Active/passive disaster recovery setups that used to rely on fragile pipelines and DNS tricks are now just a matter of CRPs and policies, seamlessly integrated with Azure Front Door. Observability is unified: Azure Monitor, Defender for Cloud, and Fleet policies work together to help you catch issues early before they turn into severe outages. Scalability is ready for growth: Whether you’re managing 3 clusters today or 30 tomorrow, Fleet offers a consistent and reliable way to handle them.\nMost importantly, Fleet doesn’t require you to change how your developers work. Your application manifests, Helm charts, and GitOps pipelines remain familiar; Fleet extends them smoothly across multiple clusters in a controlled and secure manner.\nFor platform engineers, this means less urgent firefighting and more time to drive innovation. For compliance teams, it provides clear and enforceable guardrails. And for developers, it ensures their applications land correctly every single time, without adding extra complexity. If you’re just getting started:\nSet up a Fleet hub in a test subscription. Onboard two AKS clusters in a test Azure subscription to set up a Fleet hub and serve as a centralized management point. Then, onboard two Azure Kubernetes Service (AKS) clusters located in different regions to ensure global distribution and high availability. Deploy a straightforward workload, such as a sample web application, on these clusters. Test various Cluster Resource Placement (CRP) selectors, including PickAll (which selects all available clusters), PickFixed (which targets a specific fixed cluster), and PickN (which targets a specified number of clusters), to evaluate flexible workload placement strategies.\nNext, incorporate Azure Front Door to efficiently manage and route multi-region traffic, enabling a seamless user experience across different geographies. Use GitOps practices leveraging tools like Argo CD or Flux to automate all deployment and configuration processes, ensuring continuous delivery and infrastructure as code.\nBy following this setup, typically achievable within a couple of hours, you’ll gain insight into why Fleet is considered a robust and scalable tool for managing enterprise Kubernetes environments across multiple regions with automation, flexibility, and high availability.\nIf you enjoyed this article, follow me for more deep dives into Azure Kubernetes, Fleet, and modern cloud-native operations.\n","permalink":"https://wolkwacht.nl/posts/azure-kubernetes-chronicles-6/","summary":"\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*LANSyYGz8foib5-v-RrorA.png\"\u003e\u003c/p\u003e\n\u003ch2 id=\"introduction-why-centralized-management-of-aks-clustersmatters\"\u003eIntroduction: Why Centralized Management of AKS Clusters Matters\u003c/h2\u003e\n\u003cp\u003eKubernetes adoption in enterprises has skyrocketed. What begins as a single proof-of-concept cluster often grows into dozens across various teams, environments, and regions. For organizations using Azure Kubernetes Service (AKS), this growth is even more common: different business units create clusters for their workloads, data residency rules require data to stay in specific locations, and production systems need failover capabilities across regions.\u003c/p\u003e","title":"Azure Kubernetes Chronicles 6:"},{"content":"Digital Sovereignty and the Public Cloud: Navigating Azure in a European context Introduction: Why digital sovereignty matters Over the past decade, cloud adoption has shifted from an optional innovation to the standard approach for many. In Europe, Microsoft Azure, Amazon Web Services, and Google Cloud are the foundation of digital infrastructure. However, a key question has arisen: who really controls your data and workloads?\nThis is the heart of digital sovereignty, the ability of a state, organization, or individual to shape their digital future. It’s not just about where data is stored, but also about who can access it, under what laws, and with what safeguards in place. For cloud engineers and DevOps professionals, sovereignty may have seemed abstract until it began to affect core architecture decisions, such as encryption, multi-cloud strategy, and operational governance.\nJoin us as we dive into:\n• What digital sovereignty looks like in Europe.\n• The regulatory and political challenges shaping it.\n• Clearing up common misconceptions.\n• Your Azure options.\n• Why hybrid cloud often makes the most sense.\n• The trade-offs that come with it.\n• What engineers need to prepare for.\nThe European landscape: regulation, politics, and sovereign tensions Europe’s cloud infrastructure and policy landscape are heavily influenced by a complex interplay of regulatory frameworks, data privacy laws, and strategic priorities focused on digital sovereignty. The General Data Protection Regulation (GDPR) sets a strong compliance baseline, requiring organizations to enforce strict controls over data location, transfer methods, and access permissions to protect privacy and security. However, some legal uncertainties remain, especially after the Schrems II ruling, which invalidated the EU-US Privacy Shield and created new hurdles for cross-border data transfers. Moreover, the U.S. CLOUD Act allows U.S. authorities to access data stored abroad, including European data held by U.S. cloud providers, adding complications for European companies seeking to use cloud services worldwide.\nTo address these challenges, recent EU directives such as NIS2 and DORA have been put in place to greatly improve cybersecurity resilience across critical infrastructure sectors and strengthen vendor management controls. These directives are part of a broader strategic effort to protect digital assets and ensure operational continuity. At the same time, initiatives like GAIA-X and domestic cloud providers such as OVHcloud, Scaleway, and Deutsche Telekom are being developed to promote digital sovereignty, lowering reliance on foreign cloud services, and fostering a secure and innovative digital economy within Europe.\nHowever, despite these regional advancements, these solutions often face significant obstacles in scaling, maturing, and achieving the operational robustness needed to match the capabilities of established hyperscalers like Amazon Web Services, Microsoft Azure, and Google Cloud. These global cloud giants have extensive infrastructure, advanced technological ecosystems, and a broad service portfolio that enable them to serve diverse enterprise needs worldwide.\nThe difference in size and development affects the regional solutions’ competitive position and their ability to integrate smoothly into the large, connected global cloud ecosystem. Overcoming these issues is essential for securing European digital sovereignty, staying competitive, and encouraging innovation in cloud technology. The hyperscaler dilemma: Azure’s benefits and sovereignty challenges\nMicrosoft Azure is widely regarded as one of the most trusted cloud platforms in Europe, primarily due to its robust data security and compliance features. Its EU Data Boundary initiative guarantees that customer data stored and processed across Azure, Microsoft 365, Dynamics 365, and Power Platform remains within the EU/EFTA regions, complying with regional data sovereignty laws. However, as a U.S.-based company, Azure remains subject to the CLOUD Act, meaning that U.S. authorities could potentially access data stored in EU data centers under lawful orders. Many organizations depend heavily on Azure\u0026rsquo;s Platform-as-a-Service (PaaS) offerings, which can lead to vendor lock-in, making it difficult to switch providers. This reliance results in operational inertia, as teams, governance frameworks, automation processes, and tooling are often heavily centered around Azure, creating significant challenges when attempting to adopt alternative cloud solutions or migrate workloads.\nClearing up misconceptions about digital sovereignty When discussing digital sovereignty, the conversation often becomes entangled in a mix of legal terms, political statements, and technical jargon. The real meat of the issue, though, lies somewhere in between: the practical reality of how tech choices impact strategy, compliance, and growth. This is where misconceptions take hold. Some think sovereignty is merely a legal issue, while others view it as a technical formality, and still others assume it means giving up innovation altogether. To make progress, we need to sort fact from fiction and recognize sovereignty as a blend of tech integration, innovation, and long-term strategic planning.\nMyth 1: “If my data is stored in an EU data center, I am considered sovereign over it.”\nReality: Although the physical location of data centers is important, true data sovereignty also relies on the legal jurisdiction. For instance, the CLOUD Act can still apply, impacting data stored in EU data centers. Recognizing both physical and legal factors is essential for maintaining data sovereignty.\nMyth 2: “Going sovereign means abandoning the cloud.”\nReality: It is a common misconception that pursuing sovereignty requires leaving cloud environments. In fact, organizations can remain within cloud platforms like Azure while ensuring data sovereignty through advanced technical strategies such as implementing customer-managed encryption keys, utilizing confidential computing for secure data processing, and leveraging the EU Data Boundary to comply with regional data regulations. These options enable a balanced approach that combines the benefits of cloud scalability and innovation with strict adherence to sovereignty and compliance requirements, supporting both business agility and strong security.\nMyth 3: “European providers match Azure feature-for-feature.”\nReality: While sovereign clouds ensure regulatory compliance and meet regional data sovereignty needs, they currently lack the same level of advanced artificial intelligence features, extensive global platform-as-a-service (PaaS) options, or the vibrant developer community that Azure offers. This often affects their suitability for businesses looking for cutting-edge technology and broad international scalability.\nMyth 4: “Hybrid is too complex to be sovereign.”\nReality: With the emergence of advanced tools like Azure Arc, Kubernetes, and standards from the Cloud Native Computing Foundation (CNCF), managing hybrid and multi-cloud environments has become easier and more manageable. These technologies offer unified control, automate complex processes, and lower technical barriers, thereby simplifying operations and enabling organizations to maintain sovereignty and control over their hybrid infrastructures.\nMyth 5: “Digital sovereignty is solely a legal issue.”\nThis reality also significantly influences key business and technical decisions. These decisions encompass critical areas, including system architecture, encryption method choices, workload placement strategies, and ensuring portability across various platforms. Addressing these aspects is crucial for maintaining control, enhancing security, and enabling flexibility in digital operations.\nStrategic Paths for Azure-Centric Enterprises If you’re deeply invested in Azure today, here are your pragmatic paths forward:\nStay with Azure — Sovereign by Design Business Perspective: Leverage Azure’s strategic European data centers to ensure compliance with regional data sovereignty laws, enhancing customer trust and enabling smoother regulatory approvals. Technical Perspective: Utilize EU/EFTA regions and Azure’s EU Data Boundary to store data within specified jurisdictions, applying customer-managed encryption, confidential compute, and double-key models to ensure data security and privacy. Additional Control Measures: Rely on contractual commitments such as EU Standard Contractual Clauses to formalize data protection obligations, strengthening control over data handling and compliance. Hybrid Sovereignty Business Perspective: Balance innovation and compliance by running sensitive data and critical workloads in sovereign environments while deploying less sensitive, innovative workloads on Azure, optimizing costs and agility. Technical Perspective: Manage workloads through Azure Arc, enabling centralized governance across multiple environments, ensuring consistent security policies and compliance. Example Scenario: A banking institution processes sensitive payment data within a sovereign cloud, ensuring data sovereignty, while deploying mobile and web applications in Azure for scalability and rapid development. Multi-Cloud / Exit Strategy Business Perspective: Reduce dependency on a single cloud provider by deploying applications across Azure and other EU cloud providers, enhancing resilience, negotiating power, and avoiding vendor lock-in. Technical Perspective: Maintain portability and flexibility using containerization, Kubernetes, and Infrastructure as Code (IaC) practices to facilitate seamless migration and interoperability. Trade-offs: Accept increased complexity and potentially higher costs in exchange for greater independence and resilience. Why Hybrid Cloud often makes the most sense Hybrid architecture might appear like a compromise at first, but in the area of data sovereignty and enterprise technology, it usually stands as the most balanced and strategic choice.\nAdvantages include:\nGreater control over sensitive and critical data, ensuring compliance with local regulations and security policies. Fostering innovation by leveraging cloud services like Azure, which offers advanced tools and scalable infrastructure. Avoiding the pitfalls of choosing only hyperscalers or solely sovereign solutions, thus maintaining flexibility.\nHowever, there are challenges to consider: Managing operational complexity, especially in identity management, networking configurations, and policy alignment across different environments. Increased costs associated with maintaining and operating dual or hybrid environments. Necessity for robust governance frameworks to ensure security, compliance, and consistent policies across all platforms. Real-World Examples:\nGovernment: Deploy a secure citizen registry system on sovereign infrastructure, using Azure Sovereign cloud services to ensure jurisdictional compliance and improved security. This includes encrypted data storage, identity management, and access controls through Azure Active Directory.\nHealthcare: Deploying protected patient data management systems on local cloud environments, in compliance with healthcare regulations like HIPAA, and additionally, integrating AI-driven diagnostic tools using Azure Machine Learning to support clinicians with quick and accurate disease detection.\nManufacturing: Deploying edge computing devices for real-time IoT data collection and processing on-site, combined with advanced analytics and data visualization on Azure Synapse Analytics for predictive maintenance, operational efficiency, and supply chain optimization.\nBusiness \u0026amp; Political trade-offs Digital sovereignty comes with undeniable trade-offs:\nMost enterprises will land somewhere in between — balancing speed and innovation with regulatory needs and political reality.\nWhat Engineers need to prepare for If you’re designing the future of your technology infrastructure, consider adopting comprehensive security and deployment strategies that align with modern business demands:\n• Implement end-to-end encryption protocols, including customer key management and confidential computing, to ensure data security and build customer trust.\n• Prioritize workload placement based on strategic considerations, making sure not all functions are confined to a single cloud provider like Azure. This approach allows for flexibility, cost savings, and risk reduction.\n• Invest in gaining expertise in Azure Arc, hybrid cloud governance, and multi-cloud orchestration tools to manage diverse environments efficiently and securely.\n• Design your infrastructure for portability using Infrastructure as Code (IaC), containerization, and open standards such as those from the Cloud Native Computing Foundation (CNCF). This prevents vendor lock-in and promotes agility.\n• **Understand and translate sovereignty requirements (**regulatory or political) into specific architectural decisions, ensuring compliance while maintaining operational resilience.\nBy applying these principles, your architecture can support scalable, secure, and compliant business operations that adapt to the changing cloud landscape.\nConclusion: Pragmatism over purism Today, digital sovereignty has become a crucial strategic focus in adopting and deploying cloud computing services, shifting from a niche concern to a core requirement for organizations operating in highly regulated or geopolitically sensitive areas. These organizations need to develop a nuanced approach that balances key factors such as:\n• Innovation: utilizing cloud platforms like Azure, which offers a comprehensive suite of cloud services including compute, storage, AI, and IoT integrations, all supported by a vast global network footprint. This enables quick development and scaling of applications while allowing the localization of services as needed.\n• Control: ensuring compliance with legal, data residency, and privacy regulations such as GDPR, HIPAA, or regional data sovereignty laws by implementing detailed access controls, data encryption, and audit logging within both the cloud environment and sovereign-specific infrastructure.\n• Cost Optimization: managing budgets while considering the deployment of specialized sovereign infrastructure, such as private clouds or on-premises solutions, which may involve hardware procurement, maintenance, and integration with existing legacy systems.\n• Delivery Speed: balancing the agility provided by cloud services with the complexity of regulatory requirements, which calls for automated provisioning, configuration management, and continuous integration/continuous deployment (CI/CD) pipelines that include compliance checks.\n• Political and Regulatory Alignment: supporting initiatives like the Digital Single Market or European Cloud initiatives to ensure compliance with regional policies without compromising the enterprise’s global competitiveness.\nRather than choosing solely between hyperscalers or sovereign solutions, a hybrid cloud architecture often provides the best approach by combining the scalability, extensive ecosystem, and advanced services of providers like Azure with dedicated sovereign infrastructure for sensitive workloads.\nAs cloud engineers, architects, and DevOps professionals, our roles are evolving. We are now responsible for translating strategic, legal, and political directives into scalable, automated infrastructure and deployment pipelines. Designing with sovereignty in mind isn’t about rejecting cloud computing but about enhancing and customizing it to meet specific national or organizational sovereignty requirements, ensuring compliance without sacrificing innovation or agility.\nRelevant Links \u0026amp; Resources: Europe’s push for digital sovereignty and challenges (TechRadar/FT) CLOUD Act implications and EU conflict (GDPR vs CLOUD Act) — clarity on legal tension. Microsoft EU Data Boundary overview — ensures data resides and processes within EU/EFTA regions. AWS European Sovereign Cloud Microsoft Sovereign Cloud ","permalink":"https://wolkwacht.nl/posts/digital-sovereignty-and-the-public-cloud/","summary":"\u003ch2 id=\"digital-sovereignty-and-the-public-cloud-navigating-azure-in-a-europeancontext\"\u003eDigital Sovereignty and the Public Cloud: Navigating Azure in a European context\u003c/h2\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*mK8O3BO5KGnJVE4b4dyHUQ.png\"\u003e\u003c/p\u003e\n\u003ch2 id=\"introduction-why-digital-sovereignty-matters\"\u003eIntroduction: Why digital sovereignty matters\u003c/h2\u003e\n\u003cp\u003eOver the past decade, cloud adoption has shifted from an optional innovation to the standard approach for many. In Europe, Microsoft Azure, Amazon Web Services, and Google Cloud are the foundation of digital infrastructure. However, a key question has arisen: who really controls your data and workloads?\u003c/p\u003e\n\u003cp\u003eThis is the heart of digital sovereignty, the ability of a state, organization, or individual to shape their digital future. It’s not just about where data is stored, but also about who can access it, under what laws, and with what safeguards in place. For cloud engineers and DevOps professionals, sovereignty may have seemed abstract until it began to affect core architecture decisions, such as encryption, multi-cloud strategy, and operational governance.\u003c/p\u003e","title":"Digital Sovereignty and the Public Cloud:"},{"content":"Harnessing Chaos: Implementing Chaos Engineering with Azure Chaos Studio and Gremlin on AKS Chaos Engineering is a crucial practice for modern cloud operations, enabling teams to identify hidden weaknesses and potential failures in complex systems before they become a problem. By deliberately causing failures in a controlled setting, you can observe how your system responds, measure its resilience, and ultimately strengthen its robustness. This practice really stands out in cloud-native environments like Azure Kubernetes Service (AKS), where the details and interconnections can sometimes obscure vulnerabilities.\nAzure Chaos Studio is a set of cloud-native tools designed explicitly for Azure environments, seamlessly integrating, scaling, and providing insights directly within the Azure ecosystem. When paired with Gremlin — a widely used chaos engineering platform — organizations can craft thorough chaos engineering strategies that enhance system reliability, security, and operational performance.\nThis blog will discuss the architecture, practical setup steps, scripts, and validation techniques, all closely aligned with the principles outlined in Azure’s Well-Architected Framework. Through real examples and tested scripts, you’ll discover how to leverage the power of controlled chaos to build resilient and dependable cloud solutions.\nChaos Engineering and Azure’s Well-Architected Framework: Chaos Engineering directly supports several core pillars of Azure’s Well-Architected Framework, helping organizations maintain high-quality cloud services. Below is a detailed exploration of how Chaos Engineering aligns with specific pillars of this framework:\nReliability Chaos Engineering directly contributes to the reliability pillar by proactively discovering and mitigating potential failures. It enables teams to identify weaknesses in system design and implementation before they become outages. Through controlled fault injection, teams can ensure systems gracefully handle disruptions, leading to improved reliability metrics such as Mean Time Between Failures (MTBF) and Mean Time To Recover (MTTR).\nAzure Chaos Studio experiments, such as pod disruption or node stress tests, help validate redundancy and failover strategies, ensuring resilience in Azure Kubernetes Service (AKS) deployments.\nSecurity By simulating real-world security threats or disruptions, Chaos Engineering validates security controls and incident response procedures. It ensures that security mechanisms, like identity and access management (IAM), network isolation, and threat detection, function correctly under abnormal conditions. Azure Chaos Studio and Gremlin provide scenarios for network latency and resource exhaustion, testing the robustness of security boundaries and alerting systems.\nOperational Excellence Chaos Engineering enhances operational excellence by refining processes, practices, and tools for managing cloud infrastructure. Regular chaos experiments promote continuous learning, helping teams to improve incident response, monitoring, and automation practices. This disciplined approach supports consistent operational performance and adaptability, enabling teams to identify and rectify issues promptly. Azure Chaos Studio integrates directly into operational dashboards and monitoring solutions, ensuring that experiments contribute meaningfully to operational insights and improvements. By aligning Chaos Engineering practices with Azure’s Well-Architected Framework, organizations ensure they build robust, secure, and efficiently managed cloud-native applications and infrastructure.\nOverview of Azure Chaos Studio Azure Chaos Studio is a fully managed service offered by Microsoft that simplifies the process of implementing chaos engineering practices within Azure cloud environments. By allowing teams to perform controlled, deliberate fault injections into their cloud services, Azure Chaos Studio helps identify potential weaknesses, enhances service resilience, and facilitates proactive operational management.\nCapabilities and Key Features\nComprehensive Chaos Library: Pre-defined chaos experiments that simulate common disruptions like resource exhaustion, network latency, service outages, and application faults. Integration with Azure Ecosystem: Native support for various Azure resources such as Azure Kubernetes Service (AKS), Azure App Service, Virtual Machines, and Azure Functions. Custom Experimentation: Users can create custom chaos experiments explicitly tailored to their environments and scenarios. Observability and Analysis: Built-in monitoring and reporting capabilities that provide insights and actionable recommendations for improvement. Supported Azure Services:\nAzure Kubernetes Service (AKS) Azure Virtual Machines Azure App Service Azure Cosmos DB Azure Functions Azure Service Bus Integration Points and Prerequisites:\nAzure Role-Based Access Control (RBAC) to manage permissions securely. Azure Monitor integration for real-time tracking and alerts. Chaos Agents installation for orchestrating experiments within Kubernetes clusters. Prerequisites include enabled resource providers in Azure and appropriate access permissions for chaos experiment resources. Azure Chaos Studio significantly simplifies the setup and execution of chaos experiments, making it an essential tool for maintaining robust and reliable cloud infrastructures on Azure.\nOverview of Gremlin: Gremlin is a robust and widely adopted chaos engineering platform that enables organizations to safely and systematically conduct chaos experiments across various infrastructures, including cloud, containers, and Kubernetes. Gremlin helps engineering teams proactively identify and address resilience gaps, validate service-level objectives (SLOs), and strengthen overall system reliability.\nKey Features and Capabilities:\nDiverse Attack Library: Extensive range of built-in chaos scenarios such as CPU, memory, disk, network disruptions, and process termination. Platform Agnostic: Supports multiple cloud providers, container environments, Kubernetes clusters, and hybrid deployments. Granular Control and Scheduling: Allows detailed control over experiment parameters and timing, including scenario scheduling, automated chaos tests, and targeted experiments. Rich Observability: Integrated monitoring and reporting that supports detailed analysis and continuous improvement practices. Differences and Synergies with Azure Chaos Studio:\nWhile Azure Chaos Studio is tightly integrated within the Azure ecosystem, Gremlin provides broader cross-platform support, including hybrid and multi-cloud environments. Gremlin’s advanced scenarios complement Azure Chaos Studio, providing more profound and more complex fault injection capabilities. By combining both platforms, organizations can leverage Azure’s native integrations for streamlined operations and Gremlin’s flexibility for broader, more complex scenarios. When to Consider Using Gremlin:\nMulti-cloud or hybrid environments that require a single chaos platform. Complex or large-scale chaos experiments that go beyond Azure-native capabilities. Teams seeking advanced scheduling, automation, and comprehensive reporting functionalities. Architecture: Integrating Azure Chaos Studio with AKS: Integrating Azure Chaos Studio with Azure Kubernetes Service (AKS) provides a structured and efficient approach for performing chaos experiments on containerized applications. The architecture involves several key components and configurations designed to ensure seamless integration, security, and observability.\nArchitectural Components: Azure Kubernetes Service (AKS) Cluster: Hosts the containerized applications and workloads. Chaos Agent: A Kubernetes-based component deployed onto AKS, responsible for executing the chaos experiments defined in Azure Chaos Studio. Azure Chaos Studio Resources: Includes Chaos Experiments, Targets, and Actions managed via Azure portal or Azure CLI. Azure Role-Based Access Control (RBAC): Ensures secure and appropriate access levels for chaos resources and AKS integration. Azure Monitor and Log Analytics: Provides visibility and insights through telemetry and logging of chaos experiment outcomes. Architectural Diagram: [User] \u0026lt;--\u0026gt; [Azure Portal/Azure CLI] | v [Azure Chaos Studio] | v [Chaos Experiments] ---\u0026gt; [Chaos Agent] | v [AKS Cluster] | v [Containerized Workloads] | v [Azure Monitor \u0026amp; Log Analytics] Explanation of Integration Workflow:\nDefine Chaos Experiments: Configure experiments within Azure Chaos Studio, specifying scenarios like pod failures, resource stress, or network issues. Deploy Chaos Agent: Install the Chaos Agent into your AKS cluster, which interacts directly with Kubernetes resources to perform experiments. Execute Experiments: Trigger experiments from the Azure portal or Azure CLI. The Chaos Agent applies the defined chaos actions to targeted resources within AKS. Monitor and Analyze: Use Azure Monitor and Log Analytics to gather insights, observe experiment outcomes, and measure application resilience. This architecture ensures a secure, efficient, and observable approach to chaos engineering within AKS environments, helping teams proactively improve reliability and resilience.\nStep-by-Step Walkthrough: Implementing Chaos Engineering on AKS with Azure Chaos Studio: Step 1: Setting up AKS Cluster\nProvision an AKS cluster via the Azure Portal or Azure CLI: az group create --name chaosdemo --location westeurope az aks create --resource-group chaosdemo --name akschaos --node-count 3 --generate-ssh-keysAKSCluster --node-count 3 --generate-ssh-keys Ensure kubectl is configured: az aks get-credentials --resource-group chaosdemo --name akschaos Deploy the sample app to disrupt kubectl create deployment nginx --image=nginx kubectl scale deployment nginx --replicas=2 Validate that the sample app is running kubectl get pods -n default Step 2: Configuring Azure Chaos Studio\nTo use Chaos Studio with Azure Kubernetes Service, Chaos Studio currently depends on Chaos Mesh, a free, open-source chaos engineering platform for Kubernetes. To add Chaos Mesh to the AKS cluster, use the following commands:\nhelm repo add chaos-mesh https://charts.chaos-mesh.org helm repo update kubectl create ns chaos-testing helm install chaos-mesh chaos-mesh/chaos-mesh --namespace=chaos-testing --set chaosDaemon.runtime=containerd --set chaosDaemon.socketPath=/run/containerd/containerd.sock Verify that Chaos Mesh was installed successfully.\nkubectl get pods -n chaos-testing Register required providers\naz provider register --namespace Microsoft.Chaos az provider register --namespace Microsoft.ContainerService Check the registration status and ensure that both are Registered\naz provider show --namespace Microsoft.Chaos --query \u0026#34;registrationState\u0026#34; az provider show --namespace Microsoft.ContainerService --query \u0026#34;registrationState\u0026#34; Assign Chaos Studio permissions:\nNavigate to Azure portal \u0026gt; Chaos Studio \u0026gt; Targets. Select the AKS cluster. Choose enable targets Step 4: Defining and Executing Chaos Experiments Select the Experiments tab in Chaos Studio. In this view, you can see and manage all your chaos experiments. Select Create \u0026gt; New experiment. Fill in the Subscription, Resource Group, and Location where you want to deploy the chaos experiment. Give your experiment a name. Select Next: Experiment designer. You’re now in the Chaos Studio experiment designer. The experiment designer lets you build your experiment by adding steps, branches, and faults. Give a friendly name to your Step and Branch, then select Add action \u0026gt; Add fault. Select AKS Chaos Mesh Pod Chaos from the dropdown list. Fill in Duration with the number of minutes you want the failure to last, and jsonSpec with the following information: To formulate your Chaos Mesh jsonSpec: Refer to the Chaos Mesh documentation for a specific fault type, such as PodChaos. Formulate the YAML configuration for that fault type by using the Chaos Mesh documentation. apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: pod-kill-example namespace: chaos-testing spec: action: pod-kill mode: one selector: namespaces: default Use a YAML-to-JSON converter, such as this one, to convert the Chaos Mesh YAML to JSON and minimize it. {\u0026#34;action\u0026#34;:\u0026#34;pod-failure\u0026#34;,\u0026#34;mode\u0026#34;:\u0026#34;all\u0026#34;,\u0026#34;selector\u0026#34;:{\u0026#34;namespaces\u0026#34;:[\u0026#34;default\u0026#34;]}} Paste the minimized JSON into the jsonSpec field in the portal.\nSelect next: Target Resources, and select the AKS cluster.\nReview and Create, Create\nAfter the experiment is created, select the experiment and choose run.\nWhen the experiment status is changed to \u0026lsquo;running\u0026rsquo;, select \u0026lsquo;Details\u0026rsquo; and \u0026lsquo;History\u0026rsquo;.\nStep 5: Visualize the result\nAzure Chaos Studio emits diagnostic logs and metrics, which must be explicitly configured for export to Azure Monitor.\nGo to the Azure Portal.\nNavigate to Chaos Studio \u0026gt; Experiments.\nSelect your Chaos Experiment.\nUnder the Monitoring section, choose Diagnostic settings.\nClick Add diagnostic setting.\nProvide a name (e.g., ChaosStudioLogs). Enable logging categories such as: Experiment execution details Experiment resource operations Select a destination, typically:\nLog Analytics Workspace (recommended for deeper analysis). Click Save. Verify Data Flow in Azure Monitor Logs (Log Analytics)\nEnsure Chaos Studio logs are being sent correctly:\nNavigate to your Log Analytics Workspace in Azure Portal Under General, select Logs. Run a simple KQL query to verify incoming logs: #kql AzureDiagnostics | where ResourceProvider == \u0026#34;MICROSOFT.CHAOS\u0026#34; | sort by TimeGenerated desc | limit 50 Logs appearing indicates successful integration.\nUseful KQL Queries are the starting point for the dashboards or monitoring:\nRecent Experiment Results\n#kql AzureDiagnostics | where ResourceProvider == \u0026#34;MICROSOFT.CHAOS\u0026#34; | project TimeGenerated, OperationName, ExperimentName = resourceName_s, ResultDescription | order by TimeGenerated desc | limit 100 Experiment Failure Details:\n#kql AzureDiagnostics | where ResourceProvider == \u0026#34;MICROSOFT.CHAOS\u0026#34; | where Level == \u0026#34;Error\u0026#34; | project TimeGenerated, ExperimentName = resourceName_s, ResultDescription, Level, OperationName | order by TimeGenerated desc Visualize Data with Azure Monitor Dashboards\nCreate insightful visualizations by leveraging Azure Monitor dashboards:\n1. In your Log Analytics workspace, select Logs.\n2. Run your KQL query.\n3. Once satisfied, select Pin to dashboard.\n4. Choose an existing dashboard or create a new one.\nSuggested visualizations include:\n• Time charts for experiment executions and outcomes.\n• Pie charts to summarize experiment success/failure ratios.\nExample KQL for a pie chart of experiment results:\n#kql AzureDiagnostics | where ResourceProvider == \u0026#34;MICROSOFT.CHAOS\u0026#34; | summarize count() by ResultDescription | render piechart Set Up Alerts for Chaos Studio Events\nConfigure proactive monitoring and notifications:\n1. Navigate to your Log Analytics workspace or directly from Azure Monitor.\n2. Go to Alerts \u0026gt; Create alert rule.\n3. Define condition with a query, e.g., alert on failed experiments:\n#kql AzureDiagnostics | where ResourceProvider == \u0026#34;MICROSOFT.CHAOS\u0026#34; | where Level == \u0026#34;Error\u0026#34; Set thresholds, evaluation intervals, and define action groups for notifications. Continuous Monitoring and Improvements\n• Regularly review dashboards and logs to spot recurring failures or weaknesses in resilience.\n• Adjust and improve Chaos Experiments based on insights.\nIntegrating Gremlin for Enhanced Capabilities: Integrating Gremlin with AKS, alongside Azure Chaos Studio, provides extended flexibility and advanced capabilities, which are especially beneficial for complex scenarios or multi-cloud deployments. Gremlin complements Azure Chaos Studio by offering additional scenarios and enhanced control over chaos experiment executions.\nStep-by-Step Gremlin Integration with AKS\nStep 1: Creating a Gremlin Account\nSign up for a Gremlin account at Gremlin Sign-up. Step 2: Installing Gremlin Agent on AKS\nObtain Gremlin credentials from the Gremlin portal. Deploy Gremlin using Helm: kubectl create namespace gremlin helm repo add gremlin https://helm.gremlin.com helm install gremlin gremlin/gremlin --namespace gremlin \\ --set gremlin.secret.managed=true \\ --set gremlin.teamID=\u0026lt;YOUR_TEAM_ID\u0026gt; \\ --set gremlin.teamSecret=\u0026lt;YOUR_TEAM_SECRET\u0026gt; Step 3: Validate Gremlin Agent Deployment\nConfirm pods are running: kubectl get pods -n gremlin Step 4: Execute Gremlin Chaos Experiments\nUse Gremlin UI or CLI to define and execute experiments such as resource stress or network latency. Example CLI command to create a CPU stress experiment: gremlin attack cpu --length 120 --cores 2 Example Gremlin Scenarios Complementary to Azure Chaos Studio:\nAdvanced Network Attacks\nDNS Failure Packet Loss Network Blackhole Comprehensive Resource Saturation\nDisk I/O saturation Memory exhaustion CPU overload Best Practices and Lessons Learned: Implementing Chaos Engineering effectively requires thoughtful preparation and disciplined execution. Below are best practices and key lessons learned from real-world implementations:\nBest Practices Start Small and Incrementally Scale\nBegin with small-scale experiments on non-critical environments. Gradually scale up to production environments as confidence and maturity grow. Define Clear Objectives\nClearly define the scope and goals of each chaos experiment. Establish measurable success criteria that align with business objectives and strategic learning objectives (SLOs). Ensure Observability and Monitoring\nIntegrate experiments with robust monitoring systems such as Azure Monitor and Log Analytics. Keep detailed logs and metrics to accurately analyze results and evaluate their effect on performance. Communicate Across Teams\nInform and involve stakeholders across development, operations, security, and management teams. Document and communicate experiment schedules, expected outcomes, and contingency plans. Automate Chaos Experiments Automate experiments using CI/CD pipelines or scheduled runs for consistent and repeatable chaos testing. Utilize scripting and Infrastructure as Code (IaC) to maintain control and versioning of experiments. Lessons Learned: Understand System Dependencies Chaos experiments often uncover unexpected dependencies. Clearly map and understand system relationships and dependencies before executing large-scale experiments. Expect the Unexpected\nAlways have rollback and recovery plans ready. Experiments can have unanticipated consequences, so it is essential to prepare thoroughly for a rapid recovery. Continuous Learning and Improvement\nTreat Chaos Engineering as a continuous process of learning and improving system reliability. Regularly review experiment outcomes and update architecture and practices accordingly. Align with Incident Response Procedures\nChaos Engineering should align closely with existing incident response procedures. Use chaos experiments to validate and improve incident response plans and training. By following these best practices and incorporating lessons learned, organizations can maximize the effectiveness of Chaos Engineering, significantly enhancing the resilience, reliability, and security of their cloud infrastructure and applications.\nReferences and Resources: To further explore Chaos Engineering with Azure Chaos Studio and Gremlin, consider reviewing the following resources and documentation:\nOfficial Documentation:\nAzure Chaos Studio Documentation Gremlin Documentation Azure Well-Architected Framework:\nReliability Pillar Security Pillar Operational Excellence Pillar Azure Kubernetes Service (AKS):\nAzure Kubernetes Service (AKS) Documentation AKS Best Practices Community and External Resources:\nPrinciples of Chaos Engineering (Chaos Engineering Book) Gremlin Slack GitHub Repository for Azure Chaos Studio Examples Recommended Learning and Training: Microsoft Learn Modules on Azure Chaos Studio Gremlin Certification Blogs and Articles: Azure Blog — Chaos Engineering These resources provide comprehensive knowledge, best practices, practical tutorials, and community insights, empowering you to effectively adopt and leverage Chaos Engineering practices in your cloud-native environments\nConclusion: Chaos Engineering isn\u0026rsquo;t just for the big players anymore; it’s something that everyone using cloud services should consider. With tools like Azure Chaos Studio and Gremlin, Microsoft Azure gives you a solid way to set up, control, and fine-tune your chaos experiments at any scale. By bringing these tools into your Azure Kubernetes Service (AKS) setup, you can spot and fix vulnerabilities ahead of time, check if your architecture choices hold up, and keep boosting your system’s resilience. The combination of Azure\u0026rsquo;s tools and third-party options enables teams to address a wide range of issues, from basic pod failures to complex network slowdowns and resource shortages. As your crew dives into chaos engineering, keep in mind that it’s not just about running experiments—it\u0026rsquo;s about learning from them. Every time you test things out, you have an opportunity to refine your systems and processes, making them more solid, secure, and efficient. Start small, keep a close eye on everything, automate wisely, and adhere to the Azure Well-Architected Framework. With these groovy principles, you can dive into the chaos with confidence, knowing it’ll help build more stable systems over time. Now\u0026rsquo;s the time to inject some chaos into your DevOps and CloudOps workflow. Check out Azure Chaos Studio. Give Gremlin a shot. Break stuff—on purpose—and get ready to create better systems because of it!\n","permalink":"https://wolkwacht.nl/posts/harnessing-chaos-implementing-chaos-engineering-with-azure-chaos-studio-and-gremlin-on-aks/","summary":"\u003ch2 id=\"harnessing-chaos-implementing-chaos-engineering-with-azure-chaos-studio-and-gremlin-onaks\"\u003e\u003cstrong\u003eHarnessing Chaos: Implementing Chaos Engineering with Azure Chaos Studio and Gremlin on AKS\u003c/strong\u003e\u003c/h2\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*BvBXMZRFGfRJPf2RkujuAw.png\"\u003e\u003c/p\u003e\n\u003cp\u003eChaos Engineering is a crucial practice for modern cloud operations, enabling teams to identify hidden weaknesses and potential failures in complex systems before they become a problem. By deliberately causing failures in a controlled setting, you can observe how your system responds, measure its resilience, and ultimately strengthen its robustness. This practice really stands out in cloud-native environments like Azure Kubernetes Service (AKS), where the details and interconnections can sometimes obscure vulnerabilities.\u003cbr\u003e\nAzure Chaos Studio is a set of cloud-native tools designed explicitly for Azure environments, seamlessly integrating, scaling, and providing insights directly within the Azure ecosystem. When paired with Gremlin — a widely used chaos engineering platform — organizations can craft thorough chaos engineering strategies that enhance system reliability, security, and operational performance.\u003cbr\u003e\nThis blog will discuss the architecture, practical setup steps, scripts, and validation techniques, all closely aligned with the principles outlined in Azure’s Well-Architected Framework. Through real examples and tested scripts, you’ll discover how to leverage the power of controlled chaos to build resilient and dependable cloud solutions.\u003c/p\u003e","title":"Harnessing Chaos: Implementing Chaos Engineering with Azure Chaos Studio and Gremlin on AKS"},{"content":"Azure Kubernetes Chronicles part 5: Autoscaling with KEDA Autoscaling is a transformative approach for modern cloud-native application architectures, particularly within Kubernetes and microservices environments. As the adoption of these technologies accelerates, implementing automated workload adjustments based on real-time demand becomes imperative. This capability enhances user experience and ensures continuous high availability and cost efficiency. Consider the diverse applications: whether deploying a customer-facing web application, processing extensive data sets, or orchestrating backend task queues, the proficiency in automating workload scaling is essential for operational success and resource optimization.\nKubernetes provides robust auto-scaling capabilities through tools such as the Horizontal Pod Autoscaler (HPA), Vertical Pod Autoscaler (VPA), and Cluster Autoscaler. These tools function similarly to a personal assistant for managing workloads, dynamically adjusting resource allocation based on crucial metrics like CPU utilization, memory consumption, and other performance indicators.\nHowever, in modern applications — particularly those that operate asynchronously, are event-driven, or utilize messaging protocols — traditional resource metrics may not sufficiently capture the nuances of workload requirements.\nThis article will delve into strategies to effectively leverage Kubernetes’ auto-scaling features to optimize application performance and resource efficiency in contemporary environments.\nWhy Autoscaling Matters In our fast-paced digital world, where smooth and instant experiences have become part of everyday life, autoscaling is your secret weapon for keeping applications running smoothly while being kind to your budget. Whether preparing for the exciting rush of Black Friday or enjoying some peace during quieter times, your applications need to adapt in real-time to meet these ever-changing demands. We have to ensure your tech is always ready for whatever comes next!\nAutoscaling helps achieve three essential goals in modern cloud-native environments:\nPerformance and Availability\nUsers expect fast response times, minimal latency, and consistent uptime. Applications that cannot scale to meet user demand risk degraded performance, application crashes, or even outages. Autoscaling ensures that the necessary resources are available when needed, enhancing the user experience and ensuring service level objectives (SLOs) are met.\nResource Efficiency and Cost Optimization\nCloud resources are not free. Autoscaling helps strike a balance between resource allocation and cost by dynamically adjusting compute power to match actual usage. This means you avoid overprovisioning during low-traffic periods and underprovisioning when demand spikes.\nThis can result in substantial cost savings for organizations running large-scale distributed systems or multiple microservices, especially when leveraging features like KEDA’s scale-to-zero capabilities, which completely shut down idle workloads.\nOperational Simplicity and Automation\nManual scaling is inefficient, error-prone, and doesn’t scale with the complexity of modern applications. Autoscaling enables you to automate resource provisioning, reduce operational overhead, and free up engineering time to focus on delivering business value rather than managing infrastructure.\nFurthermore, autoscaling aligns perfectly with GitOps and Infrastructure as Code (IaC) principles, making it easier to codify, version, and track infrastructure changes across environments.\nUse Case Scenarios Where Autoscaling is Critical: E-commerce platforms scaling during flash sales and high-traffic campaigns. SaaS applications responding to dynamic user interactions. Data processing systems handling batch or stream-based workloads. IoT platforms ingest unpredictable volumes of telemetry data. Event-driven microservices processing jobs from queues, topics, or HTTP triggers. Native Kubernetes Autoscaling Techniques Kubernetes offers built-in autoscaling features integral to managing resources in a dynamic environment. These mechanisms include the Horizontal Pod Autoscaler (HPA), Vertical Pod Autoscaler (VPA), and Cluster Autoscaler. Each serves a distinct purpose and targets different scaling dimensions.\nHorizontal Pod Autoscaler (HPA) HPA automatically adjusts the number of pods in a deployment, replica set, or stateful set based on observed CPU utilization (or other select metrics like memory or custom metrics).\nHow it works: HPA uses metrics collected from the Metrics Server (or Prometheus adapter) to make scaling decisions. For example, if the average CPU usage exceeds a defined threshold, HPA increases the number of pods. Example manifest: apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: webapp-hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: webapp minReplicas: 2 maxReplicas: 10 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70 Limitations: HPA is resource-centric. It doesn’t handle event-driven workloads well (e.g., scaling based on queue length or incoming events). Vertical Pod Autoscaler (VPA) VPA automatically adjusts CPU and memory requests/limits for containers within pods to optimize resource usage. It’s ideal for workloads that cannot be scaled horizontally or where resource usage patterns vary significantly.\nModes of Operation:\nAuto: VPA can update pod resources and restart them automatically. Initial: Sets recommendations only when a pod is first created. Off: VPA monitors usage and provides recommendations, but doesn’t enforce changes. Example VPA Manifest: apiVersion: autoscaling.k8s.io/v1 kind: VerticalPodAutoscaler metadata: name: backend-vpa spec: targetRef: apiVersion: apps/v1 kind: Deployment name: backend updatePolicy: updateMode: Auto Limitations: VPA restarts pods when applying new resource values, which may not be suitable for high-availability workloads unless carefully orchestrated. Cluster Autoscaler Cluster Autoscaler operates at the infrastructure level and adjusts the number of nodes in a cluster. It scales out when pods are unschedulable due to resource constraints and scales in when nodes are underutilized.\nKey Features:\nWorks with AKS, GKE, and EKS. Scales only node pools with autoscaling enabled. Example (AKS CLI): az aks nodepool update \\ --resource-group myResourceGroup \\ --cluster-name myAKSCluster \\ --name nodepool1 \\ --enable-cluster-autoscaler \\ --min-count 1 \\ --max-count 5 Limitations: Cluster autoscaling can take several minutes and may not react quickly to sudden workload spikes. Depending on your workload characteristics, these autoscaling mechanisms can be used independently or in combination. However, none inherently support autoscaling based on external event sourceslike queues or message buses, where KEDA adds its unique value.\nThe Limitations of Native Autoscaling While Kubernetes’ native autoscaling mechanisms are powerful and essential for many workloads, they have inherent limitations that reduce their effectiveness in modern, event-driven, and microservice-heavy environments. Let’s explore some of the most significant gaps and constraints.\nResource-Centric Metrics Only\nHPA and VPA rely heavily on CPU and memory utilization as primary indicators for scaling. While this is useful for compute-bound applications, it doesn’t capture real application pressure in event-driven systems.\nExample: A message queue-backed service may experience a massive influx of messages. However, if the current pods are underutilized (CPU-wise), HPA won’t scale the workload, even though the backlog is growing.\nThis creates a bottleneck where messages accumulate and latency increases, leading to delayed processing and a poor user experience.\nLack of Event Awareness\nNone of the native autoscalers in Kubernetes can natively respond to external signals like queue length, HTTP request volume, database entries, or cloud events.\nThis severely limits their use in event-driven architectures where these external signals represent the application load. Without this context, scaling decisions are essentially blind to the system’s actual needs.\nSlow Reaction Times\nCluster Autoscaler can take several minutes to provision new nodes. While it’s excellent for optimizing node pool sizes and preventing resource waste, it’s too slow for latency-sensitive applications that require rapid response to surges in demand.\nSimilarly, HPA’s default polling interval (30 seconds) and gradual scale-up strategy may not be sufficient for workloads that spike quickly and require immediate action.\nNo Support for Scale-to-Zero\nA critical feature for cost efficiency in asynchronous workloads is the ability to scale to zero when there is no work to do. Native autoscaling does not support this concept — HPA requires a minimum of one replica, and Cluster Autoscaler will not remove the last node if it would render the cluster unschedulable.\nThis represents a significant inefficiency for workloads that remain idle most of the time but must respond quickly when triggered.\nComplex Configuration for Custom Metrics\nWhile HPA supports custom metrics through adapters such as the Prometheus Adapter, setting this up can be complex and prone to errors. It also necessitates maintaining a separate monitoring and metrics collection infrastructure for scaling decisions.\nThis overhead often becomes a barrier for teams that need quick, flexible autoscaling capabilities.\nWorkload Compatibility Gaps\nSome workloads do not scale well using traditional resource-based indicators:\nJobs and batch processes Queue consumers Stateful services These workloads require scaling decisions based on external conditions (like queue depth or job count) rather than internal pod metrics. Native Kubernetes autoscalers are not optimized for such patterns.\nInconsistent Behavior Across Cloud Providers\nWhile Kubernetes is cloud-agnostic, autoscaling behavior can vary depending on how a managed Kubernetes service (like AKS, EKS, GKE) implements metrics collection, cluster scaling policies, and integration with cloud-native services.\nThis inconsistency complicates hybrid or multi-cloud strategies where uniform autoscaling behavior is required.\nThese limitations underscore the need for a more extensible and event-aware autoscaling system that integrates with external services and supports advanced scaling patterns. This is precisely where KEDA (Kubernetes Event-driven Autoscaling) steps in.\nIntroducing KEDA: Kubernetes Event-driven Autoscaling Kubernetes Event-driven Autoscaling (KEDA) is a lightweight, open-source component that brings event-based autoscaling to Kubernetes. It allows applications to scale dynamically based on the number of events needing to be processed, whether those events are messages in a queue, rows in a database, or custom metrics from a monitoring system.\nOriginally developed by Microsoft and Red Hat, KEDA has since evolved into a robust and widely adopted project under the Cloud Native Computing Foundation (CNCF).\nKEDA effectively bridges the gap between external event sources and Kubernetes’ native autoscaling framework (HPA). It does this by exposing custom metrics to Kubernetes and, when necessary, automatically launching or shutting down workloads, including scale-to-zero scenarios, which native Kubernetes cannot do independently.\nWhat is KEDA? KEDA is a Kubernetes-based event-driven autoscaler that enables fine-grained, real-time autoscaling for container workloads. Unlike the default HPA, which scales pods based on resource metrics, KEDA enables workloads to scale based on external data sources like message queues, event streams, HTTP request rates, etc.\nKEDA does this in two primary ways:\nMetrics Adapter: KEDA is a metrics provider that feeds custom metrics into Kubernetes’ HPA via the Metrics API. HPA uses these metrics to make scaling decisions. Activation Controller: KEDA can activate and deactivate Kubernetes Deployments, including scaling them to zero and back again when event thresholds are crossed. KEDA Architecture At a high level, KEDA consists of the following components:\nOperator: Watches for custom KEDA resources (ScaledObjects, ScaledJobs) and manages scaling behavior. Metrics Adapter: Exposes custom metrics to the Kubernetes Metrics API for use by HPA. Trigger Scalers: Interfaces that define how to pull metric data from external sources. These include built-in scalers for Azure Service Bus, RabbitMQ, Kafka, Prometheus, AWS SQS, and more. Architecture Diagram\nCore Concepts\nScaledObject: A Kubernetes custom resource that defines how to scale a particular Deployment based on an event source. Each ScaledObject includes:\nThe target deployment to scale. The scaling trigger type (e.g., Azure Service Bus, Kafka). Trigger metadata such as connection strings, queue names, and thresholds. ScaledJob: A special resource designed for one-time jobs or batch processing. KEDA launches short-lived pods that perform work based on events.\nTriggers: The core mechanism KEDA uses to evaluate when and how to scale. They are defined by source type and contain metadata such as polling interval, threshold, and authentication information\nExample ScaledObject YAML:\napiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: azure-queue-scaler spec: scaleTargetRef: name: queue-processor minReplicaCount: 0 maxReplicaCount: 10 triggers: - type: azure-queue metadata: queueName: myqueue connectionFromEnv: AzureWebJobsStorage queueLength: \u0026#34;5\u0026#34; Supported Event Sources KEDA supports over 50 scalers out-of-the-box. Popular scalers include:\nAzure Service Bus Azure Storage Queues Azure Event Hubs Kafka RabbitMQ AWS SQS and CloudWatch PostgreSQL and MySQL Prometheus queries Redis Streams Cron (time-based triggers) For a complete list, refer to: https://keda.sh/docs/\nBenefits of Using KEDA on Azure Kubernetes Service (AKS) KEDA is such a fantastic extension for Kubernetes, and when combined with Azure Kubernetes Service (AKS), it truly shines! Microsoft Azure offers a fantastic managed experience for KEDA, simplifying the processes of integration, operation, and scaling of production workloads. Below are the key benefits of using KEDA specifically on AKS:\nFirst-Party AKS Integration and Managed Add-on Support\nKEDA is available as a native add-on for AKS, which dramatically simplifies the installation and lifecycle management process. You can enable KEDA directly via the Azure CLI, ARM templates, or Bicep.\naz aks enable-addons \\ - addons keda \\ - name myAKSCluster \\ - resource-group myResourceGroup Azure manages the underlying KEDA components, including the operator and metrics server, providing better stability, supportability, and seamless integration with Azure RBAC and identity management.\nAzure-Native Event Source Support\nKEDA offers deep integration with Azure services out of the box, including:\nAzure Storage Queues Azure Service Bus (Queues and Topics) Azure Event Hubs Azure Monitor (via custom metrics) Azure Blob Trigger (via Event Grid) This allows AKS workloads to respond natively to Azure ecosystem signals, without requiring additional wrappers, shims, or bridge services.\nSeamless Identity and Secret Management\nKEDA on AKS can leverage:\nAzure AD Pod Identity (for secure access to services without connection strings) Azure Key Vault Provider for Secrets Store CSI Driver (for securely injecting credentials) Managed Identities (for tightly scoped and rotated permissions) These integrations help reduce the risk of credential leaks, simplify compliance with enterprise-grade security policies, and support the implementation of zero trust.\nScale-to-Zero for Cost Efficiency\nWith KEDA, AKS workloads can scale down to zero replicas when idle. This is especially powerful for workloads that only run occasionally or during business hours.\nExample use cases:\nLine-of-business apps are active only during working hours. Batch processing jobs are triggered via queue messages. Seasonal workloads with unpredictable traffic patterns. When combined with Azure Spot VMs and cluster autoscaler, KEDA enables highly cost-efficient architectures.\nAdvanced Telemetry and Monitoring Support\nKEDA automatically integrates with Azure Monitor by running on AKS and can be further enhanced with Prometheus and Grafana setups. You can visualize metrics such as:\nNumber of active messages in a queue Replica count trends Time spent at peak vs idle These insights help fine-tune your scaling policies and provide visibility into autoscaling performance.\nSimplified Developer and Ops Experience\nWith Azure-native tooling (Azure CLI, Bicep, Azure Monitor, AKS diagnostics), engineers can easily provision and manage KEDA. It fits well into both GitOps and IaC strategies, enabling teams to:\nAutomate deployments Version and review scaling configs Audit autoscaling activity Microsoft Support and Enterprise Compliance\nKEDA is officially supported by Microsoft as part of the AKS ecosystem, which means you can escalate issues, file support tickets, and receive help under your enterprise support agreements.\nFurthermore, AKS clusters running KEDA can be integrated with:\nAzure Policy for compliance enforcement Azure Defender for threat protection Azure Arc for hybrid observability These benefits make KEDA on AKS a compelling choice for modern applications that need fast, flexible, and secure autoscaling capabilities across various event sources.\nDeploying KEDA on AKS There are two primary ways to deploy KEDA on Azure Kubernetes Service (AKS):\nUsing the built-in AKS add-on is the recommended and most straightforward approach. Using Helm for custom scenarios where greater configuration control is needed. Both options offer a seamless integration into your AKS cluster. Let’s explore each approach.\nPrerequisites\nBefore you begin, ensure the following:\nYou have an existing AKS cluster running Kubernetes 1.20 or higher. You have the Azure CLI and kubectl installed and configured. You have Contributor or higher access to the AKS cluster’s resource group. Verify Azure CLI version:\naz version Check kubectl context:\nkubectl config get-contexts Option 1: Enabling the KEDA Add-on (Recommended)\nThe simplest and most supported way to deploy KEDA on AKS is using the built-in AKS add-on:\naz aks enable-addons \\ - addons keda \\ - name \u0026lt;AKS_CLUSTER_NAME\u0026gt; \\ - resource-group \u0026lt;RESOURCE_GROUP\u0026gt; Note: Replace \u0026lt;AKS\\_CLUSTER\u0026gt; and \u0026lt;RESOURCE\\_GROUP\u0026gt; with the actual values.\nThis will:\nDeploy the KEDA operator and metrics server to the kube-system namespace. Configure appropriate permissions via Azure RBAC. Enable automatic updates and patching via Azure. You can verify the KEDA components are running using:\nkubectl get pods -n kube-system | grep keda You should see pods like keda-operator and keda-metrics-apiserver in a Running state.\nOption 2: Installing KEDA Manually with Helm\nHelm allows you to customize your KEDA installation (e.g., custom namespaces, scaling intervals, Prometheus integration) if you need more flexibility.\nStep 1: Add the KEDA Helm repository\nhelm repo add kedacore https://kedacore.github.io/charts helm repo update Step 2: Create a dedicated namespace (optional but recommended)\nkubectl create namespace keda Step 3: Install KEDA with Helm\nhelm install keda kedacore/keda \\ - namespace keda \\ - set prometheus.metricServer.enabled=true Step 4: Confirm installation\nkubectl get all -n keda This should list the KEDA operator deployment, service, and metrics server.\nPost-Deployment Checks\nRegardless of your installation method:\nEnsure your Metrics Server is functioning:\nkubectl top pods Confirm the CRDs (Custom Resource Definitions) for ScaledObject and ScaledJob are installed:\nkubectl get crds | grep keda Expected output:\nscaledobjects.keda.sh scaledjobs.keda.sh triggerauthentications.keda.sh With KEDA deployed and operational, your AKS cluster can now scale workloads dynamically based on external event sources.\nDeploying an Event-Driven Application with KEDA Now that KEDA is installed and operational, let’s walk through a hands-on example. In this section, we’ll deploy an event-driven application that uses Azure Storage Queues and scales dynamically using KEDA’s ScaledObject. mechanism.\nThis example demonstrates how to:\nCreate an Azure Storage Queue Deploy a queue processor in AKS. Configure a KEDA ScaledObject to autoscale based on queue length. Step 1: Set Up Azure Resources\nEnsure the Azure CLI is logged in and targeting the correct subscription and region:\naz login az account set --subscription \u0026#34;YourSubscriptionName\u0026#34; Create a resource group:\naz group create --name demo-keda-rg --location westeurope Create a storage account:\naz storage account create \\ --name demokedastorageacct \\ --resource-group demo-keda-rg \\ --location westeurope \\ --sku Standard_LRS Retrieve the storage account connection string:\naz storage account show-connection-string \\ --name demokedastorageacct \\ --resource-group demo-keda-rg \\ --query connectionString --output tsv Create a queue:\naz storage queue create \\ --name ordersqueue \\ --account-name demokedastorageacct Step 2: Create Kubernetes Secrets for the Queue Connection\nStore the Azure Storage connection string securely in AKS:\nkubectl create secret generic azure-queue-secret \\ --from-literal=AzureWebJobsStorage=\u0026#34;\u0026lt;your-connection-string\u0026gt;\u0026#34; Replace \u0026lt;your-connection-string\u0026gt; with the value from the previous step.\nStep 3: Deploy the Queue Processor Application\nHere is a sample Kubernetes deployment manifest:\napiVersion: apps/v1 kind: Deployment metadata: name: queue-processor spec: replicas: 1 selector: matchLabels: app: queue-processor template: metadata: labels: app: queue-processor spec: containers: - name: queue-processor image: myregistry.azurecr.io/queue-processor:latest env: - name: AzureWebJobsStorage valueFrom: secretKeyRef: name: azure-queue-secret key: AzureWebJobsStorage Apply the deployment:\nkubectl apply -f queue-processor-deployment.yaml Ensure the pod is running:\nkubectl get pods -l app=queue-processor Step 4: Define a KEDA ScaledObject\nHere’s a basic ScaledObject manifest that uses the Azure Storage Queue scaler:\napiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: queue-scaler spec: scaleTargetRef: name: queue-processor minReplicaCount: 0 maxReplicaCount: 10 triggers: - type: azure-queue metadata: queueName: ordersqueue connectionFromEnv: AzureWebJobsStorage queueLength: \u0026#34;5\u0026#34; Apply the ScaledObject:\nkubectl apply -f queue-scaler.yaml Verify the ScaledObject:\nkubectl get scaledobject Step 5: Generate Load\nYou can enqueue messages to the storage queue manually to trigger scaling:\nfor i in {1..20}; do az storage message put \\ --account-name demokedastorageacct \\ --queue-name ordersqueue \\ --content \u0026#34;Message $i\u0026#34; echo \u0026#34;Message $i enqueued\u0026#34; done Check the number of replicas:\nkubectl get deployment queue-processor Within moments, KEDA will detect the backlog and scale the queue-processor deployment accordingly.\nMonitoring and Observability with KEDA KEDA integrates with Kubernetes’ metrics pipeline, making monitoring autoscaling activity straightforward through standard observability tools like the Kubernetes Metrics Server, Prometheus, and Grafana.\nMetrics Server Integration\nThe Kubernetes Metrics Server collects resource usage metrics from pods and nodes. While KEDA provides custom metrics, the Metrics Server is still essential for HPA to make decisions.\nTo verify the Metrics Server is running:\nkubectl get deployment metrics-server -n kube-system Check live metrics:\nkubectl top pods If this returns metrics, the Metrics Server is operational. If not, you may need to install or troubleshoot it.\nEnabling KEDA Metrics for Prometheus\nIf you’ve installed KEDA via Helm with Prometheus metrics enabled:\nhelm upgrade --install keda kedacore/keda \\ --namespace keda \\ --set prometheus.metricServer.enabled=true This exposes a /metrics endpoint on the keda-metrics-apiserver that Prometheus can scrape.\nTo expose metrics, ensure a ServiceMonitor is defined (if you are using the Prometheus Operator):\napiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: keda-servicemonitor labels: release: prometheus spec: selector: matchLabels: app: keda-operator namespaceSelector: matchNames: - keda endpoints: - port: http interval: 30s Ensure the KEDA metrics API server is reachable:\nkubectl get svc -n keda Viewing Metrics in Grafana\nOnce Prometheus is scraping metrics from KEDA, you can build Grafana dashboards using metrics like:\nkeda_scaledobject_scaler_queueLength keda_scaledobject_scaled_replicas keda_scaledobject_active You can import a prebuilt dashboard or create your own using Prometheus queries to correlate queue depth with pod replicas over time.\nAzure Monitor Integration\nIf using the AKS-managed KEDA add-on, Azure Monitor will automatically collect some metrics. To visualize them:\nGo to Azure Portal \u0026gt; AKS Cluster \u0026gt; Insights \u0026gt; Workloads. Use Log Analytics queries for deeper inspection: KubePodInventory | where ContainerName == \u0026#34;keda-operator\u0026#34; This allows you to trace KEDA activity and correlate it with application logs.\nMonitoring is critical for understanding the behavior of your autoscalers and optimizing for performance and cost.\nTroubleshooting and Debugging KEDA Troubleshooting KEDA involves inspecting logs, validating ScaledObject definitions, and checking event source connectivity. Because KEDA spans multiple components — operator, metrics server, trigger scalers — debugging often involves Kubernetes-level diagnostics and external system checks.\nCheck the KEDA Operator Logs\nThe operator is the control plane component that manages autoscaling. Inspect logs to see if scaling decisions are being made:\nkubectl logs -l app=keda-operator -n keda Look for lines like:\nSuccessfully updated deployment ... from 1 to 3 replicas Verify ScaledObject Configuration\nEnsure the ScaledObject is applied correctly and recognized:\nkubectl get scaledobjects.keda.sh kubectl describe scaledobject \u0026lt;name\u0026gt; Validate the scaling trigger and metadata — incorrect queueName, missing credentials, or misconfigured pollingIntervalare common issues.\nCheck Metrics Server Availability\nThe metrics server must be available and returning data. Run:\nkubectl top pods If you receive an error, ensure the metrics server is installed and running:\nkubectl get deployment metrics-server -n kube-system Confirm Trigger Activity\nFor event-driven triggers (like Azure Queues), validate that messages exist in the queue and the queue name matches the configuration.\nIf the queueLength remains 0, scaling won\u0026rsquo;t occur.\nYou can use KEDA logs to trace trigger execution:\nkubectl logs -l app=keda-operator -n keda | grep Trigger Ensure Permissions Are Set Correctly\nMissing RBAC roles or incorrect Azure identity configuration can prevent KEDA from authenticating with event sources. Use:\nkubectl describe serviceaccount keda-operator -n keda kubectl get clusterrolebinding | grep keda Monitor Metrics API Server\nIf Prometheus scraping is enabled and the KEDA Metrics API server is exposed, ensure it’s operational:\nkubectl get pods -n keda | grep keda-metrics-apiserver kubectl port-forward svc/keda-metrics-apiserver 9022:9022 -n keda curl http://localhost:9022/metrics This endpoint should return a list of KEDA metrics.\nThorough diagnostics and logs are critical when working with KEDA.\nConclusion: Scaling Smarter with KEDA on AKS Autoscaling has become a must-have rather than just a nice feature, especially for creating resilient, responsive, and cost-effective cloud-native applications. Although Kubernetes offers robust built-in autoscalers like HPA, VPA, and the Cluster Autoscaler, they sometimes don’t quite meet the demands of today’s event-driven workloads, where external signals play a crucial role in determining scaling needs. That’s where KEDA steps in as a game-changer. By enabling autoscaling based on real-world events like queue length, HTTP requests, or database state, KEDA transforms your AKS workloads into intelligent, reactive systems that can scale dynamically — and even scale to zero when idle.\nThanks to its native integration with Azure Kubernetes Service, robust support for Azure event sources, and seamless compatibility with popular enterprise tools like Prometheus, Azure Monitor, and GitOps workflows, KEDA makes event-driven scaling powerful and effortlessly manageable.\nWhether you’re navigating fluctuating traffic, managing asynchronous tasks, or developing agile microservices, KEDA equips your team to scale workloads effectively, ensuring precision, efficiency, and responsiveness. Now it’s your turn: dive into hands-on deployment, observe the system’s performance, and start crafting customized autoscaling strategies designed specifically for your workload’s requirements. Happy scaling!\n","permalink":"https://wolkwacht.nl/posts/azure-kubernetes-chronicles-part-5/","summary":"\u003ch2 id=\"azure-kubernetes-chronicles-part-5-autoscaling-withkeda\"\u003eAzure Kubernetes Chronicles part 5: Autoscaling with KEDA\u003c/h2\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*xPGw1bvlmgmhhc6YGI_uMw.jpeg\"\u003e\u003c/p\u003e\n\u003cp\u003eAutoscaling is a transformative approach for modern cloud-native application architectures, particularly within Kubernetes and microservices environments. As the adoption of these technologies accelerates, implementing automated workload adjustments based on real-time demand becomes imperative. This capability enhances user experience and ensures continuous high availability and cost efficiency. Consider the diverse applications: whether deploying a customer-facing web application, processing extensive data sets, or orchestrating backend task queues, the proficiency in automating workload scaling is essential for operational success and resource optimization.\u003c/p\u003e","title":"Azure Kubernetes Chronicles part 5:"},{"content":"\nAs a cloud professional, I constantly seek ways to streamline application delivery while reducing operational overhead. Whether you’re building APIs, microservices, or event-driven backends, choosing the right platform can significantly impact speed to market and maintainability.\nIntroducing Azure Container Apps (ACA) — a fully managed, serverless platform designed specifically for running containerized workloads without the complexity of orchestrating and managing Kubernetes infrastructure. Built on top of Kubernetes and powered by open-source technologies like Dapr and KEDA (Kubernetes Event Driven Autoscaling), ACA offers the best of both worlds: the power and flexibility of containers with the simplicity and developer focus of a Platform-as-a-Service (PaaS).\nI’ll walk you through the ACA experience from code to cloud in this blog. We’ll cover everything from setting up your environment and deploying container images to managing revisions, securing access with VNets and Private Link, and implementing advanced monitoring with Azure Monitor and Application Insights. We’ll even explore modern deployment strategies like Blue/Green and dive into distributed tracing with Dapr and OpenTelemetry.\nWhether you’re coming from an Azure Kubernetes Services (AKS) background or exploring ACA for greenfield projects, this guide will give you hands-on insight into how ACA fits into your cloud architecture toolbox.\nWhat is Azure Container Apps? Azure Container Apps is a fully managed serverless platform that enables developers to run containerized applications without managing the underlying infrastructure. It is designed for microservices, event-driven applications, and background tasks, allowing you to focus on writing code rather than handling orchestration details. Built on Kubernetes, ACA leverages open-source projects like Dapr for microservices and KEDA for event-driven scaling, making it a versatile solution for cloud-native app development.\nKey Features Serverless Scaling: Automatically scales applications based on HTTP traffic, events, or custom metrics. Supports scaling down to zero when no activity is detected, reducing costs. Built-in Ingress and Load Balancing: Provides out-of-the-box support for HTTPS ingress and load balancing across app revisions. Environment Scoping: Container Apps are deployed within an environment that can span virtual networks, providing network isolation and scalability. Integrated Open-Source Technologies: Supports Dapr for building microservices, KEDA for autoscaling, and Envoy for ingress. Support for CI/CD Pipelines: Easily integrates with GitHub Actions or Azure DevOps for streamlined build and deployment workflows. Built-In Secret Management: Use Azure Key Vault or environment variables to manage application secrets securely. Azure Container Apps architecture\nHow It Works Deployment: You deploy container images (from ACR, Docker Hub, etc.) to ACA via Azure CLI, ARM/Bicep, Terraform, or GitHub Actions. Runtime: ACA runs your container in a managed environment. You don’t manage any VMs, nodes, or Kubernetes resources. Scaling: KEDA automatically scales your app in and out based on workload. It supports scale to zero and back to n instances. Ingress: Envoy provides HTTPS ingress and routes traffic to the correct revision of your app (ACA uses a revision-based deployment model). Networking: You can enable internal-only access, integrate with VNets, or expose the app publicly. Observability: Logs and metrics are collected via Azure Monitor and Application Insights (optional). Azure Container Apps vs Azure Kubernetes Service: Which One to Choose? When building containerized applications on Azure, two popular options are Azure Container Apps (ACA) and Azure Kubernetes Service (AKS). Both support modern cloud-native workloads, but they differ significantly in complexity, control, and use cases.\nAbstraction vs Control\nACA is a fully managed, serverless container platform built on Kubernetes, but it abstracts away the Kubernetes layer entirely. You don’t need to manage clusters, pods, or nodes. You define your app, environment, and scaling rules, and Azure handles the rest.\nIn contrast, AKS is a managed Kubernetes service that gives you full control over your cluster and workloads. You manage node pools, configure deployments, manage Helm charts, and operate like a traditional Kubernetes cluster.\nScaling and Operations\nACA uses KEDA natively to autoscale apps based on HTTP traffic, queue length, CPU/memory usage, or custom events. It can scale to zero, making it ideal for bursty or event-driven workloads.\nAKS supports KEDA and HPA (Horizontal Pod Autoscaler), but you’re responsible for configuring and maintaining the scaling rules and cluster autoscaling.\nUse Cases\nUse Azure Container Apps if you want:\nSimplicity with containerized apps Event-driven microservices Serverless scale-to-zero capability Focus on code, not infrastructure Use Azure Kubernetes Service if you need:\nFine-grained control over Kubernetes resources Complex service meshes, operators, or custom CRDs Stateful workloads with persistent volumes Multi-tenant architectures or service meshes (Istio, Linkerd) Getting Started with Azure Container Apps Let’s walk through deploying a containerized application using ACA. We’ll start with a basic Hello World app and later expand with ACR (Azure Container Registry), scaling rules, VNet integration, and monitoring.\nPrerequisites An active Azure subscription Azure CLI installed (*az — version*) Docker installed (*docker — version*) GitHub account (for optional CI/CD integration) Walkthrough: Building, Pushing, and Deploying a Custom Image with ACR Create a Resource Group\naz group create --name my-container-apps --location westeurope This grouping logically organizes your Azure resources and allows you to manage them collectively. Choose a region close to your users for lower latency.\nCreate an Azure Container Registry (ACR)\naz acr create \\ --name myAcrRegistry \\ --resource-group my-container-apps \\ --sku Basic \\ --admin-enabled true ACR is a private container registry used for storing your Docker images. Admin-enabled allows for username/password authentication, which is necessary in this example.\nLog in to ACR\naz acr login --name myAcrRegistry This authenticates your Docker CLI with the ACR instance, so you can push images to it.\nBuild and Push the Docker Image\ndocker build -t myacrregistry.azurecr.io/hello-world:v1 . docker push myacrregistry.azurecr.io/hello-world:v1 Ensure your image name matches the ACR login server format. To confirm, use:\naz acr show --name myAcrRegistry --query loginServer -o tsv Create a Container App Environment\naz containerapp env create \\ --name my-aca-env \\ --resource-group my-container-apps \\ --location westeurope This creates a Container App Environment, which acts as a logical boundary or isolation layer for your Azure Container Apps.\nA logical boundary is all about keeping resources in the same environment organized with common settings and infrastructure, like networking, security policies, logging, and metrics. For instance, the Container App Environment gives you a simple way to run and manage multiple Container Apps together, making handling their operations and configurations easier. When you set up a Container App Environment, you group related Container Apps in a way that simplifies the management of shared resources. This helps keep things tidy by clearly separating different app groups or stages, like production vs. development.\nDeploy the App Using ACR Image\nThe following Azure CLI command deploys an Azure Container App using a container image stored in an Azure Container Registry (ACR):\naz containerapp create \\ --name hello-acr-app \\ --resource-group my-container-apps \\ --environment my-aca-env \\ --image myacrregistry.azurecr.io/hello-world:v1 \\ --target-port 80 \\ --ingress external \\ --registry-server myacrregistry.azurecr.io \\ --registry-username $(az acr credential show --name myAcrRegistry --query username -o tsv) \\ --registry-password $(az acr credential show --name myAcrRegistry --query passwords[0].value -o tsv) Explanation of Command Flags: — name hello-acr-app: This flag specifies the unique name for your Container App within the resource group. Choose a clear, descriptive identifier that reflects the app’s purpose.\n— resource-group my-container-apps: Defines the Azure Resource Group where your Container App is created. A resource group logically groups Azure resources together, making it easier to manage, monitor, and control access permissions collectively.\n— environment my-aca-env: Associates the app with a previously created Container App Environment. The environment acts as a logical container that provides shared infrastructure and configuration such as networking (VNet integration), logging, monitoring, and security policies for multiple Container Apps.\n— image myacrregistry.azurecr.io/hello-world:v1: Specifies the container image source from Azure Container Registry (ACR). The provided format includes:\nRegistry URL (myacrregistry.azurecr.io): This is the fully-qualified URL of your private Azure Container Registry.\nImage name and tag (hello-world:v1): Indicates the container image (hello-world) and the version/tag (v1). Specifying a tag ensures you’re deploying a specific, known version of your containerized application.\n— target-port 80: This flag specifies the port number your container listens on internally. Azure Container Apps will forward incoming requests from the ingress to this port on your container. Ensure this matches the port your app configures to listen on within the container.\n— ingress external: Enables ingress to the Container App from the internet. Setting it as external configures a publicly accessible URL endpoint, allowing external clients or users to access your application over HTTP(S).\nOther possible values include:\ninternal: Only accessible within the environment or VNet. external: Publicly accessible on the internet. — registry-server myacrregistry.azurecr.io: Explicitly declares the registry server hosting your container images. This is the server URL for the Azure Container Registry.\n— registry-username and — registry-password: These flags provide credentials required by Azure Container Apps to securely authenticate and pull container images from a private Azure Container Registry (ACR).\nView App URL\naz containerapp show \\ --name hello-acr-app \\ --resource-group my-container-apps \\ --query properties.configuration.ingress.fqdn \\ --o tsv This gives you the public endpoint of your running containerized app. Visit it in your browser to test the deployment.\nFull ACR to ACA workflow\nManaging and Monitoring Azure Container Apps Environment and Revision Management Azure Container Apps (ACA) environments provide a structured way to host multiple containerized applications. Each ACA environment can include multiple container apps, with optional network isolation to enhance security. Whenever you change the configuration of an app — for example, updating environment variables, scaling rules, or container images — ACA automatically creates a new revision of the app. Each revision represents a specific, deployable snapshot of your application’s configuration and container state.\nTo manage revisions, ACA distinguishes between ‘latest’ and ‘stable’ revisions:\nLatest Revision: The most recent deployment or configuration change you’ve applied to the container app. It’s typically used for testing new features or updates in a controlled manner.\nStable Revision: A specific revision explicitly marked as stable, representing a reliable, verified state of your application. You typically route the majority of production traffic to this revision.\nACA supports advanced deployment strategies, such as canary deployments, by enabling traffic splitting between revisions. For instance, you can route traffic evenly (e.g., 50/50) between your latest revision and the stable revision to evaluate the performance and stability of recent changes before fully adopting them.\nYou can list all revisions of your app with:\naz containerapp revision list \\ -- name hello-acr-app \\ To split traffic equally between the latest and stable revisions, you can run:\naz containerapp update \\ --name hello-acr-app \\ --resource-group my-container-apps \\ --traffic-weight latest=50 stable=50 This approach allows controlled exposure of new features or configurations, reducing risk and ensuring seamless user experiences.\nMonitoring with Azure Monitor As your applications scale and evolve, observability becomes crucial for maintaining reliability and performance. Azure Container Apps integrates natively with Azure Monitor, Log Analytics, and Application Insights, providing deep insights into logs, metrics, revisions, scaling events, and errors — all without custom instrumentation.\nLet’s break down how to monitor and visualize ACA telemetry with Kusto Query Language (KQL) and Azure Dashboards.\nWalkthrough: Creating a Dashboard for ACA Monitoring\nOpen Azure Dashboard\nNavigate to the Azure Portal In the search bar, type “Dashboard” Click “New Dashboard” Give it a name like ACA Monitoring Pin a Log Query to the Dashboard\nOpen your Log Analytics Workspace (linked to ACA diagnostics). Go to Logs. Run the following sample KQL query: #kql ContainerAppConsoleLogs | where AppName == \u0026#34;hello-acr-app\u0026#34; | where Log_s contains \u0026#34;GET\u0026#34; | project TimeGenerated, AppName, RevisionName, Log_s | sort by TimeGenerated desc Click “Pin to Dashboard” (top right) → Select your dashboard.\nTip: You can filter for POST, errors, or even specific custom log statements based on Log_s.\nStep 3: Add Metrics to Dashboard\nYou can also pin standard metrics like CPU usage, memory usage, and requests per second:\nGo to your Container App resource Navigate to Metrics Select: Metric namespace: ContainerApp Metric: CpuUsageSeconds, MemoryWorkingSetBytes, or Requests Set aggregation (e.g., average, max) Click Pin to dashboard Repeat for different metrics to build a holistic view.\nStep 4: Layout the Dashboard\nOnce all tiles are pinned, go back to the dashboard Click “Edit” Resize and organize tiles logically (logs on one side, metrics on another) Save layout Advanced KQL Queries for ACA Logs\nOnce diagnostics are enabled, logs are streamed to your Log Analytics workspace under tables like:\nContainerAppConsoleLogs ContainerAppSystemLogs ContainerAppHttpLogs Here are some practical examples:\nQuery 1: View Most Recent Logs\n# kql ContainerAppConsoleLogs | where AppName == \u0026#34;hello-acr-app\u0026#34; | sort by TimeGenerated desc | limit 50 Query 2: Filter Logs for Errors or Warnings\n# kql ContainerAppConsoleLogs | where AppName == \u0026#34;hello-acr-app\u0026#34; | where Log_s has_any (\u0026#34;error\u0026#34;, \u0026#34;exception\u0026#34;, \u0026#34;fail\u0026#34;) | sort by TimeGenerated desc Query 3: Analyze Request Traffic by Revision\n# kql ContainerAppHttpLogs | where AppName == \u0026#34;hello-acr-app\u0026#34; | summarize count() by RevisionName, bin(TimeGenerated, 5m) | render timechart This is useful for spotting traffic spikes or comparing load across revisions.\nQuery 4: CPU and Memory Trends\n# kql ContainerAppSystemLogs | where AppName == \u0026#34;hello-acr-app\u0026#34; | summarize avg(CpuUsageSeconds), avg(MemoryWorkingSetBytes) by bin(TimeGenerated, 1m) | render timechart A sample Azure Dashboard JSON can be imported directly into the Azure Portal. It includes key visualizations for CPU usage, memory usage, request traffic, and a Kusto query log view for ACA logs.\n# Azure Dashboard JSON for ACA Monitoring { \u0026#34;properties\u0026#34;: { \u0026#34;lenses\u0026#34;: { \u0026#34;0\u0026#34;: { \u0026#34;order\u0026#34;: 0, \u0026#34;parts\u0026#34;: { \u0026#34;0\u0026#34;: { \u0026#34;position\u0026#34;: { \u0026#34;x\u0026#34;: 0, \u0026#34;y\u0026#34;: 0, \u0026#34;rowSpan\u0026#34;: 6, \u0026#34;colSpan\u0026#34;: 6 }, \u0026#34;metadata\u0026#34;: { \u0026#34;inputs\u0026#34;: [ { \u0026#34;name\u0026#34;: \u0026#34;resourceType\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;microsoft.app/containerapps\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;resourceName\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;\u0026lt;REPLACE-WITH-YOUR-APP-NAME\u0026gt;\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;metricNamespace\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;microsoft.app/containerapps\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;chartTitle\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;CPU Usage (sec)\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;metrics\u0026#34;, \u0026#34;value\u0026#34;: [ { \u0026#34;metricName\u0026#34;: \u0026#34;CpuUsageSeconds\u0026#34;, \u0026#34;aggregation\u0026#34;: \u0026#34;Average\u0026#34; } ] } ], \u0026#34;type\u0026#34;: \u0026#34;Extension/MetricsChartPart\u0026#34; } }, \u0026#34;1\u0026#34;: { \u0026#34;position\u0026#34;: { \u0026#34;x\u0026#34;: 6, \u0026#34;y\u0026#34;: 0, \u0026#34;rowSpan\u0026#34;: 6, \u0026#34;colSpan\u0026#34;: 6 }, \u0026#34;metadata\u0026#34;: { \u0026#34;inputs\u0026#34;: [ { \u0026#34;name\u0026#34;: \u0026#34;resourceType\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;microsoft.app/containerapps\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;resourceName\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;\u0026lt;REPLACE-WITH-YOUR-APP-NAME\u0026gt;\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;metricNamespace\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;microsoft.app/containerapps\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;chartTitle\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;Memory Usage (bytes)\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;metrics\u0026#34;, \u0026#34;value\u0026#34;: [ { \u0026#34;metricName\u0026#34;: \u0026#34;MemoryWorkingSetBytes\u0026#34;, \u0026#34;aggregation\u0026#34;: \u0026#34;Average\u0026#34; } ] } ], \u0026#34;type\u0026#34;: \u0026#34;Extension/MetricsChartPart\u0026#34; } }, \u0026#34;2\u0026#34;: { \u0026#34;position\u0026#34;: { \u0026#34;x\u0026#34;: 0, \u0026#34;y\u0026#34;: 6, \u0026#34;rowSpan\u0026#34;: 8, \u0026#34;colSpan\u0026#34;: 12 }, \u0026#34;metadata\u0026#34;: { \u0026#34;inputs\u0026#34;: [ { \u0026#34;name\u0026#34;: \u0026#34;query\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;ContainerAppHttpLogs\\n| where AppName == \\\u0026#34;hello-acr-app\\\u0026#34;\\n| summarize count() by bin(TimeGenerated, 5m)\\n| render timechart\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;version\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;KqlItem/1.0\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;queryType\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;Kusto\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;resourceType\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;microsoft.operationalinsights/workspaces\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;resourceId\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;\u0026lt;REPLACE-WITH-YOUR-LOG-WORKSPACE-RESOURCE-ID\u0026gt;\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;partTitle\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;HTTP Request Count\u0026#34; } ], \u0026#34;type\u0026#34;: \u0026#34;Extension/KqlPart\u0026#34; } }, \u0026#34;3\u0026#34;: { \u0026#34;position\u0026#34;: { \u0026#34;x\u0026#34;: 0, \u0026#34;y\u0026#34;: 14, \u0026#34;rowSpan\u0026#34;: 10, \u0026#34;colSpan\u0026#34;: 12 }, \u0026#34;metadata\u0026#34;: { \u0026#34;inputs\u0026#34;: [ { \u0026#34;name\u0026#34;: \u0026#34;query\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;ContainerAppConsoleLogs\\n| where AppName == \\\u0026#34;hello-acr-app\\\u0026#34;\\n| sort by TimeGenerated desc\\n| project TimeGenerated, Log_s\\n| take 50\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;version\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;KqlItem/1.0\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;queryType\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;Kusto\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;resourceType\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;microsoft.operationalinsights/workspaces\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;resourceId\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;\u0026lt;REPLACE-WITH-YOUR-LOG-WORKSPACE-RESOURCE-ID\u0026gt;\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;partTitle\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;Latest Console Logs\u0026#34; } ], \u0026#34;type\u0026#34;: \u0026#34;Extension/KqlPart\u0026#34; } } } } }, \u0026#34;metadata\u0026#34;: { \u0026#34;model\u0026#34;: { \u0026#34;timeRange\u0026#34;: { \u0026#34;value\u0026#34;: { \u0026#34;relative\u0026#34;: { \u0026#34;duration\u0026#34;: 3600000 } }, \u0026#34;type\u0026#34;: \u0026#34;MsPortalFx.Composition.Configuration.ValueTypes.TimeRange\u0026#34; } } } }, \u0026#34;name\u0026#34;: \u0026#34;aca-monitoring-dashboard\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;Microsoft.Portal/dashboards\u0026#34;, \u0026#34;location\u0026#34;: \u0026#34;westeurope\u0026#34;, \u0026#34;tags\u0026#34;: {} } Instructions for Importing\nGo to Dashboard in the Azure Portal Click “New Dashboard” → “Upload JSON”. Paste the above JSON into a text file and upload it. Replace: \u0026lt;REPLACE-WITH-YOUR-APP-NAME\u0026gt; — your ACA app name (e.g., hello-acr-app) \u0026lt;REPLACE-WITH-YOUR-LOG-WORKSPACE-RESOURCE-ID\u0026gt; — your Log Analytics Workspace resource ID (you can get this from az monitor log-analytics workspace show …) This is an enhanced version of the Azure Dashboard JSON with an alert example included. The alert will monitor CPU usage and trigger when the average CPU exceeds 80% over a 5-minute window.\nThis includes:\nCPU \u0026amp; Memory charts HTTP request timechart Recent console logs A tile linking to active alerts for easy visibility # Azure Dashboard JSON with Alert Monitoring { \u0026#34;properties\u0026#34;: { \u0026#34;lenses\u0026#34;: { \u0026#34;0\u0026#34;: { \u0026#34;order\u0026#34;: 0, \u0026#34;parts\u0026#34;: { \u0026#34;0\u0026#34;: { \u0026#34;position\u0026#34;: { \u0026#34;x\u0026#34;: 0, \u0026#34;y\u0026#34;: 0, \u0026#34;rowSpan\u0026#34;: 6, \u0026#34;colSpan\u0026#34;: 6 }, \u0026#34;metadata\u0026#34;: { \u0026#34;inputs\u0026#34;: [ { \u0026#34;name\u0026#34;: \u0026#34;resourceType\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;microsoft.app/containerapps\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;resourceName\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;\u0026lt;REPLACE-WITH-YOUR-APP-NAME\u0026gt;\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;metricNamespace\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;microsoft.app/containerapps\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;chartTitle\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;CPU Usage (sec)\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;metrics\u0026#34;, \u0026#34;value\u0026#34;: [ { \u0026#34;metricName\u0026#34;: \u0026#34;CpuUsageSeconds\u0026#34;, \u0026#34;aggregation\u0026#34;: \u0026#34;Average\u0026#34; } ] } ], \u0026#34;type\u0026#34;: \u0026#34;Extension/MetricsChartPart\u0026#34; } }, \u0026#34;1\u0026#34;: { \u0026#34;position\u0026#34;: { \u0026#34;x\u0026#34;: 6, \u0026#34;y\u0026#34;: 0, \u0026#34;rowSpan\u0026#34;: 6, \u0026#34;colSpan\u0026#34;: 6 }, \u0026#34;metadata\u0026#34;: { \u0026#34;inputs\u0026#34;: [ { \u0026#34;name\u0026#34;: \u0026#34;resourceType\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;microsoft.app/containerapps\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;resourceName\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;\u0026lt;REPLACE-WITH-YOUR-APP-NAME\u0026gt;\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;metricNamespace\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;microsoft.app/containerapps\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;chartTitle\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;Memory Usage (bytes)\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;metrics\u0026#34;, \u0026#34;value\u0026#34;: [ { \u0026#34;metricName\u0026#34;: \u0026#34;MemoryWorkingSetBytes\u0026#34;, \u0026#34;aggregation\u0026#34;: \u0026#34;Average\u0026#34; } ] } ], \u0026#34;type\u0026#34;: \u0026#34;Extension/MetricsChartPart\u0026#34; } }, \u0026#34;2\u0026#34;: { \u0026#34;position\u0026#34;: { \u0026#34;x\u0026#34;: 0, \u0026#34;y\u0026#34;: 6, \u0026#34;rowSpan\u0026#34;: 8, \u0026#34;colSpan\u0026#34;: 12 }, \u0026#34;metadata\u0026#34;: { \u0026#34;inputs\u0026#34;: [ { \u0026#34;name\u0026#34;: \u0026#34;query\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;ContainerAppHttpLogs\\n| where AppName == \\\u0026#34;hello-acr-app\\\u0026#34;\\n| summarize count() by bin(TimeGenerated, 5m)\\n| render timechart\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;version\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;KqlItem/1.0\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;queryType\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;Kusto\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;resourceType\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;microsoft.operationalinsights/workspaces\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;resourceId\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;\u0026lt;REPLACE-WITH-YOUR-LOG-WORKSPACE-RESOURCE-ID\u0026gt;\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;partTitle\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;HTTP Request Count\u0026#34; } ], \u0026#34;type\u0026#34;: \u0026#34;Extension/KqlPart\u0026#34; } }, \u0026#34;3\u0026#34;: { \u0026#34;position\u0026#34;: { \u0026#34;x\u0026#34;: 0, \u0026#34;y\u0026#34;: 14, \u0026#34;rowSpan\u0026#34;: 10, \u0026#34;colSpan\u0026#34;: 12 }, \u0026#34;metadata\u0026#34;: { \u0026#34;inputs\u0026#34;: [ { \u0026#34;name\u0026#34;: \u0026#34;query\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;ContainerAppConsoleLogs\\n| where AppName == \\\u0026#34;hello-acr-app\\\u0026#34;\\n| sort by TimeGenerated desc\\n| project TimeGenerated, Log_s\\n| take 50\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;version\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;KqlItem/1.0\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;queryType\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;Kusto\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;resourceType\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;microsoft.operationalinsights/workspaces\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;resourceId\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;\u0026lt;REPLACE-WITH-YOUR-LOG-WORKSPACE-RESOURCE-ID\u0026gt;\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;partTitle\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;Latest Console Logs\u0026#34; } ], \u0026#34;type\u0026#34;: \u0026#34;Extension/KqlPart\u0026#34; } }, \u0026#34;4\u0026#34;: { \u0026#34;position\u0026#34;: { \u0026#34;x\u0026#34;: 0, \u0026#34;y\u0026#34;: 24, \u0026#34;rowSpan\u0026#34;: 6, \u0026#34;colSpan\u0026#34;: 12 }, \u0026#34;metadata\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;Extension/AzureResourceHealthPart\u0026#34;, \u0026#34;inputs\u0026#34;: [ { \u0026#34;name\u0026#34;: \u0026#34;resourceType\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;microsoft.insights/metricalerts\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;resourceIds\u0026#34;, \u0026#34;value\u0026#34;: [ \u0026#34;\u0026lt;REPLACE-WITH-YOUR-ALERT-RULE-RESOURCE-ID\u0026gt;\u0026#34; ] }, { \u0026#34;name\u0026#34;: \u0026#34;partTitle\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;Active Alerts (CPU \u0026gt; 80%)\u0026#34; } ] } } } } }, \u0026#34;metadata\u0026#34;: { \u0026#34;model\u0026#34;: { \u0026#34;timeRange\u0026#34;: { \u0026#34;value\u0026#34;: { \u0026#34;relative\u0026#34;: { \u0026#34;duration\u0026#34;: 3600000 } }, \u0026#34;type\u0026#34;: \u0026#34;MsPortalFx.Composition.Configuration.ValueTypes.TimeRange\u0026#34; } } } }, \u0026#34;name\u0026#34;: \u0026#34;aca-monitoring-dashboard-alerts\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;Microsoft.Portal/dashboards\u0026#34;, \u0026#34;location\u0026#34;: \u0026#34;westeurope\u0026#34;, \u0026#34;tags\u0026#34;: {} } Replace the following placeholders in the script:\n\u0026lt;REPLACE-WITH-YOUR-APP-NAME\u0026gt; -\u0026gt; Your ACA app name (e.g., hello-acr-app) \u0026lt;REPLACE-WITH-YOUR-LOG-WORKSPACE-RESOURCE-ID\u0026gt; -\u0026gt; Resource ID of your Log Analytics Workspace \u0026lt;REPLACE-WITH-YOUR-ALERT-RULE-RESOURCE-ID\u0026gt; -\u0026gt; Resource ID of your Azure Monitor Metric Alert for CPU threshold Dapr and Distributed Tracing in Azure Container Apps When building microservices, it’s common to encounter service discovery, state management, pub/sub, and distributed tracing. This is where Dapr (Distributed Application Runtime) excels, and Azure Container Apps makes it easy to enable with just a few flags.\nWhat is Dapr?\nDapr is an open-source, portable runtime that helps developers build resilient, stateless, and stateful microservices. It abstracts away the boilerplate code needed for service-to-service communication, pub/sub messaging, and external state store integrations.\nIn ACA, enabling Dapr launches a sidecar container next to your app. This sidecar handles:\nService-to-service calls over HTTP/gRPC Pub/Sub messaging with built-in brokers State management (Redis, Cosmos DB, etc.) Input/output bindings Secrets and configuration Distributed tracing Enabling Dapr in Azure Container Apps\nLet’s walk through enabling Dapr for your app and viewing tracing information.\nEnable Dapr on your container app\naz containerapp update \\ --name hello-acr-app \\ --resource-group my-container-apps \\ --enable-dapr \\ --dapr-app-id hellodapr \\ --dapr-app-port 80 Explanation of flags:\n— enable-dapr: Activates the Dapr sidecar — dapr-app-id: A unique logical name used for service discovery (other services can call this app via http://hellodapr) — dapr-app-port: The internal port your container listens on (used by Dapr for ingress) Once enabled, your app and the Dapr sidecar run together inside the ACA environment.\nMaking Service-to-Service Calls with Dapr\nIf you had another ACA with Dapr enabled (frontend for example), it could call the hello-acr-app like this:\nPOST http://hellodapr/v1.0/invoke/hellodapr/method/hello This allows for automatic service discovery, retries, and circuit breaking — no service mesh or Kubernetes config is needed.\nDistributed Tracing with Dapr + OpenTelemetry\nDapr supports OpenTelemetry out of the box, meaning you can trace requests across services and visualize them in tools like:\nAzure Monitor / Application Insights Jaeger Zipkin OpenTelemetry Collector Step 2: View Traces in Azure Monitor\nIf you enable diagnostics (as described earlier), Dapr will emit trace spans into your Log Analytics workspace.\nSample KQL query to view traces:\n# kql traces | where customDimensions[\u0026#34;dapr.component\u0026#34;] has \u0026#34;http\u0026#34; | project timestamp, name, customDimensions, operation_Name | sort by timestamp desc You can use this to:\nSee inbound/outbound calls Identify slow dependencies Correlate requests across microservices Optional: Link to Application Insights\nIf you want advanced visualizations, live metrics, and performance monitoring:\nCreate an Application Insights instance Link it via environment variables or diagnostic settings: Set APPINSIGHTS_INSTRUMENTATIONKEY Or enable via az monitor diagnostic-settings Tips for Operational Monitoring\nUse Log Alerts in Azure Monitor to trigger actions (email, Teams, ITSM). Group metrics and logs in Workbooks for better team visibility. Export to Power BI for cross-environment reporting. Use Application Insights for advanced tracing if needed (e.g., via Dapr/OpenTelemetry). Visualize with Workbooks\nUse Azure Workbooks to build custom dashboards combining:\nLogs from Dapr sidecar Metrics from ACA (CPU, memory, scaling) Tracing spans from OpenTelemetry This gives a unified view of microservice health and dependencies.\nVNet Integration and Private Link for Secure Access Integrating Azure Container Apps with a Virtual Network (VNet) allows you to securely connect your apps to private Azure services, databases, and on-premises networks. Adding Private Link to the setup ensures your environment is accessible via private IP addresses, enhancing security by avoiding exposure to the public internet.\nStep-by-Step: VNet and Private Link Setup Create a Virtual Network and Subnet\nFirst, create a virtual network with a dedicated subnet for ACA. The subnet must have at least a /23 CIDR to ensure enough IPs are available.\naz network vnet create \\ --resource-group my-container-apps \\ --name my-vnet \\ --address-prefix 10.0.0.0/16 \\ --subnet-name my-aca-subnet \\ --subnet-prefix 10.0.1.0/23 Deploy the ACA Environment into the VNet Subnet\nNow associate the ACA environment with the subnet you just created. This allows your container apps to securely communicate with other resources inside the VNet.\naz containerapp env create \\ --name my-aca-env \\ --resource-group my-container-apps \\ --location westeurope \\ --infrastructure-subnet-resource-id \\ \u0026#34;/subscriptions/\u0026lt;subscription-id\u0026gt;/resourceGroups/my-container-apps/providers/Microsoft.Network/virtualNetworks/my-vnet/subnets/my-aca-subnet\u0026#34; Make sure to replace \u0026lt;subscription-id\u0026gt; with your actual subscription ID.\nAccess PaaS Resources from ACA via VNet\nYour container apps can now reach resources like Azure SQL, Cosmos DB, or Redis that are VNet-integrated or configured with Private Endpoints. Use environment variables to inject connection strings.\nEnable Private Link (Optional)\nSet up a private endpoint to ensure your ACA environment is reachable only via private IPs. This involves creating a private DNS zone and associating it with the VNet.\nDiagram: ACA with VNet + Private Link\nBlue/Green Deployment Strategy in ACA Blue/Green deployments reduce downtime and risk by maintaining two environments: one for current production (blue) and one for the new version (green). ACA natively supports this model through revision-based traffic routing.\nBlue/Green with Revisions\nDeploy the Initial (Blue) Version\naz containerapp create \\ --name myapp \\ --resource-group my-container-apps \\ --environment my-aca-env \\ --image myacr.azurecr.io/myapp:v1 \\ --ingress external \\ --target-port 80 \\ --revisions-mode multiple \\ --revision-suffix v1 This sets up the app with the revision suffix v1 as the “blue” environment.\nDeploy the New (Green) Version\naz containerapp update \\ --name myapp \\ --resource-group my-container-apps \\ --image myacr.azurecr.io/myapp:v2 \\ --revision-suffix v2This creates a new revision, `v2,` without affecting traffic yet. Split Traffic Between Blue and Green\naz containerapp ingress traffic set \\ --name myapp \\ --resource-group my-container-apps \\ --revision-weight v1=50 v2=50 This allows to test the green version in production with 50% traffic while monitoring for errors.\nFinalize Green Deployment\nOnce the green version proves stable, route 100% of traffic to it:\naz containerapp ingress traffic set \\ --name myapp \\ --resource-group my-container-apps \\ --revision-weight v1=0 v2=100 Then, optionally deactivate the older revision to save costs:\naz containerapp revision deactivate \\ --name myapp \\ --resource-group my-container-apps \\ --revision v1 Blue/Green Traffic Routing\nBest Practices Use Application Insights to monitor response times and exceptions during the split. Rollback by shifting traffic back to the stable revision. Automate the steps using GitHub Actions or Azure DevOps Pipelines. More Resources\nNetworking in Azure Container App Private Endpoint Guide Traffic Splitting in Azure Container Apps Final Thoughts Azure Container Apps is a powerful addition to Microsoft’s cloud-native portfolio. It elegantly abstracts away the operational burden of Kubernetes while giving developers a robust environment to run containers at scale. With features like revision management, Dapr integration, event-driven scaling, and seamless CI/CD pipeline support, ACA is designed to help teams move fast — without compromising on observability, flexibility, or security.\nWhile it’s not meant to replace Azure Kubernetes Service (AKS), ACA shines in scenarios where simplicity, rapid iteration, and cost-effectiveness are priorities. Think of APIs, microservices, background jobs, or burst workloads that benefit from scale-to-zero capabilities.\nAs the platform continues to evolve, we see more enterprise-grade features like VNet integration, Private Link support, and Application Insights tracing, making ACA an increasingly compelling option for startups and enterprises.\nIf you’re building cloud-native apps and want to offload infrastructure management while still leveraging containerization best practices, ACA is well worth exploring. Don’t forget to check the roadmap, test performance, and cost models for your specific use case, and integrate early with monitoring tools to maintain production readiness.\nDo you have thoughts or questions or want to share how you use ACA in real-world projects? I’d love to hear from you — drop a comment below or connect with me on LinkedIn. Let’s build better, faster, and smarter in the cloud.\n","permalink":"https://wolkwacht.nl/posts/from-code-to-cloud-a-hands-on-guide-to-azure-container-apps/","summary":"\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*cOYo_ORmMItNjJcD3xKyJg.png\"\u003e\u003c/p\u003e\n\u003cp\u003eAs a cloud professional, I constantly seek ways to streamline application delivery while reducing operational overhead. Whether you’re building APIs, microservices, or event-driven backends, choosing the right platform can significantly impact speed to market and maintainability.\u003c/p\u003e\n\u003cp\u003eIntroducing Azure Container Apps (ACA) — a fully managed, serverless platform designed specifically for running containerized workloads without the complexity of orchestrating and managing Kubernetes infrastructure. Built on top of Kubernetes and powered by open-source technologies like \u003ca href=\"https://dapr.io\"\u003eDapr\u003c/a\u003e and \u003ca href=\"https://keda.sh/\"\u003eKEDA\u003c/a\u003e (Kubernetes Event Driven Autoscaling), ACA offers the best of both worlds: the power and flexibility of containers with the simplicity and developer focus of a Platform-as-a-Service (PaaS).\u003c/p\u003e","title":"From Code to Cloud: A Hands-On Guide to Azure Container Apps"},{"content":"Azure Kubernetes Chronicles part 4: Dataprotection Welcome back to the Azure Kubernetes Chronicles! In previous episodes, we explored network-related topics like container network interfaces, eBPF, and observability in Azure Kubernetes Service (AKS). But securing an AKS cluster goes beyond networking and runtime security — it extends to protecting, recovering, and managing data.\nIn this episode, we’re shifting gears to focus on a topic that’s equally critical but often underestimated: data protection in AKS. While Kubernetes brings agility and scalability, it also introduces complexities in securing persistent storage, managing sensitive secrets, and ensuring compliance with evolving regulations like NIS2 and DORA.\nWhether you’re dealing with accidental data loss, ransomware threats, or compliance challenges, having a robust backup, recovery, and encryption strategy is non-negotiable. Misconfigurations, untested restore procedures, and reliance on default security settings can quickly turn an operational hiccup into a full-blown crisis.\nWhy Data Protection Matters in Azure Kubernetes In today’s digital landscape, data breaches have become an all-too-common occurrence, with numerous incidents reported daily around the globe. The frequency of these attacks is alarming, highlighting that anyone can fall victim, regardless of their size or industry. The critical question has shifted from “if” a data breach will occur to “when” it will happen. As we navigate this new reality, understanding the implications and taking proactive measures is essential for safeguarding our valuable information.\nKubernetes workloads often rely heavily on persistent storage, databases, and secrets management. If these components aren’t properly secured or backed up, attackers have a clear path to valuable data.\nAnd let’s not forget compliance. If your organization handles regulated information — think healthcare, finance, or personal data under GDPR — then solid data protection practices aren’t optional; they’re mandatory. Misconfigured AKS clusters can inadvertently expose sensitive data, risking not just reputation damage but legal and financial penalties, too.\nCommon Pitfalls in AKS Data Protection A big misconception around Azure Kubernetes Service (AKS) is that it automatically covers all data protection bases. Spoiler: it doesn’t. AKS has some great built-in security features, but leaving things on default can quickly backfire. Misconfigured persistent storage, skipping encryption, or neglecting proper RBAC (Role-Based Access Control) setups are all easy ways to accidentally expose your cluster’s data.\nAnother pitfall is mishandling of Kubernetes secrets. Kubernetes secrets are base64 encoded — not encrypted — by default. It’s way too common to see sensitive info like API keys or passwords sitting openly in ConfigMaps or static environment variables, often without proper rotation policies. This kind of shortcut leaves your systems wide open if anyone breaches your cluster.\nBackup strategies also frequently miss the mark. Many teams trust cloud-provider snapshots without realizing these might not be application-aware or reliable enough for critical data. Without a tested and validated restore plan, backups become a false sense of security, and recovery can turn into chaos exactly when you need calm.\nNIS2 and DORA: What They Mean for Data Protection in AKS Regulations aren’t just legal hurdles — they’re here to make sure we don’t wake up one day to find our critical data lost or stolen. In the world of Kubernetes, particularly in Azure Kubernetes Service (AKS), data protection is more than just ticking compliance boxes; it’s about making sure your workloads are resilient, secure, and recoverable.\nTwo key regulations are shaping how organizations approach data security in the EU: NIS2 (Network and Information Security Directive 2) and DORA (Digital Operational Resilience Act). While they target different industries, they both push for stronger cybersecurity, backup strategies, and disaster recovery plans — which are must-haves for anyone running Kubernetes in production.\nWhat NIS2 Means for Your AKS Setup\nYou need to have solid backup and recovery plans. If something goes wrong — be it a cyberattack, a misconfiguration, or accidental deletion — you must restore your workloads quickly.\nIncident response and reporting are a must. If an attack or outage happens, NIS2 requires that it be reported promptly.\nData security (encryption and access control) needs to be rock solid. NIS2 mandates the proper protection of sensitive data, meaning no more storing secrets in plain text.\nWhat DORA Means for AKS in Financial Services\nYou can’t just “hope” your backups work — you need to test them. DORA mandates regular testing of disaster recovery plans to prove that your backups are useful.\nThird-party cloud risks need to be managed. Financial institutions using cloud services must ensure that their providers meet resilience and security requirements.\nRecovery Time Objectives (RTO) and Recovery Point Objectives (RPO) matter. DORA forces organizations to define how quickly they can recover from a failure (RTO) and how much data loss is acceptable (RPO).\nNIS2 and DORA aren’t just another set of bureaucratic rules — they’re a wake-up call to take data protection seriously. Whether you’re managing cloud services under NIS2 or financial workloads under DORA, your AKS backup and recovery strategy is critical.\nSolutions: Backup and Restore Strategies for AKS When you’re protecting your data in AKS, having a solid backup and restore strategy is essential. Start by using Kubernetes-native tools like Velero, which are built specifically to work smoothly with Kubernetes clusters. Velero helps automate and manage backup operations, including snapshots of cluster states, resource definitions, and persistent volume data. It is easy to schedule regular backups, configure custom backup schedules, and even integrate hooks for database consistency.\nIf there is a need for more enterprise-level features and more management options, then solutions like Veeam Kasten K10 is a good choice. These tools offer advanced capabilities like policy-driven backup automation, incremental backups, application-aware snapshots, and intuitive dashboards that simplify management and reporting. They also handle backups across multiple clusters and clouds, making them ideal for complex environments.\nThese backup tools cover all the bases by backing up both cluster resources and persistent volumes, ensuring comprehensive protection. Plus, they simplify the entire restore and disaster recovery process, making it easier when things inevitably go sideways. With features like point-in-time recovery and granular restores, you can recover exactly what you need, quickly and accurately.\nFirst, we look at how Velero is installed on an Azure Kubernetes Cluster. After that, we take a quick look at Veeam Kasten.\nInstall Velero\nCreate an Azure storage account to store the backups from Velero\n# Create Storage Account az storage account create \\ --name blogvelerobackup \\ --resource-group aks-blog-rg \\ --sku Standard_LRS \\ --kind StorageV2 \\ --access-tier Hot \\ --location westeurope \\ --default-action Allow Velero consists of a client part, Velero cli, and a server part. Velero cli is used to manage the Velero server installation. We start by installing the Velero cli.\n# for macOS (Homebrew) brew install velero # direct binary download VELERO_VERSION=v1.13.2 wget https://github.com/vmware-tanzu/velero/releases/download/$VELERO_VERSION/velero-$VELERO_VERSION-darwin-amd64.tar.gz tar -xvf velero-$VELERO_VERSION-darwin-amd64.tar.gz sudo mv velero-$VELERO_VERSION-darwin-amd64/velero /usr/local/bin/ Velero uses a file to authenticate with the storage account. We have to put the Azure Storage Account Key in a credential file.\naz storage account keys list \\ --resource-group aks-blog-rg \\ --account-name blogvelerobackup \\ --query \u0026#39;[0].value\u0026#39; -o tsv vim credentials-velero Paste the following content into the file\nAZURE_STORAGE_ACCOUNT_ACCESS_KEY=\u0026lt;YOUR_STORAGE_ACCOUNT_ACCESS_KEY\u0026gt; Set the permissions for safety on the credentials-velero file\nchmod 600 credentials-velero Install Velero using the following script\n# install-velero.sh #!/bin/bash RESOURCE_GROUP=aks-blog-rg AZURE_STORAGE_ACCOUNT=blogvelerobackup BLOB_CONTAINER=velero # Create storage container (using Azure AD login) az storage container create \\ --name $BLOB_CONTAINER \\ --account-name $AZURE_STORAGE_ACCOUNT \\ --auth-mode login # Install Velero with Azure plugin velero install \\ --provider azure \\ --plugins velero/velero-plugin-for-microsoft-azure:v1.7.0 \\ --bucket $BLOB_CONTAINER \\ --secret-file ./credentials-velero \\ --backup-location-config resourceGroup=$RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT \\ --snapshot-location-config resourceGroup=$RESOURCE_GROUP,subscriptionId=$(az account show --query id -o tsv) Check the Velero installation\nkubectl get pods -n velero To perform a backup, run the following command\nvelero backup create \u0026lt;backup-name\u0026gt; --include-namespaces \u0026lt;namespace\u0026gt; To restore from the backup use\nvelero restore create --from-backup \u0026lt;backup-name\u0026gt; Install Veeam Kasten\nkubectl create namespace kasten-io helm repo add kasten https://charts.kasten.io/ helm repo update helm install k10 kasten/k10 --namespace kasten-io Access the Kasten Dashboard via port forwarding\nkubectl --namespace kasten-io port-forward service/gateway 8080:8000 By using port forwarding, we can open the Kasten Dashboard on http://localhost:8080.\nIn the dashboard, you can configure policies to create backups for the applications in the Azure Kubernetes cluster. The dashboard offers a lot of different options to configure the disaster recovery scenarios.\nBut backups alone aren’t enough. Encryption and secure secret management are also important. Possible options for safely storing your secrets are solutions like Azure Key Vault or HashiCorp Vault. Both of these solutions can integrate with Azure Kubernetes. Next to using a Vault solution, it is also important to configure Role-Based Access Control (RBAC). Make sure that the right permissions are assigned and monitor them regularly.\nInjecting secrets from Azure Key Vault into Azure Kubernetes Service (AKS) workloads Set up Azure Key Vault and store a secret Configure AKS to access Key Vault secrets Deploy an application to AKS that uses the injected secret Set up Azure Key Vault\n# Create a Key Vault az keyvault create --name BlogKeyVault --resource-group aks-blog-rg --location westeurope # Create role assigment az role assignment create \\ --role \u0026#34;Key Vault Secrets Officer\u0026#34; \\ --assignee \u0026#34;\u0026lt;Tenant-id\u0026gt;\u0026#34; \\ --scope \u0026#34;/subscriptions/c9465047-a812-42e7-a53b-739940940898/resourceGroups/aks-blog-rg/providers/Microsoft.KeyVault/vaults/BlogKeyVault\u0026#34; # Add a secret az keyvault secret set --vault-name BlogKeyVault --name MySecret --value \u0026#34;SuperSecretPassword\u0026#34; Connect Azure Kubernetes Service with Azure Key Vault\nWe will use Azure Workload Identity, a secure and recommended way to authenticate pods to Azure resources.\nEnable workload identity on your AKS cluster\naz aks update -n aks-blog-cluster -g aks-blog-rg --enable-oidc-issuer --enable-workload-identity Create an Azure Managed Identity\nIDENTITY_CLIENT_ID=$(az identity show -n AKSIdentity -g aks-blog-rg --query clientId -o tsv) IDENTITY_OBJECT_ID=$(az identity show -n AKSIdentity -g aks-blog-rg --query principalId -o tsv) az role assignment create \\ --role \u0026#34;Key Vault Secrets User\u0026#34; \\ --assignee-object-id \u0026#34;$IDENTITY_OBJECT_ID\u0026#34; \\ --assignee-principal-type ServicePrincipal \\ --scope \u0026#34;$(az keyvault show --name BlogKeyVault --query id -o tsv)\u0026#34;… Create a Kubernetes service account and link it to the Azure identity\n# aks-sa.yaml apiVersion: v1 kind: ServiceAccount metadata: name: akv-service-account namespace: default annotations: azure.workload.identity/client-id: \u0026#34;\u0026lt;IDENTITY_CLIENT_ID\u0026gt;\u0026#34; kubectl apply -f aks-sa.yaml Federated Credential setup\nWe have enabled workload identity in AKS, so we have to configure a federated identity credential for the earlier created Managed Identity ( AKSIdentity)\n# Set variables AKS_OIDC_ISSUER=\u0026#34;$(az aks show -n aks-blog-cluster -g aks-blog-rg --query \u0026#34;oidcIssuerProfile.issuerUrl\u0026#34; -o tsv)\u0026#34; IDENTITY_NAME=\u0026#34;AKSIdentity\u0026#34; NAMESPACE=\u0026#34;default\u0026#34; SERVICE_ACCOUNT_NAME=\u0026#34;akv-service-account\u0026#34; RESOURCE_GROUP=\u0026#34;aks-blog-rg\u0026#34; SUBSCRIPTION_ID=\u0026#34;$(az account show --query id -o tsv)\u0026#34; # Create federated identity credntial az identity federated-credential create \\ --name aks-keyvault-fic \\ --identity-name \u0026#34;$IDENTITY_NAME\u0026#34; \\ --resource-group \u0026#34;$RESOURCE_GROUP\u0026#34; \\ --issuer \u0026#34;$AKS_OIDC_ISSUER\u0026#34; \\ --subject \u0026#34;system:serviceaccount:$NAMESPACE:$SERVICE_ACCOUNT_NAME\u0026#34; \\ --audiences \u0026#34;api://AzureADTokenExchange\u0026#34; Injecting Secrets into Your Application\nWe will use the Azure Key Vault Secrets Provider CSI Driver. To be able to use it, we have to enable it in AKS\naz aks enable-addons --addons azure-keyvault-secrets-provider --name aks-blog-cluster --resource-group aks-blog-rg Create a deployment YAML that injects your secret\napiVersion: apps/v1 kind: Deployment metadata: name: my-app spec: replicas: 1 selector: matchLabels: app: my-app template: metadata: labels: app: my-app spec: serviceAccountName: akv-service-account containers: - name: my-app-container image: nginx env: - name: SECRET_FROM_KEYVAULT valueFrom: secretKeyRef: name: my-secret key: MySecret volumeMounts: - name: secrets-store-inline mountPath: \u0026#34;/mnt/secrets-store\u0026#34; readOnly: true volumes: - name: secrets-store-inline csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: \u0026#34;azure-kvname\u0026#34; --- apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: azure-kvname spec: provider: azure parameters: usePodIdentity: \u0026#34;false\u0026#34; keyvaultName: BlogKeyVault tenantId: \u0026#34;983d915c-b881-4803-bf11-a81f8e51e3bc\u0026#34; clientID: \u0026#34;b5e1174d-952a-44d3-8651-476a83c2babe\u0026#34; objects: | array: - | objectName: MySecret objectType: secret secretObjects: - secretName: my-secret type: Opaque data: - objectName: MySecret key: MySecret Apply the deployment\nkubectl apply -f deployment.yaml Check the name of your pod with kubectl get pods -n default. Exec into your container.\nkubectl exec -it \u0026lt;new-pod-name\u0026gt; -- printenv SECRET_FROM_KEYVAULT You should see as output\nSuperSecretPassword Another key aspect that often is overlooked is monitoring. Backups aren’t the most exciting thing in the world, so it’s easy to set them up and just assume they’ll work when needed. It is also important to keep an eye on your backup performance and health. Azure Monitor is one of the tools that can do just that by giving you detailed insights into what’s happening behind the scenes. It highlights any anomalies, performance issues, or even backups that aren’t completing as expected.\nAnd, of course, don’t underestimate the importance of testing your restore procedures regularly. It is easy to assume everything is working fine. But there is no worse feeling than confidently heading into a disaster recovery scenario, only to discover the backups were silently failing the whole time. Schedule periodic test restores, make sure the process is smooth, and confirm you can get your data back quickly and reliably. That extra bit of testing can make all the difference when disaster strikes.\nConclusion Data protection in Azure Kubernetes Service (AKS) isn’t just a compliance checkbox — it’s a critical component of running resilient, secure, and recoverable workloads. Misconfigurations, overlooked backups, and poor secret management can turn small mistakes into major incidents.\nBy implementing a solid backup and restore strategy with tools like Velero or Veeam Kasten, securing secrets with Azure Key Vault or HashiCorp Vault, and aligning with regulations like NIS2 and DORA, you can ensure your AKS clusters remain protected against both operational failures and security threats.\nBut don’t stop at implementation — regularly test your restore processes, monitor backup health, and refine your disaster recovery plan. The worst time to find out your backups aren’t working is when you need them the most.\nBackups are like seatbelts: you hope you never need them, but you’ll be grateful they’re there when disaster strikes.\nStay tuned for the next episode of Azure Kubernetes Chronicles! 🚀\n","permalink":"https://wolkwacht.nl/posts/azure-kubernetes-chronicles-part-4/","summary":"\u003ch2 id=\"azure-kubernetes-chronicles-part-4-dataprotection\"\u003eAzure Kubernetes Chronicles part 4: Dataprotection\u003c/h2\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*LrOkjjVsm52vAudzwHunEw.png\"\u003e\u003c/p\u003e\n\u003cp\u003eWelcome back to the \u003cem\u003eAzure Kubernetes Chronicles\u003c/em\u003e! In previous episodes, we explored network-related topics like container network interfaces, eBPF, and observability in Azure Kubernetes Service (AKS). But securing an AKS cluster goes beyond networking and runtime security — it extends to protecting, recovering, and managing data.\u003c/p\u003e\n\u003cp\u003eIn this episode, we’re shifting gears to focus on a topic that’s equally critical but often underestimated: \u003cstrong\u003edata protection in AKS\u003c/strong\u003e. While Kubernetes brings agility and scalability, it also introduces complexities in securing persistent storage, managing sensitive secrets, and ensuring compliance with evolving regulations like NIS2 and DORA.\u003c/p\u003e","title":"Azure Kubernetes Chronicles part 4"},{"content":"Azure Kubernetes Chronicles part 3: Observability Welcome back to the Azure Kubernetes Chronicles! In our previous posts, we explored the basics of Container Network Interfaces (CNIs) and had a first glance at how eBPF is changing networking within Azure Kubernetes Service (AKS). If you haven’t had a chance to read it yet, you might want to catch up here.\nIn this episode, we’re diving even deeper into the labyrinth of Kubernetes by shining a light on a topic that’s both critical and fascinating: Observability!\nWhat is Observability? Observability in cloud-native environments refers to the ability to gain deep insights into the internal state of systems based on external outputs. It extends beyond traditional monitoring by focusing on three core pillars: metrics, logs, and traces. Metrics provide quantitative data on system performance, logs capture detailed event records, and traces map the flow of requests across distributed components. Observability is critical for modern cloud operations because it enables engineers to diagnose issues proactively, optimize performance, and ensure reliability. Unlike traditional monitoring, which relies on predefined thresholds, observability allows teams to explore unknown failure modes by analyzing real-time telemetry data.\neBPF and observability eBPF (Extended Berkeley Packet Filter) is revolutionizing observability by enabling lightweight and high-performance monitoring directly within the Linux kernel. eBPF allows programs to run in a sandboxed manner within the kernel, collecting granular data about system behavior without modifying application code. This makes it highly effective for deep observability in Kubernetes environments, as it can track network traffic, system calls, and performance metrics with minimal overhead. With eBPF, cloud engineers can gain insights into issues like latency spikes, security anomalies, and inefficient resource usage in real time, making it a powerful tool for modern observability solutions.\nImplementation in Azure Azure Monitor for containers provides comprehensive logging, metrics, and distributed tracing for AKS workloads, enabling engineers to detect performance bottlenecks and failures efficiently. Prometheus and Grafana can be integrated for custom metrics visualization, while Azure Log Analytics aggregates logs from nodes and applications. Additionally, eBPF-powered solutions like Cilium enhance network observability and security within AKS clusters. By leveraging these tools, cloud architects and engineers can gain full-stack visibility into their Kubernetes environments, ensuring operational excellence in cloud-native applications.\n# Create the resource group az group create --name aks-blog-rg --location swedencentral # Deploy the cluster az aks create \\ --name aks-blog-cluster \\ --resource-group aks-blog-rg \\ --location swedencentral \\ --tier standard \\ --kubernetes-version 1.29 \\ --os-sku AzureLinux \\ --node-count 3 \\ --load-balancer-sku standard \\ --network-plugin azure \\ --network-plugin-mode overlay \\ --network-dataplane cilium \\ --network-policy cilium \\ --enable-managed-identity \\ --enable-azure-monitor-metrics \\ --enable-acns \\ --generate-ssh-keys # Get AKS credentials az aks get-credentials --name aks-blog-cluster --resource-group aks-blog-rg The script will deploy an Azure Kubernetes Cluster with the following configuration:\nDeployed in Sweden Central Kubernetes version 1.29 Azure Linux for the nodes Standard Tier 3 nodes Cilium for dataplane and network policies Advanced Container Networking Services Enabled Azure managed Prometheus and Grafana Azure offers fully managed services for both Prometheus and Grafana, streamlining observability while reducing operational overhead.\nAzure Monitor Managed Service for Prometheus Azure Monitor Managed Service for Prometheus is a fully managed, scalable implementation of the popular open-source monitoring system, Prometheus. Designed for cloud-scale observability, it integrates seamlessly with Azure Kubernetes Service (AKS) and Azure Arc-enabled Kubernetes clusters, providing high availability, automatic updates, and long-term data retention (up to 18 months). Unlike self-managed Prometheus, this service removes the burden of managing Prometheus infrastructure while ensuring compliance with enterprise security and governance policies.\nOne key difference between Azure’s managed Prometheus and the open-source version is case insensitivity, which may impact existing users who rely on case-sensitive metric naming. Organizations using custom dashboards or alerts that depend on case-sensitive labels should evaluate compatibility before migrating.\nPros:\n✅ Fully managed — no need to operate and scale Prometheus manually.\n✅ Deep integration with Azure services (e.g., Azure Monitor, AKS).\n✅ Automatic updates, high availability, and up to 18 months of data retention.\nCons:\n❌ Case insensitivity may cause issues for teams relying on case-sensitive metrics.\n❌ Costs may be higher than self-hosted Prometheus in certain use cases.\nAzure Managed Grafana Azure Managed Grafana is a fully managed visualization and analytics service based on the open-source Grafana project. It enables teams to create real-time dashboards and alerts using data sources like Azure Monitor, Azure Managed Prometheus, and self-hosted Prometheus instances. The service provides pre-configured dashboards for common Azure workloads, reducing the setup time for monitoring AKS, virtual machines, and application services.\nSince Azure Managed Grafana is a fully managed service, it eliminates the operational burden of managing a self-hosted Grafana instance. However, customization options may be more restricted compared to self-hosted Grafana, particularly for organizations needing extensive plugin support, authentication mechanisms, or fine-grained access control.\nPros:\n✅ Fully managed, reducing operational complexity.\n✅ Pre-built dashboards and Azure-native integrations for quick insights.\n✅ Supports both Azure-managed and self-hosted Prometheus backends.\nCons:\n❌ Limited customization and plugin support compared to self-hosted Grafana.\n❌ Costs may add up for large-scale usage compared to a self-hosted solution.\nWe start with creation of the Azure Monitor resource.\naz resource create \\ --resource-group aks-blog-rg \\ --namespace microsoft.monitor \\ --resource-type accounts \\ --name AKS-blog-monitor \\ --location swedencentral \\ --properties \u0026#39;{}\u0026#39; Next step is the creation of the Grafana instance and putting the Azure Monitor and Azure Managed Grafana id’s in variables.\naz grafana create \\ --name AKS-blog-Grafana \\ --resource-group aks-blog-rg grafanaId=$(az grafana show \\ --name AKS-blog-Grafana \\ --resource-group aks-blog-rg \\ --query id \\ --output tsv) azuremonitorId=$(az resource show \\ --resource-group aks-blog-rg \\ --name AKS-blog-monitor \\ --resource-type \u0026#34;Microsoft.Monitor/accounts\u0026#34; \\ --query id \\ --output tsv) Now that we are going to use the resource id’s that we have put in variables to link the Azure Monitor and Azure Managed Grafana to the Azure Kubernetes Cluster.\naz aks update \\ --name aks-blog-cluster \\ --resource-group aks-blog-rg\\ --enable-azure-monitor-metrics \\ --azure-monitor-workspace-resource-id $azuremonitorId \\ --grafana-resource-id $grafanaId After this step, we check if the monitor pod’s are runnning.\nkubectl get pods -o wide -n kube-system | grep ama- The output should look something like this\nMicrosoft has provided sample dashboards as part of the Advanced Container Networking Services to get you started. To open the dashboards go to the portal and search for Managed Grafana. Open the link to Grafana. Go to Dashboards and select the folder Azure Managed Prometheus. There you can find the dashboards.\nWhat is Cilium Hubble? Cilium Hubble is an observability platform built on top of Cilium, an open-source networking and security project powered by eBPF. Hubble extends Cilium’s capabilities by providing real-time monitoring, security visibility, and troubleshooting insights for Kubernetes workloads.\nAt its core, Hubble enables fine-grained, service-level visibility into network traffic, DNS queries, and application-layer interactions. It provides a UI and CLI to visualize network flows, security policies, and performance metrics, making it a crucial tool for modern cloud-native environments.\nKey Features of Cilium Hubble Deep Network Observability: Provides detailed insights into L3-L7 traffic, including HTTP, gRPC, and Kafka protocols. eBPF-Powered Efficiency: Uses eBPF to collect telemetry data with minimal performance overhead. Flow Visibility \u0026amp; Service Dependency Graphs: Visualizes network traffic between Kubernetes pods, namespaces, and services. Security Policy Audit \u0026amp; Enforcement: Helps validate network security policies and troubleshoot enforcement issues. Distributed Tracing: Supports integrations with Jaeger and OpenTelemetry to provide distributed tracing capabilities. CLI and UI Dashboards: Allows engineers to explore network flows, filter logs, and analyze data in real time. Pros of Using Cilium Hubble ✅ Kubernetes-Native: Designed specifically for Kubernetes environments, making it an ideal fit for cloud-native applications.\n✅ Lightweight \u0026amp; High Performance: eBPF ensures efficient data collection without introducing significant latency.\n✅ Enhanced Security Visibility: Helps identify policy violations, unauthorized access attempts, and suspicious traffic patterns.\n✅ Comprehensive Observability: Offers deep insights into network behavior, security posture, and application interactions.\n✅ Seamless Integrations: Works well with Prometheus, Grafana, OpenTelemetry, and security tools like Falco.\nCons of Using Cilium Hubble ❌ Learning Curve: Requires familiarity with Cilium, eBPF, and Kubernetes networking concepts.\n❌ Operational Complexity: Managing and configuring Hubble at scale may require additional expertise.\n❌ Resource Overhead: While efficient, eBPF-based monitoring can still introduce some resource consumption.\n❌ Limited Outside Kubernetes: Primarily designed for Kubernetes environments, making it less useful for non-containerized workloads.\nIn the script that we used to deploy the Kubernetes clusters we have chosen Cilium for the dataplane and the network policies. Hubble is installed as part of the Cilium installation.\nTo visualize the Hubble flow we can use the Hubble Cli or we can use Hubble UI.\nVisualize using the Hubble Cli First step is installing the Hubble Cli.\n# Set environment variables export HUBBLE_VERSION=v1.16.3 export HUBBLE_ARCH=amd64 #Install Hubble CLI if [ \u0026#34;$(uname -m)\u0026#34; = \u0026#34;aarch64\u0026#34; ]; then HUBBLE_ARCH=arm64; fi curl -L - fail - remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum} sha256sum - check hubble-linux-${HUBBLE_ARCH}.tar.gz.sha256sum sudo tar xzvfC hubble-linux-${HUBBLE_ARCH}.tar.gz /usr/local/bin rm hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum} Next step is checking if the Hubble pods are running.\nkubectl get pods -o wide -n kube-system -l k8s-app=hubble-relay or Cilium status To connect the Hubble Cli to the Hubble Relay, we have to forward the port\nkubectl port-forward -n kube-system svc/hubble-relay --address 127.0.0.1 4245:443 Securing communications between the Hubble Relay server and its clients is critical. By leveraging Mutual TLS (mTLS), both the server and client authenticate each other using digital certificates, establishing a robust and trusted connection. To enable the Hubble client to retrieve flow data, you must first obtain the necessary certificates and then configure the client to use them. The certificates can be applied with the following commands:\n#!/usr/bin/env bash set -euo pipefail set -x # Directory where certificates will be stored CERT_DIR=\u0026#34;$(pwd)/.certs\u0026#34; mkdir -p \u0026#34;$CERT_DIR\u0026#34; declare -A CERT_FILES=( [\u0026#34;tls.crt\u0026#34;]=\u0026#34;tls-client-cert-file\u0026#34; [\u0026#34;tls.key\u0026#34;]=\u0026#34;tls-client-key-file\u0026#34; [\u0026#34;ca.crt\u0026#34;]=\u0026#34;tls-ca-cert-files\u0026#34; ) for FILE in \u0026#34;${!CERT_FILES[@]}\u0026#34;; do KEY=\u0026#34;${CERT_FILES[$FILE]}\u0026#34; JSONPATH=\u0026#34;{.data[\u0026#39;${FILE//./\\\\.}\u0026#39;]}\u0026#34; # Retrieve the secret and decode it kubectl get secret hubble-relay-client-certs -n kube-system -o jsonpath=\u0026#34;${JSONPATH}\u0026#34; | base64 -d \u0026gt; \u0026#34;$CERT_DIR/$FILE\u0026#34; # Set the appropriate hubble CLI config hubble config set \u0026#34;$KEY\u0026#34; \u0026#34;$CERT_DIR/$FILE\u0026#34; done hubble config set tls true hubble config set tls-server-name instance.hubble-relay.cilium.io After the running the script we have to check if the secrets were generated.\nkubectl get secrets -n kube-system | grep hubble- The hubble observe command supports various parameters that help refine and filter flow data for better analysis. Key parameters include — namespace to limit results to a specific Kubernetes namespace, — pod to filter flows for a particular pod, and — service to track traffic related to a specific Kubernetes service. You can also use — src and — dst to filter based on source or destination IPs, CIDRs, or Kubernetes identities. For protocol-based filtering, the — protocol flag allows you to isolate traffic for TCP, UDP, or ICMP flows. Additionally, the — port option focuses on traffic using a specific port, which is useful for debugging application-layer issues.\nFor deeper insights, the — type parameter helps categorize flows into allowed (L3_L4), dropped (DROP), or error-related (L7) events. The ‘— json’ flag outputs raw JSON for advanced parsing, while ’ — since’ and ‘— until’ allow querying historical flow data within a specific time range. The ‘— follow’ option streams live flow events in real time, making it useful for active debugging. These parameters make hubble observe a flexible and powerful tool for monitoring network activity, troubleshooting connectivity issues, and enforcing security policies in Kubernetes environments.\nVisualizing using the Hubble UI The Hubble UI provides a graphical interface for visualizing network traffic within a Kubernetes cluster, offering a more intuitive alternative to command-line monitoring. It presents real-time flow data with interactive graphs that display service-to-service communication, network policies, and packet flow details. Users can filter flows based on namespaces, pods, services, or specific endpoints, making it easier to analyze traffic patterns and troubleshoot connectivity issues. The UI also categorizes flows into allowed, dropped, and forwarded events, helping teams quickly identify misconfigurations or security policy violations.\nBefore we install the Hubble UI, we start with the deployment of an application.\nFor that we use Stan’s Robot Shop, by IBM, a sample microservices application designed for demonstrating observability and monitoring capabilities. It includes multiple services such as web, payment, and database services. For more information about the application, check out this link.\nStan’s Robot Shop offers a Helm chart for streamlined deployment. Follow these steps to deploy the application.\n#Clone the repository git clone https://github.com/instana/robot-shop.git cd robot-shop/K8s/helm Create the namespace to deploy the application in.\nkubectl create namespace robot-shop The Helm Chart will be used to will deploy all the necessary services and components of Stan’s Robot Shop into the AKS cluster.\nhelm install robot-shop --namespace robot-shop Next step is enabling the Hubble-UI. For that we have to save the following text in hubble-ui.yaml\napiVersion: v1 kind: ServiceAccount metadata: name: hubble-ui namespace: kube-system --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: hubble-ui labels: app.kubernetes.io/part-of: retina rules: - apiGroups: - networking.k8s.io resources: - networkpolicies verbs: - get - list - watch - apiGroups: - \u0026#34;\u0026#34; resources: - componentstatuses - endpoints - namespaces - nodes - pods - services verbs: - get - list - watch - apiGroups: - apiextensions.k8s.io resources: - customresourcedefinitions verbs: - get - list - watch - apiGroups: - cilium.io resources: - \u0026#34;*\u0026#34; verbs: - get - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: hubble-ui labels: app.kubernetes.io/part-of: retina roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: hubble-ui subjects: - kind: ServiceAccount name: hubble-ui namespace: kube-system --- apiVersion: v1 kind: ConfigMap metadata: name: hubble-ui-nginx namespace: kube-system data: nginx.conf: | server { listen 8081; server_name localhost; root /app; index index.html; client_max_body_size 1G; location / { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; # CORS add_header Access-Control-Allow-Methods \u0026#34;GET, POST, PUT, HEAD, DELETE, OPTIONS\u0026#34;; add_header Access-Control-Allow-Origin *; add_header Access-Control-Max-Age 1728000; add_header Access-Control-Expose-Headers content-length,grpc-status,grpc-message; add_header Access-Control-Allow-Headers range,keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout; if ($request_method = OPTIONS) { return 204; } # /CORS location /api { proxy_http_version 1.1; proxy_pass_request_headers on; proxy_hide_header Access-Control-Allow-Origin; proxy_pass http://127.0.0.1:8090; } location / { try_files $uri $uri/ /index.html /index.html; } # Liveness probe location /healthz { access_log off; add_header Content-Type text/plain; return 200 \u0026#39;ok\u0026#39;; } } } --- kind: Deployment apiVersion: apps/v1 metadata: name: hubble-ui namespace: kube-system labels: k8s-app: hubble-ui app.kubernetes.io/name: hubble-ui app.kubernetes.io/part-of: retina spec: replicas: 1 selector: matchLabels: k8s-app: hubble-ui template: metadata: labels: k8s-app: hubble-ui app.kubernetes.io/name: hubble-ui app.kubernetes.io/part-of: retina spec: serviceAccountName: hubble-ui automountServiceAccountToken: true containers: - name: frontend image: mcr.microsoft.com/oss/cilium/hubble-ui:v0.12.2 imagePullPolicy: Always ports: - name: http containerPort: 8081 livenessProbe: httpGet: path: /healthz port: 8081 readinessProbe: httpGet: path: / port: 8081 resources: {} volumeMounts: - name: hubble-ui-nginx-conf mountPath: /etc/nginx/conf.d/default.conf subPath: nginx.conf - name: tmp-dir mountPath: /tmp terminationMessagePolicy: FallbackToLogsOnError securityContext: {} - name: backend image: mcr.microsoft.com/oss/cilium/hubble-ui-backend:v0.12.2 imagePullPolicy: Always env: - name: EVENTS_SERVER_PORT value: \u0026#34;8090\u0026#34; - name: FLOWS_API_ADDR value: \u0026#34;hubble-relay:443\u0026#34; - name: TLS_TO_RELAY_ENABLED value: \u0026#34;true\u0026#34; - name: TLS_RELAY_SERVER_NAME value: ui.hubble-relay.cilium.io - name: TLS_RELAY_CA_CERT_FILES value: /var/lib/hubble-ui/certs/hubble-relay-ca.crt - name: TLS_RELAY_CLIENT_CERT_FILE value: /var/lib/hubble-ui/certs/client.crt - name: TLS_RELAY_CLIENT_KEY_FILE value: /var/lib/hubble-ui/certs/client.key livenessProbe: httpGet: path: /healthz port: 8090 readinessProbe: httpGet: path: /healthz port: 8090 ports: - name: grpc containerPort: 8090 resources: {} volumeMounts: - name: hubble-ui-client-certs mountPath: /var/lib/hubble-ui/certs readOnly: true terminationMessagePolicy: FallbackToLogsOnError securityContext: {} nodeSelector: kubernetes.io/os: linux volumes: - configMap: defaultMode: 420 name: hubble-ui-nginx name: hubble-ui-nginx-conf - emptyDir: {} name: tmp-dir - name: hubble-ui-client-certs projected: defaultMode: 0400 sources: - secret: name: hubble-relay-client-certs items: - key: tls.crt path: client.crt - key: tls.key path: client.key - key: ca.crt path: hubble-relay-ca.crt --- kind: Service apiVersion: v1 metadata: name: hubble-ui namespace: kube-system labels: k8s-app: hubble-ui app.kubernetes.io/name: hubble-ui app.kubernetes.io/part-of: retina spec: type: ClusterIP selector: k8s-app: hubble-ui ports: - name: http port: 80 targetPort: 8081 After we have saved the file, we can apply the yaml using kubectl\nkubectl apply -f hubble-ui.yaml To make the Hubble UI reachable we forward the port\nkubectl -n kube-system port-forward svc/hubble-ui 12000:80 We can now open the Hubble UI using the browser and connect to http://localhost:/12000/ . The Hubble UI presents all the available namespaces in the cluster. We select the robot-shop namespace to see traffic between the different parts off the application.\nIt is possible to filter the traffic on all, forwarded or dropped traffic.\nConclusion In closing, observability isn’t just another buzzword — it’s a critical part of managing modern cloud infrastructure. By using tools like eBPF, Cilium Hubble, and Azure’s managed Prometheus and Grafana services, you get real, actionable insights into your AKS clusters. This isn’t just about spotting issues after they happen; it’s about building a proactive approach that lets you diagnose problems, optimize performance, and secure your environment before challenges escalate.\nWhat we’ve explored here shows that while these tools add incredible power to your operations, they also demand a good balance of technical know-how and thoughtful planning. I encourage you to dive in, experiment with these solutions, and adapt them to fit your specific needs.\nThanks for reading. I encourage you to experiment with these tools and share your experiences and I look forward to sharing more on this journey as we continue exploring the ever-evolving world of Azure Kubernetes. Stay tuned for the next episode of Azure Kubernetes Chronicles!\n","permalink":"https://wolkwacht.nl/posts/azure-kubernetes-chronicles-part-3/","summary":"\u003ch2 id=\"azure-kubernetes-chronicles-part-3-observability\"\u003eAzure Kubernetes Chronicles part 3: Observability\u003c/h2\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*aYn2CkTsEdRyUrz9NyeSYA.png\"\u003e\u003c/p\u003e\n\u003cp\u003eWelcome back to the \u003cem\u003eAzure Kubernetes Chronicles\u003c/em\u003e! In our previous posts, we explored the basics of \u003cstrong\u003eContainer Network Interfaces (CNIs)\u003c/strong\u003e and had a first glance at how \u003cstrong\u003eeBPF\u003c/strong\u003e is changing networking within Azure Kubernetes Service (AKS). If you haven’t had a chance to read it yet, you might want to catch up \u003ca href=\"http://jurgenallewijn.nl\"\u003ehere\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eIn this episode, we’re diving even deeper into the labyrinth of Kubernetes by shining a light on a topic that’s both critical and fascinating: Observability!\u003c/p\u003e","title":"Azure Kubernetes Chronicles part 3"},{"content":"Azure Kubernetes Chronicles Networking using eBPF Welcome back to the Azure Kubernetes Chronicles, a blog series dedicated to unraveling the complexities of Kubernetes on Azure. In the first edition, we explored the fundamentals of the Container Network Interface (CNI), its role in Kubernetes networking, and how it powers communication within clusters. Building on that foundation, this next episode delves into the transformative impact of eBPF (extended Berkeley Packet Filter) when integrated with CNI. By combining these technologies, we unlock new possibilities for enhancing networking, security, and observability in Kubernetes environments, particularly on Azure Kubernetes Service (AKS).\nIn this blog post, we’ll examine how eBPF elevates the capabilities of CNI, empowering Azure Kubernetes Service users to build secure, high-performing, and observable clusters. Let’s dive into the synergy between CNI and eBPF and see how this duo is shaping the future of Kubernetes networking.\neBPF in Kubernetes environments Managing networking, security, and observability at scale can be challenging. eBPF (extended Berkeley Packet Filter) provides a powerful way to solve these challenges by enabling efficient, safe, and customizable kernel-level programmability. With tools like Cilium or Calico, eBPF has become a critical technology for improving performance and security in Kubernetes clusters, including those running on managed services like Azure Kubernetes Service (AKS).\nWhat is eBPF? eBPF, short for extended Berkeley Packet Filter, is a powerful technology that allows you to run sandboxed programs directly within the Linux kernel, without having to change kernel source code or load custom kernel modules. Originally created for packet filtering, eBPF has evolved into a versatile framework for safely extending the kernel’s capabilities. It can be used to instrument the kernel to gather detailed metrics, observe network traffic, enhance security, and even modify system behaviour on the fly.\nThe key feature of eBPF is its ability to run custom code within the kernel in a secure and efficient manner. This code is verified before execution to ensure it won’t crash the system or harm kernel stability. The power of eBPF lies in its ability to interact with the kernel, allowing developers to tap into the operating system and get insights that would be otherwise challenging to obtain.\nHow eBPF Works eBPF programs are loaded into the kernel from user space, where they can attach to specific events such as system calls, kernel functions, or network packets. Once attached, these programs can observe, measure, or alter how the system behaves. The kernel ensures that the eBPF programs are safe to run through a verification process, which prevents bugs or vulnerabilities from being introduced.\nFor example, an eBPF program can be used to monitor every file open operation across the system, providing insights into what files are being accessed and by which processes. Or it can attach to network events to analyze traffic patterns in real time without having to use separate tools or network taps.\nBenefits of eBPF Performance and Efficiency Since eBPF programs run inside the kernel, they eliminate the need for user-space context switches, which can be a source of significant overhead in traditional monitoring and tracing tools. This makes eBPF ideal for scenarios where high performance is required, such as observability and network filtering.\nFlexibility eBPF provides unparalleled flexibility to developers and operators. Instead of relying on static kernel modules or third-party tools, you can write custom programs to collect the data you need or modify system behaviour to suit specific requirements. This flexibility allows for more creative and responsive solutions to complex performance and security problems.\nUnified Tooling for Observability, Security, and Networking eBPF has blurred the lines between different domains of system management. Observability, security, and networking can now all be addressed using the same tooling. Tools like Cilium, bpftrace, and Falco leverage eBPF to provide powerful capabilities, from enforcing network policies to gathering in-depth system metrics.\nReal-time Insights eBPF allows you to get real-time insights into your system — whether it’s monitoring latency, tracking packet drops, or gathering metrics on CPU utilization. Because eBPF programs run directly in the kernel, they have immediate access to the information you need, providing insights as soon as an event occurs.\nComparison between different network options Key Points for Kubenet\nvirtual ethernet (veth) Pair \u0026amp; Linux Bridge: Each pod gets a veth pair connecting its network namespace to a Linux bridge on the host. NAT and Routing: Kubenet sets up simple NAT (using iptables) and routing rules so that pod IPs (typically from a separate CIDR) can reach and be reached by the external network. Simplicity: It is designed for basic networking requirements, making it easier to configure but less feature-rich than advanced solutions. Key Points for CNI without eBPF\nveth Pair: Provides connectivity from the container to the host. Traditional Processing: Packet handling is done in the kernel via standard components, possibly with extra context switches and userspace involvement. Key Points for CNI with eBPF\nIn-Kernel Processing: eBPF programs attached at hooks like XDP or TC handle packet manipulation directly in the kernel. Lower Overhead: Eliminates many of the extra steps in traditional processing, leading to lower latency and higher throughput. Dynamic and Programmable: eBPF can be updated or reprogrammed at runtime without service interruptions. Microsoft Azure Kubernetes Service and eBPF Azure Kubernetes Service (AKS) integrates eBPF capabilities to enhance observability, networking, and security within Kubernetes environments. Microsoft, through tools like Cilium, enables eBPF-powered networking within AKS, offering improved performance, visibility, and control.\nMicrosoft offers two options to install a CNI with eBPF support. The first is called BYOCNI. This option gives all the freedom to configure the CNI as you wish. Downside is that it doesn’t fully integrate with the Azure stack and Microsoft is not responsible for supporting and maintaining the implementation. This requires more in depth knowledge of the CNI. To make use of this option first install AKS with the — network-plugin parameter with the parameter value of none. After the AKS installation the nodes will have a not ready status. At this the moment Cilium or Calico or any other CNI can be installed and configured.\nThe second option is Azure CNI powered by Cilium. With this option, AKS manages the configuration of Cilium. This option can support most use cases however there are some limitations. For instance this option doesn’t support Windows nodes and Cilium L7 policy enforcement is disabled.\nAzure CNI Powered by Cilium can be deployed using two different methods for assigning pod IPs:\nAssign IP addresses from an overlay network (similar to Azure CNI Overlay mode) Assign IP addresses from a virtual network (similar to existing Azure CNI with Dynamic Pod IP Assignment) Pro’s and cons of the two scenario’s can be found here. For this blog the virtual network scenario was used.\nBefore we are able deploy anything on Azure we need create a resource group at the location where we want to deploy the cluster.\naz group create — name AKS-blog — location westeurope\nCreate the Resource Group az group create --name AKS-blog --location westeurope Create a Virtual Network and Subnets az network vnet create \\ --resource-group AKS-blog \\ --location westeurope \\ --name vnet-aks \\ --address-prefixes 10.0.0.0/8 -o none az network vnet subnet create \\ --resource-group AKS-blog \\ --vnet-name vnet-aks \\ --name nodesubnet \\ --address-prefixes 10.240.0.0/16 -o none az network vnet subnet create \\ --resource-group AKS-blog \\ --vnet-name vnet-aks \\ --name podsubnet \\ --address-prefixes 10.241.0.0/16 -o none Deploy the AKS Cluster with Cilium az aks create \\ --name AKS-blog-cluster \\ --resource-group AKS-blog \\ --location westeurope \\ --max-pods 250 \\ --network-plugin azure \\ --vnet-subnet-id /subscriptions/c9465047-a812-42e7-a53b-739940940898/resourceGroups/AKS-blog/providers/Microsoft.Network/virtualNetworks/vnet-aks/subnets/nodesubnet \\ --pod-subnet-id /subscriptions/c9465047-a812-42e7-a53b-739940940898/resourceGroups/AKS-blog/providers/Microsoft.Network/virtualNetworks/vnet-aks/subnets/podsubnet \\ --network-dataplane cilium \\ --generate-ssh-keys Verify the Cluster Get credentials and check the cluster status:\naz aks get-credentials --name AKS-blog-cluster --resource-group AKS-blog kubectl get nodes Cilium provides a cli which you can use among other things for checking the health status of the Cilium installation within the cluster.\nTo make the impact of Cilium visible, the demo application Online Boutique is used. First step is cloning the repository containing the scripts.\ngit clone https://github.com/GoogleCloudPlatform/microservices-demo.git Next is the deployment of the script.\nkubectl apply -f ./release/kubernetes-manifests.yaml After the installation is completed, check if all pods are running.\nBy implementing a policy we can restrict traffic between the different parts of the application.\nUsing Cilium, we can enforce granular network policies to control pod communication.\nRestrict Traffic to the Checkout Service apiVersion: \u0026#34;cilium.io/v2\u0026#34; kind: CiliumNetworkPolicy metadata: name: \u0026#34;allow-frontend-to-checkout\u0026#34; spec: endpointSelector: matchLabels: app: checkoutservice ingress: - fromEndpoints: - matchLabels: app: frontend Scope of the Policy: The endpointSelector field in the Cilium policy determines which pods the policy applies to. Specifically, this selector matches pods labeled with app: checkout service. As a result, all pods with this label are considered the “protected” endpoints. The policy effectively wraps these pods in a security boundary, ensuring that any incoming traffic must comply with the defined rules.\nIngress Rules: The ingress section outlines the conditions under which traffic is permitted to reach the selected pods (checkout service). In this case, the policy explicitly allows traffic only from pods that carry the label app: frontend. This condition ensures that only requests originating from these specific pods are authorized to communicate with checkout service.\nAllowed Traffic Sources: The fromEndpoints key within the policy further narrows down allowed traffic by specifying the exact source pods. It filters based on labels, matching only those pods labeled app: frontend. This tightens access control, ensuring that traffic from any other pod — even if it exists within the same namespace or cluster — is denied.\nWhat Happens After Applying the Policy\nAllowed Connections: Pods labeled with app: frontend are permitted to send traffic to checkoutservice without any restrictions. Communication flows smoothly for these pods.\nBlocked Connections: All other pods, including those with incorrect labels, no labels, or entirely different labels (e.g., app: analytics or app: user-service), are denied access to checkout service. Their connection attempts are dropped without any response.\nWhitelist Model Enforcement: This policy enforces a strict whitelist approach, where only explicitly permitted traffic is allowed, blocking all other connections by default.\nTo simulate blocked or allowed traffic, a simple pod with utilities like curl or wget to test connectivity can be used.\nDeploy a test pod using an image that includes curl, such as curlimages/curl\nkubectl run test-curl --image=curlimages/curl:latest --restart=Never -- sleep 3600 This creates a pod with curl preinstalled. In order to test the connection, the ip-adress of the frontend is needed.\nkubectl get svc -n default Exec into the Pod and Test Connectivity\nkubectl exec -it test-curl -- curl http://20.61.152.55 As seen in the screenshot the curl command fails to connect. To allow the curl command to work a label is added to the pod. If the curl command is repeated after applying the label, the curl command is allowed as shown in the above image.\nBefore applying the Policy:\n• By default, Kubernetes clusters allow unrestricted communication between pods unless a network policy is enforced.\n• Any pod, regardless of its role or labels, could freely access checkoutservice. This permissive behaviour could lead to unintended access, posing security risks.\n• Unauthorized pods or potentially malicious actors within the cluster could exploit this unrestricted access to compromise sensitive operations handled by checkoutservice.\nAfter the applying Policy\nAllowed Traffic: Pods labeled with app: frontend can continue communicating with checkoutservice as expected, with no interruptions.\nBlocked Traffic: Pods without the correct label (e.g., app: analytics or app: user-service), or pods from entirely different namespaces, will fail to connect to checkoutservice. Connection attempts from these sources will result in dropped packets, meaning they won’t even reach the pod.\nIngress Control: The policy is enforced by Cilium using eBPF (extended Berkeley Packet Filter) at the kernel level. This means that every packet destined for checkout service is inspected, and those that fail to meet the criteria (i.e., not originating from app: frontend) are dropped before reaching the pod.\nConclusion As organizations continue to embrace Kubernetes for their containerized workloads, mastering networking becomes crucial for building scalable, secure, and high-performing clusters. This edition of the Azure Kubernetes Chronicles delves into the essential role of the Container Network Interface (CNI) and the transformative capabilities of eBPF. Together, these technologies empower Kubernetes users to achieve efficient pod-to-pod communication, enforce granular network policies, and enhance cluster observability.\nWith Azure Kubernetes Service (AKS) integrating tools like Cilium, businesses can harness the power of eBPF to bring advanced networking, security, and performance enhancements to their clusters. Whether deploying a straightforward overlay network or exploring the flexibility of a virtual network setup, understanding these concepts lays the groundwork for effective Kubernetes management.\nIn this blog, we demonstrated how to deploy an AKS cluster with Cilium, apply a network policy to secure traffic, and use eBPF-powered tools for efficient data flow and protection. These tools not only improve the security posture of your cluster but also simplify complex configurations for better manageability.\nStay tuned for the next edition, where we’ll explore the powerful monitoring and observability capabilities of eBPF, providing deeper insights into cluster operations and performance. With Azure Kubernetes Service and eBPF, the future of Kubernetes networking and observability is not just efficient but transformative.\n","permalink":"https://wolkwacht.nl/posts/azure-kubernetes-chronicles-part-2/","summary":"\u003ch2 id=\"azure-kubernetes-chronicles-networking-usingebpf\"\u003eAzure Kubernetes Chronicles Networking using eBPF\u003c/h2\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*zKH6hV5Ff_ztvKU1V53HNw.png\"\u003e\u003c/p\u003e\n\u003cp\u003eWelcome back to the \u003cem\u003eAzure Kubernetes Chronicles\u003c/em\u003e, a blog series dedicated to unraveling the complexities of Kubernetes on Azure. In the first edition, we explored the fundamentals of the Container Network Interface (CNI), its role in Kubernetes networking, and how it powers communication within clusters. Building on that foundation, this next episode delves into the transformative impact of eBPF (extended Berkeley Packet Filter) when integrated with CNI. By combining these technologies, we unlock new possibilities for enhancing networking, security, and observability in Kubernetes environments, particularly on Azure Kubernetes Service (AKS).\u003c/p\u003e","title":"Azure Kubernetes Chronicles"},{"content":"Azure Kubernetes Chronicles Container Network Interfaces In today’s cloud landscape, Kubernetes has become one of the backbones of modern application development, empowering businesses to deploy, scale, and manage containerized applications effortlessly. But with great power comes great complexity! That’s where Azure Kubernetes Service (AKS) comes into play — a managed Kubernetes service designed to make your container orchestration journey smoother and more productive.\nThis blog series, Azure Kubernetes Chronicles, is here for platform engineers, CloudOps professionals and Cloud architects who are beginning their Kubernetes journey or looking to streamline their operations to guide you through the essentials and beyond, as we dive into some of the tools, features, and best practices for leveraging Azure Kubernetes to its fullest potential.In this first part the focus lies on the Container Network Interface (CNI).\nWhat is a CNI? The CNI (Container Network Interface) is a cloud-native standard for managing container networking. It provides a specification and framework for configuring network interfaces in Linux containers, ensuring that containers can communicate with each other, services within the cluster, and external systems.\nCNI is not specific to Kubernetes — it is a general-purpose solution — but it has become a cornerstone of Kubernetes networking. Kubernetes relies on the CNI standard to abstract network setup and management.\nIn Kubernetes, each Pod (the smallest deployable unit) operates in its own network namespace, requiring:\nAn IP address to communicate with other pods and services. Routing rules to enable East-West (intra-cluster) and North-South (external) traffic. The CNI standard ensures that these networking needs are consistently met across different infrastructure providers and environments.\nHow does CNI framework integrates with Kubernetes: The CNI Specification:\nThe CNI defines a simple contract between the container runtime (e.g., Docker, containerd) and a network plugin. It uses two primary operations: Add: Attach a network interface to a container when it is created. Delete: Remove the interface when the container is destroyed. The Kubernetes Networking Model:\nKubernetes imposes certain networking requirements, and CNI plugins help satisfy them:\nPod-to-Pod Communication: Every pod should be able to reach every other pod without NAT. Pod-to-Service Communication: Pods must be able to connect to Kubernetes services. Cluster External Access: Pods and services must be accessible from outside the cluster. Interaction with kubelet\nWhen Kubernetes creates a pod, the kubelet invokes the CNI plugin to configure networking. The plugin handles IP allocation, DNS configuration, and routing setup. How CNIs Handle Key Networking Aspects IP Address Management:\nCNI’s assign IP addresses to pods using either:\nHost-local methods (local IP pools). IPAM plugins (IP Address Management systems). Routing:\nCNIs configure routing tables to ensure traffic flows correctly between pods, nodes, and external systems.\nNetwork Policies\nKubernetes Network Policies define rules for pod communication. CNI plugins (e.g., Calico, Cilium) enforce these policies at runtime. Service Discovery and DNS\nCNIs integrate with Kubernetes CoreDNS to handle service discovery for pods. CNI Plugins Different CNI plugins are available, catering to various use cases. Here’s an in-depth look at popular options:\nCalico Type: Layer 3 networking and network policy engine.\nFeatures:\nSupports advanced network security policies. Can operate in both overlay and non-overlay modes. Integrates with eBPF for performance optimization. Use Case: Enterprises requiring fine-grained network policies and scalability.\nFlannel Type: Simple overlay network.\nFeatures:\nLightweight and easy to set up. Uses VXLAN for encapsulation. Minimalist compared to other CNIs. Use Case: Small-to-medium-sized clusters with straightforward networking needs.\nCilium Type: Layer 3 networking with eBPF.\nFeatures:\nHigh observability and security. Granular traffic control using eBPF. Advanced load balancing and service mesh integrations. Use Case: Modern microservices architectures prioritising performance and security.\nAzure Kubernetes Services and CNI Microsoft offers different CNI option to suppor the Azure Kubernetes Service: Azure CNI, Azure CNI Overlay, Azure CNI Powered by Cilium and BYOCNI (Bring Your Own CNI). In this post, we’ll explore: Azure CNI, Azure CNI Overlay, and Kubenet. We will dive deeper in Azure CNI Powered by Cilium in our next post.\nAzure CNI Azure CNI (Container Networking Interface) is the default networking option for AKS. It provides a seamless integration with Azure Virtual Network (VNet), ensuring each pod gets its own IP address from the subnet associated with the AKS cluster.\nKey Features\nFull VNet Integration: Pods are directly assigned IPs from the Azure VNet, allowing native communication with other Azure resources. Security and Compliance: Azure policies and NSGs (Network Security Groups) can be applied at the pod level for fine-grained control. Scalability: Ideal for workloads requiring high throughput and low latency. Use Cases\nEnterprises with stringent compliance or security requirements. Workloads that need direct integration with other Azure services like Azure SQL Database or Storage Accounts. Scenarios requiring large-scale, high-performance applications. Challenges\nIP Exhaustion: Each pod consumes an IP address from the VNet, which can lead to IP exhaustion in large clusters. Complexity in Subnet Management: Requires careful planning of subnet sizes, especially in high-density environments. Azure CNI Overlay Azure CNI Overlay is a newer addition designed to address the limitations of Azure CNI, particularly around IP exhaustion. Instead of assigning each pod an IP directly from the VNet, it uses an overlay network to assign pod IPs.\nKey Features\nOverlay Networking: Pods are assigned IPs from a different address space (an internal overlay network), conserving VNet IP addresses. Efficient Resource Utilization: Supports larger cluster sizes without requiring extensive subnet planning. High Performance: Optimized for low latency and high throughput workloads. Use Cases\nScenarios where subnet IP exhaustion is a concern. High-density workloads with a need for more pods per node. Teams looking for simplified IP address management. Challenges\nOverlay Overhead: Introduces slight overhead due to encapsulation, which may marginally affect network latency. Limited Adoption: As a newer option, it may require additional testing for niche use cases. Restrictions\nYou can’t use Application Gateway as an Ingress Controller (AGIC) for an Overlay cluster. You can’t use Application Gateway for Containers for an Overlay cluster. Virtual Machine Availability Sets (VMAS) aren’t supported for Overlay. Kubenet Kubenet is a basic CNI option that relies on Kubernetes’ built-in network components. It configures pod networking using NAT (Network Address Translation) and route tables.\nKey Features\nSimple Architecture: Minimal configuration and dependencies. Low IP Consumption: Pods communicate using NAT, which doesn’t require assigning individual IPs from the VNet. Cost Efficiency: Suitable for smaller clusters and test environments. Use Cases\nDevelopment or testing environments with limited networking requirements. Scenarios with small-scale, low-performance workloads. Challenges\nLimited Integration: No direct integration with Azure services, as pods don’t get their own VNet IPs. Manual Route Management: Requires explicit route table configuration for pod communication. Unlike Azure CNI clusters, multiple kubenet clusters can’t share a subnet. AKS doesn’t apply Network Security Groups (NSGs) to its subnet and doesn’t modify any of the NSGs associated with that subnet. Scalability Constraints: Less suitable for large-scale or complex applications. Comparison of CNI Options View code on GitHub Gist\nKey Considerations When Choosing a CNI When selecting a CNI for your AKS cluster, consider the following factors:\nCluster Size and Density: Azure CNI Overlay is a better fit for high-density clusters, while Azure CNI suits mid-sized clusters with integration needs. Integration with Azure Resources: If direct communication with Azure services is critical, Azure CNI is the preferred option. IP Management: Azure CNI Overlay is ideal for scenarios where IP exhaustion is a concern. Performance Requirements: For workloads requiring high throughput and low latency, Azure CNI and Azure CNI Overlay are better suited than Kubenet. Setting Up and Testing Your CNI Configuration Introduction to the Demo In this section, we’ll walk through a practical demonstration of setting up an AKS cluster with Azure CNI and testing its networking configuration. This hands-on approach will help strengthen your understanding of CNIs and their integration with Kubernetes on Azure.\nCreate the Resource Group\naz group create --name AKS-blog-rg --location westeurope Create the cluster\nUse the parameter— network-plugin azure to create the cluster with the network configured. After deployment get the credentials.\naz aks create \\ --resource-group AKS-blog-rg \\ --name aks-cluster \\ --network-plugin azure \\ --generate-ssh-keys az aks get-credentials --resource-group AKS-blog-rg --name aks-cluster Now we have downloaded the credentials, we can check the health status of the nodes, pods and services that are installed.\nDeploy a Sample Application\nDeploy a simple application in your cluster for testing, use a utility pod that has them pre-installed and verify that the pod is running.\nkubectl run busybox --image=busybox --restart=Never --command -- sleep 3600 kubectl get pods -n default Test Pod-to-Pod Communication\nTo verify that pods can communicate with each other:\nDeploy a service and expose the service on port 80\nkubectl create deployment nginx --image=nginx kubectl expose deployment nginx --port=80 --target-port=80 --name=nginx-service --type=ClusterIP Testing with wget\nRetrieve the ClusterIP of the service\nkubectl get svc nginx Note the ip adress of the service (e.g. 10.0.0.150).\nUse wget inside the busybox pod to send a request. Replace \u0026lt;ClusterIP\u0026gt; with the adress from the previous step. You should see the default NGINX welcome page HTML content.\nkubectl exec -it busybox -- sh wget -qO- http://\u0026lt;ClusterIP\u0026gt;:80 Validate External Access\nTo validate if your application is accessible externally:\nDeploy a sample nginx application if not already done.\nkubectl create deployment nginx --image=nginx Expose the deployment using a LoadBalancer service.\nkubectl expose deployment nginx --port=80 --target-port=80 --name=nginx-service --type=LoadBalancer Retrieve the external IP assigned by the LoadBalancer\nkubectl get service nginx-service Access the application in your browser or using wget\nwget http://\u0026lt;external-ip\u0026gt; -q -O - Replace \u0026lt;external-ip\u0026gt; with the external IP obtained in the previous step. You should see the application’s response, indicating that external access is working correctly.\nConclusion Container Network Interfaces (CNIs) form the backbone of networking in Kubernetes clusters, ensuring seamless communication within and beyond the cluster. Whether you prioritize performance, scalability, or simplicity, Azure Kubernetes Service offers a range of CNI options tailored to meet diverse workload needs.\nIn the next installment of the Azure Kubernetes Chronicles, we’ll explore the power of eBPF in relation to networking. Stay tuned!\n","permalink":"https://wolkwacht.nl/posts/azure-kubernetes-chronicles-part-1/","summary":"\u003ch2 id=\"azure-kubernetes-chronicles-container-network-interfaces\"\u003e\u003cstrong\u003eAzure Kubernetes Chronicles\u003c/strong\u003e Container Network Interfaces\u003c/h2\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*0Jz4Ue2j1MYsupA0BSIsrw.png\"\u003e\u003c/p\u003e\n\u003cp\u003eIn today’s cloud landscape, Kubernetes has become one of the backbones of modern application development, empowering businesses to deploy, scale, and manage containerized applications effortlessly. But with great power comes great complexity! That’s where \u003ca href=\"https://learn.microsoft.com/en-us/azure/aks/what-is-aks?WT.mc_id=MVP_323616\"\u003eAzure Kubernetes Service (AKS)\u003c/a\u003e comes into play — a managed Kubernetes service designed to make your container orchestration journey smoother and more productive.\u003c/p\u003e\n\u003cp\u003eThis blog series, \u003cem\u003eAzure Kubernetes Chronicles\u003c/em\u003e, is here for platform engineers, CloudOps professionals and Cloud architects who are beginning their Kubernetes journey or looking to streamline their operations to guide you through the essentials and beyond, as we dive into some of the tools, features, and best practices for leveraging Azure Kubernetes to its fullest potential.In this first part the focus lies on the Container Network Interface (CNI).\u003c/p\u003e","title":"Azure Kubernetes Chronicles"},{"content":"Challenging the Cloud Cost-Saving Myth: A Closer Look at Azure and cost control Created by Microsoft Designer\nIntroduction In today’s rapidly evolving digital landscape, the migration to cloud computing is often heralded as a golden pathway to significant cost savings, increased efficiency, and unparalleled scalability. This popular belief has led countless organisations to embark on cloud migration journeys with the expectation of automatic reductions in IT expenditures. However, the reality of cloud computing costs is far more complex, asking for a closer examination to understand the true impact on cost.\nThe goal of this blog is not to advise organisations against adopting cloud services but to equip them with the knowledge and strategies needed to make informed decisions. By understanding the different layers of cloud cost management and the specific challenges and opportunities presented by the public cloud, businesses can leverage the cloud to not only achieve technological and operational benefits but also realize potential cost savings through strategic planning and ongoing optimization.\nIn challenging the cost-saving myth, the goal is to drive a smarter, more layered dialogue about cloud computing — one that acknowledges the potential for cost savings while recognising the complexities involved in achieving them. This balanced perspective is crucial for organisations looking to make the most of their cloud investments. In this blog the focus is on Microsoft Azure but most of the recommendations also apply to the other hyperscalers.\nThe Cloud Cost-Saving Myth The belief that migrating to the cloud inherently leads to cost reduction is widespread, but where did this perception come from? Many organisations initially see the cloud as a way to eliminate the need for physical infrastructure, reduce maintenance efforts, and benefit from scalability. The cloud promises flexibility — the ability to scale resources up or down based on demand — which can indeed result in more efficient resource usage and, ultimately, cost savings. However, the assumption that these benefits will automatically translate into lower overall IT costs is often misguided.\nThe reality is that cloud cost management requires a comprehensive understanding of both the technical and financial aspects of cloud services. For instance, while organisations can save on hardware costs, they often face increased costs in other areas, such as data transfer, storage, and the need for skilled personnel to manage cloud infrastructure effectively. These factors contribute to the complexity of cloud cost management, and without proper planning, cloud costs can easily spiral out of control.\nCloud-Agnostic Challenges to Cost Savings Hidden Costs of Cloud Migration: While the cloud provides an array of benefits, the migration process itself often brings unexpected expenses. Data transfer fees, complex pricing models, and costs related to re-architecting applications to suit a cloud environment are frequently underestimated or overlooked. For example, data egress fees, which are incurred when moving data out of the cloud, can quickly add up if not carefully managed. Additionally, applications often need to be refactored or re-architected to take advantage of cloud-native features, which requires development effort and expertise.\nManagement and Operational Overheads: The successful management of cloud resources requires specialized knowledge and experience. Cloud environments demand continuous monitoring to ensure that resources are used efficiently, and this involves additional costs in the form of skilled cloud engineers or managed service providers. Without proper cloud governance, organisations may find themselves facing increased operational costs due to underutilization or overprovisioning of cloud services. Tools like Azure Monitor and Azure Automation can help mitigate these challenges, but they require expertise to configure and maintain effectively.\nCompliance and Security Costs: Ensuring data security and maintaining compliance with various regulations can also drive up costs. These investments are necessary, especially when dealing with sensitive data, but they must be factored into the overall cost analysis. Cloud platforms like Azure offer security services such as Azure Security Center and Azure Sentinel, which provide enhanced security monitoring and threat detection. However, using these services comes with additional costs, and organizations must also consider the expense of maintaining compliance with industry standards like GDPR, HIPAA , or PCI DSS.\nAzure-Specific Considerations Azure Pricing Model Complexity: Azure, like other major cloud platforms, comes with a complex pricing structure. Service tiers, region-based pricing, and options like reserved instances all add layers of complexity that need to be navigated carefully to avoid unnecessary costs. Azure services are billed based on usage metrics such as compute hours, storage consumption, and data transfers, and understanding these pricing variables is crucial. For example, Azure Virtual Machines pricing varies based on machine size, region, current contracts and reserved instance options. Azure also offers Spot VM’s, which can provide significant savings but come with the risk of being evicted if capacity demands increase.\nOptimization Tools and Strategies on Azure: Azure offers several tools, such as Azure Cost Management, Azure Advisor, and Azure Resource Graph, to help monitor and optimize cloud expenditures. Azure Cost Management provides insights into spending patterns, and Azure Advisor offers recommendations for optimizing costs by identifying underutilized resources or suggesting cheaper service options. However, effectively using these tools requires a learning curve and continuous effort. For example, tagging resources properly is essential for gaining visibility into resource consumption across different departments, and this requires establishing proper governance and tagging policies. Learn more about tagging strategies on Azure.\nCase Studies: There are numerous real-world examples of organisations migrating to Azure, expecting cost savings, but discovering that their initial costs actually increased. One example is a company that migrated its on-premises workloads to Azure without optimising them for the cloud. This led to high compute costs because the workloads were not right-sized or scaled appropriately. These cases highlight the importance of strategic planning and a thorough understanding of Azure’s services before migration. Organizations should perform detailed cost modeling and proof-of-concept testing to identify potential cost implications before fully committing to a cloud migration.\nAchieving True Cost Savings on Azure Strategic Migration Planning: To achieve meaningful cost savings, it’s crucial to conduct a thorough assessment before migration. This includes selecting the appropriate Azure services and understanding their pricing models, as well as identifying potential scalability options that align with business needs. For example, leveraging Platform-as-a-Service (PaaS) offerings like Azure App Service can be more cost-effective compared to traditional Infrastructure-as-a-Service (IaaS) Virtual Machines. The migration strategy should also consider the use of Azure Migrate to assess on-premises workloads and identify dependencies, allowing for a more efficient migration process.\nFinOps Practices: FinOps is an emerging financial management practice that aims to bring together financial accountability and cloud spending. Applying FinOps principles to Azure can help organisations maximize the value of their cloud investments by aligning financial processes with cloud operations. FinOps practices include budgeting, forecasting, and tracking cloud expenses in real-time. Azure Cost Management plays a key role in implementing FinOps, providing tools to track spending and allocate costs to different teams or projects. Establishing a culture of cost accountability and empowering engineering teams to take ownership of cloud spending is essential for successful FinOps adoption. Learn more about FinOps.\nOngoing Optimization: Realizing cost savings in the cloud is an ongoing mission. Organisations need to review and adjust their resource usage regularly. Azure’s native optimisation tools are instrumental in this continuous process, ensuring that cloud spending aligns with actual needs. For example, Azure Automation can be used to automatically shut down non-critical VMs during off-hours, reducing costs. Azure Policy can be employed to enforce best practices, such as restricting the deployment of expensive resource types or ensuring that all resources are tagged properly for cost allocation purposes. Regular audits of cloud resources and the use of Azure Reserved Instances for predictable workloads can also lead to substantial cost savings.\nGreen Cloud Computing Green Cloud Computing in Azure: Sustainability Meets Cost Efficiency Green cloud computing focuses on minimising environmental impact while maintaining optimal performance and cost-efficiency. Microsoft Azure, as a leader in this space, combines sustainability with tools that help organisations manage and reduce cloud costs effectively.\nEnergy Efficiency and Cost Reduction Azure’s energy-efficient data centers employ:\nAdvanced Cooling Techniques: Reducing the need for energy-intensive cooling systems, which translates into lower costs. Optimized Hardware Utilization: Ensuring hardware operates at peak efficiency to reduce energy waste and costs. These practices help customers benefit from economies of scale, reducing their overall expenses.\nCarbon-Aware and Cost-Aware Computing Azure’s carbon-aware computing optimizes workloads based on the availability of renewable energy. This not only lowers emissions but also offers cost benefits by utilizing off-peak electricity rates. (Carbon-Aware Computing)\nMoving Beyond the Myth Balanced Approach: A balanced approach to cloud computing recognizes both its strengths and its challenges. Cloud services offer significant technological and operational advantages — such as agility, innovation, and enhanced collaboration — but the promise of reduced IT costs is only achievable with strategic planning and consistent optimization efforts. Organisations should evaluate the Total Cost of Ownership (TCO) of their cloud investments, considering both direct and indirect costs, such as training, compliance, and operational overheads.\nDecision-Making Factors: Organisations should not view cloud adoption solely through the lens of cost savings. Instead, they should also consider factors like improved agility, the potential for innovation, and the ability to gain a competitive edge in the market. Cloud services can accelerate product development cycles, provide access to advanced analytics and AI capabilities, and enable global reach. These benefits often outweigh the pure cost considerations, particularly when organisations leverage cloud-native technologies like Azure Kubernetes Service (AKS), Azure Functions, or Azure Cognitive Services to drive innovation.\nConclusion Challenging the myth of automatic cost savings from cloud migration is crucial for organisations aiming to make informed decisions about their IT strategies. While the cloud offers numerous benefits, including scalability, flexibility, and potential cost efficiencies, these are only realized through careful planning, diligent management, and ongoing optimization.\nOrganisations looking to migrate to the cloud should do so with a critical eye and a comprehensive strategy. By understanding the nuances of cloud cost management, businesses can make cloud investments that not only enhance their operations but also deliver tangible value.\nIf you’re interested in further reading, I have compiled resources on cloud cost optimization and FinOps practices. Additionally, if your business needs support in optimizing your cloud strategy, consider consulting services to help you make the most of your Azure or other cloud investments.\nAzure Cost Management and Billing Documentation Azure Migration Azure Cloud Adoption Framework Azure Total Cost of Ownership (TCO) Calculator Azure FinOps Toolkit FinOps Foundation Resources ","permalink":"https://wolkwacht.nl/posts/challenging-the-cloud-cost-saving-myth/","summary":"\u003ch2 id=\"challenging-the-cloud-cost-saving-myth-a-closer-look-at-azure-and-costcontrol\"\u003eChallenging the Cloud Cost-Saving Myth: A Closer Look at Azure and cost control\u003c/h2\u003e\n\u003cp\u003e\u003cimg alt=\"Created by Microsoft Designer\" loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*V2WpNtNQlUGbkk1LaqgCjQ.jpeg\"\u003e\n\u003cem\u003eCreated by Microsoft Designer\u003c/em\u003e\u003c/p\u003e\n\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003eIn today’s rapidly evolving digital landscape, the migration to cloud computing is often heralded as a golden pathway to significant cost savings, increased efficiency, and unparalleled scalability. This popular belief has led countless organisations to embark on cloud migration journeys with the expectation of automatic reductions in IT expenditures. However, the reality of cloud computing costs is far more complex, asking for a closer examination to understand the true impact on cost.\u003c/p\u003e","title":"Challenging the Cloud Cost-Saving Myth"},{"content":"Zero Trust in the Cloud A Simple Path to Securing Cloud Infrastructure Created by Microsoft Designer\nIntroduction In the world of cybersecurity, traditional network perimeter security models are no longer sufficient to protect against increasingly sophisticated threats. As cloud adoption continues to grow, ensuring robust security measures for both data and resources is critical. Enter Zero Trust — a security framework that shifts the paradigm from implicit trust to “never trust, always verify.” In this blog post, we will explore what Zero Trust is, the pros and cons of this security approach, and how to implement Zero Trust in both Microsoft Azure (Azure) and Amazon Web Services (AWS) environments.\nWhy Zero Trust? The Zero Trust model addresses several key security challenges that organizations face today, including insider threats, phishing attacks, and ransomware. Traditional security models often assume that anything inside the network is trustworthy, which leaves organizations vulnerable to internal threats and sophisticated social engineering attacks. Insider threats can occur when employees misuse their access privileges, either intentionally or unintentionally, to compromise sensitive data. Phishing attacks are also common, exploiting users’ trust to gain access to credentials or sensitive information. Ransomware attacks, meanwhile, have become a major threat, with attackers encrypting valuable data and demanding payment to restore access. By adopting Zero Trust principles, organizations can mitigate these risks by enforcing continuous verification, least privilege access, and monitoring for anomalous activities, ensuring that no one entity is trusted by default.\nUnderstanding Zero Trust Zero Trust is a security concept centered around the idea that no entity — whether inside or outside of an organization’s network — should be trusted by default. Instead, every access request is verified, authenticated, and authorized, regardless of its origin. The Zero Trust model requires continuous monitoring, identity verification, and strict control over access to ensure data and applications remain secure.\nThe National Institute of Standards and Technology (NIST) defines Zero Trust as a collection of concepts and ideas designed to minimize uncertainty in enforcing accurate, least privilege per-request access decisions in information systems and services. At its core, Zero Trust assumes that threats may exist both inside and outside traditional network boundaries, necessitating a “trust nothing, verify everything” approach to security.\nThe Seven Pillars of Zero Trust The Zero Trust framework is built around seven key pillars that help organizations secure their environments:\nUser Identity: Verify and authenticate every user with strong identity and access management (IAM) practices, such as multi-factor authentication (MFA) and conditional access policies, to ensure only authorized users gain access.\nDevice Security: Ensure that devices accessing the network are secure and compliant by implementing device management and endpoint security solutions to maintain the integrity of connected devices.\nNetwork Security: Segment and isolate the network to reduce the attack surface. Use network micro-segmentation and enforce least privilege access to limit lateral movement in case of a breach.\nApplication Security: Secure applications by verifying their integrity, restricting access, and monitoring their behavior. Application-layer controls, such as Web Application Firewalls (WAFs), help protect applications from threats.\nData Security: Classify, encrypt, and protect sensitive data, ensuring that only authorized users and applications can access or modify it. Data protection mechanisms should apply whether data is at rest, in transit, or in use.\nVisibility and Analytics: Continuously monitor activities across the environment to detect anomalies and potential threats. Leverage security analytics and threat intelligence to identify and respond to suspicious activities in real-time.\nAutomation and Orchestration: Automate security responses and policy enforcement to reduce human error and improve the efficiency of security operations. Automated workflows help ensure consistent adherence to security policies across the organization.\nPros and Cons of Zero Trust Like with a lot of things in life also Zero Trust comes with pro’s and con’s. This is a sum up of some of them.\nPros Enhanced Security: By removing implicit trust and enforcing verification for each access request, Zero Trust significantly reduces the attack surface, making it much harder for malicious actors to compromise the system.\nReduced Lateral Movement: Zero Trust controls reduce lateral movement by segmenting access to resources. If an attacker breaches one part of the network, they cannot easily move to other parts of the environment without undergoing strict security checks.\nBetter Compliance: Zero Trust helps organizations meet regulatory requirements such as GDPR, HIPAA, and PCI-DSS by enforcing policies around data access and monitoring user activities. It provides comprehensive visibility and control over data access, which supports compliance.\nAgility: Zero Trust principles align well with modern cloud infrastructure, allowing organizations to dynamically grant access based on identity and context, without relying on traditional static network boundaries.\nCons Complexity: Implementing Zero Trust requires a deep understanding of existing IT infrastructure, cloud services, user identities, and access policies. Organizations may need to re-engineer their networks and security architecture to align with Zero Trust principles.\nOperational Overhead: Zero Trust often requires additional layers of verification, leading to increased management and maintenance efforts. The constant monitoring and verification could also impact user experience, particularly for remote or high-volume access environments.\nInitial Investment: Implementing a Zero Trust framework may involve additional costs, such as identity management tools, multi-factor authentication (MFA), endpoint security solutions, and network micro-segmentation.\nImplement Zero Trust in the cloud environment All the major hyperscalers have solutions and tools which will help with the implementation of Zero Trust.\nZero Trust on Azure Microsoft Azure offers an extensive set of tools and services to help organizations implement Zero Trust across their cloud infrastructure. The Zero Trust implementation in Azure is built around six pillars: identity, endpoints, data, applications, network, and infrastructure.\nFor more information, visit the official Azure Zero Trust page.\nIdentity and Access Management (IAM): Azure Active Directory (Azure AD) forms the backbone of identity management in Azure’s Zero Trust model. By using features like Azure AD Conditional Access, Multi-Factor Authentication (MFA), and Identity Protection, Azure enables organizations to enforce strict access policies and continuously monitor identity behavior.\nConditional Access Policies: Conditional Access policies help to enforce adaptive access control based on the user’s identity, device state, location, and risk level. For instance, administrators can configure policies to allow access only from trusted locations or compliant devices, ensuring only verified entities can connect.\nNetwork Segmentation: Azure Virtual Network (VNet) and Network Security Groups (NSGs) provide segmentation capabilities to create isolated network segments. This helps limit the lateral movement of potential attackers, as each network segment is restricted by rules that grant the least privilege access.\nEndpoint Protection: Azure integrates with Microsoft Defender for Endpoint to help secure end-user devices and servers. Defender provides proactive threat protection, endpoint detection and response (EDR), and application control to identify suspicious activities and prevent breaches.\nApplication and Data Security: Azure ensures data security through services like Azure Information Protection, which classifies and encrypts sensitive data, and Azure Key Vault, which manages secrets, keys, and certificates securely.\nZero Trust on AWS Amazon Web Services (AWS) also offers a range of tools and services that align with the Zero Trust framework, helping organizations protect their cloud resources and data.\nFor more information, visit the official AWS Zero Trust page\nIdentity and Access Management (IAM): AWS IAM enables organizations to create and manage AWS users and groups securely. It allows for implementing the principle of least privilege by assigning granular access policies to different services and resources. AWS Single Sign-On (SSO) can also be used to enforce centralized identity control.\nMulti-Factor Authentication (MFA): AWS provides MFA to add an additional layer of security to user accounts and privileged operations. MFA ensures that users verify their identity using something they know (password) and something they have (a hardware or virtual MFA device).\nNetwork Micro-Segmentation: AWS supports network segmentation through services like Virtual Private Cloud (VPC) and Security Groups. VPCs can be used to create isolated environments, while Security Groups can be leveraged to set granular inbound and outbound traffic controls, thereby reducing the attack surface.\nAWS PrivateLink and VPC Endpoint: AWS PrivateLink allows you to securely access services and applications over the AWS network without exposing data to the public internet. This helps enforce a Zero Trust approach by minimizing external exposure.\nMonitoring and Logging: AWS CloudTrail and Amazon GuardDuty are essential for monitoring user activities and identifying suspicious behavior. CloudTrail captures API activity, while GuardDuty leverages machine learning to detect potential threats and anomalies, providing visibility into access patterns and reducing the risk of breaches.\nConclusion Zero Trust is an essential security model for organizations adopting cloud services, as it assumes that threats exist both within and outside of an organization’s environment. By leveraging Zero Trust, organizations can minimize security risks, enhance data protection, and ensure compliance with stringent regulatory standards. Although Zero Trust implementation can be complex and requires a careful alignment of technology and processes, hyperscalers like Azure and AWS offer a range of tools and services that facilitate the transition to this modern security framework.\nBy implementing Zero Trust in cloud environments, organizations can create a strong security posture that protects against modern cyber threats, reduces the risk of data breaches, and helps maintain control over sensitive data. Whether you’re planning to secure your cloud infrastructure, meet compliance requirements, or prevent malicious actors from exploiting vulnerabilities, Zero Trust is a forward-looking approach that can address the dynamic challenges of cloud security.\n","permalink":"https://wolkwacht.nl/posts/zero-trust-in-the-cloud/","summary":"\u003ch2 id=\"zero-trust-in-thecloud\"\u003eZero Trust in the Cloud\u003c/h2\u003e\n\u003ch3 id=\"a-simple-path-to-securing-cloud-infrastructure\"\u003eA Simple Path to Securing Cloud Infrastructure\u003c/h3\u003e\n\u003cp\u003e\u003cimg alt=\"Created by Microsoft Designer\" loading=\"lazy\" src=\"https://cdn-images-1.medium.com/max/800/1*T5Ar0cKlBnXq_aTnkH8XTw.png\"\u003e\n\u003cem\u003eCreated by Microsoft Designer\u003c/em\u003e\u003c/p\u003e\n\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003eIn the world of cybersecurity, traditional network perimeter security models are no longer sufficient to protect against increasingly sophisticated threats. As cloud adoption continues to grow, ensuring robust security measures for both data and resources is critical. Enter Zero Trust — a security framework that shifts the paradigm from implicit trust to “never trust, always verify.” In this blog post, we will explore what Zero Trust is, the pros and cons of this security approach, and how to implement Zero Trust in both Microsoft Azure (Azure) and Amazon Web Services (AWS) environments.\u003c/p\u003e","title":"Zero Trust in the Cloud"},{"content":"In 2023 a lot has happened. The company I work for Luminis became part of Yuma. I achieved several certifications. Next to attending conferences I also was a speaker at several conferences. This year I was for the first time a guest in a podcast. In the technology space also a lot happened. With ChatGPT, Bard, Dall-E, Copilot and others AI became a very hot topic. There was also a lot of attention for sustainability, how can IT help in reducing the carbon foot print.\nCertifications I started the year with KCNA: Kubernetes and Cloud Native Associate. After that I focussed on the security oriented Microsoft Azure exams. I passed the following 2 exams:\nMicrosoft Certified: Azure Security Engineer Associate SC-100: Microsoft Cybersecurity Architect By passing those two exams I also achieved the status of Microsoft Certified: Cybersecurity Architect Expert. Later in the year I also renewed both of them as Microsoft certifications are valid for a year and a half before expiring it is possible to renew them. I also renewed two other Microsoft exams, Microsoft Certified: Azure Administrator Associate and Microsoft Certified: Azure Solutions Architect Expert. After the Microsoft exams, I followed the cloud security path. I passed the ISC2 Certified in Cybersecurity (CC).\nConferences and meetups In 2023, next to attending several conferences and meetups I also was a speaker at some of them. For me the conference season started with the Kubernetes Community Days Amsterdam on the 23rd and the 24th of February. At this conference I gave, together with Dinant Paardenkooper, a workshop Kubernetes 201 and also had talk AKS unlighted, but what about Security and Multi-tenancy?. Read this blog post for a recap of the conference. On the 8th of March, I spoke at the meetup session The Azure journey of Gemeente Amsterdam, organized by the Dutch Azure Meetup, about the new data platform that the municipality of Amsterdam is building on Azure. In April I went, from the 18th to the 21st, to The Azure Day on Kubernetes and KubeCon Europe. For more details on this event, you can read my blogpost. On the 1st of June I attended the AWS Summit in Amsterdam. Next event in June were the Devopsdays Amsterdam, where I gave the sessions AKS unlighted, but what about Compliancy, cost and Multi-tenancy? together with Dinant Paardenkooper. Next to speaker and attendee I also manned the Luminis booth. Also my colleague Lutske de Leeuw gave a workshop Machine Learning 101. You can read more about my experience on the Devopsdays in this blog post. During the Microsoft event, App Modernization Customer Success Stories Conference - Do More With Less!, I gave together with Dinant Paardenkooper the presentation Gemeente Amsterdam - \u0026ldquo;AKS unlighted, but what about Compliancy, cost and Multi-tenancy?\u0026rdquo; on the 26th of June. On the 20th of September I attended Edgecase 2023. Very interesting conference around the usecases of Kubernetes in the edge. I really liked the customer case of Chick-fil-a. On the 12th of October, I went to meetup session Empowering Sustainability and AI Advancements with Azure OpenAI Service at the Luminis office organized by the Dutch Azure Meetup.\nPodcast and webinar This year was also the first year that I participated in two podcasts. The first podcast was a Luminis Tech Talk which was hosted by Nico Krijnen and together with my colleague Edi Recica we talked about Cloud Security. I also participated in a podcast of the Nederlandse Kubernetes Podcast together with Dinant Paardenkooper. This podcast is at this moment not yet online.\nFor Yuma the company which Luminis I gave a webinar Container Security. In this webinar I explained what container security is and why it is important.\n2024 In 2024 my goal is to focus more on cloud security. I have started with my CCSP certification. I have no concrete ideas about talks or sessions I want to give this year but I have some ideas. My plan is also blog more frequently this year.\n","permalink":"https://wolkwacht.nl/posts/recapandpreview/","summary":"\u003cp\u003eIn 2023 a lot has happened. The company I work for \u003ca href=\"https://www.luminis.eu\"\u003eLuminis\u003c/a\u003e became part of \u003ca href=\"https://weareyuma.com/\"\u003eYuma\u003c/a\u003e. I achieved several certifications. Next to attending conferences I also was a speaker at several conferences. This year I was for the first time a guest in a podcast. In the technology space also a lot happened. With ChatGPT, Bard, Dall-E, Copilot and others AI became a very hot topic. There was also a lot of attention for sustainability, how can IT help in reducing the carbon foot print.\u003c/p\u003e","title":"Recap of 2023"},{"content":"Introduction In the rapidly changing cloud landscape, security and privacy are key concerns for businesses and individuals alike. Confidential Compute turns out to be a possible solution, ensuring sensitive data remains private and secure even in shared computing environments. In this blog post, I want to explain the concept of Confidential Compute in general and explore the specifics of how Azure has implemented confidential compute and how this can help with cloud sovereignty.\nUnderstanding Confidential Compute NIST definition for confidential compute: Hardware-enabled features that isolate and process encrypted data in memory so that the data is at less risk of exposure and compromise from concurrent workloads or the underlying system and platform. Source: NISTIR 8320\nConfidential computing is a set of technologies that protect data in use, even when it is being processed. In traditional computing, data is typically encrypted when it is at rest (stored on disk) and in transit (moving across a network), but it is often decrypted and exposed in memory when it is being processed by the CPU. Confidential computing aims to address this vulnerability by providing a secure enclave for sensitive data during processing. This is achieved through the use of Trusted Execution Environments (TEEs), which are isolated areas of a computer\u0026rsquo;s memory that are designed to protect against unauthorized access. These enclaves allow for the execution of code and processing of data in a secure environment, shielding it from the rest of the system and even from privileged software.\nAMD Secure Encrypted Virtualization (SEV) is a technology developed by AMD that provides hardware-based memory encryption for virtual machines (VMs). SEV uses a unique key to encrypt the memory of each VM, which is managed by the AMD Secure Processor. This helps to isolate guests and the hypervisor from one another, thereby providing an additional layer of security. SEV requires enablement in the guest operating system and hypervisor.\nIntel SGX (Software Guard Extensions) and AMD SEV (Secure Encrypted Virtualization) are examples of hardware-based technologies that enable confidential computing. These technologies provide a secure execution environment for applications by creating isolated enclaves within the CPU. Additionally, there are software-based approaches, such as Microsoft\u0026rsquo;s Azure Confidential Computing and Open Enclave SDK, which provide a platform-independent way to build confidential computing solutions.\nIntel Software Guard Extensions (SGX) is a set of instruction codes implementing trusted execution environment that are built into some Intel central processing units (CPUs). They allow user-level and operating system code to define protected private regions of memory, called enclaves. SGX is designed to be useful for implementing secure remote computation, secure web browsing, and digital rights management (DRM). Other applications include concealment of proprietary algorithms and of encryption keys. SGX involves encryption by the CPU of a portion of memory (the enclave). Data and code originating in the enclave are decrypted on the fly within the CPU, protecting them from being examined or read by other code, including code running at higher privilege levels such the operating system and any underlying hypervisors. While this can mitigate many kinds of attacks, it does not protect against side-channel attacks. SGX was first introduced in 2015 with the sixth generation Intel Core microprocessors based on the Skylake microarchitecture. Support for SGX in the CPU is indicated in CPUID “Structured Extended feature Leaf”, EBX bit 02, but its availability to applications requires BIOS / UEFI support and opt-in enabling which is not reflected in CPUID bits. This complicates the feature detection logic for applications. Emulation of SGX was added to an experimental version of the QEMU system emulator in 2014. In 2021, Intel deprecated SGX from the 11th and 12th generation Intel Core Processors, but development continues on Intel Xeon for cloud and enterprise use. Source: Wikipedia\nConfidential computing offers several benefits for safer handling of sensitive data while in use\nAdded security in shared, untrusted, or unfamiliar environments: Confidential computing techniques and technologies give your sensitive data more safeguards, regardless of the computing environment. Secure data input and output: Confidential computing ensures that data is encrypted in memory and processed only after verifying the cloud environment as a trusted execution environment, preventing data access by cloud providers, malicious administrators, and authorized software. Equal focus on security and cloud computing capabilities: Confidential computing enables entirely new security standards that prevent breaches, malware, malicious insiders, and keep hackers out. Remote quality assurance capabilities: Confidential computing allows for remote quality assurance capabilities, which can help organizations save time and money. Easier detection and prevention of unauthorized access: Confidential computing provides a secure environment for processing data in the cloud, which makes it easier to detect and prevent unauthorized access. Compatibility with data privacy and compliance requirements: Confidential computing is compatible with data privacy and compliance requirements, such as the General Data Protection Regulation (GDPR) and the Health Insurance Portability and Accountability Act (HIPAA). Protection for data in use: Confidential computing protects data in use by performing computation in a hardware-based, attested Trusted Execution Environment (TEE), which prevents unauthorized access or modification of applications and data during computation. Azure and Confidential Compute Azure Confidential Computing is a Microsoft Azure service that provides a secure environment for processing sensitive data. It is designed to protect data in use by performing computations in a hardware-based Trusted Execution Environment (TEE). Confidential computing is an industry term defined by the Confidential Computing Consortium (CCC), a foundation dedicated to defining and accelerating the adoption of confidential computing. The CCC defines confidential computing as the protection of data in use by performing computations in a hardware-based, attested Trusted Execution Environment (TEE). These TEEs prevent unauthorized access or modification of applications and data during computation, thereby always protecting data . Azure Confidential Computing provides a range of features that enable customers to configure and protect their data and resources in ways that help them comply with their specific regulatory, security, and sovereignty requirements . Some of the key features include:\nConfidential secrets management Confidential analytics services Control over access to customer workloads Protection of proprietary business logic, analytics functions, machine learning algorithms, or entire applications Elimination of the single largest barrier of encryption - encryption while in use Protection of sensitive or highly regulated data sets and application workloads in a secure public cloud platform Benefits of Azure\u0026rsquo;s Confidential Compute:\nData Confidentiality Protects sensitive data throughout its lifecycle, from processing to storage. Mitigates the risk of data breaches and unauthorized access.\nRegulatory Compliance Helps organizations comply with strict data protection regulations by providing a robust security framework.\nVersatility Supports a wide range of workloads, from traditional applications to modern containerized environments.\nDeveloper Friendly Azure Confidential Compute integrates seamlessly with popular development tools and frameworks, making it accessible for developers.\nAzure Confidential Computing can be used on-premises or in a hybrid cloud environment by using Azure Stack HCI, which is a hyper-converged infrastructure (HCI) cluster solution that runs virtualized Windows and Linux workloads in a hybrid on-premises environment . Azure Stack HCI provides a consistent Azure experience on-premises, enabling customers to run Azure services on-premises and in the cloud. In addition, Azure Arc enables customers to manage servers, Kubernetes clusters, and applications across on-premises, multi-cloud, and edge environments. Azure Arc provides a single control plane for managing resources across environments, enabling customers to use Azure services anywhere.\nCurrent Confidential Compute features (General available) Azure Confidential VMs Azure Confidential VMs provide a secure foundation for sensitive workloads. By leveraging Intel SGX (Software Guard Extensions) technology, these virtual machines create a confidential enclave where data can be processed in a secure and isolated environment. This ensures that even the cloud service provider cannot access the data during processing.\nAzure Kubernetes Service (AKS) with Confidential Nodes For containerized workloads, Azure introduces Confidential Nodes in AKS. This allows organizations to run confidential workloads in Kubernetes clusters, maintaining the same level of security as Confidential VMs. This is particularly valuable for applications that rely on container orchestration.\nAzure Attestation Azure Attestation is a service that remotely validates the trustworthiness of a confidential enclave. It enables developers to ensure that their code is running within a secure environment, protecting against potential threats. This attestation capability enhances the overall security posture of applications leveraging Confidential Compute.\nAzure IoT Edge with Confidential Compute Extending the benefits of Confidential Compute to the edge, Azure IoT Edge enables secure and private processing of data on IoT devices. This is crucial for scenarios where real-time processing is required without compromising on security.\nCloud for Sovereignty The term cloud sovereignty refers to the concept of ensuring that data and operations in the cloud are subject to the laws and regulations of the country or region in which they are located. It involves addressing concerns related to data governance, legal jurisdiction, and compliance with local regulations when using cloud computing services.\nCloud sovereignty is particularly relevant when organizations or governments want to maintain control over their data and ensure that it is subject to the legal frameworks of their own jurisdiction. Some countries or industries have specific regulations regarding data storage, processing, and access, and cloud sovereignty aims to address these requirements.\nAspects of cloud sovereignty are:\nData Location Organizations may have legal or regulatory obligations to store certain types of data within specific geographic boundaries. Cloud sovereignty involves ensuring that data is stored in data centers located within the jurisdiction specified by applicable laws.\nData Access and Control Organizations may want to retain control over who can access their data and under what circumstances. This involves understanding and negotiating the terms of service with cloud providers to ensure compliance with legal requirements related to data access and control.\nLegal and Regulatory Compliance Cloud sovereignty addresses the need to comply with local laws and regulations, including data protection laws, privacy regulations, and other legal requirements that may vary from one jurisdiction to another.\nSecurity and Compliance Standards Ensuring that the cloud infrastructure and services adhere to security and compliance standards that are recognized or mandated by the jurisdiction is a crucial aspect of cloud sovereignty.\nRisk Management Organizations need to assess and manage the risks associated with using cloud services, taking into account the legal, regulatory, and geopolitical considerations that may impact the sovereignty of their data.\nMicrosoft Cloud for Sovereignty\nMicrosoft Cloud for Sovereignty is a cloud solution that enables, for instance public sector customers to build and digitally transform workloads in the Microsoft Cloud while supporting a variety of compliance, security, and policy requirements. It features platform capabilities that unlock greater resiliency, agility, and security while offering greater control over data and increased transparency to the operational and governance processes of the cloud. The solution is designed to help mitigate sovereignty risks by providing a resilient, scalable, and agile platform for (public sector) customers to deploy their sovereign workloads by combining the power of the global Azure platform with several sovereignty capabilities. These capabilities include data residency, confidential computing, document classification, and hybrid deployments. The solution also provides customers with access to local partners who have technical and industry experience to help them plan, incorporate, govern, and operate their cloud environments\nSovereign Landing Zone\nAs part of the Cloud for Sovereignty, Microsoft offers the Sovereign Landing Zone (SLZ). This landing zone is based on the Azure Landing Zone. The difference in the architecture is that you get next to corp and online landingzones also a confidential corp and confidential online landing zone. Using policy as code, you set specific security and privacy related policies on the confidential landing zones.\nAnother difference between \u0026rsquo;normal\u0026rsquo; corp and online landingzone and the confidential corp and online landingzone is that the latter will only allow services based on Azure confidential computing (ACC). ACC reduces the need for trust across various aspects of the compute cloud infrastructure. Azure confidential computing minimizes trust in both the system processes, such as the host OS kernel and the hypervisor, and human operators that include the VM admin and the host administrator. With the guardrails and compliance enforcement of Microsoft Cloud for Sovereignty, ACC provides enhanced protection of sovereign, sensitive or highly regulated data and workloads within the scalability, flexibility, availability, and services of the Azure platform. The SLZ is currently in preview. More information regarding the SLZ can be found here. For instructions how to configure and or deploy the Sovereign Landing Zone, visit Github.\nNew Azure confidential compute features in Q4 In Q4 of 2023, Microsoft announced during Microsoft several new features in the confidential compute domain. Most of them are still in preview.\nMicrosoft Azure Managed Confidential Consortium Framework, now in preview, is a new Azure service that will offer execution of the Microsoft Confidential Consortium Framework (CCF) open-source SDKs as a managed service, eliminating the need for developers to stand up their own infrastructure to support a CCF API endpoint. Developers will be able to more easily build and manage confidential multi-party applications with decentralized trust on a secured and governed network of trusted execution environments. For more information read this blog\nThe confidential virtual machine (VM) option for Azure Databricks is now generally available. Customers seeking to better ensure privacy of personally identifiable information (PII) or other sensitive data while analyzing that data in Azure Databricks can now do so by specifying AMD-based confidential VMs when creating an Azure Databricks cluster. Running a customer’s Azure Databricks cluster on Azure confidential VMs enables Azure Databricks customers to confidently analyze their sensitive data in Azure. For more information read this blog\nThe DCesv5-series and ECesv5-series confidential virtual machines (VMs) are now in preview. Featuring 4th Gen Intel® Xeon® Scalable processors, these VMs are backed by an all-new hardware-based trusted execution environment called Intel® Trust Domain Extensions (TDX). Organizations will be able to use these VMs to seamlessly bring confidential workloads to the cloud without any code changes to their applications. For more information read this blog\nNew features and services for Azure confidential virtual machines (VMs) include Red Hat Enterprise Linux (RHEL) 9.3 support, Disk Integrity Tool, temporary disk encryption, new region support and trusted launch as default in PowerShell for all Azure Gen 2 VMs. RHEL 9.3 support for AMD SEV-SNP confidential VMs will allow Azure customers to specify the RHEL 9.3 image as the guest operating system (OS) for AMD-based confidential VMs. This will ensure any sensitive data processed by their RHEL guest OS is protected in use, in memory. Azure AMD-based confidential VMs provide a strong, hardware-enforced boundary that hardens the protection of the guest OS against host operator access and other Azure tenants. These VMs are designed to help ensure that data in use, in memory, is protected from unauthorized users using encryption keys generated by the underlying chipset and inaccessible to Azure operators. RHEL 9.3 support for AMD SEV-SNP confidential VMs is in preview.\nDisk Integrity Tool for Intel TDX confidential VMs will allow customers to measure and attest to a disk in their confidential VM. The tooling comes as an Azure CLI extension that a user can install in their own trusted environment to run a few simple commands to protect the disk. When such integrity protected disks are used for confidential VM deployments, after the VM boots, users will be able to cryptographically attest that OS disk’s root/system partition contents are secure and as expected before processing any confidential workloads. Disk Integrity Tool for AMD SEV-SNP confidential VMs is in preview.\nTemporary disk encryption for AMD SEV-SNP confidential VMs will allow Azure customers to encrypt the temporary disk attached to their AMD-based confidential VMs using customer-managed keys. This ensures any sensitive data on those disks is protected at rest. Temporary disk encryption for AMD SEV-SNP confidential VMs is in preview.\nNew region support for AMD SEV-SNP confidential VMs is now generally available in the following new regions: Southeast Asia, Central India, East Asia, Italy North, Switzerland North, Japan East, Germany West Central and UAE North.\nTrusted launch as default in PowerShell for all Azure Gen 2 VMs, now generally available, hardens Azure Virtual Machines with security features that allow administrators to deploy virtual machines with verified and signed bootloaders, OS kernels and a boot policy. This is accomplished via such trusted launch features as secure boot, vTPM and boot integrity monitoring that protect against boot kits, rootkits and kernel-level malware. For more information read this blog\nThe NCCv5 series confidential virtual machines with NVIDIA H100 Tensor Core GPUs, in preview, will be the first and only cloud offering of its kind that will allow AI developers to deploy their GPU-powered applications confidentially. This will ensure data in both CPU and GPU memory is always encrypted by using keys generated by hardware and is protected from unauthorized alteration. Data scientists needing to train their models and gain insights from multiple third-party data sources will be able do so while ensuring personal data and AI models are kept private and provide evidence of their confidentiality through attestation reports. For more information read this blog\nConfidential containers on Azure Kubernetes Service (AKS) is the first cloud service offering pod-level isolation and memory encryption in a managed Kubernetes service based on the open-source Kata containers project and powered by AMD SEV-SNP. Organizations will be able to migrate their most sensitive container workloads to the cloud without any code changes, while protecting their data in memory from external and internal threats. Confidential containers on AKS is now in preview. For more information read this blog\nConclusion In conclusion, confidential computing is a technology that helps in providing a secure environment for processing data in the cloud and is here to stay. The technology used is relatively new, so I think there will be a lot of innovation in the near future. Microsoft offers with Azure Confidential Computing in combination with Cloud for Sovereignty a set of services which can support various use cases for securing data in use in different industries, such as government, financial services, and healthcare. If we look at the announcements at Ignite around confidential compute that Microsoft will not stop here but continue to expand on the confidential compute offerings. I think that in the near future, confidential compute will not be limited to the above mentioned use cases but will become the new normal same as disk encryption.\n","permalink":"https://wolkwacht.nl/posts/confidentialcompute/","summary":"\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003eIn the rapidly changing cloud landscape, security and privacy are key concerns for businesses and individuals alike. Confidential Compute turns out to be a possible solution, ensuring sensitive data remains private and secure even in shared computing environments. In this blog post, I want to explain the concept of Confidential Compute in general and explore the specifics of how Azure has implemented confidential compute and how this can help with cloud sovereignty.\u003c/p\u003e","title":"Confidential Computing"},{"content":"From the 21st of June till the 23rd of June the Devopsdays Amsterdam were held in Pakhuis de Zwijger in Amsterdam. There is a first time for everything and for me this I attended the Devopsdays in three roles:\nAttendee, for me this was the third time I attended the Devopsdays in Pakhuis de Zwijger. (Gold) Sponsor, Luminis was one of the gold sponsors. In 2019 the company I worked for at that moment was also a gold sponsor. Speaker, this was the first time for me to speak at Devopsdays. Devopdays starts with a workshop day and after that two days of sessions. Next to that there is a lot of stuff going on what makes my Devopsdays in my opinion an conference everyone should attend. I will give a recap of some of sessions that I thought were interesting and also some focus on my different roles during the conference. Workshop day (Wednesday, 21-06-2023) There were a lot of workshops to choose from. I attended the following workshops:\nDomain Driven Devops by Andrew Clay Shafer. On paper this looked like a very interesting session. But as Andrew told us at the beginning, this workshop normally takes a whole day and now there were only two hours available. So there was no handson part but mostly sharing of knowledge and insights by Andrew. First Steps to Full Lifecycle Security in your DevOps pipeline with Open Source Tools by Anais Urlichs. Anais started her workshop with explaining how you could integrate security in your devops environment. In the hands-on workshop we used Trivy, an open source solution developed by Aquasec, to do all the excercises. If you want to play with Trivy you can find here a repo with some demo excercises. Machine learning 101: Where to begin? by my colleague Lutske de Leeuw. This workshop consisted of two parts. In first part, Lutske gave a presentation explaining the basics of machine learning. In the second part of the workshop, we could start with getting our hands dirty and start with exercises ourselves and start gaining a basic knowledge. First day of sessions (Thursday, 22-06-2023) This day started with the organisation from Devopsdays welcoming us to the event. After that we started with the Keynote - Long live the imposter syndrome! by Julia Sullivan. She started with explaining why imposter syndrome is really a wrong name. During the keynote, there was also participation part of the attendees. First excersise was to tell one of your (biggest) fears with your neighbour. The idea behind this is that talking about your fear(s) helps you and make you a stronger person. Second exercise was staring in the eyes of your neighbour for 30 seconds in silence. Very interesting keynote. Next speaker was Busra Koken with her talk The Art of Turning Incidents into Opportunities. Busra told about her oops moment in which you deploy something and it breaks. Important lesson is that you are not alone and that in devops team there is room for making mistakes, solving it as team and learn from it. After the first brake it was also booth time. During Thursday and Friday is was also at the booth of Luminis to tell what Luminis is all about. A lot of attendees visited the booth. Had some good conversations with people. At the booth is was possible to build yourself as Lego mini figure. You could choose your own head(face), hairstyle/color, and the color of the pants. You could also add a laptop to your figure. This also was a reason for some people to vist the booth more then once. We also promoted our latest whitepaper on the Well Architected Framework. During the lunch break, Dinant Paardenkooper and I, had our interview with Joep Piscaer for the livestream to promote our talk. Nice opportunity that was offered by the organisation. In the afternoon more time for networking with people and talking with other sponsors like Sysdig and Isovalent. Also spoke with some of my former colleagues from Cegeka. Thursday is also the day of the barbecue. The food was very good and also the Devhops beer, Gin Weizen, from Gebrouwen door Vrouwen was very good. After the barbecue there was a bingo organised where could win amongst other things boxes of Lego. The program ended with karaoke.\nSecond day of sessions and also closing day (Friday, 23-06-2023) The second day started with the keynote - There are no soft skills: making empathy actionable by Sharon Steed. This was very powerful keynote in which Sharon shared some personal stories. I was really impressed by the presentation and I think that if I look at myself I can also have some actions to take as it comes to empathy.\nIn the afternoon around 15:00 it was our, Dinant Paardenkooper and me, turn to give our talk AKS Unlighted - but what about cost, compliancy and multi-tenancy. We spoke about how we have implemented a Shared AKS solution on the Azure cloud platform of the city of Amsterdam. We explained some of the challenges we faced on the road of delivering this solution. We also shared the reason why we developed a shared AKS solution. We talked about the pitfalls, the lessons learned and the takeaways. Thea Schukken sumarised our talk in the drawing below.\nConclusion We had a lot of luck with the weather. Apart from some not worth mentioning raindrops around the barbecue the weather was great. The food and drinks were great and also Pakhuis de Zwijger is a real nice location. I want to thank the team of Devopsdays Amsterdam for organising such an inspiring event and also want to thank the community for making it possible to have such good time and have good talks. I also want to thank Megin Zondervan for the beautiful photo\u0026rsquo;s. I have to say that speaking once at Devopsdays, tastes like more. Already thinking for topic to talk about next year or maybe submit a workshop. I can recommend to everyone to submit a talk to devopsdays. Doesn\u0026rsquo;t matter if you are an experienced speaker or just starting. It is a safe and fun environment. If you can\u0026rsquo;t wait until next year, you can always go to Devopsdays Eindhoven. If you want speak there, you can send in your cfp here.\nIf you were not able to attend Devopsdays of part of it, just missed it or want to see everything again, you can view both sessions days via Youtube. On the day 2 stream you can also view the presentation AKS Unlighted - but what about cost, compliancy and multi-tenancy? which I gave together with Dinant Paardenkooper. The links to the streams can be found here: day 1, Thursday 22-3-203 and day 2, Friday 23-06-2023 If you want to download the presentation, that can be found here.\n","permalink":"https://wolkwacht.nl/posts/devopsdays2023/","summary":"\u003cp\u003eFrom the 21st of June till the 23rd of June the Devopsdays Amsterdam were held in Pakhuis de Zwijger in Amsterdam. There is a first time for everything and for me this I attended the Devopsdays in three roles:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAttendee, for me this was the third time I attended the Devopsdays in Pakhuis de Zwijger.\u003c/li\u003e\n\u003cli\u003e(Gold) Sponsor, \u003ca href=\"http://luminis.eu\"\u003eLuminis\u003c/a\u003e was one of the gold sponsors. In 2019 the company I worked for at that moment was also a gold sponsor.\u003c/li\u003e\n\u003cli\u003eSpeaker, this was the first time for me to speak at Devopsdays.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eDevopdays starts with a workshop day and after that two days of sessions. Next to that there is a lot of stuff going on what makes my Devopsdays in my opinion an conference everyone should attend. I will give a recap of some of sessions that I thought were interesting and also some focus on my \u003cem\u003edifferent roles\u003c/em\u003e during the conference.\n\u003cimg alt=\"Pakhuis de Zwijger\" loading=\"lazy\" src=\"/images/2023/001-DevOpsdAys-B.jpg\"\u003e\u003c/p\u003e","title":"Devopsdays Amsterdam 2023"},{"content":"Introduction This year KubeCon / CloudNativeCon Europe was held in the Netherlands at the RAI conference center in Amsterdam. KubeCon started with the different sponsor hosted and CNCF hosted colocated events like ArgoCon, CiliumCon, Observability day and Azure Day with Kubernetes. Very cool that it was hosted in the Netherlands, felt as home game. In this post I want to give a recap of the Azure Day with Kubernetes and KubeCon with some of the highlights, announcements and interesting sessions that I have attended. Videos of all the colocated events are or will become available in the next days. Watch this channel for the all the talks of KubeCon and other events like KCD.\nAzure Day on Kubernetes Microsoft was one of the sponsors that hosted a colocated event, Azure Day with Kubernetes. This day had a fully packed schedule with lots of information, tips and announcements. I want to thank Jorge Palma, Michael Withrow, Pavneet Signh, Allison Ford, Kaysie Yu and Alvin Li for the updates and presentations. If you want to watch the videos because you missed them or just watch them again, register at https://azuredaywithkubernetes2023.com/.\nAzure Kubernetes Service Long Term Support (LTS) Microsoft will offer long time support, 2 years, on Kubernetes starting with version 1.27.1. This an addition on N-2 support which is currently in place. LTS offers the ability to return to the upstream version, also upgrade to the next AKS LTS supported version. LTS version is based on a fork of the upstream EOL. For now supported by Microsoft but aim is to have LTS supported by the community.\nAzure Workload Identity (general available) Azure AD Workload Identity is the highly awaited next iteration of Azure AD Pod Identity that enables Kubernetes applications to access Azure cloud resources securely with Azure Active Directory based on annotated service accounts. Azure AD Workload Identity uses Kubernetes primitives to associate managed identities for Azure resources and identities in Azure Active Directory (AAD) with pods.\nAzure CNI overlay (general available) Azure CNI overlay addresses performance, scalability and IP exhaustion challenges while using traditional Azure Container Networking Interface (CNI).\nDifference between Azure CNI Overlay and Kubenet\nArea Azure CNI Overlay Kubenet Cluster Scale 5000 nodes and 250 pods/node 400 nodes and 250 pods/node Network Configuration Simple - no additional configuration needed for pod networking Complex - requires route tables and UDRS on cluster subnet for pod networking Pod connectivity performance performance on par with VMs in a VNet Additional hops adds minor latency Network Dataplanes Azure and Cilium (eBPF) Azure Kubernetes Network Policies Azure Network Policies, Calico, Cilium Calico OS platforms supported Linux and Windows Linux only Azure CNI Powered by Cilium (public preview) Azure CNI Powered by Cilium combines the robust control plane of Azure CNI with the dataplane of Cilium to provide high-performance networking and security. Cilium Enterpise is now available in the Azure Marketplace. Azure Kubernetes Fleet Manager (preview) With the increasing growth of applying Azure Kubernetes Service (AKS) in environments, it becomes more difficult for the operations teams to handle these environments in a uniform way. With Fleet Manager you can manage Kubernetes cluster at scale, ease the upgrade proces, centrally manage the policies and manage north-south load balancer orchestrates traffic flow across workloads deployed in multiple member clusters of the fleet. For more information about Fleet manager follow the link or for the road map click here.\nAzure Monitor managed service for Prometheus (public preview) Azure Monitor managed service for Prometheus is a component of Azure Monitor Metrics. Azure Monitor managed service for Prometheus allows you to collect and analyze metrics at scale using a Prometheus-compatible monitoring solution, based on the Prometheus project from the Cloud Native Compute Foundation. This fully managed service allows you to use the Prometheus query language (PromQL) to analyze and alert on the performance of monitored infrastructure and workloads without having to operate the underlying infrastructure. For more information about this service follow the link.\nAKS service mesh addon for Istio (public preview) The AKS addon for service mesh builds on top of open source Istio and provides additional benefits such as compatibility testing done between Istio with supported versions of AKS, managed external/internal ingresses, and scaling of Istio control plane components. For more information and how to deploy this addon follow the link.\nBack up Azure Kubernetes Service using Azure Backup (public preview) Azure Backup now supports Backup for AKS, which is available in public preview. This solution simplifies the backup and restore of containerized applications and data. It allows customers to configure scheduled backup for both cluster state and application data, with fine-grained control. Backup for AKS is aligned with the Container Storage Interface (CSI) to offer Kubernetes-aware backup capabilities.\nConfidential Compute Azure Kubernetes Service can make use of the capabilities offered by confidential compute. Azure Confidential Computing offers next to the encryption of data at rest and data in transit also encryption of data in use. This offers protection against third parties accessing data without consent.\nConfidential Containers (CoCo) (preview) Confidential containers on Azure Kubernetes Service (AKS) are leveraging Kata confidential containers and build further on the capabilities offered by confidential compute. More information about this open-source project can be found here. Kata Containers are making use of nested virtualisation. Every pods runs on his own lightweight vm. In this way the application is isolated from the parent VM (AKS node) and from the OS admin of the node.\nMore information about confidential containers can be found in these two links:\nPreview support for Kata VM Isolated Containers on AKS for Pod Sandboxing Aligning with Kata Confidential Containers to achieve zero trust operator deployments with AKS For a complete overview of what the AKS product team is working, check the public roadmap\nKubectl OpenAI plugin This project is a kubectl plugin to generate and apply Kubernetes manifests using OpenAI GPT.\nKubernetes Copilot (experimental) Kubernetes Copilot is powered by OpenAI can help in auditing security issues and diagnose problems.\nKubeCon Europe 2023 Some nice statistics:\nKubeCon was sold out 10.000 attendees 58% of the attendees was for the first time at KubeCon 159 CNCF projects 1300 maintainers, 200k contributors 155 ambassadors in 2023 406 community groups 24 Kubernetes Community Days Two new certifications were announced:\nKubernetes and Cloud Security Associate (available in Q3 2023) Certified GitOps Associate For more information regarding the availability for these exams check the links.\nKubeCon/ CloudNativeCon Europe 2024 will be in Paris from 19-22 of March.\nDuring KubeCon there was a lot of focus on sustainability. During one of the keynotes,Jorge Palma spoke about Building a Sustainable Carbon-Aware Cloud. Jorge announced the availability of the carbon-aware-keda operator. A recap of the keynote can be found in this blog. This operator can help scale Kubernetes workloads based on carbon intensity. Use cases are workloads that can handle interruptions like for instance ML training jobs. in the Github repo you can find more information about to implement this operator. More information information can also be found here:\nhttps://github.com/Azure/sustainability https://github.com/Azure/Kubernetes-carbon-intensity-exporter https://aka.ms/k8sonazure Also Kristina Devochko spoke, in the first breakout session after the keynote, about this topic in her talk Be the Change Our Planet Seeks: How YOU Can Contribute to Running Environment-Friendly Workloads on Kubernetes. Kristina showed the current situation in the world by showing graphs and numbers. She addressed how we as individuals using Kubernetes can play a role in climate change by for instance adopting green coding or lean coding. More information can be found on the site of the Green Software Foundation. Like the shared responsibility applies technical level, it also applies on a sustainability level. Kristina also showed some ways to create visibility by implementing or using tools like OpenCost which shows next to cost also your carbon footprint, the sustainability pillar with in the well architected frameworks by Microsoft or carbon emissions calculators like cloud-carbon-footprint.\nSome other interesing sessions:\nThe Next Episode in Workload Isolation: Confidential Containers - Jeremi Piotrowksi, Microsoft Adopting Network Policies in Highly Secure Environments - Raymond de Jong, Isovalent Gardens and Glaciers: Saving Knowledge Through Succession - Emily Fox, Security Engineer, Apple Unlocking Argo CD\u0026rsquo;s Hidden Tools for Chaos Engineering - Featuring VCluster and More - Dan Garfield \u0026amp; Brandon Philips, Codefresh Building a Succesful Business in Cloud Native - Liz Rice,Isovalent; Guillermo Rauch, Vercel; Kelsey Hightower, Google; Sheng Liang, Acorn Labs; Tom Manvill, Kasten by Veeam It not only getting stars on Github. The open source software you develop is not the part where will earn a lot of money. You have to create business value. Frederick Kautz spoke during his session Trust No system: The Unsettling Reality of Zero-Trust about what the buzzword zero-trust means. How do we handle trust in cloud environment? One of the newest technologies to to lookout for is SPIFFE/SPIRE. Giles Heron from Cisco spoke about Media Streaming Mesh. In his session he talked about the issues that occur when you watch for instance a soccer match via streaming which is then lagging behind broadcast for the more traditional ways of watching. Media mesh on Kubernetes solves this issue.\nFor me it was an interesting conference. Learned a lot of new things. Met a lot of old and new friends, also reconnected with people that I didn\u0026rsquo;t see for some time. Did a lot of networking with people from Tigera, Codefresh, Aqua, Traefik, Suse, Isovalent, Sysdig, Paolo Alto, Spectro Cloud and VMware. As a result of the number of participants I was not able to always attend the sessions that I had planned for. Luckily I can watch those sessions back when they come available online.\n","permalink":"https://wolkwacht.nl/posts/kubeconeu/","summary":"\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003eThis year KubeCon / CloudNativeCon Europe was held in the Netherlands at the RAI conference center in Amsterdam. KubeCon started with the different sponsor hosted and CNCF hosted colocated events like ArgoCon, CiliumCon, Observability day and Azure Day with Kubernetes. Very cool that it was hosted in the Netherlands, felt as home game. In this post I want to give a recap of the Azure Day with Kubernetes and KubeCon with some of the highlights, announcements and interesting sessions that I have attended. Videos of all the colocated events are or will become available in the next days. Watch this \u003ca href=\"ttps://www.youtube.com/@cncf/videos\"\u003echannel\u003c/a\u003e for the all the talks of KubeCon and other events like KCD.\u003c/p\u003e","title":"Azure Day on Kubernetes + KubeCon/CloudNativeCon Europe 2023 "},{"content":"Introduction Recently Azure Kubernetes Service (AKS) Edge Essentials became general available. AKS Edge Essentials is the latest addition to the AKS portfolio which already consist of Azure Kubernetes Services and AKS hybrid cluster. The product was originally announced at Ignite 2022 as AKS light. With AKS Edge, Microsoft makes it possible to run AKS on local Windows machines or Edge devices running the Windows operating system.\nAKS Edge Essentials includes the following features, managed by Microsoft:\nA lightweight, CNCF-conformant K8S and K3S distribution that is supported and managed by Microsoft. The key difference between AKS on HCI and AKS on Windows is that AKS on Windows has minimal compute and memory requirements (4 GB RAM and 2 vCPUs). Each Kubernetes node runs in its own Hyper-V isolated virtual machine and includes many features to help secure your container infrastructure. Microsoft-maintained Linux and Windows worker nodes virtual machine images adhere to security best practices. Microsoft also refreshes these images monthly with the latest security updates. Simplified installation experience with PowerShell cmdlets and agents to enable provisioning and control of VMs and infrastructure. Microsoft provides automatic updates for your Kubernetes deployment, so you stay up-to-date with the latest available Kubernetes versions. Requirements It is possible to run AKS Edge Essentials on a virtual machine in Azure. When creating the virtual machine make sure it supports nested virtualisation.\nAt the time of writing this review running on nested virtualisation was still in preview.\nOS Requirements Currently the following operating systems are supported, Windows 10/11 IoT Enterprise/Enterprise/Pro.\nHardware Requirements Specs Local cluster Arc-connected cluster and GitOps Host OS Windows 10/11 IoT Enterprise/Enterprise/Pro and Windows Server 2019, 2022 Total physical memory 4 GB with at least 2.5 GB free 8 GB with at least 4.5 GB free CPU 2 vCPUs, clock speed at least 1.8 GHz 4 vCPUs, clock speed at least 1.8 GHz Disk space At least 14 GB free At least 14 GB free Setup I tested AKS essentials on my laptop running Windows 11 Home edition. This not a supported version by Microsoft. For testing the installation it is ok but if you want run in production, my recommendation is to run a Microsoft supported version of the base operating system. The home editions are out of the box not able to install the Hyper-V feature which is needed. It is however possible to add the feature.\nTo install the hyper-v feature follow these steps\nOpen a text editor, for instance Notepad Copy the code below in the text editor pushd \u0026#34;%~dp0\u0026#34; dir /b %SystemRoot%\\servicing\\Packages\\*Hyper-V*.mum \u0026gt;hv-home.txt for /f %%i in (\u0026#39;findstr /i . hv-home.txt 2^\u0026gt;nul\u0026#39;) do dism /online /norestart /add-package:\u0026#34;%SystemRoot%\\servicing\\Packages\\%%i\u0026#34; del hv-home.txt Dism /online /enable-feature /featurename:Microsoft-Hyper-V -All /LimitAccess /ALL pause Save the file as a batch file, for instance hyperv.bat Right-click the hyperv.bat file and select the Run as administrator option. Hyper-V and all of its required components are now installed. Reboot your machine after the installation is finished. After that Hyper-V functionality is available. The Hyper-V feature can be turned off or on like any other Windows feature. Now Hyper-V is installed, the installation of AKS Edge Essentials can continue.\nThe installation consist of two parts:\nSetup AKS Edge Essentials offers two options. The first is based on K3S and the second is based on K8S. If you want to know more about K3S, you can read about in some of my earlier posts or go to link at the bottom. Microsoft offers a msi package for both options. They offer also a zip file which makes it possible to add a Windows node to the cluster. This is still experimental. Before we can deploy the AKS cluster we first have to install the msi package and optionally the Windows node. After the installation is completed the AKS Edge modules have to be imported and verified.\nDeployment For the deployment, there are two options available:\nSingle machine deployment Full deployment this is still experimental A machine that is running AKS Edge Essentials is running only one node (Linux vm) and optional one Windows vm. A full deployment makes it possible to have a seperation of the control plane and the worker node. AKS Edge doesn\u0026rsquo;t support autoscaling which is available in AKS For this node a license is required if it is not used for test or development purposes. I chose the single machine deployment as I only have one Windows device available. After the deployment is succesful the optional Windows node can be added. The adding of the node is done by scaling the cluster.\nI started with the K3S installation. After the installation completed I also added the metrics server and the local path provisioner. One of the differences between the K3S and the K8S deployment is the cni which is installed. K3S is installed with Flannel and K8S is installed with Calico.\nI didn\u0026rsquo;t test the optional Windows node as the memory in my machine was not sufficient for running a Windows node next to the Linux node. I also didn\u0026rsquo;t test the Microsoft Arc integration as I have no access to such a Microsoft Arc environment.\nConclusion Azure Kubernetes Edge Essentials is a very nice addition to the Azure Kubernetes offering. It extends Kubernetes to Edge devices that are running Windows operating system. Currently the usefulness of the application is limited as lot of the features are still experimental, like Calico support for K3S. The integration with Microsoft Arc is the reason that AKS Edge essentials is an interesting solution. In use cases where you have large amounts of Windows devices and there is a requirement for running cloud native workloads in containers, running AKS Edge essentials integrated with Microsoft might be the way for creating an environment that can be managed in a consistent and secure way.\nReference links Microsoft AKS Edge Essentials AKS Edge Github repo K3S Local path Project Calico ","permalink":"https://wolkwacht.nl/posts/aks-edge/","summary":"\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003eRecently Azure Kubernetes Service (AKS) Edge Essentials became general available. AKS Edge Essentials is the latest addition to the AKS portfolio which already consist of Azure Kubernetes Services and AKS hybrid cluster. The product was originally announced at Ignite 2022 as AKS light. With AKS Edge, Microsoft makes it possible to run AKS on local Windows machines or Edge devices running the Windows operating system.\u003c/p\u003e\n\u003cp\u003eAKS Edge Essentials includes the following features, managed by Microsoft:\u003c/p\u003e","title":"AKS Edge Essentials"},{"content":"On the 23rd and 24th of February the Kubernetes Community Days (KCD) were organised by the Dutch Kubernetes meetup group backup by the CNCF. The first KCD of its kind was organised in 2019 in the Netherlands. Nowadays it is an CNCF supported event organised by the local community in a country. There are KCD events everywhere in the world. A successful KCD event consists of keynotes, workshops and presentations round the topic Kubernetes. It is not only talks. Important part is also the social component which focuses on the community. For instance a a rather large group of the attendees went after the event to a karaoke bar.This year at the of the conference was a vegan barbecue followed by a dance party which was free for all attendees and after 23:00 everybody was welcome. The location was the ‘party-location’ Westerunie.\nFor me this was the first time that I was not only an attendee but also a speaker. The first day I needed to do nothing so I was able to network and attend talks. The event started with the keynote Scaling the 4th Industrial Revolution of Sarah Polan of Hashicorp in which she made the comparison between the 1st, 2nd, and 3rd Industrial Revolution with the current 4th revolution and the patterns that are applicable in the past and still are.\nAfter this session I attended several other sessions. The sessions that I want to highlight are the following, the session of the ING about their Container Hosting journey by Robbin Siepman. Very nice insights on the journey of the ING and how to build there container platform. Other session that I want to highlight is the session from Kristina Devochko, the session Managed Kubernetes Service: Day Zero Survival Pack. This session contained some very good tips which helps you when you start with Managed Kubernetes. Between the sessions I had some good talks with people from the community. Also visited the booths of the different sponsors.\nThe second day of the event started with keynote Cloud to on-prem and back again: on-prem leanings from a cloud and Kubernetes native company, by Gijs van der Voort. I find \u0026lsquo;real life\u0026rsquo; scenarios very interesting and found the challenges and the choices made to overcome the challenges very interesting to see. The choices between on-premise and in the cloud and the effects of these choices were interesting to see.\nThat morning it was also time for the workshop Kubernetes 201 by Luminis. I gave this workshop together with Dinant Paardenkooper. The goal of the workshop was to be a follow up on the workshop Kubernetes 101 which took place at Thursday. In the workshop we used managed Kubernetes. Originally the workshop was based on Azure Kubernetes Service (AKS) but in the week before we added also Elastic Kubernetes Service (EKS). Both deployment were done from the cloud shell that is provided by AWS and Azure, so that you don\u0026rsquo;t have to install anything locally. For the deployment of EKS we made use of the tool eksutil which deploys a lot of the requirements for you and makes removing the environment also more easy. During the deployment of the Kubernetes environment, which takes around 10 minutes, we played a quiz which contained 15 basic questions around Kubernetes. The top 3 received a small gift. After the quiz we did a quick refresh of some of the commands used by Kubernetes and of the components. After that we continued with the exercises and focussed on exposing an application by using NGINX as load balancer. Next were the different deployment models and we finished up with some basic logging and monitoring. Upfront I thought it would not be enough material but it turned out that we had enough to fill up the slot of 1,5 hour. The workshop exercises are available on Github so if you want to take a look, they can be found here\nAfter the workshop it was time for a nice lunch and some time for our talk later that afternoon. Because of the workshop I was not able to attend Service MESH without the MESS by Raymond de Jong which I hope to see when it is available on the KCD YouTube channel.\nThe presentation, AKS unlighted, but what about Security and Multi-tenancy?, I also co-presented with Dinant Paardenkooper. In this talk we wanted to tell how we created a multi-tenant AKS cluster, for the municipality of Amsterdam, but still be compliant with regulations that apply for a local government. The talk was on the main stage and was also streamed on the KCD YouTube channel. The presentations will be available here so it is possible to see the talk. Our talk was composed of two parts, the business reason and the technical solution. In the business part we talked about the reasons behind the creation of multi-tenant cluster, like lack of Kubernetes operation and security knowledge within the teams and the cost aspect of running a Kubernetes environment. After that we spoke about the architecture and the decisions we made to come to a design. Next part was the more technical part in which we spoke about the translation of the design into an implementation. We addressed the challenges that we faced and showed a picture of how the solution now looks like at what a workload team gets when they will make use of our AKS service. We ended the presentation with the takeaways.\nThe event was closed with an excellent vegan barbecue. I really liked the vegan shredded chicken burger. In conclusion, for me the event was a succes. Attended some good sessions, reconnected with some people form the community I had not seen or spoken for some time. I want to thank the KCD team for the organisation.\n","permalink":"https://wolkwacht.nl/posts/kcd/","summary":"\u003cp\u003eOn the 23rd and 24th of February the Kubernetes Community Days (KCD) were organised by the \u003ca href=\"https://www.meetup.com/dutch-kubernetes-meetup/\"\u003eDutch Kubernetes meetup\u003c/a\u003e group backup by the CNCF. The first KCD of its kind was organised in 2019 in the Netherlands. Nowadays it is an CNCF supported event organised by the local community in a country. There are KCD events everywhere in the world.  A successful KCD event consists of keynotes, workshops and presentations round the topic Kubernetes. It is not only talks. Important part is also the social component which focuses on the community. For instance a a rather large group of the attendees went after the event to a karaoke bar.This year at the of the conference was a vegan barbecue followed by a dance party which was free for all attendees and after 23:00 everybody was welcome.  The location was the ‘party-location’  \u003ca href=\"https://www.westerunie.nl\"\u003eWesterunie\u003c/a\u003e.\u003c/p\u003e","title":"Kubernetes Community Days  Amsterdam 2023"},{"content":"Short update: I\u0026rsquo;m happy to announce that I have been admitted to the Calico Big Cats Ambassador program. Thanks to Tigera, Chris Tomkins and Project Calico. In the last months I have been busy with gettting my Calico certifications. At the moment I have the Calico Certified Operator:Level 1 and Calico Certified Operator: AWS Expert. I\u0026rsquo;m still pursuing the Calico Certified Operator: eBPF. My plan is to finish this training in the coming week or next week.\nAs a Calico Big Cats Ambassador I plan to participate in the @projectcalico community. At the moment I\u0026rsquo;m also in the process of getting Calico working on my rPI based K3S cluster. As soon as that is working I will blog about my approach in how I got it working.\n","permalink":"https://wolkwacht.nl/posts/calicobigcat/","summary":"\u003cp\u003eShort update: I\u0026rsquo;m happy to announce that I have been admitted to the Calico Big Cats Ambassador program. Thanks to Tigera, Chris Tomkins and Project Calico. In the last months I have been busy with gettting my Calico certifications. At the moment I have the Calico Certified Operator:Level 1 and Calico Certified Operator: AWS Expert. I\u0026rsquo;m still pursuing the Calico Certified Operator: eBPF. My plan is to finish this training in the coming week or next week.\u003c/p\u003e","title":"Calico Big Cats Ambassador program"},{"content":"I started 2021 with a focus on Microsoft Azure and also passed two exams which I also mention in earlier posts. In August I changed jobs. At my current employer there is more a focus on AWS as a cloud provider. Unfortunately my AWS certification had expired, so I had to start over again. My goal for 2021 was to pass at least the Solutions Architect Associate exam. I achieved that goal and also passed to other exams.\nAWS Certified Cloud Practioner AWS Certified Cloud Solutions Architect - Associate AWS Certified Cloud Developer - Associate In the preperation to passing the exams I made use of different sources. I used the training provided by A Cloud Guru. I also used the training and practice excercise provided by Whizlabs. The training that I really found useful were the training(s) by Stephane Maarek which can be found on Udemy. And of course the certification guides and the Skillbuilder resources provided by AWS itself. In the end I found the Developer exam the most challenging as that covers parts that I mostly don\u0026rsquo;t focus on. The Solution Architect exam (SAA-C02) was also more challenging then the SAA-C01 which I took some years ago but I like the way that AWS is updating their exams and the way the validate your knowledge.\nNext to the certifications I also passed two AWS accreditations\nAWS Partner: Accreditation (Technical) AWS Partner: Accreditation (Business) For this year 2022, my goal is to pass at least the AWS Certified Security - Specialty and AWS Certified Solutions Architect Professional. We will see how that goes :).\nFor more information about the certifications and accreditations you can check them on Credly.com\n","permalink":"https://wolkwacht.nl/posts/aws-certifications/","summary":"\u003cp\u003eI started 2021 with a focus on Microsoft Azure and also passed two exams which I also mention in earlier posts. In August I changed jobs. At my current employer there is more a focus on AWS as a cloud provider. Unfortunately my AWS certification had expired, so I had to start over again. My goal for 2021 was to pass at least the Solutions Architect Associate exam. I achieved that goal and also passed to other exams.\u003c/p\u003e","title":"AWS Certifications"},{"content":"Last year I passed my az-104 certification and at that moment I thought I would first focus on achieving az-400. At the beginning of this year I decided that it was smarter to first pass the Microsoft Certified: Azure Solutions Architect Expert certification. Reason for that was that I already took the class last year for az-303 which is on of the two exams that you have to pass. So in the beginning of March of this year I passed my az-303 exam and at the end of March I also passed my az-304 exam. View my AZ-303 credential on Credly.\nView my AZ-304 credential on Credly.\nAs I already mentioned in my post around az-104, there was an overlap in knowledge between az-104 and az-303. In preperation of this course I used the blueprint on the Microsoft site. Next to that I used the training videos and the practice questions that are offered by Whizlabs. Because I already pass the az-104 exam I found this exam not very difficult. Next step was az-304. For this I also used the blueprint on the Microsoft site and the information offered on Microsoft Learn. Next to that I also used the the training videos and the practice questions that are offered by Whizlabs.\nView my Azure Solutions Architect Expert credential on Credly.\nNext step is az-400 which I need to achieve the Microsoft Certified: DevOps Engineer Expert status. Goal is to achieve this before the first of June because of the changes in the certification renewal policy and how long your certification will be valid. More information about that can be found here\n","permalink":"https://wolkwacht.nl/posts/az-303-304/","summary":"\u003cp\u003eLast year I passed my az-104 certification and at that moment I thought I would first focus on achieving az-400.\nAt the beginning of this year I decided that it was smarter to first pass the Microsoft Certified: Azure Solutions Architect Expert certification. Reason for that was that I already took the class last year for az-303 which is on of the two exams that you have to pass.\nSo in the beginning of March of this year I passed my az-303 exam and at the end of March I also passed my az-304 exam.\n\u003ca href=\"https://www.credly.com/badges/40ddfaf2-baa8-4a83-b92d-225f6181cc89/public_url\"\u003eView my AZ-303 credential on Credly\u003c/a\u003e.\u003c/p\u003e","title":"Microsoft Certified: Azure Solutions Architect Expert"},{"content":"Today I passed my first Azure exam for this year, Microsoft Certified: Azure Administrator Associate. My goal is to achieve two other certificates in the coming two months, Microsoft Certified: DevOps Engineer Expert and Microsoft Certified: Azure Solutions Architect Expert. For the first certification this is one of the required exams. Next will be AZ-400 Designing and Implementing Microsoft DevOps Solutions\nIn preparation for this exam I read the blueprint and the modules offered on Microsoft Learn. Also I used AZ-104 Microsoft Azure Administrator Exam Certification 2020 by Scott Duffy on Udemy.com and I also followed AZ-303: Microsoft Azure Architect Technologies training which was offered by my company and has some overlap with AZ-104 topic wise. Next to that I used a free account on Azure to get hands on experience with Azure. The exam itself is not difficult and consist only of 65 multiple choice questions. I had some questions where I was not allowed to go back to the earlier question and didn\u0026rsquo;t show up in the question review screen. The last four questions where around a case study. All in all a very doable exam if you have some hands on experience and of course make sure that you read the questions! View my Azure Administrator Associate credential on Credly.\n","permalink":"https://wolkwacht.nl/posts/azure-104/","summary":"\u003cp\u003eToday I passed my first Azure exam for this year, Microsoft Certified: Azure Administrator Associate. My goal is to achieve two other certificates in the coming two months, Microsoft Certified: DevOps Engineer Expert and Microsoft Certified: Azure Solutions Architect Expert. For the first certification this is one of the required exams. Next will be AZ-400 Designing and Implementing Microsoft DevOps Solutions\u003c/p\u003e\n\u003cp\u003eIn preparation for this exam I read the blueprint and the modules offered on Microsoft Learn. Also I used AZ-104 Microsoft Azure Administrator Exam Certification 2020 by Scott Duffy on Udemy.com and I also followed AZ-303: Microsoft Azure Architect Technologies training which was offered by my company and has some overlap with AZ-104 topic wise. Next to that I used a free account on Azure to get hands on experience with Azure.\nThe exam itself is not difficult and consist only of 65 multiple choice questions. I had some questions where I was not allowed to go back to the earlier question and didn\u0026rsquo;t show up in the question review screen. The last four questions where around a case study.\nAll in all a very doable exam if you have some hands on experience and of course make sure that you read the questions!\n\u003ca href=\"https://www.credly.com/badges/50c56420-3879-42a8-82a6-c60517d6fe92/public_url\"\u003eView my Azure Administrator Associate credential on Credly\u003c/a\u003e.\u003c/p\u003e","title":"Microsoft Certified: Azure Administrator Associate"},{"content":"\nThis is the first day of KubeCon + CloudNativeCon Europe Virtual. Originally this event was planned for the end of March/ beginning of April in Amsterdam. I was really looking forward to it because Amsterdam is very near where I live. Because of covid-19 is wast decided to postpone the event and finally they made it a virtual event. Despite of the fact that it is now a virtual event I’m still happy that Kubecon + CloudNativeCon is being held. I think the CNCF, sponsors, speakers and everybody that is involved with creating this event deserve kudos for organising it. I’m ready to attend the sessions, visit the virtual booth’s and participate in this event. I will not be able to fully focus this whole week on KubeCon so I making choices in which sessions I will follow. During the conference I will share topics that interest me.\nToday I had not a lot of time. Tried Captain Kube\u0026rsquo;s quiz, watched some demo\u0026rsquo;s in the demo theater and I attended the session Welcome to CloudLand! An Illustrated Intro to the Cloud Native Landscape by Kaslin Fields, Google. Very interesting presentation explaining the Cloud Native Landscape in an easy way supported by illustrations.\n","permalink":"https://wolkwacht.nl/posts/kubecon-cloudnativecon-europe-2020/","summary":"\u003cp\u003e\u003cimg loading=\"lazy\" src=\"/images/2020/kubeconbanner.png\"\u003e\u003c/p\u003e\n\u003cp\u003eThis is the first day of KubeCon + CloudNativeCon Europe Virtual. Originally this event was planned for the end of March/ beginning of April in Amsterdam. I was really looking forward to it because Amsterdam is very near where I live. Because of covid-19 is wast decided to postpone the event and finally they made it a virtual event. Despite of the fact that it is now a virtual event I’m still happy that Kubecon + CloudNativeCon is being held. I think the CNCF, sponsors, speakers and everybody that is involved with creating this event deserve kudos for organising  it. I’m ready to attend the sessions, visit the virtual booth’s and participate in this event. I will not be able to fully focus this whole week on KubeCon so I making choices in which sessions I will follow. During the conference I will share topics that interest me.\u003c/p\u003e","title":"KubeCon / CloudNativeCon Europe Virtual"},{"content":"Last week I completed Certified Rancher Operator: Level 1. I already had some experience with the products of Rancher Labs. I wrote some posts about the K3S product and I have implemented a Rancher based solution at a customer. Next to that I was part of team tht build a Rancher based platform to host different teams in a muli tenant setup. I already heard that Rancher was thinking about certifications for their products. I have to say that I like their approach. It is very easy to start. You have to go the Rancher Academy and create a account. Currently there is one training available but during the training there are references being made to the next training and at the end of the course they talk about some of the topics the level 2 training will contain. The training is a mix of presentations and demo\u0026rsquo;s, via Youtube videos, and excercises you can do. The excercises are not mandetory but will help you understand the material better. The trainers Adrian Goins and William Jimenez are doing a good job in presenting the material. It easy to follow and complete because the videos are split up in small chunks so that you are able to complete parts of the training without having to block large parts of your day. After you have completed the training and passed the final test you get a certificate. You can see my certificate here\nI can recommend this training to everybody who wants to do more with Rancher and Kubernetes. Personally I\u0026rsquo;m looking forward to level 2 :) .\n","permalink":"https://wolkwacht.nl/posts/rancher-certificate/","summary":"\u003cp\u003eLast week I completed Certified Rancher Operator: Level 1. I already had some experience with the products of Rancher Labs. I wrote some posts about the K3S product and I have implemented a Rancher based solution at a customer. Next to that I was part of team tht build a Rancher based platform to host different teams in a muli tenant setup.\nI already heard that Rancher was thinking about certifications for their products. I have to say that I like their approach. It is very easy to start. You have to go the \u003ca href=\"https://academy.rancher.com/courses/course-v1:RANCHER+K101+2019/about\"\u003eRancher Academy\u003c/a\u003e and create a account. Currently there is one training available but during the training there are references being made to the next training and at the end of the course they talk about some of the topics the level 2 training will contain.\nThe training is a mix of presentations and demo\u0026rsquo;s, via Youtube videos, and excercises you can do. The excercises are not mandetory but will help you understand the material better. The trainers Adrian Goins and William Jimenez are doing a good job in presenting the material. It easy to follow and complete because the videos are split up in small chunks so that you are able to complete parts of the training without having to block large parts of your day.\nAfter you have completed the training and passed the final test you get a certificate. You can see my \u003ca href=\"https://academy.rancher.com/certificates/e5a3c33866e64b9d94178c3af8942079\"\u003ecertificate here\u003c/a\u003e\u003c/p\u003e","title":"Certified Rancher Operator"},{"content":"In 2015 I started with a personal blog which was called Mindmelt.nl. This blog was based on Wordpress and up until now my blog was running there.Last year I talked with some colleagues about which platform they were using and some of them mentioned Hugo. I really like Hugo and also the fact that when you can check your post local before publishing, push it to a git repository and have it automatically being hosted online. So with the \u0026lsquo;intelligent lockdown\u0026rsquo; we are in due to Corvid-19, I thought it was a good idea to migrate my current blog, based on Wordpress, to a new blog based on Hugo. I also decided to change the domain name from Mindmelt.nl to JurgenAllewijn.nl. When I decided to move to a new blog, I didn\u0026rsquo;t want to loose all my old blogposts so I tried some of the tools which were mentioned on the Hugo website. I was not very happy with the results of the tools I tried. Then I came over an article about migrating from Wordpress to Hugo by Christopher Kirk-Nielsen. As written in his article he used a tool called blog2md. First I logged on to my Wordpress site and exported all my posts to an xml file. With blog2md I converted this xml to markdown so that I had all my post as seperate markdown files. For the images I had to choose another approach. I installed the plugin WP file manager on my Wordpress site. With WP File Manager you get access to your Wordpress site. You can then select all images in the Upload directory and download them as an archive. I had now all the components I needed to start my new blog. First I installed Hugo using Homebrew:\nbrew install hugo After Hugo I installed my new site:\nhugo new site mysite cd mysite I then had to choose a theme. On the Hugo website are a lot of different teams. I was looking for a \u0026lsquo;clean\u0026rsquo; responsive theme which also supported dark mode. A friend of mine pointed me to the Loveit theme which was meeting my requirements. This theme is well documented and maintained. My goal was to push my blog to Gitlab and then make use of Netlify to host it. The result is that when I now push an update to my gitlab repository, the change is picked up bij Netlify and my blog is automatically rebuild.\ngit init git submodule -b master add https://github.com/dillonzq/LoveIt.git themes/LoveIt Next step was adopting the config.toml to reflect the configuration that I wanted. As a base I used the config.toml example that was being provided by the LoveIT theme. I then copied all the blogpost markdown files, from my old blog, to the /content/posts directory. Blog2md did a good job in converting all my posts to markdown but all the image references were still pointing to my old blog. I have not found a way yet to easily convert the image references so I\u0026rsquo;m in the process of changing them manually and also checking the posts. I copied the iamges that I downloaded from my old blog, that were still used, to the /images/ directory. After that was done I ran:\nhugo serve With this command Hugo starts your website/blog locally and hosts it at port 1313 so you can check if everything looks ok. When I was satisfied I ran the command:\nhugo Hugo now creates a public folder and generates the files, containing all static content and assets for my blog. I could now use the git commands to push it to my git repository on Gitlab. To have my blog hosted on Netlify I used the instructions on the Hugo website and linked my repository on to my Netlify account. As extra security options I activated MFA on Netlify.\nI\u0026rsquo;m very happy with the results so far. I still have a lot of posts I have to check for references to the mindmelt site.\nUpdate: I moved to another theme as the Loveit theme is no longer supported. I\u0026rsquo;m now using Hugo Clarity\n","permalink":"https://wolkwacht.nl/posts/blog_updated/","summary":"\u003cp\u003eIn 2015 I started with a personal blog which was called Mindmelt.nl. This blog was based on Wordpress and up until now my blog was running there.Last year I talked with some colleagues about which platform they were using and some of them mentioned \u003ca href=\"https://gohugo.io\"\u003eHugo\u003c/a\u003e. I really like Hugo and also the fact that when you can check your post local before publishing, push it to a git repository and have it automatically being hosted online. So with the \u0026lsquo;intelligent lockdown\u0026rsquo; we are in due to Corvid-19, I thought it was a good idea to migrate my current blog, based on Wordpress, to a new blog based on Hugo. I also decided to change the domain name from Mindmelt.nl to JurgenAllewijn.nl.\nWhen I decided to move to a new blog, I didn\u0026rsquo;t want to loose all my old blogposts so I tried some of the tools which were mentioned on the Hugo website. I was not very happy with the results of the tools I tried. Then I came over an article about migrating from Wordpress to Hugo by \u003ca href=\"https://www.smashingmagazine.com/2019/05/switch-wordpress-hugo/\"\u003eChristopher Kirk-Nielsen\u003c/a\u003e. As written in his article he used a tool called \u003ca href=\"https://github.com/palaniraja/blog2md\"\u003eblog2md\u003c/a\u003e.\nFirst I logged on to my Wordpress site and exported all my posts to an xml file. With blog2md I converted this xml to markdown so that I had all my post as seperate markdown files. For the images I had to choose another approach. I installed the plugin WP file manager on my Wordpress site.\n\u003cimg alt=\"WP File manager\" loading=\"lazy\" src=\"/images/2020/wpfileman.png\"\u003e\nWith WP File Manager you get access to your Wordpress site. You can then select all images in the Upload directory and download them as an archive.\nI had now all the components I needed to start my new blog.\nFirst I installed Hugo using Homebrew:\u003c/p\u003e","title":"Blog updated and migrated"},{"content":"If you install k3s with the default settings it also installs Traefik as a load balancer. Traefik also offers a dashboard which is very easy to enable. If you go on your k3s machines to the path /var/lib/rancher/k3s/server/manifests you can find their traefik.yaml. To enable the Traefik dashboard you have to add dashboard.enabled: \u0026ldquo;true\u0026rdquo; to the yaml.\nroot@k3s-master-1:/var/lib/rancher/k3s/server/manifests# cat traefik.yaml apiVersion: helm.cattle.io/v1 kind: HelmChart metadata: name: traefik namespace: kube-system spec: chart: https://%{KUBERNETES_API}%/static/charts/traefik-1.77.1.tgz set: rbac.enabled: \u0026#34;true\u0026#34; ssl.enabled: \u0026#34;true\u0026#34; metrics.prometheus.enabled: \u0026#34;true\u0026#34; kubernetes.ingressEndpoint.useDefaultPublishedService: \u0026#34;true\u0026#34; dashboard.enabled: \u0026#34;true\u0026#34; root@k3s-master-1:/var/lib/rancher/k3s/server/manifests# cat traefik.yaml apiVersion: helm.cattle.io/v1 kind: HelmChart metadata: name: traefik namespace: kube-system spec: chart: https://%{KUBERNETES_API}%/static/charts/traefik-1.77.1.tgz set: rbac.enabled: \u0026#34;true\u0026#34; ssl.enabled: \u0026#34;true\u0026#34; metrics.prometheus.enabled: \u0026#34;true\u0026#34; kubernetes.ingressEndpoint.useDefaultPublishedService: \u0026#34;true\u0026#34; dashboard.enabled: \u0026#34;true\u0026#34; After a few minutes you will see some extra pods getting started.\nroot@k3s-master-1:~# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE helm-install-traefik-4lz62 0/1 Completed 0 14d coredns-66f496764-46dpj 1/1 Running 0 14d svclb-traefik-kc6sx 3/3 Running 0 14d svclb-traefik-tmwv6 3/3 Running 6 14d svclb-traefik-24dm6 3/3 Running 3 14d svclb-traefik-xxbcd 3/3 Running 3 14d svclb-traefik-8n5cq 3/3 Running 3 14d svclb-traefik-xqf2g 3/3 Running 3 14d helm-install-traefik-btn4j 0/1 Completed 0 14d helm-install-traefik-cv7mj 0/1 Completed 0 12d traefik-7f759dfc78-4ds69 1/1 Running 0 12d metrics-server-5f476d6468-99f2s 1/1 Running 1 12d svclb-traefik-82wj2 3/3 Running 6 14d Also a new endpoint is added for the Traefik Dahsboard.\nroot@k3s-master-1:~# kubectl get endpoints -n kube-system NAME ENDPOINTS AGE kube-dns 10.42.0.9:53,10.42.0.9:53,10.42.0.9:9153 14d traefik-dashboard 10.42.5.7:8080 12d traefik 10.42.5.7:80,10.42.5.7:8080,10.42.5.7:443 14d metrics-server 10.42.3.7:443 12d You can the browse to the dashboard on port 8080.\nFor more information about how to use Traefik and the Traefik dashboard you can go to Containous.\n","permalink":"https://wolkwacht.nl/posts/k3s-enable-traefik-dashboard/","summary":"\u003cp\u003eIf you install k3s with the default settings it also installs Traefik as a load balancer. Traefik also offers a dashboard which is very easy to enable. If you go on your k3s machines to the path /var/lib/rancher/k3s/server/manifests you can find their traefik.yaml. To enable the Traefik dashboard you have to add \u003cem\u003edashboard.enabled: \u0026ldquo;true\u0026rdquo;\u003c/em\u003e to the yaml.\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003eroot@k3s-master-1:/var/lib/rancher/k3s/server/manifests# cat traefik.yaml\napiVersion: helm.cattle.io/v1\nkind: HelmChart\nmetadata:\n  name: traefik\n  namespace: kube-system\nspec:\n  chart: https://%{KUBERNETES_API}%/static/charts/traefik-1.77.1.tgz\n  set:\n    rbac.enabled: \u0026#34;true\u0026#34;\n    ssl.enabled: \u0026#34;true\u0026#34;\n    metrics.prometheus.enabled: \u0026#34;true\u0026#34;\n    kubernetes.ingressEndpoint.useDefaultPublishedService: \u0026#34;true\u0026#34;\n    dashboard.enabled: \u0026#34;true\u0026#34;\nroot@k3s-master-1:/var/lib/rancher/k3s/server/manifests# cat traefik.yaml\napiVersion: helm.cattle.io/v1\nkind: HelmChart\nmetadata:\n  name: traefik\n  namespace: kube-system\nspec:\n  chart: https://%{KUBERNETES_API}%/static/charts/traefik-1.77.1.tgz\n  set:\n    rbac.enabled: \u0026#34;true\u0026#34;\n    ssl.enabled: \u0026#34;true\u0026#34;\n    metrics.prometheus.enabled: \u0026#34;true\u0026#34;\n    kubernetes.ingressEndpoint.useDefaultPublishedService: \u0026#34;true\u0026#34;\n    dashboard.enabled: \u0026#34;true\u0026#34;\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eAfter a few minutes you will see some extra pods getting started.\u003c/p\u003e","title":"k3s: Enable Traefik dashboard"},{"content":" Today I got the email from Google saying that I passed the Google Cloud Associate Engineer certification. Last Friday I took the exam at a Kryterion test center and got the pass score but had to wait till the final approval from Google.\nAt my current job I don\u0026rsquo;t work with Google Cloud but more with Microsoft Azure, AWS and data center environments. Google certification was on my list of certifications I wanted to look into. When A Cloud Guru also started with the offering of course around Google Cloud I bought it. I already knew them from the AWS courses and I really like their approach. I started also with the training but didn\u0026rsquo;t finish it.\nIn the end of June I read about the Google certification challenge and saw this as an opportunity in achieving my first Google certification. In this challenge you have to pass the certification within 3 months. So I started the challenge. Google has an recommended learning in which they offer some Qwiklab quests, and Coursera training, and a practice exam.\nThe Qwiklab quests are really useful to get a first experience with Google Cloud. The Cloud Architecture Quests were a bit more challenging as you have to use the knowledge which you acquired in the GCP Essentials Quest. I did these is the first two weeks. I also started reading the Associate Cloud Engineer Study Guide eBook. This eBook was important parts in passing my exam. The eBook has also practice questions but even better they offer an online bank with test questions which you get access to if you have bought the book. These question are not a brain dump or exam dump but help you get familiar with choosing the right solutions that Google Cloud offers. I also watched parts of the training I bought from a Cloud Guru. I also used parts the training from Google that are offered on Coursera.\nWith all IT based exams hands-on experience is very useful and the Google Cloud Certification is no exception. Of course you have the labs offered to by Qwiklabs, but you also make use of the free trial offered by Google where you can $300,00 credits which you can use within Google cloud.\nThe exam consist of 50 questions. The exam is not really difficult in my opinion but it is very important to take your time to read the questions very carefully. Look for the answer that is the best practice from Google and choose the answer that is the best solution for what was stated in the question. So don\u0026rsquo;t choose the question that might be a case of \u0026lsquo;over engineering\u0026rsquo;.\nGood luck with the exam!\n","permalink":"https://wolkwacht.nl/posts/google-certified-assoicate-cloud-engineer/","summary":"\u003cp\u003e\u003cimg alt=\"google badge\" loading=\"lazy\" src=\"/images/2019/google-cloud-certified-cloud-engineer-150x150.png\"\u003e\nToday I got the email from Google saying that I passed the Google Cloud Associate Engineer certification. Last Friday I took the exam at a Kryterion test center and got the pass score but had to wait till the final approval from Google.\u003c/p\u003e\n\u003cp\u003eAt my current job I don\u0026rsquo;t work with Google Cloud but more with Microsoft Azure, AWS and data center environments. Google certification was on my list of certifications I wanted to look into. When \u003ca href=\"https://acloud.guru\"\u003eA Cloud Guru\u003c/a\u003e also started with the offering of course around Google Cloud I bought it. I already knew them from the AWS courses and I really like their approach. I started also with the training but didn\u0026rsquo;t finish it.\u003c/p\u003e","title":"Google Certified Associate Cloud Engineer"},{"content":"As I already explained in my earlier post, this was my first KubeCon | CloudNativeCon that I attended. What I liked about KubeCon was the diversity in sessions and that it was often a difficult choice per time slot which session to attend. For me this congress was next to attending sessions also about networking with vendors and other attendees. There were a lot of good opportunities and I spoke to a lot of people.\nFor me KubeCon started on Monday. I picked up my badge at the Fira in the afternoon and in the evening, my colleague and I were invited to a dinner with the guys and girls from Rancher Labs.This dinner gave us the possibility to already meet the people from Rancher Labs but also talk to some of the other attendees of Kubecon and users of Rancher products.\nThe Tuesday started with a talk with the guys from Twistlock. After that is was time for the first keynote sessions of this edition of Kubecon. The keynotes started with talk by Dan Kohn and Cheryl Hung from the Cloud Native Computing Foundation.After that Brian Liles came on stage. He gave some updates from CNCF projects.\nAfter the keynotes and I went for a quick strawl around the Sponsor Showcase. `it is interesting to see that a lot of tech companies are now jumping into Kubernetes. Some of the companies, like for instance SAP, surprised me with their offering around Kubernetes. As you look at the CNCF Cloud Native Landscape poster you see a lot of choice in applications and this was also reflected in the particpants in the sponsor showcase. At sponsor showcase I of course also gathered some swag like a lot of stickers, t-shirts, socks, etc.\nThen in it was time for some session. First session was by Andrew Martin from Control Plane. Andrew spoke about security around CI/CD. After that I went to an intro session about [Helm](http:// 33:46 Intro: Helm - Michelle Noorali \u0026amp; Matt Fisher, Microsoft). After this session tit was time for lunch and another stroll over the sponsor showcase. Had some good talks with some of the sponsors. One of the other interesting sessions in the afternoon was [Streamling Kubernetes Application CI/CD with Bazel.](http://Streamlining Kubernetes Application CI/CD with Bazel - Gregg Donovan \u0026amp; Chris Love) My last session for the day before the keynote sessions was the session [Kubectl Apply 2019: Defense against the Dark arts](http://Kubectl Apply 2019: Defense Against the Dark Arts - Phillip Wittrock \u0026amp; Jennifer Buckley, Google). During the closing keynotes of that day, Gabe Monroy of Microsoft announced [Service Mesh Interface](http://Democratizing Service Mesh on Kubernetes - Gabe Monroy, Microsoft \u0026amp; CNCF Board Member). You can read more info about smi here.\nThe second day of the conference started with keynotes by, the opening by Kubecon host Bryan Liles. After that came an interesting keynote by David Xia, Keynote: [How Spotify Accidentally Deleted All its Kube Clusters with No User Impact](Keynote: How Spotify Accidentally Deleted All its Kube Clusters with No User Impact - David Xia). Interesting story about how learning from mistakes is more important than playing the \u0026lsquo;blame game\u0026rsquo;. Other interesting session of that day were, M3 and Prometheus, Monitoring at Planet Scale for Everyone - Rob Skillington, Uber and Build a Kubernetes Based Cloud Native Storage Solution From Scratch - Sheng Yang, Rancher Labs. During the second day I also visited the sponsor showcase as there were so many interesting vendors to talk to.\nThursday was the last day of Kubecon. I had some good sessions this day like the keynote from ABN AMRO by Laura Rehorst and Testing your K8s apps with KIND. After attending the session about KIND I installed it on my Macbook. Very cool product. I also installed Rancher Rio on the cluster created by KIND. Also a very cool product. Next to the sessions I had a meeting scheduled with Rancher about, amongst other things, the roadmap around k3s, Rio which was just released and shared experiences with the products of Rancher.\nAll the sessions of Kubecon are recorded and put on Youtube. Here is my top 5 of sessions that I attended and I liked:\nKeynote: From COBOL to Kubernetes: A 250 Year Old Bank\u0026rsquo;s Cloud-Native Journey - Laura Rehorst Testing your K8s apps with KIND - Benjamin Elder, Google \u0026amp; James Munnelly, Jetstack.io Build a Kubernetes Based Cloud Native Storage Solution From Scratch - Sheng Yang, Rancher Labs Grafana Loki: Like Prometheus, But for logs. - Tom Wilkie, Grafana Labs Rootless, Reproducible, and Hermetic: Secure Container Build Showdown - Andrew Martin, Control Plane Summary:\nIt was a very interesting conference. Had a lot of good talks with vendors and attendees. Also saw a lot of interesting sessions. It is now important to follow up on the conversations I had and use the knowledge I gathered in the different sessions Looking forward to the next KubeCon | CloudNativeCon Europe.\n","permalink":"https://wolkwacht.nl/posts/kubecon-cloudnativecon-europe-2019-2/","summary":"\u003cp\u003eAs I already explained in my earlier \u003ca href=\"https://www.jurgenallewijn.nl/kubecon-cloudnativecon-europe-2019-1/\"\u003epost\u003c/a\u003e, this was my first KubeCon | CloudNativeCon that I attended. What I liked about KubeCon was the diversity in sessions and that it was often a difficult choice per time slot which session to attend. For me this congress was next to attending sessions also about networking with vendors and other attendees. There were a lot of good opportunities and I spoke to a lot of people.\u003c/p\u003e","title":"KubeCon | CloudNativeCon Europe 2019 part 2"},{"content":"\nThis year I\u0026rsquo;m going to KubeCon | CloudNativeCon which is held in Barcelona from Monday, May 20, 2019 - Thursday, May 23. This is my first KubeCon and I\u0026rsquo;m really looking forward to the sessions and the meetings I have scheduled\nHere is list of tips which kind of apply to every conference I have attended.\nWear comfortable shoes Take a powerbank/ adapters with you Build your agenda using the agenda builder Leave time between session. You need time to process the information you get from different sessions Take time to network with other attendees or vendors Check the sponsors on the Sponsor showcase Install the Sched app to have your schedule on your mobile Enjoy the event :) You can book your agenda via this link, https://kccnceu19.sched.com/ . Below is my schedule as it currently is within the app. I selected a session for every slot but that will not be doable as I have also meetings planned and like I said in my tips you should als to time to process the information and of course visit the booths of the sponsors. So this schedule will definitely change during the conference.\nMAY 21 • TUESDAY\nKeynote: Stitching Things Together – Dan Kohn, Executive Director, Cloud Native Computing Foundation Hall 6\nSpeakers: Dan Kohn\nKeynote: 2.66 Million - Cheryl Hung, Director of Ecosystem, Cloud Native Computing Foundation Hall 6\nSpeakers: Cheryl Hung\nKeynote: CNCF Project Update - Bryan Liles, Senior Staff Engineer, VMware Hall 6\nSpeakers: Bryan Liles\nSponsored Keynote: Network, Please Evolve – Vijoy Pandey, VP/CTO Cloud, Cisco Hall 6\nSpeakers: Vijoy Pandey\nKeynote: Getting Started in the Kubernetes Community - Lucas Käldström, CNCF Ambassador, Independent \u0026amp; Nikhita Raghunath, Software Engineer, Loodse Hall 6\nSpeakers: Lucas Käldström, Nikhita Raghunath\nSponsor ShowcaseSponsor Showcase, Hall 7\nKeynote: Closing Remarks - Bryan Liles, Senior Staff Engineer, VMware Hall 6\nSpeakers: Bryan Liles\nCoffee BreakSponsor Showcase, Hall 7\nRootless, Reproducible, and Hermetic: Secure Container Build Showdown - Andrew Martin, Control Plane Hall 8.1 G2\nSpeakers: Andrew Martin\nKubernetes Failure Stories and How to Crash Your Clusters - Henning Jacobs, Zalando SE Hall 8.0 A1\nSpeakers: Henning Jacobs\nLunch (Provided)Hall 7 + 8.1\nTutorial: Building Security into Kubernetes Deployment Pipelines - Michael Hough, IBM \u0026amp; Sam Irvine, ControlPlane Hall 8.0 D2\nSpeakers: Michael Hough, Sam Irvine\nStreamlining Kubernetes Application CI/CD with Bazel - Gregg Donovan, Etsy.com, Inc. \u0026amp; Chris Love, CNM Consulting Hall 8.0 F3\nSpeakers: Gregg Donovan, Chris Love\nIstio Multi-Cluster Service Mesh Patterns Explained - Daniel Berg \u0026amp; Ram Vennam, IBM Hall 8.0 A1\nSpeakers: Dan Berg, Ram Vennam\nCoffee BreakSponsor Showcase, Hall 7\nKubectl Apply 2019: Defense Against the Dark Arts - Phillip Wittrock \u0026amp; Jennifer Buckley, Google Hall 8.0 B3\nSpeakers: Jennifer Buckley, Phillip Wittrock\nUsing K8s Audit Logs to Secure Your Cluster - Mark Stemm, Sysdig Hall 8.0 A1\nSpeakers: Mark Stemm\nKeynote: Welcome Remarks - Janet Kuo, Software Engineer, Google Hall 6\nSpeakers: Janet Kuo\nSponsored Keynote: To Be Announced (Microsoft)Hall 6\nKeynote: Kubernetes Project Update - Janet Kuo, Software Engineer, Google Hall 6\nSpeakers: Janet Kuo\nSponsored Keynote: Recursive Kubernetes: Cluster API and Clusters as Cattle - Joe Beda, Principal Engineer, VMware Hall 6\nSpeakers: Joe Beda\nKeynote: Reperforming a Nobel Prize Discovery on Kubernetes - Ricardo Rocha, Computing Engineer \u0026amp; Lukas Heinrich, Physicist, CERN Hall 6\nSpeakers: Ricardo Rocha, Lukas Heinrich\nSponsored Keynote: Expanding the Kubernetes Operator Community - Rob Szumski, Principal Product Manager for OpenShift, Red Hat Hall 6\nSpeakers: Rob Szumski\nKeynote: End User Awards - Cheryl Hung, Director of Ecosystem, CNCF Hall 6\nSpeakers: Cheryl Hung\nKeynote: Closing Remarks - Janet Kuo, Software Engineer, Google Hall 6\nSpeakers: Janet Kuo\nK8s Boothday PartySponsor Showcase, Hall 7\nMAY 22 • WEDNESDAY\nThe New Stack Pancake Breakfast, Sponsored by VMware Hall 8.0 D1\nModerators: Alex Williams, Joab Jackson, Speakers: Pere Monclus\nWelcome CoffeeLink Hall 6/7 (Foyer space between Hall 6 \u0026amp; 7)\nKeynote: Opening Remarks - Bryan Liles, Senior Staff Engineer, VMware Hall 6\nSpeakers: Bryan Liles\nKeynote: How Spotify Accidentally Deleted All its Kube Clusters with No User Impact - David Xia, Infrastructure Engineer, Spotify Hall 6\nSpeakers: David Xia\nSponsored Keynote: Building a Bigger Tent: Cloud Native, Cultural Change and Complexity - Bob Quillin, VP Developer Relations, Oracle Cloud Hall 6\nSpeakers: Bob Quillin\nKeynote: A Journey to a Centralized, Globally Distributed Platform – Katie Gamanji, Cloud Platform Engineer, Condé Nast International Hall 6\nSpeakers: Katie Gamanji\nSponsored Keynote: What I Learned Running 10,000+ Kubernetes Clusters - Jason McGee, IBM Fellow, IBM Hall 6\nSpeakers: Jason McGee\nKeynote: Debunking the Myth: Kubernetes Storage is Hard - Saad Ali, Senior Software Engineer, Google Hall 6\nSpeakers: Saad Ali\nKeynote: Closing Remarks - Bryan Liles, Senior Staff Engineer, VMware Hall 6\nSpeakers: Bryan Liles\nCoffee BreakSponsor Showcase, Hall 7\nM3 and Prometheus, Monitoring at Planet Scale for Everyone - Rob Skillington, Uber Hall 8.0 A1\nSpeakers: Rob Skillington\nBuild a Kubernetes Based Cloud Native Storage Solution From Scratch - Sheng Yang, Rancher Labs Hall 8.1 G2\nSpeakers: Sheng Yang\nLunch (Provided)Hall 7 + 8.1\nThe Magic of Kubernetes Self-Healing Capabilities - Saad Ali, Google Hall 8.0 B1\nSpeakers: Saad Ali\nContainer Forensics: What to Do When Your Cluster is a Cluster - Maya Kaczorowski \u0026amp; Ann Wallace, Google Hall 8.0 B1\nSpeakers: Maya Kaczorowski, Ann Wallace\nCoffee BreakSponsor Showcase, Hall 7\nGrow with Less Pains - Meshing From Monolith to Microservices - Leo LIang, Cruise Automation Hall 8.0 F3\nSpeakers: LEO LIANG\nImproving Availability for Stateful Applications in Kubernetes - Michelle Au, Google Hall 8.0 B3\nSpeakers: Michelle Au\nExtending Knative for Fun and Profit - Matt Moore \u0026amp; Ville Aikas, Google Hall 8.1 G2\nSpeakers: Ville Aikas, Matt Moore\nCaller ID in Kubernetes - Michael Danese, Google Hall 8.0 C1\nSpeakers: Michael Danese\nAll Attendee Party at Poble EspanyolPoble Espanyol (Av. Francesc Ferrer i Guàrdia, 13 08038 Barcelona)\nMAY 23 • THURSDAY\nWelcome CoffeeLink Hall 6/7 (Foyer space between Hall 6 \u0026amp; 7)\nKeynote: Opening Remarks - Janet Kuo, Software Engineer, Google Hall 6\nSpeakers: Janet Kuo\nKeynote: Kubernetes - Don\u0026rsquo;t Stop Believin\u0026rsquo; – Bryan Liles, Senior Staff Engineer, VMware Hall 6\nSpeakers: Bryan Liles\nKeynote: From COBOL to Kubernetes: A 250 Year Old Bank\u0026rsquo;s Cloud-Native Journey - Laura Rehorst, Product Owner - Stratus Platform, ABN AMRO Bank NV \u0026amp; Mike Ryan, DevOps Consultant, backtothelab.io Hall 6\nSpeakers: Laura Rehorst, Mike Ryan\nKeynote: Metrics, Logs \u0026amp; Traces; What Does the Future Hold for Observability? - Tom Wilkie, VP Product, Grafana Labs \u0026amp; Frederic Branczyk, Software Engineer, Red Hat Hall 6\nSpeakers: Frederic Branczyk, Tom Wilkie\nKeynote: Closing Remarks - Bryan Liles, Senior Staff Engineer, VMware \u0026amp; Janet Kuo, Software Engineer, Google Hall 6\nSpeakers: Bryan Liles, Janet Kuo\nCoffee BreakSponsor Showcase, Hall 7\nIntro + Deep Dive: Prometheus - Julius Volz, Prometheus \u0026amp; Richard Hartmann, SpaceNet Hall 8.0 D4\nSpeakers: Richard Hartmann, Julius Volz\nDIY Pen-Testing for Your Kubernetes Cluster - Liz Rice, Aqua Security Hall 8.0 B1\nSpeakers: Liz Rice\nLunch (Provided)Hall 7 + 8.1\nHelm 3: Navigating To Distant Shores - Bridget Kromhout \u0026amp; Jessica Deen, Microsoft Hall 8.0 A1\nSpeakers: Jessica Deen, Bridget Kromhout\nDeploy, Scale and Extend Jaeger - Louis-Etienne Dorval, Ticketmaster Hall 8.0 C4\nSpeakers: Louis-Etienne Dorval\nSharing is Caring: Your Kubernetes Cluster, Namespaces, and You - Amy Chen \u0026amp; Eryn Muetzel, VMware Hall 8.0 A1\nSpeakers: Amy Chen, Eryn Muetzel\nCoffee BreakSponsor Showcase, Hall 7\nDeep Dive: KubeEdge - Cindy Xing \u0026amp; Zefeng Wang, Huawei Hall 8.1 G1\nSpeakers: Kevin Wang, Cindy Xing\nHow Does Google Release Kubernetes in GKE - Kobi Magnezi \u0026amp; Josh Hoak, Google Hall 8.0 A1\nSpeakers: Josh Hoak, Kobi Magnezi\n","permalink":"https://wolkwacht.nl/posts/kubecon-cloudnativecon-europe-2019-1/","summary":"\u003cp\u003e\u003cimg loading=\"lazy\" src=\"/images/2019/kubeconcloudnsativecon.jpeg\"\u003e\u003c/p\u003e\n\u003cp\u003eThis year I\u0026rsquo;m going to KubeCon | CloudNativeCon which is held in Barcelona from Monday, May 20, 2019 - Thursday, May 23. This is my first KubeCon and I\u0026rsquo;m really looking forward to the sessions and the meetings I have scheduled\u003c/p\u003e\n\u003cp\u003eHere is list of tips which kind of apply to every conference I have attended.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eWear comfortable shoes\u003c/li\u003e\n\u003cli\u003eTake a powerbank/ adapters with you\u003c/li\u003e\n\u003cli\u003eBuild your agenda using the agenda builder\u003c/li\u003e\n\u003cli\u003eLeave time between session. You need time to process the information you get from different sessions\u003c/li\u003e\n\u003cli\u003eTake time to network with other attendees or vendors\u003c/li\u003e\n\u003cli\u003eCheck the sponsors on the Sponsor showcase\u003c/li\u003e\n\u003cli\u003eInstall the Sched app to have your schedule on your mobile\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eEnjoy the event :)\u003c/strong\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eYou can book your agenda via this link, \u003ca href=\"https://kccnceu19.sched.com/\"\u003ehttps://kccnceu19.\u003c/a\u003e\u003ca href=\"https://kccnceu19.sched.com/\"\u003esched\u003c/a\u003e\u003ca href=\"https://kccnceu19.sched.com/\"\u003e.com/\u003c/a\u003e . Below is my schedule as it currently is within the app. I selected a session for every slot but that will not be doable as I have also meetings planned and like I said in my tips you should als to time to process the information and of course visit the booths of the sponsors. So this schedule will definitely change during the conference.\u003c/p\u003e","title":"KubeCon / CloudNativeCon Europe 2019 part 1"},{"content":"In my post about first experience with k3s I blogged about setting up my k3s based cluster on Raspberry PI\u0026rsquo;s. Since that post I have added two more nodes Raspberry Pi\u0026rsquo;s and also updated to the 0.3.0 version of k3s.\npi@k3s-master-1:~ $ kubectl get nodes NAME STATUS ROLES AGE VERSION k3s-master-1 Ready master 4h11m v1.13.5-k3s.1 k3s-node-1 Ready node 129m v1.13.5-k3s.1 k3s-node-2 Ready node 118m v1.13.5-k3s.1 k3s-node-3 Ready node 119m v1.13.5-k3s.1 pi@k3s-master-1:~ $ Next step for me was getting the Kubernetes Dashboard up and running. I used the information from Web UI (Dashboard) First downloaded I the kubernetes-dashboard.yaml\ncurl -sfL https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml \u0026gt; kubernetes-dashboard.yaml and changed the image as it was pointing to the amd version and replaced it with the arm version.\nspec: containers: - name: kubernetes-dashboard image: k8s.gcr.io/kubernetes-dashboard-arm:v1.10.1 After that I copied the yaml file to the /var/lib/rancher/k3s/server/manifests directory and the pod was created. To access the pod you have to run the command kubectl proxy. This makes it possible to access the dashboard from the local host only. It is possible to access the dashboard from a machine out of the cluster. To make it work you have to setup a ssl tunnel.\nssh -L8001:localhost:8001 \u0026lt;ip-adress of the master\u0026gt; After that you can access the dashboard via this link: http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/\nIn my environment, I have selected the option Token and followed the instructions for creating a token as described here. As they mention there it is a sample user with all permissions so in productions you would have to make other choices.\nNext step was adding load balancing. Out of the box you can use nodeport to expose ports to the outside. This has however limitations. So I added, like a lot of other people, MetalLB. MetalLB can be run in two modes, layer-2 mode and bgp mode. I chose the layer-2 mode as this is very easy to install. You only have to download a YAML manifest.\ncurl -sfL https://raw.githubusercontent.com/google/metallb/v0.7.3/manifests/metallb.yaml \u0026gt; /var/lib/rancher/k3s/server/manifests/metallb.yaml By placing the file in /var/lib/rancher/k3s/server/manifests, it will be automatically applied. After that you have to\nwrite a config map to metallb-system/config. I chose a small ip-range.\napiVersion: v1 kind: ConfigMap metadata: namespace: metallb-system name: config data: config: | address-pools: - name: pod-ralm protocol: layer2 addresses: - 192.168.2.240-192.168.2.250 To bind a service to a specific IP, you can use the loadBalancerIP parameter in your service manifest:\napiVersion: apps/v1beta2 kind: Deployment metadata: name: nginx spec: selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1 ports: - name: http containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: nginx spec: ports: - name: http port: 80 protocol: TCP targetPort: 80 selector: app: nginx type: LoadBalancer This YAML is the example provided MetalLB in the tutorial. After the pod is running, you can look at the nginx service with kubectl get service nginx:\npi@k3s-master-1:~ $ kubectl get service nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx LoadBalancer 10.43.145.246 192.168.2.241 80:30820/TCP 31m When you curl http://192.168.2.241 you should see the default nginx page: “Welcome to nginx!”\n","permalink":"https://wolkwacht.nl/posts/k3s-kubernetes-dashboard-load-balancer/","summary":"\u003cp\u003eIn \u003ca href=\"https://www.jurgenallewijn.nl/k3s-lightweight-kubernetes-distribution-first-experience/\"\u003emy post about first experience with k3s\u003c/a\u003e I blogged about setting up my k3s based cluster on Raspberry PI\u0026rsquo;s. Since that post I have added two more nodes Raspberry Pi\u0026rsquo;s and also updated to the \u003ca href=\"https://twitter.com/Rancher_Labs/status/1111810281443901450\"\u003e0.3.0\u003c/a\u003e version of k3s.\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003epi@k3s-master-1:~ $ kubectl get nodes\nNAME           STATUS   ROLES    AGE     VERSION\nk3s-master-1   Ready    master   4h11m   v1.13.5-k3s.1\nk3s-node-1     Ready    node     129m    v1.13.5-k3s.1\nk3s-node-2     Ready    node     118m    v1.13.5-k3s.1\nk3s-node-3     Ready    node     119m    v1.13.5-k3s.1\npi@k3s-master-1:~ $\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eNext step for me was getting the Kubernetes Dashboard up and running. I used the information from \u003ca href=\"https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/\"\u003eWeb UI (Dashboard)\u003c/a\u003e First downloaded I the kubernetes-dashboard.yaml\u003c/p\u003e","title":"k3s: Kubernetes Dashboard + load balancer"},{"content":"\nRancher has released v0.2.0.of k3s. Information of the new release is in this article. As soon as I have some time and I have added two extra nodes to the cluster, I\u0026rsquo;m going to add the cluster to my Rancher server. I have upgraded my Raspberry Pi\u0026rsquo;s to v0.2.0. The process of upgrading is really simple. Just download the new version and replace the old version of k3s with the new version and restart.\npi@k3s-master-1:~ $ kubectl get nodes NAME STATUS ROLES AGE VERSION k3s-master-1 Ready master 6d5h v1.13.4-k3s.1 k3s-node-1 Ready node 6d4h v1.13.4-k3s.1 pi@k3s-master-1:~ $ Release v0.2.0 of k3s adds several important enhancements and addresses numerous bugs found by our community. Many of these improvements are the direct result of community members opening issues, submitting PRs, and testing fixes.\nFeatures and Enhancements Support arbitrary CRI implementations [#107] - Users can now configure k3s to use cri-o and other CRI implementations that are not packaged into k3s. Support for preloading images [#92] - Users can now have node agents load docker images from a location on the host at startup, eliminating the need to pull images from a remote location. Upgrade to Kubernetes v1.13.4 [#95] - Update to the latest release of Kubernetes. Support k3s on Rancher [#69] - Users can now import k3s clusters into Rancher (supported in Rancher v2.2.0-rc3 and later). Support agent options in server command [#73] - Users can now set any of the options available to agents when starting the k3s server node. Support the ability to run k3s as non-root user [#38] - User can now run the k3s server as a non-root user. Support the ability to read node token from a file [#98] - Users can now have the node-agent read its token from a file rather than passing it as a string. Bug fixes Fixed an issue where preloaded deployment manifests fail to deploy if no namespace is specified #151 Fixed an issue where changes to helm chart values or values.yaml aren\u0026rsquo;t always triggering an upgrade #187 Fixed an issue where nodes with uppercase hostnames hang indefinitely #160 Fixed an issue where containerd log level environment variable is not respected #188 Fixed an issue where node-token path doesn\u0026rsquo;t resolve for root user in agent scripts #189 Fixed an issue where traefik is not listed in the \u0026ndash;no-deploy flag\u0026rsquo;s help text #186 Fixed an issue where changing cluster CIDR was not possible #93 Fixed an issue where k3s systemd service should wait until the server is ready #57 Fixed an issue where test volume mount e2e fails for k3s image #45 Fixed an issue where component status is not accurate #126 Fixed an issue where install script fails if wget is not available #48 Added the ability to dynamically install the latest release of k3s #47 source: https://github.com/rancher/k3s/releases/tag/v0.2.0\n","permalink":"https://wolkwacht.nl/posts/k3s-release-v0-2-0-released/","summary":"\u003cp\u003e\u003cimg alt=\"K3S\" loading=\"lazy\" src=\"/images/2019/k3s-150x137.png\"\u003e\u003c/p\u003e\n\u003cp\u003eRancher has released v0.2.0.of k3s. Information of the new release is in this article. As soon as I have some time and I have added two extra nodes to the cluster, I\u0026rsquo;m going to add the cluster to my Rancher server. I have upgraded my Raspberry Pi\u0026rsquo;s to v0.2.0. The process of upgrading is really simple. Just download the new version and replace the old version of k3s with the new version and restart.\u003c/p\u003e","title":"k3s: release v0.2.0 released"},{"content":"\nRancher introduced alsmost week ago k3s, a lightweight Kubernetes Distribution. In the YouTube video below you hear Shannon Williams and Darren Shepherd from Rancher talk about K3S, what it is, the usecases and demo of K3S.\nhttps://youtu.be/5-5t672vFi4\nk3sis a fully compliant, production-grade Kubernetes distribution that maintains an absolutely tiny footprint. Weighing in at less than 40 MB, it only needs 512 MB of RAM to run. This means it’s perfect for all kinds of computing that requires a minimal about of memory and space.\nk3s is designed for Edge computing, IoT, CI, and ARM. Even if you’re working with something as small as a Raspberry Pi, k3s allows developers to utilize Kubernetes for production workloads. It simplifies operations, reducing the dependencies and steps needed to run a production Kubernetes cluster.\nInstallation is a breeze, considering that k3s is packaged as a single binary with less than 40 MB. However, security isn’t an afterthought, since TLS certificates are generated by default to make sure that all communication is secure by default.\nInstallation\nAs k3s is built for running on hardware like the Raspberry Pi,I thought it would be interesting to take a closer look at the product and install it on my Raspberry Pi.\nMy home setup\nMy home setup is not where it should be yet but two Pi\u0026rsquo;s is enough to start with. My plan is to add some more PI\u0026rsquo;s in the future so I will be able to have multiple nodes and when k3s also supports HA, I can also add an extra master.\nCurrently I use the following equipment for my k3s environment\nRaspberry PI 3B+ (2 at the moment. 2 other will be added later) 16 GB SD card (2) TP-Link TL-SG105 - Switch Anker PowerPort+ 5 Binnen Zwart So I started with preparing the Raspberry Pi\u0026rsquo;s. First I downloaded raspbian-stretch-lite from Raspbian.org. Then I used Etcher from Balena to flash the SD-cards with the image I downloaded. I used 2018-11-13-raspbian-stretch-lite.\nBefore I powered on the Raspberry Pi\u0026rsquo;s I mounted the sd-cards again and created a file in the root of the boot volume so that I could ssh to the PI\u0026rsquo;s\nThe following steps should be run on all Raspberry Pi\u0026rsquo;s that will be part of the cluster.\ntouch ssh After that I powered on the Raspberry Pi\u0026rsquo;s and connected to them using ssh. To make it myself easy I assigned ip-addresses based on the mac-addresses of the Raspberries in my router. The default password for the PI user is raspberry.\nssh pi@192.168.2.100 #this is the ip-address of my Raspberry that will run as master The next step was setting up the host name, changing the password and setting the ip configuration. The\nchanging of the host name and password can be done by raspi-config.\nsudo raspi-config After changing the host name, choose Finish and reboot the Pi. Next stepping was setting up the network configuration. The network configuration can be configured in /etc/dhcpcd.conf\nprofile static_eth0 static ip_address=192.168.2.100/24 # replace this with your node\u0026#39;s ip-address static routers=192.168.2.254 # replace this with the router address static domain_name_servers=8.8.8.8 Next step is turning off swap.\ndphys-swapfile swapoff \u0026amp;\u0026amp; \\ dphys-swapfile uninstall \u0026amp;\u0026amp; \\ update-rc.d dphys-swapfile remove Next step is adding the following line to /boot/cmdline.txt.\nDon\u0026rsquo;t add any new lines! After saving the file reboot and login.\ncgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory These step should only be run on the Pi that will have the role of master\ncurl -sfL https://get.k3s.io | sh - # Check for Ready node, takes maybe 30 seconds k3s kubectl get node Default k3s doesn\u0026rsquo;t assign roles to the nodes and allows for pods to be scheduled on the master. If you want you can change that with the following commands\n# label node as master kubectl label node mymasternode kubernetes.io/role=master kubectl label node mymasternode node-role.kubernetes.io/master=\u0026#34;\u0026#34; # exclude master from scheduling pods kubectl taint nodes mymasternode node-role.kubernetes.io/master=effect:NoSchedule On the node run the following commands\ncurl -fSL \u0026#34;https://github.com/rancher/k3s/releases/download/v0.1.0/k3s-armhf\u0026#34; \\ -o /usr/local/bin/k3s \u0026amp;\u0026amp; \\ chmod +x /usr/local/bin/k3s After that you start the agent\n# NODE_TOKEN comes from /var/lib/rancher/k3s/server/node-token on the master sudo k3s agent --server https://myserver:6443 --token ${NODE_TOKEN} \u0026amp; Optionally you can also set a label for the node. The commands should be run from the master node\nkubectl label node mynode kubernetes.io/role=node kubectl label node mynode node-role.kubernetes.io/node=\u0026#34;\u0026#34; You are now ready to run a pod. As first pod to run I chose Nginx. Create a file at /home/pi/nginx-test.yaml with the following content\n--- apiVersion: v1 kind: Service metadata: name: nginx-unprivileged-test namespace: default spec: type: NodePort selector: app: nginx-unprivileged-test ports: - protocol: TCP nodePort: 30123 port: 8080 name: http targetPort: 8080 --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: nginx-unprivileged-test namespace: default spec: replicas: 1 template: metadata: labels: app: nginx-unprivileged-test spec: containers: - image: nginxinc/nginx-unprivileged name: nginx-unprivileged-test ports: - containerPort: 8080 name: http livenessProbe: httpGet: path: / port: http initialDelaySeconds: 3 periodSeconds: 3 Next step is deploying to the cluster\nkubectl apply -f /home/pi/nginx-test.yaml Since this is a NodePort service, k3s will open a port on the Raspberry Pi at 30123. On my local network, the Raspberry Pi is located on 192.168.2.100\nA lot of more possibilities and stuff to find and try out. It is very easy to install Kubernetes and get a pod running. Looking forward to the upcoming releases.\nIf you want to hear more about k3s, you can attend the online meetup k3s: The Lightweight Kubernetes Distribution Built for the Edge. You can register here\nAs input for the post I used the following sites:\nhttps://k3s.io https://goo.gl/P9fyJG https://github.com/rancher/k3s Another interesting blog is K3S sur un cluster de Raspberry Pi, Blog Zenika\n","permalink":"https://wolkwacht.nl/posts/k3s-lightweight-kubernetes-distribution-first-experience/","summary":"\u003cp\u003e\u003cimg alt=\"K3S\" loading=\"lazy\" src=\"/images/2019/k3s-150x137.png\"\u003e\u003c/p\u003e\n\u003cp\u003eRancher introduced alsmost week ago k3s, a lightweight Kubernetes Distribution. In the YouTube video below you hear Shannon Williams and Darren Shepherd from Rancher talk about K3S, what it is, the usecases and demo of K3S.\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://youtu.be/5-5t672vFi4\"\u003ehttps://youtu.be/5-5t672vFi4\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://github.com/rancher/k3s\"\u003ek3s\u003c/a\u003eis a fully compliant, production-grade Kubernetes distribution that maintains an absolutely tiny footprint. Weighing in at less than 40 MB, it only needs 512 MB of RAM to run. This means it’s perfect for all kinds of computing that requires a minimal about of memory and space.\u003c/p\u003e","title":"k3s: Lightweight Kubernetes Distribution first experience"},{"content":"Earlier this week I passed my Docker exam, and so I can call myself now Docker Certified Associate. For the official certificate, check here. Achieving this certification is the first step in the path to Docker Accredited Consultant. Next step is attending the DAC (Docker Accredited Consultant) workshop which I attended in the second week of July. This certification is also a part of the shift I want to make to more cloud native and also moving away from the infrastructure layer higher up in the stack.\nPreparing for the exam Like most of the IT-based exams, only following a training doesn\u0026rsquo;t help you pass the exam. Training is of course good as base but the real added value you get from hands on experience. This also applies to the Docker exam. There are of course different ways to prepare for the exam. For me the path started already in 2016 when I followed two Docker trainingen. The certification path didn\u0026rsquo;t exist at that time. Last year Docker announced the certification and also offered new classroom based training oppurtunities. I chose not follow another classroom training but follow a training on Udemy by Brett Fisher, Docker Mastery the complete toolset of Docker Captain and on Pluralsight some of the trainingmodules provided by Nigel Poulton. I also bought the book Docker Deep Dive from Nigel Poulton. Next to the study material I installed the trial version of Docker Enterprise Edition several times on different Linux based operating systems and platforms to get experience with the products. I also made use of the exercises and possibilities of \u0026lsquo;Play with Docker. The last piece I used was the the Docker Certified Associate study guide. This guide provides you with all the information about the topics that are covered in the exam.\nNext steps So I\u0026rsquo;m now certified but I really see this as the starting point of my journey into cloud native, containerization, etc. I want to expand my knowledge with products that are part of the ecosystem around Docker. I think, I will focus first at Kubernetes, Ansible and Jenkins but also on a product like Twistlock. I believe that the strength of containerization lies in the sum of the products.\nUseful links: https://www.udemy.com\nhttps://www.pluralsight.com\nhttp://blog.nigelpoulton.com\nhttps://training.play-with-docker.com/\nhttps://success.docker.com\nHelpful materials for (DCA) Certification\nDocker Certified Associate (DCA) Certification Test Resources\nDocker Certified Associate Exam Preparation Guide .\n","permalink":"https://wolkwacht.nl/posts/docker-certified-associate/","summary":"\u003cp\u003eEarlier this week I passed my Docker exam, and so I can call myself now \u003cstrong\u003eDocker Certified Associate\u003c/strong\u003e. For the official certificate, check \u003ca href=\"https://credentials.docker.com/kofjks9i\"\u003ehere\u003c/a\u003e. Achieving this certification is the first step in the path to Docker Accredited Consultant. Next step is attending the DAC (Docker Accredited Consultant) workshop which I attended in the second week of July.  This certification is also a part of the shift I want to make to more cloud native and also moving away from the infrastructure layer higher up in the stack.\u003c/p\u003e","title":"Docker Certified Associate"},{"content":"On the 28th of June the event Containers Today was held in the Fokker Terminal in The Hague. This event was organized by Amazic and Lef Marketing. KPN ICT Consulting was one of the sponsors of this event, next to Docker, Amazic etc. A booth and a session were part of the sponsor package.\nIt was very nice weather and as a result of that sometimes a bit hot inside the building. The event was visited by around the 330 people and had a variety of sessions divided in three different tracks, Business, Operations and Developer. The event started with a keynote session by Bradley Wong, Director of Product Management within Docker. As part of Bradley\u0026rsquo;s presentation were some slides with numbers of Docker downloads, Docker Hub usage and also the number of job listings which included Docker knowledge as a requirement.\nAfter the keynote there was the first breakout session. I attended the session Dockerizing the Enterprise, Fast and Secure. This session was presented by Jean-Paul van Deursen and Wiebe de Roos from ABN AMRO. They showed diagrams of how their CI/CD environment was built and which components were used. They also spoke about the steps they had to take to get to the current situation and what there next steps were. The second breakout session started after the first coffee break. The session in the Operations track was hosted by KPN ICT Consulting. I was one of the presenters together with my colleagues Patrick Mandemaker and Mimo Amghar. The title of our session was The Good, The Bad and The Containers. A lot of people showed up for our session and the room was more than filled. Patrick started the session with some slides about KPN ICT Consulting. After that Mimo took over and told about the history and evolution of containers and what the benefits are of containers for customers of KPN. I finished the presentation by explaining what steps we took in building the platform within our hosting environment and what hurdles had to be taken. There were a lot of good questions and feedback from the audience.\nAfter the session was the lunch break. During the lunch break I spoke with some of the attendees regarding our sessions and also the possibilities and opportunities KPN ICT Consulting and KPN had to offer. I skipped the afternoon tracks and took my time to network with the vendors which were also on the expo floor, manned our booth and also spoke with other attendees.\nThe last session of the day was again presented by Bradley Wong and he gave a recap of the announcements on DockerCon 2018 which took place from the 12th till the 15th of June in San Francisco. He also gave an outlook on the direction in which Docker is going. The day ended with a reception and a raffle hosted by Docker and Amazic. For me, and I guess for everybody attending it was very good event. Met a lot of interesting people, had some good talks. Looking forward to next year.\n","permalink":"https://wolkwacht.nl/posts/containers-today-2018/","summary":"\u003cp\u003eOn the 28th of June the event Containers Today was held in the Fokker Terminal in The Hague. This event was organized by \u003ca href=\"https://www.amazic.com\"\u003eAmazic\u003c/a\u003e and \u003ca href=\"https://www.lefmarketing.com\"\u003eLef Marketing.\u003c/a\u003e KPN ICT Consulting was one of the sponsors of this event, next to Docker, Amazic etc. A booth and a session were part of the sponsor package.\u003c/p\u003e\n\u003cp\u003e\u003cimg alt=\"KPN Booth\" loading=\"lazy\" src=\"/images/2018/kpnbooth-300x225.jpg\"\u003e\u003c/p\u003e\n\u003cp\u003eIt was very nice weather and as a result of that sometimes a bit hot inside the building. The event was visited by around the 330 people and had a variety of sessions divided in three different tracks, Business, Operations and Developer. The event started with a keynote session by Bradley Wong, Director of Product Management within Docker. As part of Bradley\u0026rsquo;s presentation were some slides with numbers of Docker downloads, Docker Hub usage and also the number of job listings which included Docker knowledge as a requirement.\u003c/p\u003e","title":"Containers Today 2018"},{"content":"On the 28th of June the event Containers Today is being organized by Amazic. My employer, KPN ICT Consulting is one of the gold sponsors of this event and I will be one of the speakers during one of the breakout sessions. Hope to see you there or on the booth. Containers Today - Experience the exciting world of containers While digitization is more important than ever before and many professionals work with an innovative portfolio to realize this progress - think of clouds, data centers and application architectures - progress is being countered at the same time by existing applications and related infrastructure (legacy). The solution? With container technology, you can run applications without depending on the underlying servers and software versions, both on-premise and in the cloud, managed and secured from a single platform. Do you want to know what containerization can mean for your organization? On June 28 a.s., we will talk about containerization in one day: how does your organization benefit from container technology? What is the difference between containerization and virtualization? How do you save costs? How do you manage a hybrid infrastructure and containers in the cloud? To answer all questions as well as possible, we divide the day into three different tracks: Business Track - Developer Track - Operations Track For whom On this day we expect more than 300 Dev / DevOp\u0026rsquo;s, ITDM\u0026rsquo;s, Architects, Project Managers, CTO\u0026rsquo;s \u0026amp; CIO\u0026rsquo;s. There is something for everybody\nBusiness Track: ABN AMRO, Accenture, Docker en het Ministerie van Justitie share their experiences Developer Track: Live demo’s and presenations of GitHub, Cloudbees, F5 and Docker Operations Track: What used to take minutes, now only takes seconds. KPN ICT Consulting, Avanade and Docker elaborate about it Register now for free, there are still some tickets available.\n","permalink":"https://wolkwacht.nl/posts/containers-today/","summary":"\u003cp\u003eOn the 28th of June the event Containers Today is being organized by \u003ca href=\"https://www.amazic.com\"\u003eAmazic\u003c/a\u003e. My employer, \u003ca href=\"https://www.kpn.com/zakelijk/grootzakelijk/ict-consulting.htm\"\u003eKPN ICT Consulting\u003c/a\u003e is one of the gold sponsors of this event and I will be one of the speakers during one of the breakout sessions. Hope to see you there or on the booth.   \u003cstrong\u003eContainers Today\u003c/strong\u003e - Experience the exciting world of containers While digitization is more important than ever before and many professionals work with an innovative portfolio to realize this progress - think of clouds, data centers and application architectures - progress is being countered at the same time by existing applications and related infrastructure (legacy). The solution? With container technology, you can run applications without depending on the underlying servers and software versions, both on-premise and in the cloud, managed and secured from a single platform. Do you want to know what containerization can mean for your organization? On June 28 a.s., we will talk about containerization in one day: how does your organization benefit from container technology? What is the difference between containerization and virtualization? How do you save costs? How do you manage a hybrid infrastructure and containers in the cloud? To answer all questions as well as possible, we divide the day into three different tracks: Business Track - Developer Track - Operations Track For whom On this day we expect more than 300 Dev / DevOp\u0026rsquo;s, ITDM\u0026rsquo;s, Architects, Project Managers, CTO\u0026rsquo;s \u0026amp; CIO\u0026rsquo;s. There is something for everybody\u003c/p\u003e","title":"Containers Today announcement"},{"content":"This morning, very early I received the e-mail that I received the vExpert award. This was for me the 2nd time that I got this award. The first time I applied in December 2016 for the vExpert 2017 award, which I got. Last year I also succesful applied for the vExpert Cloud 2017, one of the three sub awards.\nHappy to be a part of the vCommunity and looking forward to again participate in the community and share information and knowlegde. Thanks to Corey Romero and his team for his efforts for making this possible.\nOfficial announcement: First we would like to say thank you to everyone who applied for the 2018 vExpert program. I’m pleased to announce the list of 2018 vExperts. Each of these vExperts have demonstrated significant contributions to the community and a willingness to share their expertise with others. Contributing is not always blogging or Twitter as there are many public speakers, book authors, CloudCred task writers, script writers, VMUG leaders, VMTN community moderators and internal champions among this group. I want to personally thank everyone who applied and point out that a “vExpert” is not a technical certification or even a general measure of VMware expertise. The judges selected people who were particularly engaged with their community and who had developed a substantial personal platform of influence in those communities. If you feel like you were not selected in error, that’s entirely possible. The judges may have overlooked or misinterpreted what you wrote in your application. Email us at vexpert@vmware.com and we can discuss your situation as well as provide feedback and guidance on what you can do to receive the award. We looked at all of the 2017 activities to determine the voting results. We will open the second half 2018 applications around May / June which will only allow for two voting periods this year. If you were selected as a vExpert 2018, we will be conducting the on-boarding throughout the next few weeks so hold tight and expect future communication from us soon. Congratulations to all the vExperts, new and returning and we’re looking forward to working with you.\nWhat is a vExpert The VMware vExpert program is VMware’s global evangelism and advocacy program. The program is designed to put VMware’s marketing resources towards your advocacy efforts. Promotion of your articles, exposure at our global events, co-op advertising, traffic analysis, and early access to beta programs and VMware’s roadmap. Each year, we bring together in the vExpert Program the people who have made some of the most important contributions to the VMware community. These are the bloggers, book authors, VMUG leaders, speakers, tool builders, community leaders and general enthusiasts. They work as IT admins and architects for VMware customers, they act as trusted advisors and implementors for VMware partners or as independent consultants, and some work for VMware itself. All of them have the passion and enthusiasm for technology and applying technology to solve problems. They have contributed to the success of us all by sharing their knowledge and expertise over their days, nights, and weekends. vExperts who participate in the program have access to private betas, free licenses, early access briefings, exclusive events, free access to VMworld conference materials online, exclusive vExpert parties at VMworld and other opportunities to interact with VMware product teams. They also get access to a private community and networking opportunities. New for vExpert 2018 New this year is a completely new vExpert website for applications, data management, vExpert directory. vExperts will be able to manage your own data (email address, employer, and many others), update your applications, download your license keys and download your vExpert Certificate. Apply at https://vexpert.vmware.com Returning vExperts, you have a different process. Please check your email for a password recovery link to reset your password for vexpert.vmware.com. If you do not have an email to reset your vexpert.vmware.com account, please use this URL to reset your password. https://vexpert.vmware.com/recover-password. Use the same email address you use to receive email for the vExpert program.\nEvangelist Path The Evangelist Path includes book authors, bloggers, tool builders, public speakers, VMTN contributors, and other IT professionals who share their knowledge and passion with others with the leverage of a personal public platform to reach many people. Employees of VMware can also apply via the Evangelist path. A VMware employee reference is recommended if your activities weren’t all in public or were in a language other than English. Customer Path The Customer Path is for leaders from VMware customer organizations. They have been internal champions in their organizations or worked with VMware to build success stories, act as customer references, given public interviews, spoken at conferences, or were VMUG leaders. A VMware employee reference is recommended if your activities weren’t all in public. VPN (VMware Partner Network) Path The VPN Path is for employees of our partner companies who lead with passion and by example, who are committed to continuous learning through accreditations and certifications and to making their technical knowledge and expertise available to many. This can take shape of event participation, video, IP generation, as well as public speaking engagements. A VMware employee reference is required for VPN Path candidates. **Questions \u0026amp; Updates **For questions about the application process or the vExpert Program, please send email to vexpert@vmware.com. Be sure to follow @vExpert for updates on the 2018 vExpert program.\n","permalink":"https://wolkwacht.nl/posts/vmware-vexpert-2018-awarded/","summary":"\u003cp\u003eThis morning, very early I received the e-mail that I received the vExpert award. This was for me the 2nd time that I got this award. The first time I applied in December 2016 for the vExpert 2017 award, which I got. Last year I also succesful applied for the vExpert Cloud 2017, one of the three sub awards.\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"/images/2018/Screenshot-2018-03-10-at-09.59.29-300x97.png\"\u003e\u003c/p\u003e\n\u003cp\u003eHappy to be a part of the vCommunity and looking forward to again participate in the community and share information and knowlegde. Thanks to Corey Romero and his team for his efforts for making this possible.\u003c/p\u003e","title":"VMware vExpert 2018 awarded"},{"content":"Today I passed my VMware Certified Professional 6 -Network Virtualization 6.2 certification. Last year I already took the course but at that moment I was too busy with other stuff to also take the exam. Now after reading again the course material, the online training Clear and Simple VMware NSX 6.2 and vSphere Virtual Networks from Rick Crisci via Udemy.com and the use of hands on labs I felt up to the exam.\nView my VMware Network Virtualization credential on Credly.\nWith achieving this certification, I also hold the double VCP status. View my double VCP credential on Credly.\nNow up to the next certification. That will be either Docker Certified Associate or my second AWS exam, AWS Certified Developer Associate. Still not sure which to do first :).\n","permalink":"https://wolkwacht.nl/posts/vmware-certified-professional-6-network-virtualization-6-2/","summary":"\u003cp\u003eToday I passed my VMware Certified Professional 6 -Network Virtualization 6.2 certification.  Last year I already took the course but at that moment I was too busy with other stuff to also take the exam. Now after reading again the course material, the online training \u003cem\u003eClear and Simple VMware NSX 6.2 and vSphere Virtual Networks\u003c/em\u003e from Rick Crisci via Udemy.com and the use of hands on labs I felt up to the exam.\u003c/p\u003e","title":"VMware Certified Professional 6 -Network Virtualization 6.2"},{"content":"Yesterday was the third and last day of VMworld 2017 Europe. I had two sessions scheduled. Both were VMware Cloud on AWS related. First session was: Protecting Virtual Machines in VMware Cloud on AWS.\nIn this session, VMware and Dell EMC also presented their capabilities for this subject. Interesting solutions in which you see the added value of AWS as an endpoint for storing backups on S3 for instance. My second session was VMware Cloud on AWS: An Architectural and Operational Deep Dive. This session was about the architecture of the environment, the vCenter permissions of the customer and how the host remediation process on AWS works. After the sessions I had lunch and had a last quick walk over the solution exchange. Before I went to the airport I spend time on filling out the surveys that I had not processed. If I summarize my visit to VMworld 2017, I think it has been an interesting event. My focus in the sessions was mostly with VMware Cloud on AWS. I had some nice discussions with employees of VMware, peers and vendors and I have gotten a lot of useful info. For me the highlights were:\nVMware Cloud on AWS sessions the VMware HCX announcement VMware Appdefense session ","permalink":"https://wolkwacht.nl/posts/vmworld-2017-day-3/","summary":"\u003cp\u003eYesterday was the third and last day of VMworld 2017 Europe. I had two sessions scheduled. Both were VMware Cloud on AWS related. First session was: \u003cem\u003eProtecting\u003c/em\u003e \u003cem\u003eVirtual Machines in VMware Cloud\u003c/em\u003e \u003cem\u003eon AWS\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"/images/2017/images-300x156.jpg\"\u003e\u003c/p\u003e\n\u003cp\u003eIn this session, VMware and Dell EMC also presented their capabilities for this subject. Interesting solutions in which you see the added value of AWS as an endpoint for storing backups on S3 for instance. My second session was \u003cem\u003eVMware Cloud on AWS: An Architectural and Operational Deep Dive\u003c/em\u003e. This session was about the architecture of the environment, the vCenter permissions of the customer and how the host remediation process on AWS works. After the sessions I had lunch and had a last quick walk over the solution exchange. Before I went to the airport I spend time on filling out the surveys that I had not processed. If I summarize my visit to VMworld 2017, I think it has been an interesting event. My focus in the sessions was mostly with VMware Cloud on AWS.  I had some nice discussions with employees of VMware, peers and vendors and I have gotten a lot of useful info. For me the highlights were:\u003c/p\u003e","title":"VMworld 2017 (Day 3)"},{"content":"\nToday I decided to not go to the general session but instead follow it on the big screen in the VMvillage. This gives you the possibility to get coffee during the keynote :). The keynote started with Pat Gelsinger, Sanjay Poonen and Ray O\u0026rsquo;Farell answering questions that the VMworld attendees could submit yesterday. After that Ray O\u0026rsquo;Farrel, Chris Wolf and Purnima Padmanabhan went more into depth about VMware HCX, Wavefront, NSX-T, VMware Pulse IoT Center and Functions as a Service (FaaS). After the keynote I attended two sessions. Both were around VMware Cloud on AWS. The last two sessions, Service Overview for VMware Cloud on AWS and VMware Cloud on AWS: A Technical Deep Dive, were really interesting. The first session was focussed on service, the support and separation of the responsibilities between the customer and VMware. They also showed the online support that was available for this product and the automatic actions that VMware takes to keep the customer environment healthy. The second session showed in more detail the components that made the VMware Cloud on AWS product. The session touched on compute, storage, networking and security. After the sessions I went back to the solution exchange and visited several booths and also spent time on networking with the vendors, colleagues and other attendees. In the evening I went to the VMware Customer Appreciation Party featuring the Kaiser Chiefs. I think it was without a doubt the best band and the VMworld party. ","permalink":"https://wolkwacht.nl/posts/vmworld-2017-europe-day-2/","summary":"\u003cp\u003e\u003cimg alt=\"badge\" loading=\"lazy\" src=\"/images/2017/badge-300x225.jpg\"\u003e\u003c/p\u003e\n\u003cp\u003eToday I decided to not go to the general session but instead follow it on the big screen in the VMvillage. This gives you the possibility to get coffee during the keynote :). The keynote started with Pat Gelsinger, Sanjay Poonen and Ray O\u0026rsquo;Farell answering questions that the VMworld attendees could submit yesterday. After that Ray O\u0026rsquo;Farrel, Chris Wolf and Purnima Padmanabhan went more into depth about VMware HCX, Wavefront, NSX-T, VMware Pulse IoT Center and Functions as a Service (FaaS). After the keynote I attended two sessions. Both were around VMware Cloud on AWS.  The last two sessions, \u003cem\u003eService Overview for VMware Cloud on AWS\u003c/em\u003e and  \u003cem\u003eVMware Cloud on AWS: A Technical Deep Dive\u003c/em\u003e, were really interesting. The first session was focussed on service, the support and separation of the responsibilities between the customer and VMware. They also showed the online support that was available for this product and the automatic actions that VMware takes to keep the customer environment healthy. The second session showed in more detail the components that made the VMware Cloud on AWS product. The session touched on compute, storage, networking and security. After the sessions I went back to the solution exchange and visited several booths and also spent time on networking with the vendors, colleagues and other attendees. In the evening I went to the VMware Customer Appreciation Party featuring the Kaiser Chiefs. I think it was without a doubt the best band and the VMworld party.\n\u003cimg loading=\"lazy\" src=\"/images/2017/DJoEznWXgAAhfgE-300x225.jpg\"\u003e\u003c/p\u003e","title":"VMworld 2017 Europe (day 2)"},{"content":"For me VMworld already started on Monday with the Partner Exchange and the TAM day. It started today with the keynote by, amongst other speakers, Pat Gelsinger and Sanjay Poonen. They talked about the strategy for VMware. Pat was saying that tech was breaking out of tech. Alan Renouf was, together with Pat Gelsinger, a way of managing your environment via VR. Following announcements were done:\nVMware Integrated Open Stack 4.0 vSphere Integrated Containers 1.2 VMware HCX After the keynote I had no session scheduled so I made a short visit to the solution exchange. Lot of interesting vendors and of course also the usual suspects. Also got some of the swag from some vendors.\nAfter lunch I attended sessions around VMware Cloud on AWS, Containers and Appdefense. A lot of interesting info and new opportunities. In the coming weeks I will write more about VMware Cloud on AWS and Appdefense.\n","permalink":"https://wolkwacht.nl/posts/vmworld-2017-europe-day-1/","summary":"\u003cp\u003eFor me VMworld already started on Monday with the Partner Exchange and the TAM day. It started today with the keynote by, amongst other speakers, Pat Gelsinger and Sanjay Poonen. They talked about the strategy for VMware. Pat was saying that \u003cem\u003etech was breaking out of tech\u003c/em\u003e. Alan Renouf was, together with Pat Gelsinger, a way of managing your environment via VR. Following announcements were done:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eVMware Integrated Open Stack 4.0\u003c/li\u003e\n\u003cli\u003evSphere Integrated Containers 1.2\u003c/li\u003e\n\u003cli\u003eVMware HCX\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eAfter the keynote I had no session scheduled so I made a short visit to the solution exchange. Lot of interesting vendors and of course also the usual suspects. Also got some of the swag from some vendors.\u003c/p\u003e","title":"VMworld 2017 Europe (day 1)"},{"content":"Today VMworld 2017 started for me. Yesterday I flew in to Barcelona and got my badge and the backpack with, amongst other things, the water bottle and the shirt. Today is the partner exchange and the tam day. I had only sessins in the partner track scheduled. My first two sessions were about VMware Cloud on AWS. The first session was less technical but the second session went into more depth about the technology behind VMware Cloud on AWS. After that I attended the general partner sessions. Amongst the speakers there were Pat Gelsinger, Maurizio Carli, Ross Brown and Brandon Sweeney. The told us about the strategy and the partner organisation. After the keynote it was time for lunch. While not everybody who will be attending VMworld was already there, it was still very crowded. After the lunch I attended two other sessions around VMware Cloud on AWS. The second of those two was very interesting as it was about the 3rd party ISV ecosystem for VMware Cloud on AWS. Looking forward to the sessions tomorrow and the opening of the solution exchange.\n","permalink":"https://wolkwacht.nl/posts/vmworld-partner-exchange/","summary":"\u003cp\u003eToday VMworld 2017 started for me. Yesterday I flew in to Barcelona and got my badge and the backpack with, amongst other things, the water bottle and the shirt. Today is the partner exchange and the tam day. I had only sessins in the partner track scheduled. My first two sessions were about VMware Cloud on AWS. The first session was less technical but the second session went into more depth about the technology behind VMware Cloud on AWS. After that I attended the general partner sessions. Amongst the speakers there were Pat Gelsinger, Maurizio Carli, Ross Brown and Brandon Sweeney. The told us about the strategy and the partner organisation. After the keynote it was time for lunch. While not everybody who will be attending VMworld was already there, it was still very crowded. \u003cimg alt=\"Partner lunch\" loading=\"lazy\" src=\"/images/2017/partnerlunch-300x225.jpg\"\u003e\u003c/p\u003e","title":"VMworld Partner Exchange"},{"content":"This year VMworld Europe will again be held in Barcelona, but unlike previous years it will not be in October but already in the second week of September. Just as last year, my employer made it possible for me to attend VMworld Europe. I have to finalize my schedule but as it available, I already started planning which sessions I would like to attend. When I have finalized my schedule I will also post it here. As my interest are shifting to the public cloud, I\u0026rsquo;m really looking forward to the session around VMware on AWS. So mostly my schedule will filled with that as there are many sessions around VMware on AWS. Next to that I\u0026rsquo;m still interested in devops and container related sessions. Interesting links: VMworld parties, gathering and events Sneak peek VMworld bag Content builder VMworld Europe First-Timer\u0026rsquo;s Guide to VMworld\n","permalink":"https://wolkwacht.nl/posts/vmworld-2017-europe/","summary":"\u003cp\u003eThis year VMworld Europe will again be held in Barcelona, but unlike previous years it will not be in October but already in the second week of September. Just as last year, my employer made it possible for me to attend VMworld Europe. I have to finalize my schedule but as it available, I already started planning which sessions I would like to attend. When I have finalized my schedule I will also post it here. As my interest are shifting to the public cloud, I\u0026rsquo;m really looking forward to the session around VMware on AWS. So mostly my schedule will filled with that as there are many sessions around VMware on AWS. Next to that I\u0026rsquo;m still interested in devops and container related sessions. Interesting links: \u003ca href=\"http://www.vbrain.info/2017/07/19/vmworld-emea-2017-parties-gatherings-events/\"\u003eVMworld parties, gathering and events\u003c/a\u003e \u003ca href=\"https://blogs.vmware.com/vmworld/2017/07/sneak-peek-2017-vmworld-backpack.html\"\u003eSneak peek VMworld bag\u003c/a\u003e \u003ca href=\"https://my.vmworld.com/scripts/catalog/eucatalog.jsp\"\u003eContent builder VMworld Europe\u003c/a\u003e \u003ca href=\"https://blogs.vmware.com/vmworld/2017/07/715.html\"\u003eFirst-Timer\u0026rsquo;s Guide to VMworld\u003c/a\u003e\u003c/p\u003e","title":"VMworld 2017 Europe"},{"content":"AWS offer a range of certification types including Associate, Professional, and Specialty. The certifications focus on gaining technical knowledge on the AWS Platform across several roles and specialties.We offer associate certifications covering the roles of Solutions Architect, Developer, and SysOps Administrator. Once you have completed an Associate Certification in any of these roles, you may progress to a Specialty in Advanced Networking or Big Data, or a Professional Certification in Solutions Architect or DevOps Engineering.Specialty Certifications offer advancement for someone interested validating their expertise in a specific area. Professional Certifications are our highest role-specific certification.\nThis week I passed my first exam, Solutions Architect Associate. To achieve this goal I used the exam blueprint from AWS and also the Well Architected Framework from AWS. Next to these documents I played around with some Qwiklabs training and for me the largest help was the online training from Ryan Kroonenburg, from A Cloud guru. Next to the online training I also used the practice exams from Ryan. Next on the list: Certified Developer Associate.\n","permalink":"https://wolkwacht.nl/posts/amazon-web-services-certification-18/","summary":"\u003cp\u003eAWS  offer a range of certification types including Associate, Professional, and Specialty. The certifications focus on gaining technical knowledge on the AWS Platform across several roles and specialties.We offer associate certifications covering the roles of Solutions Architect, Developer, and SysOps Administrator. Once you have completed an Associate Certification in any of these roles, you may progress to a Specialty in Advanced Networking or Big Data, or a Professional Certification in Solutions Architect or DevOps Engineering.Specialty Certifications offer advancement for someone interested validating their expertise in a specific area. Professional Certifications are our highest role-specific certification.\u003c/p\u003e","title":"Amazon Web Services certification 1/8"},{"content":"Yesterday VMware published the list of vExperts for 2017. This is the first time that I applied for the VMware vExpert program and I\u0026rsquo;m honored to have been chosen as one of the few vExperts this year. What is the vExpert program? The VMware vExpert program is VMware\u0026rsquo;s global evangelism and advocacy program. The program is designed to put VMware\u0026rsquo;s marketing resources towards your advocacy efforts. Promotion of your articles, exposure at our global events, co-op advertising, traffic analysis, and early access to beta programs and VMware\u0026rsquo;s roadmap. The awards are for individuals, not companies, and last for one year. Employees of both customers and partners can receive the awards. In the application, we consider various community activities from the previous year as well as the current year\u0026rsquo;s (only for 2nd half applications) activities in determining who gets awards. We look to see that not only were you active but are still active in the path you chose to apply for. Criteria If you are interested in becoming a vExpert the criteria is simple. We are looking for IT Professionals who are sharing their VMware knowledge and contributing that back to the community. The see the term \u0026ldquo;giving back\u0026rdquo; as defined as going above and beyond your day job. There are several ways to share your knowledge and engage with the community. Some of those activities are blogging, book authoring, magazine articles, CloudCred task writing, active in facebook groups, forum (VMTN as well as other non VMware) platforms, public speaking, VMUG leadership, videos and so on. vExpert Program Benefits\nInvite to our private #Slack channel vExpert certificate signed by our CEO Pat Gelsinger. Private forums on communities.vmware.com. Permission to use the vExpert logo on cards, website, etc for one year Access to a private directory for networking, etc. Exclusive gifts from various VMware partners. Private webinars with VMware partners as well as NFR\u0026rsquo;s. Access to private betas (subject to admission by beta teams). 365-day eval licenses for most products for home lab / cloud providers. Private pre-launch briefings via our blogger briefing pre-VMworld (subject to admission by product teams) Blogger early access program for vSphere and some other products. Opportunity to receive a free blogger pass to VMworld US or VMworld Europe (limited to 50 for US and 35 for EU). Featured in a public vExpert online directory. Access to vetted VMware \u0026amp; Virtualization content for your social channels. Yearly vExpert parties at both VMworld US and VMworld Europe events. Identification as a vExpert at both VMworld US and VMworld EU. Within KPN I’m trying to keep everyone up to date on all VMware is doing from an architectural perspective within our KPN SDDC and Cloud team. Thanks to Corey Romero and the VMware Social Media \u0026amp; Community Team. Grats to all the 2017 vExperts. Read the complete list here\n","permalink":"https://wolkwacht.nl/posts/vexpert-2017/","summary":"\u003cp\u003eYesterday VMware published the list of vExperts for 2017.  This is the first time that I applied for the VMware vExpert program and I\u0026rsquo;m honored to have been chosen as one of the few vExperts this year. \u003cstrong\u003eWhat is the vExpert program?\u003c/strong\u003e The VMware vExpert program is VMware\u0026rsquo;s global evangelism and advocacy program. The program is designed to put VMware\u0026rsquo;s marketing resources towards your advocacy efforts. Promotion of your articles, exposure at our global events, co-op advertising, traffic analysis, and early access to beta programs and VMware\u0026rsquo;s roadmap. The awards are for individuals, not companies, and last for one year. Employees of both customers and partners can receive the awards. In the application, we consider various community activities from the previous year as well as the current year\u0026rsquo;s (only for 2nd half applications) activities in determining who gets awards. We look to see that not only were you active but are still active in the path you chose to apply for.\n\u003cimg loading=\"lazy\" src=\"/images/2017/VMW-LOGO-vEXPERT-2017-k.png\"\u003e\u003c/p\u003e","title":"vExpert 2017"},{"content":" Today is the last day of VMworld 2016 Europe. It has been an interesting congress with a lot of topics and also a lot of ways to get information. The last day of VMworld is always the Thursday and you see already of the attendees leaving as they travel back home. The venue gets quieter by the hour. This last days I had three interesting sessions scheduled:\nYour Open-Source Datacenter VM\u0026rsquo;s and Containers: Extending Docker to vCloud Air Introducing VMware Cloud Foundation Your Open-Source Datacenter, was a session given by two enthusiastic guys, David Lloyd and Ramon Tarnavski. They talked about the advantages of Source:\nLow barrier of entry (free) Community driven (many eyeballs on code) Quality projects Drive technological innovations Prefered approach to computing Allows speed to market Choices and vendor lock-in averse Amongs other things, they spoke about RackHD. RackHD is a tool which can help you with the following things:\nDevice Discovery, logging and alerting Event based workflow Engine Firmware management System provisioning Open access with RESTful API allowing your infrastructure to be defined as code Very interesting session which much more information I will include in another presentation. After this session I followed VM\u0026rsquo;s and Containers: Extending Docker to vCloud Air. This session started with an explanation what the difference is between cloud native applications and devops. After that the way Docker and containers work. Last part of the presentation was about the value that vCloud Air can offer to running Docker workloads in combination with vSphere Integrated Containers or Photon Platform. Also a lot of interesting possibilities to further investigate. Last session of this VMworld was Introducing VMware Cloud Foundation -a Closer Look. In this session the technical details behind VMware Cloud Foundation were shared. These are the characterics of VMware Cloud Foundation:\nDelivers the next generation hyper-converged platform Combines vSphere, VSAN and NSX into a single unified platform Includes VMware SDDC Manager Natively integrates the cloud infrastructure stack Automates deployment and lifecycle of the cloud infrastructure stack Provides the universal platform for any application - traditonal and cloud native Available across private and public clouds VMware Cloud Foundation, on premise, is not a roadmap product but direct available. On the roadmap for VMware Cloud Foundation are:\nVMware Cloud Foundation Service on IBM (available in Q4) VMware vCloud Air (Beta in Q4) VMware Cloud on Amazon (available in mid 2017) ","permalink":"https://wolkwacht.nl/posts/vmworld-2016-day-3-be_tomorrow/","summary":"\u003cp\u003e\u003cimg alt=\"Roadmap\" loading=\"lazy\" src=\"/images/2016/wp-image-900823913jpg.jpg\"\u003e\nToday is the last day of VMworld 2016 Europe. It has been an interesting congress with a lot of topics and also a  lot of ways to get information. The last day of VMworld is always the Thursday and you see already of the attendees leaving as they travel back home. The venue gets quieter by the hour. This last days I had three interesting sessions scheduled:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eYour Open-Source Datacenter\u003c/li\u003e\n\u003cli\u003eVM\u0026rsquo;s and Containers: Extending Docker to vCloud Air\u003c/li\u003e\n\u003cli\u003eIntroducing VMware Cloud Foundation\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eYour Open-Source Datacenter, was a session given by two enthusiastic guys, David Lloyd and Ramon Tarnavski.  They talked about the advantages of Source:\u003c/p\u003e","title":"VMworld 2016 Day 3, be_Tomorrow"},{"content":" Today is the first day of VMworld 2016. The day starts with the general keynote session. First speaker during the general session was Jean-Pierre Brulard, general manager EMEA. He started with some statistics. This VMworld there are 10k attendees, 300 sessions. This is the first time that VMworld and a AirWatch connect are held together. The second speaker was Pat Gelsinger,CEO. Pat spoke about the world in which you control nothing but are in charge. This is a good base for the hybrid cloud. Pat talked about Digital transformation, which is according to him the new buzzword. He said he rather used Digital Business as also legacy applications are still there. Pat also announced vSphere 6.5 and VSAN 6.5. After that Pat talked about the cross cloud architecture. Part of it is the cross cloud service consisting of two parts, Cloud Foundation and a set of SAAS applications which will become available in 2017. These SAAS applications will offer you cloud control over your cross cloud environment. Pat also addressed the AWS announcement. More details below in this post and other posts to come. After the general session I did a quick walk around at the Solution Exchange which also openend today. After that I went for an early lunch because I didn\u0026rsquo;t want to miss the session VMware Cloud on AWS. Last week AWS and VMware announced this new partnership. This partnership was also mentioned during the keynote and also yesterday during Partner sessions. VMware Cloud on AWS gives customers the possibility to run their workloads in the cloud with a lot of the benefits that AWS has to offer. One of the benefits is elastic drs, which causes in times of shortage of resources to automatic provision an extra host. This includes cpu, memory and storage. VMware Cloud is based on vSphere 6.5, NSX and VSAN.VMware Cloud will be managed and operated by VMware people on AWS hardware. This new service will be available mid 2017 starting with the Oregon location. After that all other AWS locations will follow. Not all information is yet public available but more information will follow in the coming months. I think this offering helps a lot of customer in their decision for moving forward with their current environment to the cloud. For more information about VMware Cloud, you can also visit http://www.mindjudo.nl. The last session of the day is From Zero to VMware Photon Platform. VMware also made announcements around Photon Platform. Photon Platform now supports Kubernetes next to Cloud foundry. Photon Platform also support now VSAN and NSX. VSAN is not required as local storage is needed. Shared storage is only needed for storing of the images that are used by the environment. Photon Platform uses vSphere as a base but doesn\u0026rsquo;t use functionality like drs and ha.This kind of functionality should be taken care of by the application itself. Photon is primarily a command line driven environment but there is also a basic guide available. If you want to try Photon Platform you can download it here . ","permalink":"https://wolkwacht.nl/posts/vmworld-2016-day-1-be-_-tomorrow/","summary":"\u003cp\u003e\u003cimg loading=\"lazy\" src=\"/images/2016/wp-image-343263341jpg-768x286.jpg\"\u003e\nToday is the first day of VMworld 2016. The day starts with the general  keynote session. First speaker during the general session was Jean-Pierre Brulard, general manager EMEA. He started with some statistics. This VMworld there are 10k attendees, 300 sessions. This is the first time that VMworld and a AirWatch connect are held together. The second speaker was Pat Gelsinger,CEO. Pat spoke about the world in which you control nothing but are in charge. This is a good base for the hybrid cloud. Pat talked about Digital transformation, which is according to him the new buzzword. He said he rather used Digital Business as also legacy applications are still there. Pat also announced vSphere 6.5 and VSAN 6.5. After that Pat talked about the cross cloud architecture. Part of it is the cross cloud service consisting of two parts, Cloud Foundation and a set of SAAS applications which will become available in 2017. These SAAS applications will offer you cloud control over your cross cloud environment. Pat also addressed the AWS announcement. More details below in this post and other posts to come. After the general session I did a quick walk around at the Solution Exchange which also openend today.\n\u003cimg alt=\"Solution Exchange\" loading=\"lazy\" src=\"/images/2016/img_20161018_143428-768x432.jpg\"\u003e\nAfter that I went for an early lunch because I didn\u0026rsquo;t want to miss the session VMware Cloud on AWS. Last week AWS and VMware announced this new partnership. This partnership was also mentioned during the keynote and also yesterday during Partner sessions. VMware Cloud on AWS gives customers the possibility to run their workloads in the cloud with a lot of the benefits that AWS has to offer. One of the benefits is elastic drs, which causes in times of shortage of resources to automatic provision an extra host. This includes cpu,  memory and storage. VMware Cloud is based on vSphere 6.5, NSX and VSAN.VMware Cloud will be managed and operated by VMware people on AWS hardware.  This new service will be available mid 2017 starting with the Oregon location. After that all other AWS locations will follow. Not all information is yet public available but more information will follow in the coming months. I think this offering helps a lot of customer in their decision for moving forward with their current environment to the cloud.\n\u003cimg alt=\"VMware on AWS\" loading=\"lazy\" src=\"/images/2016/wp-image-840977815jpg-768x435.jpg\"\u003e\nFor more information about VMware Cloud, you can also visit \u003ca href=\"http://www.mindjudo.nl\"\u003ehttp://www.mindjudo.nl\u003c/a\u003e. The last session of the day is From Zero to VMware Photon Platform. VMware also made announcements around Photon Platform. Photon Platform now supports Kubernetes next to Cloud foundry.  Photon Platform also support now VSAN and NSX. VSAN is not required as local storage is needed. Shared storage is only needed for storing of the images that are used by the environment. Photon Platform uses vSphere as a base but doesn\u0026rsquo;t use functionality like drs and ha.This kind of functionality should be taken care of by the application itself. Photon is primarily a command line driven environment but there is also a basic guide available. If you want to try Photon Platform you can download it \u003ca href=\"https://github.com/vmware/photon-controller/wiki\"\u003ehere\u003c/a\u003e .\n\u003cimg alt=\"Photon platform\" loading=\"lazy\" src=\"/images/2016/wp-image-2143137082jpg-768x432.jpg\"\u003e\u003c/p\u003e","title":"VMworld 2016 Day 1, be_Tomorrow"},{"content":" Today was the second day of VMworld, which started with the general session. First speaker was Sanjay Poonen. Sanjay talked about the digital transformation that is already happening in healthcare and education and how the vision of VMware fits in this. After that Sanjay spoke about Workspace One. VMware Workspace ONE is a simple and secure enterprise platform that delivers and manages any app on any device by integrating identity, application and enterprise mobility management. It is available as a cloud service or for on-premises deployment. Ray \u0026lsquo;o Farrel was the second speaker on stage. His main topic was SDDC and the components it consists of within VMware, vSphere, VSAN, NSX and vRealize . He focused first on vSphere 6.5 and all the new features it offers. More info can be found here. Yan Bing Lee came on stage to talk about VSAN 6.5 and the new features it offers but also what was in the beta for successor of the just released version. More information on what is new in VSAN 6.5 can be found here and on the site of Duncan Epping. After Yan Bing, Rajiv Ramaswami came on stage to talk about NSX and the challenges that NSX helps to address. These challenges are around security, automation and application continuity. Last speaker of the general session was Kit Colbert. Kit talked about Photon Platform. Photon Platform delivers core IaaS capabilities including VMs, Networks, and Persistent Disks on-demand to developers. Resources are provisioned quickly and reliably, supporting the needs of devops tools that programmatically allocate resources at scale. Photon Platform enables you to deliver Kubernetes as a Service to multiple tenants from a single shared pool of hardware. Each tenant gets access to API, CLI and GUI tools which allow them to provision dedicated Kubernetes clusters on the fly. Users get a dedicated kubernetes cluster with strong isolation from other tenants. Photon Platform automates the provisioning and high availability of these clusters, automatically replacing failed nodes with no human intervention.The Kubernetes service will become available in Q4 of 2016. VMware will offer end to end support for Kubernetes running on Photon Platform. I will post more on Cloud Native Applications and specific Photon Platform in other post. After the general session I attended two other session. First session was VMware Cloud on AWS - a Closer Look . In this session the technical background behind VMware Cloud on AWS. Important takeaways of this sessions are:\nvCenter is used to manage the environment NSX is not needed on the local side but is preferred VMware updates, manages and supports the VMware Cloud on AWS You will get one bill for VMware Cloud and one for the AWS Services The second session I attended vSphere 6.x Host Resource Deep Dive. This presentation was given by Frank Denneman and Niels Hagoort. First Frank spoke about the cpu and memory architecture and the impact it has the choice you make for your vm\u0026rsquo;s. Statement of Frank, _Its not all about best performance but most of the time about consistent performance. That\u0026rsquo;s not so easy so get. _After memory and cpu, Frank talked about storage, How far away is your data. The session was completed by Niels who addressed the network side. ","permalink":"https://wolkwacht.nl/posts/vmworld-2016-day-2/","summary":"\u003cp\u003e\u003cimg alt=\"VMworld 2016\" loading=\"lazy\" src=\"/images/2016/img_20161019_112417.jpg\"\u003e\nToday was the second day of VMworld, which started with the general session.  First speaker was Sanjay Poonen. Sanjay talked about the digital transformation that is already happening in healthcare and education and how the vision of VMware fits in this. After that Sanjay spoke about Workspace One. VMware Workspace ONE is a simple and secure enterprise platform that delivers and manages any app on any device by integrating identity, application and enterprise mobility management. It is available as a cloud service or for on-premises deployment. Ray \u0026lsquo;o Farrel was the second speaker on stage. His main topic was SDDC  and the components it consists of within VMware, vSphere, VSAN, NSX and vRealize . He focused first on vSphere 6.5 and all the new features it offers. More info can be found \u003ca href=\"https://blogs.vmware.com/vsphere/2016/10/whats-new-in-vsphere-6-5-vcenter-server.html\"\u003ehere\u003c/a\u003e.\nYan Bing Lee came on stage to talk about VSAN 6.5 and the new features it offers but also what was in the beta for successor of the just released version. More information on what is new in VSAN 6.5 can be found \u003ca href=\"http://www.vmware.com/products/whats-new-virtual-san.html\"\u003ehere\u003c/a\u003e and on the site of \u003ca href=\"http://www.yellow-bricks.com/2016/10/18/new-virtual-san-6-5/\"\u003eDuncan Epping\u003c/a\u003e. After Yan Bing, Rajiv Ramaswami came on stage to talk about NSX and the challenges that NSX helps to address. These challenges are around security, automation and application continuity. Last speaker of the general session was Kit Colbert. Kit talked about Photon Platform. Photon Platform delivers core IaaS capabilities including VMs, Networks, and Persistent Disks on-demand to developers. Resources are provisioned quickly and reliably, supporting the needs of devops tools that programmatically allocate resources at scale.\n\u003cimg loading=\"lazy\" src=\"/images/2016/wp-1476880864965-768x427.jpg\"\u003e\nPhoton Platform enables you to deliver Kubernetes as a Service to multiple tenants from a single shared pool of hardware. Each tenant gets access to API, CLI and GUI tools which allow them to provision dedicated Kubernetes clusters on the fly. Users get a dedicated kubernetes cluster with strong isolation from other tenants. Photon Platform automates the provisioning and high availability of these clusters, automatically replacing failed nodes with no human intervention.The Kubernetes service will become available in Q4 of 2016. VMware will offer end to end support for Kubernetes running on Photon Platform.\n\u003cimg loading=\"lazy\" src=\"/images/2016/wp-1476880912480-768x430.jpg\"\u003e\nI will post more on Cloud Native Applications and specific Photon Platform in other post. After the general session I attended two other session. First session was VMware Cloud on AWS - a Closer Look . In this session the technical background behind VMware Cloud  on AWS. Important takeaways of this sessions are:\u003c/p\u003e","title":"VMworld 2016 Day 2, be_Tomorrow"},{"content":"[ Today VMworld 2016 started for me with the partner and tam day. I already checked in yesterday, but still it was an early start as I had my first session at 8:30 in the morning. During the day I was also able to change my schedule and put some sessions about VMware and AWS in I started with Building \u0026amp; Enabling a Hybrid Cloud with vCloud director - a Perspective for Service Providers. The session was focused on the use of vCloud director, what is available now and what is on the roadmap. After that I attended Route To Market Session - vCloud Network Service Provider Partners. This session focused on the possibilities that VMware offers to their partners. After this session was the general session. A lot of interessting announcements will come tomorrow during the general keynote session.Also there was some basic information about the AWS partnership. After the general session it was time for the lunch. The food was as alwys very good. The lunch was in the same room as the hands-on labs, the partner lounge and the alumni lounge. After the lunch and visiting the lounge space and the VMware store I attended the Building Value of Data Center Virtualization and Hybrid Cloud Extensibility. This was a quick talk about what were the possibilities for cost savings. The last session of the day is The Practical Path to NSX for Partners.This session was about use cases for NSX and the vision of VMware around NSX. The day closes with the Partner Exchange Networking reception. Looking forward to the sessions in the coming days when the event really starts.\n","permalink":"https://wolkwacht.nl/posts/vmworld-2016-partner-and-tam-day/","summary":"\u003cp\u003e[\u003cimg alt=\"Gran Fira\" loading=\"lazy\" src=\"/images/2016/wp-image-1277360941jpg-768x576.jpeg\"\u003e\nToday VMworld 2016 started for me with the partner and tam day. I already checked in yesterday, but still it was an early start as I had my first session at 8:30 in the morning. During the day I was also able to change my schedule and put some sessions about VMware and AWS in I started with Building \u0026amp; Enabling a Hybrid Cloud with vCloud director - a Perspective for Service Providers. The session was focused on the use of vCloud director, what is available now and what is on the roadmap. After that I attended Route To Market Session - vCloud Network Service Provider Partners. This session focused on the possibilities that VMware offers to their partners.\n\u003cimg alt=\"session\" loading=\"lazy\" src=\"/images/2016/wp-image-1457197341jpg.jpeg\"\u003e After this session was the general session. A lot of interessting announcements will come tomorrow during the general keynote session.Also there was some basic information about the AWS partnership.\n\u003cimg alt=\"self paced labs\" loading=\"lazy\" src=\"/images/2016/wp-image-1469888896jpg-768x576.jpeg\"\u003e\nAfter the general session it was time for the lunch. The food was as alwys very good. The lunch was in the same room as the hands-on labs, the partner lounge and the alumni lounge. After the lunch and visiting the lounge space and the VMware store I attended the Building Value of Data Center Virtualization and Hybrid Cloud Extensibility. This was a quick talk about what were the possibilities for cost savings. The last session of the day is The Practical Path to NSX for Partners.This session was about use cases for NSX and the vision  of VMware around NSX. The day closes with the Partner Exchange Networking reception. \u003cimg alt=\"networking reception\" loading=\"lazy\" src=\"/images/2016/wp-image-1136664213jpg-768x574.jpg\"\u003e Looking forward to the sessions in  the coming days when the event really starts.\u003c/p\u003e","title":"VMworld 2016 Partner and TAM day"},{"content":"Last Tuesday, I was together with two colleagues at the AWS Enterprise summit in the World Forum in The Hague. This was for me the second AWS summit that I attended. The event started with a keynote from Ian Massingham, Chief Evangelist (EMEA), AWS. Ian spoke about the impact that cloud has on enterprise IT. After the keynote, there was a plenary session by Hans Koolen, Senior Director, Philips IT Global Services. He posed the statement that a private cloud doesn\u0026rsquo;t exist. After the keynote and the plenary session, there was lunch break and a possibility to visit the Partner \u0026amp; Solutions Expo. During the whole event the Hands-On labs were also available. The afternoon started with two breakout tracks, management and technical. Both of the tracks consisted of 4 sessions. I followed three sessions in the management track and one session in the technical track. The first track that I followed, from the Management Track, was the session Starting your Journey to the Cloud. Very interesting session, addressing the points which you had to pay attention to when starting the journey to the cloud. The slides can be found here. After this session, I followed a Technical Track session, End-user Computing on AWS. This session was about AWS WorkSpaces, running your desktop on AWS, and also about using AWS WorkMail. The slides can be found here. During the break I visited the Partner \u0026amp; Solution Expo and spoke some of the people. After the break there were two session left. I took both of the sessions from the Management track. First I attend the session Cost Optimization at Scale. This was an interesting session addressing the possibilities that AWS offers to optimize your cost. The speaker addressed amongst other things the tools that AWS offers, the difference between instances. He also spoke about the comparison between the cost of running your own data center compared to running the workloads in the AWS cloud. Slides of this presentations can found here. As last I followed the Information Security by Design in AWS session. This was also a very interesting session. This session was, amongst other things, about compliance, AWS assurance programs and AWS compliance certifications. Very useful information when talking to customers about the cloud. The day end with a network reception and again the possibility to speak to the people on the Partner \u0026amp; Solution expo. I especially liked the last two breakout sessions. I think they were very useful and I can use the knowledge in my job when advising businesses in their journey to the cloud.\n","permalink":"https://wolkwacht.nl/posts/aws-enterprise-summit/","summary":"\u003cp\u003eLast Tuesday, I was together with two colleagues at the AWS Enterprise summit in the World Forum in The Hague. This was for me the second AWS summit that I attended.  The event started with a keynote from Ian Massingham, Chief Evangelist (EMEA), AWS. Ian spoke about the impact that cloud has on enterprise IT. After the keynote, there was a plenary session by Hans Koolen, Senior Director, Philips IT Global Services. He posed the statement that a private cloud doesn\u0026rsquo;t exist.\n\u003cimg alt=\"img_20160921_101158\" loading=\"lazy\" src=\"/images/2016/IMG_20160921_101158-590x390.jpg\"\u003e\u003c/p\u003e","title":"AWS Enterprise Summit"},{"content":"Yesterday was the 3rd birthday of Docker. Docker Randstand Meetup organised two meetup sessions. I attended both of the sessions.\nDocker 3rd birthday celebration - Introduction to Docker Docker 3rd birthday plus book signing with Adrian Mouat The meetups were in the W in Amsterdam and sponsored by Microsoft. The Introduction to Docker meetup was basically a hands-on workshop where you were you the tutorials provide by Docker to create the birthday app. If you were not familiar with Docker, there was also a getting started tutorial in which you could get acquainted with Docker and the basic commands. After you had created the app you could upload it to Docker and show that you had participated in the birthday event. During the meetup there were also pastries and good coffee available. Overall a good organized meetup.\nThe second meetup consisted of two presentations. First presentations was by Arjan Schaaf – DevOps Architect and was about Docker Network performance in the public cloud. Arjan performed different tests onMicrosoft Azure and Amazone Web Services regarding network performance. The second presentation was by Adrian Mouat – Chief Scientist @ Container Solutions and had as subject Container Orchestration with Kubernetes, Docker Swarm and Mesos/ Marathon. Adrian discussed the features of the different products and also showed a demo of each of the products in which he showed the advantages and disadvantages of the different solutions After the presentations there was the possibility to get your book Using Docker, signed by Adrian Mouat. And of course there were drinks, snacks and the cutting of the birthday cake. All in all two very good meetups, well organized.\n","permalink":"https://wolkwacht.nl/posts/docker-randstand-meetup/","summary":"\u003cp\u003eYesterday was the 3rd birthday of Docker. Docker Randstand Meetup organised two meetup sessions. I attended both of the sessions.\u003c/p\u003e\n\u003cp\u003e\u003cimg alt=\"docker\" loading=\"lazy\" src=\"/images/2016/docker.jpg\"\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eDocker 3rd birthday celebration - Introduction to Docker\u003c/li\u003e\n\u003cli\u003eDocker 3rd birthday plus book signing with Adrian Mouat\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e  \u003cimg alt=\"IMG_20160324_135955\" loading=\"lazy\" src=\"/images/2016/IMG_20160324_135955-300x225.jpg\"\u003e\u003c/p\u003e\n\u003cp\u003eThe meetups were in the W in Amsterdam and sponsored by Microsoft. The Introduction to Docker meetup was basically a hands-on workshop where you were you the tutorials provide by Docker to create the birthday app. If you were not familiar with Docker, there was also a getting started tutorial in which you could get acquainted with Docker and the basic commands. After you had created the app you could upload it to Docker and show that you had participated in the birthday event. During the meetup there were also pastries and good coffee available. Overall a good organized meetup.\u003c/p\u003e","title":"Docker Randstad Meetup: March 2016"},{"content":"My VMware certifications were going to expire on the 12th of March, To keep that from happening I took and passed the VCP 6 - Data Center Virtualization delta exam. Since I only used selfstudy to achieve my VCP 5 certification I thought it was a smart idea to follow one of the suggested training courses. I attended the VMware vSphere: What\u0026rsquo;s New [V5.5 to V6] course. Next to that course I also attended the VMware vSphere: Certified Professional Exam Preparation Workshop [V1.0]. The training and the workshop are very helpfull towards the exam but not enough. I also installed the VMware ESXi 6.0 and VMware vCenter appliance in a test environment to get some hands on experience with the product. Useful links: VCP 6 Study guide VMware practive exam VMware product walkthrough View my VMware Data Center Virtualization credential on Credly.\n","permalink":"https://wolkwacht.nl/posts/vcp6dv/","summary":"\u003cp\u003eMy VMware certifications were going to expire on the 12th of March, To keep that from happening I took and passed the VCP 6 - Data Center Virtualization  delta exam. Since I only used selfstudy to achieve my VCP 5 certification I thought it was a smart idea to follow one of the suggested training courses. I attended the VMware vSphere: What\u0026rsquo;s New [V5.5 to V6] course. Next to that course I also attended the VMware vSphere: Certified Professional Exam Preparation Workshop [V1.0]. The training and the workshop are very helpfull towards the exam but not enough. I also installed the VMware ESXi 6.0 and VMware vCenter appliance in a test environment to get some hands on experience with the product. Useful links: \u003ca href=\"https://www.vladan.fr/\"\u003eVCP 6 Study guide\u003c/a\u003e \u003ca href=\"https://mylearn.vmware.com/mgrReg/plan.cfm?plan=64181\u0026amp;ui=www_cert\"\u003eVMware practive exam\u003c/a\u003e \u003ca href=\"https://featurewalkthrough.vmware.com/\"\u003eVMware product walkthrough\u003c/a\u003e  \u003c/p\u003e","title":"Passed my VCP6-DCV certification"},{"content":" About Wolkwacht\nJurgen Allewijn Principal Consultant · Microsoft MVP\nI help organizations turn complex cloud-native technology into platforms that are secure, resilient and practical to operate.\nConnect on LinkedIn Speaking profile Building technology that works in the real world Technology is where curiosity and practical problem-solving meet for me. I work across cloud architecture, hybrid environments and cloud-native platforms, with a particular focus on Microsoft Azure, Kubernetes, platform engineering and security.\nAs a Principal Consultant at Yuma, I help teams navigate cloud transformation from architecture through implementation. That means designing resilient systems, creating effective platform capabilities, and making sure reliability, automation and clarity remain part of the engineering conversation.\nPrincipal Consultant at Yuma Microsoft MVP since 2024 Azure, Kubernetes and Open Source Cloud and Datacenter Management Enterprise and Platform Security More than twenty years in IT My Microsoft journey began in 1999 with a Windows NT 4.0 Workstation certification. Since then, I have worked through the shifts from Novell networks and Active Directory to VMware, public cloud and cloud-native architecture. You can view my complete Microsoft certification transcript.\nCommunity and speaking Sharing knowledge is a fundamental part of my work. I speak at internal and public events about Azure, Kubernetes, GitOps, cloud-native operations, security and platform engineering. Wolkwacht is where I turn those experiences into practical articles, architectural perspectives and lessons learned in the field.\nExplore my public speaking sessions or visit my Microsoft MVP profile.\nWhy the name Wolkwacht? Wolk means cloud. Wacht means watch, guard or lookout. Together they describe an approach to technology: stay observant, remain curious and be ready for what comes next.\nWolkwacht is rooted in Dutch heritage and shaped by modern cloud engineering. It reflects the work of watching over cloud platforms, guiding organizations through change and keeping an eye on the fast-moving cloud-native landscape.\nBeyond the keyboard Away from work, I enjoy walking with my dog, playing games such as World of Warcraft and Baldur\u0026rsquo;s Gate 3, and flying drones. Aerial photography gives me a different perspective on castles, ancient fortifications and the landscapes around them.\n","permalink":"https://wolkwacht.nl/about/","summary":"\u003csection class=\"about-hero\" aria-labelledby=\"about-title\"\u003e\n  \u003cdiv class=\"about-hero__portrait\"\u003e\n    \u003cimg src=\"/images/profile/jurgenallewijn.jpg\" width=\"360\" height=\"360\" alt=\"Jurgen Allewijn\"\u003e\n  \u003c/div\u003e\n  \u003cdiv class=\"about-hero__content\"\u003e\n    \u003cp class=\"about-hero__eyebrow\"\u003eAbout Wolkwacht\u003c/p\u003e\n    \u003ch1 id=\"about-title\"\u003eJurgen Allewijn\u003c/h1\u003e\n    \u003cp class=\"about-hero__role\"\u003ePrincipal Consultant · Microsoft MVP\u003c/p\u003e\n    \u003cp class=\"about-hero__lede\"\u003eI help organizations turn complex cloud-native technology into platforms that are secure, resilient and practical to operate.\u003c/p\u003e\n    \u003cdiv class=\"about-hero__links\"\u003e\n      \u003ca class=\"about-hero__primary\" href=\"https://www.linkedin.com/in/jallewijn/\" target=\"_blank\" rel=\"noopener noreferrer\"\u003eConnect on LinkedIn\u003c/a\u003e\n      \u003ca href=\"https://sessionize.com/jurgen-allewijn/\" target=\"_blank\" rel=\"noopener noreferrer\"\u003eSpeaking profile\u003c/a\u003e\n    \u003c/div\u003e\n  \u003c/div\u003e\n\u003c/section\u003e\n\u003ch2 id=\"building-technology-that-works-in-the-real-world\"\u003eBuilding technology that works in the real world\u003c/h2\u003e\n\u003cp\u003eTechnology is where curiosity and practical problem-solving meet for me. I work across cloud architecture, hybrid environments and cloud-native platforms, with a particular focus on Microsoft Azure, Kubernetes, platform engineering and security.\u003c/p\u003e","title":"About"},{"content":" Wolkwacht · Aerial field notes\nA different perspective Castles, fortifications and landscapes reveal a different story when seen from above.\nArchitecture Landscape History from above Stories written into the landscape Flying gives me a different perspective on familiar places. I am drawn to castles, fortifications and landscapes—subjects whose shapes, boundaries and history become clearer from the air. This is the quieter, visual side of Wolkwacht: field notes made with altitude, patience and changing light.\nThe aircraft I currently fly two compact DJI drones. Each has a distinct role: one for spontaneous exploration and one for more deliberate image-making.\nDJI Flip Fly More Combo The DJI Flip is my compact option for spontaneous flights. Its low weight, enclosed propellers and quick setup make it useful when the opportunity matters more than carrying a complete camera kit.\nDJI Mini 5 Pro Fly More Combo The DJI Mini 5 Pro is my camera-first option. Its larger sensor, flexible gimbal and obstacle sensing make it better suited to detailed landscapes, difficult light and carefully composed photographs.\nFlight archive An evolving collection of views from my flights. Select a photograph to open the full-resolution image.\n","permalink":"https://wolkwacht.nl/drones/","summary":"\u003csection class=\"drone-hero\" style=\"--drone-hero-image: url('/drones/dji_fly_20260809_090452_0033_1786264338963_pano_hu_e2b9d443fad94abb.jpg')\" aria-labelledby=\"drone-hero-title\"\u003e\n  \u003cdiv class=\"drone-hero__content\"\u003e\n    \u003cp class=\"drone-hero__eyebrow\"\u003eWolkwacht · Aerial field notes\u003c/p\u003e\n    \u003ch1 id=\"drone-hero-title\"\u003eA different perspective\u003c/h1\u003e\n    \u003cp class=\"drone-hero__lede\"\u003eCastles, fortifications and landscapes reveal a different story when seen from above.\u003c/p\u003e\n    \u003cdiv class=\"drone-hero__details\" aria-label=\"Gallery themes\"\u003e\n      \u003cspan\u003eArchitecture\u003c/span\u003e\n      \u003cspan\u003eLandscape\u003c/span\u003e\n      \u003cspan\u003eHistory from above\u003c/span\u003e\n    \u003c/div\u003e\n  \u003c/div\u003e\n\u003c/section\u003e\n\n\u003ch2 id=\"stories-written-into-the-landscape\"\u003eStories written into the landscape\u003c/h2\u003e\n\u003cp\u003eFlying gives me a different perspective on familiar places. I am drawn to castles, fortifications and landscapes—subjects whose shapes, boundaries and history become clearer from the air. This is the quieter, visual side of Wolkwacht: field notes made with altitude, patience and changing light.\u003c/p\u003e","title":"Aerial Field Notes"},{"content":"Thanks for subscribing to Cloud Signals, the Wolkwacht newsletter.\nButtondown has sent a confirmation email to the address you entered. Open that email and click the confirmation link to complete your subscription.\nIf it does not arrive within a few minutes, check your spam or promotions folder.\nReturn to Wolkwacht\n","permalink":"https://wolkwacht.nl/newsletter/check-your-inbox/","summary":"\u003cp\u003eThanks for subscribing to \u003cstrong\u003eCloud Signals\u003c/strong\u003e, the Wolkwacht newsletter.\u003c/p\u003e\n\u003cp\u003eButtondown has sent a confirmation email to the address you entered. Open that\nemail and click the confirmation link to complete your subscription.\u003c/p\u003e\n\u003cp\u003eIf it does not arrive within a few minutes, check your spam or promotions folder.\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"/\"\u003eReturn to Wolkwacht\u003c/a\u003e\u003c/p\u003e","title":"Check your inbox"},{"content":"If you\u0026rsquo;d like to get in touch, reaching out to me on LinkedIn is the simplest way. Feel free to also follow or send me a message on Bluesky or Mastodon. I look forward to connecting with you!\nLinkedIn GitHub Bluesky Mastodon Medium ","permalink":"https://wolkwacht.nl/contact/","summary":"\u003cp\u003eIf you\u0026rsquo;d like to get in touch, reaching out to me on LinkedIn is the simplest way. Feel free to also follow or send me a message on Bluesky or Mastodon. I look forward to connecting with you!\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"https://www.linkedin.com/in/jallewijn/\"\u003eLinkedIn\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/Jurgen-Allewijn\"\u003eGitHub\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://bsky.app/profile/jurgenallewijn.nl\"\u003eBluesky\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://hachyderm.io/@jurgen\"\u003eMastodon\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://medium.com/@jurgenallewijn\"\u003eMedium\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e","title":"Contact"},{"content":"Website analytics Wolkwacht uses Umami Analytics to understand how visitors use this website. This includes aggregated information such as page views, referring websites, browser and device types, and approximate geographic location.\nThe standard Umami tracker does not use cookies and is not intended to identify individual visitors or track visitors across different websites.\nAnalytics information is used solely to understand website traffic and improve the content on Wolkwacht.\nFor more information, see the Umami privacy policy.\nNewsletter Wolkwacht uses Buttondown to manage newsletter subscriptions and send newsletter emails. When you subscribe, Buttondown processes the information you enter in the signup form, such as your email address, as well as technical information needed to provide and secure the service.\nThis information is used only to manage your subscription and send the newsletter. You can unsubscribe at any time using the link in any newsletter email.\nExternal links This website contains links to external websites. Wolkwacht is not responsible for the privacy practices or content of those websites.\nContact For questions about privacy or the processing of information through this website, contact me through the channels listed on the Contact page.\nChanges This privacy statement may be updated when the website or the services it uses change.\nLast updated: September 6, 2026.\n","permalink":"https://wolkwacht.nl/privacy/","summary":"\u003ch2 id=\"website-analytics\"\u003eWebsite analytics\u003c/h2\u003e\n\u003cp\u003eWolkwacht uses Umami Analytics to understand how visitors use this website.\nThis includes aggregated information such as page views, referring websites,\nbrowser and device types, and approximate geographic location.\u003c/p\u003e\n\u003cp\u003eThe standard Umami tracker does not use cookies and is not intended to identify\nindividual visitors or track visitors across different websites.\u003c/p\u003e\n\u003cp\u003eAnalytics information is used solely to understand website traffic and improve\nthe content on Wolkwacht.\u003c/p\u003e\n\u003cp\u003eFor more information, see the\n\u003ca href=\"https://umami.is/privacy\"\u003eUmami privacy policy\u003c/a\u003e.\u003c/p\u003e","title":"Privacy"},{"content":"Your subscription is confirmed. Welcome to Cloud Signals!\nYou will receive occasional practical insights about Azure, Kubernetes, cloud security, and digital sovereignty. No noise, and you can unsubscribe at any time using the link in every email.\nIn the meantime, you can browse the latest articles or explore topics by tag.\n","permalink":"https://wolkwacht.nl/newsletter/welcome/","summary":"\u003cp\u003eYour subscription is confirmed. Welcome to \u003cstrong\u003eCloud Signals\u003c/strong\u003e!\u003c/p\u003e\n\u003cp\u003eYou will receive occasional practical insights about Azure, Kubernetes, cloud\nsecurity, and digital sovereignty. No noise, and you can unsubscribe at any\ntime using the link in every email.\u003c/p\u003e\n\u003cp\u003eIn the meantime, you can \u003ca href=\"/posts/\"\u003ebrowse the latest articles\u003c/a\u003e or explore\n\u003ca href=\"/tags/\"\u003etopics by tag\u003c/a\u003e.\u003c/p\u003e","title":"Welcome to Cloud Signals"}]