When Kubernetes Lies: Seeing the Truth Inside AKS with Inspektor Gadget
Cloud platforms are built on abstractions.
We tell developers that services talk to services. We tell security teams that network policies enforce isolation. We tell architects that identity is the new perimeter and that zero trust ensures nothing moves unless explicitly allowed. In Azure Kubernetes Service, these abstractions become even more powerful. Managed control planes, private clusters, Cilium networking, workload identity, Azure Firewall, private endpoints, and policy enforcement create the feeling of a highly controlled, highly predictable platform.
And most of the time, that feeling is justified.
Until the day it isn’t.
Beneath every abstraction, whether it’s a YAML file, Azure policy, or other layer, there’s always a Linux kernel executing system calls. Containers function as processes; pods are simply cgroups; and network policies boil down to packet-filtering decisions. When issues arise unexpectedly in production, it’s the kernel that holds the definitive truth.
The problem is that most cloud tooling never shows you that truth.
Metrics reveal symptoms, logs provide interpretations, network tools display flows between pods, and security tools show alerts. However, when it comes to answering the crucial question, what actually happened?, these layers of abstraction often become obstacles rather than helpful solutions. This is where kernel-level observability changes how you operate Azure Kubernetes Service.
This is where Inspektor Gadget becomes one of the most valuable tools you can add to an AKS platform.
The gap between architecture and reality
Modern AKS environments consist of layered controls making them complex systems. A single production cluster might incorporate Azure CNI or overlay networking, Cilium or kube-proxy, private DNS zones, outbound routing via NAT Gateway or Azure Firewall, workload identity for Azure resources, admission control policies, and various observability pipelines feeding into Azure Monitor and Log Analytics.
From a design perspective, this aligns perfectly with our goals. We achieve defense-in-depth, least privilege, segmentation, auditability, and centralized monitoring. The architecture diagrams are clear, policies appear to be enforced, and dashboards indicate a healthy system.
However, runtime behavior is distinct from architecture; it refers to what processes actually perform.
When a compromised container opens a shell and begins probing internal services, this activity might not be clearly visible in application logs. Similarly, if a third-party library makes silent calls to external endpoints, metrics alone may not reveal the reason. Additionally, a misconfiguration that causes a workload to resolve public DNS instead of private DNS can introduce issues several layers removed from the initial cause.
At larger scales, these blind spots become more significant. In regulated settings, their importance grows even further. Regulations such as NIS2, DORA, and GDPR now require organizations to show operational control, not merely their intended configuration. It’s no longer sufficient to understand what your cluster is designed to do; you must also provide proof of its actual behavior.
That evidence lives in the kernel.
What Inspektor Gadget really is
Inspektor Gadget is a sandbox project under the Cloud Native Computing Foundation that aims to introduce kernel-level observability into Kubernetes. The official project website can be found at: https://www.inspektor-gadget.io/
The source code and releases are maintained on GitHub: https://github.com/inspektor-gadget/inspektor-gadget
Inspektor Gadget leverages eBPF, a Linux technology that enables small programs to run safely within the kernel. These programs are linked to specific kernel events, such as process execution, network connections, DNS resolution, file access, or system calls. Since the observation occurs directly within the kernel, visibility is precise and comprehensive, unaffected by layers above.
What sets Inspektor Gadget apart from standard eBPF tools is its Kubernetes awareness. The kernel-collected events are enhanced with Kubernetes-specific details, including namespace, pod name, container name, and labels. This allows you to identify which workload created processes and sockets, rather than just seeing anonymous data.
On AKS, this is especially important. The best practice in Azure is to avoid allowing SSH access to nodes, and changing the host is not recommended. Inspektor Gadget runs as a privileged DaemonSet, functioning within the supported Kubernetes security framework while still exposing kernel-level events.
This provides an uncommon level of insight into cloud environments: direct visibility into container activity. To install Inspektor Gadget on Azure Kubernetes Service, you need a Linux-based AKS cluster and the necessary permissions to deploy resources and run cluster-wide tools. Once your local setup is connected to the cluster, install Inspektor Gadget using Helm.
Example script for deploying Azure Kubernetes with Azure CNI Overlay and Cilium:
#!/usr/bin/env bash
set -euo pipefail
# -----------------------------------------------------------------------------
# AKS + Azure CNI Overlay + Cilium dataplane + ACNS (AKS-managed Hubble Relay)
# + Azure Linux node OS
#
# References:
# - Create with ACNS + Cilium dataplane: https://learn.microsoft.com/azure/aks/how-to-configure-container-network-logs
# - ACNS overview: https://learn.microsoft.com/azure/aks/advanced-container-networking-services-overview
# - Azure Linux 3 guidance: https://learn.microsoft.com/azure/azure-linux/how-to-enable-azure-linux-3
# -----------------------------------------------------------------------------
# ====== EDIT THESE ======
LOCATION="${LOCATION:-swedencentral}" # Choose an AKS-supported region (e.g. swedensouth, westeurope, northeurope)
RESOURCE_GROUP="${RESOURCE_GROUP:-rg-aks-acns-cilium-se}"
CLUSTER_NAME="${CLUSTER_NAME:-aks-acns-cilium-se}"
NODE_COUNT="${NODE_COUNT:-2}"
NODE_VM_SIZE="${NODE_VM_SIZE:-Standard_D4s_v5}"
POD_CIDR="${POD_CIDR:-192.168.0.0/16}"
DISABLE_SSH="${DISABLE_SSH:-true}" # Set to 'true' to disable SSH access to nodes
SKIP_KUBECONFIG="${SKIP_KUBECONFIG:-false}" # Set to 'true' to skip downloading kubeconfig
KUBECONFIG_PATH="${KUBECONFIG_PATH:-${HOME}/.kube/config}" # Path to save kubeconfig
# Kubernetes version:
# ACNS+Cilium features are supported on newer versions (docs show 1.33+ examples).
# Set to an available version in your region if you want to pin.
K8S_VERSION="${K8S_VERSION:-}"
# Azure Linux:
# --os-sku AzureLinux will default to Azure Linux 3.0 on sufficiently new AKS versions.
# If you MUST pin AzureLinux3 explicitly (without changing k8s version), check your CLI supports it.
OS_SKU="${OS_SKU:-AzureLinux}"
# =======================
require() { command -v "$1" >/dev/null 2>&1 || { echo "Missing dependency: $1" >&2; exit 1; }; }
require az
require kubectl
require curl
require tar
require sha256sum
echo "==> Azure login check"
az account show >/dev/null 2>&1 || az login >/dev/null
echo "==> Preflight: validate AKS availability in ${LOCATION}"
if ! az aks get-versions --location "${LOCATION}" -o none 2>/dev/null; then
echo "AKS is not available or the location is invalid: ${LOCATION}" >&2
echo "Try another region, e.g. swedensouth, or check your subscription permissions." >&2
exit 1
fi
echo "==> Create resource group: ${RESOURCE_GROUP} (${LOCATION})"
az group create --name "${RESOURCE_GROUP}" --location "${LOCATION}" -o none
echo "==> Create AKS cluster (Azure CNI overlay + Cilium dataplane + ACNS + Azure Linux)"
AKS_CREATE_ARGS=(
--name "${CLUSTER_NAME}"
--resource-group "${RESOURCE_GROUP}"
--location "${LOCATION}"
--node-count "${NODE_COUNT}"
--node-vm-size "${NODE_VM_SIZE}"
--network-plugin azure
--network-plugin-mode overlay
--pod-cidr "${POD_CIDR}"
--network-dataplane cilium
--enable-acns
--os-sku "${OS_SKU}"
)
if [[ "${DISABLE_SSH}" != "true" ]]; then
AKS_CREATE_ARGS+=( --generate-ssh-keys )
fi
if [[ -n "${K8S_VERSION}" ]]; then
AKS_CREATE_ARGS+=( --kubernetes-version "${K8S_VERSION}" )
fi
az aks create "${AKS_CREATE_ARGS[@]}" -o none
echo "==> Get kubeconfig"
if [[ "${SKIP_KUBECONFIG}" == "true" ]]; then
echo " (Skipping kubeconfig download)"
else
mkdir -p "$(dirname "${KUBECONFIG_PATH}")"
az aks get-credentials \
--resource-group "${RESOURCE_GROUP}" \
--name "${CLUSTER_NAME}" \
--file "${KUBECONFIG_PATH}" \
--overwrite-existing -o none
echo " Kubeconfig saved to: ${KUBECONFIG_PATH}"
export KUBECONFIG="${KUBECONFIG_PATH}"
fi
echo "==> Verify nodes (ensure OS is Azure Linux)"
kubectl get nodes -o wide
echo "==> Verify Cilium pods exist (AKS-managed Cilium runs in kube-system)"
kubectl -n kube-system get pods | grep -E "^cilium-" >/dev/null
echo "==> Verify ACNS-managed Hubble Relay is running"
# Microsoft docs use label k8s-app=hubble-relay
kubectl get pods -n kube-system -l k8s-app=hubble-relay -o wide
echo "==> Install Hubble CLI (as per Microsoft docs)"
# Pin the version shown in docs; update if your cluster components require a newer one.
HUBBLE_VERSION="${HUBBLE_VERSION:-v1.16.3}"
HUBBLE_ARCH="amd64"
if [[ "$(uname -m)" == "aarch64" || "$(uname -m)" == "arm64" ]]; then HUBBLE_ARCH="arm64"; fi
tmpdir="$(mktemp -d)"
trap 'rm -rf "${tmpdir}"' EXIT
(
cd "${tmpdir}"
curl -L --fail --remote-name-all \
"https://github.com/cilium/hubble/releases/download/${HUBBLE_VERSION}/hubble-linux-${HUBBLE_ARCH}.tar.gz" \
"https://github.com/cilium/hubble/releases/download/${HUBBLE_VERSION}/hubble-linux-${HUBBLE_ARCH}.tar.gz.sha256sum"
sha256sum --check "hubble-linux-${HUBBLE_ARCH}.tar.gz.sha256sum"
# Install to /usr/local/bin if writable, else ~/.local/bin
install_dir="/usr/local/bin"
if [[ ! -w "${install_dir}" ]]; then
install_dir="${HOME}/.local/bin"
mkdir -p "${install_dir}"
export PATH="${install_dir}:${PATH}"
fi
tar xzvfC "hubble-linux-${HUBBLE_ARCH}.tar.gz" "${install_dir}"
)
echo "==> Port-forward Hubble Relay (leave this running in a separate terminal when observing flows)"
echo " Command:"
echo " kubectl port-forward -n kube-system svc/hubble-relay --address 127.0.0.1 4245:443"
echo
echo "==> After port-forward is running, you can observe flows with:"
echo " hubble status"
echo " hubble observe --follow"
echo
# -----------------------------------------------------------------------------
# Optional: Deploy Hubble UI (Microsoft manifest pattern) + port-forward
# This expects the secret 'hubble-relay-client-certs' to exist (ACNS-managed Hubble creates it).
# -----------------------------------------------------------------------------
echo "==> Deploy Hubble UI (optional but recommended)"
cat <<'YAML' | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
name: hubble-ui
namespace: kube-system
labels:
app.kubernetes.io/part-of: retina
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: hubble-ui
labels:
app.kubernetes.io/part-of: retina
rules:
- apiGroups: [""]
resources: ["namespaces", "pods", "services"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: hubble-ui
labels:
app.kubernetes.io/part-of: retina
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: hubble-ui
subjects:
- kind: ServiceAccount
name: hubble-ui
namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
name: hubble-ui-nginx
namespace: kube-system
data:
nginx.conf: |
server {
listen 8081;
server_name _;
gzip on;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
root /usr/share/nginx/html;
location / {
try_files $uri $uri/ /index.html;
}
location /healthz {
add_header Content-Type text/plain;
return 200 'ok';
}
}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: hubble-ui
namespace: kube-system
labels:
k8s-app: hubble-ui
app.kubernetes.io/name: hubble-ui
app.kubernetes.io/part-of: retina
spec:
replicas: 1
selector:
matchLabels:
k8s-app: hubble-ui
template:
metadata:
labels:
k8s-app: hubble-ui
app.kubernetes.io/name: hubble-ui
app.kubernetes.io/part-of: retina
spec:
serviceAccountName: hubble-ui
automountServiceAccountToken: true
nodeSelector:
kubernetes.io/os: linux
containers:
- name: frontend
image: mcr.microsoft.com/oss/cilium/hubble-ui:v0.12.2
imagePullPolicy: Always
ports:
- name: http
containerPort: 8081
livenessProbe:
httpGet:
path: /healthz
port: 8081
readinessProbe:
httpGet:
path: /
port: 8081
volumeMounts:
- name: hubble-ui-nginx-conf
mountPath: /etc/nginx/conf.d/default.conf
subPath: nginx.conf
- name: tmp-dir
mountPath: /tmp
- name: backend
image: mcr.microsoft.com/oss/cilium/hubble-ui-backend:v0.12.2
imagePullPolicy: Always
env:
- name: EVENTS_SERVER_PORT
value: "8090"
- name: FLOWS_API_ADDR
value: "hubble-relay:443"
- name: TLS_TO_RELAY_ENABLED
value: "true"
- name: TLS_RELAY_SERVER_NAME
value: ui.hubble-relay.cilium.io
- name: TLS_RELAY_CA_CERT_FILES
value: /var/lib/hubble-ui/certs/hubble-relay-ca.crt
- name: TLS_RELAY_CLIENT_CERT_FILE
value: /var/lib/hubble-ui/certs/client.crt
- name: TLS_RELAY_CLIENT_KEY_FILE
value: /var/lib/hubble-ui/certs/client.key
livenessProbe:
httpGet:
path: /healthz
port: 8090
readinessProbe:
httpGet:
path: /healthz
port: 8090
ports:
- name: grpc
containerPort: 8090
volumeMounts:
- name: hubble-ui-client-certs
mountPath: /var/lib/hubble-ui/certs
readOnly: true
volumes:
- name: hubble-ui-nginx-conf
configMap:
name: hubble-ui-nginx
- name: tmp-dir
emptyDir: {}
- name: hubble-ui-client-certs
projected:
defaultMode: 0400
sources:
- secret:
name: hubble-relay-client-certs
items:
- key: tls.crt
path: client.crt
- key: tls.key
path: client.key
- key: ca.crt
path: hubble-relay-ca.crt
---
apiVersion: v1
kind: Service
metadata:
name: hubble-ui
namespace: kube-system
labels:
k8s-app: hubble-ui
app.kubernetes.io/name: hubble-ui
app.kubernetes.io/part-of: retina
spec:
type: ClusterIP
selector:
k8s-app: hubble-ui
ports:
- name: http
port: 80
targetPort: 8081
YAML
echo
echo "==> Hubble UI deployed. To access it:"
echo " kubectl -n kube-system port-forward svc/hubble-ui 12000:80"
echo " then open: http://localhost:12000/"
echo
echo "DONE."
First, retrieve cluster credentials.
az aks get-credentials \
--resource-group rg-aks-acns-cilium-se \
--name aks-acns-cilium-se
Add the Helm repository and install the DaemonSet.
helm repo add inspektor-gadget https://inspektor-gadget.github.io/charts
helm repo update
helm install ig inspektor-gadget/gadget \
--namespace gadget \
--create-namespace
After a short moment, verify that the pods are running on all nodes.
kubectl get pods -n gadget
At this point, kernel probes are active across the cluster. Nothing has changed in your applications. No sidecars were injected. No code was modified. Yet the cluster is now capable of exposing runtime behavior that previously required node access.
To query this data, install the kubectl plugin via Krew.
kubectl krew install gadget
You can confirm the installation with:
kubectl gadget version
The cluster is now ready for kernel-level observation.
Seeing processes instead of containers
One of the simplest but most powerful demonstrations is tracing process execution.
Run:
kubectl gadget run trace_exec
Then start a test workload in another terminal.
kubectl run demo --image=busybox -- sleep 3600
kubectl exec -it demo -- sh
Inside the shell, run a simple command like ls. The trace will immediately display the process execution, including the namespace, pod, and container details. Many engineers realize what Inspektor Gadget reveals at this moment. Typically, Kubernetes hides processes behind container boundaries, but with kernel tracing, you can see exactly what runs within those boundaries. Unexpected shells, debugging tools, package installers, or custom binaries become instantly visible.
This level of clarity is especially transformative for incident response.
Understanding runtime behavior through DNS and TCP
Most production issues in distributed systems eventually involve networking. But networking problems often start with name resolution. A service might resolve a public endpoint instead of a private one. A library might call an undocumented telemetry endpoint. A compromised container might attempt to communicate with a command-and-control server.
To observe DNS activity across the cluster:
kubectl gadget run trace_dns
Similarly, TCP connections can be monitored with:
kubectl gadget run trace_tcp
These traces show which workloads resolve specific domains and the connections they make. Since the data originates from the kernel, it accurately represents real runtime behavior rather than just the declared configuration.
In AKS environments with private endpoints, outbound filtering, and strict egress controls, this visibility enables platform teams to verify that traffic flows align with the architectural design.
Where Cilium Hubble fits into the picture
Many AKS clusters already utilize Cilium and Hubble for network observability. Hubble offers valuable insights into network traffic between pods and services, including policy verdicts and, in some cases, application-layer information.
However, Hubble operates at the network layer, whereas Inspektor Gadget operates at the kernel layer.
Hubble can identify when a pod is connected to an external IP address. Inspektor Gadget determines which process opened the socket. Hubble indicates whether the traffic was allowed or denied, while Inspektor Gadget shows the binary and command responsible for generating the traffic.
In practice, the two tools work together. Hubble determines whether the network policy functioned correctly, while Inspektor Gadget evaluates the workload’s behavior.
In zero-trust architectures, both perspectives are necessary.
More information about Hubble can be found at:
https://docs.cilium.io/en/stable/gettingstarted/hubble/
A real production scenario
Consider a situation that many platform teams eventually face.
A firewall administrator observes outbound traffic from an AKS subnet to public IP addresses that are not listed in the approved dependency list. The applications seem to be functioning normally. There are no clear errors in the logs, metrics are normal, and no security alerts have been triggered.
The platform team receives the question: which workload is responsible for this traffic?
Instead of searching through logs or reviewing application code, the team begins with runtime observation.
Creating a realistic workload in the payments namespace
Start by creating the namespace
kubectl create namespace payments
Now deploy a small worker pod that continuously performs DNS lookups and outbound HTTPS calls. This simulates the behavior of a service reaching external dependencies.
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: payments-worker
namespace: payments
labels:
app: payments-worker
spec:
restartPolicy: Always
containers:
- name: worker
image: wbitt/network-multitool
command: ["/bin/sh","-c"]
args:
- |
echo "Starting payments worker simulation..."
while true; do
# DNS lookups
nslookup example.com >/dev/null 2>&1 || true
nslookup kubernetes.default.svc.cluster.local >/dev/null 2>&1 || true
# Outbound HTTPS call
curl -s https://example.com >/dev/null 2>&1 || true
# Simulate periodic command execution
/bin/sh -c "date > /tmp/heartbeat"
sleep 10
done
EOF
Observing DNS activity
In one terminal, start the DNS trace.
kubectl gadget run trace_dns -n payments
Within seconds, repeated queries appear for an unfamiliar domain.
Observing outbound connections
In another terminal, observe TCP activity.
kubectl gadget run trace_tcp:latest -n payments
You will see outbound connections to public IP addresses on port 443. In a real incident, this is where platform teams start asking important questions:
Why is this workload talking to the internet?
Is this expected behavior?
Does it match the architecture?
Observing process execution
Finally, observe process activity.
kubectl gadget run trace_exec:latest -n payments
The trace reveals a shell process executing a command periodically inside a container that was supposed to run only a Java application.
At this point, the investigation moves from speculation to evidence. The workload image can be inspected, the deployment history reviewed, and containment actions taken with confidence.
This is the difference between observing symptoms and observing reality.
When finished, clean up:
kubectl delete namespace payments
Cleanup script for removing the AKS Cluster
#!/usr/bin/env bash
set -euo pipefail
# -----------------------------------------------------------------------------
# Cleanup for the AKS + ACNS + Cilium + Hubble UI deployment script
#
# What it does:
# 1) Deletes the optional Hubble UI resources created in kube-system (if present)
# 2) Deletes the AKS cluster (optional; default true)
# 3) Deletes the resource group (optional; default true) ✅ recommended
# 4) Optionally removes the kubeconfig entry/context for the cluster
#
# Usage:
# chmod +x cleanup-aks-acns.sh
# ./cleanup-aks-acns.sh
#
# Optional flags:
# ./cleanup-aks-acns.sh --force
# ./cleanup-aks-acns.sh --no-wait
# DELETE_RG=false ./cleanup-aks-acns.sh
# DELETE_AKS=false ./cleanup-aks-acns.sh
# REMOVE_KUBECONFIG=true ./cleanup-aks-acns.sh
#
# Docs:
# - az group delete: https://learn.microsoft.com/cli/azure/group#az-group-delete
# - az aks delete: https://learn.microsoft.com/cli/azure/aks#az-aks-delete
# -----------------------------------------------------------------------------
# Must match your deploy defaults (override via env vars)
LOCATION="${LOCATION:-swedencentral}"
RESOURCE_GROUP="${RESOURCE_GROUP:-rg-aks-acns-cilium-se}"
CLUSTER_NAME="${CLUSTER_NAME:-aks-acns-cilium-se}"
KUBECONFIG_PATH="${KUBECONFIG_PATH:-${HOME}/.kube/config}"
# Behavior toggles
DELETE_AKS="${DELETE_AKS:-true}" # delete cluster explicitly (not needed if deleting RG, but useful if DELETE_RG=false)
DELETE_RG="${DELETE_RG:-true}" # recommended: deletes everything created by deployment
DELETE_HUBBLE_UI="${DELETE_HUBBLE_UI:-true}"
REMOVE_KUBECONFIG="${REMOVE_KUBECONFIG:-false}" # remove kubeconfig context/user/cluster entries
FORCE=false
NO_WAIT=false
for arg in "$@"; do
case "$arg" in
--force) FORCE=true ;;
--no-wait) NO_WAIT=true ;;
esac
done
require() { command -v "$1" >/dev/null 2>&1 || { echo "Missing dependency: $1" >&2; exit 1; }; }
require az
echo "========================================"
echo " Cleanup: AKS + ACNS + Cilium + Hubble UI"
echo " Location : ${LOCATION}"
echo " Resource Group: ${RESOURCE_GROUP}"
echo " Cluster : ${CLUSTER_NAME}"
echo "========================================"
echo
# Azure login
az account show >/dev/null 2>&1 || az login >/dev/null
# Confirm deletion (unless forced)
if [[ "${FORCE}" != "true" ]]; then
echo "This cleanup may delete:"
[[ "${DELETE_HUBBLE_UI}" == "true" ]] && echo " - Hubble UI resources in kube-system (Deployment/Service/ConfigMap/RBAC)"
[[ "${DELETE_AKS}" == "true" ]] && echo " - AKS cluster: ${CLUSTER_NAME} (if it exists)"
[[ "${DELETE_RG}" == "true" ]] && echo " - Resource group: ${RESOURCE_GROUP} (and ALL contained resources) [recommended]"
[[ "${REMOVE_KUBECONFIG}" == "true" ]] && echo " - kubeconfig entries for ${CLUSTER_NAME} in ${KUBECONFIG_PATH}"
echo
read -r -p "Type the resource group name to confirm: " CONFIRM
if [[ "${CONFIRM}" != "${RESOURCE_GROUP}" ]]; then
echo "Confirmation failed. Aborting."
exit 1
fi
fi
# If we plan to touch Kubernetes objects, try to get credentials (best-effort)
if [[ "${DELETE_HUBBLE_UI}" == "true" ]]; then
if command -v kubectl >/dev/null 2>&1; then
echo "==> Attempting to delete Hubble UI resources from kube-system (best-effort)"
# best-effort kubeconfig usage
export KUBECONFIG="${KUBECONFIG_PATH}"
# Try to get credentials (won't fail the script if cluster is already gone)
az aks get-credentials -g "${RESOURCE_GROUP}" -n "${CLUSTER_NAME}" --file "${KUBECONFIG_PATH}" --overwrite-existing -o none 2>/dev/null || true
# Delete resources created by the deploy script (ignore if not found)
kubectl -n kube-system delete svc hubble-ui --ignore-not-found || true
kubectl -n kube-system delete deploy hubble-ui --ignore-not-found || true
kubectl -n kube-system delete cm hubble-ui-nginx --ignore-not-found || true
kubectl delete clusterrolebinding hubble-ui --ignore-not-found || true
kubectl delete clusterrole hubble-ui --ignore-not-found || true
kubectl -n kube-system delete sa hubble-ui --ignore-not-found || true
else
echo "==> kubectl not found; skipping Hubble UI Kubernetes resource cleanup."
echo " (If you delete the resource group, these will be removed anyway.)"
fi
fi
# Delete AKS cluster (optional)
if [[ "${DELETE_AKS}" == "true" && "${DELETE_RG}" != "true" ]]; then
echo "==> Deleting AKS cluster '${CLUSTER_NAME}' (resource group '${RESOURCE_GROUP}')"
if [[ "${NO_WAIT}" == "true" ]]; then
az aks delete -g "${RESOURCE_GROUP}" -n "${CLUSTER_NAME}" --yes --no-wait || true
echo " AKS deletion started (no-wait)."
else
az aks delete -g "${RESOURCE_GROUP}" -n "${CLUSTER_NAME}" --yes || true
echo " AKS deletion completed."
fi
else
if [[ "${DELETE_RG}" == "true" ]]; then
echo "==> Skipping explicit AKS delete because resource group deletion will remove it."
fi
fi
# Delete resource group (recommended)
if [[ "${DELETE_RG}" == "true" ]]; then
echo "==> Deleting resource group '${RESOURCE_GROUP}' (this removes ALL resources in it)"
if [[ "${NO_WAIT}" == "true" ]]; then
az group delete --name "${RESOURCE_GROUP}" --yes --no-wait
echo " Resource group deletion started (no-wait)."
else
az group delete --name "${RESOURCE_GROUP}" --yes
echo " Resource group deletion completed."
fi
fi
# Optionally remove kubeconfig entries
if [[ "${REMOVE_KUBECONFIG}" == "true" ]]; then
if command -v kubectl >/dev/null 2>&1; then
echo "==> Removing kubeconfig entries for '${CLUSTER_NAME}' from ${KUBECONFIG_PATH}"
export KUBECONFIG="${KUBECONFIG_PATH}"
# AKS usually uses contexts like "<cluster-name>" or "<cluster-name>-admin"
# We'll remove any context that contains the cluster name.
mapfile -t contexts < <(kubectl config get-contexts -o name 2>/dev/null | grep -F "${CLUSTER_NAME}" || true)
for ctx in "${contexts[@]}"; do
kubectl config delete-context "${ctx}" >/dev/null 2>&1 || true
done
# Try to remove cluster entry and user entry that commonly match the cluster name
# These names vary, so we do best-effort matching.
mapfile -t clusters < <(kubectl config get-clusters 2>/dev/null | tail -n +2 | grep -F "${CLUSTER_NAME}" || true)
for c in "${clusters[@]}"; do
kubectl config delete-cluster "${c}" >/dev/null 2>&1 || true
done
mapfile -t users < <(kubectl config get-users 2>/dev/null | tail -n +2 | grep -F "${CLUSTER_NAME}" || true)
for u in "${users[@]}"; do
kubectl config unset "users.${u}" >/dev/null 2>&1 || true
done
echo " kubeconfig cleanup complete."
else
echo "==> kubectl not found; skipping kubeconfig cleanup."
fi
fi
echo
echo "Cleanup finished."
Operational and governance considerations
Kernel-level observability is a powerful capability and must be handled with care. Access to Inspektor Gadget should be restricted using Kubernetes RBAC, and its deployment should adhere to operational procedures similar to those for other privileged diagnostic tools.
In most organizations, this responsibility typically falls to platform engineering, security operations, or incident response teams instead of general developers.
From a compliance perspective, the value is considerable. Inspektor Gadget offers proof of runtime behavior, which can aid forensic investigations, fulfill audit requirements, and support zero-trust validation.
It enables teams to show not just policy configurations but also that workloads operate within expected limits.
Conclusion: closing the gap between design and execution
Cloud architecture has grown highly complex. We develop identity-focused access systems, segmented networks, encrypted communication channels, and policy-based deployment pipelines. Azure Kubernetes Service offers managed infrastructure that ensures these designs are scalable and dependable.
However, architecture reflects intent; security, reliability, and compliance ultimately rely on execution.
Execution happens in the kernel.
Inspektor Gadget offers more than just another observability tool for your platform. It reestablishes visibility into the layer where containers transition into processes, where policies turn into packet decisions, and where abstractions fade away.
For platform teams managing AKS at scale, this alters the approach to incident investigation, validation of zero-trust assumptions, and demonstration of operational control.
When integrated with tools such as Azure Monitor, Microsoft Defender for Containers, and Cilium Hubble, kernel-level visibility completes the observability stack. Metrics reveal performance, logs clarify behavior, network flows illustrate communication, and the kernel provides the underlying reality.
Once you’ve viewed your cluster from that angle, it becomes hard to depend solely on abstractions.
Ultimately, Kubernetes may describe the system you aimed to create, but the kernel reveals the system that is actually running.