Production Readiness for Dapr on AKS: Upgrades, Disaster Recovery, and Lessons Learned

Production readiness starts where most tutorials end. Installing Dapr, deploying sidecars, and proving the building blocks are necessary milestones, but they do not answer the operational questions: How will you upgrade without moving every workload at once? Which state must survive a lost cluster? Can another region actually receive traffic? How quickly can you undo a bad change?

Those questions cannot be answered by Dapr alone. The operating model spans the Dapr control plane, application sidecars, Kubernetes resources, Azure dependencies, data stores, identity, networking, and deployment automation.

Outcome: Build a staged Dapr upgrade plan, promote Dapr resources with GitOps, classify every piece of recoverable state, choose an active-passive or active-active AKS design, rehearse a regional recovery, and finish with an opinionated production checklist.

This is Part 10 and the capstone of the Dapr on Kubernetes and AKS series. It assumes the security, capacity, rollout, and autoscaling baseline from Part 9.

Production is a recovery capability

A production platform should be able to perform four different recoveries:

A backup addresses only part of this table. A second cluster addresses only part of it too. A reliable design links all four to explicit recovery time and recovery point objectives.

Step 1: Define recovery objectives before selecting tools

For each user-facing capability, write down:

• RTO: how long it may remain unavailable;
• RPO: how much committed data may be lost;
• degraded mode: what still works while a dependency is unavailable;
• recovery owner: who decides when failover is required and who is responsible for executing it;
• proof: the most recent drill that demonstrated the objective.

Do this per business journey, not once for the entire cluster. Checkout may need a five-minute RTO while a reporting workflow can tolerate several hours. Those requirements lead to different Azure service tiers, replication modes, standby capacity, and costs.

Create a state inventory before designing DR:

Do not describe Kubernetes as the system of record when the actual business state lives in Cosmos DB, Service Bus, Key Vault, or a Scheduler volume.

Step 2: Capture an upgrade baseline

Download the upgrade preflight script, make it executable, and run it against the intended cluster:

chmod +x dapr-upgrade-preflight.sh
./dapr-upgrade-preflight.sh

The script is read-only. It records Kubernetes and Dapr versions, control-plane health, mTLS expiry, Dapr-enabled Deployments, current sidecar images, Dapr resources, recent warning events, and Scheduler PVCs.

Also capture workload-specific evidence:

kubectl get hpa,pdb \
  --all-namespaces

kubectl get pods \
  --all-namespaces \
  --field-selector=status.phase!=Running

kubectl top pods \
  --all-namespaces \
  --containers

Record baseline request rate, error rate, latency, sidecar CPU and memory, queue depth, actor placement stability, and workflow/job throughput. Without a baseline, “the upgrade succeeded” means only that the command returned successfully.

Before choosing a target version:

1. Read every intervening Dapr release note and breaking-change notice.
2. Check the supported upgrade path; older releases can require intermediate upgrades.
3. Confirm Kubernetes compatibility and component version requirements.
4. Test CRDs, Configuration, Component, Resiliency, and subscription resources with the target version.
5. Confirm the target version is available through the installation method you actually use.

The AKS extension version can include a Microsoft build suffix and does not necessarily equal the upstream runtime tag. Do not invent a version string or hard-code an Azure REST API version from an old example.

Step 3: Rehearse in a production-like cluster

Use at least this promotion path:

local validation → integration cluster → preproduction → production canary → production fleet

Preproduction should use the same Dapr installation method, HA mode, network policy, Workload ID, storage class, admission policy, telemetry, and representative traffic patterns as production. A single-node development cluster cannot expose zone placement, disruption-budget, surge-capacity, or persistent-volume problems.

The rehearsal must include more than a happy-path invocation:

• service invocation with retries and ACLs;
• state reads, writes, ETags, and transactions;
• pub/sub delivery, retry, dead-letter, and duplicate handling;
• actor activation, reminders, and rebalancing if actors are used;
• running and resuming workflows;
• secret resolution and workload identity token renewal;
• telemetry export; and
• a rollback while traffic continues.

Keep the exact commands and expected signals in the change record. If a test cannot be repeated, it is not an upgrade gate.

Step 4: Upgrade the control plane through its owner

First identify who owns the installation:

helm list \
  --namespace dapr-system

az k8s-extension show \
  --cluster-type managedClusters \
  --cluster-name "$AKS_CLUSTER_NAME" \
  --resource-group "$AKS_RESOURCE_GROUP" \
  --name dapr \
  --output table

Use only one lifecycle owner. Do not run Helm or dapr upgrade over an extension-managed installation.

AKS Dapr extension

Production environments should pin an approved version rather than automatically accepting every minor update. After confirming an available target version, update the extension:

TARGET_DAPR_EXTENSION_VERSION="<approved-extension-version>"

az k8s-extension update \
  --cluster-type managedClusters \
  --cluster-name "$AKS_CLUSTER_NAME" \
  --resource-group "$AKS_RESOURCE_GROUP" \
  --name dapr \
  --auto-upgrade-minor-version false \
  --version "$TARGET_DAPR_EXTENSION_VERSION"

Microsoft’s current guidance exposes version discovery through az k8s-extension extension-types. If that preview command fails because the installed CLI extension uses an unsupported Azure API version, update the relevant CLI extension or use the AKS extension view in the Azure portal rather than substituting a guessed REST API version.

Helm-managed Dapr

Keep the complete values file in version control and pass it on every upgrade:

TARGET_DAPR_VERSION="<approved-dapr-version>"

helm repo update

helm upgrade dapr dapr/dapr \
  --namespace dapr-system \
  --version "$TARGET_DAPR_VERSION" \
  --values values-production.yaml \
  --wait

Helm resets values that are not supplied. The reviewed values file is therefore part of the release artifact, not a convenience.

CLI-managed Dapr

For a CLI-managed cluster:

dapr upgrade \
  --kubernetes \
  --runtime-version "<approved-dapr-version>"

After any control-plane upgrade:

dapr status --kubernetes

kubectl get pods \
  --namespace dapr-system \
  --output wide

kubectl get events \
  --namespace dapr-system \
  --sort-by='.lastTimestamp'

Do not immediately restart every application. Hold the control plane at the new version while existing sidecars run, execute the compatibility tests, and watch errors and latency. Observe the supported version-skew window for the target release.

Step 5: Roll out sidecars as application releases

Updating the control plane does not replace daprd in existing pods. Sidecars receive the new image only when the application pod is recreated.

List Dapr-enabled Deployments:

kubectl get deployments \
  --all-namespaces \
  --output go-template='{{range .items}}{{if index .spec.template.metadata.annotations "dapr.io/enabled"}}{{.metadata.namespace}}{{"\t"}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}'

Restart one low-risk workload first:

kubectl rollout restart deployment/<canary-deployment> \
  --namespace <canary-namespace>

kubectl rollout status deployment/<canary-deployment> \
  --namespace <canary-namespace> \
  --timeout 10m

Verify its sidecar image:

kubectl get pods \
  --namespace <canary-namespace> \
  --selector app=<canary-app-label> \
  --output go-template='{{range .items}}{{.metadata.name}}{{"\n"}}{{range .spec.containers}}{{if eq .name "daprd"}}{{"  "}}{{.image}}{{"\n"}}{{end}}{{end}}{{end}}'

Then advance in batches. Respect each Deployment’s surge capacity, disruption budget, topology constraints, actor-drain requirements, and downstream capacity. Pause between batches long enough to observe SLO signals.

For actor hosts, ordinary container termination can interrupt deactivation and temporarily route calls to a departing pod. Use the Dapr-recommended drain settings and a tested preStop/termination-grace-period design. Do not enable Kubernetes native sidecars for actor hosts without checking current Dapr limitations; termination ordering can prevent clean actor deactivation.

Rollback is two separate decisions

A bad application-sidecar rollout can usually be reversed with the application’s deployment history or a previous Git commit. A control-plane rollback uses the owning installation tool and a supported target version. Reverting only one side can leave an unsupported version combination.

Before rollback, ask:

• Did the upgrade change CRDs or persisted formats?
• Did a new component version write data an older version cannot understand?
• Are older sidecars still supported by the current control plane?
• Will rolling back Placement or Scheduler cause actor redistribution?

“Use the previous version” is not a rollback plan until these questions have tested answers.

Step 6: Put Dapr resources under GitOps

Dapr resources are application infrastructure. Store Component, Configuration, Resiliency, Subscription, and access-policy resources beside the workload or in a clearly owned platform repository.

A practical repository shape is:

clusters/
  production-west/
    platform/
    dapr-resources/
    applications/
  production-north/
    platform/
    dapr-resources/
    applications/
base/
  applications/
  dapr-resources/

Reconcile in this order:

cluster prerequisites and Dapr CRDs
            ↓
namespaces, identities, policies, secret providers
            ↓
Dapr components, configurations, resiliency, subscriptions
            ↓
application deployments

Download the Flux ordering example. It creates two Flux Kustomization resources: applications wait for Dapr resources to reconcile successfully.

GitOps should contain references, not credentials. Use Workload ID and Key Vault, keep immutable image digests in overlays, and validate component schemas before merge. Treat prune: true carefully: deleting a component from Git can immediately remove it from running sidecars because Dapr resource hot reload is enabled by default.

Promote the same commit or immutable artifact across environments. Environment overlays should change only values that genuinely differ, such as namespaces, resource identifiers, replica limits, and regional endpoints. Recopying YAML between branches invites invisible drift.

Step 7: Design the second cluster as an independent stamp

Dapr is cluster-local. An app ID in one AKS cluster is not automatically discoverable from another cluster, and Dapr does not replicate component data between clusters.

A multi-region stamp normally contains:

• an independent AKS cluster and Dapr control plane;
• the same GitOps-controlled workload and Dapr resources;
• regional ingress, egress, private endpoints, monitoring, and identities;
• an image source available in that region;
• explicitly replicated or recoverable data services; and
• global traffic routing such as Azure Front Door for HTTP/HTTPS or Traffic Manager for DNS-based/non-HTTP routing.

Choose the operating mode from requirements:

Active-active does not mean that every data service can accept writes in both regions. Design idempotency, conflict handling, ordering, and ownership for each stateful dependency.

Step 8: Recover data, not only manifests

GitOps can recreate a Component object, but it cannot recreate the data behind that component.

State stores and actors

Back up and replicate the backing state store using its native capabilities. Actor state shares that dependency, so test that restored actor keys, ETags, reminders, and placement behaviour remain valid. Never copy a production database into a recovery environment without considering encryption, privacy, and external side effects.

Pub/sub and Service Bus

For Azure Service Bus Premium, Geo-Replication protects metadata and message data and is the preferred regional recovery feature for most workloads. Metadata Geo-Disaster Recovery replicates entities and configuration but not messages. Standard tier has no native multi-region feature; application-level replication is required when message preservation matters.

Whichever design you choose, recreate and verify private endpoints, DNS, managed identities, and RBAC in the secondary region. Service Bus metadata failover does not automatically reproduce every surrounding security and network setting.

Scheduler, jobs, reminders, and workflows

Fresh Dapr 1.18+ installations persist Scheduler’s embedded etcd data on PVCs, with a default 16Gi claim. Clusters upgraded from older versions can retain a 1Gi claim. Scheduler stores Jobs API data and participates in actor reminders and workflows, so include these PVCs in backup, capacity monitoring, and recovery tests.

Inspect them with:

kubectl get persistentvolumeclaims \
  --namespace dapr-system \
  --selector app=dapr-scheduler-server

Confirm the actual labels in your installed version if the selector returns nothing. A snapshot existing in a vault is not proof of recovery; restore it into an isolated cluster and demonstrate that scheduled work resumes without unintended duplicate effects.

Kubernetes resources and volumes

Prefer rebuilding stateless cluster resources from infrastructure as code and GitOps. Use Azure Backup for AKS when Kubernetes objects or persistent volumes contain state that cannot be reconstructed. Define namespace scope, retention, cross-region availability, trusted access, and restore sequencing from the RPO/RTO—not from the default wizard settings.

Step 9: Run a recovery drill

Download the recovery drill checklist and adapt it to your workload.

A useful drill has evidence for every phase:

1. Declare the simulated failure and start the RTO clock.
2. Freeze risky changes and identify the last known good Git commit and recovery points.
3. Confirm the recovery cluster, Dapr control plane, identities, policies, and monitoring.
4. Restore or promote stateful Azure services in dependency order.
5. Reconcile Dapr resources, then applications.
6. Run synthetic reads and writes without public traffic.
7. Verify trace, metrics, logs, queue processing, actor/workflow behaviour, and business invariants.
8. Shift a small traffic percentage, observe, then complete the traffic move.
9. Record actual RTO, actual RPO, data reconciliation, and manual steps.
10. Decide how to repair and re-establish redundancy before declaring the incident closed.

Run failback as well as failover. Many systems can move traffic once but have no safe method to return or rebuild the failed region.

Step 10: Make cost part of reliability

Dapr cost is distributed across the platform:

sidecar cost = replicas × per-sidecar CPU/memory
rollout headroom = surge replicas × complete pod request
control plane = HA replicas + Scheduler persistent disks
telemetry = metric cardinality + trace sampling + log ingestion/retention
DR = second cluster + replicated data services + global routing

Optimize from measurements. Reducing every sidecar request can create throttling and slow incidents; keeping every standby cluster at full production size can waste money. Warm standby, scheduled scale changes, Savings Plans, regional data tiers, and telemetry retention are business trade-offs tied to RTO and RPO.

Track cost per workload or transaction alongside reliability. A sidecar platform is valuable when shared capabilities remove more engineering and operational cost than they add in runtime resources.

What this series taught us

Several lessons recur across all ten parts:

1. Dapr is an application runtime, not a complete platform. Kubernetes and Azure still own scheduling, network boundaries, identity, data durability, and regional architecture.
2. The sidecar is part of the workload. Include it in resources, rollouts, scaling, observability, security reviews, and incident response.
3. Abstraction does not remove infrastructure semantics. ETags, message delivery, partitioning, broker failover, database consistency, and identity propagation still matter.
4. Start deny-by-default before the component catalog grows. Retrofitting scopes and ACLs after every app can load every component is painful.
5. Prefer managed identity over stored credentials. Workload ID narrows ownership and makes rotation an identity concern rather than a secret-distribution project.
6. A second cluster is not DR until state and traffic move. Reconciliation, replication, routing, and human decisions must all be rehearsed.
7. Pin production versions and promote evidence. Automatic minor upgrades are useful in development, but production change needs a target, gates, rollback criteria, and observation time.

The production gate

Before calling the platform ready, require evidence that:

• supported Dapr, Kubernetes, component, and SDK versions are recorded;
• control-plane and sidecar upgrades have passed preproduction and canary gates;
• manifests and Dapr resources reconcile from Git into an empty cluster;
• credentials are absent from Git and workloads use narrow identities;
• state, messages, Scheduler data, and secrets have explicit recovery mechanisms;
• multi-region routing and dependency promotion have been tested;
• actual RTO and RPO are measured in a drill;
• monitoring detects control-plane, sidecar, data, and failover problems;
• complete-pod and surge capacity are included in cost and scheduling; and
• the team knows who may trigger rollback, failover, and failback.

Cleanup

This post does not create cloud resources automatically. The preflight script is read-only, and the Flux example is a template containing placeholders. If you applied the Flux resources in a test cluster, remove only the named test resources after confirming their scope:

kubectl delete kustomization apps dapr-resources \
  --namespace flux-system

kubectl delete gitrepository platform-config \
  --namespace flux-system

Do not delete a production Flux configuration, backup vault, recovery cluster, replicated data service, or Dapr control plane as tutorial cleanup.

Takeaway

Dapr has a real learning curve. At first, the promise of calling a local sidecar for service invocation, state, messaging, secrets, and actors sounds straightforward. The complexity becomes visible when you connect those building blocks to Kubernetes, Azure identities, network policies, external data services, observability, scaling, upgrades, and disaster recovery. I found that learning Dapr is less about memorising its APIs and more about understanding where Dapr’s responsibility ends and the application’s or platform’s responsibility begins.

That understanding develops gradually. Start with one building block, inspect what the sidecar actually does, and deliberately break the example before adding the next capability. The troubleshooting in this series was as valuable as the successful deployments: configuration mistakes, identity problems, timeouts, and version differences reveal how the system really works. Trying to adopt every Dapr feature at once would hide those lessons behind too many moving parts.

After working through the series, I see Dapr not as a shortcut around distributed-systems engineering, but as a consistent foundation for it. It can remove repeated plumbing and give teams a shared model, but production readiness still requires careful ownership, staged upgrades, tested recovery, and operational practice. Use the series index to follow the same path, then adapt the examples into standards that fit your own applications and platform.

Sources and further reading

• Upgrade Dapr on Kubernetes
• Dapr Kubernetes production guidelines
• Dapr supported releases and upgrade paths
• Install and operate the Dapr extension for AKS
• Configure the Dapr extension
• Dapr resource updates and hot reload
• Persist Dapr Scheduler jobs
• GitOps with Flux on AKS
• AKS baseline architecture
• AKS multi-region baseline
• AKS active-passive disaster recovery
• Azure Service Bus Geo-Replication
• Azure Backup for AKS FAQ