Two applications communicating through their Dapr sidecars on Kubernetes

A direct HTTP call between two Kubernetes services seems simple. The caller needs a hostname, a timeout, and sufficient error handling to survive a disappearing pod. Soon it also needs retries, tracing, metrics, transport security, and an answer to which failures are safe to retry. Repeating that work in every service and language is where simple calls become distributed-systems plumbing.

Dapr service invocation moves much of that plumbing into the sidecars. The caller sends a standard HTTP or gRPC request to its local Dapr endpoint and identifies the destination by app ID. Dapr discovers a healthy destination instance, preserves tracing context, communicates between sidecars using mTLS on Kubernetes, and forwards the request to the destination application.

This post outlines how to set up two small services on AKS. The frontend application calls checkout without needing to know specific Kubernetes Service names or pod addresses. We will trigger the chain, examine both sidecars, add timeout, retry, and circuit-breaker policies, and intentionally disable the destination to observe the failure behavior.

Outcome: a request enters the frontend sidecar, reaches the frontend application, returns to its local sidecar, resolves the checkout app ID, then crosses to a checkout sidecar, and finally reaches one of two checkout replicas.

This is Part 4 of the Dapr on Kubernetes and AKS series. It builds on From OrbStack to AKS: Installing Dapr on a Managed Cluster, but the same sample also runs on the local Kubernetes environment from Part 2.

Why use the sidecar instead of a direct Kubernetes call?

Kubernetes already provides service discovery. An application can call http://checkout.dapr-demo.svc.cluster.local, and a Service can distribute traffic across checkout pods. Dapr does not invalidate that mechanism. It changes where you define application-level communication behavior.

With Dapr, the application calls localhost:3500 and names the destination checkout. The sidecars handle:

  • app-ID-based discovery and load balancing across instances;
  • mTLS between Dapr sidecars on supported hosted platforms;
  • propagation of distributed tracing context;
  • request metrics and consistent diagnostic metadata;
  • HTTP and gRPC invocation; and
  • declarative timeouts, retries, and circuit breakers for supported paths.

The trade-off involves adding another hop and runtime dependency. A service still requires careful API design, idempotency, authorization, capacity planning, and proper error handling. Dapr manages transport behavior centrally but doesn’t make an unreliable downstream process into a reliable business transaction.

Follow one request

A frontend application calling checkout through two local Dapr sidecars

The path matters because there are two separate application-to-sidecar interactions:

  1. A client invokes frontend through the frontend pod’s Dapr sidecar.
  2. The frontend application receives /orders/42 on port 8080.
  3. Frontend calls its own sidecar at 127.0.0.1:3500 and targets app ID checkout.
  4. Dapr applies the outbound resiliency policy and resolves a checkout instance.
  5. The frontend sidecar calls a checkout sidecar over the cluster network.
  6. The checkout sidecar forwards /orders/42 to its application on port 8080.

The frontend code contains neither a checkout hostname nor an AKS-specific address. That is the portability boundary: the app ID remains stable while the platform decides where instances run.

The sample

The companion manifest contains:

  • one namespace named dapr-demo;
  • a frontend Deployment with one replica;
  • a checkout Deployment with two replicas;
  • a ConfigMap containing two dependency-free Python HTTP services; and
  • a Dapr Resiliency resource scoped to the frontend app ID.

Both applications use Python’s standard library, so there is no package installation at startup. The image is deliberately public to keep this exercise focused on invocation. For a production workload, build and scan your own image, store it in ACR, and pin it by digest as described in Part 3.

Download the complete service-invocation manifest. The important annotations on each pod template are:

annotations:
  dapr.io/enabled: "true"
  dapr.io/app-id: "checkout"
  dapr.io/app-port: "8080"

dapr.io/enabled requests injection. dapr.io/app-id gives every replica the same logical identity. dapr.io/app-port tells the sidecar where the application accepts inbound requests inside the pod.

Check the cluster before deploying

Confirm that kubectl points at the intended cluster and that the Dapr control plane is healthy:

kubectl config current-context
kubectl get pods --namespace dapr-system
dapr status --kubernetes

If you do not use the Dapr CLI, the Kubernetes check is sufficient. Do not continue until the control-plane pods are ready.

Deploy the sample:

kubectl apply \
  --filename service-invocation.yaml

kubectl rollout status \
  deployment/checkout \
  --namespace dapr-demo

kubectl rollout status \
  deployment/frontend \
  --namespace dapr-demo

Verify the injected containers:

kubectl get pods \
  --namespace dapr-demo \
  --output jsonpath='{range .items[*]}{.metadata.name}{"  "}{.spec.containers[*].name}{"\n"}{end}'

Each line should contain an application container and daprd. The two checkout pods share the app ID checkout; Dapr can select either healthy instance.

Invoke the complete chain

Forward local port 3500 to the Dapr sidecar in the frontend pod:

kubectl port-forward \
  deployment/frontend \
  3500:3500 \
  --namespace dapr-demo

Keep that terminal open. From another terminal, invoke the frontend app through Dapr:

curl --fail --show-error \
  http://localhost:3500/v1.0/invoke/frontend/method/orders/42

The response comes from checkout even though the original destination was frontend:

{"orderId": "42", "status": "accepted", "servedBy": "checkout"}

The frontend application makes its downstream call with a normal HTTP request to its local sidecar:

dapr_url = (
    "http://127.0.0.1:3500"
    "/v1.0/invoke/checkout/method"
    f"{self.path}"
)

It does not need a Dapr SDK. SDKs can improve developer ergonomics, but the HTTP API is the actual contract.

You can also call checkout directly through the same frontend sidecar:

curl --fail --show-error \
  http://localhost:3500/v1.0/invoke/checkout/method/orders/99

That distinction is useful while debugging: a successful direct checkout invocation proves discovery and the destination path, while a failed call through frontend points you toward frontend code or its inbound path.

Inspect both sides of the call

Application and sidecar logs answer different questions. Start with the applications:

kubectl logs \
  deployment/frontend \
  --container frontend \
  --namespace dapr-demo

kubectl logs \
  --selector app=checkout \
  --container checkout \
  --namespace dapr-demo \
  --prefix

Then inspect the sidecars when discovery, routing, certificates, or policy behavior is suspect:

kubectl logs \
  deployment/frontend \
  --container daprd \
  --namespace dapr-demo \
  --tail 100

For a single request, correlate the application response, sidecar log entries, metrics, and trace ID rather than reading one container in isolation. A later post in this series will connect that telemetry to Azure Monitor and managed Prometheus.

Put resiliency on the destination

Dapr resiliency resources separate policy from application code. The sample applies this policy only to calls made by the frontend app ID and targets the checkout destination:

apiVersion: dapr.io/v1alpha1
kind: Resiliency
metadata:
  name: service-invocation
  namespace: dapr-demo
scopes:
  - frontend
spec:
  policies:
    timeouts:
      checkoutTimeout: 2s
    retries:
      checkoutRetry:
        policy: exponential
        duration: 100ms
        maxInterval: 1s
        maxRetries: 3
    circuitBreakers:
      checkoutCircuitBreaker:
        maxRequests: 1
        interval: 10s
        timeout: 30s
        trip: consecutiveFailures >= 5
  targets:
    apps:
      checkout:
        timeout: checkoutTimeout
        retry: checkoutRetry
        circuitBreaker: checkoutCircuitBreaker

Do not paste the YAML directly into your shell. If you deployed the complete service-invocation.yaml earlier, this resource was applied with the rest of the sample. Confirm it exists:

kubectl get resiliencies.dapr.io \
  --namespace dapr-demo

kubectl get resiliencies.dapr.io \
  service-invocation \
  --namespace dapr-demo \
  --output yaml

If you are building the example one file at a time, save the YAML block as service-invocation-resiliency.yaml, then apply and inspect it:

kubectl apply \
  --filename service-invocation-resiliency.yaml

kubectl describe resiliencies.dapr.io \
  service-invocation \
  --namespace dapr-demo

The Resiliency resource must be in the same namespace as the applications. The scopes entry selects the frontend sidecar as the policy consumer, while targets.apps.checkout attaches the named policies to outbound calls whose destination app ID is checkout.

The three mechanisms solve different problems:

  • Timeouts limit how long a call may occupy resources.
  • Retries give transient failures another chance, using exponential backoff here.
  • Circuit breakers stop repeatedly calling an unhealthy destination and later allow a limited recovery probe.

Do not copy retry counts blindly. Retrying a GET such as this sample is usually easier to reason about than retrying a payment-creating POST. For mutating operations, use idempotency keys or another deduplication design before enabling retries. Also remember that the caller’s total latency budget must include every attempt and its backoff.

In current Dapr releases, Kubernetes sidecars can hot-reload updated resiliency resources. Keep the manifest under source control even when hot reload removes the need for a manual restart.

Run a controlled failure drill

With port forwarding still active, remove all checkout application instances:

kubectl scale \
  deployment/checkout \
  --replicas 0 \
  --namespace dapr-demo

Invoke the chain several times and include response headers:

for REQUEST_NUMBER in 1 2 3 4 5 6; do
  printf '\nrequest %s\n' "$REQUEST_NUMBER"
  curl --include \
    http://localhost:3500/v1.0/invoke/frontend/method/orders/42
done

The requests now fail, but they fail through the policy-controlled path rather than hanging indefinitely. Logs from the frontend sidecar show the useful detail: no healthy destination, retry activity, timeouts, or an open circuit depending on timing and runtime version.

Restore checkout and wait for it to become available:

kubectl scale \
  deployment/checkout \
  --replicas 2 \
  --namespace dapr-demo

kubectl rollout status \
  deployment/checkout \
  --namespace dapr-demo

After the breaker timeout and successful recovery probe, calls succeed again. The exercise is intentionally small, but it exposes a production question: can operators distinguish an application error, an unavailable destination, an exhausted retry budget, and an open circuit from the telemetry they retain?

Namespaces change the destination name

Within one namespace, checkout is enough. Across namespaces, qualify the destination as <app-id>.<namespace>:

checkout.production

The corresponding HTTP invocation path is:

/v1.0/invoke/checkout.production/method/orders/42

Qualification is routing, not authorization. A cross-namespace name does not by itself grant access. Use Dapr access-control policies, Kubernetes network policy, namespace boundaries, and workload identity according to the trust model. Reusing the same app ID in several namespaces can be valid, but every caller must be clear about which environment or team boundary it targets.

HTTP or gRPC?

HTTP is the lowest-friction choice for this tutorial. It works with curl, requires no generated client, and can wrap an existing HTTP endpoint by adding the dapr-app-id header or using the /v1.0/invoke/.../method/... path.

gRPC is attractive when the application already uses protobuf contracts, needs streaming, or benefits from strongly typed clients and lower serialization overhead. Dapr supports native gRPC proxying so an existing gRPC service can retain its proto contract.

The choice is not purely performance. Check feature parity against the Dapr version you operate. As of Dapr 1.18, Dapr’s resiliency policies are not supported for service invocation through gRPC. If those declarative policies are central to your design, verify the current limitation before selecting the protocol and decide where gRPC resilience will live.

What Dapr does not decide for you

Service invocation reduces repeated infrastructure code, but teams still own the boundaries around it:

  • Design idempotent operations before applying retries.
  • Set deadlines from an end-to-end latency budget, not one hop at a time.
  • Avoid retry multiplication across ingress, application libraries, Dapr, and downstream services.
  • Treat app IDs as stable service identities with naming ownership.
  • Define authorization separately from discovery.
  • Monitor retry volume and open circuits; hidden recovery work still consumes capacity.
  • Load-test with sidecar CPU and memory limits representative of production.

A useful rule is to make one layer responsible for each behavior. If Dapr owns the checkout retry policy, remove an overlapping generic retry from the frontend HTTP client unless the combination is deliberately modeled.

Clean up

When you are finished, remove the sample namespace and everything in it:

kubectl delete namespace dapr-demo

The daily building-block pattern

This example captures the pattern behind Dapr’s other building blocks: applications call a stable local API, while declarative platform resources select behavior and infrastructure. For service invocation, the stable name is an app ID and the platform responsibilities are discovery, secure transport, telemetry, and resilience.

The practical rules are:

  1. Call the local sidecar rather than embedding platform discovery in application code.
  2. Treat app IDs and namespaces as governed service identities.
  3. Apply timeout, retry, and circuit-breaker policies to explicit destinations.
  4. Make mutating endpoints idempotent before retrying them.
  5. Diagnose the application and both sidecars as one request path.
  6. Confirm HTTP and gRPC feature parity for the Dapr version you operate.

Next in the series, we will use Dapr’s state API to store cart data while keeping the application independent of the backing state store.

Return to the Dapr on Kubernetes and AKS series index.

Sources