Multi-Region AKS Deployments Using Workload Identity
Design and implement secure multi-region Azure Kubernetes Service architectures with Workload Identity federation for seamless cross-region authentication and failover.
I’ve been working with multi-region Kubernetes deployments for years, and one question keeps coming up: how do you handle identity and authentication across regions without creating security vulnerabilities or operational nightmares? Azure Workload Identity has become my go-to solution for this challenge.
Let me share what I’ve learned from deploying production multi-region AKS clusters with proper identity federation, and why this approach has replaced the older AAD Pod Identity model in every architecture I design today.
Why Multi-Region AKS?
Before diving into implementation, let’s talk about why you’d even consider multi-region deployments. It’s not just about checking a compliance box.
Legitimate reasons I’ve seen:
- Business continuity: Your application can’t tolerate regional outages (looking at you, financial services)
- Latency optimization: Serving global users with sub-100ms response times
- Data residency: Regulatory requirements forcing data to stay within specific geographies
- Load distribution: Scaling beyond single-region capacity limits
When NOT to go multi-region:
- You’re just starting out (master single-region first)
- Your traffic patterns don’t justify the complexity
- You haven’t solved state management and data replication
- Your team lacks the operational maturity to manage distributed systems
Multi-region is powerful but expensive—both in infrastructure costs and engineering complexity. Make sure you actually need it.
Understanding Azure Workload Identity
Workload Identity is Azure’s modern approach to giving Kubernetes workloads secure access to Azure resources without managing credentials. It replaced AAD Pod Identity in late 2022, and honestly, it’s much better.
Why Workload Identity Replaced AAD Pod Identity
I’ve migrated multiple clusters from AAD Pod Identity to Workload Identity. Here’s what changed:
AAD Pod Identity (the old way):
- Required Node Managed Identity or VMSS Identity
- Used a complex webhook and identity assignment system
- Had race conditions and timing issues at scale
- Required the NMI (Node Managed Identity) DaemonSet
- Didn’t work well with virtual nodes or serverless scenarios
Workload Identity (the new way):
- Uses OpenID Connect (OIDC) federation
- No cluster infrastructure components required
- Works with AKS, EKS, GKE, or any OIDC-enabled cluster
- Better security through short-lived tokens
- Simpler to troubleshoot and reason about
The key insight: instead of assigning identities to nodes, Workload Identity federates trust between Kubernetes service accounts and Azure Managed Identities using OIDC tokens. It’s cleaner, more secure, and actually works reliably.
Architecture Overview
Here’s a typical multi-region AKS architecture with Workload Identity that I’ve deployed multiple times:
graph TB
subgraph Global["Global Layer"]
tm[Traffic Manager]
acr[Container Registry]
end
subgraph RegionA["East US Region"]
aksA[AKS Cluster]
podsA[Application Pods]
miA[Managed Identity]
kvA[Key Vault]
storageA[Storage Account]
end
subgraph RegionB["West Europe Region"]
aksB[AKS Cluster]
podsB[Application Pods]
miB[Managed Identity]
kvB[Key Vault]
storageB[Storage Account]
end
subgraph Shared["Shared Data"]
sql[Azure SQL]
cosmos[Cosmos DB]
end
tm -->|Route Traffic| aksA
tm -->|Route Traffic| aksB
aksA -.->|Pull Images| acr
aksB -.->|Pull Images| acr
podsA -->|OIDC Token| miA
podsB -->|OIDC Token| miB
miA -->|Access| kvA
miA -->|Access| storageA
miA -->|Connect| sql
miA -->|Connect| cosmos
miB -->|Access| kvB
miB -->|Access| storageB
miB -->|Connect| sql
miB -->|Connect| cosmos
Notice how each region has its own managed identity, but they all federate through OIDC. This gives you regional isolation while maintaining consistent authentication patterns.
Implementation: Step-by-Step
Step 1: Enable Workload Identity on AKS Clusters
First, you need to enable OIDC issuer and Workload Identity on your clusters. This is a one-time operation per cluster.
# Enable on existing cluster (East US)
az aks update \
--resource-group rg-aks-prod-eastus \
--name aks-prod-eastus \
--enable-oidc-issuer \
--enable-workload-identity
# Enable on existing cluster (West Europe)
az aks update \
--resource-group rg-aks-prod-westeu \
--name aks-prod-westeu \
--enable-oidc-issuer \
--enable-workload-identity
# Get the OIDC issuer URL (you'll need this later)
az aks show \
--resource-group rg-aks-prod-eastus \
--name aks-prod-eastus \
--query "oidcIssuerProfile.issuerUrl" \
-o tsv
The OIDC issuer URL looks something like: https://eastus.oic.prod-aks.azure.com/00000000-0000-0000-0000-000000000000/
Step 2: Create Multi-Region Infrastructure with Terraform
Here’s my production-grade Terraform setup for multi-region AKS with Workload Identity:
# Variables for multi-region deployment
variable "regions" {
type = map(object({
location = string
address_space = list(string)
}))
default = {
primary = {
location = "eastus"
address_space = ["10.1.0.0/16"]
}
secondary = {
location = "westeurope"
address_space = ["10.2.0.0/16"]
}
}
}
# Resource Groups per region
resource "azurerm_resource_group" "aks" {
for_each = var.regions
name = "rg-aks-prod-${each.value.location}"
location = each.value.location
tags = {
Environment = "Production"
Region = each.key
ManagedBy = "Terraform"
}
}
# VNets per region
resource "azurerm_virtual_network" "aks" {
for_each = var.regions
name = "vnet-aks-${each.value.location}"
location = each.value.location
resource_group_name = azurerm_resource_group.aks[each.key].name
address_space = each.value.address_space
}
# AKS Subnet per region
resource "azurerm_subnet" "aks" {
for_each = var.regions
name = "snet-aks-nodes"
resource_group_name = azurerm_resource_group.aks[each.key].name
virtual_network_name = azurerm_virtual_network.aks[each.key].name
address_prefixes = [cidrsubnet(each.value.address_space[0], 4, 0)]
}
# AKS Clusters with Workload Identity enabled
resource "azurerm_kubernetes_cluster" "aks" {
for_each = var.regions
name = "aks-prod-${each.value.location}"
location = each.value.location
resource_group_name = azurerm_resource_group.aks[each.key].name
dns_prefix = "aks-prod-${each.value.location}"
kubernetes_version = "1.28.3"
default_node_pool {
name = "system"
node_count = 3
vm_size = "Standard_D4s_v5"
vnet_subnet_id = azurerm_subnet.aks[each.key].id
enable_auto_scaling = true
min_count = 3
max_count = 10
os_sku = "Ubuntu"
}
identity {
type = "SystemAssigned"
}
# Enable Workload Identity and OIDC
oidc_issuer_enabled = true
workload_identity_enabled = true
network_profile {
network_plugin = "azure"
network_policy = "azure"
load_balancer_sku = "standard"
}
tags = {
Environment = "Production"
Region = each.key
}
}
# User-Assigned Managed Identities for workloads
resource "azurerm_user_assigned_identity" "workload" {
for_each = var.regions
name = "id-aks-workload-${each.value.location}"
location = each.value.location
resource_group_name = azurerm_resource_group.aks[each.key].name
}
# Federated Identity Credentials (this is the magic)
resource "azurerm_federated_identity_credential" "workload" {
for_each = var.regions
name = "fic-aks-app-${each.value.location}"
resource_group_name = azurerm_resource_group.aks[each.key].name
parent_id = azurerm_user_assigned_identity.workload[each.key].id
audience = ["api://AzureADTokenExchange"]
issuer = azurerm_kubernetes_cluster.aks[each.key].oidc_issuer_url
subject = "system:serviceaccount:default:workload-identity-sa"
}
# Grant permissions to Azure resources
# Example: Key Vault access
resource "azurerm_key_vault" "regional" {
for_each = var.regions
name = "kv-aks-${each.value.location}-${random_string.suffix.result}"
location = each.value.location
resource_group_name = azurerm_resource_group.aks[each.key].name
tenant_id = data.azurerm_client_config.current.tenant_id
sku_name = "standard"
enable_rbac_authorization = true
}
resource "azurerm_role_assignment" "kv_secrets_user" {
for_each = var.regions
scope = azurerm_key_vault.regional[each.key].id
role_definition_name = "Key Vault Secrets User"
principal_id = azurerm_user_assigned_identity.workload[each.key].principal_id
}
# Output the configuration details
output "aks_oidc_issuers" {
value = {
for k, v in azurerm_kubernetes_cluster.aks : k => v.oidc_issuer_url
}
}
output "workload_identity_client_ids" {
value = {
for k, v in azurerm_user_assigned_identity.workload : k => v.client_id
}
}
Step 3: Configure Kubernetes Service Accounts
Now deploy the Kubernetes-side configuration. This is where you link your pods to the Azure Managed Identity.
# service-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: workload-identity-sa
namespace: default
annotations:
# This annotation links the K8s SA to the Azure Managed Identity
azure.workload.identity/client-id: "CLIENT_ID_FROM_TERRAFORM_OUTPUT"
---
apiVersion: v1
kind: Pod
metadata:
name: my-app
namespace: default
labels:
azure.workload.identity/use: "true" # This label triggers injection
spec:
serviceAccountName: workload-identity-sa
containers:
- name: app
image: myregistry.azurecr.io/my-app:latest
env:
- name: AZURE_CLIENT_ID
value: "CLIENT_ID_FROM_TERRAFORM_OUTPUT"
- name: AZURE_TENANT_ID
value: "YOUR_TENANT_ID"
- name: AZURE_FEDERATED_TOKEN_FILE
value: /var/run/secrets/azure/tokens/azure-identity-token
Deploy this to both regions:
# Apply to East US cluster
kubectl apply -f service-account.yaml --context aks-prod-eastus
# Apply to West Europe cluster
kubectl apply -f service-account.yaml --context aks-prod-westeu
Step 4: Test the Identity Federation
Here’s a quick test to verify everything works:
apiVersion: v1
kind: Pod
metadata:
name: identity-test
namespace: default
labels:
azure.workload.identity/use: "true"
spec:
serviceAccountName: workload-identity-sa
containers:
- name: test
image: mcr.microsoft.com/azure-cli:latest
command:
- /bin/bash
- -c
- |
echo "Testing Workload Identity..."
az login --federated-token "$(cat $AZURE_FEDERATED_TOKEN_FILE)" \
--service-principal \
-u $AZURE_CLIENT_ID \
-t $AZURE_TENANT_ID
echo "Logged in successfully!"
az account show
echo "Testing Key Vault access..."
az keyvault secret list --vault-name YOUR_KV_NAME
sleep 3600
Run this in both regions and check the logs. If authentication works, you’re golden.
Cluster Failover Strategies
Now let’s talk about the hard part: what happens when a region goes down?
Active-Active with Traffic Manager
This is my preferred approach for truly critical workloads:
How it works:
- Both regions serve production traffic simultaneously
- Traffic Manager or Front Door routes based on performance or weighted distribution
- Applications must handle eventual consistency
- Database writes need careful coordination (active-active Cosmos DB or conflict resolution)
Failover behavior:
- Automatic: Traffic Manager detects cluster health issues and routes away
- Manual: You can adjust weights or disable endpoints
- RTO: Under 5 minutes typically
- RPO: Depends on your data replication strategy
# Azure Traffic Manager for AKS multi-region
resource "azurerm_traffic_manager_profile" "aks" {
name = "tm-aks-prod"
resource_group_name = azurerm_resource_group.shared.name
traffic_routing_method = "Performance" # or "Weighted", "Priority"
dns_config {
relative_name = "aks-prod"
ttl = 30
}
monitor_config {
protocol = "HTTPS"
port = 443
path = "/health"
expected_status_code_ranges = ["200-202", "301-302"]
interval_in_seconds = 30
timeout_in_seconds = 10
tolerated_number_of_failures = 3
}
}
resource "azurerm_traffic_manager_azure_endpoint" "eastus" {
name = "endpoint-eastus"
profile_id = azurerm_traffic_manager_profile.aks.id
weight = 100
target_resource_id = azurerm_public_ip.aks_ingress_eastus.id
priority = 1
}
resource "azurerm_traffic_manager_azure_endpoint" "westeu" {
name = "endpoint-westeu"
profile_id = azurerm_traffic_manager_profile.aks.id
weight = 100
target_resource_id = azurerm_public_ip.aks_ingress_westeu.id
priority = 2
}
Active-Passive with DNS Failover
For workloads that can’t handle active-active complexity:
How it works:
- One region (primary) handles all traffic
- Secondary region is warm standby
- Manual or automated DNS failover on primary failure
- Simpler data consistency story
Trade-offs:
- RTO: 5-15 minutes (DNS propagation + manual intervention)
- RPO: Depends on replication lag
- Lower cost (secondary can be scaled down)
- Requires runbook and testing
Data Plane vs Control Plane Considerations
This is where many architects get tripped up. Let me clarify:
Control Plane (AKS managed):
- Microsoft handles regional redundancy
- 99.95% SLA with availability zones
- Your kubectl commands go through regional API endpoints
- Control plane outages are rare but possible
Data Plane (your workloads):
- This is what you’re responsible for
- Node failures, zone failures, regional failures
- Your multi-region setup protects against this
Key insight: Even in a single-region deployment, spread nodes across availability zones. Multi-region is for regional disasters, not typical node failures.
# Availability zones in node pool
resource "azurerm_kubernetes_cluster_node_pool" "app" {
name = "app"
kubernetes_cluster_id = azurerm_kubernetes_cluster.aks["primary"].id
vm_size = "Standard_D8s_v5"
zones = ["1", "2", "3"] # Spread across AZs
enable_auto_scaling = true
min_count = 3
max_count = 30
}
Cross-Region Service Routing
Your application architecture heavily influences routing strategy:
Stateless Applications (easier)
- Traffic Manager/Front Door at the edge
- Each region handles requests independently
- Session affinity if needed (but avoid if possible)
- Cache warming strategies for cold failover
Stateful Applications (harder)
- Database replication lag is your enemy
- Consider async replication with conflict resolution
- Or synchronous replication with latency penalty
- Read replicas in each region, single write region
My recommendation: Design for eventual consistency where possible. Strong consistency across regions is expensive and slow.
Regional ACR Access and Image Replication
Don’t forget about container images. Your pods need images, and pulling across regions is slow.
# Geo-replicated Azure Container Registry
resource "azurerm_container_registry" "shared" {
name = "acrprodshared"
resource_group_name = azurerm_resource_group.shared.name
location = "eastus"
sku = "Premium" # Required for geo-replication
admin_enabled = false
georeplications {
location = "westeurope"
tags = {}
}
georeplications {
location = "southeastasia"
tags = {}
}
}
# Grant AKS pull access with Managed Identity
resource "azurerm_role_assignment" "acr_pull" {
for_each = var.regions
scope = azurerm_container_registry.shared.id
role_definition_name = "AcrPull"
principal_id = azurerm_kubernetes_cluster.aks[each.key].kubelet_identity[0].object_id
}
Image pull optimization:
- Use ACR geo-replication to place images near clusters
- Implement image pull secrets with Workload Identity
- Monitor image pull times and failures
- Consider image caching strategies for frequently pulled images
Key Challenges and How to Address Them
Challenge 1: Latency Between Regions
The problem: Cross-region calls add 50-200ms depending on distance.
Solutions:
- Minimize cross-region calls in critical paths
- Use asynchronous processing where possible
- Cache aggressively at the edge
- Consider regional data partitioning
Challenge 2: RBAC Scoping Across Regions
The problem: Managing consistent RBAC across multiple clusters is painful.
Solutions:
# Use GitOps to maintain consistent RBAC
# Example: Flux kustomization for multi-cluster RBAC
# flux-system/rbac-overlay.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: rbac
namespace: flux-system
spec:
interval: 10m
sourceRef:
kind: GitRepository
name: fleet-infra
path: ./clusters/overlays/rbac
prune: true
validation: client
Maintain a single source of truth in Git, deploy consistently with Flux or ArgoCD.
Challenge 3: Secret Management Across Regions
The problem: Secrets need to be available in both regions but kept in sync.
Solutions:
# Use External Secrets Operator with Workload Identity
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: azure-keyvault
namespace: default
spec:
provider:
azurekv:
authType: WorkloadIdentity
vaultUrl: "https://kv-prod-eastus.vault.azure.net"
serviceAccountRef:
name: workload-identity-sa
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: app-secrets
namespace: default
spec:
refreshInterval: 1h
secretStoreRef:
name: azure-keyvault
kind: SecretStore
target:
name: app-secrets
creationPolicy: Owner
data:
- secretKey: database-password
remoteRef:
key: db-password
Deploy this same configuration to both regions, each pointing to its regional Key Vault. Use Key Vault replication or duplicate secrets.
Challenge 4: Cost Management
The problem: Running multiple regions doubles (or triples) your infrastructure costs.
Solutions:
- Right-size your secondary regions (you don’t need 1:1 capacity for warm standby)
- Use Azure Reservations for predictable compute costs
- Implement auto-scaling aggressively
- Monitor cross-region data transfer (it’s expensive)
# Cost-optimized secondary region node pool
resource "azurerm_kubernetes_cluster_node_pool" "app_secondary" {
name = "app"
kubernetes_cluster_id = azurerm_kubernetes_cluster.aks["secondary"].id
vm_size = "Standard_D4s_v5" # Smaller than primary
enable_auto_scaling = true
min_count = 2 # Lower minimum
max_count = 20 # Lower maximum
priority = "Spot" # Use Spot VMs for non-critical secondary
eviction_policy = "Delete"
spot_max_price = 0.05
}
Operational Recommendations for Multi-Region AKS
After running multi-region clusters in production, here’s what actually matters:
1. Implement comprehensive health checks
- Don’t rely on basic HTTP 200 responses
- Test actual dependencies (database, external APIs, Key Vault)
- Use Kubernetes readiness and liveness probes properly
2. Practice failover regularly
- Monthly failover drills (at minimum)
- Automated failover testing in lower environments
- Document actual RTO/RPO measurements
- Maintain runbooks for manual failover
3. Monitor cross-region replication lag
# Prometheus alert for replication lag
- alert: DatabaseReplicationLagHigh
expr: |
pg_replication_lag_seconds > 30
for: 5m
labels:
severity: warning
annotations:
summary: "Database replication lag is high"
description: "Replication lag is {{ $value }}s for {{ $labels.instance }}"
4. Use GitOps for consistency
- Single source of truth for cluster configuration
- Automated sync to all regions
- Drift detection and remediation
- Clear promotion path: dev → staging → prod regions
5. Implement proper observability
# Distributed tracing across regions
apiVersion: v1
kind: ConfigMap
metadata:
name: otel-collector-config
data:
config.yaml: |
receivers:
otlp:
protocols:
grpc:
http:
processors:
batch:
attributes:
actions:
- key: cluster.region
value: ${CLUSTER_REGION}
action: insert
exporters:
azuremonitor:
instrumentation_key: ${APPINSIGHTS_KEY}
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch, attributes]
exporters: [azuremonitor]
Tag all telemetry with region information so you can troubleshoot cross-region issues.
6. Capacity planning per region
- Don’t assume symmetric load distribution
- Plan for full load on single region (N+1 redundancy)
- Monitor and alert on capacity thresholds
- Have auto-scaling configured and tested
7. Documentation and team training
- Every team member should understand the multi-region architecture
- Runbooks for common failure scenarios
- Clear escalation paths for regional incidents
- Regular architectural reviews
8. Cost allocation and optimization
# Tag resources for cost tracking
tags = {
Environment = "Production"
Region = "Primary"
CostCenter = "Platform"
DR-Purpose = "Active-Active"
ReviewDate = "2025-Q1"
}
Track costs per region, review quarterly, optimize based on actual usage patterns.
Key Takeaways
Multi-region AKS with Workload Identity is powerful but complex:
- Workload Identity is the modern standard – simpler, more secure, and more reliable than AAD Pod Identity
- OIDC federation is elegant – no infrastructure components, just trust relationships
- Plan your failover strategy upfront – active-active vs active-passive has huge architectural implications
- Data consistency is hard – eventual consistency makes everything easier if your business can accept it
- Cross-region networking matters – latency, data transfer costs, and routing complexity add up quickly
- Test your failover – a disaster recovery plan you haven’t tested is just expensive documentation
- GitOps is essential – maintaining consistency across regions manually doesn’t scale
- Cost management is critical – multi-region can easily double or triple your Azure bill
The combination of multi-region AKS with Workload Identity gives you a robust, secure foundation for globally distributed applications. The key is understanding your actual requirements for availability, latency, and consistency, then designing the simplest architecture that meets those needs.
Start with single-region, add availability zones, and only go multi-region when you have clear business justification and operational maturity to support it. When you do make that jump, Workload Identity makes the authentication story much cleaner than previous solutions.