PDB Management Operator - Operational Runbook¶
Table of Contents¶
Overview¶
The PDB Management Operator is a Kubernetes operator that automatically manages Pod Disruption Budgets (PDBs) for deployments based on annotations and policies. It provides three enforcement modes: strict, flexible, and advisory.
Key Features¶
Dual Configuration Model: Supports both annotation-based and policy-based configuration
Policy Enforcement: Three modes (strict, flexible, advisory) with priority resolution
Maintenance Windows: Automatic PDB suspension during maintenance windows
Comprehensive Observability: Metrics, tracing, and structured logging
Performance Optimized: Caching layer and adaptive circuit breaker
Architecture¶
Components¶
AvailabilityPolicy Controller: Manages custom CRDs and policy enforcement
Deployment Controller: Watches deployments and creates/updates PDBs
PDB Controller: Manages PDB lifecycle and consistency
Webhook Server: Validates and defaults AvailabilityPolicy CRDs
Cache Layer: Optimizes performance with policy caching
Circuit Breaker: Adaptive client for API call resilience
Data Flow¶
Deployment Created/Updated
↓
Check Annotations
↓
Apply Policies (if any)
↓
Create/Update PDB
↓
Record Metrics & Events
Installation & Deployment¶
Prerequisites¶
Kubernetes 1.21+
cert-manager (for webhook TLS)
Prometheus (for metrics)
OpenTelemetry Collector (for tracing)
Quick Start¶
# Deploy to canvas namespace
kubectl apply -k config/default/
# Verify installation
kubectl get pods -n canvas -l app.kubernetes.io/name=pdb-management
kubectl get crd availabilitypolicies.availability.oda.tmforum.org
Production Deployment¶
# 1. Install cert-manager (if not already installed)
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cert-manager.yaml
# 2. Deploy with webhooks and TLS
kubectl apply -k config/certmanager/
kubectl apply -k config/webhook/
kubectl apply -k config/default/
# 3. Verify webhook certificates
kubectl get certificate -n canvas
kubectl get validatingwebhookconfiguration
kubectl get mutatingwebhookconfiguration
Configuration Options¶
Flag |
Default |
Description |
|---|---|---|
|
|
Enable admission webhooks |
|
|
Webhook server port |
|
|
Max concurrent reconciliations |
|
|
Enable OpenTelemetry tracing |
|
|
Log level (debug, info, error) |
Monitoring & Observability¶
Metrics¶
The operator exposes Prometheus metrics on port 8443:
# Port forward to access metrics
kubectl port-forward -n canvas svc/controller-manager-metrics-service 8443:8443
# Query metrics
curl -k https://localhost:8443/metrics
Key Metrics¶
pdb_management_reconciliations_total: Total reconciliation attemptspdb_management_pdb_created_total: PDBs createdpdb_management_pdb_updated_total: PDBs updatedpdb_management_policy_cache_hits: Cache hit ratepdb_management_reconciliation_duration_seconds: Reconciliation latency
Logging¶
The operator uses structured JSON logging with trace correlation:
# View operator logs
kubectl logs -n canvas -l app.kubernetes.io/name=pdb-management -f
# Filter by trace ID
kubectl logs -n canvas -l app.kubernetes.io/name=pdb-management | jq 'select(.trace_id)'
Log Levels¶
debug: Detailed reconciliation stepsinfo: Normal operations and state changeserror: Errors and failures
Tracing¶
OpenTelemetry tracing is enabled by default:
# View traces in Jaeger/Zipkin
# Configure your tracing backend to collect from otel-collector
Health Checks¶
# Check operator health
kubectl get pods -n canvas -l app.kubernetes.io/name=pdb-management -o wide
# Check readiness
kubectl get endpoints -n canvas controller-manager-metrics-service
Troubleshooting¶
Common Issues¶
1. PDB Not Created¶
Symptoms: Deployment has annotations but no PDB is created
Diagnosis:
# Check operator logs
kubectl logs -n canvas -l app.kubernetes.io/name=pdb-management | grep -i "deployment"
# Check deployment annotations
kubectl get deployment <name> -o yaml | grep -A5 -B5 annotations
# Check for errors
kubectl get events -n canvas --sort-by='.lastTimestamp'
Solutions:
Verify annotations are correct:
pdb-management.oda.tmforum.org/availability-classCheck if policies are blocking creation
Verify operator has RBAC permissions
2. Webhook Failures¶
Symptoms: CRD operations fail with webhook errors
Diagnosis:
# Check webhook service
kubectl get svc -n canvas webhook-service
# Check webhook certificates
kubectl get certificate -n canvas serving-cert
# Check webhook logs
kubectl logs -n canvas -l app.kubernetes.io/name=pdb-management | grep -i webhook
Solutions:
Verify cert-manager is installed and running
Check certificate validity:
kubectl describe certificate serving-cert -n canvasRestart operator if certificates are invalid
3. High Memory Usage¶
Symptoms: Operator pods using excessive memory
Diagnosis:
# Check resource usage
kubectl top pods -n canvas -l app.kubernetes.io/name=pdb-management
# Check cache statistics
kubectl logs -n canvas -l app.kubernetes.io/name=pdb-management | grep -i "cache"
Solutions:
Increase memory limits in deployment
Reduce cache size or TTL
Check for memory leaks in logs
4. Slow Reconciliation¶
Symptoms: PDBs take long time to be created/updated
Diagnosis:
# Check reconciliation metrics
curl -k https://localhost:8443/metrics | grep reconciliation_duration
# Check API server latency
kubectl logs -n canvas -l app.kubernetes.io/name=pdb-management | grep -i "circuit"
Solutions:
Increase
--max-concurrent-reconcilesCheck API server performance
Verify network connectivity
Debug Mode¶
Enable debug logging for detailed troubleshooting:
# Patch deployment to enable debug logging
kubectl patch deployment controller-manager -n canvas --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/args/2", "value": "--log-level=debug"}]'
Emergency Access¶
If operator is completely down:
# Manual PDB creation (emergency only)
kubectl apply -f - <<EOF
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: emergency-pdb
namespace: <namespace>
spec:
minAvailable: 1
selector:
matchLabels:
app: <app-name>
EOF
Maintenance & Upgrades¶
Backup Procedures¶
Before upgrades, backup critical resources:
# Backup CRDs
kubectl get crd availabilitypolicies.availability.oda.tmforum.org -o yaml > backup-crd.yaml
# Backup policies
kubectl get availabilitypolicy --all-namespaces -o yaml > backup-policies.yaml
# Backup operator configuration
kubectl get deployment controller-manager -n canvas -o yaml > backup-deployment.yaml
Upgrade Procedures¶
Rolling Update (Recommended)¶
# 1. Backup current state
kubectl get availabilitypolicy --all-namespaces -o yaml > pre-upgrade-policies.yaml
# 2. Update operator image
kubectl set image deployment/controller-manager -n canvas manager=<new-image>
# 3. Monitor upgrade
kubectl rollout status deployment/controller-manager -n canvas
# 4. Verify functionality
kubectl get pdb --all-namespaces
Blue-Green Deployment¶
# 1. Deploy new version to different namespace
kubectl apply -k config/default/ -n canvas-new
# 2. Verify new version works
kubectl get pods -n canvas-new
# 3. Switch traffic
kubectl patch svc webhook-service -n canvas -p '{"spec":{"selector":{"app.kubernetes.io/name":"pdb-management-new"}}}'
# 4. Remove old deployment
kubectl delete deployment controller-manager -n canvas
Configuration Changes¶
Update Resource Limits¶
kubectl patch deployment controller-manager -n canvas --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/resources", "value": {"limits": {"cpu": "1000m", "memory": "1Gi"}, "requests": {"cpu": "200m", "memory": "512Mi"}}}]'
Update Concurrency Settings¶
kubectl patch deployment controller-manager -n canvas --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/args/3", "value": "--max-concurrent-reconciles=10"}]'
Emergency Procedures¶
Operator Down¶
Immediate Actions:
# Check operator status kubectl get pods -n canvas -l app.kubernetes.io/name=pdb-management # Check for crash loops kubectl describe pod -n canvas -l app.kubernetes.io/name=pdb-management
Restart Operator:
kubectl delete pods -n canvas -l app.kubernetes.io/name=pdb-management
Manual PDB Management (if needed):
# Create PDBs manually for critical deployments kubectl apply -f emergency-pdbs.yaml
Webhook Failures¶
Disable Webhooks Temporarily:
kubectl delete validatingwebhookconfiguration availabilitypolicy-validating-webhook-configuration kubectl delete mutatingwebhookconfiguration availabilitypolicy-mutating-webhook-configuration
Fix Certificate Issues:
kubectl delete certificate serving-cert -n canvas kubectl apply -f config/certmanager/certificate.yaml
Re-enable Webhooks:
kubectl apply -f config/webhook/
Data Corruption¶
Restore from Backup:
kubectl apply -f backup-policies.yaml kubectl apply -f backup-crd.yaml
Reconcile All Resources:
# Trigger reconciliation for all deployments kubectl annotate deployment --all pdb-management.oda.tmforum.org/reconcile=true
Performance Tuning¶
Resource Optimization¶
Memory Tuning¶
# Recommended memory settings for different cluster sizes
# Small cluster (< 100 deployments)
resources:
limits:
memory: "512Mi"
cpu: "500m"
requests:
memory: "256Mi"
cpu: "100m"
# Large cluster (> 500 deployments)
resources:
limits:
memory: "2Gi"
cpu: "1000m"
requests:
memory: "1Gi"
cpu: "500m"
Cache Tuning¶
// Cache configuration for different workloads
// High-traffic clusters
cache := NewPolicyCache(500, 10*time.Minute)
// Low-traffic clusters
cache := NewPolicyCache(100, 5*time.Minute)
Concurrency Tuning¶
# Adjust based on cluster size and API server capacity
# Small clusters
--max-concurrent-reconciles=3
# Large clusters
--max-concurrent-reconciles=10
Network Tuning¶
# Circuit breaker settings for different network conditions
# Stable networks
--circuit-breaker-threshold=5
--circuit-breaker-timeout=30s
# Unstable networks
--circuit-breaker-threshold=3
--circuit-breaker-timeout=60s
Security¶
RBAC Best Practices¶
# Minimal RBAC for operator
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: pdb-management-operator
rules:
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: ["policy"]
resources: ["poddisruptionbudgets"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
Network Policies¶
# Restrict operator network access
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: pdb-management-network-policy
namespace: canvas
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: pdb-management
policyTypes:
- Ingress
- Egress
ingress:
- from:
- namespaceSelector:
matchLabels:
name: kube-system
ports:
- protocol: TCP
port: 8443
Pod Security¶
The operator already implements restricted security context:
securityContext:
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- "ALL"
Disaster Recovery¶
Backup Strategy¶
Automated Backups¶
#!/bin/bash
# backup-operator.sh
DATE=$(date +%Y%m%d-%H%M%S)
BACKUP_DIR="/backups/pdb-management/$DATE"
mkdir -p $BACKUP_DIR
# Backup CRDs
kubectl get crd availabilitypolicies.availability.oda.tmforum.org -o yaml > $BACKUP_DIR/crd.yaml
# Backup all policies
kubectl get availabilitypolicy --all-namespaces -o yaml > $BACKUP_DIR/policies.yaml
# Backup operator configuration
kubectl get deployment controller-manager -n canvas -o yaml > $BACKUP_DIR/deployment.yaml
# Compress backup
tar -czf $BACKUP_DIR.tar.gz $BACKUP_DIR
Recovery Procedures¶
# 1. Restore CRDs
kubectl apply -f backup/crd.yaml
# 2. Restore policies
kubectl apply -f backup/policies.yaml
# 3. Restore operator
kubectl apply -f backup/deployment.yaml
# 4. Verify recovery
kubectl get availabilitypolicy --all-namespaces
kubectl get pdb --all-namespaces
Multi-Cluster Recovery¶
For multi-cluster environments:
# Export from source cluster
kubectl config use-context source-cluster
kubectl get availabilitypolicy --all-namespaces -o yaml > policies-source.yaml
# Import to target cluster
kubectl config use-context target-cluster
kubectl apply -f policies-source.yaml
Testing Recovery¶
# Test backup/restore procedure
kubectl create namespace test-recovery
kubectl apply -f test-policy.yaml -n test-recovery
# Create backup
./backup-operator.sh
# Simulate disaster
kubectl delete namespace test-recovery
# Restore and verify
kubectl apply -f backup/policies.yaml
kubectl get availabilitypolicy -n test-recovery
Support Contacts¶
Primary On-Call: [Contact Information]
Secondary On-Call: [Contact Information]
Escalation: [Contact Information]