PDB Management Operator - Performance and Caching¶
Overview¶
The PDB Management Operator implements sophisticated performance optimization strategies to efficiently handle large-scale deployments (200+ deployments) with minimal resource overhead. This document details the comprehensive caching system, adaptive circuit breaker, and performance optimization techniques.
Performance Architecture¶
Multi-Level Caching System¶
The operator implements a three-tier caching strategy:
flowchart TB
subgraph CachingArch["Caching Architecture"]
subgraph Caches["Cache Tiers"]
PolicyCache["<b>Policy Cache</b><br/>5m TTL<br/>• Individual policies<br/>• 100 entries<br/>• Key-based lookup"]
ListCache["<b>List Cache</b><br/>5m TTL<br/>• Bulk policy queries<br/>• Expensive operations<br/>• Auto-invalidation"]
MaintenanceCache["<b>Maintenance Cache</b><br/>1m TTL<br/>• Window checks<br/>• Short TTL<br/>• Time-sensitive"]
end
subgraph Eviction["Adaptive Eviction & Cleanup"]
LRU["LRU eviction when cache full"]
Cleanup["Background cleanup every 2.5 minutes"]
Stats["Statistics tracking<br/>(hits/misses/evictions)"]
end
end
PolicyCache --> Eviction
ListCache --> Eviction
MaintenanceCache --> Eviction
1. Policy Cache¶
Purpose: Cache individual AvailabilityPolicy lookups to reduce API calls.
Configuration:
Size: 100 entries
TTL: 5 minutes
Eviction: LRU (Least Recently Used)
Implementation Details:
// Cache entry structure
type policyCacheEntry struct {
policy *availabilityv1alpha1.AvailabilityPolicy
timestamp time.Time
}
// Thread-safe operations with RWMutex
pc.mu.RLock()
defer pc.mu.RUnlock()
Cache Keys: namespace/policyname format for individual policies.
2. List Cache¶
Purpose: Cache expensive list operations when fetching all policies.
Configuration:
TTL: 5 minutes (same as policy cache)
Key:
"all-policies"for full policy listsAutomatic Invalidation: Invalidated when any policy changes
Optimization: When a list is cached, individual policies are also cached automatically.
3. Maintenance Window Cache¶
Purpose: Cache maintenance window calculations with short TTL.
Configuration:
TTL: 1 minute (shorter due to time-sensitive nature)
Purpose: Avoid repeated time calculations
Adaptive Circuit Breaker¶
Overview¶
The operator implements an adaptive circuit breaker that learns cluster characteristics and automatically adjusts parameters based on cluster performance.
Learning Phases¶
flowchart TD
subgraph CB["Circuit Breaker Phases"]
subgraph Phase1["Phase 1: LEARNING (2 minutes)"]
L1["Probe cluster performance"]
L2["Collect baseline metrics"]
L3["Conservative default settings"]
L4["Monitor: P50, P95, P99, error rates"]
end
subgraph Phase2["Phase 2: ADAPTATION (ongoing, every 30s)"]
A1["Adjust timeout based on P95 response times"]
A2["Modify failure thresholds based on error rates"]
A3["Scale max requests based on throughput"]
A4["Continuous optimization"]
end
subgraph Phase3["Phase 3: PROTECTION (when needed)"]
P1["Circuit opens on threshold breach"]
P2["Fail-fast to prevent cascade failures"]
P3["Automatic recovery testing"]
P4["Dynamic threshold adjustment"]
end
Phase1 --> Phase2
Phase2 --> Phase3
Phase3 -.->|"Recovery"| Phase2
end
Adaptive Configuration¶
Base Settings (Conservative defaults):
baseMaxRequests: 5
baseTimeout: 30s
baseFailureRatio: 0.6
baseConsecutiveFailures: 5
Adaptation Parameters:
learningPeriod: 2m # Initial learning phase
adjustmentInterval: 30s # How often to adjust settings
maxTimeoutMultiplier: 3.0 # Maximum timeout increase
minTimeoutMultiplier: 0.5 # Minimum timeout decrease
Cluster Performance Probing¶
Initial Probe Operations:
List Namespaces - Lightweight operation for baseline
List Deployments - Standard operation testing
List Policies - CRD-specific operation testing
Metrics Collected:
Response Time Percentiles: P50, P95, P99
Error Rate: Percentage of failed operations
Throughput: Requests per second capacity
Performance Optimization Techniques¶
1. Change Detection¶
Purpose: Avoid unnecessary reconciliations by detecting actual changes.
Implementation:
// State tracking per deployment
lastDeploymentState map[types.NamespacedName]string
// Change detection
func (r *DeploymentReconciler) hasDeploymentStateChanged(ctx context.Context,
deployment *appsv1.Deployment, config *AvailabilityConfig) (bool, error) {
currentState := r.calculateDeploymentStateHash(deployment, config)
r.mu.RLock()
lastState, exists := r.lastDeploymentState[namespacedName]
r.mu.RUnlock()
return !exists || lastState != currentState, nil
}
Tracked Changes:
Deployment generation
Availability class changes
Policy configuration changes
Replica count changes
2. Batch Processing¶
Event Predicates: Filter unnecessary events at the controller level.
deploymentPredicate := predicate.Funcs{
UpdateFunc: func(e event.UpdateEvent) bool {
// Skip if only status changed
oldDeployment := e.ObjectOld.(*appsv1.Deployment)
newDeployment := e.ObjectNew.(*appsv1.Deployment)
if oldDeployment.Generation == newDeployment.Generation {
return oldDeployment.DeletionTimestamp == nil &&
newDeployment.DeletionTimestamp != nil
}
return true
},
}
3. Resource Optimization¶
Memory Management:
Efficient object reuse
Deep copy only when necessary
Limited cache sizes with eviction
CPU Optimization:
Concurrent reconcilers (configurable: 3-10)
Non-blocking cache operations
Efficient predicate filtering
Performance Monitoring¶
Key Metrics¶
# Cache Performance
pdb_management_cache_hits_total{cache_type="policy"}
pdb_management_cache_misses_total{cache_type="policy"}
# Reconciliation Performance
pdb_management_reconciliation_duration_seconds{controller="deployment-pdb"}
pdb_management_reconciliation_errors_total{controller="deployment-pdb"}
# Circuit Breaker Status
pdb_management_circuit_breaker_state{operation="create-pdb",state="open|closed|half-open"}
# Retry Metrics
pdb_management_retry_attempts_total{operation,error_type,attempt}
pdb_management_retry_exhausted_total{operation,error_type}
pdb_management_retry_success_after_retry_total{operation,attempts}
# Multi-Policy Resolution
pdb_management_multi_policy_matches_total{namespace,policy_count}
pdb_management_policy_tie_breaks_total{namespace,resolution_method}
# Webhook Status
pdb_management_webhook_status{status,reason}
# Resource Usage
pdb_management_deployments_managed{namespace,availability_class}
process_resident_memory_bytes
go_goroutines
Performance Targets¶
Metric |
Target |
Description |
|---|---|---|
Cache Hit Rate |
> 90% |
Policy cache effectiveness |
Reconciliation Time |
< 100ms avg |
Average reconciliation duration |
Memory Usage |
< 100MB |
Operator memory consumption |
CPU Usage |
< 100m |
Steady-state CPU usage |
Error Rate |
< 1% |
Reconciliation error percentage |
Grafana Dashboard Queries¶
Cache Hit Rate:
rate(pdb_management_cache_hits_total[5m]) /
(rate(pdb_management_cache_hits_total[5m]) + rate(pdb_management_cache_misses_total[5m])) * 100
Average Reconciliation Time:
rate(pdb_management_reconciliation_duration_seconds_sum[5m]) /
rate(pdb_management_reconciliation_duration_seconds_count[5m])
Circuit Breaker Health:
pdb_management_circuit_breaker_state{state="open"} == 1
Configuration and Tuning¶
Command-Line Flags¶
The operator supports extensive configuration via CLI flags:
Cache Configuration¶
Flag |
Default |
Description |
|---|---|---|
|
|
TTL for cached policies |
|
|
Maximum number of policies to cache |
|
|
TTL for cached maintenance window results |
Retry Configuration¶
Flag |
Default |
Description |
|---|---|---|
|
|
Maximum retry attempts for transient errors |
|
|
Initial delay between retries |
|
|
Maximum delay between retries |
|
|
Backoff multiplier for retry delays |
Other Configuration¶
Flag |
Default |
Description |
|---|---|---|
|
|
Concurrent reconciliations |
|
|
Enable admission webhooks |
Tuning for Different Environments¶
Small Clusters (< 50 deployments)¶
resources:
requests:
memory: "64Mi"
cpu: "50m"
limits:
memory: "128Mi"
cpu: "200m"
args:
- --policy-cache-size=50
- --max-concurrent-reconciles=3
Medium Clusters (50-200 deployments)¶
resources:
requests:
memory: "128Mi"
cpu: "100m"
limits:
memory: "256Mi"
cpu: "500m"
args:
- --policy-cache-size=100
- --max-concurrent-reconciles=5
Large Clusters (200+ deployments)¶
resources:
requests:
memory: "256Mi"
cpu: "200m"
limits:
memory: "512Mi"
cpu: "1000m"
args:
- --policy-cache-size=200
- --max-concurrent-reconciles=10
- --policy-cache-ttl=10m
Retry Configuration Tuning¶
Default (Stable Networks)¶
args:
- --retry-max-attempts=5
- --retry-initial-delay=100ms
- --retry-max-delay=30s
- --retry-backoff-factor=2.0
Unstable Networks (More Aggressive Retries)¶
args:
- --retry-max-attempts=10
- --retry-initial-delay=200ms
- --retry-max-delay=60s
- --retry-backoff-factor=1.5
Fast Fail (Low Latency Requirements)¶
args:
- --retry-max-attempts=3
- --retry-initial-delay=50ms
- --retry-max-delay=5s
- --retry-backoff-factor=2.0
Troubleshooting Performance Issues¶
Cache Performance Issues¶
Symptoms: High cache miss rate (< 80%)
Diagnosis:
# Check cache metrics
kubectl port-forward -n canvas svc/controller-manager-metrics-service 8443:8443
curl -s https://localhost:8443/metrics | grep cache
# Look for high miss rate
pdb_management_cache_misses_total / (pdb_management_cache_hits_total + pdb_management_cache_misses_total)
Solutions:
Increase cache size: Set
--policy-cache-size=200Increase TTL: Set
--policy-cache-ttl=10mCheck policy churn: Reduce policy update frequency
High Reconciliation Latency¶
Symptoms: Reconciliation duration > 500ms
Diagnosis:
# Check reconciliation metrics
kubectl logs -n canvas deployment/pdb-management-controller-manager | \
jq 'select(.details.durationMs != null) | .details.durationMs' | \
sort -n | tail -10
Solutions:
Increase concurrency: Set
--max-concurrent-reconciles=10Optimize predicates: Ensure proper event filtering
Check API server latency: Monitor cluster performance
Retry Exhaustion¶
Symptoms: Operations failing after all retries
Diagnosis:
# Check retry metrics
curl -s https://localhost:8443/metrics | grep retry
# Look for exhausted retries
kubectl logs -n canvas deployment/pdb-management-controller-manager | \
grep -i "retry.*exhausted"
Solutions:
Increase max attempts: Set
--retry-max-attempts=10Increase max delay: Set
--retry-max-delay=60sfor transient network issuesCheck cluster health: Monitor API server and etcd performance
Memory Issues¶
Symptoms: Operator OOMKilled or high memory usage
Diagnosis:
# Check memory metrics
kubectl top pod -n canvas -l app.kubernetes.io/name=pdb-management
# Check for memory leaks
kubectl exec -n canvas deployment/pdb-management-controller-manager -- \
curl http://localhost:8080/debug/pprof/heap > heap.prof
Solutions:
Increase memory limits: Set higher resource limits
Reduce cache size: Lower
--policy-cache-sizevalueReduce TTL: Lower
--policy-cache-ttlto reduce memory usage
Performance Testing¶
Built-in Performance Test Suite¶
The operator includes comprehensive performance testing tools:
# Run basic performance test (100 deployments)
./toolsforlocal/unified-performance-test.sh --basic 100
# Run cache performance test
./toolsforlocal/unified-performance-test.sh --cache
# Run advanced stress test (10 minutes)
./toolsforlocal/unified-performance-test.sh --advanced 600
# Run circuit breaker test
./toolsforlocal/unified-performance-test.sh --circuit-breaker
# Run all tests
./toolsforlocal/unified-performance-test.sh --all
Custom Performance Testing¶
Load Test Configuration:
# Generate test load
for i in {1..200}; do
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: perf-test-$i
annotations:
oda.tmforum.org/availability-class: "standard"
spec:
replicas: 3
selector:
matchLabels:
app: perf-test-$i
template:
metadata:
labels:
app: perf-test-$i
spec:
containers:
- name: app
image: nginx:alpine
resources:
requests:
cpu: 1m
memory: 1Mi
EOF
done
Performance Validation:
# Measure PDB creation time
start_time=$(date +%s)
kubectl wait --for=condition=Ready pdb --all --timeout=300s
end_time=$(date +%s)
echo "PDB creation took $((end_time - start_time)) seconds for 200 deployments"
# Target: < 5 minutes for 200 PDBs
Best Practices¶
1. Cache Optimization¶
Monitor cache hit rates regularly
Adjust cache size based on cluster size
Use appropriate TTL values (not too short, not too long)
2. Resource Planning¶
Start conservative with resource requests
Monitor actual usage and adjust limits
Use horizontal pod autoscaling for high-load clusters
3. Monitoring Setup¶
Set up alerts for cache miss rates > 20%
Monitor reconciliation latency with P95/P99 percentiles
Track circuit breaker state changes
4. Deployment Strategy¶
Gradual rollout in large clusters
Test performance in staging environments
Monitor during updates and rollbacks
The performance optimization features in this operator enable it to efficiently manage hundreds of deployments while maintaining low latency and resource usage. The adaptive circuit breaker and multi-level caching system work together to provide both resilience and efficiency at scale.