View Observability Data¶
Table of Contents¶
Observability in ODA Canvas¶
The ODA Canvas supports three pillars of observability, each tied to the component specification:
Metrics (Prometheus): Each component declares a management function ExposedAPI with apiType: prometheus or apiType: openmetrics. This management API exposes a /metrics endpoint. The API Operator creates a ServiceMonitor resource that tells Prometheus where to scrape. Example from the test data (feature-definition-and-test-kit/testData/productcatalog-v1/templates/component-productcatalog.yaml):
managementFunction:
exposedAPIs:
- name: metrics
apiType: prometheus
implementation: <release>-productcatalogmanagement-sm
path: /<release>-productcatalogmanagement/metrics
port: 4000
Traces (Jaeger via OpenTelemetry): Components send OTLP traces to an OpenTelemetry Collector, which forwards them to Jaeger. The collector URL is configured via the OTL_EXPORTER_TRACE_PROTO_COLLECTOR_URL environment variable in the component’s deployment.
Dashboards (Grafana): Grafana connects to Prometheus as a data source, enabling dashboard creation for component metrics.
ODA Components → OTLP HTTP (:4318) → OpenTelemetry Collector → Jaeger
ODA Components → /metrics endpoint → Prometheus (via ServiceMonitor) → Grafana
Auto-Discover Services¶
The observability stack is optional and may not be installed. Auto-discover using the helper script:
kubectl get svc -A -o json | python <scripts>/discover_observability.py
This script finds Prometheus, Grafana, Jaeger, and OpenTelemetry services across namespaces, shows service names/namespaces/types/ports, suggests port-forward commands, and prints installation instructions if nothing is found.
Handling Missing Observability Stack¶
If no services are found, provide installation instructions. Full docs: https://github.com/tmforum-oda/oda-canvas/tree/main/charts/observability-stack
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add opentelemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
helm repo update
cd charts
helm dependency build observability-stack
helm install observability ./observability-stack --create-namespace --namespace monitoring
Configuring the Canvas to use ServiceMonitors:
helm upgrade --install canvas charts/canvas-oda -n canvas \
--set api-operator-istio.deployment.openMetricsImplementation=ServiceMonitor \
--create-namespace
Key defaults:
Parameter |
Default |
|---|---|
|
|
|
|
|
|
|
|
Jaeger storage |
In-memory, 50,000 traces max |
Prometheus retention |
15 days |
Check ServiceMonitor CRD availability:
kubectl get crd servicemonitors.monitoring.coreos.com 2>&1
ServiceMonitor Configuration¶
Verify the api-operator-istio is configured to create ServiceMonitors:
kubectl get configmap api-operator-istio-configmap -n canvas -o jsonpath='{.data.OPENMETRICS_IMPLEMENTATION}'
ServiceMonitor— correct, no action needed.DataDogAnnotationorPrometheusAnnotation— change toServiceMonitor:
kubectl patch configmap api-operator-istio-configmap -n canvas --type merge -p '{"data":{"OPENMETRICS_IMPLEMENTATION":"ServiceMonitor"}}'
kubectl rollout restart deployment -n canvas -l app.kubernetes.io/name=api-operator-istio
After changing, existing components need redeployment (helm delete + helm install) for ServiceMonitors. Set permanently:
helm upgrade canvas charts/canvas-oda -n canvas \
--set api-operator-istio.deployment.openMetricsImplementation=ServiceMonitor
Port-Forward and Open UIs¶
Default namespace for observability is monitoring. Use the discovered namespace from auto-discovery.
Before starting port-forward, check if ports are already in use from a prior session. On PowerShell:
# Check all three ports at once
@(9090, 3000, 16686) | ForEach-Object {
$conn = Get-NetTCPConnection -LocalPort $_ -ErrorAction SilentlyContinue
if ($conn) { "Port $_ is ALREADY IN USE (PID $($conn[0].OwningProcess))" }
else { "Port $_ is available" }
}
On bash:
for port in 9090 3000 16686; do
if ss -tlnp 2>/dev/null | grep -q ":$port " || lsof -i :"$port" >/dev/null 2>&1; then
echo "Port $port is ALREADY IN USE"
else
echo "Port $port is available"
fi
done
Only start port-forward for ports that are not already in use. If a port is already forwarded, skip it and proceed to open the UI.
# Prometheus (default port 9090)
kubectl port-forward svc/observability-prometheus-prometheus 9090:9090 -n monitoring
# Grafana (default port 3000, service port 80)
kubectl port-forward svc/observability-grafana 3000:80 -n monitoring
# Jaeger UI (default port 16686)
kubectl port-forward svc/observability-jaeger-query 16686:16686 -n monitoring
Run port-forward as a background process. Open UIs in VS Code Simple Browser:
Prometheus:
http://localhost:9090Grafana:
http://localhost:3000— Default password:prom-operator. Retrieve from secret:kubectl get secret observability-grafana -n monitoring -o jsonpath="{.data.admin-password}" | base64 -d
Jaeger:
http://localhost:16686
Proactive Prometheus Guidance¶
When the user opens Prometheus or asks about it, proactively offer:
Generate metrics data if
product_catalog_api_counterhas low/zero values — suggestexercise_catalog_api.py.Useful PromQL queries:
Query
Description
product_catalog_api_counterCurrent totals by notification event type
rate(product_catalog_api_counter[5m])Event rate per second over 5 minutes
sum by (NotificationEvent) (rate(product_catalog_api_counter[5m]))Rate grouped by event type (best for Graph tab)
sum(rate(product_catalog_api_counter{NotificationEvent=~".*CreationNotification"}[5m]))Aggregate creation rate
sum(rate(product_catalog_api_counter{NotificationEvent=~".*RemoveNotification"}[5m]))Aggregate removal rate
sum(rate(product_catalog_api_counter[5m])) * 60Total events per minute
container_cpu_usage_seconds_total{namespace="components"}CPU usage of component pods
container_memory_working_set_bytes{namespace="components"}Memory usage of component pods
Tip: Use
rate()queries in the Graph tab with a 15m–1h range. Use instant totals in the Table tab.Workflow suggestion when no recent data exists:
Run
exercise_catalog_api.py --rounds 3Wait 30 seconds for Prometheus to scrape
Enter
sum by (NotificationEvent) (rate(product_catalog_api_counter[5m]))in Graph tabSet time range to 15 minutes
Exercising the API for Metrics¶
Generate notification events for the product_catalog_api_counter Prometheus metric:
python <scripts>/exercise_catalog_api.py <api-base-url> [--rounds N] [--cleanup]
The API base URL is the ExposedAPI URL for the productcatalogmanagement core function API. For release r2:
# One round: creates 14 resources across 4 types, deletes half
python <scripts>/exercise_catalog_api.py https://localhost/r2-productcatalogmanagement/tmf-api/productCatalogManagement/v4
# 3 rounds for more data
python <scripts>/exercise_catalog_api.py https://localhost/r2-productcatalogmanagement/tmf-api/productCatalogManagement/v4 --rounds 3
# Clean up
python <scripts>/exercise_catalog_api.py https://localhost/r2-productcatalogmanagement/tmf-api/productCatalogManagement/v4 --cleanup
Resource Type |
Create Notification |
Delete Notification |
Count |
|---|---|---|---|
|
CategoryCreationNotification |
CategoryRemoveNotification |
5 |
|
CatalogCreationNotification |
CatalogRemoveNotification |
2 |
|
ProductSpecificationCreationNotification |
ProductSpecificationRemoveNotification |
4 |
|
ProductOfferingCreationNotification |
ProductOfferingRemoveNotification |
3 |
Each round: ~14 creation events and ~7 deletion events. Resources tagged [exercise-catalog-api], removable with --cleanup.
After running, verify:
kubectl exec -n components deploy/<release>-metricsapi -- wget -qO- http://localhost:4000/<component>/metrics
Proactive Jaeger Troubleshooting¶
When the user opens Jaeger or asks about traces, always verify components are sending traces to the correct OpenTelemetry Collector URL. Reference example components default to a DataDog agent URL that doesn’t exist in the reference Canvas.
Step 1: Check current collector URL:
kubectl get deploy <release>-prodcatapi -n components \
-o jsonpath="{.spec.template.spec.containers[0].env[?(@.name=='OTL_EXPORTER_TRACE_PROTO_COLLECTOR_URL')].value}"
Step 2: Find correct URL:
kubectl get svc -A | Select-String 'opentelemetry-collector'
Correct format: http://<collector-service>.<namespace>.svc.cluster.local:4318/v1/traces
Reference stack default: http://observability-opentelemetry-collector.monitoring.svc.cluster.local:4318/v1/traces
Step 3: Fix if wrong (points to datadog-agent or non-existent service):
helm upgrade <release> oda-components/productcatalog -n components \
--set api.otlp.protobuffCollector.url=http://observability-opentelemetry-collector.monitoring.svc.cluster.local:4318/v1/traces \
--reuse-values
Use
--reuse-valuesto preserve other settings (e.g.component.dependentAPIs.enabled=true).
Step 4: Verify traces flowing:
python <scripts>/exercise_catalog_api.py https://localhost/<release>-productcatalogmanagement/tmf-api/productCatalogManagement/v4
# PowerShell:
(Invoke-RestMethod http://localhost:16686/api/services).data
# bash:
curl -s http://localhost:16686/api/services | python -m json.tool
Component should appear as <release>-productcatalogmanagement in Jaeger services.
Common symptoms of misconfigured collector URL:
Jaeger shows no services or only
jaeger-all-in-oneComponent logs show no errors (traces silently dropped)
OpenTelemetry Collector logs show only
MetricsExporterentries, noTracesExporter
Kubernetes Observability Resources¶
kubectl get pods -n monitoring
kubectl get svc -n monitoring
kubectl get servicemonitors -A 2>&1
kubectl get pods -n monitoring -l app.kubernetes.io/name=opentelemetry-collector
kubectl logs -n monitoring deployment/observability-opentelemetry-collector
If kubectl get servicemonitors returns “the server doesn’t have a resource type”, ServiceMonitor-based metrics discovery is not available.
For UC012 BDD tests, port-forward the collector:
kubectl port-forward -n monitoring deployment/observability-opentelemetry-collector 8888:8888
Architecture Reference¶
ODA Components → OTLP HTTP (:4318) → OpenTelemetry Collector → Jaeger Collector (:4317) → Jaeger UI
ODA Components → /metrics endpoint → Prometheus (ServiceMonitor discovery) → Grafana dashboards
After explaining the architecture, note how this connects to the component specification:
The metrics endpoint is defined in
spec.managementFunction.exposedAPIs[]withapiType: prometheus— this is how the component tells the Canvas where to scrape metricsThe trace collector URL is an infrastructure configuration, not part of the Component CRD — it’s injected via Helm values or environment variables
ServiceMonitors are Kubernetes resources created by the API Operator when it processes an ExposedAPI with
apiType: prometheusorapiType: openmetrics
To see ServiceMonitors created for components:
kubectl get servicemonitors -n components
Contextual Next Steps¶
Present these options using ask_questions:
# |
Next Step |
|---|---|
1 |
Exercise the API for metrics — Generate traffic to populate Prometheus counters |
2 |
View ExposedAPIs — Check which components expose prometheus/openmetrics management APIs |
3 |
Run BDD tests (UC012) — Validate the Canvas observability configuration |
4 |
View deployed components — Check component status and API readiness |
5 |
Return to main menu |