ODA Canvas Observability Stack

Watch: Part 5, Observability in the ODA Canvas

A 9:44 video in the ODA Canvas training series. Watch it with chapters and a transcript, or on YouTube.

This Helm chart deploys a complete open-source observability stack for the TM Forum ODA Canvas, providing monitoring, alerting, and distributed tracing capabilities for ODA Components.

The observability stack installs ServiceMonitor resources for the OpenMetrics configuration. To configure the Canvas to use ServiceMonitor, set the openMetricsImplementation to ServiceMonitor. e.g. for the Istio API Operator option, use:

helm upgrade --install canvas charts/canvas-oda -n canvas --set api-operator-istio.deployment.openMetricsImplementation=ServiceMonitor --create-namespace

Chart-of-Charts

This chart-of-charts includes:

  • Prometheus: Metrics collection and storage

  • Grafana: Visualization and dashboards

  • Jaeger: Distributed tracing

  • OpenTelemetry Collector: Telemetry data processing and forwarding

Prerequisites

  • Kubernetes 1.19+

  • Helm 3.0+

  • At least 4GB of available memory for the observability stack

Installation

Quick Start

# Add the required Helm repositories
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add opentelemetry https://open-telemetry.github.io/opentelemetry-helm-charts  
helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
helm repo update

# Install the observability stack
cd /charts
helm dependency build observability-stack
helm install observability ./observability-stack --create-namespace --namespace monitoring

Custom Installation

# Install with custom values
helm install observability ./observability-stack \
  --create-namespace \
  --namespace monitoring \
  --values my-values.yaml

Configuration

The following table lists the configurable parameters and their default values:

Parameter

Description

Default

global.namespace

Namespace for observability components

monitoring

prometheus.enabled

Enable Prometheus and Grafana

true

opentelemetry.enabled

Enable OpenTelemetry Collector

true

jaeger.enabled

Enable Jaeger

true

kube-prometheus-stack.prometheus.prometheusSpec.retention

Metrics retention period

15d

kube-prometheus-stack.grafana.adminPassword

Grafana admin password

admin

jaeger.allInOne.args

Jaeger configuration arguments

Memory storage with 50000 traces

For Grafana, if no password is set, it will default to prom-operator.

Accessing the Services

Prometheus

kubectl port-forward svc/observability-prometheus-prometheus 9090:9090 -n monitoring

Visit: http://localhost:9090

Grafana

kubectl port-forward svc/observability-grafana 3000:80 -n monitoring

Visit: http://localhost:3000

Get admin password:

kubectl get secret observability-grafana -n monitoring -o jsonpath="{.data.admin-password}" | base64 -d

Jaeger

kubectl port-forward svc/observability-jaeger-query 16686:16686 -n monitoring

Visit: http://localhost:16686

BDD Testing

The Usecase UC012 contains feature tests for the observability. For the tests to work, you need to port-forward the observability-opentelemetry-collector deployment.

kubectl port-forward -n monitoring deployment/observability-opentelemetry-collector 8888:8888

ODA Component Integration

Metrics Collection

When you install a component with a openmetrics ExposedAPI, the Canvas will configure whatever Observability services you have installed. The reference implementation using Prometheus, Grafana, Jaeger amd OpenTelemetry Collector will create a ServiceMonitor resource in your component namespace. Something like:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-component-metrics
  namespace: components
spec:
  selector:
    matchLabels:
      app: my-component
  endpoints:
  - port: metrics
    path: /metrics
    interval: 30s

Distributed Tracing

Configure your ODA Component with these environment variables:

env:
- name: OTEL_EXPORTER_OTLP_ENDPOINT
  value: "http://observability-opentelemetry-collector.monitoring.svc.cluster.local:4318/v1/traces"
- name: OTEL_RESOURCE_ATTRIBUTES
  value: "service.name=my-component"

Storage Considerations

This chart uses in-memory storage for Jaeger and default storage for Prometheus. For production deployments, consider:

  • Configuring persistent storage for Prometheus

  • Using external storage backends for Jaeger (Elasticsearch, Cassandra)

  • Implementing proper backup strategies

Scaling

To scale the observability stack:

  1. Prometheus: Configure multiple replicas and sharding

  2. Grafana: Configure multiple replicas behind a load balancer

  3. Jaeger: Switch to distributed deployment with external storage

  4. OpenTelemetry: Scale the collector as a DaemonSet or increase replicas

Troubleshooting

Common Issues

  1. Pods stuck in Pending state: Check resource requirements and node capacity

  2. Metrics not appearing: Verify ServiceMonitor labels and selectors

  3. Traces not showing: Check OpenTelemetry collector logs and endpoint configuration

Useful Commands

# Check all pods
kubectl get pods -n monitoring

# Check services
kubectl get svc -n monitoring

# Check ServiceMonitor resources
kubectl get servicemonitor -A

# View logs
kubectl logs -n monitoring deployment/observability-opentelemetry-collector

Uninstallation

helm uninstall observability -n monitoring
kubectl delete namespace monitoring

Demonstration

Using the example Product Catalog component, you can observe the metrics and telemetry as follows:

Open-Metrics

The example component implements an optional metrics API supporting the open metrics standard (formerly the prometheus de-facto standard). This metrics endpoint provides business metrics about all the Create/Update/Delete events for all the Product Catalog resources (Catalog, Category, Product Offering, Product Offering Price, Product Specification).

The observability-stack includes a Prometheus observability service that can scrape the metrics API and report on these business events. For example the screenshot below shows a graph of the rate of Catalog Create events with the query rate(product_catalog_api_counter{NotificationEvent="CatalogCreationNotification"}[5m]).

alt text

Open-Telemetry

The component also generates Open-Telemetry events that can either be logged to the console using otlp.console.enabled:true or sent to an Open-Telemetry protobuffCollector. You can set this in the values.yaml file as follows:

  otlp:
    console:
      enabled: false
    protobuffCollector:
      enabled: true
      url: http://observability-opentelemetry-collector.monitoring.svc.cluster.local:4318/v1/traces

With telemetry enabled, you can view traces of executions in Jaeger:

alt text

And drill down into an individual trace:

alt text

Contributing

Please read the CONTRIBUTING.md file for details on how to contribute to this project.

Support

For support, please create an issue in the ODA Canvas repository.