Pushing metrics using Python OT SDK

Overview

This documentation provides a step-by-step guide to instrumenting a Python application to send metrics to Turo's Metrics Ingestion System using OpenTelemetry. This enables you to collect, send, and analyze telemetry data from your application.

Prerequisites

• Python 3.6 or higher
• OpenTelemetry SDK for Python

Installation

Install the necessary OpenTelemetry packages using pip:

pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp

Setup

Export the credentials gathered from Turo.

You can get the metrics endpoint and bearer token from the Turo app. For a step-by-step guide check the Model Registration guide

export TURO_METRICS_ENDPOINT="http://turo.quantumblack.com/model/<your-model>/metrics"
export TURO_AUTH_TOKEN="<your-secret-token>"

OpenTelemetry Configuration

Configure OpenTelemetry in your Python application. Create a file named otel_config.py:

# otel_config.py
import os
from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader, ConsoleMetricExporter
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.resources import SERVICE_NAME, Resource

turo_metrics_endpoint = os.getenv('TURO_METRICS_ENDPOINT')
turo_auth_token = os.getenv('TURO_AUTH_TOKEN')

# Configure the OTLP Metric Exporter to send metrics to the ingestion system
exporter = OTLPMetricExporter(
    endpoint=turo_metrics_endpoint,
    headers={"Authorization": f"Bearer {turo_auth_token}"}
)

# Create a Periodic Exporting Metric Reader
metric_reader = PeriodicExportingMetricReader(exporter)

# Create a Console Metric Exporter
console_exporter = ConsoleMetricExporter()
console_reader = PeriodicExportingMetricReader(console_exporter, export_interval_millis=3000)

## Optionally you can set the Metric reader to export metrics only on ad-hoc basis.
# metric_reader = PeriodicExportingMetricReader(exporter=exporter, export_interval_millis=math.inf)

# Create a Resource to associate with all metrics
resource = Resource.create(attributes={
    SERVICE_NAME: "example-service",
    "service.instance.id": "instance-1"
})

# Set the Meter Provider
metrics.set_meter_provider(MeterProvider(resource=resource, metric_readers=[metric_reader, console_reader]))

# Get a Meter
meter = metrics.get_meter(__name__)

# Optional: Collect metrics on an ad-hoc basis
def collect_metrics():
    metric_reader.collect()

# Example usage: Call collect_metrics() to send metrics immediately

Instrumenting Your Application

Instrument your application to create and record metrics. Below is an example of how to instrument a simple python application that outputs random cpu usage:

# main.py
from otel_config import meter, collect_metrics
from time import sleep
import random

# Create a counter instrument
cpu_usage_counter = meter.create_gauge(
    name="cpu_usage",
    description="CPU usage in percentage",
    unit="%"
)

# Function to record CPU usage
def record_cpu_usage():
    while True:
        # Simulate CPU usage
        cpu_usage = random.uniform(0, 100)
        
        # Record the CPU usage
        cpu_usage_counter.set(cpu_usage, {"host.name": "host1"})

        # Optionally send metrics immediately - Also use if set interval to math.inf to allow ad-hoc metrics export.
        collect_metrics()

        # Sleep for a short interval
        sleep(3)

if __name__ == "__main__":
    record_cpu_usage()

Running the Application

Ensure the environment variables for the Metrics Ingestion System are configured, then run your application:

python main.py

The application should output an OTLP payload for the generated metrics. Remove the console_reader from the metric_readers list in otel_config.py to remove the console output.

Example Output

{
    "resource_metrics": [
        {
            "resource": {
                "attributes": {
                    "telemetry.sdk.language": "python",
                    "telemetry.sdk.name": "opentelemetry",
                    "telemetry.sdk.version": "1.24.0",
                    "service.name": "example-service",
                    "service.instance.id": "instance-1"
                },
                "schema_url": ""
            },
            "scope_metrics": [
                {
                    "scope": {
                        "name": "otel_config",
                        "version": "",
                        "schema_url": ""
                    },
                    "metrics": [
                        {
                            "name": "cpu_usage",
                            "description": "CPU usage in percentage",
                            "unit": "%",
                            "data": {
                                "data_points": [
                                    {
                                        "attributes": {
                                            "host.name": "host1"
                                        },
                                        "start_time_unix_nano": 0,
                                        "time_unix_nano": 1721128037561857000,
                                        "value": 2.123770961342475
                                    }
                                ]
                            }
                        }
                    ],
                    "schema_url": ""
                }
            ],
            "schema_url": ""
        }
    ]
}

Logs and Debugging

Enable logging in OpenTelemetry to troubleshoot issues. Add the following to your otel_config.py:

import logging

logging.basicConfig(level=logging.DEBUG)