Mastering OpenTelemetry in Python on Azure: Troubleshooting Common Issues

Table of Contents

This article provides guidance on diagnosing and resolving typical issues encountered when using OpenTelemetry with Python applications deployed on Microsoft Azure, particularly when exporting telemetry to Azure Monitor. Implementing effective monitoring is crucial for application health and performance, and understanding common pitfalls can significantly streamline the troubleshooting process. We will explore various scenarios and provide actionable steps to ensure your telemetry data flows correctly and accurately reflects your application’s behavior.

Troubleshooting Checklist¶

When facing issues with OpenTelemetry telemetry not appearing in Azure Monitor, or exhibiting unexpected behavior like duplication or missing data, a structured approach is essential. Starting with fundamental checks can often quickly pinpoint the root cause. This checklist provides a starting point for your investigation.

Enable Diagnostic Logging¶

One of the most effective first steps in troubleshooting any telemetry issue is enabling verbose logging within your OpenTelemetry setup. The Azure Monitor Exporter for OpenTelemetry, as well as other OpenTelemetry components, utilize the standard Python logging library for internal messages. These logs contain valuable information about the exporter’s activity, any errors encountered during telemetry processing, or issues connecting to the ingestion service.

By default, the Python logging library’s root logger is often configured to a WARNING level or higher, meaning informational or debug messages are suppressed. You need to explicitly configure the logging level to DEBUG or INFO to see the detailed logs generated by OpenTelemetry and the Azure Monitor Exporter. This allows you to observe the telemetry items being processed, attempts to export data, and any failures that occur during these operations. Configuring logging early in your application’s lifecycle is important to capture setup and initialization messages.

The following Python code snippet demonstrates how to configure the root logger to capture logs of all severity levels (DEBUG and above) and direct them to both the console (standard output) and a file for later analysis. This comprehensive logging setup is invaluable during development and initial deployment troubleshooting.

import logging
import sys

# Configure the root logger to output DEBUG level messages and higher
# Format the output to include timestamp, level, and message
logging.basicConfig(
    format="%(asctime)s:%(levelname)s:%(name)s:%(message)s",
    level=logging.DEBUG,
    stream=sys.stdout # Also direct logs to standard output
)

# Optionally, create a file handler for persistent logging
# logger = logging.getLogger() # Get the root logger
# file_handler = logging.FileHandler("opentelemetry_debug.log")
# file_handler.setLevel(logging.DEBUG)
# formatter = logging.Formatter("%(asctime)s:%(levelname)s:%(name)s:%(message)s")
# file_handler.setFormatter(formatter)
# logger.addHandler(file_handler)

print("Logging configured to DEBUG level...")

# Example of logging different levels
logging.debug("This is a debug message from your application.")
logging.info("This is an informational message.")
logging.warning("This is a warning message.")
logging.error("This is an error message.")

# Your application logic and OpenTelemetry setup follow here
# ...

Remember that for production environments, enabling DEBUG level logging everywhere might generate excessive log volume. Consider using a DEBUG level during troubleshooting and reverting to INFO or WARNING for normal operations, or implementing dynamic logging level configuration. Analyzing these logs can reveal if telemetry is being generated by your application, whether the exporter is attempting to send it, and if there are any errors during transmission, such as authentication failures or network issues.

Test Connectivity Between Your Application Host and the Ingestion Service¶

Telemetry data collected by your OpenTelemetry setup is ultimately sent to the Azure Monitor ingestion endpoints. These endpoints are specific URLs that receive and process the telemetry streams (traces, metrics, logs). If your application host cannot establish a connection to these endpoints, no telemetry will reach Azure Monitor, regardless of whether it’s correctly generated by your application. Network connectivity issues are a very common cause of missing telemetry.

To diagnose connectivity problems, you can perform tests directly from the server or container hosting your Python application. Tools like curl, telnet, or PowerShell’s Invoke-RestMethod can be used to attempt connections to the relevant Azure Monitor ingestion endpoints. The specific endpoint depends on the region where your Application Insights resource is deployed. You can find the correct endpoint URL in the Application Insights resource’s configuration blade in the Azure portal. Common endpoints follow patterns like https://<region>.ingest.monitor.azure.com/.

You can use curl to simulate sending data or just test basic connectivity over HTTPS. For example, a simple curl -v https://<your-ingestion-endpoint-url> can show you the connection handshake details and any errors. Firewalls (both server-side and network-level), proxy servers, and incorrect DNS resolution are frequent culprits for failed connections. Ensure that outbound connections over port 443 (HTTPS) are permitted from your application host to the Azure Monitor ingestion endpoints.

Using telnet <ingestion-endpoint-hostname> 443 can quickly verify if a TCP connection can be established to the endpoint’s port. If telnet fails, it strongly indicates a network blockage. If telnet succeeds but HTTPS requests (via curl or your application) fail, the issue might be related to TLS/SSL certificates, proxy configuration, or HTTP protocol issues, although this is less common with standard libraries and SDKs. Consulting the Azure documentation for Application Insights ingestion endpoints and required network configurations is highly recommended.

Avoid Duplicate Telemetry¶

Receiving duplicate telemetry in Azure Monitor can inflate your costs, skew your metrics, and make analyzing traces and logs confusing. Duplicate data is almost always caused by your application code or environment configuration resulting in multiple OpenTelemetry exporters or processors being initialized and running simultaneously for the same type of telemetry (traces, metrics, or logs). Each active exporter/processor will attempt to send the same data to the backend, leading to duplication.

It is crucial to ensure that your application initialization code sets up the OpenTelemetry pipeline (including providers, processors, and exporters) only once during the application’s startup phase. Re-initializing the pipeline on every request or within a loop will inevitably lead to duplicate data streams. Let’s explore common scenarios where this happens.

Duplicate Trace Logs in Azure Functions¶

Azure Functions environments sometimes have built-in logging integrations that automatically capture standard output or specific logging frameworks (like Python’s logging) and send them to Azure Monitor. When you also enable the logging instrumentation provided by the azure-monitor-opentelemetry distribution within your function code, you effectively have two separate mechanisms sending the same log messages, resulting in duplicates.

The native logging instrumentation in Azure Functions is often sufficient for capturing basic trace logs from your application’s use of Python’s logging. To prevent duplication, you should rely on the Azure Functions’ built-in logging collection and disable the specific OpenTelemetry logging exporter provided by the distribution. This can be achieved by setting the OTEL_LOGS_EXPORTER environment variable to None. This tells the Azure Monitor OpenTelemetry distribution not to configure its own log exporter, while still allowing the native Azure Functions logging to collect messages.

By setting this environment variable, you are instructing the configure_azure_monitor() function or equivalent setup logic to skip the initialization of the OpenTelemetry log exporter. This allows you to leverage the distributed tracing and metrics capabilities of the OpenTelemetry distribution without creating redundant log entries.

# Example environment variable setting for Azure Functions
OTEL_LOGS_EXPORTER=None

Ensure this environment variable is correctly configured in your Azure Function app settings in the Azure portal or through your deployment process. This is generally the cleanest way to manage logging duplication in this specific environment.

Duplicate Telemetry in “Always On” Azure Functions¶

Azure Functions has a setting called “Always On” which, when enabled, keeps the function app instances warm and running in the background even when not actively processing requests. While beneficial for reducing cold start latency, this behavior can cause issues with stateful libraries like OpenTelemetry if not managed carefully. If your function code initializes the OpenTelemetry providers (trace, metric, log) using configure_azure_monitor() every time the function handler is invoked, and “Always On” keeps the process alive, you might end up with multiple instances of exporters running within the same worker process over time.

Imagine a timer-triggered function that runs every five minutes. If “Always On” is enabled, the Python process might not shut down completely between runs. Each time the function triggers, it calls configure_azure_monitor(), which registers new exporters with the global OpenTelemetry providers. After an hour, you could have twelve sets of exporters attempting to send data, leading to severe duplication of all telemetry types (traces, metrics, logs).

To mitigate this, you have two primary options:
1. Disable the “Always On” setting if low latency isn’t critical for that specific function. This ensures the worker process is more likely to shut down between invocations, clearing the in-memory state.
2. If “Always On” is necessary, you must ensure that OpenTelemetry providers and exporters are initialized only once per worker process lifetime, not per function invocation. Alternatively, you can manually shut down the providers at the end of each function execution to clean up resources and prevent exporter accumulation.

Manually shutting down providers involves accessing the global provider instances and calling their shutdown() methods. This signals the exporters to flush any pending telemetry and clean up resources. While possible, managing this reliably in a serverless environment like Azure Functions can be tricky, as you need to be sure you get the correct global instances and handle potential errors during shutdown.

# Example of manually shutting down providers
from opentelemetry import metrics, trace, logs

# ... your function logic ...

def your_function_handler(...):
    # ... your OpenTelemetry instrumentation ...
    # Ensure providers are initialized ONLY ONCE per process,
    # OR manage shutdown carefully per invocation if necessary.

    # Example: Call shutdown at the end of processing
    # Note: This might be complex to get right in all scenarios
    # and might impact performance if called frequently.
    try:
        metrics.get_meter_provider().shutdown()
        trace.get_tracer_provider().shutdown()
        logs.get_logger_provider().shutdown()
        print("OpenTelemetry providers shut down.")
    except Exception as e:
        print(f"Error during OpenTelemetry shutdown: {e}")

The recommended approach for “Always On” functions is to structure your code so that configure_azure_monitor() or your custom OpenTelemetry setup runs only when the worker process starts, perhaps outside the main request/trigger handler function in a global scope or a dedicated initialization routine that is guaranteed to run once.

Azure Workbooks and Jupyter Notebooks¶

Interactive data analysis environments like Azure Workbooks and Jupyter Notebooks present similar challenges to Azure Functions with “Always On”. When you execute cells in a notebook, the Python kernel (the process running your code) often maintains its state between cell executions. If you call configure_azure_monitor() in one cell, the providers and exporters are initialized in the kernel’s process. If you run another cell that also calls configure_azure_monitor() without restarting the kernel or explicitly managing the OpenTelemetry state, you will register new exporters, leading to duplicate telemetry for subsequent operations.

Simply clearing the output or re-running a cell doesn’t necessarily reset the underlying Python kernel process or its state. To prevent duplicate telemetry when using OpenTelemetry instrumentation within notebooks or workbooks, ensure that you:
1. Initialize OpenTelemetry only in the first relevant cell that requires instrumentation, and avoid re-running that initialization cell multiple times without a kernel restart.
2. Restart the kernel frequently, especially before re-running setup code.
3. Manually manage provider shutdown as discussed for Azure Functions if restarting the kernel isn’t feasible or desired between operations.

Be mindful of the kernel’s lifecycle and how state is preserved. If you are running multiple experiments or analysis steps that involve telemetry, consider using separate notebook sessions or restarting the kernel between runs that include OpenTelemetry setup.

Missing Requests Telemetry from FastAPI or Flask Apps¶

If you are using a web framework like FastAPI or Flask and find that telemetry for incoming HTTP requests (which typically populate the ‘requests’ table in Azure Monitor) is missing, while other telemetry (like custom logs or metrics) is present, the issue is likely related to the order of imports and initialization. OpenTelemetry’s auto-instrumentation for libraries like FastAPI and Flask works by “patching” or “hooking” into key parts of these libraries when they are imported. This allows the instrumentation to intercept requests and create spans/traces automatically.

For this patching to be effective, the OpenTelemetry instrumentation setup (specifically the call to configure_azure_monitor() or manual configuration with relevant instrumentors) must happen before the framework’s core components are imported and initialized. If you import fastapi.FastAPI or flask.Flask and create your application instance before calling configure_azure_monitor(), the auto-instrumentation might miss the opportunity to patch the necessary parts of the framework.

Consider the following incorrect examples, where the framework class is imported before the OpenTelemetry configuration:

# Incorrect FastAPI example - Instrumentation might fail
from azure.monitor.opentelemetry import configure_azure_monitor
from fastapi import FastAPI # FastAPI imported first!

configure_azure_monitor() # Called after FastAPI is potentially loaded

app = FastAPI()
# ... route definitions ...

# Incorrect Flask example - Instrumentation might fail
from azure.monitor.opentelemetry import configure_azure_monitor
from flask import Flask # Flask imported first!

configure_azure_monitor() # Called after Flask is potentially loaded

app = Flask(__name__)
# ... route definitions ...

In these cases, the configure_azure_monitor() function attempts to apply instrumentation patches, but the target parts of FastAPI or Flask might have already been loaded and initialized without the hooks.

The correct approach is to ensure configure_azure_monitor() is called before the framework classes (fastapi.FastAPI, flask.Flask) are accessed. This gives the auto-instrumentation a chance to apply its patches during or immediately after the framework modules are imported.

Here are the corrected examples:

# Correct FastAPI example - Call configure_azure_monitor first
from azure.monitor.opentelemetry import configure_azure_monitor

configure_azure_monitor() # Call this BEFORE importing/accessing FastAPI

from fastapi import FastAPI # Now import FastAPI

app = FastAPI(__name__) # Create app instance
# ... route definitions ...

# Correct Flask example - Call configure_azure_monitor first
from azure.monitor.opentelemetry import configure_azure_monitor

configure_azure_monitor() # Call this BEFORE importing/accessing Flask

from flask import Flask # Now import Flask

app = Flask(__name__) # Create app instance
# ... route definitions ...

Alternatively, you can import the top-level module and then call configure_azure_monitor(), ensuring the instrumentation is set up before you access the framework’s contents to create the application instance:

# Correct FastAPI example - Import module then configure
from azure.monitor.opentelemetry import configure_azure_monitor
import fastapi # Import the whole module

configure_azure_monitor() # Configure OpenTelemetry

app = fastapi.FastAPI(__name__) # Access FastAPI after configuration
# ... route definitions ...

# Correct Flask example - Import module then configure
from azure.monitor.opentelemetry import configure_azure_monitor
import flask # Import the whole module

configure_azure_monitor() # Configure OpenTelemetry

app = flask.Flask(__name__) # Access Flask after configuration
# ... route definitions ...

Adhering to this import order is critical for the OpenTelemetry auto-instrumentation to successfully hook into the web framework’s request processing lifecycle and generate the necessary request telemetry. If you still face issues, double-check that you have installed the required instrumentation packages (e.g., opentelemetry-instrumentation-fastapi, opentelemetry-instrumentation-flask) and that they are compatible with your framework version.

Further Troubleshooting Steps¶

Beyond the common issues listed above, several other factors can lead to problems with OpenTelemetry in Python on Azure.

Configuration Errors¶

Incorrect configuration is a frequent source of issues. Ensure your Azure Monitor connection string or instrumentation key is correctly set via the APPLICATIONINSIGHTS_CONNECTION_STRING environment variable. Typos, incorrect values, or missing the variable entirely will prevent the exporter from sending data to the right destination. Verify the environment variable is accessible by your application process in the deployment environment.

Sampling Configuration¶

OpenTelemetry supports various sampling mechanisms to reduce the volume of telemetry data. If sampling is configured incorrectly, or if a high sampling rate is applied, you might observe missing data, especially for less frequent operations. By default, the Azure Monitor distribution might use a form of adaptive sampling or a trace ID ratio sampler. Check your sampling configuration (e.g., OTEL_TRACE_SAMPLER, OTEL_TRACE_SAMPLER_ARG) to ensure it aligns with your data retention needs. A sampler like AlwaysOnSampler can be useful during initial setup and troubleshooting to ensure all data is sent, before switching to a more cost-effective sampler for production.

Missing Instrumentation Packages¶

OpenTelemetry relies on specific “instrumentation” packages (e.g., opentelemetry-instrumentation-requests, opentelemetry-instrumentation-django) to automatically collect telemetry from popular libraries and frameworks. If you are using a library but its corresponding instrumentation package is not installed and enabled (either via configure_azure_monitor() or manual registration), you will not get automatic telemetry from that library. Verify that all necessary opentelemetry-instrumentation-* packages are included in your project’s dependencies.

Performance Issues¶

Instrumenting your application adds some overhead. While OpenTelemetry is designed to be lightweight, complex instrumentation or high-volume telemetry can impact application performance. If you notice significant latency increases or resource consumption after adding OpenTelemetry, profile your application to identify bottlenecks. This might involve adjusting sampling rates, optimizing custom instrumentation code, or ensuring the exporter is configured correctly (e.g., using batching).

Context Propagation Failures¶

In distributed systems, ensuring that trace context (like trace and span IDs) is propagated across service boundaries is crucial for end-to-end tracing. If context propagation is not working correctly (e.g., due to missing propagation headers in HTTP requests between services, or issues with message queue integrations), traces will appear broken in Azure Monitor, with individual spans showing up but not linked together in a single trace. Ensure relevant propagators are configured and that your inter-service communication mechanisms are carrying the necessary trace headers (like traceparent and tracestate).

Here is a mermaid diagram illustrating a simplified OpenTelemetry data flow:

```mermaid
graph LR
A[Application Code] → B(Instrumentation Library A);
A → C(Instrumentation Library B);
A → D(Manual Instrumentation);
B → E(API Calls);
C → E;
D → E;
E → F{OpenTelemetry API};
F → G[OpenTelemetry SDK];
G → H[Processor];
H → I[Exporter];
I → J[Azure Monitor Ingestion Endpoint];
J → K[Azure Monitor / Application Insights];
K → L[Analytics / Visualization];

subgraph Telemetry Pipeline
    F --> G --> H --> I;
end

subgraph Azure
    J --> K --> L;
end

style Telemetry Pipeline fill:#f9f,stroke:#333,stroke-width:2px
style Azure fill:#ccf,stroke:#333,stroke-width:2px

```

This diagram visualizes how telemetry generated by your application, whether through automatic instrumentation, manual code, or library hooks, flows through the OpenTelemetry API and SDK pipeline (processors, exporters) before being sent to Azure Monitor. Troubleshooting can involve checking each stage of this flow.

Summary of Common Issues and Solutions¶

Issue	Common Causes	Troubleshooting Steps
No telemetry received	Incorrect Connection String/Instrumentation Key; Network Connectivity; Exporter not initialized/started	Verify APPLICATIONINSIGHTS_CONNECTION_STRING; Test network connectivity (cURL, telnet); Check logs for exporter errors.
Duplicate Telemetry	Multiple exporters/processors initialized; “Always On” functions; Notebook state; SDK/Agent conflicts	Ensure configure_azure_monitor() runs once; Set OTEL_LOGS_EXPORTER=None; Manage state in notebooks/functions; Check for multiple agents.
Missing Requests (Web App)	Incorrect import order for web framework (FastAPI/Flask); Missing instrumentation package	Call configure_azure_monitor() BEFORE importing framework classes/modules; Ensure opentelemetry-instrumentation-<framework> is installed.
Missing specific data (e.g., DB calls)	Missing instrumentation package for the specific library; Sampling filter	Install relevant opentelemetry-instrumentation-<library>; Review sampling configuration.
Broken Traces	Context propagation failures; Missing instrumentation for inter-service communication	Verify propagators are configured; Check headers (traceparent, tracestate) in requests; Ensure instrumentation for communication libraries.
High Performance Overhead	Excessive telemetry volume; Inefficient instrumentation; Exporter batching issues	Adjust sampling rate; Optimize custom instrumentation; Review exporter batch configuration; Profile application.
Errors in Logs	Configuration issues; Network problems; SDK internal errors	Enable DEBUG logging for OpenTelemetry/Exporter; Analyze detailed log messages for specific error details.

Troubleshooting OpenTelemetry involves a combination of checking configuration, verifying code logic, examining network conditions, and analyzing detailed diagnostic logs. By systematically working through these areas, you can effectively identify and resolve most issues encountered when integrating OpenTelemetry Python applications with Azure Monitor.

We encourage you to share your experiences and challenges in the comments section below. What were the most difficult OpenTelemetry issues you faced in Python on Azure? Do you have any tips or tricks that helped you solve them? Your insights can help the community.