The OpenTelemetry Collector is frequently treated as passive plumbing in observability stacks, sitting silently between applications and backends. This assumption creates a dangerous blind spot: when the Collector itself fails, applications may appear healthy while telemetry data silently vanishes. Exporters can stall, queues can fill, and memory leaks can creep upward until the process restarts, all without triggering application-level alerts. As Vivek Anandaraman notes, the system responsible for monitoring everything else must itself be monitored to ensure the integrity of the data being delivered.
Enabling internal telemetry shifts the mindset from merely observing the source to validating the delivery pipeline. The Collector exposes metrics via a Prometheus endpoint, configured under service.telemetry.metrics. Setting the level to normal or detailed reveals critical operational data, including resource usage, receiver acceptance rates, processor throughput, and exporter success. A basic configuration binds this endpoint to port 8888, though production environments should carefully restrict network exposure, potentially binding to localhost if the monitoring system runs locally, to avoid unnecessary attack surfaces.
The technical value lies in distinguishing between intentional data reduction and accidental loss. Metrics like otelcol_receiver_refused_spans or otelcol_processor_dropped_spans indicate ingestion or processing failures, while otelcol_exporter_send_failed_spans and growing otelcol_exporter_queue_size signal backend connectivity issues or bottlenecks. For instance, a steady climb in otelcol_process_memory_rss serves as an early warning for potential OOM kills, whereas a sudden drop in accepted spans might indicate a receiver misconfiguration rather than a lack of traffic. These signals allow engineers to differentiate between a quiet system and a broken one.
Engineers should establish baselines before configuring alerts, focusing on sustained deviations rather than transient spikes. The next step is integrating these metrics into existing dashboards, ensuring that the health of the observability infrastructure is visible alongside application performance. If the pipeline is not trustworthy, the incident data it provides is equally suspect, making internal telemetry a prerequisite for reliable operations. Good read!
[Read More]