Analyze live traffic metrics

​

Capture and analyze live profiling data

1. In the navigation menu, click Go to Admin and then select Analyzers > Live Telemetry.
2. Click the Metrics tab. To profile traces instead, see Analyze live traffic traces.
3. Click Capture live data to begin gathering statistics for data that Observability Platform accepted for matching. Click Capture live data again to pause the capture. Live Telemetry Analyzer automatically pauses the capture after three minutes. The Labels table contains these columns:
   • Label Keys: Label keys in the captured metrics.
   • Unique Values: Number of unique values for the respective label key.
   • Appears In: The percentage of metrics you’re viewing that have the matching label key.
   The Label values table contains columns for the selected grouping labels and these columns:
   • Avg. DPPS: Average data points per second (DPPS), calculated over the previous 15 seconds.
   • Current DPPS: Current data points per second.
   You can make changes to the groupings and filters while profiling.
4. To modify the displayed data, select an option from the Data phase menu to show data in different stages along the pathway from ingestion to persistence. For example, select Rejected by drop rule to view all data that Observability Platform dropped because of a configured drop rule.
5. To display data for a specific pool only, select a pool from the Pool menu to choose a specific metric pool. Use the Priority menu to narrow the filter to a specific pool priority.

​

Analyze metrics

• Review the metric names that generate the most data points per second (Avg. DPPS or Current DPPS). If those metrics are unfamiliar to you or are expensive, these might be candidates to roll up or drop.
• Ensure your drop and rollup rules are working as expected by reviewing your rolled up metrics, or ensuring that a dropped metric no longer displays.
• Group metrics by job to identify the specific scrape jobs generating the most metrics. Filter for each job, and analyze the job’s individual metrics to find opportunities for reduction. Metrics from the same job are often used together, letting you investigate metrics from a single job more quickly.
• Review metrics isolated to single environments. For example, metrics available only in development or production environment metrics. These are likely to have different metric workload shapes from each other.

​

Ingestion stages and phases

• Received: Not selectable.
• Rejected by drop rule: Metrics dropped due to drop rules. This option is relevant only for the Ingestion phase.
• Rejected by ingest limit: Metrics that dropped due to exceeding the ingestion rate limit.
• Accepted for matching: Metrics that aren’t dropped prior to ingestion.

• Rejected by persist limit: Metrics not sent to permanent storage due to persistence limits.
• Accepted for storage: Metrics sent to storage.
• Stored: Not selectable.

​

Special request metadata

• __metric_type__ displays on the incoming metric’s Chronosphere metric type. Valid values are cumulative_counter, cumulative_exponential_histogram, delta_counter, delta_exponential_histogram, gauge, or measurement. This is the recommended method for determining an incoming metric’s type.
• __metric_source__ displays on the incoming metric’s source format. Valid values are carbon, chrono_azure, chrono_cloudflare, chrono_gcp, chrono_integration, cloudwatch_metric_stream, dogstatsd, open_metrics, open_telemetry, prometheus, signalfx, statsd, tagged_statsd, or wavefront.

• __m3_prom_type__ displays the incoming metric’s Prometheus metric type. Valid values are counter, gauge, histogram, gauge_histogram, summary, info, state_set, or quantile.

• __otel_type__ displays the incoming metric’s OpenTelemetry metric type. Valid values are sum, monotonic_sum, gauge, histogram, exp_histogram, or summary.
• __otel_temporality__ displays the incoming metric’s OpenTelemetry temporality. Valid values are delta or cumulative.
• DEPRECATED: __m3_type__ displays on the incoming metric’s legacy M3 type, if any. Valid values are counter, gauge, or timer.

​

Inspect matched data for aggregation

1. In the navigation menu, click Go to Admin and then select Analyzers > Live Telemetry.
2. Select the Metrics tab, and click Capture live data.
3. In the Data phase menu, under Aggregation, select Matched for aggregation.
4. Click the Samples tab. The tab label includes a count of captured samples.
5. In the Filter then press Enter field, enter key=value pairs, key!=value pairs for exclusions, or a bare token to match any field. Press Enter to apply the filter.
6. Scan the Timestamp, Value, and Metric table columns, plus any dynamic metadata columns populated for your traffic. Click a row to open a detailed view with labels and metadata in a separate pane.
7. Click Enable or disable sample streaming to pause updates and hold the current buffer, or resume streaming when you want updated samples.
8. To discard captured samples and reset the list, click Drain current samples.

​

Group and filter metrics

sign_up{location="placeA"}
sign_up{location="placeB"}
login{version="v0.1.0"}

​

Identify rules that generate metrics

1. In the Data phase menu, select Rejected by drop rule.
2. Under the Labels section, select the __drop_rule_slug__ label. The drop rule slug names display in the Label values table.
3. Click the arrow icon to navigate directly to the drop rule that caused the metrics to be dropped.

1. In the Data phase menu, select Accepted for storage.
2. Under the Labels section, select the __rollup_rule_slug__ label. The aggregation rule slug names display in the Label values table.
3. Click the arrow icon to navigate directly to the aggregation rule that produced the aggregated metric.

​

Troubleshoot missing metrics

• Examine the filters to ensure they’re not dropping the metrics you’re searching for.
• Review the Collectors dashboard and ensure metrics are being scraped by the Collectors.
• See metric limits for more information.