> ## Documentation Index
> Fetch the complete documentation index at: https://docs.chronosphere.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Trace Metrics
export const TUsageAnalyzer = () => <>
Telemetry Usage Analyzer
;
Use Trace Explorer to identify the root cause of an issue. As you define your query
and determine what caused the problem you're investigating, you can create a
*trace metric* from your query to display that metric in a dashboard for faster
discovery. You can then configure monitors and alerts on that metric to notify other
on-call engineers if that metric identifies anomalies in your trace data.
This ability to navigate from a trace metric to a specific query in Trace Explorer
helps other on-call engineers address problems faster by using predefined queries to
explore issues.
Use the following metric types to help track information for collected traces:
* **Counter metric**: Increments any time a trace that matches a filter gets
collected. This metric also includes a label describing whether or not the trace
contains an error.
* **Histogram metric**: Collects information about the distribution of request
durations for traces that match a filter.
Chronosphere Observability Platform provides a default **Trace Metrics** dashboard
with panels that display the requests, errors, and durations of requests associated
with a created trace metric. The dashboard also includes a service map of the
requests.
Observability Platform generates any trace metrics you've created before applying
behaviors or [tail sampling rules](/control/shaping/sample-traces/tail-sampling), which determine
whether to persist the trace.
To explore tracing data in Observability Platform, you must either install and configure the
[Chronosphere Collector](/ingest/metrics-traces/collector/install/tracing) or the
[OpenTelemetry Collector](/ingest/metrics-traces/otel/otel-ingest) to receive trace data from your
services.
## View existing trace metrics
You can view trace metrics in Observability Platform and open the related query in
Trace Explorer.
To return a list of defined trace metrics without additional information, use
Chronoctl.
Viewing a trace metric opens the related query in Trace Explorer. Observability
Platform takes the contextual information in metrics from a dashboard and uses it to
build links to search for traces. Clicking the link opens Trace Explorer and replaces
the variables with matching criteria defined in the link.
To view your existing trace metrics:
1. In the navigation menu, click ** Go to Admin**
and then select
** Control > Trace Metrics**.
2. In the **Rule Name** column, click the name of a trace metric to view the dashboard
for that metric.
3. From the dashboard, click anywhere in one of the graph panels to pin that
selection. In the pinned popup, click the and then click the
three vertical dots icon and choose one of these
options:
* **Analyze anomaly (DDx)** opens
[Metrics Differential Diagnosis (DDx)](/investigate/analyze/differential-diagnosis/metrics)
and identifies labels and values that strongly correlate to the trace metric.
* **View similar logs** opens Logs Explorer with results filtered to the selected
trace metric.
* **View similar traces** opens Trace Explorer with results filtered to the
selected trace metric.
See [common panel elements](/observe/dashboards/panels#common-panel-elements) for an
explanation of the other available tools.
To view a list of all available trace metrics using [Chronoctl](/tooling/chronoctl),
use the `chronoctl trace-metrics-rules list` command:
```shell theme={null}
chronoctl trace-metrics-rules list
```
A complete list of the available trace metrics displays in your terminal.
To view a single trace metrics rule, use the `chronoctl trace-metrics-rules read`
command, followed by the slug of the rule you want to view. For example, the
following command returns a trace metrics rule with the slug `telemetry-sdk`:
```shell theme={null}
chronoctl trace-metrics-rules read telemetry-sdk
```
To complete this action with the Chronosphere API, use the
[`ListTraceMetricsRules`](/tooling/api-info/definition/operations/ListTraceMetricsRules)
endpoint.
To view a single trace metrics rule instead of a complete list of rules, use the
[`ReadTraceMetricsRule`](/tooling/api-info/definition/operations/ReadTraceMetricsRule)
endpoint.
Because the Chronosphere API requires authentication, include an API token with your
`curl` request, as shown in the following example. For more details, see
[Create an API token](/tooling/api-info#create-an-api-token).
```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null}
export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"
curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
-X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"
```
Replace the following:
* *`TOKEN`*: Your API token.
* *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance.
* *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`.
* *`ENDPOINT_PATH`*: The specific endpoint you want to access.
## Create a trace metric
Observability Platform uses trace queries as the basis for trace metrics. You define
a search query in Trace Explorer, and create a trace metric based on the matching
criteria.
You can search for any labels you add to your trace metric in
[Metrics Explorer](/investigate/querying/metrics/explorer) when investigating query
requests and responses.
Creating a trace metric is the same as creating a new metric in your system. Any trace
metrics you create impact your metrics license consumption, which display in the
[License Overview](/observe/dashboards/managed-dashboards#licensing-dashboards).
Because of that potential impact, avoid including high cardinality dimensions in your
trace metric. As a protective measure, trace metrics capture data for only the first
1,000 unique series observed.
Selecting a dimension to group by with high cardinality can overflow the 1,000 value
trace metric cardinality limit, and cause incorrect data to display in your trace
metric dashboard.
To create a trace metric, you must have administrative privileges.
To create a new trace metric:
1. In the navigation menu select
** Explorers > Trace Explorer**.
2. Apply one or more filters to
[define a search](/investigate/querying/traces#define-a-search). Trace
Explorer applies the filters represented in the query before updating metrics for
the created trace metric.
You can group your trace query search results by up to
[three dimensions](/investigate/querying/traces#group-and-narrow-results)
that can include service, operation, and any other span tag. Dimensions to group
by are inherited from the defined search query.
Trace metrics don't support regular expression operators, such as the match (`=~`)
or doesn't match (`!~`) operators.
3. Click the three vertical dots icon, and then select
**Create Trace Metric**.
4. In the **Create trace metric** dialog, enter a display name (the equivalent of
Prometheus metric `__name__`) and system name (the trace metric **Rule Name**) for
your trace metric.
5. In the **Group by labels** section, enter a metric label to display the aggregated
results of your query grouped by that attribute.
For each dimension in the **Group by labels** field, you must define a metric
label key. Use the same key as the span tag, or a key that aligns with your
existing metrics data. Observability Platform adds metric label keys to your trace
metric, and associates the label value with the span tag.
For example, if you group your query by the span tag **Service** and enter
**container\_service** as the label key, Observability Platform adds a label to
your trace metric where the label values equal the span tag **Service**
values. You can then query by that label anywhere you search for metric labels,
such as in Metrics Explorer or .
6. Optional: Expand the **Static metric labels** section and enter key/value pairs to
add static labels to your trace metric. Static labels are like metadata you add to
your trace metric.
7. Click **Create**.
After creating the trace metric, the **Trace Metric Created** dialog displays.
It can take several minutes for the dashboard to display data about the trace
metric.
8. Click **Go to Trace Metrics** to view the list of available trace metrics. When
your trace metric is available, you can select it from this list to view a
dashboard for the trace metric that includes requests, errors, durations, and a
topology map.
9. When viewing your trace metric, click a data point in any of the provided graphs
and then click **Query Traces** to open Trace Explorer with the query you defined
for your trace metric.
When creating your trace metric with Chronoctl, consider the following behaviors of
the `slug` property:
* If you don't provide a `slug`, Chronoctl generates one based on the `name` field.
* You can modify the `name` after creating your trace metric, but can't
modify the `slug`.
* The `metric_name` and `slug` must match.
To use [Chronoctl](/tooling/chronoctl) to create a trace metric, use the
`chronoctl trace-metrics-rules` command:
```shell theme={null}
chronoctl trace-metrics-rules scaffold
```
1. Run the following command to generate a sample trace metric configuration you can
use as a template:
```shell theme={null}
chronoctl trace-metrics-rules scaffold
```
In the template, `kind: TraceMetricsRule` defines an individual trace metric.
You can redirect the results (using the redirection operator `>`) to a file for
editing.
2. With a completed definition, submit it with:
```shell /FILE_NAME/ theme={null}
chronoctl trace-metrics-rules create -f FILE_NAME
```
Replace *`FILE_NAME`* with the name of the YAML definition file you want to use.
See the [Chronoctl example](#chronoctl-trace-metric-example) for a completed trace
metrics resource.
When you run `terraform plan` to generate an execution plan, Chronosphere automatically
tests configurations that include notification policies by submitting them as dry runs.
For details, see the
[Terraform provider](/tooling/infrastructure/terraform#validate-plans-with-dry-runs)
documentation.
To create a new trace metric with [Terraform](/tooling/infrastructure/terraform):
1. Create or edit a Terraform file and add the definition by using the
`chronosphere_trace_metrics_rule` type, followed by a name in a resource
declaration.
Refer to [Trace metric rule fields](#trace-metric-rule-fields) for a list of all
supported fields.
2. Run this command to apply the changes:
```shell theme={null}
terraform apply
```
See the [Terraform example](#terraform-trace-metric-example) for a completed trace
metric resource.
To complete this action with the Chronosphere API, use the
[`CreateTraceMetricsRule`](/tooling/api-info/definition/operations/CreateTraceMetricsRule)
endpoint.
Because the Chronosphere API requires authentication, include an API token with your
`curl` request, as shown in the following example. For more details, see
[Create an API token](/tooling/api-info#create-an-api-token).
```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null}
export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"
curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
-X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"
```
Replace the following:
* *`TOKEN`*: Your API token.
* *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance.
* *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`.
* *`ENDPOINT_PATH`*: The specific endpoint you want to access.
### Chronoctl trace metric example
The following resource defines a trace metric that matches spans that includes both
the `ordering-svc` and the `POST/submit-order` operation, a tag of `env=production`,
and a duration between two and three seconds. The metric also groups by the
`ordering-svc` key.
Use the `scope_filter` field to add a secondary layer of filtering after the initial
trace metrics filter. This additional filtering is comparable to using the
[span statistics](/investigate/querying/traces/features#span-statistics) feature in
Trace Explorer to return information about specific spans within the specified trace
filter.
```yaml theme={null}
api_version: v1/config
kind: TraceMetricsRule
spec:
name: Ordering service uswest1 error traces
slug: ordering-service-uswest1-error-traces
metric_name: order-svc-uswest1-errors
group_by:
label: svc
key:
named_key: ordering-svc
type: SERVICE
trace_filter:
span:
tags:
key: env
value:
value: production
match: EXACT
operation:
value: POST/submit-order
match: EXACT
service:
value: ordering-svc
match: EXACT
trace:
duration:
min_secs: 2
max_secs: 3
error:
value: true
scope_filter:
span_scopes:
- service:
value: ordering-svc
match: EXACT
match_type: INCLUDE
- operation:
value: /ordering.Ordering/Checkout
match: EXACT
- duration:
min_secs: 2s
```
### Terraform trace metric example
The following Terraform resource definition defines a trace metric that matches spans
that include both the `ordering-svc` and the `POST/submit-order` operation, a tag of
`env=production`, and a duration between two and three seconds. The metric also
groups by the `ordering-svc` key.
Use the `scope_filter` field to add a secondary layer of filtering after the initial
trace metrics filter. This additional filtering is comparable to using the
[span statistics](/investigate/querying/traces/features#span-statistics) feature in
Trace Explorer to return information about specific spans within the specified trace
filter.
```terraform theme={null}
resource "chronosphere_trace_metrics_rule" "ordering_service_metric" {
{
trace_metrics_rule: {
name = "Ordering service uswest1 error traces"
metric_name = "order-svc-uswest1-errors"
group_by {
label = "svc"
key {
type = "SERVICE"
named_key = "ordering-svc"
}
}
trace_filter {
span {
tags {
key = "env"
value {
value = "production"
match = "EXACT"
}
}
operation {
value = "POST/submit-order"
match = "EXACT"
}
service {
value = "ordering-svc"
match = "EXACT"
}
}
trace {
duration {
min_secs = 2
max_secs = 3
}
error {
value = true
}
}
scope_filter {
span_scopes {
service {
value = "ordering-svc"
match = "EXACT"
}
match_type = "INCLUDE"
}
}
}
}
}
```
## Update a trace metric
Select from the following methods to update trace metrics.
To update a trace metric, you must have administrative privileges.
To edit or update a trace metric:
1. In the navigation menu, click ** Go to Admin**
and then select
** Control > Trace Metrics**.
2. To the right of the date of the trace metric, click the
three vertical dots icon, and then click **Edit**.
3. In the **Edit trace metric** dialog, edit the query that generates the trace
metric or make any other needed changes.
4. Click **Save** to save your changes.
To update a trace metric using [Chronoctl](/tooling/chronoctl):
1. Return a list of all available trace metrics:
```shell theme={null}
chronoctl trace-metrics-rules list
```
2. Modify the YAML file for the trace metrics you want to update.
3. Apply your changes:
```shell theme={null}
chronoctl apply -f FILE_NAME.yaml
```
Replace *`FILE_NAME`* with the name of your trace metric YAML file.
Chronoctl updates the trace metric's properties if it has the same slug.
To edit a trace metric using [Terraform](/tooling/infrastructure/terraform):
1. Modify the resource's existing properties.
2. Run `terraform apply` to apply the changes.
```shell theme={null}
terraform apply
```
To complete this action with the Chronosphere API, use the
[`UpdateTraceMetricsRule`](/tooling/api-info/definition/operations/UpdateTraceMetricsRule)
endpoint.
Because the Chronosphere API requires authentication, include an API token with your
`curl` request, as shown in the following example. For more details, see
[Create an API token](/tooling/api-info#create-an-api-token).
```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null}
export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"
curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
-X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"
```
Replace the following:
* *`TOKEN`*: Your API token.
* *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance.
* *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`.
* *`ENDPOINT_PATH`*: The specific endpoint you want to access.
## Delete a trace metric
Select from the following methods to delete trace metrics.
To delete a trace metric, you must have administrative privileges.
To delete a trace metric:
1. In the navigation menu, click ** Go to Admin**
and then select
** Control > Trace Metrics**.
2. To the right of the date of the trace metric, click the
three vertical dots icon, and then click **Delete**.
To delete a trace metric using [Chronoctl](/tooling/chronoctl):
1. Return a list of all available notification policies:
```shell theme={null}
chronoctl trace-metrics-rules list
```
2. Run the following command to delete the trace metric:
```shell theme={null}
chronoctl trace-metrics-rules delete SLUG
```
Replace *`SLUG`* with the slug of the trace metric you want to delete.
Chronoctl deletes the trace metric with the specified slug.
To delete a resource that's managed by [Terraform](/tooling/infrastructure/terraform):
1. Edit your Terraform configuration file to remove the pre-existing resource
definition.
2. Run this command to remove the resource from Observability Platform:
```shell theme={null}
terraform apply
```
To complete this action with the Chronosphere API, use the
[`DeleteTraceMetricsRule`](/tooling/api-info/definition/operations/DeleteTraceMetricsRule)
endpoint.
Because the Chronosphere API requires authentication, include an API token with your
`curl` request, as shown in the following example. For more details, see
[Create an API token](/tooling/api-info#create-an-api-token).
```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null}
export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"
curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
-X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"
```
Replace the following:
* *`TOKEN`*: Your API token.
* *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance.
* *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`.
* *`ENDPOINT_PATH`*: The specific endpoint you want to access.
Observability Platform removes the metric from the **Trace Metrics** page. Deleting a
trace metric also removes access to the metric in the Trace Metrics dashboard.
## Trace metric rule fields
A trace metric rule consists of the following fields, which are properties of the
`trace_metrics_rule` object. Each field is required unless otherwise noted. See
the [CreateTraceMetricsRule](/tooling/api-info/definition/operations/CreateTraceMetricsRule)
API for a complete list of supported fields.
* `group_by`: Labels for grouping and narrowing search results to specific
attributes. See [Group and narrow results](/investigate/querying/traces#group-and-narrow-results)
for more information about grouping related attributes.
* `key`: The key to group by.
* `named_key`: The name of the key to group by.
* `type`: The type of key to group by.
* `label`: The dimension for displaying the aggregated results of your query
grouped by that attribute in resulting trace metrics. Defaults to the selected
service.
* `histogram_buckets_seconds`: Optional: An array of custom buckets measured in
seconds for duration histogram metrics. You can set these if the default buckets
aren't appropriate for your data. The default buckets are `[0.005, 0.01, 0.025,
0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]`.
* `metric_labels`: Optional: Static labels applied to the metrics, defined as
an object of one or more label name/value pairs.
* `metric_name`: The metric's name as stored in the metrics database. Use this name
to refer to the metric in PromQL queries, such as in dashboards, monitors, and the
Metrics Explorer. The `metric_name` must follow
[Prometheus metric naming rules](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels).
Although you can modify the `metric_name` after creation, any data points generated
*before* the name change aren't renamed. The new metric name is instead used to
generate new data points starting from that point in time. The old metric name
doesn't add new data, but you can still query it for past data points.
* `name`: The name of the trace metric rule. You can modify this name after creating
the trace metric rule.
* `slug`: Optional: The slug for the trace metric rule. After you create the rule,
the slug is immutable. If omitted, Observability Platform generates a `slug` at
creation time that's based on the `name`.
* `trace_filter`: A filter object that evaluates traces against several criteria and
emits metrics for only those that match. The filter's capabilities are similar to
the filter in the [Trace Explorer](/investigate/querying/traces), except it
supports only exact matches on string fields instead of regular expressions. This
object supports additional
[optional object fields](#optional-trace-filter-object-fields).
### Optional trace filter object fields
The required `trace_filter` object supports the following optional fields, and you
can use any combination of them:
Unlike most Terraform resources, several child fields of `trace` and `span` are
objects with a single field named `value`.
* `trace`: Applies trace-level filtering. You can specify only one trace filter.
* `duration`: Matches traces with a specified duration. You can specify only one
of `min_secs` and `max_secs`.
* `min_secs`: Matches traces with a duration greater than or equal to this
value.
* `max_secs`: Matches traces with a duration less than or equal to this value.
* `error`: Object with a single field named `value`. If specified, matches traces
with an error flag equal to this Boolean value. Refer to
[assign values with `value`](#assign-values-with-value) for more information.
* `span`: Applies span-level filtering. You can specify multiple span filters, but
Observability Platform evaluates a trace filter match only if *all* `span` filters match.
* `service`: Object with a single field named `value`. Matches spans with a service
field equal to this value. Refer to
[assign values with `value`](#assign-values-with-value) for more information.
* `operation`: Object with a single field named `value`. Matches spans with an
operation field equal to this value. Refer to
[assign values with `value`](#assign-values-with-value) for more information.
* `parent_service`: Object with a single field named `value`. Matches spans whose
parent span's service field is equal to this value. Refer to
[assign values with `value`](#assign-values-with-value) for more information.
* `parent_operation`: Object with a single field named `value`. Matches spans whose
parent span's operation field is equal to this value. Refer to
[assign values with `value`](#assign-values-with-value) for more information.
* `duration`: Matches spans with a specified duration. You can specify only one of
`min_secs` and `max_secs`.
* `min_secs`: Matches spans with a duration greater than or equal to this
value.
* `max_secs`: Matches spans with a duration less than or equal to this value.
* `error`: Object with a single field named `value`. If specified, matches traces
with an error flag equal to this Boolean value. Refer to
[assign values with `value`](#assign-values-with-value) for more information.
* `tags`: Matches based on a span's tag keys and values. You must specify the `key`, and
you can specify multiple tag filters, but Observability Platform evaluates a
span filter match only if *all* `tag` filters match.
* `key`: Matches spans whose tags match the specified key.
* `value`: Object with a single field named `value`. Matches spans when the tag
with a matching key also matches the specified value. If omitted, Observability
Platform evaluates *all* spans matching the key as matches. Refer to
[assign values with `value`](#assign-values-with-value) for more information.
* `span_count`: Specifies how many spans can match. You can set only one of `min`
and `max`. By default, Observability Platform evaluates a `span` filter as a
match only if at least one span matches.
* `min`: If set, *at least* this number of spans must match the parent span
filter's conditions for Observability Platform to evaluate the entire trace
filter as a match.
* `max`: If set, Observability Platform evaluates a trace filter match only if
the number of spans matching the parent span filter's conditions is equal to
or less than the `max` value.
* `match_type`: Specifies the span filter's match type. Valid values are
`"include"` and `"exclude"`. The `"include"` match type is the default, and
evaluates a trace filter match if *all* its fields match *any* span, or multiple
spans if you specify a `span_count`. An `"exclude"` match type evaluates a trace
filter match if *no* spans within that trace match *all* of the span filter's
conditions.
### Assign values with `value`
If a child field of a `trace` or `span` are objects with a single field named
`value`, assign the value to the required child `value` field of these
objects, instead of directly assigning the value like a field.
For example, to set the `error` field's value in a `trace` filter, use:
```terraform theme={null}
trace {
error {
value = true
}
}
```
Also, the value of a `tag` filter, unlike the key, is also an object with a `value`
field:
```terraform theme={null}
span {
tag {
key = "region"
value = {
value = "us-east"
}
}
}
```
Although these objects have only the one `value` field, this structure lets
Chronosphere plan future features for these criteria. For more examples of this
structure, see the [filter examples](#filter-examples).
### Filter examples
Matches traces marked as `error` that also took more than five seconds:
```terraform theme={null}
trace_filter {
trace {
duration {
min_secs = 5
}
error {
value = true
}
}
}
```
Matches traces with at least one span from the service named `"cupcake-factory"`:
```terraform theme={null}
trace_filter {
span {
service {
value = "cupcake-factory"
}
}
}
```
Matches traces with at least one span tagged `region:us-east` *and* at least one
span tagged `region:us-west`:
```terraform theme={null}
trace_filter {
span {
tag {
key = "region"
value = {
value = "us-east"
}
}
}
span {
tag {
key = "region"
value = {
value = "us-west"
}
}
}
}
```
Matches traces with at least one span tagged *both* `region:us-east` *and*
`stack:production`:
```terraform theme={null}
trace_filter {
span {
tag {
key = "region"
value = {
value = "us-east"
}
}
tag {
key = "stack"
value = {
value = "production"
}
}
}
```
Matches traces with *at least* 10 spans containing the `"db-query"` operation:
```terraform theme={null}
trace_filter {
span {
operation {
value = "db-query"
}
span_count {
min = 10
}
}
}
```