Use trace behaviors with datasets

Controlling the tracing data you process and persist is necessary to manage your license consumption.

After creating datasets, you can apply trace behaviors to set sampling rates for your datasets without needing to write fine-grained sampling rules. While datasets are the underpinning of behaviors, the behaviors themselves let you change the sampling rates of one or more datasets from either the Chronosphere Observability Platform app or by using Chronoctl.

Create the right datasets for your organization and then use behaviors to manage the data that those datasets generate. Use the built-in behavior types to control sampling rates and apply them across datasets. Observability Platform includes an unassigned dataset you can use for experimentation. Assign the baseline behavior to the unassigned dataset, start sending traces to Observability Platform, and as you learn about how your datasets generate data, create additional datasets to split out your services.

As you learn more about your trace data, you can edit facets of the baseline behavior to modify your sampling strategy and more clearly define which traces to drop and which to keep. In most cases, you want to assertively drop less interesting, lower-value traces and keep more interesting, higher-value traces. In the Observability Platform app, you can quickly modify the baseline sampling strategy and apply it across one or more datasets.

Trace behaviors implement head sampling rules by default, which means you need only set the percentage at which you want to keep processed data. For example, if you set the head sampling rate to 100%, you're keeping 100% of traces. If you don't set a behavior for a dataset, Observability Platform uses defined tail sampling rules. However, you can use trace datasets and behaviors without needing to write tail sampling rules.

Behaviors are part of the Trace Control Plane, which also includes datasets and head and tail sampling rules. You need administrative access to use the Trace Control Plane.

To access trace behaviors:

1. Click Go to Admin.
2. In the navigation menu select Control > Trace Control Plane.
3. Select the Behaviors tab to view and manage behaviors.

Trace behavior types

Trace behaviors can be one of the following types:

• Baseline behavior: Sample data in your datasets using data-driven best practices. Select which facets to apply to your low-value and high-value trace data, and modify the criteria as you learn more about the characteristics of your trace data.

  The baseline behavior is mostly proactive, and helps to identify what trace data to bring into Observability Platform and what data to drop.

• Allow behavior: Sample your data at 100% to allow all traces.

  The allow behavior can be both proactive and reactive. For example, you might want more high-fidelity data during a deploy to catch any issues (proactive), or allow all traces from a specific service or operation when debugging issues (reactive).

• Deny behavior: Sample your data at 0% to block all traces.

  The deny behavior can be both proactive and reactive. For example, drop all traces from a service because the data isn't useful (proactive), or stop traces from a dataset that's currently generating too much data so you don't exceed your license limit (reactive).

View behaviors

You can view and filter available trace behaviors using Observability Platform, and return the trace behavior definition using Chronoctl.

Manage assigned behaviors

You can manage the assigned behaviors for a dataset on two levels:

• Assign a main behavior to define the primary behavior for a dataset.

• Assign an override behavior to temporarily override the main behavior.

  You can assign only one main behavior and one override behavior to a dataset.

Both the main and override behaviors can use any of the trace behavior types, which are baseline, allow, and deny. You can set the override behavior to start immediately, or schedule it to start at a future time.

When managing assigned behaviors, you can set the shaping order for overlapping trace datasets. The shaping order determines the priority order to apply behaviors when traces in one dataset overlap with traces in another dataset. For example, if a trace belongs to more than one dataset with an assigned behavior, Observability Platform uses the behavior assigned to the dataset that's first in the shaping order.

The shaping order applies only when the selected behavior is active.

You can assign behaviors to a dataset using the Observability Platform app or by using Chronoctl.

Customize your sampling strategy

You can edit the facets of the baseline behavior to modify the tail sampling strategy you want to apply to your datasets. There are two main parts of the tail sampling strategy with different facets you can modify: drop less interesting, low-value traces and keep more interesting, high-value traces.

• Less interesting traces might be ones that denote success. These traces indicate that your app is working as designed, but you likely don't need to keep most of them.
• More interesting traces might be error traces, which indicate an issue that requires attention. Keeping more interesting traces helps when you're debugging issues and need to identify the source of the problem.

There are several facets you can configure: successful traces, failed traces, small traces, large traces, fast traces, and slow traces. Although each part of the baseline sampling strategy includes a set of facets by default, you can move facets from the low-value section to the high-value section, and the reverse. This flexibility lets you shape your sampling strategy as you learn more about your trace data and determine what information is valuable to your organization and what isn't. You can disable some of the facets if you don't want to use them in your baseline strategy.

For example, you don't want to keep most of your successful traces because you know they're successful, and you want to sample only a portion of your very small traces. In the low-value section of the sampling strategy, you set the sample rate for successful traces to 0.1%, and then set the sample rate for small traces to 10%.

In the high-value section of the sampling strategy, you define the criteria for the traces you want to keep. The facets for this part of the strategy might include error traces, large traces, slow traces, and fast traces. You can select the number of spans to keep, the duration for slow and fast traces, and the sampling rate for each facet.

When you assign your customized baseline behavior to a dataset, Observability Platform applies the following match criteria in order:

1. First, all traces in the dataset get matched against the behavior's low-value trace criteria. If a trace meets one or more of these criteria, the lowest possible sample rate applies.
2. Second, all remaining traces in the dataset get matched against the high-value trace criteria. If a trace meets one or more of these criteria, the highest possible sampling rate applies.

Edit the baseline behavior

To edit the baseline behavior:

1. In the Observability Platform app, click Go to Admin.

2. In the navigation menu select Control > Trace Control Plane.

3. Click the Behaviors tab, and then click the Baseline behavior.

4. On the Behavior details panel, click Edit.

5. On the Edit baseline behavior panel, define your sampling methodology:

   • Head sampling: Specify the sample rate for head sampling to capture only a portion of possible traces from originating services.

     To specify a sample rate for head sampling in the baseline behavior, you must instrument head sampling first.

   • Tail sampling: Specify the sample rate for each of the facets to drop low-value traces and keep high-value traces. Enable which facets to include in your tail sampling strategy, and specify a sampling rate for each facet.

     To move facets between the low-value and high-value sections of the tail sampling strategy, select Low or High from the Sample type dropdown for each facet you enable.

6. Click Save to save the changes to your baseline behavior.