Metric naming conventions
Metric names of ingested CloudWatch metrics in Observability Platform follow the prefix naming pattern:-
<namespace>: The namespace is lowercased, and Observability Platform replaces all forward slash (/) and period (.) characters in the CloudWatch namespace with underscores (_). All AWS service namespaces follow the naming conventionAWS/<ServiceName>, where<ServiceName>is replaced with the service name. In Observability Platform, the ingested metrics therefore begin withaws_<servicename>. For a list of AWS services and their respective namespaces, see AWS services that publish CloudWatch metrics. If you create custom metrics, the namespace you set for the metric correspondingly becomes the metric name prefix in Observability Platform. -
<MetricName>: Observability Platform preserves the CloudWatch metric name’s case. -
<statistic>: Observability Platform appends the CloudWatch statistic’s name (count,sum,maximum,minimum,average). If you define additional statistics for a metric, Observability Platform appends the corresponding CloudWatch metric statistic name (pXX).
Metric labeling conventions
Observability Platform adds CloudWatch metric dimensions as labels to the time series following the patterndimension_<DimensionName>. For examples, see
Example metric names.
Metric limitations
Most AWS services publish metrics to CloudWatch in or near real-time. However, a subset of AWS services, such as Amazon S3 daily storage metrics for buckets in CloudWatch, produce metrics based on a daily schedule with a data point timestamp older than the two-hour late-arriving data point limit. Observability Platform generally expects data points to be timestamped at the time they are sent, and enforces a two-hour age limit on ingested data points. Check the send interval for a specific AWS service metric to ensure it publishes metrics more frequently than the two-hour ingestion age limit.Stream resource attributes
CloudWatch Metric Streams include OpenTelemetry Protocol (OTLP) resource attributes, which Observability Platform merges into the time series. Observability Platform replaces periods (.) with underscores (_) in attribute key names.
Amazon Data Firehose includes the following resource attributes in every post:
aws_exporter_arn: The Amazon Resource Name (ARN) of the CloudWatch Metric Stream, which serves as the unique metric writer instance identifier.cloud_account_id: The account ID of the Amazon Data Firehose sending the stream, such as123456789.cloud_provider: The value is alwaysaws.cloud_region: The AWS region of the Amazon Data Firehose sending the stream, such asus-east-2.
Add custom resource attributes using stream parameters
You can define custom key:value pairs as parameters for Amazon Data Firehose to include in each HTTP call. Observability Platform treats all additional parameters as resource attributes and merges them into the time series.Your custom parameters take precedence over the default CloudWatch metrics resource
attributes. To avoid accidentally overwriting CloudWatch-provided values, do not
add custom parameters with key names that conflict with default CloudWatch key names.
- Terraform
- AWS Management Console
To configure parameters using Terraform:
- Edit the Terraform module.
-
In the
aws_kinesis_firehose_delivery_streamresource definition, modify therequest_configurationblock to define additionalcommon_attributes. The AWS Management Console names this setting Parameters, while the API name iscommon_attributes. For example, this configures thecontent_encodingparameter toGZIPand defines twocommon_attributes,testnameandtestname2:
Example metric names
Given a CloudWatch metric with the following attributes:- The namespace
AWS/EBS(AWS Service) - The metric name
VolumeReadBytes - The dimension
VolumeId - The custom Firehose destination parameter
environment - The CloudWatch metric resource attributes
aws_exporter_arn,cloud_account_id,cloud_provider, andcloud_region
- The namespace
Buildkite(custom metrics) - The metric name
RunningJobsCount - The custom Firehose destination parameter
environment - No dimension
- CloudWatch resource metric attributes
aws_exporter_arn,cloud_account_id,cloud_provider, andcloud_region
Drop CloudWatch Metric Stream metrics
When you ingest CloudWatch Metric Streams, you generate metrics that consume some of your Standard Metrics License capacity. To determine how this might affect license consumption, configure a drop rule before configuring CloudWatch Metric Stream ingestion.Create rules to drop CloudWatch Metrics
This example Chronoctl YAML resource definition creates a drop rule that drops all metrics from CloudWatch Metric Streams except for metrics about the Metric Stream itself.- In the AWS Management Console, go to Streams.
- Edit the Metric Stream.
- Under Metrics to be streamed, include or exclude namespaces.
View drop rule metrics
To view how many data points per second that Observability Platform is dropping with the example CloudWatch Metric Streams drop rule, use the following PromQL query:Configure CloudWatch Metric Streams
The following diagram shows the architecture and data flow from your AWS account to Chronosphere. In each of your AWS regions where you want to stream data from, a CloudWatch Metric Streams instance sends data to an AWS Data Firehose, which forwards that data to the AWS Data Firehose ingest endpoint running in your Observability Platform tenant. Observability Platform processes the CloudWatch metrics and makes them available for use in queries, monitors, and dashboards.CloudWatch roles and permissions
To use CloudWatch Metric Streams in Observability Platform, you must configure a CloudWatch Metric Stream in each AWS account and region. The account you use to set up the CloudWatch Metric Stream must either have theCloudWatchFullAccess policy
and iam:PassRole permission, or it must have the following list of permissions:
iam:PassRolecloudwatch:PutMetricStreamcloudwatch:DeleteMetricStreamcloudwatch:GetMetricStreamcloudwatch:ListMetricStreamscloudwatch:StartMetricStreamscloudwatch:StopMetricStreamsiam:CreateRoleiam:PutRolePolicy
Observability Platform authentication
You must also create or use the API token of an Observability Platform restricted service account with write-only permission. For more information, see Create a restricted service account. You must also provide your Observability Platform organization name, which is the name of the subdomain that you use to access Observability Platform. For example, if your team usesexample.chronosphere.io, your team’s organization name is example.
AWS resources and IAM roles
The AWS setup process automatically creates the following resources and IAM roles as part of creating a metrics stream:AWS resources
- S3 Bucket: A bucket will be created to store data processed by the Kinesis Firehose delivery stream.
- CloudWatch Log Group: A log group will be created to capture logs related to the Kinesis Firehose delivery stream.
- Kinesis Firehose Delivery Stream: A Kinesis Firehose delivery stream will be created with configurations to send data to Observability Platform through an HTTP endpoint, and store backup data in the S3 bucket.
- IAM Role for S3: An IAM role will be created with the following permissions
for the Kinesis Firehose to access the S3 bucket and CloudWatch Logs:
s3:AbortMultipartUploads3:GetBucketLocations3:GetObjects3:ListBuckets3:ListBucketMultipartUploadss3:PutObjectlogs:PutLogEvents
IAM roles
An IAM role to allow CloudWatch Metric Streams to publish data to the Kinesis Firehose delivery stream will be created with the following permissions:firehose:PutRecordfirehose:PutRecordBatch
Apply the configuration
You can configure CloudWatch Metric Streams either manually in the AWS Management Console or by using Terraform.Before configuring metric ingestion, you can set up a drop rule to drop all metrics
sent by CloudWatch Metric Streams. Doing this avoids unexpected license
consumption changes. For examples, see
Drop CloudWatch Metric Stream metrics.
- Terraform
- AWS Management Console
-
Set values for the following environment variables, and modify the following
Terraform data, and resources to apply the required settings.
Verify CloudWatch Metric Stream ingestion
After setup, data can take from 5 to 10 minutes to arrive to Observability Platform. To verify functionality, check the operational dashboards in the AWS Management Console for the Metric Stream and Amazon Data Firehose.Check the CloudWatch Metrics Ingestion & Health dashboard
The CloudWatch Metrics Ingestion & Health dashboard displays operational information about the health of your CloudWatch Metrics Streams integration with Observability Platform. The CloudWatch Metric Streams and Data Firehose panel groups rely on CloudWatch metrics sent from those services to Observability Platform. To populate these charts, include metrics from theAWS/Firehose and
AWS/CloudWatch/MetricStreams namespaces in your CloudWatch Metrics Streams
configuration.
- In Observability Platform, go to Dashboards.
- In the search bar, enter CloudWatch, and then click on the CloudWatch Metrics Ingestion & Health dashboard. The Observability Platform metrics ingestion panel group displays information about the CloudWatch metrics Observability Platform received.
- Check the Data Firehose records received by Amazon Resource Name chart to confirm that the ingestion API received Data Firehose records.
- Check the CloudWatch metric updates received chart to confirm the number of CloudWatch metric updates Observability Platform extracts from the Data Firehose records.
- Confirm that the Transformed metrics chart shows no Rejected Data Points
- Check the Unique time series by AWS metric namespace chart to confirm that metrics from the AWS namespaces you want are in the stream that Observability Platform received.
Check the Metric Stream Dashboard
In the AWS Management Console, check the Metric Updates chart for specific metrics to validate that metrics are streaming.- In the AWS Management Console, go to CloudWatch > Metric Streams.
- Click
chronosphere-cloudwatch-metric-streamto view the status and operational statistics. - Verify that the Status is
Running. - Verify how many updates have been sent in the Metric Updates chart. If the stream is working, the chart should report a non-zero number of updates.
- Verify whether any errors were reported in the Errors chart. The value should be 0.
Check the Amazon Data Firehose status
In the AWS Management Console, check the status of several charts to ensure that metrics are streaming.- In the AWS Management Console, go to Amazon Data Firehose > Firehose Streams.
- Select
PUT-CW-STREAM-CHRONOSPHEREto view status and operational statistics for the Amazon Data Firehose. - The Incoming bytes, Incoming put requests, and Incoming records charts should all report non-zero values.
- The HTTP endpoint delivery success chart should report a 100% successful metric count.
- The Records delivered to HTTP endpoint chart should report a non-zero value.
Query metrics about AWS CloudWatch Metric Streams
You can also query for metrics about AWS CloudWatch Metric Streams in Observability Platform. These metrics won’t appear in the Observability Platform Metrics Explorer if you’ve defined a drop rule to drop all AWS metrics. Modify the rule to allow some metrics, such as all metrics from theAWS/CloudWatch/MetricStreams namespace. For
examples, see Drop CloudWatch Metric Stream metrics.
- In Observability Platform, go to Explore > Metrics Explorer.
- In the query box, enter
aws_cloudwatch_metricstreams`to view a list of AWS metrics received from the CloudWatch Metric Stream. - Select the metric you want to query to add it to the query prompt.