Ingest Google Cloud metrics

Google Cloud (opens in a new tab) integration connects Chronosphere with Google Cloud to ingest metrics from Google Cloud projects.

Chronosphere uses Google's service account impersonation (opens in a new tab) as the authentication mechanism to connect Google Cloud with Chronosphere. Service account impersonation is trust-based, and uses short-lived credentials so that one Google Cloud service account principal can impersonate another.

Set up Google Cloud integration

1. Request infrastructure from Chronosphere.

   Chronosphere support must create infrastructure for your Google Cloud integration for use with your tenant. Before proceeding, contact Chronosphere support.

2. Modify the Google Cloud domain restriction constraint.

   If your Google Cloud organization restricts identities by domain (opens in a new tab), you must add the Chronosphere customer identity (C04aozwp9) as an allowed value in your policy.

3. Create a Google Cloud service account.

   A Google Cloud service account isn't a Chronosphere service account. Chronosphere requires a Google Cloud service account in the user's Google Cloud project to provide the Chronosphere application access to metrics using service account impersonation.

   For each Google Cloud project, create a service account (opens in a new tab) with the monitoring.viewer role.

   To instead grant a more restricted set of permissions, implementing the principle of least privilege, create a custom role with permissions for monitoring.timeSeries.list and monitoring.metricDescriptors.list.

4. Add the Chronosphere principal to the Google Cloud service account.

   Each Google Cloud service account must grant access to the Chronosphere principal to impersonate them. Grant the principal the iam.serviceAccountTokenCreator role.

5. Configure the integration in Chronosphere.

   Initialize the integration using the Google Cloud integration API by applying the configuration with Chronoctl or Terraform.

   For each Google Cloud project, supply the Google Cloud service account email (SERVICE-ACCOUNT@PROJECT-ID.iam.gserviceaccount.com) in the request to configure the integration, along with specific metrics to ingest using metric prefixes. See Metric Prefix Search for examples.

Example configuration

The following code provides an example for creating a single Google Cloud service account in the user's Google Cloud project, and enables Chronosphere to impersonate and gain access.

locals {
  // Email address of Chronosphere tenant-specific principal.
  chronosphere_sa_email = "SERVICE-ACCOUNT@PROJECT-ID.iam.gserviceaccount.com"
 
  // Google Cloud project containing monitoring data to be ingested into Chronosphere.
  monitoring_project_id = "PROJECT_ID"
}
 
// Service account that will allow Chronosphere tenant-specific principal
// to impersonate it.
resource "google_service_account" "chronosphere" {
  project    = local.monitoring_project_id
  account_id = "chronosphere"
}
 
// Custom role with minimal set of permissions for principle of least privilege
// resource "google_project_iam_custom_role" "chronosphere" {
  project     = local.monitoring_project_id
  role_id     = "Google Cloud Integration Metrics Viewer"
  title       = "Google Cloud Integration Metrics Viewer"
  description = "Role for granting view access for Google Cloud integration"
  permissions = [
    "monitoring.metricDescriptors.list",
    "monitoring.timeSeries.list",
  ]
}
 
// The service account has the minimal set of permissions for access to the project
// containing the metric data so that it can retrieve time series data from the
// Google Cloud API.
resource "google_project_iam_member" "chronosphere" {
  project = local.monitoring_project_id
  role    = google_project_iam_custom_role.chronosphere.id
  member  = google_service_account.chronosphere.member
}
 
// The service account provides the Chronosphere tenant-specific principal with
// roles/iam.serviceAccountTokenCreator access so that it can impersonate it. Only
// the Chronosphere tenant-specific principal will be able to perform this
// impersonation.
data "google_iam_policy" "chronosphere" {
  binding {
    role    = "roles/iam.serviceAccountTokenCreator"
    members = ["serviceAccount:${chronosphere_sa_email}"]
  }
}
 
// Assign the token creator permission to the service account.
resource "google_service_account_iam_policy" "chronosphere" {
  service_account_id = google_service_account.chronosphere.name
  policy_data        = data.google_iam_policy.chronosphere.policy_data
}
 

Use metrics scopes for projects

Chronosphere encourages users to use metrics scoping for projects (opens in a new tab) in their Google Cloud environment, particularly for users that manage five or more Google Cloud projects. Metrics scopes have benefits over configuring individual Google Cloud projects for Google Cloud integration:

• Google Cloud API cost savings: With metrics scoping, Chronosphere ingests metrics using the least number of API requests to Google Cloud. Metrics scoped projects introduce a fan in/out model of ingesting metrics from the underlying Google Cloud projects that will pack time series data points for each metric into the smallest set of API responses. The Google Cloud API cost savings are significant, particularly for users managing multiple Google Cloud projects.

• Configuration simplicity: With metrics scoping, users can manage a smaller set of Google Cloud service accounts and configuration in Chronosphere. This simplifies management of the Google Cloud integration.

Set up Chronosphere to receive Google Cloud data

After configuring Google Cloud to enable Chronosphere to access metrics, you must configure Chronosphere to receive and process those metrics.

View Google Cloud integrations

To list Google Cloud integrations with Chronoctl, use the command:

chronoctl gcp-metrics-integrations list

To view a Google Cloud integration with Chronoctl, use the command:

chronoctl gcp-metrics-integrations read SLUG

SLUG is the unique identifier of the Google Cloud integration.

Create or update a Google Cloud integration

You can create or update your Google Cloud Platform integration with Chronosphere by applying a configuration file with Chronoctl or Terraform.

chronoctl gcp-metrics-integrations create --filename FILENAME

chronoctl gcp-metrics-integrations update --filename FILENAME

api_version: v1/config
kind: GcpMetricsIntegration
spec:
  name: NAME
  slug: SLUG
  service_account:
    client_email: EMAIL
  metric_groups:
    - project_id: PROJECT_ID
      prefixes:
        - PREFIX

Replace the following:

• NAME: (string) The name of the Google Cloud integration.
• SLUG: (string) The unique identifier of the Google Cloud integration.
• EMAIL: (string) The email address of the customer's Google Cloud service account of the form SERVICE_ACCOUNT@PROJECT-ID.iam.gserviceaccount.com.
• PROJECT_ID: (string) The ID of the customer's Google Cloud project to ingest metrics from.
• PREFIX: (list(string)) A list of metric prefixes to ingest from the Google Cloud project.

Delete a Google Cloud integration

To delete a Google Cloud integration with Chronoctl, use the command:

chronoctl gcp-metrics-integrations delete SLUG

Metric information

The following sections contain information about Google Cloud metrics information and how Chronosphere uses or displays it.

Metrics availability

On average, Google Cloud metrics display in Chronosphere with a delay on the order of minutes from the metric timestamp in Google Cloud. These metrics appear delayed due to the following factors:

• Latency delay: Most Google Cloud metrics have a delay (opens in a new tab) before they're made available for querying by Google Cloud API.

• Sample rate: Google Cloud metrics sample (opens in a new tab) at different rates. Most metrics sample at 60-second intervals, with some as long as once per day.

Due to these delays, configure monitors to expect a delay in arrival. Use the PromQL offset modifier (opens in a new tab) to account for delays.

Metric names

In Chronosphere, Google Cloud metrics (opens in a new tab) have the following name structure:

gcp_<MONITORED-RESOURCE-NAME>_<FULLY-QUALIFIED-METRIC-NAME>

where:

• MONITORED-RESOURCE-NAME is the monitored resource type (opens in a new tab) in Google Cloud.

• FULLY-QUALIFIED-DOMAIN-NAME is the metric name in Google Cloud, with characters like . and / convert to _ to be Prometheus compatible. The following examples are fully qualified metric names in Google Cloud before metric name sanitization:

  • bigquery.googleapis.com/job/num_in_flight
  • cloudsql.googleapis.com/database/available_for_failover
  • kubernetes.io/container/accelerator/memory_total

The following is an example of how to map a CloudSQL metric name in Google Cloud to a metric name in Chronosphere:

For this metric, cloudsql_database is the monitored resource name and cloudsql.googleapis.com/database/active_directory/domain_reachable is the fully qualified metric name. In Chronosphere, the metric name is gcp_cloudsql_database_cloudsql_googleapis_com_database_active_directory_domain_reachable.

Finding Google Cloud metrics in Metrics Explorer

Use Metrics Explorer to find and review the status of your ingested metrics.

• All Google Cloud metrics start with the prefix gcp_. Searching for this prefix displays all Google Cloud metrics in the platform.

• Substring search is possible. For example, if the original Google Cloud metric name contains a substring like domain_reachable, searching for the substring returns the Google Cloud metric, along with other metrics containing the substring.

Metric mappings

Google Cloud metrics map closely to Chronosphere metric types. Metric mappings are defined on the metric types page.

Metric prefix search

Google Cloud integration scrapes metrics by metric prefix search using the Monitoring Query Language (MQL) (opens in a new tab). This following table provides configuration examples for Google Cloud integration: