OBSERVABILITY PLATFORM
Investigate
Logs
Route logs from Google Cloud Platform

Route logs from Google Cloud Platform

This feature is available only to specific Chronosphere Observability Platform users, and has not been announced or officially released. Do not share or discuss this feature, or information about it, with anyone outside of your organization.

Google Cloud Platform provides sinks (opens in a new tab), which control how you can route log data to supported destinations. Use the Google Cloud Pub/Sub (opens in a new tab) destination to route log data from Google Cloud Platform to Chronosphere Observability Platform.

Complete the following steps to route logs from Google Cloud Platform:

  1. Create a sink for logs.
  2. Create a service account.
  3. Create a pull subscription.
  4. Contact Chronosphere Support.

If you want to use Terraform to manage routing Google logs to Observability Platform, refer to the Terraform example. After applying this configuration, Terraform generates the information that you supply to Chronosphere Support.

Create a sink for logs

First, you need to create a sink that defines the service type and destination to route your logs.

  1. In the Google Cloud Logs Console, in the left navigation, click Log Router to open the Log router (opens in a new tab) page.

  2. Follow the steps in the Google Cloud documentation to create a sink (opens in a new tab).

  3. Select Cloud Pub/Sub topic as the sink service.

  4. Optional: Enter a filter expression that matches the log entries you want to include.

    See the Logging query language (opens in a new tab) in the Google Cloud documentation for information on how to construct queries.

  5. Complete the remaining steps to create the sink.

Next, create a service account.

Create a service account

After creating a sink, you create a service account in Google Cloud that Observability Platform impersonates.

  1. In the Google Cloud console, create a service account (opens in a new tab).

  2. Add the Observability Platform principal to the Google Cloud service account.

    Each Google Cloud service account must grant access to the Observability Platform principal to impersonate them. The Observability Platform principal format is:

    gcp-logs-ADDRESS@chronosphere-production-a|b|c.iam.gserviceaccount.com
    • ADDRESS: Your company name prefixed to your Observability Platform instance that ends in .chronosphere.io. For example, MY_COMPANY.chronosphere.io.

    Your cluster can vary (a, b, or c). Check with your account team to ensure you have the correct format.

    Grant the principal the iam.serviceAccountTokenCreator role.

  3. Enter information for the remaining fields to finish creating the service account.

Next, create a pull subscription.

Create a pull subscription

Add a pull subscription (opens in a new tab) to the Pub/Sub destination you previously created. Observability Platform initiates requests to the Pub/Sub server to retrieve messages.

  1. In the Google Cloud console, Create a pull subscription (opens in a new tab).

  2. In the Delivery type menu, select Pull.

  3. Configure the following recommended settings:

    SettingValue
    Message retention duration23 hours*
    Expiration periodNever expire
    Acknowledgement deadline60 seconds
    Exactly once deliveryNo
    Message orderingNo
    Dead letteringNo
    Retry policyRetry after exponential backoff delay

    *Set Message retention duration to less than 24 hours to avoid storage costs.

  4. Grant access to the Pub/Sub you created by assigning the following roles to your new internal service account:

    • roles/pubsub.subscriber
    • roles/pubsub.viewer

Lastly, contact Chronosphere and provide the details about your pull subscription.

Contact Chronosphere Support

After completing the previous steps, contact Chronosphere Support and provide the following information:

  • The name of your new Google Cloud service account email, such as:

    SERVICE-ACCOUNT@PROJECT-ID.iam.gserviceaccount.com
    • SERVICE-ACCOUNT is the name of the Google Cloud service account you created in create a service account.
    • PROJECT-ID is the ID of your Google Cloud project.

  • The name of the logging sink subscription you created, such as

    projects/PROJECT-ID/subscriptions/SUBSCRIPTION

Chronosphere can then enable the logging integration to start routing your Google Cloud logs to Observability Platform.

Terraform example

The following code provides an example for creating a single Google Cloud service account in the a Google Cloud project, and enables Observability Platform to impersonate and gain access.

locals {
  // Email address of your Chronosphere Logs tenant-specific principal.
  chronosphere_sa_email = "gcp-logs-ADDRESS@chronosphere-production-b.iam.gserviceaccount.com"
 
  // Google Cloud project containing logging data to be ingested into
  // Observability Platform.
  logging_project_id = "PROJECT_ID"
 
  // Organization ID of the Google Cloud organization containing the project.
  org_id = "ORGANIZATION_ID"
}
 
// Creates Pub/Sub topic for logging sink.
resource "google_pubsub_topic" "main" {
  name    = "logging_sink"
  project = local.project_id
}
 
// Creates a new logging sink.
resource "google_logging_organization_sink" "main" {
  org_id           = local.org_id
  name             = "all_logs_sink"
  destination      = "pubsub.googleapis.com/${google_pubsub_topic.main.id}"
  include_children = true
}
 
// Grants sink permission to roles so they can write to the Pub/Sub topic.
data "google_iam_policy" "topic" {
  binding {
    role = "roles/pubsub.publisher"
    members = [
      google_logging_organization_sink.main.writer_identity,
    ]
  }
}
 
// Binds the role to a new topic.
resource "google_pubsub_topic_iam_policy" "main" {
  project     = local.project_id
  topic       = google_pubsub_topic.main.name
  policy_data = data.google_iam_policy.topic.policy_data
}
 
// Creates a new Pub/Sub subscription.
resource "google_pubsub_subscription" "main" {
  name    = "all_logs_sink"
  project = local.project_id
  topic   = google_pubsub_topic.main.name
 
  message_retention_duration = "23h"
  ack_deadline_seconds = 60
 
  retry_policy {
    minimum_backoff = "10s"
  }
 
  enable_message_ordering    = false
  enable_exactly_once_delivery = false
}
 
// Creates a policy granting the new Chronosphere logs service account access to
// the subscription.
data "google_iam_policy" "subscription" {
  binding {
    role = "roles/pubsub.viewer"
    members = [
      google_service_account.chronosphere_logs.member
    ]
  }
 
  binding {
    role    = "roles/pubsub.subscriber"
    members = [
      google_service_account.chronosphere_logs.member
    ]
  }
}
 
// Binds the new policy to the subscription.
resource "google_pubsub_subscription_iam_policy" "main" {
  project      = local.project_id
  subscription = google_pubsub_subscription.main.name
  policy_data  = data.google_iam_policy.subscription.policy_data
}
 
// Service account that lets an Observability Platform tenant-specific principal
// to impersonate it.
resource "google_service_account" "chronosphere_logs" {
  project    = local.project_id
  account_id = "chronosphere-logs"
}
 
// The service account provides the Observability Platform tenant-specific principal with
// roles/iam.serviceAccountTokenCreator access so that it can impersonate it. Only
// the Observability Platform tenant-specific principal can perform this
// impersonation.
data "google_iam_policy" "chronosphere_logs" {
  binding {
    role    = "roles/iam.serviceAccountTokenCreator"
    members = ["serviceAccount:${local.chronosphere_sa_email}"]
  }
}
 
// Assigns the token creator permission to the service account.
resource "google_service_account_iam_policy" "chronosphere_logs" {
  service_account_id = google_service_account.chronosphere_logs.name
  policy_data        = data.google_iam_policy.chronosphere_logs.policy_data
}
External tasksLive Telemetry Analyzer