Skip to main content
Requires Chronosphere Telemetry Pipeline v2.72.0 or later.
The Google Cloud PubSub Input source plugin (name: gcp_pubsub) ingests data from Google Cloud Pub/Sub into a telemetry pipeline. You can use this plugin to ingest LogEntry objects generated from a GCP Logging Sink with a pull-type topic subscription. This is a push-based source plugin.

​
Supported telemetry types

The for Chronosphere Telemetry Pipeline supports these telemetry types:
LogsMetricsTraces

​
Configuration parameters

Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the Name column to describe these parameters. Pipeline configuration files use the items in the Key column as YAML keys.

​
General

NameKeyDescriptionDefault
Subscription IDsubscription_idRequired. This value can be either your Pub/Sub subscription ID or your fully qualified subscription name. However, if you don’t use a fully qualified subscription name, you must also set the project_id parameter.none
Project IDproject_idYour Google Cloud project ID. This parameter is required if the value of subscription_id isn’t a fully qualified subscription name.none

​
Advanced

NameKeyDescriptionDefault
nonemem_buf_limitFor pipelines with the Deployment or DaemonSet workload type only. Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit’s rules for unit sizes. If unspecified, no limit is enforced. In addition to this parameter, the subscription client for the Google Cloud PubSub Input source plugin has its own queue limit.none

​
Authentication

Use one of the following methods to authenticate with Google Cloud.

​
Use GKE workload identity

Use this method to provide credentials to ADC through an attached service account.
This method is supported only if you installed Telemetry Pipeline in a GKE cluster. Additionally, this method assumes that Telemetry Pipeline is installed in the same GCP project as your Pub/Sub topic subscription.
  1. Identify the Kubernetes namespace and service account associated with your Core Instance.
  2. In Google Cloud, create an IAM service account, or identify an existing IAM service account to associate with your Kubernetes service account.
  3. Grant your IAM service account the roles/pubsub.subscriber role: 
    gcloud projects add-iam-policy-binding IAM_SA_PROJECT_ID \
    --member "serviceAccount:IAM_SA_NAME@IAM_SA_PROJECT_ID.iam.gserviceaccount.com" \
     --role "roles/pubsub.subscriber"
    Replace the following values:
    • IAM_SA_PROJECT_ID: The Google Cloud project ID associated with your IAM service account.
    • IAM_SA_NAME: The name of your IAM service account.
  4. Create an IAM allow policy that gives your Kubernetes service account access to impersonate the IAM service account: 
    gcloud iam service-accounts add-iam-policy-binding IAM_SA_NAME@IAM_SA_PROJECT_ID.iam.gserviceaccount.com \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:IAM_SA_PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME]"
    Replace the following values:
    • IAM_SA_PROJECT_ID: The Google Cloud project ID associated with your IAM service account.
    • IAM_SA_NAME: The name of your IAM service account.
    • NAMESPACE: The Kubernetes namespace associated with your pipeline.
    • KSA_NAME: The Kubernetes service account associated with your pipeline.
  5. Annotate your Kubernetes service account with the name and project ID of your IAM policy: 
    kubectl annotate serviceaccount KSA_NAME \
    --namespace NAMESPACE \
    iam.gke.io/gcp-service-account=IAM_SA_NAME@IAM_SA_PROJECT_ID.iam.gserviceaccount.com
    Replace the following values:
    • IAM_SA_PROJECT_ID: The Google Cloud project ID associated with your IAM service account.
    • IAM_SA_NAME: The name of your IAM service account.
    • NAMESPACE: The Kubernetes namespace associated with your pipeline.
    • KSA_NAME: The Kubernetes service account associated with your pipeline.
  6. In Pipeline CLI, use the --service-account flag when you create or update a pipeline that contains this plugin: 
    calyptia {create|update} pipeline --service-account KSA_NAME
    Replace KSA_NAME with the Kubernetes service account associated with your pipeline.

​
Use GCP service account credentials

Use this method to provide credentials to ADC through the GOOGLE_APPLICATION_CREDENTIALS environment variable.
This method is supported for all Telemetry Pipeline installation methods, including Telemetry Pipeline installations that aren’t running in a GKE cluster.
  1. If you don’t already have an IAM service account in GCP, create a new IAM service account.
  2. Create a new IAM service account key to use with Telemetry Pipeline.
  3. Download the IAM key that you created. This download should be a JSON file.
  4. In your GKE cluster, create a Kubernetes secret from the JSON file that contains your IAM key.
  5. Use the --secret-volume-mount flag when you create or update the pipeline that contains this plugin: 
    calyptia {create|update} pipeline --secret-volume-mount SECRET:KEY=GOOGLE_APPLICATION_CREDENTIALS
    Replace the following values:
    • SECRET: The name of the secret you created.
    • KEY: The key that contains your credentials in that secret.
    Don’t replace GOOGLE_APPLICATION_CREDENTIALS with any other value.

​
Queue limit

The subscription client for the GCP PubSub source plugin has a 300 MB queue limit. This means the plugin will pause fetching data from Google Pub/Sub if its subscription queue exceeds 300 MB. If other constraints or limits are placed on your pipeline, the GCP PubSub source plugin might stop fetching data before this queue limit is reached. For example, if you use mem_buf_limit or resources.storage.backlogMemLimit to enforce a memory limit, and that memory limit is reached, those values take precedence over the 300 MB queue limit.