GCP PubSub Input source plugin - Chronosphere Documentation

This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform.

Requires Chronosphere Telemetry Pipeline v2.72.0 or later.

The Google Cloud PubSub Input source plugin (name: gcp_pubsub) ingests data from Google Cloud Pub/Sub into a telemetry pipeline. You can use this plugin to ingest LogEntry objects generated from a GCP Logging Sink with a pull-type topic subscription. This is a push-based source plugin.

Supported telemetry types

The for Chronosphere Telemetry Pipeline supports these telemetry types:

Logs	Metrics	Traces

Configuration parameters

Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the Name column to describe these parameters. Pipeline configuration files use the items in the Key column as YAML keys.

General

Name	Key	Description	Default
Subscription ID	`subscription_id`	Required. This value can be either your Pub/Sub subscription ID or your fully qualified subscription name. However, if you don’t use a fully qualified subscription name, you must also set the `project_id` parameter.	none
Project ID	`project_id`	Your Google Cloud project ID. This parameter is required if the value of `subscription_id` isn’t a fully qualified subscription name.	none

Advanced

Name	Key	Description	Default
none	`mem_buf_limit`	Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit’s rules for unit sizes. If unspecified, no limit is enforced. For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet workload type. To learn more, see the v2 backpressure guide. For v3 pipelines, this parameter is independent from the OpenTelemetry `memory_limiter` and `batch` processors. In addition to this parameter, the subscription client for the Google Cloud PubSub Input source plugin has its own queue limit.	none

Authentication

Use one of the following methods to authenticate with Google Cloud.

Use GKE workload identity

Use this method to provide credentials to ADC through an attached service account.

This method is supported only if you installed Telemetry Pipeline in a GKE cluster. Additionally, this method assumes that Telemetry Pipeline is installed in the same GCP project as your Pub/Sub topic subscription.

Identify the Kubernetes namespace and service account associated with your Core Instance.
In Google Cloud, create an IAM service account, or identify an existing IAM service account to associate with your Kubernetes service account.
Grant your IAM service account the roles/pubsub.subscriber role:
gcloud projects add-iam-policy-binding IAM_SA_PROJECT_ID \ --member "serviceAccount:IAM_SA_NAME@IAM_SA_PROJECT_ID.iam.gserviceaccount.com" \ --role "roles/pubsub.subscriber"
Replace the following values:
- IAM_SA_PROJECT_ID: The Google Cloud project ID associated with your IAM service account.
- IAM_SA_NAME: The name of your IAM service account.
Create an IAM allow policy that gives your Kubernetes service account access to impersonate the IAM service account:
gcloud iam service-accounts add-iam-policy-binding IAM_SA_NAME@IAM_SA_PROJECT_ID.iam.gserviceaccount.com \ --role roles/iam.workloadIdentityUser \ --member "serviceAccount:IAM_SA_PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME]"
Replace the following values:
- IAM_SA_PROJECT_ID: The Google Cloud project ID associated with your IAM service account.
- IAM_SA_NAME: The name of your IAM service account.
- NAMESPACE: The Kubernetes namespace associated with your pipeline.
- KSA_NAME: The Kubernetes service account associated with your pipeline.
Annotate your Kubernetes service account with the name and project ID of your IAM policy:
kubectl annotate serviceaccount KSA_NAME \ --namespace NAMESPACE \ iam.gke.io/gcp-service-account=IAM_SA_NAME@IAM_SA_PROJECT_ID.iam.gserviceaccount.com
Replace the following values:
- IAM_SA_PROJECT_ID: The Google Cloud project ID associated with your IAM service account.
- IAM_SA_NAME: The name of your IAM service account.
- NAMESPACE: The Kubernetes namespace associated with your pipeline.
- KSA_NAME: The Kubernetes service account associated with your pipeline.
In Pipeline CLI, use the --service-account flag when you create or update a pipeline that contains this plugin:
calyptia {create|update} pipeline --service-account KSA_NAME
Replace KSA_NAME with the Kubernetes service account associated with your pipeline.

Use GCP service account credentials

Use this method to provide credentials to ADC through the GOOGLE_APPLICATION_CREDENTIALS environment variable.

This method is supported for all Telemetry Pipeline installation methods, including Telemetry Pipeline installations that aren’t running in a GKE cluster.

If you don’t already have an IAM service account in GCP, create a new IAM service account.
Create a new IAM service account key to use with Telemetry Pipeline.
Download the IAM key that you created. This download should be a JSON file.
In your GKE cluster, create a Kubernetes secret from the JSON file that contains your IAM key.
Use the --secret-volume-mount flag when you create or update the pipeline that contains this plugin:
calyptia {create|update} pipeline --secret-volume-mount SECRET:KEY=GOOGLE_APPLICATION_CREDENTIALS
Replace the following values:
- SECRET: The name of the secret you created.
- KEY: The key that contains your credentials in that secret.
Don’t replace GOOGLE_APPLICATION_CREDENTIALS with any other value.

Queue limit

The subscription client for the GCP PubSub source plugin has a 300 MB queue limit. This means the plugin will pause fetching data from Google Pub/Sub if its subscription queue exceeds 300 MB. If other constraints or limits are placed on your pipeline, the GCP PubSub source plugin might stop fetching data before this queue limit is reached. For example, if you use mem_buf_limit or resources.storage.backlogMemLimit to enforce a memory limit for a v2 pipeline, and that memory limit is reached, those values take precedence over the 300 MB queue limit.

​Supported telemetry types

​Configuration parameters

​General

​Advanced

​Authentication

​Use GKE workload identity

​Use GCP service account credentials

​Queue limit

Supported telemetry types

Configuration parameters

General

Advanced

Authentication

Use GKE workload identity

Use GCP service account credentials

Queue limit