Google Cloud BigQuery destination plugin
The Google Cloud BigQuery destination plugin lets you configure Chronosphere Telemetry Pipeline to send your log data and metrics to Google Cloud BigQuery.
Supported telemetry types
This plugin supports these telemetry types:
| Logs | Metrics | Traces |
|---|---|---|
Configuration parameters
Use the parameters in this section to configure your plugin. The Telemetry Pipeline web interface uses the values in the Name column to describe the parameters. Items in the Key column are the YAML keys to use in pipeline configuration files.
General
| Name | Key | Description | Default |
|---|---|---|---|
| Google Service Credentials Path | google_service_credentials | Required. The Service Credentials file lets Telemetry Pipeline communicate directly with Google Cloud Services. For information about how to set up service credentials, see the Google Cloud Observability documentation (opens in a new tab). | none |
| Google Project Id | project_id | The project ID containing the BigQuery dataset to stream into. If the service file is provided then the project ID is taken from there. | none |
| Existing Data Set ID | dataset_id | The dataset ID of the BigQuery dataset to write into. This dataset must exist in your project. | none |
| Existing Table ID | table_id | The table ID of the BigQuery table to write into. This table must exist in the specified dataset and the schema must match the output. | none |
Advanced
| Name | Key | Description | Default |
|---|---|---|---|
| Skip Invalid Rows | skip_invalid_rows | If true, insert all valid rows of a request, even if invalid rows exist. If false, cause the entire request to fail if any invalid rows exist. Accepted values: true, false. | false |
| Ignore Unknown Values | ignore_unknown_values | Accept rows that contain values that don't match the schema, and ignores unknown values. When set to false, treats unknown values as errors. Accepted values: true, false. | false |
| Enable Workload Identity Federation | enable_workload_identity_federation | Enables workload identity federation as an alternative authentication method. Can't be used with service account credentials file or environment variable. AWS is the only supported identity provider. Accepted values: true, false. | false |
| AWS Region for BigQuery | aws_region | Google Cloud region for BigQuery. | none |
| GCP Project Number | project_number | Google Cloud project number where the identity provider was created. Used to construct the full resource name of the identity provider. | none |
| GCP Pool Id | pool_id | Google Cloud workload identity pool where the identity provider was created. Used to construct the full resource name of the identity provider. | none |
| GCP Provider Id | provider_id | Google Cloud workload identity provider. Used to construct the full resource name of the identity provider. Only AWS accounts are supported. | none |
Security and TLS
| Name | Key | Description | Default |
|---|---|---|---|
| TLS | tls | Enable or disable TLS/SSL support. Accepted values: true, false. | false |
| TLS Certificate Validation | tls.verify | Enable or disable TLS/SSL certificate validation. TLS must be enabled for certificates to be validated. Accepted values: off, on. | on |
| TLS Debug Level | tls.debug | Set TLS debug verbosity level. Accepted values: 0 (No debug), 1 (Error), 2 (State change), 3 (Informational), 4 (Verbose). | 1 |
| CA Certificate File Path | tls.ca_file | Absolute path to CA certificate file. | none |
| Certificate File Path | tls.crt_file | Absolute path to certificate file. | none |
| Private Key File Path | tls.key_file | Absolute path to private key file. | none |
| Private Key Path Password | tls.key_passwd | Password for private key file. | none |
| TLS SNI Hostname Extension | tls.vhost | Hostname to be used for TLS SNI extension. | none |
Advanced Networking
| Name | Key | Description | Default |
|---|---|---|---|
| DNS Mode | net.dns.mode | Select the primary DNS connection type, which can be TCP or UDP. | none |
| DNS Resolver | net.dns.resolver | Select the primary DNS connection type, which can be LEGACY or ASYNC. | none |
| Prefer IPv4 | net.dns.prefer_ipv4 | Prioritize IPv4 DNS results when trying to establish a connection. Accepted values: true, false. | false |
| Keepalive | net.keepalive | Enable or disable Keepalive support. Accepted values: true, false. | true |
| Keepalive Idle Timeout | net.keepalive_idle_timeout | Set maximum time allowed for an idle Keepalive connection. | 30s |
| Max Connect Timeout | net.connect_timeout | Set maximum time allowed to establish a connection, which includes the TLS handshake. | 10s |
| Max Connect Timeout Log Error | net.connect_timeout_log_error | On connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message. Accepted values: true, false. | true |
| Max Keepalive Recycle | net.keepalive_max_recycle | Set maximum number of times a keepalive connection can be used before it's retired. | 2000 |
| Source Address | net.source_address | Specify network address to bind for data traffic. | none |