Google Cloud Big Query

Google Cloud BigQuery destination plugin

The Google Cloud BigQuery destination plugin lets you configure Chronosphere Telemetry Pipeline to send your log data and metrics to Google Cloud BigQuery.

Supported telemetry types

This plugin supports these telemetry types:

LogsMetricsTraces

Configuration parameters

Use the parameters in this section to configure your plugin. The Telemetry Pipeline web interface uses the values in the Name column to describe the parameters. Items in the Key column are the YAML keys to use in pipeline configuration files.

General

NameKeyDescriptionDefault
Google Service Credentials Pathgoogle_service_credentialsRequired. The Service Credentials file lets Telemetry Pipeline communicate directly with Google Cloud Services. For information about how to set up service credentials, see the Google Cloud Observability documentation (opens in a new tab).none
Google Project Idproject_idThe project ID containing the BigQuery dataset to stream into. If the service file is provided then the project ID is taken from there.none
Existing Data Set IDdataset_idThe dataset ID of the BigQuery dataset to write into. This dataset must exist in your project.none
Existing Table IDtable_idThe table ID of the BigQuery table to write into. This table must exist in the specified dataset and the schema must match the output.none

Advanced

NameKeyDescriptionDefault
Skip Invalid Rowsskip_invalid_rowsIf true, insert all valid rows of a request, even if invalid rows exist. If false, cause the entire request to fail if any invalid rows exist. Accepted values: true, false.false
Ignore Unknown Valuesignore_unknown_valuesAccept rows that contain values that don't match the schema, and ignores unknown values. When set to false, treats unknown values as errors. Accepted values: true, false.false
Enable Workload Identity Federationenable_workload_identity_federationEnables workload identity federation as an alternative authentication method. Can't be used with service account credentials file or environment variable. AWS is the only supported identity provider. Accepted values: true, false.false
AWS Region for BigQueryaws_regionGoogle Cloud region for BigQuery.none
GCP Project Numberproject_numberGoogle Cloud project number where the identity provider was created. Used to construct the full resource name of the identity provider.none
GCP Pool Idpool_idGoogle Cloud workload identity pool where the identity provider was created. Used to construct the full resource name of the identity provider.none
GCP Provider Idprovider_idGoogle Cloud workload identity provider. Used to construct the full resource name of the identity provider. Only AWS accounts are supported.none

Security and TLS

NameKeyDescriptionDefault
TLStlsEnable or disable TLS/SSL support. Accepted values: true, false.false
TLS Certificate Validationtls.verifyTurn TLS/SSL certificate validation on or off. TLS must be on for this setting to be enabled.on
TLS Debug Leveltls.debugSet TLS debug verbosity level. Accepted values: 0 (No debug), 1 (Error), 2 (State change), 3 (Informational), 4 (Verbose).1
CA Certificate File Pathtls.ca_fileAbsolute path to CA certificate file.none
Certificate File Pathtls.crt_fileAbsolute path to certificate file.none
Private Key File Pathtls.key_fileAbsolute path to private key file.none
Private Key Path Passwordtls.key_passwdOptional password for tls.key_file file.none
TLS SNI Hostname Extensiontls.vhostHostname to be used for TLS SNI extension.none

Advanced Networking

NameKeyDescriptionDefault
DNS Modenet.dns.modeSelect the primary DNS connection type, which can be TCP or UDP.none
DNS Resolvernet.dns.resolverSelect the primary DNS connection type, which can be LEGACY or ASYNC.none
Prefer IPv4net.dns.prefer_ipv4Prioritize IPv4 DNS results when trying to establish a connection. Accepted values: true, false.false
Keepalivenet.keepaliveEnable or disable Keepalive support. Accepted values: true, false.true
Keepalive Idle Timeoutnet.keepalive_idle_timeoutSet maximum time allowed for an idle Keepalive connection.30s
Max Connect Timeoutnet.connect_timeoutSet maximum time allowed to establish a connection, which includes the TLS handshake.10s
Max Connect Timeout Log Errornet.connect_timeout_log_errorOn connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message. Accepted values: true, false.true
Max Keepalive Recyclenet.keepalive_max_recycleSet maximum number of times a keepalive connection can be used before it's retired.2000
Source Addressnet.source_addressSpecify network address to bind for data traffic.none