Apache Spark

​

Supported telemetry types

​

Prerequisites

• Make the driver web interface for each Spark app reachable from the Collector. Point targets at app driver web interfaces, not Spark standalone masters, Apache Hadoop Yet Another Resource Negotiator (YARN) ResourceManagers, or Apache Mesos masters.

​

Configure

1. Choose how the Collector finds Spark drivers. The Apache Spark integration runs by default and discovers Kubernetes driver pods labeled spark-role=driver that expose port 4040. It also discovers Kubernetes Pods and Services that carry autodiscovery annotations for Spark. For more information, see autodiscovery.
2. Optional: Configure static driver targets for Spark apps outside Kubernetes. A non-empty endpoints list turns off discovery for that integration instance, and the Collector collects only the listed targets. To run discovery and static targets together, add a named integration instance. For example, add the following to the values.yaml for your Helm chart:
   config:
     integrations:
       spark: {}
       spark/external:
         endpoints:
           - endpoint: "spark-driver.example.com:4040"
   
   The empty spark entry keeps discovery active. spark/external is an independent, named instance that collects from the listed driver.

config:
  integrations:
    spark:
      enabled: false

​

Validate

1. In Live Telemetry Analyzer, filter for __name__=cxdot.integration.target.health and confirm the metric reports 1 for each Spark target. The cxdot.integration.name and cxdot.integration.check labels have the value spark, and cxdot.integration.target identifies the driver.
2. In Metrics Explorer, run the following query:
   count by ("server.address", "server.port", "spark.application.name") ({"spark.executor.task.total"})
   
   Confirm the query returns a value for each active Spark app.

​

Troubleshooting

• No app metrics appear for a standalone master, YARN ResourceManager, or Apache Mesos master: Configure each app’s driver web interface as a target. The integration doesn’t use cluster managers to discover apps.

​

Configuration reference

​

Optional settings

• enabled Type: boolean. Optional. Default: true. Whether to enable this Spark integration instance. If true, the Collector runs the instance. If false, the Collector doesn’t run it.
• endpoints Type: array of object. Optional. Static Spark app driver web interface targets. When this list is empty or omitted, the integration uses automatic target discovery. When the list contains targets, this integration instance collects from exactly those targets and doesn’t automatically discover targets. Configure a second named instance to use both methods. Specify each target as host:port to use HTTP, or as a full HTTP or HTTPS URL. The URL must identify the web interface root and must not include a Spark REST API path. Configure app driver web interfaces only. The integration doesn’t discover apps through cluster managers.
• endpoints[].endpoint Type: string. Required. Spark app driver web interface target as host:port or a full HTTP or HTTPS URL. For host:port, the Collector uses HTTP.
• application_names Type: array of string. Optional. Default: []. Spark app names to collect from each target. When the list is empty, the Collector collects every app reported by the target.
• collection_interval Type: duration. Optional. Default: 30s. How often the Collector collects metrics from each Spark target.
• timeout Type: duration. Optional. Default: 30s. Maximum time allowed to collect metrics from one Spark target. The value must not exceed collection_interval.
• tls Type: object. Optional. TLS certificate and client authentication settings for HTTPS connections to Spark targets.
• tls.ca_file Type: string. Optional. Path to the CA cert. For a client this verifies the server certificate. For a server this verifies client certificates. If empty uses system root CA. (optional)
• tls.ca_pem Type: string. Optional. In memory PEM encoded cert. (optional)
• tls.cert_file Type: string. Optional. Path to the TLS cert to use for TLS required connections. (optional)
• tls.cert_pem Type: string. Optional. In memory PEM encoded TLS cert to use for TLS required connections. (optional)
• tls.cipher_suites Type: array of string. Optional. CipherSuites is a list of TLS cipher suites that the TLS transport can use. If left blank, a safe default list is used. See https://go.dev/src/crypto/tls/cipher_suites.go for a list of supported cipher suites.
• tls.curve_preferences Type: array of string. Optional. contains the elliptic curves that will be used in an ECDHE handshake, in preference order Defaults to empty list and “crypto/tls” defaults are used, internally.
• tls.include_system_ca_certs_pool Type: boolean. Optional. If true, load system CA certificates pool in addition to the certificates configured in this struct.
• tls.insecure Type: boolean. Optional. In gRPC and HTTP when set to true, this is used to disable the client transport security. See https://godoc.org/google.golang.org/grpc#WithInsecure for gRPC. Please refer to https://godoc.org/crypto/tls#Config for more information. (optional, default false)
• tls.insecure_skip_verify Type: boolean. Optional. InsecureSkipVerify will enable TLS but not verify the certificate.
• tls.key_file Type: string. Optional. Path to the TLS key to use for TLS required connections. (optional)
• tls.key_pem Type: string. Optional. In memory PEM encoded TLS key to use for TLS required connections. (optional)
• tls.max_version Type: string. Optional. MaxVersion sets the maximum TLS version that is acceptable. If not set, refer to crypto/tls for defaults. (optional)
• tls.min_version Type: string. Optional. MinVersion sets the minimum TLS version that is acceptable. If not set, TLS 1.2 will be used. (optional)
• tls.reload_interval Type: duration. Optional. ReloadInterval specifies the duration after which the certificate will be reloaded If not set, it will never be reloaded (optional)
• tls.server_name_override Type: string. Optional. ServerName requested by client for virtual hosting. This sets the ServerName in the TLSConfig. Please refer to https://godoc.org/crypto/tls#Config for more information. (optional)
• tls.tpm Type: object. Optional. Trusted platform module configuration
• tls.tpm.auth Type: string. Optional. Authorization value for the trusted platform module key.
• tls.tpm.enabled Type: boolean. Optional. Whether to use a trusted platform module for the TLS private key. If true, the Collector loads the key from the configured device or socket. If false, the Collector uses the configured key file or in-memory key.
• tls.tpm.owner_auth Type: string. Optional. Owner authorization value for the trusted platform module.
• tls.tpm.path Type: string. Optional. The path to the TPM device or Unix domain socket. For instance /dev/tpm0 or /dev/tpmrm0.