> ## Documentation Index
> Fetch the complete documentation index at: https://docs.chronosphere.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Apache Spark

> Apache Spark application metrics read from a driver's REST API, including executor memory and task throughput, garbage collection, shuffle volume, and persisted RDD storage.

The Apache Spark integration requires CXDOT Collector 1.4.0 or greater.

[Apache Spark](https://spark.apache.org/) is a unified engine for large-scale data
analytics. Use the Apache Spark integration to collect app, job, stage, executor,
and storage metrics from Spark drivers running in your environment.

The Apache Spark integration supports Apache Spark 3.4.0 or greater.

## Supported telemetry types

The Apache Spark integration supports these telemetry types:

|   Type  | Supported |
| :-----: | :-------: |
|   Logs  |     No    |
| Metrics |    Yes    |
|  Traces |     No    |
|  Events |     No    |

## Prerequisites

The Apache Spark integration has the following prerequisites:

* Make the driver web interface for each Spark app reachable from the Collector.
  Point targets at app driver web interfaces, not Spark standalone masters,
  Apache Hadoop Yet Another Resource Negotiator (YARN) ResourceManagers, or
  Apache Mesos masters.

## Configure

To configure the Apache Spark integration, follow these steps:

1. Choose how the Collector finds Spark drivers. The Apache Spark integration
   runs by default and discovers
   Kubernetes driver pods labeled `spark-role=driver` that expose port `4040`.
   It also discovers Kubernetes Pods and Services that carry

   autodiscovery annotations for Spark. For more information, see
   [autodiscovery](https://docs.chronosphere.io/ingest/cxdot-collector/autodiscovery).

2. Optional: Configure static driver targets for Spark apps outside Kubernetes.
   A non-empty `endpoints` list turns off discovery for that integration instance,
   and the Collector collects only the listed targets. To run discovery and
   static targets together, add a named integration instance. For example, add
   the following to the `values.yaml` for your Helm chart:

   ```yaml theme={null}
   config:
     integrations:
       spark: {}
       spark/external:
         endpoints:
           - endpoint: "spark-driver.example.com:4040"
   ```

   The empty `spark` entry keeps discovery active. `spark/external` is an
   independent, named instance that collects from the listed driver.

To disable the default Spark integration instance, set `enabled` to `false`. For
example, add the following to the `values.yaml` for your Helm chart:

```yaml theme={null}
config:
  integrations:
    spark:
      enabled: false
```

### Validate

To validate the Apache Spark integration, follow these steps:

1. In Live Telemetry Analyzer, filter for
   `__name__=cxdot.integration.target.health` and confirm the metric reports `1`
   for each Spark target. The `cxdot.integration.name` and
   `cxdot.integration.check` labels have the value `spark`, and
   `cxdot.integration.target` identifies the driver.

2. In Metrics Explorer, run the following query:

   ```text theme={null}
   count by ("server.address", "server.port", "spark.application.name") ({"spark.executor.task.total"})
   ```

   Confirm the query returns a value for each active Spark app.

### Troubleshooting

* No app metrics appear for a standalone master, YARN ResourceManager, or
  Apache Mesos master: Configure each app's driver web interface as a target.
  The integration doesn't use cluster managers to discover apps.

For more information about diagnosing a failing integration, see
[Troubleshooting](https://docs.chronosphere.io/ingest/cxdot-collector/troubleshooting).

## Configuration reference

Configure one Apache Spark integration instance with the following settings. In Helm values,
place these settings under `config.integrations.spark`. In a Collector configuration file, place
them under `cxdot.integrations.spark`.

### Optional settings

* **`enabled`**
  Type: `boolean`. Optional. Default: `true`.
  Whether to enable this Spark integration instance. If true, the Collector runs the instance.
  If false, the Collector doesn't run it.

* **`endpoints`**
  Type: `array of object`. Optional.
  Static Spark app driver web interface targets. When this list is empty or omitted, the
  integration uses automatic target discovery. When the list contains targets, this integration
  instance collects from exactly those targets and doesn't automatically discover targets.
  Configure a second named instance to use both methods. Specify each target as `host:port` to
  use HTTP, or as a full HTTP or HTTPS URL. The URL must identify the web interface root and
  must not include a Spark REST API path. Configure app driver web interfaces only. The
  integration doesn't discover apps through cluster managers.

* **`endpoints[].endpoint`**
  Type: `string`. Required.
  Spark app driver web interface target as `host:port` or a full HTTP or HTTPS URL. For
  `host:port`, the Collector uses HTTP.

* **`application_names`**
  Type: `array of string`. Optional. Default: `[]`.
  Spark app names to collect from each target. When the list is empty, the Collector collects
  every app reported by the target.

* **`collection_interval`**
  Type: `duration`. Optional. Default: `30s`.
  How often the Collector collects metrics from each Spark target.

* **`timeout`**
  Type: `duration`. Optional. Default: `30s`.
  Maximum time allowed to collect metrics from one Spark target. The value must not exceed
  `collection_interval`.

* **`tls`**
  Type: `object`. Optional.
  TLS certificate and client authentication settings for HTTPS connections to Spark targets.

* **`tls.ca_file`**
  Type: `string`. Optional.
  Path to the CA cert. For a client this verifies the server certificate. For a server this
  verifies client certificates. If empty uses system root CA. (optional)

* **`tls.ca_pem`**
  Type: `string`. Optional.
  In memory PEM encoded cert. (optional)

* **`tls.cert_file`**
  Type: `string`. Optional.
  Path to the TLS cert to use for TLS required connections. (optional)

* **`tls.cert_pem`**
  Type: `string`. Optional.
  In memory PEM encoded TLS cert to use for TLS required connections. (optional)

* **`tls.cipher_suites`**
  Type: `array of string`. Optional.
  CipherSuites is a list of TLS cipher suites that the TLS transport can use. If left blank, a
  safe default list is used. See
  [https://go.dev/src/crypto/tls/cipher\_suites.go](https://go.dev/src/crypto/tls/cipher_suites.go)
  for a list of supported cipher suites.

* **`tls.curve_preferences`**
  Type: `array of string`. Optional.
  contains the elliptic curves that will be used in an ECDHE handshake, in preference order
  Defaults to empty list and "crypto/tls" defaults are used, internally.

* **`tls.include_system_ca_certs_pool`**
  Type: `boolean`. Optional.
  If true, load system CA certificates pool in addition to the certificates configured in this
  struct.

* **`tls.insecure`**
  Type: `boolean`. Optional.
  In gRPC and HTTP when set to true, this is used to disable the client transport security. See
  [https://godoc.org/google.golang.org/grpc#WithInsecure](https://godoc.org/google.golang.org/grpc#WithInsecure)
  for gRPC. Please refer to
  [https://godoc.org/crypto/tls#Config](https://godoc.org/crypto/tls#Config) for more
  information. (optional, default false)

* **`tls.insecure_skip_verify`**
  Type: `boolean`. Optional.
  InsecureSkipVerify will enable TLS but not verify the certificate.

* **`tls.key_file`**
  Type: `string`. Optional.
  Path to the TLS key to use for TLS required connections. (optional)

* **`tls.key_pem`**
  Type: `string`. Optional.
  In memory PEM encoded TLS key to use for TLS required connections. (optional)

* **`tls.max_version`**
  Type: `string`. Optional.
  MaxVersion sets the maximum TLS version that is acceptable. If not set, refer to crypto/tls
  for defaults. (optional)

* **`tls.min_version`**
  Type: `string`. Optional.
  MinVersion sets the minimum TLS version that is acceptable. If not set, TLS 1.2 will be used.
  (optional)

* **`tls.reload_interval`**
  Type: `duration`. Optional.
  ReloadInterval specifies the duration after which the certificate will be reloaded If not set,
  it will never be reloaded (optional)

* **`tls.server_name_override`**
  Type: `string`. Optional.
  ServerName requested by client for virtual hosting. This sets the ServerName in the TLSConfig.
  Please refer to [https://godoc.org/crypto/tls#Config](https://godoc.org/crypto/tls#Config) for
  more information. (optional)

* **`tls.tpm`**
  Type: `object`. Optional.
  Trusted platform module configuration

* **`tls.tpm.auth`**
  Type: `string`. Optional.
  Authorization value for the trusted platform module key.

* **`tls.tpm.enabled`**
  Type: `boolean`. Optional.
  Whether to use a trusted platform module for the TLS private key. If true, the Collector loads
  the key from the configured device or socket. If false, the Collector uses the configured key
  file or in-memory key.

* **`tls.tpm.owner_auth`**
  Type: `string`. Optional.
  Owner authorization value for the trusted platform module.

* **`tls.tpm.path`**
  Type: `string`. Optional.
  The path to the TPM device or Unix domain socket. For instance /dev/tpm0 or /dev/tpmrm0.


## Related topics

- [Apache](/ingest/xcor/integrations/collector/apache.md)
- [Tomcat](/ingest/xcor/integrations/collector/tomcat.md)
- [Kafka destination plugin](/ingest/pipeline/plugins/destination-plugins/kafka.md)
- [Datagen source plugin](/ingest/pipeline/plugins/source-plugins/datagen.md)
- [Kafka source plugin](/ingest/pipeline/plugins/source-plugins/kafka.md)