> ## Documentation Index
> Fetch the complete documentation index at: https://docs.chronosphere.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting pipelines

Use this information to help troubleshoot your pipelines.

<Note>
  If you need additional assistance, contact [Chronosphere Support](/support).
  As part of the support process, Chronosphere recommends downloading and running the
  [`support.sh`](https://raw.githubusercontent.com/calyptia/charts/master/support.sh)
  script, which gathers information about your Telemetry Pipeline installation
  and packages this information into an archive that you can attach to your ticket.

  However, be aware that the output generated by the `support.sh` script can include
  any information related to your Telemetry Pipeline installation, including the
  values of secrets or authentication credentials you've stored. Before
  sending this output to Chronosphere, be sure to redact any sensitive
  information accordingly.
</Note>

## Problems with data ingestion

If you're having trouble with data ingestion in your pipeline, follow the troubleshooting
procedures that correspond to your source type:

* **Push-based**: Agents, network devices, and services send data through various
  protocols (including TCP, HTTP, UDP, and OTLP) to a pipeline.
* **Pull-based**: A pipeline retrieves data from a SaaS service, Kafka, Prometheus,
  cloud storage, or other location.

### Push-based sources

To troubleshoot pushed-based sources, follow each of these procedures:

{/* vale PANW.WordList = NO */}

1. [Simplify your pipeline](#simplify-push).
2. [Examine your service, host, port, and protocol](#examine-your-service-host-port-and-protocol).
3. [Review your firewalls](#firewalls-push).
4. [Verify your SSL/TLS settings](#verify-ssltls).

{/* markdownlint-enable MD051 */}

<h4 id="simplify-push">
  Simplify your pipeline
</h4>

Simplifying your pipeline can help you narrow down the
cause of any issues. You can either perform these actions on an existing pipeline,
or create a new test pipeline that meets these criteria.

1. Choose one source to keep for your pipeline, then remove all other sources.
2. Set your pipeline's destination to [standard output](/ingest/pipeline/plugins/destination-plugins/standard-output).
   This destination lets you view real-time logs generated by your pipeline.
3. Turn off or remove any active processing rules.
4. Review the logs generated by your pipeline. You can use any of the following
   methods to do so:

   <Tabs>
     <Tab title="Web" id="ui-push">
       1. Sign in to the [Telemetry Pipeline UI](https://core.calyptia.com/)
       2. Open the project that contains the pipeline whose logs you want to view.
       3. Go to **Core Instances**, then click the name of the Core Instance associated
          with the pipeline whose logs you want to view.
       4. In the **Data Pipelines** section, click the name of your pipeline.
       5. In the [**Pipeline Output**](/ingest/pipeline/navigate#pipeline-output-v2-pipelines) section,
          click **Get Latest Logs**.
     </Tab>

     <Tab title="kubectl" id="kubectl-push">
       1. Run the following command to retrieve the replicas of a pipeline:

          ```shell theme={null}
          kubectl get pods
          ```

       2. To display logs in your terminal, run the following command:

          ```shell theme={null}
          kubectl logs POD
          ```

          Replace *`POD`* with the name of the pod where your pipeline is deployed.

       3. Optional: Run the following command to follow a stream of data
          in real time:

          ```shell theme={null}
          kubectl logs -f POD
          ```

          Replace *`POD`* with the name of the pod where your pipeline is deployed.
     </Tab>
   </Tabs>

#### Examine your service, host, port, and protocol

To examine your service, host, port, and protocol, perform these steps:

1. Run the following command to view the services configured by your
   pipeline's Core Instance:

   ```shell theme={null}
   kubectl get svc
   ```

2. Review the output of the previous step to find information about your pipeline,
   including its host, port, and protocol. For example, a pipeline named `mypipe`
   in a Core Instance named `instance` would have the following service name and
   associated information:

   `instance-mypipe-PORT-PROTOCOL-UUID`

3. If the output of step 1 includes a `pending` notice for the external IP,
   recreate the pipeline.

4. If you're working in a Kubernetes cluster and don't see any active services,
   ensure your cluster policies and governance tools aren't blocking
   service creation.

{/* vale PANW.Acronyms = NO */}

<h4 id="firewalls-push">
  Review firewalls
</h4>

To review your firewalls, perform these steps:

1. If you're working in a cloud environment, certain inbound ports might be blocked
   by default. Communicate with your cloud provider to confirm that traffic can flow
   through the required port and protocol for your pipeline.

2. If your Linux server or Kubernetes cluster is deployed in a virtual private
   cloud (VPC), confirm that external push-based sources can access the host through
   the VPC and load balancer.

{/* vale PANW.WordList = YES */}

#### Verify SSL/TLS

To verify your SSL/TLS settings, perform these steps:

1. Confirm that your pipeline is using SSL/TLS version 1.2 or later.

2. Confirm that the certificate for your pipeline and the certificate for your
   push-based source are the same.

3. Use the `tls.verify off` setting to temporarily turn off certificate validation,
   which helps confirm or rule out issues with a parent certificate.

### Pull-based sources

{/* vale PANW.WordList = NO */}

To troubleshoot pull-based sources, follow each of these procedures:

1. [Simplify your pipeline](#simplify-pull).
2. [Test connectivity](#test-connectivity).
3. [Review your firewalls](#firewalls-pull).
4. [Validate credentials and authorization](#validate-credentials-and-authorization).

{/* markdownlint-enable MD051 */}

<h4 id="simplify-pull">
  Simplify your pipeline
</h4>

Simplifying your pipeline can help you narrow down the
cause of any issues. You can either perform these actions on an existing pipeline,
or create a new test pipeline that meets these criteria.

1. Choose one source to keep for your pipeline, then remove all other sources.

2. Set your pipeline's destination to [standard output](/ingest/pipeline/plugins/destination-plugins/standard-output).
   This destination lets you view real-time logs generated by your pipeline.

3. Turn off or remove any active processing rules.

4. Review the logs generated by your pipeline. You can use any of the following
   methods to do so:

   <Tabs>
     <Tab title="Web" id="ui-pull">
       1. Sign in to the [Telemetry Pipeline UI](https://core.calyptia.com/)

       2. Open the project that contains the pipeline whose logs you want to view.

       3. Go to **Core Instances**, then click the name of the Core Instance where you
          deployed the pipeline whose logs you want to view.

       4. In the **Data Pipelines** section, click the name of your pipeline.

       5. In the [**Pipeline Output**](/ingest/pipeline/navigate#pipeline-output-v2-pipelines)
          section, click **Get Latest Logs**.
     </Tab>

     <Tab title="kubectl" id="kubectl-pull">
       1. Run the following command to retrieve the replicas of a pipeline:

          ```shell theme={null}
          kubectl get pods
          ```

       2. To display logs in your terminal, run the following command:

          ```shell theme={null}
          kubectl logs POD
          ```

          Replace *`POD`* with the name of the pod where your pipeline is deployed.

       3. Optional: Run this command to follow a stream of data
          in real time:

          ```shell theme={null}
          kubectl logs -f POD
          ```

          Replace *`POD`* with the name of the pod where your pipeline is deployed.
     </Tab>
   </Tabs>

#### Test connectivity

To test connectivity, perform these steps:

1. From the server where your Core Instance is deployed, run this command to confirm
   your pipeline can access the push-based source:

   ```shell theme={null}
   ping SOURCE
   ```

   Replace *`SOURCE`* with the IP or host of the push-based source.

2. From the server of your push-based source, run the following command to confirm
   that the source can access your pipeline:

   ```shell theme={null}
   ping PIPELINE
   ```

   Replace *`PIPELINE`* with the IP or host of your pipeline.

<h4 id="firewalls-pull">
  Review firewalls
</h4>

Confirm there aren't any firewalls blocking outbound traffic from the push-based
source.

#### Validate credentials and authorization

Confirm your push-based source isn't generating HTTP `401 Unauthorized` status codes
or any other errors that indicate issues with credentials and authorization.

## Problems sending data to a destination

If you're having trouble using your pipeline to send data to a destination,
follow each of these troubleshooting procedures:

1. [Simplify your pipeline](#simplify-destination).
2. [Review your logs for console errors](#review-your-logs-for-console-errors).

<h3 id="simplify-destination">
  Simplify your pipeline
</h3>

Simplifying your pipeline can help you narrow down the
cause of any issues. You can either perform these actions on an existing pipeline,
or create a new test pipeline that meets these criteria.

1. Set your pipeline's source to [Mock Data](/ingest/pipeline/plugins/source-plugins/mock-data).
   This source sends `{"message": "dummy"}` to the specified output at
   one-second intervals.

2. Add [standard output](/ingest/pipeline/plugins/destination-plugins/standard-output) as a
   destination. This destination lets you view real-time logs generated by your pipeline.

3. Turn off or remove any active processing rules.

4. Review the logs generated by your pipeline. You can use any of the following
   methods to do so:

   <Tabs>
     <Tab title="Web" id="ui-destination">
       1. Sign in to the [Telemetry Pipeline UI](https://core.calyptia.com/)

       2. Open the project that contains the pipeline whose logs you want to view.

       3. Go to **Core Instances**, then click the name of the Core Instance where you
          deployed the pipeline whose logs you want to view.

       4. In the **Data Pipelines** section, click the name of your pipeline.

       5. In the [**Pipeline Output**](/ingest/pipeline/navigate#pipeline-output-v2-pipelines)
          section, click **Get Latest Logs**.
     </Tab>

     <Tab title="kubectl" id="kubectl-destination">
       1. Run the following command to retrieve the replicas of a pipeline:

          ```shell theme={null}
          kubectl get pods
          ```

       2. To display logs in your terminal, run the following command:

          ```shell theme={null}
          kubectl logs POD
          ```

          Replace *`POD`* with the name of the pod where your pipeline is deployed.

       3. Optional: Run the following command to follow a stream of data in real time:

          ```shell theme={null}
          kubectl logs -f POD
          ```

          Replace *`POD`* with the name of the pod where your pipeline is deployed.
     </Tab>
   </Tabs>

### Review your logs for console errors

Your pipeline's logs might display certain errors. Use the following table
to determine what each error means and how to troubleshoot it.

| Console output                                                          | Cause                                   | How to troubleshoot                                                                                                                                                                                                                    |
| ----------------------------------------------------------------------- | --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `400: Error`                                                            | Incorrect data format                   | • Verify the format of the logs you're routing.                                                                                                                                                                                        |
| `401: Unauthorized`                                                     | Incorrect credentials when routing data | • Review your authorization settings.<br /> • Review your pipeline secrets.                                                                                                                                                            |
| SSL/TLS error                                                           | Certificate error                       | • Verify your certificate settings.<br /> • Temporarily turn off TLS, then see if the issue persists.                                                                                                                                  |
| No log messages displayed in output                                     | Data routing or pipeline thrashing      | • Ensure you're running the right query in your backend.<br />• Add `log_level: debug` as a parameter to your pipeline output.<br /> • Use `kubectl get` and `kubectl describe` to review the performance and attributes of your pods. |
| `200: OK` or `204: No Content`, but no log messages displayed in output | Wrong query                             | • Ensure you're running the right query in your backend.                                                                                                                                                                               |