Troubleshooting pipelines

Use this information to help troubleshoot your pipelines.

Problems with data ingestion

If you're having trouble with data ingestion in your pipeline, follow the troubleshooting procedures that correspond to your source type:

• Push-based: Agents, network devices, and services send data through various protocols (including TCP, HTTP, UDP, and OTLP) to a pipeline.
• Pull-based: A pipeline retrieves data from a SaaS service, Kafka, Prometheus, cloud storage, or other location.

Push-based sources

To troubleshoot pushed-based sources, follow each of these procedures:

1. Simplify your pipeline.
2. Examine your service, host, port, and protocol.
3. Review your firewalls.
4. Verify your SSL/TLS settings.

Simplify your pipeline

Simplifying your pipeline can help you narrow down the cause of any issues. You can either perform these actions on an existing pipeline, or create a new test pipeline that meets these criteria.

1. Choose one source to keep for your pipeline, then remove all other sources.

2. Set your pipeline's destination to standard output. This destination lets you view realtime logs generated by your pipeline.

3. Turn off or remove any active processing rules.

4. Review the logs generated by your pipeline. You can use any of the following methods to do so:

   Webkubectl
   1. Sign in to the Telemetry Pipeline UI (opens in a new tab)
   2. Go to Core Instances, then click the name of the Core Instance where you deployed the pipeline whose logs you'd like to view.
   3. In the Data Pipelines section, click the name of your pipeline.
   4. In the Pipeline Output section, click Get Latest Logs.

Examine your service, host, port, and protocol

To examine your service, host, port, and protocol, perform these steps:

1. Run the following command to view the services configured by your pipeline's Core Instance:

   kubectl get svc

2. Review the output of the previous step to find information about your pipeline, including its host, port, and protocol. For example, a pipeline named mypipe in a Core Instance named instance would have the following service name and associated information:

   instance-mypipe-PORT-PROTOCOL-UUID

3. If the output of step one includes a pending notice for the external IP, recreate the pipeline.

4. If you're working in a Kubernetes cluster and don't see any active services, ensure your cluster policies and governance tools aren't blocking service creation.

Review firewalls

To review your firewalls, perform these steps:

1. If you're working in a cloud environment, certain inbound ports might be blocked by default. Communicate with your cloud provider to confirm that traffic can flow through the required port and protocol for your pipeline.

2. If your Linux server or Kubernetes cluster is deployed in a virtual private cloud (VPC), confirm that external push-based sources can access the host through the VPC and load balancer.

Verify SSL/TLS

To verify your SSL/TLS settings, perform these steps:

1. Confirm that your pipeline is using SSL/TLS version 1.2 or greater.

2. Confirm that the certificate for your pipeline and the certificate for your push-based source are the same.

3. Use the tls.verify off setting to temporarily turn off certificate validation, which helps confirm or rule out issues with a parent certificate.

Pull-based sources

To troubleshoot pull-based sources, follow each of these procedures:

1. Simplify your pipeline.
2. Test connectivity.
3. Review your firewalls.
4. Validate credentials and authorization.

Simplify your pipeline

Simplifying your pipeline can help you narrow down the cause of any issues. You can either perform these actions on an existing pipeline, or create a new test pipeline that meets these criteria.

1. Choose one source to keep for your pipeline, then remove all other sources.

2. Set your pipeline's destination to standard output. This destination lets you view realtime logs generated by your pipeline.

3. Turn off or remove any active processing rules.

4. Review the logs generated by your pipeline. You can use any of the following methods to do so:

   Webkubectl

   1. Sign in to the Telemetry Pipeline UI (opens in a new tab)

   2. Go to Core Instances, then click the name of the Core Instance where you deployed the pipeline whose logs you'd like to view.

   3. In the Data Pipelines section, click the name of your pipeline.

   4. In the Pipeline Output section, click Get Latest Logs.

Test connectivity

To test connectivity, perform these steps:

1. From the server where your Core Instance is deployed, run this command to confirm your pipeline can access the push-based source:

   ping SOURCE

   Replace SOURCE with the IP or host of the push-based source.

2. From the server of your push-based source, run the following command to confirm that the source can access your pipeline:

   ping PIPELINE

   Replace PIPELINE with the IP or host of your pipeline.

Review firewalls

Confirm there aren't any firewalls blocking outbound traffic from the push-based source.

Validate credentials and authorization

Confirm your push-based source isn't generating HTTP 401 Unauthorized status codes or any other errors that indicate issues with credentials and authorization.

Problems sending data to a destination

If you're having trouble using your pipeline to send data to a destination, follow each of these troubleshooting procedures:

1. Simplify your pipeline.
2. Review your logs for console errors.

Simplify your pipeline

Simplifying your pipeline can help you narrow down the cause of any issues. You can either perform these actions on an existing pipeline, or create a new test pipeline that meets these criteria.

1. Set your pipeline's source to Mock Data. This source sends {"message": "dummy"} to the specified output at one-second intervals.

2. Add standard output as a destination. This destination lets you view realtime logs generated by your pipeline.

3. Turn off or remove any active processing rules.

4. Review the logs generated by your pipeline. You can use any of the following methods to do so:

   Webkubectl

   1. Sign in to the Telemetry Pipeline UI (opens in a new tab)

   2. Go to Core Instances, then click the name of the Core Instance where you deployed the pipeline whose logs you'd like to view.

   3. In the Data Pipelines section, click the name of your pipeline.

   4. In the Pipeline Output section, click Get Latest Logs.

Review your logs for console errors

Your pipeline's logs might display certain errors. Use the following table to determine what each error means and how to troubleshoot it.