Troubleshoot Monitoring Integrations

Review the common troubleshooting scenarios you might encounter while getting a Monitor integration working and see what you can do if an integration does not report metics after installation.

If you are directed to this page from the Sysdig Monitor app, your agent deployment might include a configuration that:

  • Prohibits the use of Monitoring Integrations
  • Affect the metrics you are currently collecting

Ensure that you meet the prerequisites to successfully use Monitoring Integrations. For technical assistance, contact Sysdig Support.

Check Prerequisites

To use Monitoring Integrations, ensure the following prerequisites are met:

  • Sysdig agent v12.0.0+

  • If you have clusters with more than 50 nodes and don’t have the prom_service_discovery option enabled:

    • Enabling the latest Prometheus features might create an additional connection to the Kubernetes API server from each Sysdig agent in your environment. The surge in agent connections can increase the CPU and memory load in your API servers. Therefore, ensure that your API servers are suitably sized to handle the increased load in large clusters.

  • Remove the following manual configurations in the dragent.yaml file because they might interfere with those provided by Sysdig:

    • use_promscrape
    • promscrape_fastproto
    • prom_service_discovery
    • prometheus.max_metrics
    • prometheus.ingest_raw
    • prometheus.ingest_calculated

  • The sysdig_sd_configs configuration is no longer supported. Remove the existing prometheus.yaml if it includes the sysdig_sd_configs configuration.

If you are not currently using Prometheus metrics in Sysdig Monitor, you can skip the following steps:

  • If you are using a custom Prometheus process_filter in dragent.yaml to trigger scraping, see Migrating from Promscrape V1 to V2.

  • If you are using service annotations or container labels to find scrape targets, you may need to create new scrape_configs in prometheus.yaml , preferably based on Kubernetes pods service discovery. This configuration can be complicated in certain environments and therefore we recommend that you contact Sysdig support for help.

  • Some integrations require secrets and other resources available in the correct namespace in order for it to work. Integrations such as database exporters might require you to create a user and provide special permissions in the database to connect to the endpoint and generate metrics.

Verify the Exporter is Running

Some integrations require exporters. If the problematic integration uses an exporter, ensure the pods corresponding to the exporter are running correctly. You can check this after installing the integration. If the exporter is installed as a sidecar of the application (such as Nginx), verify that the exporter container is added to the pod.

You can check the status of the pods with the Kubernetes dashboard Pod Status & Performance, or with the following command:

kubectl get pods --namespace=<namespace>

Additionally, if the container has problems and cannot start, use this command to check the description of the pod for error messages:

kubectl describe pod <pod-name> --namespace=<namespace>

Verify Metrics Are Generated

Check whether a running exporter is generating metrics by accessing the metrics endpoint:

kubectl port-forward <pod-name> <pod-port> <local-port> --namespace=<namespace>
curl http://localhost:<local-port>/metrics

This is also valid for applications that don’t need an exporter to generate their own metrics.

If the exporter is not generating metics, there could be problems accessing or authenticating with the application. Check the logs associated with the pods:

kubectl logs <pod-name> --namespace=<namespace>

If the application is instrumented and is not generating metrics, check if the Prometheus metrics option or the module is activated.

Verify Sysdig Agent Is Scraping Metrics

If an application doesn’t need an exporter to generate metrics, check if it has the default Prometheus annotations.

Additionally, you can check if the Sysdig agent can access the metrics endpoint with the following command:

kubectl exec <sysdig-agent-pod-name> --namespace=sysdig-agent -- /bin/sh -c "curl http://<exporer-pod-ip>:<pod-port>/metrics"

Select the Sysdig Agent pod in the same node than the pod used to scrape.

Contact Support

If your problems persist after troubelshooting, contact Sysdig Support.