Add Agent Files

After generating the files, add them to the Support ticket you created.

Debug the Agent Start Command or Manifest File

Many agent connection problems are due to transcription errors in the agent start command or manifest files.

This is especially true with truncated access keys and when using the ADDITIONAL_CONF parameter in a container agent installation. Always cut and paste, and then modify, the Sysdig example commands or manifest files.

The Helm values, docker start, and native agent run commands are available in the Secure UI. To find them:

1. Log in to Sysdig Secure.

2. Navigate to Integrations > Data Sources page.

3. Select Add Agent in the top right corner.

If you are not an Admin level user you will not see the installation tab, in this case, please request the instructions from the admin.

• Try running the agent from the command line using the ‘docker start’ or curl command.

  1. Ensure that you have the agent container ID.

     docker ps -a
     

  2. Stop the container.

     docker stop <container_id>
     

  3. Debug the container. To do so, create a directory and copy log files to that directory.

     mkdir test_log
     docker cp sysdig-agent:/opt/draios/logs ./test_log/
     

  4. Once debugging is complete, start the agent container.

     docker start <container_id>
     

• Send in the command used and initial agent output.

• Tip: if using docker start to debug the agent live, remove the ‘-d' option so the output will display on the console.

Sysdig Agent Configuration File and Logs

When troubleshooting issues such as metrics not reporting, agent connection failures, or abnormal behavior, it is essential to collect the Sysdig Agent configuration and log files. Provide the Sysdig Support team with context-specific diagnostic data to ensure faster and more effective assistance. The method for gathering these files depends on your deployment type, such as Kubernetes, Docker, or native Linux.

General Guidance

• Restart the agent before collecting logs to capture critical startup information.
  The method for restarting the agent depends on your deployment:

  • Kubernetes:
    Delete the relevant agent Pod to restart it. Kubernetes will automatically create a new Pod:

    kubectl -n NAMESPACE delete pod sysdig-agent-AGENT_ID
    

  • Docker:

    docker restart sysdig-agent
    

  • Linux service-based agents:

    service dragent restart
    

• File locations differ by deployment type:

  • In Kubernetes, configuration is typically sourced from a ConfigMap and logs reside in the agent pod’s filesystem.
  • In Docker deployments, configuration and logs are within the running container.
  • In native Linux installations, files are found directly on the host system.

• Typical file paths (depending on deployment):

  • Main configuration file: /opt/draios/etc/dragent.yaml
  • Log directory: /opt/draios/logs/
  • Primary log file: /opt/draios/logs/draios.log

  The agent rotates the log file when it reaches 10MB, retaining the 10 most recent log files, each with a timestamp.

  Always provide the host name, pod name, or relevant VM/instance where the files were collected.

Collecting Files by Deployment

Kubernetes Deployments

• Obtain the agent configuration (from the ConfigMap):

  kubectl -n NAMESPACE get configmap sysdig-agent -o yaml > sysdig-agent-configmap.yaml
  

• Collect all agent logs:

  kubectl -n NAMESPACE cp sysdig-agent-AGENT_ID:/opt/draios/logs/. .
  

  The above command will grab ALL the log files (including the rollovers) and place them in the directory where the command was run.

• For agent Pods not in Running/Ready state, attach the output of:

  kubectl -n NAMESPACE describe pod sysdig-agent-AGENT_ID > describe-pod.txt
  

• Optionally, export the DaemonSet configuration for cluster-level troubleshooting:

  kubectl -n NAMESPACE get daemonset sysdig-agent -o yaml > sysdig-agent-daemonset.yaml
  

Docker Deployments

• Copy the configuration file and all log files from the running Sysdig agent container:

  docker cp sysdig-agent:/opt/draios/etc/dragent.yaml ./dragent.yaml
  docker cp sysdig-agent:/opt/draios/logs/. ./logs/
  

Linux Agents (.rpm/.deb Packages)

• Copy the configuration and log files from the host to your workstation:

  cp /opt/draios/etc/dragent.yaml ./dragent.yaml
  cp -r /opt/draios/logs ./logs/
  

What to Attach to Your Support Case

• The agent configuration file, dragent.yaml, as applicable
• All files from /opt/draios/logs/ or container/pod equivalents

For Kubernetes:

• Agent logs
• sysdig-agent-configmap.yaml
• sysdig-agent-daemonset.yaml
• describe-pod.txt for non-running pods
• Helm values.yaml file used to deploy the agent
• If OOMKill is occurring, collect logs from the previous container using --previous

Also include:

• Any other logs or files relevant to the issue
• The hostname, pod name, or instance name where the files were collected

Compress the files into a .zip or .tar.gz archive and attach them to your support case.

Debug Log Level

If requested by Support, enable debug logging in the agent configuration before collecting new logs. For more information, see Debug log level available.

Kubernetes: Check for Delegated Agents

With agents deployed in a Kubernetes environment, it can be useful to know which agents are “delegated” to poll the Kubernetes API server.

To do this, you must generate agent log files (see above) and search for ‘delegated’.

Logs are stored in /opt/draios/logs/draios.log.

• For agent versions 0.77.0 or later: you will see log entries stating whether your agent is/is not the delegated node and the exact address of the delegated node.

  Search for the string “Delegated’.

• For agents before version 0.77.0: issue the command:

  kubectl get nodes
  

  By default, the top two addresses automatically became delegated polling agents.

• Alternatively, enabling debug mode in agent versions 0.64.0 and beyond will show the node list in any agent logs as well.

Docker Swarm: Find the Docker Swarm Master

When using a Sysdig agent inside a Docker Swarm, metadata (information about inventory, labels, ext..) is collected from the Docker Swarm masters.

Effort should be made to collect the logs from the masters, and more specifically the “leader” when a problem with Docker Swarm inventory is suspected.

To determine which Docker Swarm nodes are masters and leaders:

1. Run the command:

   docker node ls
   

   Example:

   $ docker node ls
   ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
   789sgbfdbtcdtn9npbnkuz worker1 Ready Active
   76avknaakzbvmwlnh5jr3g * manager1 Ready Active Leader
   

2. Under the Manager Status column, you can note which node is the “Leader”.