Collect Troubleshooting Data
When experiencing issues, you can collect troubleshooting data that can
help the support team. The data can be collected by hand, or Sysdig
provides a very simple get_support_bundle.sh script that takes as an
argument the namespace where Sysdig is deployed and will generate a
tarball containing some information (mostly log files). The script is
located in the GitHub repository.
$ ./scripts/get_support_bundle.sh sysdigcloud
Getting support logs for sysdigcloud-api-1477528018-4od59
Getting support logs for sysdigcloud-api-1477528018-ach89
Getting support logs for sysdigcloud-cassandra-2987866586-fgcm8
Getting support logs for sysdigcloud-collector-2526360198-e58uy
Getting support logs for sysdigcloud-collector-2526360198-v1egg
Getting support logs for sysdigcloud-mysql-2388886613-a8a12
Getting support logs for sysdigcloud-redis-1701952711-ezg8q
Getting support logs for sysdigcloud-worker-1086626503-4cio9
Getting support logs for sysdigcloud-worker-1086626503-sdtrc
Support bundle generated: 1473897425_sysdig_cloud_support_bundle.tgz
Docker Connectivity Issues (IPv4/IPv6)
Some issues with IPv4 and IPv6 interconnectivity between on-premises containers and the outside world have been detected.
IP packet forwarding is governed by the ip_forward system parameter.
Packets can only pass between containers if this parameter is 1.
Usually, you will simply leave the Docker server at its default setting
--ip-forward=true and Docker will go set ip_forward to 1 for you
when the server starts up. If you set --ip-forward=false and your
system’s kernel has it enabled, the --ip-forward=false option has no
effect.
To check the setting on your kernel use:
sysctl net.ipv4.conf.all.forwarding
To turn it on use:
sysctl net.ipv4.conf.all.forwarding=1
Please see this article from docker for more details on Docker Connectivity.
Proxy/Firewall Issues
Prior to installing ensure your proxy settings are valid for the
session. You can use curl, lynx, or wget to test internet
connectivity:
export http_proxy="http://user:password@proxy_server:port"
export https_proxy="https://user:password@proxy_server:port"
echo $http_proxy
You can then attempt a curl or docker hub call to ensure outside connectivity.
Firewall
Prior to installation, you may want to disable local firewall (iptables) to rule out local connectivity issues.
However here are some details around Sysdig connectivity and backend connectivity requirements.
Sysdig Connectivity:
6443 Agent communication
443 Sysdig Monitor UI access
8800 Management console access
Here are specifics around what is used for connectivity for the Sysdig backend for on-premises solution:
https://www.replicated.com/docs/kb/supporting-your-customers/firewalls/
File Write Permissions Issues (SELINUX or APP ARMOR)
During the install, you may see errors writing to volumes such as
(/var or /opt) from either the onprem install scripts or Docker. You
should disable SELINUX (CENTOS/RHEL) or Apparmor (UBUNTU/DEBIAN)
during the course of the install so the valid directories can be
created. This can be accomplished by:
Centos (SELINUX)
From the command line, edit the /etc/sysconfig/selinux file. This file
is a symlink to /etc/selinux/config. The configuration file is
self-explanatory. Changing the value of SELINUX or
*SELINUXTYPE*changes the state of SELinux and the name of the policy
to be used the next time the system boots.
[root@host2a ~]# cat /etc/sysconfig/selinux
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - SELinux is fully disabled.
SELINUX=permissive
# SELINUXTYPE= type of policy in use. Possible values are:
# targeted - Only targeted network daemons are protected.
# strict - Full SELinux protection.
SELINUXTYPE=targeted
# SETLOCALDEFS= Check local definition changes
SETLOCALDEFS=0
See SELinux Modes for more information.
UBUNTU/Debian (AppArmor)
AppArmor can be disabled, and the kernel module unloaded by entering the following:
sudo systemctl stop apparmor.service
sudo update-rc.d -f apparmor remove
To re-enable AppArmor enter:
sudo systemctl start apparmor.service
sudo update-rc.d apparmor defaults
Advanced Troubleshooting - Firewall, IPtables, IP forwarding
In the preflight check step with Replicated, if you come across the error:
getsockopt: no route to host
Please do the following:
For CentOS 7/RedHat:
Log in as root or run these commands via sudo:
service firewalld stop
systemctl disable firewalld
sysctl -w net.ipv4.ip_forward=1
iptables -F
setenforce 0
service docker restart
For Ubuntu:
Log in as root or run these commands via sudo:
sysctl -w net.ipv4.ip_forward=1
systemctl stop apparmor.service
update-rc.d -f apparmor remove
ufw disable
iptables -F
service docker restart