Troubleshooting On-Premises Installation

Collect Troubleshooting Data

When experiencing issues, you can collect troubleshooting data that can help the support team. The data can be collected by hand, or Sysdig provides a very simple get_support_bundle.sh script that takes as an argument the namespace where Sysdig is deployed and will generate a tarball containing some information (mostly log files). The script is located in the GitHub repository.

$ ./scripts/get_support_bundle.sh sysdigcloud
Getting support logs for sysdigcloud-api-1477528018-4od59
Getting support logs for sysdigcloud-api-1477528018-ach89
Getting support logs for sysdigcloud-cassandra-2987866586-fgcm8
Getting support logs for sysdigcloud-collector-2526360198-e58uy
Getting support logs for sysdigcloud-collector-2526360198-v1egg
Getting support logs for sysdigcloud-mysql-2388886613-a8a12
Getting support logs for sysdigcloud-redis-1701952711-ezg8q
Getting support logs for sysdigcloud-worker-1086626503-4cio9
Getting support logs for sysdigcloud-worker-1086626503-sdtrc
Support bundle generated: 1473897425_sysdig_cloud_support_bundle.tgz

Docker Connectivity Issues (IPv4/IPv6)

Some issues with IPv4 and IPv6 interconnectivity between on-premises containers and the outside world have been detected.

IP packet forwarding is governed by the ip_forward system parameter. Packets can only pass between containers if this parameter is 1. Usually, you will simply leave the Docker server at its default setting --ip-forward=true and Docker will go set ip_forward to 1 for you when the server starts up. If you set --ip-forward=false and your system’s kernel has it enabled, the --ip-forward=false option has no effect.

To check the setting on your kernel use:

sysctl net.ipv4.conf.all.forwarding

To turn it on use:

sysctl net.ipv4.conf.all.forwarding=1

Please see this article from dockerfor more details on Docker Connectivity.

Proxy/Firewall Issues

Prior to installing ensure your proxy settings are valid for the session. You can use curl, lynx, or wget to test internet connectivity:

export http_proxy="http://user:password@proxy_server:port"
export https_proxy="https://user:password@proxy_server:port"
echo $http_proxy

You can then attempt a curl or docker hub call to ensure outside connectivity.

Firewall

Prior to installation, you may want to disable local firewall (iptables) to rule out local connectivity issues.

However here are some details around Sysdig connectivity and backend connectivity requirements.

Sysdig Connectivity:

6443 Agent communication

443 Sysdig Monitor UI access

8800 Management console access

Here are specifics around what is used for connectivity for the Sysdig backend for on-premises solution:

https://www.replicated.com/docs/kb/supporting-your-customers/firewalls/

File Write Permissions Issues (SELINUX or APP ARMOR)

During the install, you may see errors writing to volumes such as (/var or /opt) from either the onprem install scripts or Docker. You should disable SELINUX (CENTOS/RHEL) or Apparmor (UBUNTU/DEBIAN) during the course of the install so the valid directories can be created. This can be accomplished by:

Centos (SELINUX)

From the command line, edit the /etc/sysconfig/selinux file. This file is a symlink to /etc/selinux/config. The configuration file is self-explanatory. Changing the value of SELINUX or *SELINUXTYPE*changes the state of SELinux and the name of the policy to be used the next time the system boots.

[root@host2a ~]# cat /etc/sysconfig/selinux
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#       enforcing - SELinux security policy is enforced.
#       permissive - SELinux prints warnings instead of enforcing.
#       disabled - SELinux is fully disabled.
SELINUX=permissive
# SELINUXTYPE= type of policy in use. Possible values are:
#       targeted - Only targeted network daemons are protected.
#       strict - Full SELinux protection.
SELINUXTYPE=targeted

# SETLOCALDEFS= Check local definition changes
SETLOCALDEFS=0

See SELinux Modes for more information.

UBUNTU/Debian (AppArmor)

AppArmor can be disabled, and the kernel module unloaded by entering the following:

sudo systemctl stop apparmor.service
sudo update-rc.d -f apparmor remove

To re-enable AppArmor enter:

sudo systemctl start apparmor.service
sudo update-rc.d apparmor defaults

Advanced Troubleshooting - Firewall, IPtables, IP forwarding

In the preflight check step with Replicated, if you come across the error:

getsockopt: no route to host

Please do the following:

For CentOS 7/RedHat:

Log in as root or run these commands via sudo:

service firewalld stop
systemctl disable firewalld
sysctl -w net.ipv4.ip_forward=1
iptables -F
setenforce 0
service docker restart

For Ubuntu:

Log in as root or run these commands via sudo:

sysctl -w net.ipv4.ip_forward=1
systemctl stop apparmor.service
update-rc.d -f apparmor remove
ufw disable
iptables -F
service docker restart

Learn More

See Get Help | Using Sysdig Support (On-Prem).