Portal Server

Each article describes a different situation.

• Solution for: "failed to create fs.notify watcher: too many open files" error, execute this command on the master node machine:

  1. sudo sysctl -w fs.inotify.max_user_watches=2099999999

  2. sudo sysctl -w fs.inotify.max_user_instances=2099999999

  3. sudo sysctl -w fs.inotify.max_queued_events=2099999999

  4. sysctl -p

• Solution for authentication: If sec.yaml keys and values are the same on the UI and yaml, redeploy the sec component. But this time please only use yaml file instead of adding keys & values by hand, Rancher encrypts secrets in his own way. If key & value fields are given by hand, Rancher's encryption is not working and an authentication problem will happen.

• If the onprem.cloudcyte.com gives "cannot get/" error, please check the auth pod. Sometimes it needs to be restarted.

• If the servers have issue with connecting to the internet, please check:

  1. certificates -> "Storage -> Secrets" on rancher

  2. configmap -> "Storage -> ConfigMaps" on rancher, there should be no_proxy/NO_PROXY and http(s)_proxy/HTTP(s)_PROXY fields

  3. pods -> check the pods that have config maps proxy variables

  4. /etc/resolv.conf

  5. /etc/resolvconf/resolv.conf.d/head

  6. /etc/dnsmasq.conf

  7. /etc/network/interfaces

• Check DNS Port:

  1. ss -lptn 'sport = :53'

• Show k8s container logs of the rancher:

  • kubectl logs -f <pod_full_name>

• For applying the proxy settings again or resetting the rancher, please use rke2-manager.sh on the portal server, it is a builtin script in the portal that CyberCyte provides. With the script, many confiurations can be applied with minutes.

• If the servers have issue with communicating with the other servers, please check:

  1. /etc/resolv.conf

  2. /etc/resolvconf/resolv.conf.d/head

  3. /etc/dnsmasq.conf

  4. /etc/netplan/<file>.yaml

• Check DNS Port via executing these commands:

  • ss -lptn 'sport = :53'

• Check services via executing these commands:

  • systemctl status rke2-server

• Check server resources via executing these commands:

  • df -h

  • top

  • btop (if exists)

  • lsblk

  • ncdu (if exist)