Troubleshooting
DNS Not Working
ziti-edge-tunnel run provides a built-in nameserver for the services it is authorized to dial. The nameserver is automatically configured by systemd-resolved, if enabled.
If systemd-resolved is not enabled, you must configure your resolver to query the tunneler's nameserver. Add Ziti's nameserver to the connection manager, e.g., NetworkManager, Netplan, or by directly editing /etc/resolv.conf.
You may configure the system resolver to use the tunneler's nameserver as the first or only nameserver.
When the tunneler nameserver is the first of multiple nameservers and the requested DNS record does not match an authorized service's intercept domain name, it sets the query status to REFUSE. This implies that the caller should keep trying to resolve the domain name with other nameservers.
To use the tunneler nameserver as the only nameserver, you must specify an upstream nameserver for recursion: ziti-edge-tunnel run --dns-upstream 208.67.222.222. In this configuration, the query status from the upstream nameserver is returned, e.g., NXDOMAIN if the domain name is not found in the tunneler nameserver or the upstream nameserver.
The IP address of the nameserver (default: 100.64.0.2) is determined by the tunneler's dns-ip-range (default: 100.64.0.1/10).
Increase Log Level
Set the log level to DEBUG to identify the activity that is occurring at the same time as the problem.
# set the logLevel to "debug" in /var/lib/ziti/config.json
sudo -u ziti ziti-edge-tunnel set_log_level --loglevel DEBUG
The tunneler obeys the value of logLevel in /var/lib/ziti/config.json. The initial value may be set with run
--verbose 4, but setting this option on subsequent runs has no effect on log level.
Capture the Current Process Log
journalctl _SYSTEMD_INVOCATION_ID=$(systemctl show -p InvocationID --value ziti-edge-tunnel.service) -l \
| tee /tmp/ziti-edge-tunnel.log
Systemd Service Won't Start or Keeps Restarting
Reload the systemd service unit definitions to rule out a stale definition.
sudo systemctl daemon-reload
Inspect the service unit.
sudo systemctl cat ziti-edge-tunnel.service
Check the service status for an error message.
sudo systemctl status ziti-edge-tunnel.service
Monitor the service logs.
sudo journalctl -u ziti-edge-tunnel.service
Intercepting or Hosting Not Working
Inspect the identity and router info for a running tunneler process. This creates a file named like {{identity name}}.ziti
for each loaded identity. Each file summarizes the available services and router connections for the identity.
sudo -u ziti ziti-edge-tunnel dump -p /tmp/ziti-dump-dir/
Find tunneler's nameserver IP.
$ resolvectl --interface=ziti0 dns
Link 19 (tun0): 100.64.0.2
Query the Ziti nameserver to find the intercept IP address for a service.
dig +noall +answer my.ziti.service.example.com @100.64.0.2
The tunneler provides end-to-end TCP handshake. Test the service's ability to accept connections even if it does not provide a greeting or banner as shown in the OpenSSH server example below.
# wait up to 3 seconds for a TCP handshake on port 443
$ ncat -vzw3 100.64.0.3 443
Ncat: Connected to 100.64.0.3:443.
Ncat: 0 bytes sent, 0 bytes received in 0.08 seconds.
# wait up to 3 seconds for an OpenSSH server greeting on port 22
$ ncat -vw3 100.64.0.3 22
SSH-2.0-OpenSSH_7.4
Process Keeps Crashing
A crash may be caused by a segmentation fault. If saving a Corefile is enabled, Linux will create a core dump file
according to this pattern file: /proc/sys/kernel/core_pattern. Ubuntu configures this to use
Apport. Read more about core dumps.
Please raise a GitHub issue if the tunneler crashes.
Operation Not Permitted
Delete the /tmp/.ziti directory and restart the tunneler to solve this issue.
The tunneler may log a warning about an operation not permitted and failure to start the socket server if the hard-coded path /tmp/.ziti is not owned by the run-as user. This can happen if you first run the tunneler as root and then as a non-root user.
WARN ziti-edge-tunnel:ziti-edge-tunnel.c:1686 make_socket_path() failed to set ownership of /tmp/.ziti to 1003:1003: Operation not permitted (errno=1)
WARN ziti-edge-tunnel:ziti-edge-tunnel.c:1730 run_tunneler_loop() One or more socket servers did not properly start.
Another symptom of the same issue is this error when attempting to send a message to the IPC socket server, which is not running.
failed to connect: -111/connection refused