Debugging and troubleshooting¶

Headscale and Tailscale provide debug and introspection capabilities that can be helpful when things don't work as expected. This page explains some debugging techniques to help pinpoint problems.

Please also have a look at Tailscale's Troubleshooting guide. It offers a many tips and suggestions to troubleshoot common issues.

Tailscale¶

The Tailscale client itself offers many commands to introspect its state as well as the state of the network:

Check local network conditions: tailscale netcheck
Get the client status: tailscale status --json
Get DNS status: tailscale dns status --all
Client logs: tailscale debug daemon-logs
Client netmap: tailscale debug netmap
Test DERP connection: tailscale debug derp headscale
And many more, see: tailscale debug --help

Many of the commands are helpful when trying to understand differences between Headscale and Tailscale SaaS.

Headscale¶

Application logging¶

The log levels debug and trace can be useful to get more information from Headscale.

log:
  # Valid log levels: panic, fatal, error, warn, info, debug, trace
  level: debug

Database logging¶

The database debug mode logs all database queries. Enable it to see how Headscale interacts with its database. This also requires the application log level to be set to either debug or trace.

database:
  # Enable debug mode. This setting requires the log.level to be set to "debug" or "trace".
  debug: false

log:
  # Valid log levels: panic, fatal, error, warn, info, debug, trace
  level: debug

Metrics and debug endpoint¶

Headscale provides a metrics and debug endpoint. It allows to introspect different aspects such as:

Information about the Go runtime, memory usage and statistics
Connected nodes and pending registrations
Active ACLs, filters and SSH policy
Current DERPMap
Prometheus metrics

Keep the metrics and debug endpoint private

The listen address and port can be configured with the metrics_listen_addr variable in the configuration file. By default it listens on localhost, port 9090.

Keep the metrics and debug endpoint private to your internal network and don't expose it to the Internet.

Query metrics via http://localhost:9090/metrics and get an overview of available debug information via http://localhost:9090/debug/. Metrics may be queried from outside localhost but the debug interface is subject to additional protection despite listening on all interfaces.

Direct accessSSH port forwardingVia debug keyVia debug IP address

Access the debug interface directly on the server where Headscale is installed.

curl http://localhost:9090/debug/

Use SSH port forwarding to forward Headscale's metrics and debug port to your device.

ssh <HEADSCALE_SERVER> -L 9090:localhost:9090

Access the debug interface on your device by opening http://localhost:9090/debug/ in your web browser.

The access control of the debug interface supports the use of a debug key. Traffic is accepted if the path to a debug key is set via the environment variable TS_DEBUG_KEY_PATH and the debug key sent as value for debugkey parameter with each request.

openssl rand -hex 32 | tee debugkey.txt
export TS_DEBUG_KEY_PATH=debugkey.txt
headscale serve

Access the debug interface on your device by opening http://<IP_OF_HEADSCALE>:9090/debug/?debugkey=<DEBUG_KEY> in your web browser. The debugkey parameter must be sent with every request.

The debug endpoint expects traffic from localhost. A different debug IP address may be configured by setting the TS_ALLOW_DEBUG_IP environment variable before starting Headscale. The debug IP address is ignored when the HTTP header X-Forwarded-For is present.

export TS_ALLOW_DEBUG_IP=192.168.0.10       # IP address of your device
headscale serve

Access the debug interface on your device by opening http://<IP_OF_HEADSCALE>:9090/debug/ in your web browser.