Back Matter

Troubleshooting

Issues encountered with Tailscale for AI Fleets, and how to resolve them.

permission denied when SSHing into a Spark

The ACL policy does not grant your tailnet identity SSH access to the target machine. Work through this checklist:

  1. Is your Tailscale login in group:operators?
    Open the admin console → Access controls. Find the groups block. Your login email must appear there exactly as you authenticated with Tailscale.
  2. Does the target machine carry tag:spark?
    Admin console → Machines → find the Spark. If no tag appears, go to ⋯ → Edit ACL tags and apply tag:spark. Verify with:
    tailscale status --json | jq '.Peer[] | {name: .HostName, tags: .Tags}'
  3. Is Tailscale SSH enabled on the Spark?
    SSH into the Spark via another path (LAN or physical access) and check:
    tailscale debug prefs | grep -i runssh
    If "RunSSH": false, enable it:
    sudo tailscale up --ssh --advertise-tags=tag:spark
  4. Is the Spark visible on the tailnet?
    From your laptop: tailscale status. Both Spark hostnames should appear. If not, the Spark may have lost its connection — check sudo systemctl status tailscaled on the machine.

In our lab, spark-bundle2-1 returned permission denied while spark-bundle2-2 was accessible — the cause was that spark-bundle2-1 had not yet been tagged. Applying tag:spark in the admin console resolved it immediately.

Client/server version mismatch warning

You see something like:

Warning: client version "1.102.2-t3b69c7bd6-g..." != tailscaled server version "1.102.1-t3b69c7bd6-g..."

This is cosmetic. It appears when the tailscale CLI binary and the tailscaled background daemon are on slightly different patch versions — typically after a Homebrew update upgrades the CLI but the daemon has not restarted yet.

Fix on macOS:

sudo brew services restart tailscale

Fix on Linux:

sudo systemctl restart tailscaled

After restarting, both versions will match and the warning disappears. SSH and network connectivity are not affected while the warning is present.

Tailscale SSH is not enabled on this machine

Check the current SSH mode state:

tailscale debug prefs | grep -i runssh

If the output is "RunSSH": false, the machine is on the tailnet but will not accept inbound SSH connections through Tailscale. Enable it:

sudo tailscale up --ssh --advertise-tags=tag:spark

You do not need to re-authenticate. The command re-applies the --ssh flag to the existing connection.

Verify after:

tailscale debug prefs | grep -i runssh
# Expected: "RunSSH": true
A Spark is not appearing in tailscale status

If a machine that should be on the tailnet does not appear in tailscale status from another peer, it has lost its connection. On the Spark:

  1. Check the daemon:
    sudo systemctl status tailscaled
  2. If inactive, restart it:
    sudo systemctl restart tailscaled
    sudo tailscale up --ssh --advertise-tags=tag:spark
  3. If you see "needs login" or an authentication URL, the node key may have expired. Authenticate again via the printed URL — though note that tagged devices should not expire. If a tagged device is prompting for re-login, check that the tag is still applied in the admin console.
Error: tag:spark is not defined in the policy

The admin console rejects tag application when the tag is not declared in tagOwners. Open Access controls and confirm this block exists:

"tagOwners": {
  "tag:spark": ["autogroup:admin"]
},

Save the policy first, then return to Machines → Edit ACL tags and apply tag:spark. The error should not reappear once the tagOwners entry is present.

Machine enrolled in the wrong tailnet

If you opened the authentication URL in a browser signed into a personal Tailscale account instead of the fleet account, the machine joined the wrong tailnet. To fix it:

  1. Log out from the Spark:
    sudo tailscale logout
  2. Re-enroll with the fleet account:
    sudo tailscale up --ssh --advertise-tags=tag:spark
  3. Open the new URL in a browser signed into the correct fleet tailnet account.

After this, remove the stray device from your personal tailnet via the admin console at login.tailscale.com/admin/machines.