Chapter 1 · Part I — Fleet Setup

Introduction: The Networking Problem

You have two NVIDIA DGX Spark machines. They are powerful, quiet, and ready to run frontier-scale AI workloads. The only question is: how do you — and your team — actually get into them?

The classic answer is SSH keys distributed to everyone who needs access, firewall rules punched through for each machine, and a growing spreadsheet of who has which key on which laptop. That works fine for one machine and one person. For a small AI fleet accessed by multiple engineers, it becomes operational debt almost immediately.

The Networking Problem with a DGX Fleet

A two-node DGX Spark setup has a deceptively simple surface area, but the networking headaches compound quickly:

  • Static IPs change. If your lab network changes, every SSH config and firewall rule must be updated.
  • Key rotation is manual. When someone leaves, you must log into every machine and remove their key — a step that is easy to miss under pressure.
  • VPN complexity. Traditional VPNs introduce a hub-and-spoke bottleneck; all traffic routes through a central server even when two peers are on the same LAN.
  • No audit trail. Vanilla SSH gives you auth logs, but not a centralized view of who can reach what.
The invisible risk
The worst outcome is not a breach — it is a former team member whose key was never removed, silently retaining access to machines running proprietary models for months.

What You Want Instead

You want a system where:

  • Any authorized engineer can SSH into any Spark from any device — laptop, desktop, or cloud VM — without exchanging keys manually.
  • Access is controlled by identity, not by which machine the engineer happens to be on.
  • Revoking access for one person immediately and completely is a single admin action.
  • Adding a new Spark to the fleet automatically falls under the same access rules as every existing one.

That is exactly what Tailscale delivers.

How Tailscale Works

Tailscale builds a tailnet — a private WireGuard mesh network — across all your enrolled devices. WireGuard handles the encrypted tunnels; Tailscale handles the key exchange and peer discovery so you never have to configure WireGuard directly.

🔐
WireGuard
Modern, audited VPN protocol. All traffic between tailnet peers is encrypted end-to-end with WireGuard tunnels. Tailscale manages the key rotation automatically.
kernel module
🆔
Identity
Each user authenticates with an existing identity provider (Google, GitHub, Microsoft, etc.). No new credentials to create or manage.
OAuth / OIDC
📋
ACL Policy
A single JSON file in the admin console declares who can reach what. Groups, tags, and SSH rules are all expressed here. The policy is enforced on every node.
HuJSON
🌐
MagicDNS
Every device on the tailnet gets a stable hostname (e.g. spark-bundle2-2.tail8a84f6.ts.net) that resolves to its Tailscale IP regardless of where the machine physically is.
100.x.x.x range

Tailscale SSH

The feature used in this runbook is Tailscale SSH. When you start Tailscale with --ssh, it intercepts port 22 on the tailnet interface and replaces password/key authentication with tailnet identity checks. The ACL policy is the only authorization mechanism — no ~/.ssh/authorized_keys required.

No keys, no passwords
After setup, SSH access looks like: ssh stoke@spark-bundle2-2.tail8a84f6.ts.net. No key to specify, no password prompt. The only question the system asks is "is this device on the tailnet and does the ACL allow it to SSH here?" If yes, you're in.

Architecture Overview

The complete setup has three layers of actors: machines, users, and policy.

┌──────────────────────────────────────────────────────────────────────────────┐ │ TAILNET │ │ │ │ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌──────────────────┐ │ │ │ spark-bundle1-1 │ │ spark-bundle1-2 │ │ spark-bundle2-1 │ │ spark-bundle2-2 │ │ │ │ 100.109.xx.xx │ │ 100.104.xx.xx │ │ 100.70.xx.xx │ │ 100.67.xx.xx │ │ │ │ tag:spark │ │ tag:spark │ │ tag:spark │ │ tag:spark │ │ │ │ TS SSH ✓ │ │ TS SSH ✓ │ │ TS SSH ✓ │ │ TS SSH ✓ │ │ │ └────────┬────────┘ └────────┬────────┘ └────────┬────────┘ └────────┬─────────┘ │ │ └───────────────────┴──────────────────────┴──────────WireGuard┘ │ │ ▲ │ │ ACL: group:operators may SSH → tag:spark │ │ as user: stoke │ │ │ │ │ ┌───────────────────────────────────────────────────────────────────────────────────┐ │ │ │ macOS laptop 100.124.xx.xx mohnishbasha@ (group:operators member) │ │ │ └───────────────────────────────────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────┘

Reading this diagram:

  • Both Spark machines carry the ACL tag tag:spark. One tag rule in the policy covers both — and every future Spark you enroll.
  • The engineer's laptop is enrolled as a regular device under their Tailscale login. It is automatically a member of group:operators once their login is added to that group in the ACL.
  • All three devices communicate over encrypted WireGuard tunnels within the tailnet. No traffic goes through a central server if they can reach each other directly (NAT traversal is handled automatically).

Roles in this Setup

RoleWhoWhat they can do
Tailnet AdminAccount ownerInvite users, edit ACL policy, manage machine tags
OperatorEngineers in group:operatorsSSH into any tag:spark machine as stoke
Spark machineDGX Spark nodesAccept SSH from authorized operators via tailnet identity

Setup Sequence

StepWhereChapter
1. Install Tailscale & enable SSH on each SparkTerminal on each SparkCh 2
2. Apply tag:spark to each machineTailscale admin consoleCh 2
3. Invite operators to the tailnetTailscale admin consoleCh 3
4. Define groups, tags, and SSH rule in ACLTailscale admin consoleCh 3
5. Install Tailscale on operator laptops, loginEach operator's machineCh 4
6. SSH into Sparks via IP or MagicDNS hostnameAny operator deviceCh 4