Chapter 4 · Part I

Kubernetes Cluster with k3s

k3s is a lightweight, production-grade Kubernetes distribution by Rancher (SUSE). It packages the full Kubernetes control plane into a single binary under 100 MB, stripping components unnecessary for edge and small-cluster deployments. k3s is ideal for DGX Spark Bundle: you get complete Kubernetes API compatibility with minimal system overhead, leaving the bulk of CPU and memory free for GPU workloads.

Install k3s on Spark 1 (Master)

curl -sfL https://get.k3s.io | \
  INSTALL_K3S_EXEC="--write-kubeconfig-mode 644 --disable=traefik" \
  sh -

Configure kubectl

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
echo "export KUBECONFIG=/etc/rancher/k3s/k3s.yaml" >> ~/.bashrc

Get the Join Token

sudo cat /var/lib/rancher/k3s/server/node-token

Join Spark 2 as Worker

curl -sfL https://get.k3s.io | \
  K3S_URL=https://192.168.86.30:6443 \
  K3S_TOKEN=<TOKEN_FROM_SPARK1> \
  sh -

Assign Worker Role Label

kubectl label node spark-7229 node-role.kubernetes.io/worker=worker

Verify Cluster

kubectl get nodes
# Expected:
# NAME         STATUS   ROLES                  VERSION
# spark-720e   Ready    control-plane,master   v1.35.5+k3s1
# spark-7229   Ready    worker                 v1.35.5+k3s1

Install Helm

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
helm version

NVIDIA GPU Operator

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace
# Verify both GPUs visible
kubectl get nodes -o json | grep '"nvidia.com/gpu":'
# Expected: "nvidia.com/gpu": "1"  (one per node)

kubectl get nodes -o json | grep "nvidia.com/gpu.family"
# Expected: "nvidia.com/gpu.family": "blackwell"

Create Namespaces

kubectl create namespace core-services
kubectl create namespace monitoring
Previous← Ch 3: CUDA Updates