Chapter 5 · Part II

KubeRay for Distributed AI

KubeRay is the Kubernetes operator for Ray, an open-source framework for distributed Python workloads. It manages the lifecycle of RayClusters — head nodes, worker nodes, and autoscaling — as native Kubernetes resources. On DGX Spark Bundle, KubeRay bridges both nodes into a single Ray cluster so vLLM can split a large model across both GB10 GPUs using tensor parallelism.

Critical — ARM64 Architecture Warning
The standard Ray image (rayproject/ray) is x86-only and will fail on DGX Spark with "exec format error". Always use: 
nvcr.io/nvidia/vllm:25.09-py3
This image is ARM64-native with Ray 2.49.2, vLLM 0.10.1.1, and CUDA 13.0 built in.

Install KubeRay Operator

helm repo add kuberay https://ray-project.github.io/kuberay-helm/
helm repo update

helm install kuberay-operator kuberay/kuberay-operator \
  --namespace kuberay-system \
  --create-namespace \
  --set nodeSelector."kubernetes\\.io/hostname"=spark-720e
kubectl get pods -n kuberay-system
# kuberay-operator pod should show Running

NGC Registry Login

# On both Sparks
docker login nvcr.io
# Username: $oauthtoken
# Password: <YOUR_NGC_API_KEY>

HuggingFace Token Secret

kubectl create secret generic hf-token \
  --from-literal=token=$HF_TOKEN \
  -n core-services

Validate Cross-Node Networking

kubectl run test-spark1 \
  --image=busybox \
  --overrides='{"spec":{"nodeSelector":{"kubernetes.io/hostname":"spark-720e"}}}' \
  --command -- sleep 3600

kubectl run test-spark2 \
  --image=busybox \
  --overrides='{"spec":{"nodeSelector":{"kubernetes.io/hostname":"spark-7229"}}}' \
  --command -- sleep 3600

kubectl get pods -o wide  # get SPARK2_POD_IP
kubectl exec test-spark1 -- ping -c 4 <SPARK2_POD_IP>
kubectl delete pod test-spark1 test-spark2

Deploy RayCluster

kubectl apply -f - <<'EOF'
apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: vllm-cluster
  namespace: core-services
spec:
  rayVersion: '2.49.2'
  headGroupSpec:
    rayStartParams:
      dashboard-host: '0.0.0.0'
      num-gpus: '1'
    template:
      spec:
        nodeSelector:
          kubernetes.io/hostname: spark-720e
        containers:
        - name: ray-head
          image: nvcr.io/nvidia/vllm:25.09-py3
          command: ["/bin/bash", "-c"]
          args:
          - |
            ray start --head \
              --dashboard-host=0.0.0.0 \
              --num-gpus=1 \
              --block &
            sleep 30
            python3 -m vllm.entrypoints.openai.api_server \
              --model Qwen/Qwen2.5-7B-Instruct \
              --tensor-parallel-size 2 \
              --distributed-executor-backend ray \
              --host 0.0.0.0 \
              --port 8000 \
              --gpu-memory-utilization 0.85 \
              --max-num-seqs 4
          env:
          - name: HF_TOKEN
            valueFrom:
              secretKeyRef:
                name: hf-token
                key: token
          resources:
            limits:
              nvidia.com/gpu: "1"
              memory: "100Gi"
  workerGroupSpecs:
  - replicas: 1
    groupName: worker-group
    rayStartParams:
      num-gpus: '1'
    template:
      spec:
        nodeSelector:
          kubernetes.io/hostname: spark-7229
        containers:
        - name: ray-worker
          image: nvcr.io/nvidia/vllm:25.09-py3
          command: ["/bin/bash", "-c"]
          args:
          - |
            ray start \
              --address=vllm-cluster-head-svc.core-services.svc.cluster.local:6379 \
              --num-gpus=1 \
              --block
          resources:
            limits:
              nvidia.com/gpu: "1"
              memory: "100Gi"
EOF

Verify Ray Cluster

HEAD=$(kubectl get pods -n core-services -l ray.io/node-type=head -o name | head -1)
kubectl exec -n core-services $HEAD -- python3 -c \
  "import ray; ray.init(); print(ray.cluster_resources())"
# Expected: {'GPU': 2.0, 'CPU': 40.0, 'accelerator_type:GB10': 2.0}
Previous← Ch 4: k3s