Chapter 8 · Part II
Cluster Overview and Monitoring
Prometheus is an open-source metrics collection and alerting system that scrapes time-series data from targets such as the NVIDIA GPU Operator and Kubernetes node exporters. Grafana is the visualization layer that queries Prometheus and renders real-time dashboards. Together they form the standard Kubernetes observability stack, deployed here via the kube-prometheus-stack Helm chart.
Install Monitoring Stack
helm repo add prometheus-community \
https://prometheus-community.github.io/helm-charts
helm repo update
helm install monitoring prometheus-community/kube-prometheus-stack \
--namespace monitoringAccessing Grafana
# Get admin password
kubectl --namespace monitoring get secrets monitoring-grafana \
-o jsonpath="{.data.admin-password}" | base64 -d; echo
# Port-forward
kubectl --namespace monitoring port-forward \
$(kubectl --namespace monitoring get pod \
-l "app.kubernetes.io/name=grafana" -o name) 3000
# Open: http://localhost:3000 Login: admin / <password above>GPU Metrics
| Metric | Description |
|---|---|
DCGM_FI_DEV_GPU_UTIL | GPU compute utilization (%) |
DCGM_FI_DEV_FB_FREE | Free GPU frame buffer (MiB) |
DCGM_FI_DEV_FB_USED | Used GPU frame buffer (MiB) |
DCGM_FI_DEV_POWER_USAGE | GPU power draw (Watts) |
DCGM_FI_DEV_GPU_TEMP | GPU temperature (°C) |
Full Cluster Status
| Layer | Component | Status |
|---|---|---|
| Hardware | 2× DGX Spark (GB10, 128GB each) | Connected via QSFP |
| OS | DGX OS — Ubuntu 24.04.4 LTS | CUDA 13.0 |
| Kubernetes | k3s v1.35.5 — 2 nodes Ready | Flannel CNI |
| GPU | NVIDIA GPU Operator | 2 GPUs visible |
| Distributed | KubeRay + Ray 2.49.2 | Head + Worker across nodes |
| Inference | vLLM 0.10.1.1 | Serving on port 8000 |
| Gateway | AIBrix v0.6.0 | Routing + multi-tenancy |
| Monitoring | Prometheus + Grafana | GPU metrics via DCGM |