AIBrix AI Gateway

AIBrix is an open-source AI inference gateway from the vLLM project. It sits in front of one or more vLLM endpoints and adds request routing, per-namespace quota enforcement, agent lifecycle management, and GPU utilization-aware scheduling — turning a raw vLLM API server into a multi-tenant inference platform.

What AIBrix Provides

Step 1 — Install Dependencies

kubectl apply \
  -f https://github.com/vllm-project/aibrix/releases/download/v0.6.0/aibrix-dependency-v0.6.0.yaml \
  --server-side \
  --force-conflicts

Step 2 — Install AIBrix Core

kubectl apply \
  -f https://github.com/vllm-project/aibrix/releases/download/v0.6.0/aibrix-core-v0.6.0.yaml

Verify Components

kubectl get pods -n aibrix-system
# All 6 pods should show Running:
# aibrix-controller-manager
# aibrix-gateway-plugins
# aibrix-gpu-optimizer
# aibrix-kuberay-operator
# aibrix-metadata-service
# aibrix-redis-master

Register a Model (ModelAdapter)

kubectl apply -f - <<'EOF'
apiVersion: model.aibrix.ai/v1alpha1
kind: ModelAdapter
metadata:
  name: qwen-7b
  namespace: <your-app-namespace>
spec:
  modelName: Qwen/Qwen2.5-7B-Instruct
  replicas: 1
  podSelector:
    matchLabels:
      ray.io/node-type: head
EOF