起因是我发现我们客户在做GPU基准测试的时候，使用的是OrionX驱动科技的方案，并且它是优于原生的k8s调度方案的，那么有没有开源版本的类似的组件，可以让我来进行测试呢？
有的兄弟有的
GPU 虚拟化方案调研 [table]
“完整 GPU 虚拟化”需要替换 libcuda.so，自己做不现实
这是核心结论。要实现真正的显存隔离 + 算力比例控制，需要在 CUDA 驱动层做拦截（hook libcuda.so），这活只有趋动科技和 HAMi 这类专业项目才做得好。
NVIDIA 原生方案的局限
text 1 2 3 4 5 6 Time-Slicing: Pod A 用 10ms → Pod B 用 10ms → Pod A 用 10ms ... （轮流用，不是同时用，显存没隔离） MPS: Pod A 和 Pod B 真正同时跑 （并发了，但显存没限额，A 能把显存吃光） MPS + TS: 多组 Pod 轮流，每组内并发 （调度+执行都超分了，但显存还是没硬限制）
HAMi 的优势（相对其他方案）
[table]
为什么选 HAMi 做测试
开源免费，没有厂商锁定
CNCF 孵化项目，社区有保障
能做到显存限额 + 算力比例（MPS/TS 做不到）
K8s 原生集成，部署简单
是 OrionX 的直接开源替代
kind集群 单节点
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 networking: apiServerAddress: "0.0.0.0" apiServerPort: 6443 nodes: - role: control-plane extraMounts: - hostPath: /usr/lib/wsl/lib containerPath: /usr/lib/wsl/lib readOnly: true - hostPath: /dev containerPath: /dev - hostPath: /usr/lib/wsl/drivers containerPath: /usr/lib/wsl/drivers readOnly: true - hostPath: /proc/driver containerPath: /host/proc/driver readOnly: true kind create cluster --name gpu-test --config kind-gpu-config.yaml kubectl get nodes NAME STATUS ROLES AGE VERSION gpu-test-control-plane Ready control-plane 76m v1.32.2
环境准备 因为我使用的是笔记本（WSL2+kind）搭建单节点k8s集群，所以会需要做一些额外步骤
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 docker exec gpu-test-control-plane bash -c ' export http_proxy=http://192.168.10.238:7897 export https_proxy=http://192.168.10.238:7897 export no_proxy=deb.debian.org,debian.org rm -f /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg rm -f /etc/apt/sources.list.d/nvidia-container-toolkit.list curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --batch --yes --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \ sed "s#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g" | \ tee /etc/apt/sources.list.d/nvidia-container-toolkit.list apt-get update && apt-get install -y nvidia-container-toolkit ' docker exec gpu-test-control-plane nvidia-container-cli info NVRM version: 610.62 CUDA version: 13.3 Device Index: 0 Device Minor: 0 Model: NVIDIA GeForce RTX 4050 Laptop GPU Brand: GeForce GPU UUID: GPU-2b3d8a57-d275-92ab-c84a-638c4a872e45 Bus Location: 00000000:01:00.0 Architecture: 8.9
NVIDIA容器运行时 containerd针对于1.x版本和2.0版本的配置文件，容器运行时字段不同：
[table]
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 docker exec gpu-test-control-plane bash -c ' cat > /etc/containerd/conf.d/99-nvidia.toml << EOF [plugins."io.containerd.cri.v1.runtime".containerd] default_runtime_name = "nvidia" [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.nvidia] runtime_type = "io.containerd.runc.v2" [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.nvidia.options] BinaryName = "/usr/bin/nvidia-container-runtime" EOF cat > /etc/containerd/conf.d/98-runc.toml << EOF2 [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.runc] runtime_type = "io.containerd.runc.v2" EOF2 systemctl restart containerd ' docker exec gpu-test-control-plane crictl info 2>&1 | python3 -c " import json, sys d = json.load(sys.stdin) print(list(d['config']['containerd']['runtimes'].keys())) " ['nvidia' , 'runc' ]
Helm部署HAMi 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 helm repo add hami-charts https://project-hami.github.io/HAMi/ export https_proxy=http://192.168.10.238:7897export http_proxy=http://192.168.10.238:7897helm repo update helm search repo hami-charts NAME CHART VERSION APP VERSION DESCRIPTION hami-charts/hami 2.9.0 2.9.0 Heterogeneous AI Computing Virtualization Middl... helm pull hami-charts/hami --version 2.9.0 --untar ls hami/Chart.lock Chart.yaml README.md charts templates values.yaml resourceName: "nvidia.com/gpu" resourceMem: "nvidia.com/gpumem" resourceCores: "nvidia.com/gpucores" deviceSplitCount: 10 deviceMemoryScaling: 1 deviceCoreScaling: 1 cd hamihelm upgrade --install hami ./ -f values.yaml --create-namespace -n hami
验证：HAMi性能限制 故意超限，看会不会被拦住
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 kind load docker-image nvidia/cuda:12.0.0-base-ubuntu22.04 --name gpu-test cat > gpu-pod-test-a.yaml <<EOF apiVersion: v1 kind: Pod metadata: name: test-mem-5g spec: restartPolicy: Never containers: - name: test image: nvidia/cuda:12.0.0-base-ubuntu22.04 command: ["sh", "-c", "nvidia-smi && echo '=== 尝试分配 2GB 显存 ===' && python3 -c 'import ctypes; libcudart = ctypes.CDLL(\"libcudart.so\"); ptr = ctypes.c_void_p(); err = libcudart.cudaMalloc(ctypes.byref(ptr), 2*1024*1024*1024); print(f\"cudaMalloc 2GB: {\"成功\" if err == 0 else f\"失败 error={err}\"}\")' 2>&1 || echo 'python3 不可用，用 nvidia-smi 验证'; sleep 300"] resources: limits: nvidia.com/gpu: 1 nvidia.com/gpumem: 5000 nvidia.com/gpucores: 100 EOF cat > gpu-pod-test-b.yaml <<EOF apiVersion: v1 kind: Pod metadata: name: test-mem-1g spec: restartPolicy: Never containers: - name: test image: nvidia/cuda:12.0.0-base-ubuntu22.04 command: ["sh", "-c", "nvidia-smi && echo '=== 尝试分配 2GB 显存 ===' && python3 -c 'import ctypes; libcudart = ctypes.CDLL(\"libcudart.so\"); ptr = ctypes.c_void_p(); err = libcudart.cudaMalloc(ctypes.byref(ptr), 2*1024*1024*1024); print(f\"cudaMalloc 2GB: {\"成功\" if err == 0 else f\"失败 error={err}\"}\")' 2>&1 || echo 'python3 不可用，用 nvidia-smi 验证'; sleep 300"] resources: limits: nvidia.com/gpu: 1 nvidia.com/gpumem: 1000 nvidia.com/gpucores: 100 EOF kubectl apply -f gpu-pod-test-a.yaml kubectl apply -f gpu-pod-test-b.yaml
Pod A：拿到 1 个 vGPU，显存上限 5000MB → 占用100%
Pod B：拿到 1 个 vGPU，显存上限 1000MB → 占用100%
正常应该B拿不到，因为A已经占用100%
1 2 3 4 kubectl get pods -w NAME READY STATUS RESTARTS AGE test-mem-1g 0/1 Pending 0 99s test-mem-5g 1/1 Running 0 99s
查看kubectl describe pod test-mem-1g，可以看到是被hami拦下来了
1 2 3 4 Warning FailedScheduling 116s hami-scheduler 0/1 nodes are available: 1 NodeUnfitPod. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod. Warning FailedScheduling 84s (x2 over 115s) hami-scheduler 0/1 nodes are available: 1 NodeUnfitPod. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod. Warning FilteringFailed 26s (x6 over 117s) hami-scheduler 1 nodes CardInsufficientCore(gpu-test-control-plane) Warning FilteringFailed 26s (x6 over 117s) hami-scheduler no available node, 1 nodes do not meet
性能/业务基准测试 在开始测试之前，首先需要一个基线
[table]
场景 A — 独占模式
Pod 不设 gpumem/gpucores，独占整张卡。跑 gpu-burn 60 秒，记录 TFLOPS。
场景 B — 共享模式
Pod 设 gpumem: 3000 + gpucores: 50。跑同样的 gpu-burn 60 秒，记录 TFLOPS。
对比
[table]
↑以上指的是使用通用压测工具的思路，
但是实际生产中，会使用业务提供的脚本进行压测
