How to Run Isolated Tenant Kubernetes Clusters on Shared GPU Infrastructure

This post provides a pattern that preserves team autonomy without splitting the hardware. This solution involves a single control plane cluster with a GPU pool, GPU sharing with per-team quotas, and isolated Kubernetes control plane per team including an API server, controller, data store, syncer, and scheduler. This can be achieved using two open source tools: KAI Scheduler and vCluster.

Follow along with the tutorial steps and you’ll have three teams running real GPU Kubernetes pods in their own tenant clusters, all sharing a single physical GPU. You’ll also be able to verify that each team only sees their own workloads.

Tutorial prerequisites and notes[](#tutorial_prerequisites_and_notes)

To keep the process reproducible for users that have limited resources, this tutorial uses a cluster with one NVIDIA L40S GPU and three teams sharing fractions of it. This makes the moving parts easy to see and try. The process works the same way on a larger cluster with hundreds of GPU nodes and dozens of teams. You can scale the node pool, the queue hierarchy, and the number of tenant clusters.

KAI Scheduler is a robust, efficient, and scalable topology-aware Kubernetes scheduler that was purpose-built for optimizing GPU resource allocation for AI workloads. It’s designed to manage large-scale GPU clusters, including thousands of nodes, and a high throughput of workloads. With KAI Scheduler, you can dynamically allocate GPU resources to workloads. It can run alongside the default kube-scheduler. Any pod with schedulerName: kai-scheduler is handled by KAI Scheduler. Everything else goes through the usual kube-scheduler process.

The vCluster Kubernetes platform provisions fully isolated tenant clusters on your infrastructure or directly on bare metal. Each tenant cluster gets its own API server, custom resource definitions (CRDs), and role-based access control (RBAC), indistinguishable from a dedicated Kubernetes cluster, while sharing the underlying nodes and hardware. The virtualized control plane is invisible to tenants: no shared control plane nodes, no in-cluster agent pods, and no lateral path between environments. That makes vCluster a natural fit for GPU infrastructure where teams need their own clean cluster experience without splitting the hardware.

The vCluster shared-nodes model is used for this tutorial, so teams share the GPU node while each gets its own isolated control plane, the right fit for trusted internal teams. For untrusted tenants needing node-, network-, and storage-level separation, the same pattern extends to vCluster private nodes.

The example in this post uses three teams: NLP Team, Vision Team, and Recommender System Team. The NLP Team wants to install their own CRDs. The Vision Team wants cluster-admin to debug scheduling. The Recommender Team is on a different Kubeflow version. Nobody wants to share a kubectl context and accidentally break one another’s environments.

Using vCluster, each team gets their own isolated control plane, RBAC, namespaces, and CRDs. They can each have cluster-admin access. Underneath, all tenant clusters share the same nodes and GPUs.

Demo environment[](#demo_environment)

This demo runs on an NVIDIA Brev GPU instance on Nebius with:

• One NVIDIA L40S, 40 vCPUs, 160 GiB RAM, 256 GiB disk (48 GB VRAM)

• Ubuntu 24.04.4 LTS

• MicroK8s v1.36.2 – Kubernetes was preconfigured by Brev, including the MicroK8s gpu addon, which pre-installs the NVIDIA GPU Operator into the gpu-operator-resources namespace

• KAI Scheduler v0.16.4

• vCluster CLI 0.35.1

Note: For different setups, the cluster-creation and GPU Operator install steps will differ on GKE/EKS/AKS/vanilla k8s/k3s. Step 3 (KAI Scheduler) onward is identical on any Kubernetes that has the NVIDIA GPU Operator running with Container Device Interface (CDI) enabled.

Figure 1. KAI Scheduler and vCluster architecture on a shared Nebius host cluster