AMD GPU Passthrough

Overview

This guide describes how to assign an AMD GPU directly to an OpenNebula Virtual Machine using PCI passthrough, install AMD ROCm in the guest, and validate the configuration with PyTorch and vLLM. While the VM is running, the assigned GPU is exclusively owned by the guest VM. The AMD GPU driver and ROCm user-space components are therefore installed in the guest, not on the Host.

The generic PCI passthrough configuration is described in the Host Configuration Guide. Complete the IOMMU and VFIO device ownership configuration before proceeding with this guide. The AMD-specific VFIO binding and PCI monitoring values are configured below.

The commands below reproduce a validated environment with the following software:

Refer to the ROCm documentation and the vLLM installation documentation before using a different software combination.

Requirements

Before continuing, verify that:

• The AMD GPU is installed and visible on the Host.
• The GPU is supported by the ROCm release that will be installed in the guest. Refer to the ROCm compatibility matrix.
• IOMMU and VFIO device ownership are configured on the Host.
• The Host uses QEMU 8.2.2, or another QEMU version verified to expose the required PCIe atomic capabilities, and a Q35 machine type is available.
• An Ubuntu 24.04 Image with Python 3.12 is available.
• The Virtual Machine disk has at least 200 GB of capacity for ROCm, the Python environments, the 120B MXFP4 model, and operational headroom. The selected datastore must have enough capacity to provision this disk.
• The network selected for the Virtual Machine provides Internet access.

Configure the Host

Verify the QEMU Version

The virtual PCIe Root Port must advertise support for 32-bit, 64-bit, and 128-bit compare-and-swap atomic operations. The QEMU 6.2 packages originally supplied by Ubuntu 22.04 did not expose the required capabilities in the validated environment. QEMU 8.2.2 resolved the issue. Other QEMU versions must be verified independently.

Check the QEMU binary used by OpenNebula:

readlink -f /usr/bin/qemu-kvm-one
/usr/bin/qemu-kvm-one --version

On the validated Ubuntu 22.04 Host, QEMU 8.2.2 was installed from the Caracal Proposed pocket of the Ubuntu Cloud Archive:

sudo apt-get update
sudo apt-get install -y ubuntu-cloud-keyring software-properties-common
sudo add-apt-repository cloud-archive:caracal
sudo add-apt-repository cloud-archive:caracal-proposed
sudo apt-get update
sudo apt-get --simulate -t jammy-proposed/caracal install \
  qemu-system-x86 qemu-system-common \
  qemu-system-data qemu-utils qemu-block-extra
sudo apt-get install -y -t jammy-proposed/caracal \
  qemu-system-x86 qemu-system-common \
  qemu-system-data qemu-utils qemu-block-extra
/usr/bin/qemu-kvm-one --version

After installing the required packages, disable the Proposed pocket to prevent unrelated pre-release packages from being selected during future upgrades:

sudo add-apt-repository --remove cloud-archive:caracal-proposed
sudo apt-get update

Identify the AMD GPU Devices

List the installed AMD devices:

lspci -nn -d 1002:

Example output from the validated hardware:

05:00.0 Processing accelerators [1200]: Advanced Micro Devices, Inc. [AMD/ATI] Aqua Vanjaram [Instinct MI325X] [1002:74a5]

Record the identifiers reported for each device intended for passthrough. For the validated hardware, these are:

Verify the IOMMU group of every device intended for passthrough. Replace the example address with the address from your Host:

gpu=0000:05:00.0
group=$(basename "$(readlink "/sys/bus/pci/devices/$gpu/iommu_group")")

for device in /sys/kernel/iommu_groups/"$group"/devices/*; do
  lspci -nnk -s "$(basename "$device")"
done

Each assigned GPU must be isolated in a viable IOMMU group. Inspect every member of a group before binding it to VFIO.

Prevent Host Driver Binding

On a Host dedicated to AMD GPU passthrough, prevent amdgpu from claiming the accelerators. Append the following parameters to the Host kernel command line:

module_blacklist=amdgpu modprobe.blacklist=amdgpu

Also create /etc/modprobe.d/blacklist-amdgpu.conf with the following contents:

blacklist amdgpu

Regenerate the GRUB configuration and initramfs, then reboot the Host:

sudo update-grub
sudo update-initramfs -u
sudo reboot

Bind the AMD GPU to VFIO

Install driverctl, load VFIO, and bind the selected GPU. Replace 05:00.0 with the address from your Host:

sudo apt-get install -y driverctl
sudo modprobe vfio
sudo modprobe vfio-pci
sudo driverctl set-override 0000:05:00.0 vfio-pci

The modprobe commands load the VFIO modules only for the current Host boot. To load them automatically after every reboot, create /etc/modules-load.d/vfio.conf with the following contents:

vfio
vfio-pci

The driverctl set-override command shown above stores a persistent override and restores the vfio-pci binding after a Host reboot. For a temporary test that lasts only until the next reboot, use --nosave instead:

sudo driverctl --nosave set-override 0000:05:00.0 vfio-pci

Verify the binding:

lspci -nnk -s 05:00.0

The output must contain:

Kernel driver in use: vfio-pci

Configure OpenNebula Monitoring

On the Front-end, add the AMD vendor ID to the filter list in /var/lib/one/remotes/etc/im/kvm-probes.d/pci.conf. Restrict the configuration to the intended PCI addresses or use the device and class identifiers reported by lspci:

filter:
  - "1002:74a5:1200"

short_address:
  - "05:00.0"

The example restricts discovery to the GPU model and address used in the validated environment. Replace the vendor, device, class, and address values with those reported by your Host. To expose multiple GPUs for automatic selection, bind every exposed device to vfio-pci before deployment.

As oneadmin, synchronize the monitoring probes and request a new monitoring cycle:

onehost sync --force
onehost forceupdate <host>

Verify that OpenNebula discovered the devices:

onehost show <host>

The AMD GPUs must appear in the PCI Devices section.

Deploy the Virtual Machine

Add the following attributes to the Virtual Machine Template to assign a specific AMD GPU. Merge the MODEL and MACHINE values into existing CPU_MODEL and OS attributes when they are already defined:

CPU_MODEL = [
  MODEL = "host-passthrough"
]

OS = [
  MACHINE = "q35"
]

PCI = [
  SHORT_ADDRESS = "05:00.0"
]

Replace the PCI address with the value from your environment. Size the CPU, memory, and disk according to the workload and the requirements listed above.

For automatic device selection among multiple prepared GPUs, request the GPU by its vendor, device, and class identifiers instead. The following example uses the identifiers from the validated environment; replace them with the values reported by lspci:

PCI = [
  VENDOR = "1002",
  DEVICE = "74a5",
  CLASS  = "1200"
]

Repeat the PCI section to assign multiple AMD GPUs to the same Virtual Machine. Multi-GPU workloads see separate ROCm devices; frameworks such as vLLM can combine them using tensor, pipeline, or data parallelism. Validate peer access and RCCL before using a multi-GPU configuration.

Do not use automatic selection unless every matching GPU exposed by PCI monitoring has been prepared for passthrough and bound to vfio-pci.

Configure the Guest

After deploying the Virtual Machine, verify that the guest VM can see the assigned device. Run the following command on the command line of the guest VM (using onevm ssh ):

lspci -nnk -d 1002:

Install the AMD GPU Driver

Download and install the ROCm 7.2.4 repository package:

curl -fLO https://repo.radeon.com/amdgpu-install/7.2.4/ubuntu/noble/amdgpu-install_7.2.4.70204-1_all.deb
sudo apt-get install -y ./amdgpu-install_7.2.4.70204-1_all.deb
sudo apt-get update

Install the kernel headers, extra kernel modules, and AMD GPU driver:

sudo apt-get install -y "linux-headers-$(uname -r)" \
  "linux-modules-extra-$(uname -r)" amdgpu-dkms
dkms status

Reboot the guest operating system without stopping the OpenNebula Virtual Machine:

sudo reboot

After reconnecting, verify that amdgpu manages the device and that the KFD device exists:

lspci -nnk -d 1002:
id
ls -l /dev/kfd /dev/dri/renderD*

The user that runs ROCm applications must belong to the render and video groups. Commands run from a root shell do not require this membership. If either group is missing from the id output above, add it with:

sudo usermod -a -G render,video "$USER"

Group membership configured with usermod is persistent, but the user must start a new login session before it takes effect.

Verify PCIe Atomic Operations

Identify the PCIe Root Port immediately above the assigned GPU:

lspci -t

Inspect its PCIe capabilities, replacing 00:02.0 if necessary:

sudo lspci -vv -s 00:02.0 | grep -E "AtomicOpsCap|AtomicOpsCtl"

The Root Port must advertise 32-bit, 64-bit, and 128-bit compare-and-swap atomic completer support. For example:

AtomicOpsCap: Routing- 32bit+ 64bit+ 128bitCAS+

Routing- is expected for a Root Port directly connected to the assigned endpoint. The 32bit+, 64bit+, and 128bitCAS+ fields are the relevant capabilities.

Verify that the driver did not reject the device:

sudo dmesg | grep -E "amdgpu|PCIE atomic ops"

The output must not contain PCIE atomic ops is not supported.

Install the ROCm Runtime

Install the ROCm compute and machine-learning libraries:

sudo apt-get install -y rocminfo amd-smi-lib rocm-hip-runtime \
  roctracer rocm-ml-libraries

Register the ROCm library path with the dynamic linker:

printf '%s\n' /opt/rocm/lib | sudo tee /etc/ld.so.conf.d/rocm.conf
sudo ldconfig

The file in /etc/ld.so.conf.d/ makes the ROCm library path persistent. Running only with LD_LIBRARY_PATH=/opt/rocm/lib would affect the current command or shell session instead.

Verify the GPU with ROCm and AMD SMI:

/opt/rocm/bin/rocminfo
/opt/rocm/bin/amd-smi list
/opt/rocm/bin/amd-smi metric

rocminfo must report an agent for every assigned GPU, with the architecture and marketing name expected for the hardware. In the validated environment, it reported gfx942 and AMD Instinct MI325X.

Validate PyTorch (Optional)

This section provides an optional functional test of ROCm and the assigned GPU. Skip it if you intend to use only vLLM, which installs its own PyTorch build in a separate environment.

Install Python virtual environment support and create an isolated environment:

sudo apt-get install -y python3.12-venv
sudo python3.12 -m venv --clear /opt/pytorch-rocm
sudo chown -R "$USER":"$(id -gn)" /opt/pytorch-rocm
/opt/pytorch-rocm/bin/python -m pip install --upgrade pip setuptools wheel

Install the validated AMD Triton and PyTorch wheels:

/opt/pytorch-rocm/bin/pip install \
  'https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2.4/triton-3.5.1%2Brocm7.2.4.gita272dfa8-cp312-cp312-linux_x86_64.whl' \
  'https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2.4/torch-2.9.1%2Brocm7.2.4.lw.git39497456-cp312-cp312-linux_x86_64.whl'

Run a matrix multiplication on the GPU:

/opt/pytorch-rocm/bin/python - <<'PY'
import torch

print("PyTorch:", torch.__version__)
print("ROCm:", torch.version.hip)
print("GPU available:", torch.cuda.is_available())
print("GPU count:", torch.cuda.device_count())
print("GPU:", torch.cuda.get_device_name(0))

a = torch.randn((4096, 4096), device="cuda", dtype=torch.float16)
b = torch.randn((4096, 4096), device="cuda", dtype=torch.float16)
c = a @ b
torch.cuda.synchronize()

print("Result:", c.shape, c.dtype)
print("Finite:", torch.isfinite(c).all().item())
print("Allocated memory:", torch.cuda.memory_allocated() // 1024**2, "MiB")
PY

The test must report the expected number of assigned AMD GPUs, a finite result, and memory allocated on the GPU. The validated configuration assigned one GPU.

Install and Validate vLLM

The vLLM ROCm wheel includes a different, binary-compatible PyTorch build. Install it in a separate environment. If you completed the optional PyTorch validation above, do not modify /opt/pytorch-rocm.

The rocm723 suffix in the validated vLLM version identifies the ROCm version used to build the wheel. Although it differs from the ROCm 7.2.4 runtime installed above, this exact combination was validated in the environment described by this guide.

Install the required system libraries:

sudo apt-get install -y libopenmpi3t64 rocprofiler-sdk hsa-amd-aqlprofile
sudo ldconfig

Create the environment and install the vLLM build validated by this guide:

sudo python3.12 -m venv /opt/vllm-rocm
sudo chown -R "$USER":"$(id -gn)" /opt/vllm-rocm
/opt/vllm-rocm/bin/python -m pip install --upgrade uv

/opt/vllm-rocm/bin/uv pip install \
  --python /opt/vllm-rocm/bin/python \
  --pre 'vllm==0.28.1rc1.dev516+g9ea8f3ffc.rocm723' \
  --extra-index-url https://wheels.vllm.ai/rocm/9ea8f3ffc354901b740f0b31988900897b7221d7/rocm723 \
  --index-strategy unsafe-best-match

Check the installed packages and GPU platform:

/opt/vllm-rocm/bin/uv pip check --python /opt/vllm-rocm/bin/python

/opt/vllm-rocm/bin/python - <<'PY'
import torch
import vllm

print("vLLM:", vllm.__version__)
print("PyTorch:", torch.__version__)
print("ROCm:", torch.version.hip)
print("GPU available:", torch.cuda.is_available())
print("GPU:", torch.cuda.get_device_name(0))
print("Platform:", type(vllm.platforms.current_platform).__name__)
PY

The platform must be RocmPlatform and the expected AMD GPU name must be displayed. The validated output reported AMD Instinct MI325X.

Serve a Model

The following command downloads and serves the official openai/gpt-oss-120b MXFP4 model. The model requires approximately 65 GB of disk space and at least 80 GB of GPU memory:

sudo install -d -o "$USER" -g "$(id -gn)" /opt/huggingface

HF_HOME=/opt/huggingface \
/opt/vllm-rocm/bin/vllm serve openai/gpt-oss-120b \
  --host 127.0.0.1 \
  --port 8000 \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.80 \
  --max-num-seqs 64 \
  --language-model-only

Setting HF_HOME before a command affects only that command. To use the same model cache in future interactive sessions, add the following line to the user’s shell startup file, such as ~/.profile:

export HF_HOME=/opt/huggingface

The server command runs in the foreground and stops when its process or terminal session ends.

After the API server is ready, submit a request from another guest shell:

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "openai/gpt-oss-120b",
    "messages": [{
      "role": "user",
      "content": "Briefly explain the problem that OpenNebula solves."
    }],
    "temperature": 0,
    "max_tokens": 256
  }' | /opt/vllm-rocm/bin/python -m json.tool

Abbreviated sample output, with the response text shortened:

{
    "model": "openai/gpt-oss-120b",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "**OpenNebula\u2019s core problem:**\u202fHow to turn a collection of physical servers, storage, and networking resources into a reliable, easy\u2011to\u2011manage, on\u2011demand cloud that can serve many users and workloads without requiring a full\u2011blown public\u2011cloud stack.\n\n..."
            },
            "finish_reason": "length"
        }
    ],
    "usage": {
        "prompt_tokens": 78,
        "total_tokens": 334,
        "completion_tokens": 256
    }
}

Run a Small Serving Benchmark

With the server still running, submit a reproducible synthetic workload:

HF_HOME=/opt/huggingface \
/opt/vllm-rocm/bin/vllm bench serve \
  --backend vllm \
  --base-url http://127.0.0.1:8000 \
  --model openai/gpt-oss-120b \
  --dataset-name random \
  --num-prompts 32 \
  --random-input-len 512 \
  --random-output-len 128 \
  --ignore-eos \
  --request-rate inf \
  --max-concurrency 8 \
  --seed 42

Sample output:

============ Serving Benchmark Result ============
Successful requests:                     32
Failed requests:                         0
Maximum request concurrency:             8
Benchmark duration (s):                  10.35
Total input tokens:                      16384
Total generated tokens:                  4096
Request throughput (req/s):              3.09
Output token throughput (tok/s):         395.69
Peak output token throughput (tok/s):    664.00
Peak concurrent requests:                16.00
Total token throughput (tok/s):          1978.47
---------------Time to First Token----------------
Mean TTFT (ms):                          975.68
Median TTFT (ms):                        499.99
P99 TTFT (ms):                           2271.63
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          12.68
Median TPOT (ms):                        12.37
P99 TPOT (ms):                           15.93
---------------Inter-token Latency----------------
Mean ITL (ms):                           12.68
Median ITL (ms):                         12.40
P99 ITL (ms):                            14.00
==================================================

The validated single-GPU configuration completed all 32 requests without errors and produced approximately 396 output tokens per second. Results depend on the model, vLLM revision, VM configuration, and Host load.

Troubleshooting

PCIe Atomic Operations Are Not Supported

The guest driver fails to initialize the AMD GPU and the kernel log contains:

amdgpu: PCIE atomic ops is not supported

Verify that the VM uses Q35 and inspect the atomic operation capabilities of the Root Port above the GPU. If the completer capabilities are absent, use a supported QEMU package that exposes them. QEMU 8.2.2 was validated for this guide.

PyTorch Exits with Illegal Instruction

If import torch exits with Illegal instruction, inspect the Host or guest kernel log. In the validated environment, the fault occurred in libhipsparselt.so because the VM used the generic qemu64 CPU model, which did not expose AVX or AVX2.

Set the following VM attribute:

CPU_MODEL = [
  MODEL = "host-passthrough"
]

Restart the Virtual Machine for this change to take effect.

The GPU Disappears After Restarting the Virtual Machine

On the validated multi-GPU platform, the assigned GPU occasionally stopped responding after the Virtual Machine was stopped and started again. This behavior may be specific to the hardware topology, where the GPUs had previously been joined, and should not be assumed to affect other AMD GPU configurations.

The observed symptoms were:

• The Host reported the assigned GPU with PCI revision ff.
• The GPU was absent from the guest lspci output.
• /dev/kfd and the DRM render nodes were absent.
• PyTorch reported RuntimeError: No HIP GPUs are available.
• The Host log contained not ready after bus reset followed by giving up.

If these symptoms occur, power off the Virtual Machine and reboot the Host during a maintenance window. Avoid repeatedly rebinding, removing, rescanning, or resetting the affected GPU before consulting the hardware vendor.

A ROCm Shared Library Is Missing

If PyTorch fails with a missing ROCm library such as libMIOpen.so.1, verify that rocm-ml-libraries is installed and /opt/rocm/lib is registered with the dynamic linker.

The vLLM PyTorch wheel also requires MPI and profiling libraries from the guest operating system. Install the packages shown in the vLLM section if errors mention libmpi_cxx.so.40, librocprofiler-sdk.so.1, or libhsa-amd-aqlprofile64.so.1.