Axelera GPU Passthrough

Overview

This guide describes how to assign an Axelera Metis AI Processing Unit (AIPU) directly to an OpenNebula Virtual Machine using PCI passthrough. The device is exclusively owned by the guest while the Virtual Machine is running; the Axelera driver and Voyager SDK are therefore installed in the guest, not on the Host.

The generic PCI passthrough configuration, including IOMMU, VFIO device ownership, and PCI monitoring, is described in the Host Configuration Guide. Complete the Host configuration guide before commencing this guide.

The installation commands below reproduce a validated environment using Ubuntu 24.04, metis-dkms 1.5.7, and the Axelera runtime and development kit 1.6.0. Refer to the Voyager SDK installation guide before using a different software combination.

Requirements

Before continuing, verify that:

The Metis AIPU is installed and visible on the Host.
IOMMU and VFIO are configured on the Host.
The OpenNebula PCI monitoring probe is configured.
An Ubuntu 22.04 or Ubuntu 24.04 Image is available.
The Virtual Machine disk has at least 30 GB of capacity. The selected datastore must have enough capacity to provision this disk.
The network selected for the Virtual Machine provides Internet access.

Configure the Host

Identify and Bind the Metis Devices

List the Metis devices installed on the Host:

lspci -d 1f9d:

Example output:

63:00.0 Processing accelerators: Axelera AI Metis AIPU (rev 02)
64:00.0 Processing accelerators: Axelera AI Metis AIPU (rev 02)
65:00.0 Processing accelerators: Axelera AI Metis AIPU (rev 02)
66:00.0 Processing accelerators: Axelera AI Metis AIPU (rev 02)

Note

Systems with an outdated PCI ID database may display the devices using only their numeric identifiers:

e3:00.0 Processing accelerators [1200]: Device [1f9d:1100]
e4:00.0 Processing accelerators [1200]: Device [1f9d:1100]
e5:00.0 Processing accelerators [1200]: Device [1f9d:1100]
e6:00.0 Processing accelerators [1200]: Device [1f9d:1100]

Update the database and then run the above lspci command again:

sudo update-pciids

Both outputs represent the same 1f9d:1100 device. The -nn option can be used to display the numeric class and device IDs together with the device name.

Bind each device intended for passthrough to vfio-pci. For example:

driverctl set-override 0000:e3:00.0 vfio-pci
driverctl set-override 0000:e4:00.0 vfio-pci
driverctl set-override 0000:e5:00.0 vfio-pci
driverctl set-override 0000:e6:00.0 vfio-pci

After setting the overrides, verify the binding on the Host before deploying a Virtual Machine:

lspci -nnk -d 1f9d:

In the output of this command, every device intended for passthrough must report vfio-pci as the kernel driver in use. For example:

63:00.0 Processing accelerators [1200]: Axelera AI Metis AIPU (rev 02) [1f9d:1100]
        Kernel driver in use: vfio-pci

Configure OpenNebula Monitoring

On the Front-end, add the Axelera vendor ID to the filter list in /var/lib/one/remotes/etc/im/kvm-probes.d/pci.conf:

filter:
  - "1f9d:*"

Synchronize the monitoring probes and request a new monitoring cycle:

onehost sync --force
onehost flush <host>

Verify that OpenNebula discovered the devices:

onehost show <host>

The Metis devices must appear in the PCI Devices section. Device details may take more than one monitoring cycle to become available.

Deploy the Virtual Machine

The following excerpt shows the relevant Virtual Machine Template attributes for a specific Metis device:

CPU_MODEL = [
  MODEL = "host-passthrough"
]

DISK = [
  IMAGE = "Ubuntu 24.04",
  SIZE  = "30720"
]

PCI = [
  SHORT_ADDRESS = "e3:00.0"
]

NIC = [
  NETWORK = "host-only"
]

Replace the Image, network, and PCI address with values from your environment. The SIZE value is expressed in MB and expands the guest disk to 30 GB.

For automatic device selection, request a device by its vendor, device, and class IDs instead:

PCI = [
  VENDOR = "1f9d",
  DEVICE = "1100",
  CLASS  = "1200"
]

Repeat the PCI section to assign multiple Metis devices to the same Virtual Machine.

Important

The Voyager SDK operator build requires CPU features exposed by host-passthrough. With a generic CPU model, make operators can fail because NumPy requires the x86-64-v2 (X86_V2) instruction baseline. In the validated environment, the operator build also failed with a Q35 machine type. Leave the OS/MACHINE attribute unset so OpenNebula uses its default machine type.

Configure the Guest

After deploying the Virtual Machine, verify that the guest can see the assigned device:

lspci -nn -d 1f9d:

Install the Metis Driver

Create the APT keyring directory and install the Axelera repository signing key:

sudo install -d -m 0755 /etc/apt/keyrings
curl -fsSL https://software.axelera.ai/artifactory/api/security/keypair/axelera/public \
  | sudo gpg --dearmor -o /etc/apt/keyrings/axelera.gpg

Configure the repository matching the guest operating system:

echo "deb [signed-by=/etc/apt/keyrings/axelera.gpg] https://software.axelera.ai/artifactory/axelera-apt-source ubuntu22 main" \
  | sudo tee /etc/apt/sources.list.d/axelera.list

echo "deb [signed-by=/etc/apt/keyrings/axelera.gpg] https://software.axelera.ai/artifactory/axelera-apt-source ubuntu24 main" \
  | sudo tee /etc/apt/sources.list.d/axelera.list

Install the kernel headers and the validated Metis driver, then load the module:

sudo apt-get update
sudo apt-get install -y linux-headers-$(uname -r) metis-dkms=1.5.7
sudo modprobe metis

Verify that the module is loaded:

lsmod | grep metis

Install the Voyager SDK

Clone the SDK release matching the validated 1.6.0 runtime and install its system dependencies:

git clone --branch v1.6.0 https://github.com/axelera-ai-hub/voyager-sdk.git
cd voyager-sdk
./install-dependencies.sh
sudo apt-get install -y python3-venv

Create and activate a Python virtual environment, then install the runtime and development kit:

python3 -m venv axelera-env16
source axelera-env16/bin/activate
pip install --extra-index-url https://software.axelera.ai/artifactory/api/pypi/axelera-pypi/simple \
  axelera-rt==1.6.0 'axelera-devkit[all]==1.6.0'
make operators

Activate this environment again with source axelera-env16/bin/activate in each new shell before using the SDK.

Verify that the SDK detects the device:

axdevice

Run the Inference Pipeline

As the final verification step, still within the voyager-sdk directory and with the loaded virtual environment, run an inference pipeline with a local video file:

./inference.py -v yolov5s-v7-coco video.mp4

Replace video.mp4 with the path to the input video.

Troubleshooting

IOMMU Group Is Not Viable

If QEMU reports that the device’s IOMMU group is not viable, another device in the same group is still using a Host driver. A typical error in /var/log/one/oned.log is:

vfio 0000:e3:00.0: group 32 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.

Identify the IOMMU group and inspect every device in it:

group=$(basename "$(readlink /sys/bus/pci/devices/0000:e3:00.0/iommu_group)")

for device in /sys/kernel/iommu_groups/"$group"/devices/*; do
  echo "== $(basename "$device") =="
  lspci -nnk -s "$(basename "$device")"
done

On systems where a PCI switch in the same group is managed by the switchtec module, unloading that module may allow the group to be assigned:

sudo modprobe -r switchtec

Retry the deployment after verifying that no remaining device in the group uses a Host driver.

Warning

Unload switchtec only after confirming that the Host does not require it to manage other hardware. This change lasts until the module is loaded again or the Host reboots. Consult the hardware vendor before blacklisting the module permanently.

X11 Forwarding

To display the Voyager SDK graphical output through SSH, connect to the guest with X11 forwarding as appropriate for your security policy and install the X11 authentication tools:

sudo apt-get install -y xauth x11-apps

For example, through a jump Host:

ssh -X -J <jump-host> <user>@<guest-address>

GIVE FEEDBACK

Was this resource helpful?

Glad to hear it

Sorry to hear that