Proxmox VE

Remote install on Supermicro mainboard with ASPEED AST2500 BMC

If you try to install Proxmox 7.x remotely on a computer with Supermicro mainboard and ASPEED AST2500 BMC, you will probably not see the graphical installer, but after a while (2-3 minutes) you will see the following message:

Installation aborted - unable to continue (type exit or CTRL-D to reboot)
root@proxmox:/#

This is due to some problems with the ASPEED AST2500 graphics card and the X server configuration.

To solve this problem, you need to perform the following steps in the opened terminal session:

Xorg -configure
nano xorg.conf.new
Inside this file:
1. Find the line Load "glx" and replace it with Disable "glx"
2. After that line add Disable "glamoregl"
3. Find the line Driver "modesetting" and replace it with Driver "fbdev"
mv xorg.conf.new /etc/X11/xorg.conf
Make sure the file /etc/hosts exist and contain the line 127.0.0.1 proxmox
startx

Note

: The graphical installer will be a bit broken and some (not relevant) texts will not be displayed.

After Proxmox is successfully installed, you will be redirected back to the terminal session. To reboot type exit.

Reference (gist.github.com)

Install with /root on ZFS

Change kernel boot parameters

nano /etc/kernel/cmdline
proxmox-boot-tool refresh

No-Subscription repository

copy file

cp /etc/apt/sources.list.d/pve-enterprise.list /etc/apt/sources.list.d/pve-no-subscription.list
mv /etc/apt/sources.list.d/pve-enterprise.list /etc/apt/sources.list.d/pve-enterprise.list.disabled

change file

nano /etc/apt/sources.list.d/pve-no-subscription.list

deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription

than run

apt update

Bad memory

If you have bad memory and you know where the faulty locations are (e.g. with memtest) you can hide these locations with the kernel boot parameter memmap=<SIZE_IN_BYTES>$<START_ADDRESS>.

Example:

This kernel boot parameter hides 0x10000000 bytes (256M) starting from address 0x320000000:
memmap=0x10000000$0x320000000

PCIe passthrough

To use PCIe passthrough you have to enable IOMMU in the BIOS (Intel: VT-d).

Also you have to put some kernel boot commands:

For Intel: intel_iommu=on iommu=pt

List IOMMU groups with their PCIe and USB devices with this script.

Now you have to find the IDs from the PCIe devices you want to passthrough. The command lspci -k -nn produces something like this:

[...]
00:12.0 SATA controller [0106]: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series SATA AHCI Controller [8086:5ae3] (rev 0b)
        Subsystem: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series SATA AHCI Controller [8086:5ae3]
        Kernel driver in use: vfio-pci
        Kernel modules: ahci
[...]

In the output you have to search for the right PCIe device. When you found it, the ID is located in the square brackets [xxxx:yyyy]. E.g. in the snipped above the SATA controller has the ID [8086:5ae3].

Passthrough a SATA controller is a little bit more complex

Since the ahci driver is loaded at a very early stage and for passthrough the PCIe device must be loaded with the vfio-pci driver, the following must be done to load the vfio-pci driver before the ahci driver:

Add the line vfio-pci to the file /etc/initramfs-tools/modules.
Create the file /usr/share/initramfs-tools/scripts/init-top/load_vfio-pci with the following content (<PCIE_DEVICE_IDS is a comma separated list of all IDs which should be loaded with the vfio-pci driver):

#!/bin/sh
modprobe vfio-pci ids=<PCIE_DEVICE_IDS>

🔴 TODO_start: clean up this section

Multiple identical GPUs

https://forum.level1techs.com/t/vfio-in-2019-pop-os-how-to-general-guide-though-draft/142287

Create the file /usr/share/initramfs-tools/scripts/init-top/load_vfio-pci with the following content

#!/bin/sh

DEVS="0000:01:00.0 0000:01:00.1 0000:21:00.0 0000:21:00.1"

for DEV in $DEVS;
  do echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
done

🔴 TODO_end

Example:

For the SATA controller mentioned earlier this file would look like the following:
#!/bin/sh
modprobe vfio-pci ids=8086:5ae3

Make this file executable with chmod ug+x /usr/share/initramfs-tools/scripts/init-top/load_vfio-pci.
Edit the file nano /usr/share/initramfs-tools/scripts/init-top/udev and add load_vfio-pci to the varibale PREREQS. This should look like this PREREQS="load_vfio-pci".
For these changes to be applied, you must run update-initramfs -u -k all and reboot the system.