My PCIe switch is a Broadcom PEX88096 in the board’s one x16 slot, with four AMD Radeon AI PRO R9700s and the two RTX 3090s from my PCIe splitter post behind it, and the build post has the whole box.
The switch gives each card eight lanes and lets the cards copy to each other without going through the CPU. AMD’s ROCm calls that peer-to-peer (P2P), and it only turns it on when each card’s whole 32 GB is mapped. On Linux, that means a 32 GB Resizable BAR (base address register) on every card. Resizable BAR is the BIOS option that lets a card’s BAR, the range of the CPU’s physical address space that maps onto its memory, be the full 32 GB instead of the default 256 MB.
A few people asked for the module after the build post, so it’s on GitHub as gpu-rebar-layout. It leans on kernel internals that could change in any release, so read the README before you load it. A patch series on the mailing list fixes the sizing bug, but this board will still need the module’s resize and rescan after it lands, with its padding turned off (pad=0).
