ASRock B450 Pro4 - 9+ gpu cards - mixed rig

[jmnemonicj]

Hi everyone,

i have 10 GPUS on B450 Pro4 using 2x 1to4 usb risers in 16x a 4x PCIE slots.
I have mixed rigs with amd rx 6800 / 6700 / 6600.
When i tried to add 10th AMD card - rx580 (the 2nd gpu into 1to4 pcie riser card), I got errors like for example:

1. indicating the bad pcie riser like this one (Bad Riser example) :
   [ 50.494448] amdgpu 0000:21:00.0: [drm:amdgpu_ib_ring_tests [amdgpu]] ERROR IB test failed on gfx (-110).
   [ 50.494464] [drm:process_one_work] ERROR ib ring test failed (-110).
2. sometimes miner got stuck at:
   Auto-detected AMD OpenCL platform 0
   Auto-detected AMD OpenCL platform 1…

BUT If I add nvidia card ( Nvidia 1060 gt) using the same riser and same usb port on 1to4 riser, everything is working as expected, and 10th card is. working well:

Please what could be the problem? I haven’t tried another AMD miner yet.

Thanks for any hints.

[jmnemonicj]

So I tried another miners:
Phoenix, TeamBlackMiner, Ethminer, but every miner stuck at detecting gpus/platforms, when rx 580 added.
The card is working well itself.

The gpus are detected ok as well:

Thanks for any hint, below is dmesg init of the AMD rx580 card:

[ 48.572597] amdgpu 0000:25:00.0: enabling device (0000 → 0003)
[ 48.572673] [drm] initializing kernel modesetting (POLARIS10 0x1002:0x67DF 0x1DA2:0xE366 0xE7).
[ 48.572674] amdgpu 0000:25:00.0: amdgpu: Trusted Memory Zone (TMZ) feature not supported
[ 48.572700] [drm] register mmio base: 0xFA500000
[ 48.572701] [drm] register mmio size: 262144
[ 48.572703] [drm] PCIE atomic ops is not supported
[ 48.572713] [drm] add ip block number 0 <vi_common>
[ 48.572714] [drm] add ip block number 1 <gmc_v8_0>
[ 48.572715] [drm] add ip block number 2 <tonga_ih>
[ 48.572716] [drm] add ip block number 3 <gfx_v8_0>
[ 48.572717] [drm] add ip block number 4 <sdma_v3_0>
[ 48.572718] [drm] add ip block number 5
[ 48.572719] [drm] add ip block number 6
[ 48.572720] [drm] add ip block number 7 <uvd_v6_0>
[ 48.572721] [drm] add ip block number 8 <vce_v3_0>
[ 48.572724] kfd kfd: amdgpu: skipped device 1002:67df, PCI rejects atomics
[ 48.836082] amdgpu 0000:25:00.0: amdgpu: Fetched VBIOS from ROM BAR
[ 48.836084] amdgpu: ATOM BIOS: 113-1E366CU-S52
[ 48.836110] [drm] UVD is enabled in VM mode
[ 48.836111] [drm] UVD ENC is enabled in VM mode
[ 48.836115] [drm] VCE enabled in VM mode
[ 48.836134] [drm] GPU posting now…
[ 48.966864] [drm] vm size is 64 GB, 2 levels, block size is 10-bit, fragment size is 9-bit
[ 48.967819] amdgpu 0000:25:00.0: amdgpu: VRAM: 8192M 0x000000F400000000 - 0x000000F5FFFFFFFF (8192M used)
[ 48.967820] amdgpu 0000:25:00.0: amdgpu: GART: 256M 0x000000FF00000000 - 0x000000FF0FFFFFFF
[ 48.967827] [drm] Detected VRAM RAM=8192M, BAR=8192M
[ 48.967828] [drm] RAM width 256bits GDDR5
[ 48.967836] [drm] amdgpu: 8192M of VRAM memory ready
[ 48.967837] [drm] amdgpu: 8192M of GTT memory ready.
[ 48.967839] [drm] GART: num cpu pages 65536, num gpu pages 65536
[ 48.970095] [drm] PCIE GART of 256M enabled (table at 0x000000F400000000).
[ 48.971494] [drm] Chained IB support enabled!
[ 48.980861] amdgpu: [powerplay] hwmgr_sw_init smu backed is polaris10_smu
[ 48.985276] [drm] Found UVD firmware Version: 1.130 Family ID: 16
[ 48.998508] [drm] Found VCE firmware Version: 53.26 Binary ID: 3
[ 49.322301] [drm] Display Core initialized with v3.2.140!
[ 49.353741] [drm] UVD and UVD ENC initialized successfully.
[ 49.453711] [drm] VCE initialized successfully.
[ 49.453714] amdgpu 0000:25:00.0: amdgpu: SE 4, SH per SE 1, CU per SH 9, active_cu_number 36
[ 49.454150] amdgpu 0000:25:00.0: [drm] Cannot find any crtc or sizes
[ 49.457945] [drm] Initialized amdgpu 3.42.0 20150101 for 0000:25:00.0 on minor 10
[ 57.889841] amdgpu 0000:05:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[ 57.889843] amdgpu 0000:08:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[ 57.889845] amdgpu 0000:0b:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[ 57.889846] amdgpu 0000:0e:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[ 57.889847] amdgpu 0000:13:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[ 57.889849] amdgpu 0000:16:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[ 57.889850] amdgpu 0000:19:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[ 57.889851] amdgpu 0000:1c:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[ 57.889853] amdgpu 0000:24:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[ 57.889854] amdgpu 0000:25:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[ 68.711782] [drm:amdgpu_job_timedout [amdgpu]] ERROR ring sdma1 timeout, signaled seq=2, emitted seq=4
[ 68.711870] [drm:amdgpu_job_timedout [amdgpu]] ERROR Process information: process pid 0 thread pid 0
[ 68.711875] amdgpu 0000:25:00.0: amdgpu: GPU recovery disabled.
[ 68.711952] [drm:amdgpu_job_timedout [amdgpu]] ERROR ring sdma0 timeout, signaled seq=2, emitted seq=4
[ 68.712027] [drm:amdgpu_job_timedout [amdgpu]] ERROR Process information: process pid 0 thread pid 0
[ 68.712029] amdgpu 0000:25:00.0: amdgpu: GPU recovery disabled.

[jmnemonicj]

Finally I got it WORKING!
I had to move AMD RX580 to the dedicated PCIE port/riser and exchange RX6600 tp pcie 1to4 riser 2nd port instead.
Anyway, Phoenix or any other universal AMD/Nvidia miner was not working, so used dual amd+nvidia miner combo as you can see below:

So 11 gpus working and goint to add another one…