[gpfsug-discuss] GPFS 5.1.9.4 on Windows 11 Pro. Performance issues, write.

Tue Sep 3 13:10:50 BST 2024

Still no solution here regarding this.

Have tested other cables.
Have tested to change tcp window size, no change
Played with numa in the bios, no change
Played with hyperthreading in bios, no change

Have anyone managed to get some speed out of windows 11 and gpfs?

--

Henrik Cednert  /  + 46 704 71 89 54  /  CTO  /  OnePost (formerly Filmlance Post)

☝️ OnePost, formerly Filmlance's post-production, is now an independent part of the Banijay Group.
New name, same team – business as usual at OnePost.

________________________________
From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> on behalf of Henrik Cednert <henrik.cednert at onepost.se>
Sent: Friday, 9 August 2024 17:25
To: gpfsug-discuss at gpfsug.org <gpfsug-discuss at gpfsug.org>
Subject: [gpfsug-discuss] GPFS 5.1.9.4 on Windows 11 Pro. Performance issues, write.

VARNING: DETTA ÄR ETT EXTERNT MAIL. Klicka inte på några länkar oavsett hur legitima de verkar utan att verifiera.

Hello

I have some issues with write performance on a windows 11 pro system and I'm out of ideas here. Hopefully someone here have some bright ideas and/or experience of GPFS on Windows 11?

The system is a:

Windows 11 Pro 22H2
2 x Intel(R) Xeon(R) Gold 6418H   2.10 GHz
512 GB RAM
GPFS 5.1.9.4
Mellanox ConnectX 6 Dx
100GbE connected to Mellanox Switch with 5m Mellanox DAC.

Before deploying this workstation we had a single socket system as a test bench where we got 60 GbE in both directons with iPerf and around 6GB/sec write and 3GB/sec read from the system over GPFS (fio tests, same tests as furhter down here).

With that system I had loads of issues before getting to that point though. MS Defender had to be forcefully disabled via regedit some other tweaks. All those tweaks have been performed in this new system as well, but I can't get the proper speed out of it.

On this new system and with iPerf to the storage servers I get around 50-60GbE in both directions and send and receive.

If I mount the storage over SMB and 100GbE via the storage gateway servers I get around 3GB/sec read and write with Blackmagics Disk speed test. I have not tweaked the system for samba performande, just a test to see what it would give and part of the troubleshooting.

If I run Blackmagics diskspeed test to the GPFS mount I instead get around 700MB/sec write and 400MB/sec read.

Starting to think that the Blackmagic test might not run properly on this machine with these CPUs though. Or it's related to the mmfsd process maybe, how that threads or not threads...?

But if we instead look at fio. I have a bat script that loops through a bunch of FIO-tests. A test that I have been using over the years so that we easily can benchmark all deployed systems with the exakt same tests. The tests are named like:

seqrw-<filesize>gb-<blocksize>mb-t<threads>

The result when I run this is like the below list. Number in parenthesis is the by fio reported latency.

Job: seqrw-40gb-1mb-t1
      •     Write: 162 MB/s (6 ms)
      •     Read: 1940 MB/s (1 ms)

Job: seqrw-20gb-1mb-t2
      •     Write: 286 MB/s (7 ms)
      •     Read: 3952 MB/s (1 ms)

Job: seqrw-10gb-1mb-t4
      •     Write: 549 MB/s (7 ms)
      •     Read: 6987 MB/s (1 ms)

Job: seqrw-05gb-1mb-t8
      •     Write: 989 MB/s (8 ms)
      •     Read: 7721 MB/s (1 ms)

Job: seqrw-40gb-2mb-t1
      •     Write: 161 MB/s (12 ms)
      •     Read: 2261 MB/s (0 ms)

Job: seqrw-20gb-2mb-t2
      •     Write: 348 MB/s (11 ms)
      •     Read: 4266 MB/s (1 ms)

Job: seqrw-10gb-2mb-t4
      •     Write: 626 MB/s (13 ms)
      •     Read: 4949 MB/s (1 ms)

Job: seqrw-05gb-2mb-t8
      •     Write: 1154 MB/s (14 ms)
      •     Read: 7007 MB/s (2 ms)

Job: seqrw-40gb-4mb-t1
      •     Write: 161 MB/s (25 ms)
      •     Read: 2083 MB/s (1 ms)

Job: seqrw-20gb-4mb-t2
      •     Write: 352 MB/s (23 ms)
      •     Read: 4317 MB/s (2 ms)

Job: seqrw-10gb-4mb-t4
      •     Write: 696 MB/s (23 ms)
      •     Read: 7358 MB/s (2 ms)

Job: seqrw-05gb-4mb-t8
      •     Write: 1251 MB/s (25 ms)
      •     Read: 6707 MB/s (5 ms)

So with fio I get a very nice read speed, but the write is horrendous and I cannot find what causes it. I have looked at affinity settings for the mmfsd process but not sure I fully understand it. But no matter what I set it to, I see no difference.

I have "played" with the bios and tried with/without hyperthreading, numa and so on. And nothing affects atleast the blackmagic disk speed test.

the current settings for this host is like below. I write "current" because I have tested a few different settings here but nothing affects the write speed. maxTcpConnsPerNodeConn for sure bumped the read speed though.

nsdMaxWorkerThreads 16
prefetchPct 60
maxTcpConnsPerNodeConn 8
maxMBpS 14000

Does anyone have any suggestions or ideas on how to troubleshoot this?

Thanks

--

Henrik Cednert  /  + 46 704 71 89 54  /  CTO  /  OnePost (formerly Filmlance Post)

☝️ OnePost, formerly Filmlance's post-production, is now an independent part of the Banijay Group.
New name, same team – business as usual at OnePost.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240903/0019fcb1/attachment-0001.htm>