<div dir="ltr"><div>Hello all</div><div><br></div><div>I'd be interested to know if anyone else has experienced a problem with <span style="color:black;font-family:Arial,sans-serif;font-size:10pt">Kernel > 4.10, python >= 3.8 and Spectrum Scale (5.0.5-2).</span></div><div><span style="color:black;font-family:Arial,sans-serif;font-size:10pt"><br></span></div><div><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif"><span style="font-size:10pt;font-family:Arial,sans-serif;color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">We noticed that python shut.copy() is
failing against a GPFS mount with:</span><span style="font-size:10pt;font-family:Arial,sans-serif;color:black"><br>
<br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">BlockingIOError: [Errno 11] Resource temporarily
unavailable: 'test.file' -> 'test2.file'</span><br>
<br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">To reproduce the error:</span><br>
<br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">```</span><br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">[user@</span></span>login01<span style="font-size:10pt;font-family:Arial,sans-serif;color:black"><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">]$ module load
python-3.8.9-gcc-9.3.0-soqwnzh</span><br>
<br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">[</span></span>

user<span style="font-size:10pt;font-family:Arial,sans-serif;color:black"><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">@login01]$ truncate
--size 640MB test.file</span><br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">[</span></span>

user<span style="font-size:10pt;font-family:Arial,sans-serif;color:black"><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">@</span></span>login01<span style="font-size:10pt;font-family:Arial,sans-serif;color:black"><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">]$ python3 -c
"import shutil; shutil.copy('test.file', 'test2.file')"</span><br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">Traceback (most recent call last):</span><br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"> File "<string>", line 1,
in <module></span><br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"> File
"/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
line 418, in copy</span><br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"> copyfile(src, dst,
follow_symlinks=follow_symlinks)</span><br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"> File "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
line 275, in copyfile</span><br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"> _fastcopy_sendfile(fsrc, fdst)</span><br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"> File
"/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
line 172, in _fastcopy_sendfile</span><br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"> raise err</span><br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"> File
"/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
line 152, in _fastcopy_sendfile</span><br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"> sent = os.sendfile(outfd, infd, offset,
blocksize)</span><br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">BlockingIOError: [Errno 11] Resource temporarily
unavailable: 'test.file' -> 'test2.file'</span></span></p>

<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif"><span style="font-size:10pt;font-family:Arial,sans-serif;color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"> </span></p>

<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif"><span style="font-size:10pt;font-family:Arial,sans-serif;color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"> </span><span style="color:black;font-family:Arial,sans-serif;font-size:10pt">Investigating into why this is happening revealed
that:</span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif"><span style="font-size:10pt;font-family:Arial,sans-serif;color:black">
<br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">1. It is failing for python3.8 and above.</span><br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">2. It is happening only a GPFS mount</span><br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">3. It is happening with files whose size is
multiple of 4KB (OS Page size)</span><br>
<br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">Relevant links:</span><br>
</span><a href="https://bugs.python.org/issue43743" target="_blank" style="color:blue"><span style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(102,102,102);background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">https://bugs.python.org/issue43743</span></a><span style="font-size:10pt;font-family:Arial,sans-serif;color:black"><br>
</span><a href="https://www.ibm.com/support/pages/apar/IJ28891" target="_blank" style="color:blue"><span style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(102,102,102);background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">https://www.ibm.com/support/pages/apar/IJ28891</span></a><span style="font-size:10pt;font-family:Arial,sans-serif;color:black"><br>
<br>
<br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">Doing an strace revealed that at the lower
level, it seems to be related to the Linux Syscall of “sendfile”, which seems
to fail in these cases on GPFS.</span><br>
<br>
<br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">Strace for a 4096 B file:</span><br>
<br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">```</span><br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">sendfile(4, 3, [0] => [4096], 8388608) = 4096</span><br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">sendfile(4, 3, [4096], 8388608) = -1 EAGAIN
(Resource temporarily unavailable)</span><br>
<br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">```</span><br>
<br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">The same file on other disk:</span><br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">```</span><br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">sendfile(4, 3, [0] => [4096], 8388608) = 4096</span><br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">sendfile(4, 3, [4096], 8388608) = 0</span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif"><span style="font-size:10pt;font-family:Arial,sans-serif;color:black"><br></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif"><span style="font-size:10pt;font-family:Arial,sans-serif;color:black">IBM's "fix" for the problem of "</span><span style="color:rgb(50,50,50);font-family:ibm-plex-mono,Menlo,"DejaVu Sans Mono","Bitstream Vera Sans Mono",Courier,monospace;font-size:0.9375rem;white-space:pre-wrap">Do not use a file size which that is a multiple </span><span style="color:rgb(50,50,50);font-family:ibm-plex-mono,Menlo,"DejaVu Sans Mono","Bitstream Vera Sans Mono",Courier,monospace;font-size:0.9375rem;white-space:pre-wrap">of the page size.</span><span style="font-size:10pt;font-family:Arial,sans-serif;color:black">"  sounds really blas</span><span style="font-family:Arial,Helvetica,sans-serif;font-size:small">é.</span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:Calibri,sans-serif"><span style="font-size:10pt;font-family:Arial,sans-serif;color:black"><br>
<span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">```</span></span></p></div><div><br></div><br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><span style="font-size:small"><font color="#888888">Kind regards</font></span><span style="font-size:small"><br><div><p>Ray Coetzee<br><br></p></div><div></div></span><br></div></div></div></div></div></div>