<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" >Ray,</div>
<div dir="ltr" > </div>
<div dir="ltr" >I wonder if you are hitting the problem which was fixed on the following APAR:</div>
<div dir="ltr" > </div>
<div dir="ltr" ><a href="https://www.ibm.com/support/pages/apar/IJ28891">https://www.ibm.com/support/pages/apar/IJ28891</a></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<div dir="ltr" >  Felipe</div>
<div dir="ltr" > </div>
<div dir="ltr" >----<br>Felipe Knop knop@us.ibm.com<br>GPFS Development and Security<br>IBM Systems<br>IBM Building 008<br>2455 South Rd, Poughkeepsie, NY 12601<br>(845) 433-9314 T/L 293-9314<br> </div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: Ray Coetzee <coetzee.ray@gmail.com><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Cc:<br>Subject: [EXTERNAL] [gpfsug-discuss] Kernel > 4.10, python >= 3.8 issue<br>Date: Wed, May 26, 2021 1:38 PM<br> 
<div dir="ltr" ><div>Hello all</div>
<div> </div>
<div>I'd be interested to know if anyone else has experienced a problem with <span style="color:black;font-family:Arial,sans-serif;font-size:10pt" >Kernel > 4.10, python >= 3.8 and Spectrum Scale (5.0.5-2).</span></div>
<div> </div>
<div><p style="margin: 0px; font-size: 12pt; font-family: Calibri, sans-serif;" ><span style="font-size:10pt;font-family:Arial,sans-serif;color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >We noticed that python shut.copy() is failing against a GPFS mount with:</span><br><br><span style="font-size:10pt;font-family:Arial,sans-serif;color:black" ><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >BlockingIOError: [Errno 11] Resource temporarily unavailable: 'test.file' -> 'test2.file'</span><br><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >To reproduce the error:</span><br><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >```</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >[user@</span></span>login01<span style="font-size:10pt;font-family:Arial,sans-serif;color:black" ><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >]$ module load python-3.8.9-gcc-9.3.0-soqwnzh</span><br><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >[</span></span> user<span style="font-size:10pt;font-family:Arial,sans-serif;color:black" ><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >@login01]$ truncate --size 640MB test.file</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >[</span></span> user<span style="font-size:10pt;font-family:Arial,sans-serif;color:black" ><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >@</span></span>login01<span style="font-size:10pt;font-family:Arial,sans-serif;color:black" ><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >]$ python3 -c "import shutil; shutil.copy('test.file', 'test2.file')"</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >Traceback (most recent call last):</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" > File "<string>", line 1, in <module></span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" > File "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py", line 418, in copy</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" > copyfile(src, dst, follow_symlinks=follow_symlinks)</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" > File "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py", line 275, in copyfile</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" > _fastcopy_sendfile(fsrc, fdst)</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" > File "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py", line 172, in _fastcopy_sendfile</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" > raise err</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" > File "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py", line 152, in _fastcopy_sendfile</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" > sent = os.sendfile(outfd, infd, offset, blocksize)</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >BlockingIOError: [Errno 11] Resource temporarily unavailable: 'test.file' -> 'test2.file'</span></span></p>
<p style="margin: 0px; font-size: 12pt; font-family: Calibri, sans-serif;" ><span style="font-size:10pt;font-family:Arial,sans-serif;color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" > </span></p>
<p style="margin: 0px; font-size: 12pt; font-family: Calibri, sans-serif;" ><span style="font-size:10pt;font-family:Arial,sans-serif;color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" > </span><span style="color:black;font-family:Arial,sans-serif;font-size:10pt" >Investigating into why this is happening revealed that:</span></p>
<p style="margin: 0px; font-size: 12pt; font-family: Calibri, sans-serif;" ><br><span style="font-size:10pt;font-family:Arial,sans-serif;color:black" ><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >1. It is failing for python3.8 and above.</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >2. It is happening only a GPFS mount</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >3. It is happening with files whose size is multiple of 4KB (OS Page size)</span><br><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >Relevant links:</span></span><br><a href="https://bugs.python.org/issue43743" style="color:blue" target="_blank"><span style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(102,102,102);background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >https://bugs.python.org/issue43743</span></a><br><a href="https://www.ibm.com/support/pages/apar/IJ28891" style="color:blue" target="_blank"><span style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(102,102,102);background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >https://www.ibm.com/support/pages/apar/IJ28891</span></a><br><br><br><span style="font-size:10pt;font-family:Arial,sans-serif;color:black" ><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >Doing an strace revealed that at the lower level, it seems to be related to the Linux Syscall of “sendfile”, which seems to fail in these cases on GPFS.</span><br><br><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >Strace for a 4096 B file:</span><br><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >```</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >sendfile(4, 3, [0] => [4096], 8388608) = 4096</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >sendfile(4, 3, [4096], 8388608) = -1 EAGAIN (Resource temporarily unavailable)</span><br><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >```</span><br><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >The same file on other disk:</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >```</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >sendfile(4, 3, [0] => [4096], 8388608) = 4096</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >sendfile(4, 3, [4096], 8388608) = 0</span></span></p>
<p style="margin: 0px; font-size: 12pt; font-family: Calibri, sans-serif;" > </p>
<p style="margin: 0px; font-size: 12pt; font-family: Calibri, sans-serif;" ><span style="font-size:10pt;font-family:Arial,sans-serif;color:black" >IBM's "fix" for the problem of "</span><span style="color:rgb(50,50,50);font-family:ibm-plex-mono,Menlo,"DejaVu Sans Mono","Bitstream Vera Sans Mono",Courier,monospace;font-size:0.9375rem;white-space:pre-wrap" >Do not use a file size which that is a multiple </span><span style="color:rgb(50,50,50);font-family:ibm-plex-mono,Menlo,"DejaVu Sans Mono","Bitstream Vera Sans Mono",Courier,monospace;font-size:0.9375rem;white-space:pre-wrap" >of the page size.</span><span style="font-size:10pt;font-family:Arial,sans-serif;color:black" >"  sounds really blas</span><span style="font-family:Arial,Helvetica,sans-serif;font-size:small" >é.</span></p>
<p style="margin: 0px; font-size: 12pt; font-family: Calibri, sans-serif;" ><br><span style="font-size:10pt;font-family:Arial,sans-serif;color:black" ><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >```</span></span></p></div>
<div> </div> 

<div><div data-smartmail="gmail_signature" dir="ltr" ><div dir="ltr" ><div dir="ltr" ><div dir="ltr" ><span style="font-size:small" ><font color="#888888" >Kind regards</font></span>
<div><p><span style="font-size:small" >Ray Coetzee</span><br> </p></div>
<div> </div></div></div></div></div></div></div>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> </font></div></blockquote>
<div dir="ltr" > </div></div><BR>