<div dir="ltr">Thank you for the info Felipe.<div><br></div><div>I'll test with 5.0.5-7 in the morning. </div><div><br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><span style="font-size:small"><font color="#888888">Kind regards</font></span><span style="font-size:small"><br><div><p>Ray Coetzee<br><br></p></div><div></div></span><br></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 26, 2021 at 7:36 PM Felipe Knop <<a href="mailto:knop@us.ibm.com">knop@us.ibm.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10pt"><div dir="ltr">Ray,</div>
<div dir="ltr"> </div>
<div dir="ltr">Apologies; I should have added more details. My records show that there is a fix for IJ28891 and IJ29942, which was delivered in</div>
<div dir="ltr"> </div>
<div dir="ltr" style="margin-left:40px">5.1.0.2<br>5.0.5.5</div>
<div dir="ltr"> </div>
<div dir="ltr">"Program error" I believe is a code that indicates "needs to be fixed in our product".</div>
<div dir="ltr"> </div>
<div dir="ltr">I agree that the workaround mentioned is not "actionable" . The APAR page should have been clear that there is a fix available.</div>
<div dir="ltr"> </div>
<div dir="ltr">Regards,</div>
<div dir="ltr"> </div>
<div dir="ltr"> Felipe</div>
<div dir="ltr"> </div>
<div dir="ltr">----<br>Felipe Knop <a href="mailto:knop@us.ibm.com" target="_blank">knop@us.ibm.com</a><br>GPFS Development and Security<br>IBM Systems<br>IBM Building 008<br>2455 South Rd, Poughkeepsie, NY 12601<br>(845) 433-9314 T/L 293-9314<br> </div>
<div dir="ltr"> </div>
<div dir="ltr"> </div>
<blockquote dir="ltr" style="border-left:2px solid rgb(170,170,170);margin-left:5px;padding-left:5px;direction:ltr;margin-right:0px">----- Original message -----<br>From: Ray Coetzee <<a href="mailto:coetzee.ray@gmail.com" target="_blank">coetzee.ray@gmail.com</a>><br>To: Felipe Knop <<a href="mailto:knop@us.ibm.com" target="_blank">knop@us.ibm.com</a>><br>Cc: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>><br>Subject: [EXTERNAL] Re: [gpfsug-discuss] Kernel > 4.10, python >= 3.8 issue<br>Date: Wed, May 26, 2021 2:27 PM<br> <br> <span style="font-size:1px;color:rgb(255,255,255);line-height:1px;max-height:0px;max-width:0px;opacity:0;overflow:hidden;display:none"> Hi Felipe We looked at APAR IJ28891 & IJ29942, as both look identical but are closed as a "program error" with a workaround of "Do not use a file size which that is a multiple of the page size." Kind regards </span> <span style="font-size:1px;color:rgb(255,255,255);line-height:1px;max-height:0px;max-width:0px;opacity:0;overflow:hidden;display:none">ZjQcmQRYFpfptBannerStart</span>
<div style="margin:16px 0px;padding:8px 16px;border-radius:4px;min-width:200px;background-color:rgb(208,216,220);border-top:4px solid rgb(144,164,174)"><div style="display:inline-block;margin:0px 0px 1px;max-width:600px"><div style="line-height:18px;display:block;color:rgb(0,0,0);font-family:Arial,sans-serif;font-weight:bold;font-size:14px">This Message Is From an External Sender</div>
<div style="line-height:18px;margin-top:2px;display:block;color:rgb(0,0,0);font-weight:normal;font-family:Arial,sans-serif;font-size:12px">This message came from outside your organization.</div></div></div> <span style="font-size:1px;color:rgb(255,255,255);line-height:1px;max-height:0px;max-width:0px;opacity:0;overflow:hidden;display:none">ZjQcmQRYFpfptBannerEnd</span><br>
<div dir="ltr"><div dir="ltr">Hi Felipe
<div> </div>
<div>We looked at APAR IJ28891 & IJ29942, as both look identical but are closed as a "<span style="color:rgb(50,50,50);font-family:"IBM Plex Sans","IBM Plex Mono";font-size:16px">program error" with a workaround of "</span><span style="color:rgb(50,50,50);font-family:ibm-plex-mono,Menlo,"DejaVu Sans Mono","Bitstream Vera Sans Mono",Courier,monospace;font-size:0.9375rem;white-space:pre-wrap">Do not use a file size which that is a multiple </span><span style="color:rgb(50,50,50);font-family:ibm-plex-mono,Menlo,"DejaVu Sans Mono","Bitstream Vera Sans Mono",Courier,monospace;font-size:0.9375rem;white-space:pre-wrap">of the page size.</span><span style="color:rgb(50,50,50);font-family:"IBM Plex Sans","IBM Plex Mono";font-size:16px">"</span></div>
<div> </div>
<div><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><span style="font-size:small"><font color="#888888">Kind regards</font></span>
<div><p><span style="font-size:small">Ray Coetzee</span><br> </p></div></div></div></div></div></div></div></div>
<div><div dir="ltr">On Wed, May 26, 2021 at 6:45 PM Felipe Knop <<a href="mailto:knop@us.ibm.com" target="_blank">knop@us.ibm.com</a>> wrote:</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10pt"><div dir="ltr">Ray,</div>
<div dir="ltr"> </div>
<div dir="ltr">I wonder if you are hitting the problem which was fixed on the following APAR:</div>
<div dir="ltr"> </div>
<div dir="ltr"><a href="https://www.ibm.com/support/pages/apar/IJ28891" target="_blank">https://www.ibm.com/support/pages/apar/IJ28891</a></div>
<div dir="ltr"> </div>
<div dir="ltr"> </div>
<div dir="ltr"> Felipe</div>
<div dir="ltr"> </div>
<div dir="ltr">----<br>Felipe Knop <a href="mailto:knop@us.ibm.com" target="_blank">knop@us.ibm.com</a><br>GPFS Development and Security<br>IBM Systems<br>IBM Building 008<br>2455 South Rd, Poughkeepsie, NY 12601<br>(845) 433-9314 T/L 293-9314<br> </div>
<div dir="ltr"> </div>
<div dir="ltr"> </div>
<blockquote dir="ltr" style="border-left:2px solid rgb(170,170,170);margin-left:5px;padding-left:5px;direction:ltr;margin-right:0px">----- Original message -----<br>From: Ray Coetzee <<a href="mailto:coetzee.ray@gmail.com" target="_blank">coetzee.ray@gmail.com</a>><br>Sent by: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a><br>To: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>><br>Cc:<br>Subject: [EXTERNAL] [gpfsug-discuss] Kernel > 4.10, python >= 3.8 issue<br>Date: Wed, May 26, 2021 1:38 PM<br>
<div dir="ltr"><div>Hello all</div>
<div> </div>
<div>I'd be interested to know if anyone else has experienced a problem with <span style="color:black;font-family:Arial,sans-serif;font-size:10pt">Kernel > 4.10, python >= 3.8 and Spectrum Scale (5.0.5-2).</span></div>
<div> </div>
<div><p style="margin:0px;font-size:12pt;font-family:Calibri,sans-serif"><span style="font-size:10pt;font-family:Arial,sans-serif;color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">We noticed that python shut.copy() is failing against a GPFS mount with:</span><br><br><span style="font-size:10pt;font-family:Arial,sans-serif;color:black"><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">BlockingIOError: [Errno 11] Resource temporarily unavailable: 'test.file' -> 'test2.file'</span><br><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">To reproduce the error:</span><br><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">```</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">[user@</span></span>login01<span style="font-size:10pt;font-family:Arial,sans-serif;color:black"><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">]$ module load python-3.8.9-gcc-9.3.0-soqwnzh</span><br><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">[</span></span> user<span style="font-size:10pt;font-family:Arial,sans-serif;color:black"><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">@login01]$ truncate --size 640MB test.file</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">[</span></span> user<span style="font-size:10pt;font-family:Arial,sans-serif;color:black"><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">@</span></span>login01<span style="font-size:10pt;font-family:Arial,sans-serif;color:black"><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">]$ python3 -c "import shutil; shutil.copy('test.file', 'test2.file')"</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">Traceback (most recent call last):</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"> File "<string>", line 1, in <module></span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"> File "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py", line 418, in copy</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"> copyfile(src, dst, follow_symlinks=follow_symlinks)</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"> File "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py", line 275, in copyfile</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"> _fastcopy_sendfile(fsrc, fdst)</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"> File "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py", line 172, in _fastcopy_sendfile</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"> raise err</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"> File "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py", line 152, in _fastcopy_sendfile</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"> sent = os.sendfile(outfd, infd, offset, blocksize)</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">BlockingIOError: [Errno 11] Resource temporarily unavailable: 'test.file' -> 'test2.file'</span></span></p>
<p style="margin:0px;font-size:12pt;font-family:Calibri,sans-serif"><span style="font-size:10pt;font-family:Arial,sans-serif;color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"> </span></p>
<p style="margin:0px;font-size:12pt;font-family:Calibri,sans-serif"><span style="font-size:10pt;font-family:Arial,sans-serif;color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"> </span><span style="color:black;font-family:Arial,sans-serif;font-size:10pt">Investigating into why this is happening revealed that:</span></p>
<p style="margin:0px;font-size:12pt;font-family:Calibri,sans-serif"><br><span style="font-size:10pt;font-family:Arial,sans-serif;color:black"><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">1. It is failing for python3.8 and above.</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">2. It is happening only a GPFS mount</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">3. It is happening with files whose size is multiple of 4KB (OS Page size)</span><br><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">Relevant links:</span></span><br><a href="https://bugs.python.org/issue43743" style="color:blue" target="_blank"><span style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(102,102,102);background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">https://bugs.python.org/issue43743</span></a><br><a href="https://www.ibm.com/support/pages/apar/IJ28891" style="color:blue" target="_blank"><span style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(102,102,102);background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">https://www.ibm.com/support/pages/apar/IJ28891</span></a><br><br><br><span style="font-size:10pt;font-family:Arial,sans-serif;color:black"><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">Doing an strace revealed that at the lower level, it seems to be related to the Linux Syscall of “sendfile”, which seems to fail in these cases on GPFS.</span><br><br><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">Strace for a 4096 B file:</span><br><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">```</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">sendfile(4, 3, [0] => [4096], 8388608) = 4096</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">sendfile(4, 3, [4096], 8388608) = -1 EAGAIN (Resource temporarily unavailable)</span><br><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">```</span><br><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">The same file on other disk:</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">```</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">sendfile(4, 3, [0] => [4096], 8388608) = 4096</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">sendfile(4, 3, [4096], 8388608) = 0</span></span></p>
<p style="margin:0px;font-size:12pt;font-family:Calibri,sans-serif"> </p>
<p style="margin:0px;font-size:12pt;font-family:Calibri,sans-serif"><span style="font-size:10pt;font-family:Arial,sans-serif;color:black">IBM's "fix" for the problem of "</span><span style="color:rgb(50,50,50);font-family:ibm-plex-mono,Menlo,"DejaVu Sans Mono","Bitstream Vera Sans Mono",Courier,monospace;font-size:0.9375rem;white-space:pre-wrap">Do not use a file size which that is a multiple </span><span style="color:rgb(50,50,50);font-family:ibm-plex-mono,Menlo,"DejaVu Sans Mono","Bitstream Vera Sans Mono",Courier,monospace;font-size:0.9375rem;white-space:pre-wrap">of the page size.</span><span style="font-size:10pt;font-family:Arial,sans-serif;color:black">" sounds really blas</span><span style="font-family:Arial,Helvetica,sans-serif;font-size:small">é.</span></p>
<p style="margin:0px;font-size:12pt;font-family:Calibri,sans-serif"><br><span style="font-size:10pt;font-family:Arial,sans-serif;color:black"><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">```</span></span></p></div>
<div> </div>
<div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><span style="font-size:small"><font color="#888888">Kind regards</font></span>
<div><p><span style="font-size:small">Ray Coetzee</span><br> </p></div>
<div> </div></div></div></div></div></div></div>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> </font></div></blockquote>
<div dir="ltr"> </div></div></blockquote></div></div></blockquote>
<div dir="ltr"> </div></div><br>
</blockquote></div>