<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" >Ray,</div>
<div dir="ltr" > </div>
<div dir="ltr" >Apologies; I should have added more details. My records show that there is a fix for IJ28891 and IJ29942, which was delivered in</div>
<div dir="ltr" > </div>
<div dir="ltr" style="margin-left: 40px;" >5.1.0.2<br>5.0.5.5</div>
<div dir="ltr" > </div>
<div dir="ltr" >"Program error" I believe is a code that indicates "needs to be fixed in our product".</div>
<div dir="ltr" > </div>
<div dir="ltr" >I agree that the workaround mentioned is not "actionable" . The APAR page should have been clear that there is a fix available.</div>
<div dir="ltr" > </div>
<div dir="ltr" >Regards,</div>
<div dir="ltr" > </div>
<div dir="ltr" > Felipe</div>
<div dir="ltr" > </div>
<div dir="ltr" >----<br>Felipe Knop knop@us.ibm.com<br>GPFS Development and Security<br>IBM Systems<br>IBM Building 008<br>2455 South Rd, Poughkeepsie, NY 12601<br>(845) 433-9314 T/L 293-9314<br> </div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" data-history-expanded="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: Ray Coetzee <coetzee.ray@gmail.com><br>To: Felipe Knop <knop@us.ibm.com><br>Cc: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Subject: [EXTERNAL] Re: [gpfsug-discuss] Kernel > 4.10, python >= 3.8 issue<br>Date: Wed, May 26, 2021 2:27 PM<br> <br><!-- BaNnErBlUrFlE-HeAdEr-start --> <!-- BaNnErBlUrFlE-HeAdEr-end --> <!-- BaNnErBlUrFlE-BoDy-start --> <!-- Preheader Text : BEGIN --> <span style="display:none !important;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;max-width:0px;opacity:0;overflow:hidden;" > Hi Felipe We looked at APAR IJ28891 & IJ29942, as both look identical but are closed as a "program error" with a workaround of "Do not use a file size which that is a multiple of the page size." Kind regards </span> <!-- Preheader Text : END --> <!-- Email Banner : BEGIN --> <span style="display:none !important;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;max-width:0px;opacity:0;overflow:hidden;" >ZjQcmQRYFpfptBannerStart</span> <!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 3px 0px 16px 0px;" ><tr><td>
<table class="pfptBannerTableMSO" border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%;border-radius:4px;border-top:4px solid #90a4ae;background-color:#d0d8dc;" ><tr><td valign="top" >
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px" >
<tr><td><span class="pfptTitleMSO" style="color:#000000 !important;font-family: 'Arial', sans-serif;font-weight:bold !important;font-size:14px !important;" >
This Message Is From an External Sender
</span></td></tr>
<tr><td><span class="pfptSubtitleMSO" style="color:#000000 !important;font-weight:normal !important;font-family: 'Arial', sans-serif; font-size:12px !important;" >
This message came from outside your organization.
</span></td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]--> <!--[if !((ie)|(mso))]-->
<div style="margin:16px 0px 16px 0px; padding:8px 16px 8px 16px; border-radius: 4px; min-width: 200px;background-color: #d0d8dc; border-top: 4px solid #90a4ae;" ><div style="display: inline-block; margin: 0px 0px 1px 0px; max-width: 600px;" ><div style="color:#000000 !important;font-family: 'Arial', sans-serif !important;font-weight:bold !important;font-size:14px !important;line-height:18px;display:block;" >This Message Is From an External Sender</div>
<div style="color:#000000 !important;font-weight:normal !important;font-family: 'Arial', sans-serif !important;font-size:12px !important;line-height:18px;margin-top:2px;display:block" >This message came from outside your organization.</div></div></div><!--[endif]--> <span style="display:none !important;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;max-width:0px;opacity:0;overflow:hidden;" >ZjQcmQRYFpfptBannerEnd</span><br><!-- Email Banner : END --> <!-- BaNnErBlUrFlE-BoDy-end -->
<div dir="ltr" ><div dir="ltr" >Hi Felipe
<div> </div>
<div>We looked at APAR IJ28891 & IJ29942, as both look identical but are closed as a "<span style="color:rgb(50,50,50);font-family:"IBM Plex Sans","IBM Plex Mono";font-size:16px" >program error" with a workaround of "</span><span style="color:rgb(50,50,50);font-family:ibm-plex-mono,Menlo,"DejaVu Sans Mono","Bitstream Vera Sans Mono",Courier,monospace;font-size:0.9375rem;white-space:pre-wrap" >Do not use a file size which that is a multiple </span><span style="color:rgb(50,50,50);font-family:ibm-plex-mono,Menlo,"DejaVu Sans Mono","Bitstream Vera Sans Mono",Courier,monospace;font-size:0.9375rem;white-space:pre-wrap" >of the page size.</span><span style="color:rgb(50,50,50);font-family:"IBM Plex Sans","IBM Plex Mono";font-size:16px" >"</span></div>
<div> </div>
<div><div><div data-smartmail="gmail_signature" dir="ltr" ><div dir="ltr" ><div dir="ltr" ><div dir="ltr" ><span style="font-size:small" ><font color="#888888" >Kind regards</font></span>
<div><p><span style="font-size:small" >Ray Coetzee</span><br> </p></div></div></div></div></div></div></div></div>
<div><div dir="ltr" >On Wed, May 26, 2021 at 6:45 PM Felipe Knop <<a href="mailto:knop@us.ibm.com" target="_blank">knop@us.ibm.com</a>> wrote:</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" ><div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10pt" ><div dir="ltr" >Ray,</div>
<div dir="ltr" > </div>
<div dir="ltr" >I wonder if you are hitting the problem which was fixed on the following APAR:</div>
<div dir="ltr" > </div>
<div dir="ltr" ><a href="https://www.ibm.com/support/pages/apar/IJ28891" target="_blank">https://www.ibm.com/support/pages/apar/IJ28891</a></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<div dir="ltr" > Felipe</div>
<div dir="ltr" > </div>
<div dir="ltr" >----<br>Felipe Knop <a href="mailto:knop@us.ibm.com" target="_blank">knop@us.ibm.com</a><br>GPFS Development and Security<br>IBM Systems<br>IBM Building 008<br>2455 South Rd, Poughkeepsie, NY 12601<br>(845) 433-9314 T/L 293-9314<br> </div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote dir="ltr" style="border-left:2px solid rgb(170,170,170);margin-left:5px;padding-left:5px;direction:ltr;margin-right:0px" >----- Original message -----<br>From: Ray Coetzee <<a href="mailto:coetzee.ray@gmail.com" target="_blank">coetzee.ray@gmail.com</a>><br>Sent by: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a><br>To: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>><br>Cc:<br>Subject: [EXTERNAL] [gpfsug-discuss] Kernel > 4.10, python >= 3.8 issue<br>Date: Wed, May 26, 2021 1:38 PM<br>
<div dir="ltr" ><div>Hello all</div>
<div> </div>
<div>I'd be interested to know if anyone else has experienced a problem with <span style="color:black;font-family:Arial,sans-serif;font-size:10pt" >Kernel > 4.10, python >= 3.8 and Spectrum Scale (5.0.5-2).</span></div>
<div> </div>
<div><p style="margin:0px;font-size:12pt;font-family:Calibri,sans-serif" ><span style="font-size:10pt;font-family:Arial,sans-serif;color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >We noticed that python shut.copy() is failing against a GPFS mount with:</span><br><br><span style="font-size:10pt;font-family:Arial,sans-serif;color:black" ><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >BlockingIOError: [Errno 11] Resource temporarily unavailable: 'test.file' -> 'test2.file'</span><br><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >To reproduce the error:</span><br><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >```</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >[user@</span></span>login01<span style="font-size:10pt;font-family:Arial,sans-serif;color:black" ><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >]$ module load python-3.8.9-gcc-9.3.0-soqwnzh</span><br><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >[</span></span> user<span style="font-size:10pt;font-family:Arial,sans-serif;color:black" ><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >@login01]$ truncate --size 640MB test.file</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >[</span></span> user<span style="font-size:10pt;font-family:Arial,sans-serif;color:black" ><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >@</span></span>login01<span style="font-size:10pt;font-family:Arial,sans-serif;color:black" ><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >]$ python3 -c "import shutil; shutil.copy('test.file', 'test2.file')"</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >Traceback (most recent call last):</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" > File "<string>", line 1, in <module></span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" > File "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py", line 418, in copy</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" > copyfile(src, dst, follow_symlinks=follow_symlinks)</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" > File "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py", line 275, in copyfile</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" > _fastcopy_sendfile(fsrc, fdst)</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" > File "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py", line 172, in _fastcopy_sendfile</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" > raise err</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" > File "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py", line 152, in _fastcopy_sendfile</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" > sent = os.sendfile(outfd, infd, offset, blocksize)</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >BlockingIOError: [Errno 11] Resource temporarily unavailable: 'test.file' -> 'test2.file'</span></span></p>
<p style="margin:0px;font-size:12pt;font-family:Calibri,sans-serif" ><span style="font-size:10pt;font-family:Arial,sans-serif;color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" > </span></p>
<p style="margin:0px;font-size:12pt;font-family:Calibri,sans-serif" ><span style="font-size:10pt;font-family:Arial,sans-serif;color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" > </span><span style="color:black;font-family:Arial,sans-serif;font-size:10pt" >Investigating into why this is happening revealed that:</span></p>
<p style="margin:0px;font-size:12pt;font-family:Calibri,sans-serif" ><br><span style="font-size:10pt;font-family:Arial,sans-serif;color:black" ><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >1. It is failing for python3.8 and above.</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >2. It is happening only a GPFS mount</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >3. It is happening with files whose size is multiple of 4KB (OS Page size)</span><br><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >Relevant links:</span></span><br><a href="https://bugs.python.org/issue43743" style="color:blue" target="_blank"><span style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(102,102,102);background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >https://bugs.python.org/issue43743</span></a><br><a href="https://www.ibm.com/support/pages/apar/IJ28891" style="color:blue" target="_blank"><span style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(102,102,102);background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >https://www.ibm.com/support/pages/apar/IJ28891</span></a><br><br><br><span style="font-size:10pt;font-family:Arial,sans-serif;color:black" ><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >Doing an strace revealed that at the lower level, it seems to be related to the Linux Syscall of “sendfile”, which seems to fail in these cases on GPFS.</span><br><br><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >Strace for a 4096 B file:</span><br><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >```</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >sendfile(4, 3, [0] => [4096], 8388608) = 4096</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >sendfile(4, 3, [4096], 8388608) = -1 EAGAIN (Resource temporarily unavailable)</span><br><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >```</span><br><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >The same file on other disk:</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >```</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >sendfile(4, 3, [0] => [4096], 8388608) = 4096</span><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >sendfile(4, 3, [4096], 8388608) = 0</span></span></p>
<p style="margin:0px;font-size:12pt;font-family:Calibri,sans-serif" > </p>
<p style="margin:0px;font-size:12pt;font-family:Calibri,sans-serif" ><span style="font-size:10pt;font-family:Arial,sans-serif;color:black" >IBM's "fix" for the problem of "</span><span style="color:rgb(50,50,50);font-family:ibm-plex-mono,Menlo,"DejaVu Sans Mono","Bitstream Vera Sans Mono",Courier,monospace;font-size:0.9375rem;white-space:pre-wrap" >Do not use a file size which that is a multiple </span><span style="color:rgb(50,50,50);font-family:ibm-plex-mono,Menlo,"DejaVu Sans Mono","Bitstream Vera Sans Mono",Courier,monospace;font-size:0.9375rem;white-space:pre-wrap" >of the page size.</span><span style="font-size:10pt;font-family:Arial,sans-serif;color:black" >" sounds really blas</span><span style="font-family:Arial,Helvetica,sans-serif;font-size:small" >é.</span></p>
<p style="margin:0px;font-size:12pt;font-family:Calibri,sans-serif" ><br><span style="font-size:10pt;font-family:Arial,sans-serif;color:black" ><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial" >```</span></span></p></div>
<div> </div>
<div><div dir="ltr" ><div dir="ltr" ><div dir="ltr" ><div dir="ltr" ><span style="font-size:small" ><font color="#888888" >Kind regards</font></span>
<div><p><span style="font-size:small" >Ray Coetzee</span><br> </p></div>
<div> </div></div></div></div></div></div></div>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> </font></div></blockquote>
<div dir="ltr" > </div></div></blockquote></div></div></blockquote>
<div dir="ltr" > </div></div><BR>