<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" >dear colleagues,</div>
<div dir="ltr" >I don't think, that we can cover a specific customer setup .. .you may open a PMR to address your issues or ask your local IBM /business partner account</div>
<div dir="ltr" >but to answer your questions...</div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<div dir="ltr" >(1) pagepool can be up to 75% (by default) and it works well with RDMA , even for pagepool > 64GB</div>
<div dir="ltr" >(2) the reason for high load is hard to analyze by email ... and I doubt, that reducing this high load will be done by lowering pagepool</div>
<div dir="ltr" >(3) should nt be to tune any more... since we have recent MLX adapter/Mofed, this should be obsolete</div>
<div dir="ltr" >(4) most likely on that node this verbsPort is not existing or active (check by mmlsconfig and ibdev2netdev)</div>
<div dir="ltr" ><div dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" ><div> </div>
<div> </div>
<div>Mit freundlichen Grüßen / Kind regards</div>
<div style="font-size: 8pt; font-family: sans-serif; margin-top: 10px;" ><div><div> <br>Olaf Weiser<br> <br>IBM Systems, SpectrumScale Client Adoption<br>-------------------------------------------------------------------------------------------------------------------------------------------<br>IBM Deutschland<br>IBM Allee 1<br>71139 Ehningen<br>Phone: +49-170-579-44-66<br>E-Mail: olaf.weiser@de.ibm.com<br>-------------------------------------------------------------------------------------------------------------------------------------------<br>IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter<br>Geschäftsführung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert Janzen, Markus Koerner, Christian Noll, Nicole Reimer<br>Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940</div>
<div> </div></div></div></div></div></div></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" data-history-expanded="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: "Frederick Stock" <stockf@us.ibm.com><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: gpfsug-discuss@spectrumscale.org<br>Cc: gpfsug-discuss@spectrumscale.org<br>Subject: [EXTERNAL] Re: [gpfsug-discuss] Spectrum Scale pagepool size with RDMA<br>Date: Thu, Jul 23, 2020 1:15 PM<br>
<div dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:12pt" ><div dir="ltr" >And what version of ESS/Scale are you running on your systems (mmdiag --version)?</div>
<div dir="ltr" ><div dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" ><br><font size="2" face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" ><span style="font-size:1.143em;" >Fred<br>__________________________________________________<br>Fred Stock | IBM Pittsburgh Lab | 720-430-8821<br>stockf@us.ibm.com</span></font></div></div></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: "Yaron Daniel" <YARD@il.ibm.com><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Cc:<br>Subject: [EXTERNAL] Re: [gpfsug-discuss] Spectrum Scale pagepool size with RDMA<br>Date: Thu, Jul 23, 2020 3:09 AM<br> <br><span style=" font-size:10pt;font-family:sans-serif" >Hi</span><br><br><br><span style=" font-size:10pt;font-family:sans-serif" >What is the output for:<br>#mmlsconfig |grep -i verbs </span><br><span style=" font-size:10pt;font-family:sans-serif" >#ibstat</span><br><br><span style=" font-size:9pt;font-family:Arial" > </span><br><span style=" font-size:10pt;font-family:Arial" >Regards</span><br><span style=" font-size:9pt;font-family:Arial" > </span>
<table style="border-collapse:collapse;" width="780" > <tbody> <tr height="8" > <td colspan="4" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="780" > <div align="center" > <hr noshade="noshade" ></div> <br> <span style=" font-size:1pt;font-family:Arial" > </span></td> </tr> <tr height="8" > <td colspan="4" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="780" ><span style=" font-size:1pt;font-family:Arial" > </span></td> </tr> <tr height="8" > <td colspan="2" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="516" ><span style=" font-size:10pt;color:blue;font-family:Arial" ><b>Yaron Daniel</b></span></td> <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="168" ><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" > 94 Em Ha'Moshavot Rd</span></td> <td rowspan="3" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="96" > <div align="right" ><img src="cid:_1_08807E58088078CC00274C4DC22585AE" style="border:0px solid;" align="bottom" ></div> </td> </tr> <tr height="8" > <td colspan="2" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="516" ><span style=" font-size:10pt;color:blue;font-family:Arial" ><b>Storage Architect – IL Lab Services (Storage)</b></span></td> <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="168" ><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" > Petach Tiqva, 49527</span></td> </tr> <tr height="8" > <td colspan="2" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="516" ><span style=" font-size:10pt;color:blue;font-family:Arial" ><b>IBM Global Markets, Systems HW Sales</b></span></td> <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="168" ><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" > Israel</span></td> </tr> <tr height="8" > <td colspan="2" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="516" ><span style=" font-size:10pt;color:blue;font-family:Arial" ><b> </b></span></td> <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="168" ><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" > </span></td> <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="96" ><span style=" font-size:9pt;font-family:Arial" > </span></td> </tr> <tr height="8" > <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="90" ><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" >Phone:</span></td> <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="426" ><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" >+972-3-916-5672</span></td> <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="168" ><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" > </span></td> <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="96" ><span style=" font-size:10pt" > </span></td> </tr> <tr height="8" > <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="90" ><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" >Fax:</span></td> <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="426" ><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" >+972-3-916-5672</span></td> <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="168" ><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" > </span></td> <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="96" ><span style=" font-size:10pt" > </span></td> </tr> <tr height="8" > <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="90" ><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" >Mobile:</span></td> <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="426" ><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" >+972-52-8395593</span></td> <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="168" ><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" > </span></td> <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="96" ><span style=" font-size:10pt" > </span></td> </tr> <tr height="8" > <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="90" ><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" >e-mail:</span></td> <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="426" ><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" >yard@il.ibm.com</span></td> <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="168" ><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" > </span></td> <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="96" ><span style=" font-size:10pt" > </span></td> </tr> <tr height="8" > <td colspan="2" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="516" ><a href="Webex:%20%20%20%20%20%20%20%20%20%20%20%20%20https://ibm.webex.com/meet/yard%0dIBM%20Israel" target="_blank"><span style=" font-size:12pt;color:#5f5f5f;font-family:Times New Roman" >Webex: https://ibm.webex.com/meet/yard</span></a><br> <a href="Webex:%20%20%20%20%20%20%20%20%20%20%20%20%20https://ibm.webex.com/meet/yard%0dIBM%20Israel" target="_blank"><span style=" font-size:10pt;color:blue;font-family:Arial" ><u>IBM Israel</u></span></a></td> <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="168" > </td> <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="96" ><span style=" font-size:10pt" > </span></td> </tr> <tr height="8" > <td colspan="4" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="780" ><span style=" font-size:9pt;color:#5f5f5f;font-family:Arial" > </span></td> </tr> <tr height="8" > <td colspan="4" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="780" ><span style=" font-size:9pt;color:#5f5f5f;font-family:Arial" > </span></td> </tr> </tbody></table>
<p style="margin-top:0px;margin-Bottom:0px" > </p><br><span style=" font-size:12pt" > </span><img alt="IBM Storage for Cyber Resiliency & Modern Data Protection V1" src="cid:_1_0E57D6F80E57D2E000274C4DC22585AE" style="border:0px solid;" ><img src="cid:_2_0E57D9040E57D2E000274C4DC22585AE" style="border:0px solid;" ><img src="cid:_2_0E57DB100E57D2E000274C4DC22585AE" style="border:0px solid;" ><img src="cid:_2_0E57DD1C0E57D2E000274C4DC22585AE" style="border:0px solid;" ><img src="cid:_2_0E57DF280E57D2E000274C4DC22585AE" style="border:0px solid;" ><br><img src="cid:_2_0E57E5AC0E57E19400274C4DC22585AE" style="border:0px solid;" ><img src="cid:_2_0E58E4480E57E19400274C4DC22585AE" style="border:0px solid;" ><img src="cid:_2_0E58E6540E57E19400274C4DC22585AE" style="border:0px solid;" ><span style=" font-size:12pt" > </span><img alt="IBM Storage and Cloud Essentials" src="cid:_1_0E58E8600E57E19400274C4DC22585AE" style="border:0px solid;" ><img src="cid:_2_0E58EA6C0E57E19400274C4DC22585AE" style="border:0px solid;" ><br><br><br><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif" >From: </span><span style=" font-size:9pt;font-family:sans-serif" >Prasad Surampudi <prasad.surampudi@theatsgroup.com></span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif" >To: </span><span style=" font-size:9pt;font-family:sans-serif" >"gpfsug-discuss@spectrumscale.org" <gpfsug-discuss@spectrumscale.org></span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif" >Date: </span><span style=" font-size:9pt;font-family:sans-serif" >07/23/2020 03:34 AM</span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif" >Subject: </span><span style=" font-size:9pt;font-family:sans-serif" >[EXTERNAL] [gpfsug-discuss] Spectrum Scale pagepool size with RDMA</span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif" >Sent by: </span><span style=" font-size:9pt;font-family:sans-serif" >gpfsug-discuss-bounces@spectrumscale.org</span>
<hr noshade="noshade" ><br><br><br><span style=" font-size:12pt;font-family:Calibri" >Hi,</span><br><br><span style=" font-size:12pt;font-family:Calibri" >We have an ESS clusters with two CES nodes. The pagepool is set to 128 GB ( Real Memory is 256 GB ) on both ESS NSD servers and CES nodes as well. Occasionally we see the mmfsd process memory usage reaches 90% on NSD servers and CES nodes and stays there until GPFS is recycled. I have couple of questions in this scenario:</span>
<ol> <li value="1" ><span style=" font-size:12pt;font-family:Calibri" > What are the general recommendations of pagepool size for nodes with RDMA enabled? On, IBM knowledge center for RDMA tuning says "</span><span style=" font-size:12pt;color:#2f2f2f;font-family:Arial" >If the GPFS pagepool is set to 32 GB, then the mapping of the RDMA for this pagepool must be at least 64 GB." So, does this mean that the pagepool can't be more than half of real memory with RDMA enabled? Also, Is this the reason why mmfsd memory usage exceeds pagepool size and spikes to almost 90%?</span></li> <li value="2" ><span style=" font-size:12pt;color:#2f2f2f;font-family:Arial" >If we dont want to see high mmfsd process memory usage on NSD/CES nodes, should we decrease the pagepool size?</span></li> <li value="3" ><span style=" font-size:12pt;color:#2f2f2f;font-family:Arial" >Can we tune </span><span style=" font-size:12pt;color:#2f2f2f;font-family:Courier" >log_num_mtt parameter to limit the memory usage? Currently its set to 0 for both NSD (ppc64_le) and CES (x86_64).</span></li> <li value="4" ><span style=" font-size:12pt;color:#2f2f2f;font-family:Arial" >We also see messages like "Verbs RDMA disabled for xx.xx.xx.xx due to no matching port found" . Any idea what this message indicate? I dont see any Verbs RDMA enabled message after these warning messages. Does it get enabled automatically? </span></li></ol>
<p style="margin-top:0px;margin-Bottom:0px" ><span style=" font-size:11pt;color:#2020a0;font-family:Tahoma" ><b>Prasad Surampudi</b></span><span style=" font-size:9pt;color:#ff8141;font-family:Tahoma" >|</span><span style=" font-size:9pt;color:#000080;font-family:Tahoma" >Sr. Systems Engineer</span><span style=" font-size:9pt;color:#ff8100;font-family:Tahoma" >|</span><a href="http://www.theatsgroup.com/" target="_blank"><span style=" font-size:9pt;color:#0062e1;font-family:Tahoma" ><u>ATS Group, LLC</u></span></a></p>
<p style="margin-top:0px;margin-Bottom:0px" > </p><br><br><tt><span style=" font-size:10pt" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</span></tt><br><tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></tt><br><br><br>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> </font></div></blockquote>
<div dir="ltr" > </div></div>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> </font></div></blockquote>
<div dir="ltr" > </div></div></div>
<BR>