<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div style="font-family:arial,sans-serif">Hi Yaron,</div>
<div style="font-family:arial,sans-serif"><br>
</div>
<div style="font-family:arial,sans-serif">Please see the outputs of mmlsconfig and ibstat below:</div>
<div style="font-family:arial,sans-serif"><b><br>
</b></div>
<div style="font-family:arial,sans-serif"><b>sudo /usr/lpp/mmfs/bin/mmlsconfig |grep -i verbs</b><br>
<br>
</div>
<div style="font-family:arial,sans-serif">verbsRdmasPerNode 192<br>
verbsRdma enable<br>
verbsRdmaSend yes<br>
verbsRdmasPerConnection 48<br>
verbsRdmasPerConnection 16<br>
verbsPorts mlx5_4/1/1 mlx5_5/1/2<br>
verbsPorts mlx4_0/1/0 mlx4_0/2/0<br>
verbsPorts mlx5_0/1/1 mlx5_1/1/2<br>
verbsPorts mlx5_0/1/1 mlx5_2/1/2<br>
verbsPorts mlx5_2/1/1 mlx5_3/1/2<br>
</div>
<div style="font-family:arial,sans-serif"><br>
</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<p><span><span style="font-family: arial, sans-serif"><b>ibstat output on NSD server:</b></span></span></p>
<p><span><span style="font-family: arial, sans-serif"><b><br>
</b></span></span></p>
CA 'mlx5_0'<br>
CA type: MT4115<br>
Number of ports: 1<br>
Firmware version: 12.25.1020<br>
Hardware version: 0<br>
Node GUID: 0x506b4b03000fdb74<br>
System image GUID: 0x506b4b03000fdb74<br>
Port 1:<br>
State: Active<br>
Physical state: LinkUp<br>
Rate: 100<br>
Base lid: 0<br>
LMC: 0<br>
SM lid: 0<br>
Capability mask: 0x00010000<br>
Port GUID: 0x526b4bfffe0fdb74<br>
Link layer: Ethernet<br>
CA 'mlx5_1'<br>
CA type: MT4115<br>
Number of ports: 1<br>
Firmware version: 12.25.1020<br>
Hardware version: 0<br>
Node GUID: 0x506b4b03000fdb75<br>
System image GUID: 0x506b4b03000fdb74<br>
Port 1:<br>
State: Down<br>
Physical state: Disabled<br>
Rate: 40<br>
Base lid: 0<br>
LMC: 0<br>
SM lid: 0<br>
Capability mask: 0x00010000<br>
Port GUID: 0x526b4bfffe0fdb75<br>
Link layer: Ethernet<br>
CA 'mlx5_2'<br>
CA type: MT4115<br>
Number of ports: 1<br>
Firmware version: 12.25.1020<br>
Hardware version: 0<br>
Node GUID: 0xec0d9a0300a7e928<br>
System image GUID: 0xec0d9a0300a7e928<br>
Port 1:<br>
State: Active<br>
Physical state: LinkUp<br>
Rate: 100<br>
Base lid: 0<br>
LMC: 0<br>
SM lid: 0<br>
Capability mask: 0x00010000<br>
Port GUID: 0x526b4bfffe0fdb74<br>
Link layer: Ethernet<br>
CA 'mlx5_3'<br>
CA type: MT4115<br>
Number of ports: 1<br>
Firmware version: 12.25.1020<br>
Hardware version: 0<br>
Node GUID: 0xec0d9a0300a7e929<br>
System image GUID: 0xec0d9a0300a7e928<br>
Port 1:<br>
State: Down<br>
Physical state: Disabled<br>
Rate: 40<br>
Base lid: 0<br>
LMC: 0<br>
SM lid: 0<br>
Capability mask: 0x00010000<br>
Port GUID: 0xee0d9afffea7e929<br>
Link layer: Ethernet<br>
CA 'mlx5_4'<br>
CA type: MT4115<br>
Number of ports: 1<br>
Firmware version: 12.25.1020<br>
Hardware version: 0<br>
Node GUID: 0xec0d9a0300da5f92<br>
System image GUID: 0xec0d9a0300da5f92<br>
Port 1:<br>
State: Active<br>
Physical state: LinkUp<br>
Rate: 100<br>
Base lid: 13<br>
LMC: 0<br>
SM lid: 1<br>
Capability mask: 0x2651e848<br>
Port GUID: 0xec0d9a0300da5f92<br>
Link layer: InfiniBand<br>
CA 'mlx5_5'<br>
CA type: MT4115<br>
Number of ports: 1<br>
Firmware version: 12.25.1020<br>
Hardware version: 0<br>
Node GUID: 0xec0d9a0300da5f93<br>
System image GUID: 0xec0d9a0300da5f92<br>
Port 1:<br>
State: Active<br>
Physical state: LinkUp<br>
Rate: 100<br>
Base lid: 6<br>
LMC: 0<br>
SM lid: 1<br>
Capability mask: 0x2651e848<br>
Port GUID: 0xec0d9a0300da5f93<br>
Link layer: InfiniBand</div>
<div dir="ltr"><br>
<p><span style="font-family: arial, sans-serif"><b>ibstat output on CES server:</b></span></p>
<p><span style="font-family: arial, sans-serif"><b><br>
</b></span></p>
<p><span style="font-family: arial, sans-serif">CA 'mlx5_0'<br>
CA type: MT4115<br>
Number of ports: 1<br>
Firmware version: 12.22.4030<br>
Hardware version: 0<br>
Node GUID: 0xb88303ffff5ec6ec<br>
System image GUID: 0xb88303ffff5ec6ec<br>
Port 1:<br>
State: Active<br>
Physical state: LinkUp<br>
Rate: 100<br>
Base lid: 9<br>
LMC: 0<br>
SM lid: 1<br>
Capability mask: 0x2651e848<br>
Port GUID: 0xb88303ffff5ec6ec<br>
Link layer: InfiniBand<br>
CA 'mlx5_1'<br>
CA type: MT4115<br>
Number of ports: 1<br>
Firmware version: 12.22.4030<br>
Hardware version: 0<br>
Node GUID: 0xb88303ffff5ec6ed<br>
System image GUID: 0xb88303ffff5ec6ec<br>
Port 1:<br>
State: Active<br>
Physical state: LinkUp<br>
Rate: 100<br>
Base lid: 12<br>
LMC: 0<br>
SM lid: 1<br>
Capability mask: 0x2651e848<br>
Port GUID: 0xb88303ffff5ec6ed<br>
Link layer: InfiniBand</span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div>
<div></div>
<div></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<div style="color:rgb(34,34,34); font-family:Arial,Helvetica,sans-serif; font-size:small; text-align:start; background-color:rgb(255,255,255)">
<div>
<div>
<p><b><span style="font-size:10.5pt; font-family:Tahoma,sans-serif; color:rgb(51,51,153)">Prasad Surampudi</span></b><span style="font-size:9pt; font-family:Tahoma,sans-serif; color:rgb(247,150,70)">|</span><span style="font-size:9pt; font-family:Tahoma,sans-serif; color:navy">Sr.
Systems Engineer</span><span style="font-size:9pt; font-family:Tahoma,sans-serif; color:rgb(255,153,0)">|</span><span style="font-size:9pt; font-family:Tahoma,sans-serif; color:rgb(51,51,153)"><a href="http://www.theatsgroup.com/" target="_blank" style="color:rgb(17,85,204)"><span style="color:purple">ATS
Group, LLC</span></a></span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p><br>
</p>
</div>
</div>
</div>
<br>
</div>
</div>
</div>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> gpfsug-discuss-bounces@spectrumscale.org <gpfsug-discuss-bounces@spectrumscale.org> on behalf of gpfsug-discuss-request@spectrumscale.org
<gpfsug-discuss-request@spectrumscale.org><br>
<b>Sent:</b> Thursday, July 23, 2020 3:09 AM<br>
<b>To:</b> gpfsug-discuss@spectrumscale.org <gpfsug-discuss@spectrumscale.org><br>
<b>Subject:</b> gpfsug-discuss Digest, Vol 102, Issue 12</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">Send gpfsug-discuss mailing list submissions to<br>
gpfsug-discuss@spectrumscale.org<br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
or, via email, send a message with subject or body 'help' to<br>
gpfsug-discuss-request@spectrumscale.org<br>
<br>
You can reach the person managing the list at<br>
gpfsug-discuss-owner@spectrumscale.org<br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of gpfsug-discuss digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Spectrum Scale pagepool size with RDMA (Prasad Surampudi)<br>
2. Re: Spectrum Scale pagepool size with RDMA (Yaron Daniel)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Thu, 23 Jul 2020 00:34:02 +0000<br>
From: Prasad Surampudi <prasad.surampudi@theatsgroup.com><br>
To: "gpfsug-discuss@spectrumscale.org"<br>
<gpfsug-discuss@spectrumscale.org><br>
Subject: [gpfsug-discuss] Spectrum Scale pagepool size with RDMA<br>
Message-ID:<br>
<MN2PR13MB2976E6B29CDE6EC77B9B2B529E790@MN2PR13MB2976.namprd13.prod.outlook.com><br>
<br>
Content-Type: text/plain; charset="iso-8859-1"<br>
<br>
Hi,<br>
<br>
We have an ESS clusters with two CES nodes. The pagepool is set to 128 GB ( Real Memory is 256 GB ) on both ESS NSD servers and CES nodes as well. Occasionally we see the mmfsd process memory usage reaches 90% on NSD servers and CES nodes and stays there until
GPFS is recycled. I have couple of questions in this scenario:<br>
<br>
<br>
1. What are the general recommendations of pagepool size for nodes with RDMA enabled? On, IBM knowledge center for RDMA tuning says "If the GPFS pagepool is set to 32 GB, then the mapping of the RDMA for this pagepool must be at least 64 GB." So, does
this mean that the pagepool can't be more than half of real memory with RDMA enabled? Also, Is this the reason why mmfsd memory usage exceeds pagepool size and spikes to almost 90%?<br>
2. If we dont want to see high mmfsd process memory usage on NSD/CES nodes, should we decrease the pagepool size?<br>
3. Can we tune log_num_mtt parameter to limit the memory usage? Currently its set to 0 for both NSD (ppc64_le) and CES (x86_64).<br>
4. We also see messages like "Verbs RDMA disabled for xx.xx.xx.xx due to no matching port found" . Any idea what this message indicate? I dont see any Verbs RDMA enabled message after these warning messages. Does it get enabled automatically?<br>
<br>
<br>
Prasad Surampudi|Sr. Systems Engineer|ATS Group, LLC<<a href="http://www.theatsgroup.com/">http://www.theatsgroup.com/</a>><br>
<br>
<br>
<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200723/e2eee8fe/attachment-0001.html">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200723/e2eee8fe/attachment-0001.html</a>><br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Thu, 23 Jul 2020 10:09:17 +0300<br>
From: "Yaron Daniel" <YARD@il.ibm.com><br>
To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
Subject: Re: [gpfsug-discuss] Spectrum Scale pagepool size with RDMA<br>
Message-ID:<br>
<OFBF335AE3.531C9498-ONC22585AE.00273AFB-C22585AE.00274DB2@notes.na.collabserv.com><br>
<br>
Content-Type: text/plain; charset="iso-8859-1"<br>
<br>
Hi<br>
<br>
<br>
What is the output for:<br>
#mmlsconfig |grep -i verbs <br>
#ibstat<br>
<br>
<br>
Regards<br>
<br>
<br>
<br>
<br>
<br>
Yaron Daniel<br>
94 Em Ha'Moshavot Rd<br>
<br>
Storage Architect ? IL Lab Services (Storage)<br>
Petach Tiqva, 49527<br>
IBM Global Markets, Systems HW Sales<br>
Israel<br>
<br>
<br>
<br>
Phone:<br>
+972-3-916-5672<br>
<br>
<br>
Fax:<br>
+972-3-916-5672<br>
<br>
<br>
Mobile:<br>
+972-52-8395593<br>
<br>
<br>
e-mail:<br>
yard@il.ibm.com<br>
<br>
<br>
Webex: <a href="https://ibm.webex.com/meet/yard">https://ibm.webex.com/meet/yard</a><br>
IBM Israel<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
From: Prasad Surampudi <prasad.surampudi@theatsgroup.com><br>
To: "gpfsug-discuss@spectrumscale.org" <br>
<gpfsug-discuss@spectrumscale.org><br>
Date: 07/23/2020 03:34 AM<br>
Subject: [EXTERNAL] [gpfsug-discuss] Spectrum Scale pagepool size <br>
with RDMA<br>
Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>
<br>
<br>
<br>
Hi,<br>
<br>
We have an ESS clusters with two CES nodes. The pagepool is set to 128 GB <br>
( Real Memory is 256 GB ) on both ESS NSD servers and CES nodes as well. <br>
Occasionally we see the mmfsd process memory usage reaches 90% on NSD <br>
servers and CES nodes and stays there until GPFS is recycled. I have <br>
couple of questions in this scenario:<br>
<br>
What are the general recommendations of pagepool size for nodes with RDMA <br>
enabled? On, IBM knowledge center for RDMA tuning says "If the GPFS <br>
pagepool is set to 32 GB, then the mapping of the RDMA for this pagepool <br>
must be at least 64 GB." So, does this mean that the pagepool can't be <br>
more than half of real memory with RDMA enabled? Also, Is this the reason <br>
why mmfsd memory usage exceeds pagepool size and spikes to almost 90%?<br>
If we dont want to see high mmfsd process memory usage on NSD/CES nodes, <br>
should we decrease the pagepool size?<br>
Can we tune log_num_mtt parameter to limit the memory usage? Currently <br>
its set to 0 for both NSD (ppc64_le) and CES (x86_64).<br>
We also see messages like "Verbs RDMA disabled for xx.xx.xx.xx due to no <br>
matching port found" . Any idea what this message indicate? I dont see any <br>
Verbs RDMA enabled message after these warning messages. Does it get <br>
enabled automatically? <br>
<br>
Prasad Surampudi|Sr. Systems Engineer|ATS Group, LLC<br>
<br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org<br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Bn1XE9uK2a9CZQ8qKnJE3Q&m=3V12EzdqYBk1P235cOvncsD-pOXNf5e5vPp85RnNhP8&s=XxlITEUK0nSjIyiu9XY1DEbYiVzVbp5XHcvQPfFJ2NY&e=">https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Bn1XE9uK2a9CZQ8qKnJE3Q&m=3V12EzdqYBk1P235cOvncsD-pOXNf5e5vPp85RnNhP8&s=XxlITEUK0nSjIyiu9XY1DEbYiVzVbp5XHcvQPfFJ2NY&e=</a>
<br>
<br>
<br>
<br>
<br>
<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200723/ddd59a6a/attachment.html">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200723/ddd59a6a/attachment.html</a>><br>
-------------- next part --------------<br>
A non-text attachment was scrubbed...<br>
Name: not available<br>
Type: image/gif<br>
Size: 1114 bytes<br>
Desc: not available<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200723/ddd59a6a/attachment.gif">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200723/ddd59a6a/attachment.gif</a>><br>
-------------- next part --------------<br>
A non-text attachment was scrubbed...<br>
Name: not available<br>
Type: image/gif<br>
Size: 4105 bytes<br>
Desc: not available<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200723/ddd59a6a/attachment-0001.gif">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200723/ddd59a6a/attachment-0001.gif</a>><br>
-------------- next part --------------<br>
A non-text attachment was scrubbed...<br>
Name: not available<br>
Type: image/jpeg<br>
Size: 3847 bytes<br>
Desc: not available<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200723/ddd59a6a/attachment.jpe">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200723/ddd59a6a/attachment.jpe</a>><br>
-------------- next part --------------<br>
A non-text attachment was scrubbed...<br>
Name: not available<br>
Type: image/jpeg<br>
Size: 4266 bytes<br>
Desc: not available<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200723/ddd59a6a/attachment-0001.jpe">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200723/ddd59a6a/attachment-0001.jpe</a>><br>
-------------- next part --------------<br>
A non-text attachment was scrubbed...<br>
Name: not available<br>
Type: image/jpeg<br>
Size: 3747 bytes<br>
Desc: not available<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200723/ddd59a6a/attachment-0002.jpe">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200723/ddd59a6a/attachment-0002.jpe</a>><br>
-------------- next part --------------<br>
A non-text attachment was scrubbed...<br>
Name: not available<br>
Type: image/jpeg<br>
Size: 3793 bytes<br>
Desc: not available<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200723/ddd59a6a/attachment-0003.jpe">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200723/ddd59a6a/attachment-0003.jpe</a>><br>
-------------- next part --------------<br>
A non-text attachment was scrubbed...<br>
Name: not available<br>
Type: image/jpeg<br>
Size: 4301 bytes<br>
Desc: not available<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200723/ddd59a6a/attachment-0004.jpe">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200723/ddd59a6a/attachment-0004.jpe</a>><br>
-------------- next part --------------<br>
A non-text attachment was scrubbed...<br>
Name: not available<br>
Type: image/jpeg<br>
Size: 3739 bytes<br>
Desc: not available<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200723/ddd59a6a/attachment-0005.jpe">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200723/ddd59a6a/attachment-0005.jpe</a>><br>
-------------- next part --------------<br>
A non-text attachment was scrubbed...<br>
Name: not available<br>
Type: image/jpeg<br>
Size: 3855 bytes<br>
Desc: not available<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200723/ddd59a6a/attachment-0006.jpe">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200723/ddd59a6a/attachment-0006.jpe</a>><br>
-------------- next part --------------<br>
A non-text attachment was scrubbed...<br>
Name: not available<br>
Type: image/gif<br>
Size: 4084 bytes<br>
Desc: not available<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200723/ddd59a6a/attachment-0002.gif">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200723/ddd59a6a/attachment-0002.gif</a>><br>
-------------- next part --------------<br>
A non-text attachment was scrubbed...<br>
Name: not available<br>
Type: image/jpeg<br>
Size: 3776 bytes<br>
Desc: not available<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200723/ddd59a6a/attachment-0007.jpe">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200723/ddd59a6a/attachment-0007.jpe</a>><br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
<br>
<br>
End of gpfsug-discuss Digest, Vol 102, Issue 12<br>
***********************************************<br>
</div>
</span></font></div>
</body>
</html>