[gpfsug-discuss] Spectrum Scale pagepool size with RDMA

Thu Jul 23 08:09:17 BST 2020

Hi

What is the output for:
#mmlsconfig |grep -i verbs 
#ibstat

Regards

Yaron Daniel
 94 Em Ha'Moshavot Rd

Storage Architect ? IL Lab Services (Storage)
 Petach Tiqva, 49527
IBM Global Markets, Systems HW Sales
 Israel

Phone:
+972-3-916-5672

Fax:
+972-3-916-5672

Mobile:
+972-52-8395593

e-mail:
yard at il.ibm.com

Webex:            https://ibm.webex.com/meet/yard
IBM Israel

From:   Prasad Surampudi <prasad.surampudi at theatsgroup.com>
To:     "gpfsug-discuss at spectrumscale.org" 
<gpfsug-discuss at spectrumscale.org>
Date:   07/23/2020 03:34 AM
Subject:        [EXTERNAL] [gpfsug-discuss] Spectrum Scale pagepool size 
with RDMA
Sent by:        gpfsug-discuss-bounces at spectrumscale.org

Hi,

We have an ESS clusters with two CES nodes. The pagepool is set to 128 GB 
( Real Memory is 256 GB ) on both ESS NSD servers and CES nodes as well. 
Occasionally we see the mmfsd process memory usage reaches 90% on NSD 
servers and CES nodes and stays there until GPFS is recycled. I have 
couple of questions in this scenario:

 What are the general recommendations of pagepool size for nodes with RDMA 
enabled? On, IBM knowledge center for RDMA tuning says "If the GPFS 
pagepool is set to 32 GB, then the mapping of the RDMA for this pagepool 
must be at least 64 GB."  So, does this mean that the pagepool can't be 
more than half of real memory with RDMA enabled? Also, Is this the reason 
why mmfsd memory usage exceeds pagepool size and spikes to almost 90%?
If we dont want to see high mmfsd process memory usage on NSD/CES nodes, 
should we decrease the pagepool size?
Can we tune  log_num_mtt parameter to limit the memory usage? Currently 
its set to 0 for both NSD (ppc64_le) and CES (x86_64).
We also see messages like "Verbs RDMA disabled for xx.xx.xx.xx due to no 
matching port found" . Any idea what this message indicate? I dont see any 
Verbs RDMA enabled message after these warning messages. Does it get 
enabled automatically? 

Prasad Surampudi|Sr. Systems Engineer|ATS Group, LLC

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Bn1XE9uK2a9CZQ8qKnJE3Q&m=3V12EzdqYBk1P235cOvncsD-pOXNf5e5vPp85RnNhP8&s=XxlITEUK0nSjIyiu9XY1DEbYiVzVbp5XHcvQPfFJ2NY&e= 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20200723/ddd59a6a/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 1114 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20200723/ddd59a6a/attachment-0006.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 4105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20200723/ddd59a6a/attachment-0007.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 3847 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20200723/ddd59a6a/attachment-0016.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 4266 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20200723/ddd59a6a/attachment-0017.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 3747 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20200723/ddd59a6a/attachment-0018.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 3793 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20200723/ddd59a6a/attachment-0019.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 4301 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20200723/ddd59a6a/attachment-0020.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 3739 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20200723/ddd59a6a/attachment-0021.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 3855 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20200723/ddd59a6a/attachment-0022.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 4084 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20200723/ddd59a6a/attachment-0008.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 3776 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20200723/ddd59a6a/attachment-0023.jpe>