<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto">Thanks Ed,<div><br></div><div>The UQ team are well aware of the current limits published in the FAQ.</div><div><br></div><div>However the issue is not the number of physical nodes or the concurrent user sessions, but rather the number of SMB / NFS export mounts that Spectrum Scale supports from a single cluster or even remote mount protocol clusters is no longer enough for their research environment.</div><div><br></div><div>The current total number of Exports can not exceed 1000, which is an issue when they have multiple thousands of research project ID’s with users needing access to every project ID with its relevant security permissions.</div><div><br></div><div>Grouping Project ID’s under a single export isn’t a viable option as there is no simple way to identify which research group / user is going to request a new project ID, new project ID’s are automatically created and allocated when a request for storage allocation is fulfilled.</div><div><br></div><div>Projects ID’s (independent file sets) are published not only as SMB exports, but are also mounted using multiple AFM cache clusters to high performance instrument clusters, multiple HPC clusters or up to 5 different campus access points, including remote universities.</div><div><br></div><div>The data workflow is not a simple linear workflow</div><div>And the mixture of different types of users with requests for storage, and storage provisioning has resulted in the University creating their own provisioning portal which interacts with the Spectrum Scale data fabric (multiple Spectrum Scale clusters in single global namespace, connected via 100GB Ethernet over AFM) in multiple points to deliver the project ID provisioning at the relevant locations specified by the user / research group.</div><div><br></div><div>One point of data surfacing, in this data fabric, is the Spectrum Scale Protocols cluster that Les manages, which provides the central user access point via SMB or NFS, all research users across the university who want to access one or more of their storage allocations do so via the SMB / NFS mount points from this specific storage cluster.</div><div><br></div><div>Regards,</div><div><br></div><div><br></div><div>Andrew Beattie</div><div>File & Object Storage - Technical Lead</div><div>IBM Australia & New Zealand<br><br><div dir="ltr">Sent from my iPhone</div><div dir="ltr"><br><blockquote type="cite">On 11 Dec 2020, at 00:41, Edward Boyd <eboyd@us.ibm.com> wrote:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr">
<font face="Verdana,Arial,Helvetica,sans-serif" size="2"> <span><div>Please review the CES limits in the FAQ which states</div><div><br></div><div><div class="body"><dl class="ibm-padding-top-1"><dt class="dlterm"><a href="https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html?view=kc#gpfsclustersfaqAugust2016-gen4__maxprotoq"><strong>Q5.2:</strong></a></dt><dd class="dlentry"><strong>What are some scaling considerations for the
protocols function?</strong></dd><dt class="dlterm"><strong>A5.2:</strong></dt><dd class="dlentry">Scaling considerations for the protocols function include:
<ul class="ibm-colored-list ibm-textcolor-gray-80"><li>The number of protocol nodes.
<p>If you are using SMB
in any combination of other protocols you can configure only up to
16 protocol nodes. This is a hard limit and SMB cannot be enabled
if there are more protocol nodes. If only NFS and Object are enabled,
you can have 32 nodes configured as protocol nodes.</p></li><li>The number of client connections.
<p>A maximum of 3,000 SMB connections
is recommended per protocol node with a maximum of 20,000 SMB connections
per cluster. A maximum of 4,000 NFS connections per protocol node
is recommended. A maximum of 2,000 Object connections per protocol
nodes is recommended. The maximum number of connections depends on
the amount of memory configured and sufficient CPU. We recommend a
minimum of 64GB of memory for only Object or only NFS use cases. If
you have multiple protocols enabled or if you have SMB enabled we
recommend 128GB of memory on the system.</p></li></ul><div><a href="https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html?view=kc#maxproto" target="_blank">https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html?view=kc#maxproto</a><br></div></dd></dl></div></div>Edward L. Boyd ( Ed )<br>IBM Certified Client Technical Specialist, Level 2 Expert<br>Open Foundation, Master Certified Technical Specialist<br>IBM Systems, Storage Solutions<br>US Federal<br>407-271-9210 Office / Cell / Office / Text<br><a href="mailto:eboyd@us.ibm.com">eboyd@us.ibm.com</a> email</span><br><br><font size="2" face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" color="#000000"><font color="#990099"><a href="mailto:-----gpfsug-discuss-bounces@spectrumscale.org" target="_blank">-----gpfsug-discuss-bounces@spectrumscale.org</a> wrote: -----</font><div class="iNotesHistory" style="padding-left:5px;"><div style="padding-right:0px;padding-left:5px;border-left:solid black 2px;">To: <a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a><br>From: <a href="mailto:gpfsug-discuss-request@spectrumscale.org" target="_blank">gpfsug-discuss-request@spectrumscale.org</a><br>Sent by: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a><br>Date: 12/10/2020 07:00AM<br>Subject: [EXTERNAL] gpfsug-discuss Digest, Vol 107, Issue 13<br><br><div><font size="2" face="Courier New,Courier,monospace">Send gpfsug-discuss mailing list submissions to<br> <a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a><br><br>To subscribe or unsubscribe via the World Wide Web, visit<br> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> <br>or, via email, send a message with subject or body 'help' to<br> <a href="mailto:gpfsug-discuss-request@spectrumscale.org" target="_blank">gpfsug-discuss-request@spectrumscale.org</a><br><br>You can reach the person managing the list at<br> <a href="mailto:gpfsug-discuss-owner@spectrumscale.org" target="_blank">gpfsug-discuss-owner@spectrumscale.org</a><br><br>When replying, please edit your Subject line so it is more specific<br>than "Re: Contents of gpfsug-discuss digest..."<br><br><br>Today's Topics:<br><br> 1. Protocol limits (leslie elliott)<br> 2. Re: Protocol limits (Jan-Frode Myklebust)<br><br><br>----------------------------------------------------------------------<br><br>Message: 1<br>Date: Thu, 10 Dec 2020 08:45:22 +1000<br>From: leslie elliott <<a href="mailto:leslie.james.elliott@gmail.com" target="_blank">leslie.james.elliott@gmail.com</a>><br>To: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>><br>Subject: [gpfsug-discuss] Protocol limits<br>Message-ID:<br> <<a href="mailto:CANBv+tsnwzTH5796xMfpLmWc-aY5=kiHHLaacx-fzGdBLuPqgw@mail.gmail.com" target="_blank">CANBv+tsnwzTH5796xMfpLmWc-aY5=kiHHLaacx-fzGdBLuPqgw@mail.gmail.com</a>><br>Content-Type: text/plain; charset="utf-8"<br><br>hi all<br><br>we run a large number of shares from CES servers connected to a single<br>scale cluster<br>we understand the current supported limit is 1000 SMB shares, we run the<br>same number of NFS shares<br><br>we also understand that using external CES cluster to increase that limit<br>is not supported based on the documentation, we use the same authentication<br>for all shares, we do have additional use cases for sharing where this<br>pathway would be attractive going forward<br><br>so the question becomes if we need to run 20000 SMB and NFS shares off a<br>scale cluster is there any hardware design we can use to do this whilst<br>maintaining support<br><br>I have submitted a support request to ask if this can be done but thought I<br>would ask the collective good if this has already been solved<br><br>thanks<br><br>leslie<br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20201210/a460a862/attachment-0001.html">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20201210/a460a862/attachment-0001.html</a> ><br><br>------------------------------<br><br>Message: 2<br>Date: Thu, 10 Dec 2020 00:21:03 +0100<br>From: Jan-Frode Myklebust <<a href="mailto:janfrode@tanso.net" target="_blank">janfrode@tanso.net</a>><br>To: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>><br>Subject: Re: [gpfsug-discuss] Protocol limits<br>Message-ID:<br> <<a href="mailto:CAHwPatj8xi5Bez7M+GpqAGuOXy_P+qW87MJ4UF7Z2NxR1aeHhQ@mail.gmail.com" target="_blank">CAHwPatj8xi5Bez7M+GpqAGuOXy_P+qW87MJ4UF7Z2NxR1aeHhQ@mail.gmail.com</a>><br>Content-Type: text/plain; charset="utf-8"<br><br>My understanding of these limits are that they are to limit the<br>configuration files from becoming too large, which makes<br>changing/processing them somewhat slow.<br><br>For SMB shares, you might be able to limit the number of configured shares<br>by using wildcards in the config (%U). These wildcarded entries counts as<br>one share.. Don?t know if simimar tricks can be done for NFS..<br><br><br><br> -jf<br><br>ons. 9. des. 2020 kl. 23:45 skrev leslie elliott <<br><a href="mailto:leslie.james.elliott@gmail.com" target="_blank">leslie.james.elliott@gmail.com</a>>:<br><br>><br>> hi all<br>><br>> we run a large number of shares from CES servers connected to a single<br>> scale cluster<br>> we understand the current supported limit is 1000 SMB shares, we run the<br>> same number of NFS shares<br>><br>> we also understand that using external CES cluster to increase that limit<br>> is not supported based on the documentation, we use the same authentication<br>> for all shares, we do have additional use cases for sharing where this<br>> pathway would be attractive going forward<br>><br>> so the question becomes if we need to run 20000 SMB and NFS shares off a<br>> scale cluster is there any hardware design we can use to do this whilst<br>> maintaining support<br>><br>> I have submitted a support request to ask if this can be done but thought<br>> I would ask the collective good if this has already been solved<br>><br>> thanks<br>><br>> leslie<br>> _______________________________________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at spectrumscale.org<br>> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> <br>><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20201210/4744cdc0/attachment-0001.html">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20201210/4744cdc0/attachment-0001.html</a> ><br><br>------------------------------<br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> <br><br><br>End of gpfsug-discuss Digest, Vol 107, Issue 13<br>***********************************************<br><br></font></div></div></div></font></font><br>
<pre>_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
</pre><br></div></blockquote></div><BR>
</body></html>