<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
</head>
<body>
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr">
<p>Hi,</p>
<p><br>
</p>
<p>Nice ! didn't pay attention at the revision and the spreadsheet. If someone still have a copy somewhere it could be useful, Google didn't help :(</p>
<p><span style="font-size: 12pt;"><br>
</span></p>
<p><font size="3">We will follow your advise and start with 3 protocol nodes </font>equipped<font size="3"> with 128GB memory, 2 x 12 cores (maybe E5-2680 or E5-2670).</font><br>
</p>
<p><font size="3"><br>
</font></p>
<p><font size="3">From what I read, NFS-Ganesha mainly depend of the hardware, Linux on a SSD should be a big plus in our case.</font></p>
<p><font size="3"><br>
</font></p>
<p><font size="3">Best,</font></p>
<p><font size="3">Frank</font></p>
<p><font size="3"></font></p>
<p><font size="3"><br>
</font></p>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Jan-Frode Myklebust <janfrode@tanso.net><br>
<b>Sent:</b> Sunday, April 23, 2017 12:07:38 PM<br>
<b>To:</b> Frank Tower; gpfsug-discuss@spectrumscale.org<br>
<b>Subject:</b> Re: [gpfsug-discuss] Protocol node recommendations</font>
<div> </div>
</div>
<div>The protocol sizing tool should be available from <a href="https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/Sizing%20Guidance%20for%20Protocol%20Node/version/70a4c7c0-a5c6-4dde-b391-8f91c542dd7d">
https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/Sizing%20Guidance%20for%20Protocol%20Node/version/70a4c7c0-a5c6-4dde-b391-8f91c542dd7d</a> , but I'm getting 404 now.<br>
<br>
I think 128GB should be enough for both protocols on same nodes, and I think your 3 node suggestion is best. Better load sharing with not dedicating subset of nodes to each protocol.<br>
<br>
<br>
<br>
-jf<br>
<div class="gmail_quote">
<div dir="ltr">lør. 22. apr. 2017 kl. 21.22 skrev Frank Tower <<a href="mailto:frank.tower@outlook.com">frank.tower@outlook.com</a>>:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div id="m_378511725885416141divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr">
<p>Hi,</p>
<p><br>
</p>
<p>Thank for the recommendations. </p>
<p>Now we deal with the situation of: </p>
<p><br>
</p>
<p>- take 3 nodes with round robin DNS that handle both protocols</p>
<p>- take 4 nodes, split CIFS and NFS, still use round robin DNS for CIFS and NFS services.</p>
<p><br>
</p>
<p>Regarding your recommendations, 256GB memory node could be a plus if we mix both protocols for such case.</p>
<p><br>
</p>
<p>Is the spreadsheet publicly available or do we need to ask IBM ?</p>
<p><br>
</p>
<p>Thank for your help,</p>
<p>Frank.</p>
<p></p>
<div>
<div><br>
<div style="color:rgb(0,0,0)">
<hr style="display:inline-block;width:98%">
<div id="m_378511725885416141divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Jan-Frode Myklebust <<a href="mailto:janfrode@tanso.net" target="_blank">janfrode@tanso.net</a>><br>
<b>Sent:</b> Saturday, April 22, 2017 10:50 AM<br>
<b>To:</b> <a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a><br>
<b>Subject:</b> Re: [gpfsug-discuss] Protocol node recommendations</font>
<div> </div>
</div>
</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div id="m_378511725885416141divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr">
<div>
<div>
<div style="color:rgb(0,0,0)">
<div>That's a tiny maxFilesToCache...<br>
<br>
I would start by implementing the settings from /usr/lpp/mmfs/*/gpfsprotocolldefaul* plus a 64GB pagepool for your protocoll nodes, and leave further tuning to when you see you have issues.<br>
<br>
Regarding sizing, we have a spreadsheet somewhere where you can input some workload parameters and get an idea for how many nodes you'll need. Your node config seems fine, but one node seems too few to serve 1000+ users. We support max 3000 SMB connections/node,
and I believe the recommendation is 4000 NFS connections/node.<br>
<br>
<br>
-jf<br>
<div class="gmail_quote">
<div dir="ltr">lør. 22. apr. 2017 kl. 08.34 skrev Frank Tower <<a href="mailto:frank.tower@outlook.com" target="_blank">frank.tower@outlook.com</a>>:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div id="m_378511725885416141m_-3610594251344932013divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;margin-top:0px;margin-bottom:0px">
Hi,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;margin-top:0px;margin-bottom:0px">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;margin-top:0px;margin-bottom:0px">
We have here around 2PB GPFS (4.2.2) accessed through an HPC cluster with GPFS client on each node.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;margin-top:0px;margin-bottom:0px">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;margin-top:0px;margin-bottom:0px">
We will have to open GPFS to all our users over CIFS and kerberized NFS with ACL support for both protocol for around +1000 users</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;margin-top:0px;margin-bottom:0px">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;margin-top:0px;margin-bottom:0px">
All users have different use case and needs: </div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;margin-top:0px;margin-bottom:0px">
- some will do random I/O through a large set of opened files (~5k files)</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;margin-top:0px;margin-bottom:0px">
- some will do large write with 500GB-1TB files</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;margin-top:0px;margin-bottom:0px">
- other will arrange sequential I/O with ~10k opened files</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;margin-top:0px;margin-bottom:0px">
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;margin-top:0px;margin-bottom:0px">
NFS and CIFS will share the same server, so I through to use SSD drive, at least 128GB memory with 2 sockets.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;margin-top:0px;margin-bottom:0px">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;margin-top:0px;margin-bottom:0px">
Regarding tuning parameters, I thought at: </div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;margin-top:0px;margin-bottom:0px">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;margin-top:0px;margin-bottom:0px">
maxFilesToCache 10000<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;margin-top:0px;margin-bottom:0px">
syncIntervalStrict yes<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;margin-top:0px;margin-bottom:0px">
workerThreads (8*core)<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;margin-top:0px;margin-bottom:0px">
prefetchPct 40 (for now and update if needed)</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;margin-top:0px;margin-bottom:0px">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;margin-top:0px;margin-bottom:0px">
I read the wiki 'Sizing Guidance for Protocol Node', but I was wondering if someone could share his experience/best practice regarding hardware sizing and/or tuning parameters.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;margin-top:0px;margin-bottom:0px">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;margin-top:0px;margin-bottom:0px">
Thank by advance,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;margin-top:0px;margin-bottom:0px">
Frank</div>
</div>
</div>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">
spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>