<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div dir="ltr" >It looks like no one has attempted to answer this, so I will step in to start the conversation.</div>
<div dir="ltr" > </div>
<div dir="ltr" >There are two issues when considering how many services to run on the same nodes - in this case the NSD servers.</div>
<div dir="ltr" > </div>
<div dir="ltr" >1. Performance.</div>
<div dir="ltr" >Spectrum Scale's (nee GPFS) core differentiator is performance. The more you run on a node the more that node resource's have to be shared. Here memory bandwidth and memory space are the main ones. CPU may also be a limited resource although with modern chips this is less likely so.</div>
<div dir="ltr" >If performance is not the key delivered metric then running other things on the NSD server may be a good option so save both cost and server spawl in small datacentres.</div>
<div dir="ltr" > </div>
<div dir="ltr" >2. NFS server stability.</div>
<div dir="ltr" >pre-4.1.1, IBM used cNFS to provide multiple NFS servers in a GPFS cluster. This used traditional kernel based NFS daemons. If one hung then the whole node had to be rebooted which might have led to disruption in NSD serving if the other NSD server of a pair was already under load. With 4.1.1 came Cluster Export Services (CES) deliverd from 'Protocol Nodes'. Since there use Ganesha there would be no need to reboot this node if the NFS serving hung and in Ganesha, all NFS activity is in userspace not the kernel.</div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div dir="ltr" style="margin-top: 20px;" ><div style="font-size: 8pt; font-family: sans-serif; margin-top: 10px;" ><font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2" ><span style="font-size:1.143em;" >Daniel</span> </font>
<div><font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2" > </font></div><font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2" > </font>
<div><font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2" > </font></div><font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2" > </font>
<div><table style="border-collapse:collapse;" border="0" cellpadding="0" cellspacing="0" > <tbody> <tr> <td colspan="2" style="width:404px;padding:0cm 0cm 0cm 0cm;" > <div style="margin-bottom:0cm;margin-bottom:.0001pt;line-height:normal;page-break-after:avoid;text-autospace:none;" ><font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2" ><strong><span style="color:black;" ><span style="font-family:Arial, Helvetica, sans-serif" ><span style="font-size:10.0pt;" >Dr.Daniel Kidger</span></span></span></strong></font></div> </td> <td style="width:154px;padding:0cm 0cm 0cm 0cm;" > <div style="margin-bottom:0cm;margin-bottom:.0001pt;line-height:normal;page-break-after:avoid;text-autospace:none;" ><font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2" ><font color="#5f5f5f" ><span style="font-family:Arial, Helvetica, sans-serif" ><span style="font-size:8.0pt;" >No. 1 The Square,</span></span></font></font></div> </td> </tr> <tr> <td colspan="2" style="width:404px;padding:0cm 0cm 0cm 0cm;" > <div style="margin-bottom:0cm;margin-bottom:.0001pt;line-height:normal;page-break-after:avoid;text-autospace:none;" ><font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2" ><font color="#5f5f5f" ><span style="font-family:Arial, Helvetica, sans-serif" ><span style="font-size:8.0pt;" >Technical Specialist SDI (formerly Platform Computing)</span></span></font></font></div> </td> <td style="width:154px;padding:0cm 0cm 0cm 0cm;" > <div style="margin-bottom:0cm;margin-bottom:.0001pt;line-height:normal;page-break-after:avoid;text-autospace:none;" ><font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2" ><font color="#5f5f5f" ><span style="font-family:Arial, Helvetica, sans-serif" ><span style="font-size:8.0pt;" >Temple Quay,<br> Bristol BS1 6DG</span></span></font></font></div> </td> </tr> <tr> <td style="width:72px;padding:0cm 0cm 0cm 0cm;" > <div style="margin-bottom:0cm;margin-bottom:.0001pt;line-height:normal;page-break-after:avoid;text-autospace:none;" ><font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2" ><font color="#5f5f5f" ><span style="font-family:Arial, Helvetica, sans-serif" ><span style="font-size:8.0pt;" >Mobile:</span></span></font></font></div> </td> <td style="width:261px;padding:0cm 0cm 0cm 0cm;" > <div style="margin-bottom:0cm;margin-bottom:.0001pt;line-height:normal;page-break-after:avoid;text-autospace:none;" ><font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2" ><font color="#5f5f5f" ><span style="font-family:Arial, Helvetica, sans-serif" ><span style="font-size:8.0pt;" >+44-07818 522 266</span></span></font></font></div> </td> <td style="width:154px;padding:0cm 0cm 0cm 0cm;" > <div style="margin-bottom:0cm;margin-bottom:.0001pt;line-height:normal;page-break-after:avoid;text-autospace:none;" ><font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2" ><font color="#5f5f5f" ><span style="font-family:Arial, Helvetica, sans-serif" ><span style="font-size:8.0pt;" >United Kingdom</span></span></font></font></div> </td> </tr> <tr> <td style="width:72px;padding:0cm 0cm 0cm 0cm;" > <div style="margin-bottom:0cm;margin-bottom:.0001pt;line-height:normal;page-break-after:avoid;text-autospace:none;" ><font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2" ><font color="#5f5f5f" ><span style="font-family:Arial, Helvetica, sans-serif" ><span style="font-size:8.0pt;" >Landline:</span></span></font></font></div> </td> <td style="width:261px;padding:0cm 0cm 0cm 0cm;" > <div style="margin-bottom:0cm;margin-bottom:.0001pt;line-height:normal;page-break-after:avoid;text-autospace:none;" ><font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2" ><font color="#5f5f5f" ><span style="font-family:Arial, Helvetica, sans-serif" ><span style="font-size:8.0pt;" >+44-02392 564 121 (Internal ITN 3726 9250)</span></span></font></font></div> </td> <td style="width:154px;padding:0cm 0cm 0cm 0cm;" > <div style="margin-bottom:0cm;margin-bottom:.0001pt;line-height:normal;page-break-after:avoid;text-autospace:none;" ><font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2" > </font></div> </td> </tr> <tr> <td style="width:72px;padding:0cm 0cm 0cm 0cm;" > <div style="margin-bottom:0cm;margin-bottom:.0001pt;line-height:normal;page-break-after:avoid;text-autospace:none;" ><font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2" ><font color="#5f5f5f" ><span style="font-family:Arial, Helvetica, sans-serif" ><span style="font-size:8.0pt;" >e-mail:</span></span></font></font></div> </td> <td style="width:261px;padding:0cm 0cm 0cm 0cm;" > <div style="margin-bottom:0cm;margin-bottom:.0001pt;line-height:normal;page-break-after:avoid;text-autospace:none;" ><font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2" ><font color="#5f5f5f" ><span style="font-family:Arial, Helvetica, sans-serif" ><span style="font-size:8.0pt;" >daniel.kidger@uk.ibm.com</span></span></font></font></div> </td> <td style="width:154px;padding:0cm 0cm 0cm 0cm;" > <div style="margin-bottom:0cm;margin-bottom:.0001pt;line-height:normal;page-break-after:avoid;text-autospace:none;" ><font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2" > </font></div> </td> </tr> </tbody></table></div><font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2" > </font></div></div></div></div></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: "Buterbaugh, Kevin L" <Kevin.Buterbaugh@Vanderbilt.Edu><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Cc:<br>Subject: [gpfsug-discuss] GPFS 4.2 / Protocol nodes<br>Date: Fri, Jan 8, 2016 4:11 PM<br> <br><!--Notes ACF
<meta http-equiv="Content-Type" content="text/html; charset=utf8" >--> Happy New Year all,
<div> </div>
<div>One of my first projects on the new year is to get GPFS 4.2 up and running on our test cluster and begin testing it out in anticipation of upgrading our production cluster sometime later this year. We’re currently running 4.1.0.8 efix2 and are thinking of bypassing 4.1.1 altogether and going straight to 4.2.</div><br>We currently have 3 NSD servers also serving as CNFS servers and 1 NSD server that is not primary for any disks which serves as our SAMBA server. We are interested in going to CES.
<div> </div>
<div>Yesterday I was reading in the 4.2 FAQ and came across question 8.3, “What are some of the considerations when deploying the protocol functionality?” One of the considerations is that "several GPFS configuration aspects have not been explicitly tested with the protocol functionality” and one of those functions is, “NSD server functionality and storage attached to protocol node. We recommend that Protocol nodes do not take on these functions”.</div>
<div> </div>
<div>Really? So it is IBM’s recommendation that we buy 3 additional very beefy (2 x hex-core processors, 256 GB RAM) servers and 3 additional server licenses just to use CES? I guess I’m very surprised by that because I’m running CNFS on 3 low end servers (1 x quad-core processor, 32 GB RAM) that also serve as NSD servers to a ~700 client HPC cluster!</div>
<div> </div>
<div>If we really have to buy all that, well, we probably won’t. That’s a not insignificant chunk of change.</div>
<div> </div>
<div>Some I’m interested in hearing feedback on both: 1) do the CES servers really have to not be NSD servers, and 2) do they really need to be such high-end boxes? And both the official party line and what you can really do in the real world if you really want to (while maintaining your support agreement with IBM) are welcome! Thanks…</div>
<div> </div>
<div>Kevin</div>
<div> </div>
<div><div><div>—</div>
<div>Kevin Buterbaugh - Senior System Administrator</div>
<div>Vanderbilt University - Advanced Computing Center for Research and Education</div>
<div><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" target="_blank" >Kevin.Buterbaugh@vanderbilt.edu</a> - (615)875-9633</div>
<div> </div></div></div>
<div><font face="Default Monospace,Courier New,Courier,monospace" size="2" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank" >http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></div></blockquote>
<div dir="ltr" > </div></div>Unless stated otherwise above:<BR>
IBM United Kingdom Limited - Registered in England and Wales with number 741598. <BR>
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU<BR>
<BR>