<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" >Hey guys - I wanted to reply from the Scale development side..... </div>
<div dir="ltr" ><div dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div dir="ltr" > </div>
<div dir="ltr" >First off, consider CES as a stack and the implications of such:</div>
<div dir="ltr" >- all protocols are installed on all nodes</div>
<div dir="ltr" >- if a specific protocol is enabled (SMB, NFS, OBJ, Block), it's enabled for all protocol nodes</div>
<div dir="ltr" >- if a specific protocol is started (SMB, NFS, OBJ, Block), it's started on all nodes by default, unless manually specified. </div>
<div dir="ltr" > </div>
<div dir="ltr" >As was indicated in the e-mail chain, you don't want to be removing rpms to create a subset of nodes serving various protocols as this will cause overall issues. You also don't want to manually be disabling protocols on some nodes/not others in order to achieve nodes that are 'only serving' SMB, for instance. Doing this manual stopping/starting of protocols isn't something that will adhere to failover. </div>
<div dir="ltr" > </div>
<div dir="ltr" >===============================================================</div>
<div dir="ltr" >A few possible solutions if you want to segregate protocols to specific nodes are:</div>
<div dir="ltr" >===============================================================</div>
<div dir="ltr" >1) CES-Groups in combination with specific IPs / DNS hostnames that correspond to each protocol. </div>
<div dir="ltr" >- As mentioned, this can still be bypassed if someone attempts a mount using an IP/DNS name not set for their protocol. However, you could probably prevent some of this with an external firewall rule. </div>
<div dir="ltr" >- Using CES-Groups confines the IPs/DNS hostnames to very specific nodes</div>
<div dir="ltr" > </div>
<div dir="ltr" >2) Firewall rules</div>
<div dir="ltr" >- This is best if done external to the cluster, and at a level that can restrict specific protocol traffic to specific IPs/hostnames</div>
<div dir="ltr" >- combine this with #1 for the best results. </div>
<div dir="ltr" >- Although it may work, try to stay away from crazy firewall rules on each protocol node itself as this can get confusing very quickly. It's easier if you can set this up external to the nodes. </div>
<div dir="ltr" > </div>
<div dir="ltr" >3) Similar to above but using Node Affinity CES-IP policy - but no CES groups. </div>
<div dir="ltr" >- Upside is node-affinity will attempt to keep your CES-IPs associated with specific nodes. So if you restrict specific protocol traffic to specific IPs, then they'll stay on nodes you designate</div>
<div dir="ltr" >- Watch out for failovers. In error cases (or upgrades) where an IP needs to move to another node, it obviously can't remain on the node that's having issues. This means you may have protocol trafffic crossover when this occurs.</div>
<div dir="ltr" > </div>
<div dir="ltr" >4) A separate remote cluster for each CES protocol</div>
<div dir="ltr" >- In this example, you could make fairly small remote clusters (although we recommend 2->3nodes at least for failover purposes). The local cluster would provide the storage. The remote clusters would mount it. One remote cluster could have only SMB enabled. Another remote cluster could have only OBJ enabled. etc... </div>
<div dir="ltr" > </div>
<div dir="ltr" >------</div>
<div dir="ltr" >I hope this helps a bit....</div>
<div dir="ltr" > </div>
<div dir="ltr" ><br>Regards,<br><br>Aaron Palazzolo</div>
<div dir="ltr" >IBM Spectrum Scale Deployment, Infrastructure, Virtualization<br>9042 S Rita Road, Tucson AZ 85744<br>Phone: 520-799-5161, T/L: 321-5161<br>E-mail: aspalazz@us.ibm.com</div></div></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: gpfsug-discuss-request@spectrumscale.org<br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: gpfsug-discuss@spectrumscale.org<br>Cc:<br>Subject: gpfsug-discuss Digest, Vol 84, Issue 4<br>Date: Wed, Jan 9, 2019 7:13 AM<br>
<div><font face="Default Monospace,Courier New,Courier,monospace" size="2" >Send gpfsug-discuss mailing list submissions to<br>gpfsug-discuss@spectrumscale.org<br><br>To subscribe or unsubscribe via the World Wide Web, visit<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>or, via email, send a message with subject or body 'help' to<br>gpfsug-discuss-request@spectrumscale.org<br><br>You can reach the person managing the list at<br>gpfsug-discuss-owner@spectrumscale.org<br><br>When replying, please edit your Subject line so it is more specific<br>than "Re: Contents of gpfsug-discuss digest..."<br><br><br>Today's Topics:<br><br> 1. Re: Spectrum Scale protocol node service separation.<br> (Andi Rhod Christiansen)<br> 2. Re: Spectrum Scale protocol node service separation.<br> (Sanchez, Paul)<br><br><br>----------------------------------------------------------------------<br><br>Message: 1<br>Date: Wed, 9 Jan 2019 13:24:30 +0000<br>From: Andi Rhod Christiansen <arc@b4restore.com><br>To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Subject: Re: [gpfsug-discuss] Spectrum Scale protocol node service<br>separation.<br>Message-ID:<br><a92739a568ec4e73b68de624127d2319@B4RWEX01.internal.b4restore.com><br>Content-Type: text/plain; charset="utf-8"<br><br>Hi Simon,<br><br>It was actually also the only solution I found if I want to keep them within the same cluster ?<br><br>Thanks for the reply, I will see what we figure out !<br><br>Venlig hilsen / Best Regards<br><br>Andi Rhod Christiansen<br><br>Fra: gpfsug-discuss-bounces@spectrumscale.org <gpfsug-discuss-bounces@spectrumscale.org> P? vegne af Simon Thompson<br>Sendt: 9. januar 2019 13:20<br>Til: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Emne: Re: [gpfsug-discuss] Spectrum Scale protocol node service separation.<br><br>You have to run all services on all nodes ( ? ) actually its technically possible to remove the packages once protocols is running on the node, but next time you reboot the node, it will get marked unhealthy and you spend an hour working out why? <AHEM><br><br>But what we do to split load is have different IPs assigned to different CES groups and then assign the SMB nodes to the SMB group IPs etc ?<br><br>Technically a user could still connect to the NFS (in our case) IPs with SMB protocol, but there?s not a lot we can do about that ? though our upstream firewall drops said traffic.<br><br>Simon<br><br>From: <gpfsug-discuss-bounces@spectrumscale.org<<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">mailto:gpfsug-discuss-bounces@spectrumscale.org</a>>> on behalf of "arc@b4restore.com<<a href="mailto:arc@b4restore.com" target="_blank">mailto:arc@b4restore.com</a>>" <arc@b4restore.com<<a href="mailto:arc@b4restore.com" target="_blank">mailto:arc@b4restore.com</a>>><br>Reply-To: "gpfsug-discuss@spectrumscale.org<<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">mailto:gpfsug-discuss@spectrumscale.org</a>>" <gpfsug-discuss@spectrumscale.org<<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">mailto:gpfsug-discuss@spectrumscale.org</a>>><br>Date: Wednesday, 9 January 2019 at 10:31<br>To: "gpfsug-discuss@spectrumscale.org<<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">mailto:gpfsug-discuss@spectrumscale.org</a>>" <gpfsug-discuss@spectrumscale.org<<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">mailto:gpfsug-discuss@spectrumscale.org</a>>><br>Subject: [gpfsug-discuss] Spectrum Scale protocol node service separation.<br><br>Hi,<br><br>I seem to be unable to find any information on separating protocol services on specific CES nodes within a cluster. Does anyone know if it is possible to take, lets say 4 of the ces nodes within a cluster and dividing them into two and have two of the running SMB and the other two running OBJ instead of having them all run both services?<br><br>If it is possible it would be great to hear pros and cons about doing this ?<br><br>Thanks in advance!<br><br>Venlig hilsen / Best Regards<br><br>Andi Christiansen<br>IT Solution Specialist<br><br><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20190109/83819399/attachment-0001.html" target="_blank">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20190109/83819399/attachment-0001.html</a>><br><br>------------------------------<br><br>Message: 2<br>Date: Wed, 9 Jan 2019 14:05:48 +0000<br>From: "Sanchez, Paul" <Paul.Sanchez@deshaw.com><br>To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Subject: Re: [gpfsug-discuss] Spectrum Scale protocol node service<br>separation.<br>Message-ID:<br><53ec54bb621242109a789e51d61b1377@mbxtoa1.winmail.deshaw.com><br>Content-Type: text/plain; charset="utf-8"<br><br>The docs say: ?CES supports the following export protocols: NFS, SMB, object, and iSCSI (block). Each protocol can be enabled or disabled in the cluster. If a protocol is enabled in the CES cluster, all CES nodes serve that protocol.? Which would seem to indicate that the answer is ?no?.<br><br>This kind of thing is another good reason to license Scale by storage capacity rather than by sockets (PVU). This approach was already a good idea due to the flexibility it allows to scale manager, quorum, and NSD server nodes for performance and high-availability without affecting your software licensing costs. This can result in better design and the flexibility to more quickly respond to new problems by adding server nodes.<br><br>So assuming you?re not on the old PVU licensing model, it is trivial to deploy as many gateway nodes as needed to separate these into distinct remote clusters. You can create an object gateway cluster, and a CES gateway cluster each which only mounts and exports what is necessary. You can even virtualize these servers and host them on the same hardware, if you?re into that.<br><br>-Paul<br><br>From: gpfsug-discuss-bounces@spectrumscale.org <gpfsug-discuss-bounces@spectrumscale.org> On Behalf Of Andi Rhod Christiansen<br>Sent: Wednesday, January 9, 2019 5:25 AM<br>To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Subject: [gpfsug-discuss] Spectrum Scale protocol node service separation.<br><br>Hi,<br><br>I seem to be unable to find any information on separating protocol services on specific CES nodes within a cluster. Does anyone know if it is possible to take, lets say 4 of the ces nodes within a cluster and dividing them into two and have two of the running SMB and the other two running OBJ instead of having them all run both services?<br><br>If it is possible it would be great to hear pros and cons about doing this ?<br><br>Thanks in advance!<br><br>Venlig hilsen / Best Regards<br><br>Andi Christiansen<br>IT Solution Specialist<br><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20190109/7f9ad3f8/attachment.html" target="_blank">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20190109/7f9ad3f8/attachment.html</a>><br><br>------------------------------<br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br><br><br>End of gpfsug-discuss Digest, Vol 84, Issue 4<br>*********************************************</font><br> </div></blockquote>
<div dir="ltr" > </div></div></div><BR>