<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto">Coming to the routing point, is there any reason why you need it ? I mean, this is because GPFS trying to connect between compute nodes or a reason outside GPFS scope ?<div>If the reason is GPFS, imho best approach - without knowledge of the licensing you have - would be to use separate clusters: a storage cluster and two compute clusters.</div><div><div><div><br><div>Both compute clusters join using multicluster setup the storage cluster. There is no need both compute clusters see each other, they only need to see the storage cluster. One of the clusters using the 10G, the other cluster using the IPoIB interface.</div><div>You need at least three quorum nodes in each compute cluster but if licensing is per drive on the DSS, it is covered.</div></div><div dir="ltr"><font style="background-color: rgba(255, 255, 255, 0);">--<br>Jordi Caubet Serrabou<br>IBM Software Defined Infrastructure (SDI) and Flash Technical Sales Specialist<br>Technical Computing and HPC IT Specialist and Architect<br>Ext. Phone: <a href="tel:(+34)%20679.79.17.84" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="telephone" x-apple-data-detectors-result="20">(+34) 679.79.17.84</a> (internal 55834)<br>E-mail: <a href="mailto:jordi.caubet@es.ibm.com" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="link" x-apple-data-detectors-result="21">jordi.caubet@es.ibm.com</a></font><br style="font-family: UICTFontTextStyleTallBody; font-size: 17px; -webkit-text-size-adjust: auto;"></div><div dir="ltr"><br><blockquote type="cite">On 5 Oct 2020, at 08:19, Olaf Weiser <olaf.weiser@de.ibm.com> wrote:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr">
<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt"><div dir="ltr">let me add a few comments from some very successful large installations in Eruope</div><div dir="ltr"> </div><div dir="ltr"># InterOP</div><div dir="ltr">Even though (as Luis pointed to) , there is no support statement to run intermix DSS/ESS in general, it was ~, and is, and will be, ~ allowed for short term purposes, such as e.g migration.</div><div dir="ltr">The reason to not support those DSS/ESS mixed configuration in general is simply driven by the fact, that different release version of DSS/ESS potentially (not in every release, but sometimes) comes with different driver levels, (e.g. MOFED), OS, RDMA-settings, GPFS tuning, etc...</div><div dir="ltr">Those changes can have an impact/multiple impacts and therefore, we do not support that in general. Of course -and this would be the advice for every one - if you are faced the need to run a mixed configuration for e.g. a migration and/or e.g. cause of you need to temporary provide space etc... contact you IBM representative and settle to plan that accordingly..</div><div dir="ltr">There will be (likely) some additional requirements/dependencies defined like driver versions, OS, and/or Scale versions, but you'll get a chance to run mixed configuration - temporary limited to your specific scenario.</div><div dir="ltr"> </div><div dir="ltr"># Monitoring</div><div dir="ltr">No doubt, monitoring is essential and absolutely needed. - and/but - IBM wants customers to be very sensitive, what kind of additional software (=workload) gets installed on the ESS-IO servers. BTW, this rule applies as well to any other important GPFS node with special roles (e.g. any other NSD server etc)</div><div dir="ltr">But given the fact, that customer's usually manage and monitor their server farms from a central point of control (any 3rd party software), it is common/ best practice , that additionally monitor software(clients/endpoints) has to run on GPFS nodes, so as on ESS nodes too.</div><div dir="ltr"> </div><div dir="ltr">If that way of acceptance applies for DSS too, you may want to double check with Lenovo ?!</div><div dir="ltr"> </div><div dir="ltr"> </div><div dir="ltr">#additionally GW functions</div><div dir="ltr">It would be a hot iron, to general allow routing on IO nodes. Similar to the mixed support approach, the field variety for such a statement would be hard(==impossible) to manage. As we all agree, additional network traffic can (and in fact will) impact GPFS.</div><div dir="ltr">In your special case, the expected data rates seems to me more than ok and acceptable to go with your suggested config (as long workloads remain on that level / monitor it accordingly as you are already obviously doing) </div><div dir="ltr">Again,to be on the safe side.. contact your IBM representative and I'm sure you 'll find a way..</div><div dir="ltr"> </div><div dir="ltr"> </div><div dir="ltr"> </div><div dir="ltr">kind regards....</div><div dir="ltr">olaf</div><div dir="ltr"> </div><div dir="ltr"> </div><blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px">----- Original message -----<br>From: Jonathan Buzzard <jonathan.buzzard@strath.ac.uk><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: gpfsug-discuss@spectrumscale.org<br>Cc:<br>Subject: [EXTERNAL] Re: [gpfsug-discuss] Services on DSS/ESS nodes<br>Date: Sun, Oct 4, 2020 12:17 PM<br>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace">On 04/10/2020 10:29, Luis Bolinches wrote:<br>> Hi<br>><br>> As stated on the same link you can do remote mounts from each other and<br>> be a supported setup.<br>><br>> “ You can use the remote mount feature of IBM Spectrum Scale to share<br>> file system data across clusters.”<br>><br><br>You can, but imagine I have a DSS-G cluster, with 2PB of storage on it<br>which is quite modest in 2020. It is now end of life and for whatever<br>reason I decide I want to move to ESS instead.<br><br>What any sane storage admin want to do at this stage is set the ESS, add<br>the ESS nodes to the existing cluster on the DSS-G then do a bit of<br>mmadddisk/mmdeldisk and sit back while the data is seemlessly moved from<br>the DSS-G to the ESS. Admittedly this might take a while :-)<br><br>Then once all the data is moved a bit of mmdelnode and bingo the storage<br>has been migrated from DSS-G to ESS with zero downtime.<br><br>As that is not allowed for what I presume are commercial reasons (you<br>could do it in reverse and presumable that is what IBM don't want) then<br>once you are down the rabbit hole of one type of storage the you are not<br>going to switch to a different one.<br><br>You need to look at it from the perspective of the users. They frankly<br>could not give a monkeys what storage solution you are using. All they<br>care about is having usable storage and large amounts of downtime to<br>switch from one storage type to another is not really acceptable.<br><br><br>JAB.<br><br>--<br>Jonathan A. Buzzard Tel: +44141-5483420<br>HPC System Administrator, ARCHIE-WeSt.<br>University of Strathclyde, John Anderson Building, Glasgow. G4 0NG<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> </font><br> </div></blockquote><div dir="ltr"> </div></div><br>
<pre>_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
</pre><br></div></blockquote></div></div><BR>
Salvo indicado de otro modo más arriba / Unless stated otherwise above:<BR>
International Business Machines, S.A.<BR>
Santa Hortensia, 26-28, 28002 Madrid<BR>
Registro Mercantil de Madrid; Folio 1; Tomo 1525; Hoja M-28146<BR>
CIF A28-010791<BR>
<BR>
</body></html>