<font size=2 face="sans-serif">true, but keep in mind the setting of </font><br><font size=2 face="sans-serif">verbsRdmaMinBytes 4096</font><br><font size=2 face="sans-serif">this will define the lower border for
Rdma packets. all sizes below will take IP</font><br><br><font size=1 face="Arial"><br>Mit freundlichen Grüßen / Kind regards</font><p><font size=2 face="Arial"><b>Achim Rehor</b></font><p><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Bryan Banister <bbanister@jumptrading.com></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug main discussion
list <gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">01/24/2017 05:53 PM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
Manager nodes</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><tt><font size=2>It goes over IP, and that could be IPoIB if you have
the daemon interface or subnets configured that way, but it will go over
native IB VERBS if you have rdmaVerbsSend enabled (not recommended for
large clusters).<br><br> verbsRdmaSend<br> Enables
or disables the use of InfiniBand RDMA<br> rather than
TCP for most GPFS daemon-to-daemon<br> communication.
When disabled, only data<br> transfers
between an NSD client and NSD server<br> are eligible
for RDMA. Valid values are<br> enable or
disable. The default<br> value is
disable. The verbsRdma<br> option must
be enabled for verbsRdmaSend<br> to have
any effect.<br><br>HTH,<br>-B<br><br>-----Original Message-----<br>From: gpfsug-discuss-bounces@spectrumscale.org [</font></tt><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><tt><font size=2>mailto:gpfsug-discuss-bounces@spectrumscale.org</font></tt></a><tt><font size=2>]
On Behalf Of Simon Thompson (Research Computing - IT Services)<br>Sent: Tuesday, January 24, 2017 10:34 AM<br>To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Subject: Re: [gpfsug-discuss] Manager nodes<br><br>Thanks both. I was thinking of adding 4 (we have a storage cluster over
two DC's, so was planning to put two in each and use them as quorum nodes
as well plus one floating VM to guarantee only one sitr is quorate in the
event of someone cutting a fibre...)<br><br>We pretty much start at 128GB ram and go from there, so this sounds fine.
Would be good if someone could comment on if token traffic goes via IB
or Ethernet, maybe I can save myself a few EDR cards...<br><br>Simon<br>________________________________________<br>From: gpfsug-discuss-bounces@spectrumscale.org [gpfsug-discuss-bounces@spectrumscale.org]
on behalf of Jan-Frode Myklebust [janfrode@tanso.net]<br>Sent: 24 January 2017 15:51<br>To: gpfsug main discussion list<br>Subject: Re: [gpfsug-discuss] Manager nodes<br><br>Just some datapoints, in hope that it helps..<br><br>I've seen metadata performance improvements by turning down hyperthreading
from 8/core to 4/core on Power8. Also it helped distributing the token
managers over multiple nodes (6+) instead of fewer.<br><br>I would expect this to flow over IP, not IB.<br><br><br><br><br>-jf<br><br><br>tir. 24. jan. 2017 kl. 16.18 skrev Buterbaugh, Kevin L <Kevin.Buterbaugh@vanderbilt.edu<</font></tt><a href=mailto:Kevin.Buterbaugh@vanderbilt.edu><tt><font size=2>mailto:Kevin.Buterbaugh@vanderbilt.edu</font></tt></a><tt><font size=2>>>:<br>Hi Simon,<br><br>FWIW, we have two servers dedicated to cluster and filesystem management
functions (and 8 NSD servers). I guess you would describe our cluster
as small to medium sized ... ~700 nodes and a little over 1 PB of storage.<br><br>Our two managers have 2 quad core (3 GHz) CPU's and 64 GB RAM. They've
got 10 GbE, but we don't use IB anywhere. We have an 8 Gb FC SAN
and we do have them connected in to the SAN so that they don't have to
ask the NSD servers to do any I/O for them.<br><br>I do collect statistics on all the servers and plunk them into an RRDtool
database. Looking at the last 30 days the load average on the two
managers is in the 5-10 range. Memory utilization seems to be almost
entirely dependent on how parameters like the pagepool are set on them.<br><br>HTHAL...<br><br>Kevin<br><br>> On Jan 24, 2017, at 4:00 AM, Simon Thompson (Research Computing -
IT Services) <S.J.Thompson@bham.ac.uk<</font></tt><a href=mailto:S.J.Thompson@bham.ac.uk><tt><font size=2>mailto:S.J.Thompson@bham.ac.uk</font></tt></a><tt><font size=2>>>
wrote:<br>><br>> We are looking at moving manager processes off our NSD nodes and on
to<br>> dedicated quorum/manager nodes.<br>><br>> Are there some broad recommended hardware specs for the function of
these<br>> nodes.<br>><br>> I assume they benefit from having high memory (for some value of high,<br>> probably a function of number of clients, files, expected open files?,
and<br>> probably completely incalculable, so some empirical evidence may be
useful<br>> here?) (I'm going to ignore the docs that say you should have twice
as<br>> much swap as RAM!)<br>><br>> What about cores, do they benefit from high core counts or high clock<br>> rates? For example would I benefit more form a high core count, low
clock<br>> speed, or going for higher clock speeds and reducing core count? Or
is<br>> memory bandwidth more important for manager nodes?<br>><br>> Connectivity, does token management run over IB or only over<br>> Ethernet/admin network? I.e. Should I bother adding IB cards, or just
have<br>> fast Ethernet on them (my clients/NSDs all have IB).<br>><br>> I'm looking for some hints on what I would most benefit in investing
in vs<br>> keeping to budget.<br>><br>> Thanks<br>><br>> Simon<br>><br>> _______________________________________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at spectrumscale.org<</font></tt><a href=http://spectrumscale.org/><tt><font size=2>http://spectrumscale.org</font></tt></a><tt><font size=2>><br>> </font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<</font></tt><a href=http://spectrumscale.org/><tt><font size=2>http://spectrumscale.org</font></tt></a><tt><font size=2>><br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br><br>________________________________<br><br>Note: This email is for the confidential use of the named addressee(s)
only and may contain proprietary, confidential or privileged information.
If you are not the intended recipient, you are hereby notified that any
review, dissemination or copying of this email is strictly prohibited,
and to please notify the sender immediately and destroy this email and
any attachments. Email transmission cannot be guaranteed to be secure or
error-free. The Company, therefore, does not make any guarantees as to
the completeness or accuracy of this email or any attachments. This email
is for informational purposes only and does not constitute a recommendation,
offer, request or solicitation of any kind to buy, sell, subscribe, redeem
or perform any type of transaction of a financial product.<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br><br></font></tt><br><br><BR>