<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Using iSCSI with Spectrum Scale is definitely do-able. As with running Scale in general, your networking needs to be very solid. For iSCSI the best practice
I’m aware of is the dedicated/simple approach described by JF below: one subnet per switch (failure domain), nothing fancy like VRRP/HSRP/STP, and let multipathd do its job at ensuring that the available paths are the ones being used.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">We have also had some good experiences using
<b><i>routed</i></b> iSCSI (which fits the rackscale/hyperscale style deployment model too, but this implies that you have a good QoS plan to assure that markings are correct and any link which can become congested can’t completely starve the dedicated queue
you should be using for iSCSI. It’s also a good practice for the other TCP traffic in your non-iSCSI queue to use ECN in order to keep switch buffer utilization low. (As of today, I haven’t seen any iSCSI arrays which support ECN.) If you’re sharing arrays
with multiple clusters/filesystems (i.e. not a single workload), then I would also recommend using iSCSI arrays which support
<b>per-volume/volume-group QOS limits </b>to avoid noisy-neighbor problems in the iSCSI realm. As of today, there are even 100GbE capable all-flash solutions available which work well with Scale.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Lastly, I’d say that iSCSI might
<i>not</i> be the future… but NVMeOF hasn’t exactly given us many products ready to be the
<i>present</i>. Most of the early offerings in this space are under-featured, over-priced, inflexible, proprietary, or fragile. We are successfully using non-standards based NVMe solutions today with Scale, but they have much more stringent and sensitive
networking requirements (e.g. non-routed dedicated networking with PFC for RoCE) in order to provide reliable performance. So far, we’ve found these early offerings best-suited for single-workload use cases. I do expect this to continue to develop and improve
on price, features, reliability/fragility.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thx<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Paul<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> gpfsug-discuss-bounces@spectrumscale.org <gpfsug-discuss-bounces@spectrumscale.org>
<b>On Behalf Of </b>Jan-Frode Myklebust<br>
<b>Sent:</b> Sunday, December 16, 2018 8:46 AM<br>
<b>To:</b> gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
<b>Subject:</b> Re: [gpfsug-discuss] Anybody running GPFS over iSCSI? -<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><br>
I have been running GPFS over iSCSI, and know of customers who are also. Probably not in the most demanding environments, but from my experience iSCSI works perfectly fine as long as you have a stable network. Having a dedicated (simple) storage network for
iSCSI is probably a good idea (just like for FC), otherwise iSCSI or GPFS is going to look bad when your network admins cause problems on the shared network.<br>
<br>
<br>
-jf<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">søn. 16. des. 2018 kl. 12:59 skrev Frank Kraemer <<a href="mailto:kraemerf@de.ibm.com">kraemerf@de.ibm.com</a>>:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p><span style="font-size:10.0pt">Kevin,</span><br>
<br>
<span style="font-size:10.0pt">Ethernet networking of today is changing very fast as the driving forces are the "Hyperscale" datacenters. This big innovation is changing the world and is happening right now. You must understand the conversation by breaking
down the differences between ASICs, FPGAs, and NPUs in modern Ethernet networking.</span><br>
<br>
<span style="font-size:10.0pt">1) Mellanox has a very good answer here based on the Spectrum-2 chip</span><br>
<a href="http://www.mellanox.com/page/press_release_item?id=1933" target="_blank"><span style="font-size:10.0pt">http://www.mellanox.com/page/press_release_item?id=1933</span></a><br>
<br>
<span style="font-size:10.0pt">2) Broadcom's answer to this is the 12.8 Tb/s StrataXGS Tomahawk 3 Ethernet Switch Series</span><br>
<a href="https://www.broadcom.com/products/ethernet-connectivity/switching/strataxgs/bcm56980-series" target="_blank"><span style="font-size:10.0pt">https://www.broadcom.com/products/ethernet-connectivity/switching/strataxgs/bcm56980-series</span></a><br>
<a href="https://globenewswire.com/news-release/2018/10/24/1626188/0/en/Broadcom-Achieves-Mass-Production-on-Industry-Leading-12-8-Tbps-Tomahawk-3-Ethernet-Switch-Family.html" target="_blank"><span style="font-size:10.0pt">https://globenewswire.com/news-release/2018/10/24/1626188/0/en/Broadcom-Achieves-Mass-Production-on-Industry-Leading-12-8-Tbps-Tomahawk-3-Ethernet-Switch-Family.html</span></a><br>
<br>
<span style="font-size:10.0pt">3) Barefoots Tofinio2 is another valid answer to this problem as it's programmable with the P4 language (important for Hyperscale Datacenters)</span><br>
<a href="https://www.barefootnetworks.com/" target="_blank"><span style="font-size:10.0pt">https://www.barefootnetworks.com/</span></a><br>
<br>
<span style="font-size:10.0pt">The P4 language itself is open source. There’s details at
<a href="http://p4.org" target="_blank">p4.org</a>, or you can download code at GitHub:
</span><a href="https://github.com/p4lang/" target="_blank"><span style="font-size:10.0pt">https://github.com/p4lang/</span></a><br>
<br>
<span style="font-size:10.0pt">4) The last newcomer to this party comes from Innovium named Teralynx</span><br>
<a href="https://innovium.com/products/teralynx/" target="_blank"><span style="font-size:10.0pt">https://innovium.com/products/teralynx/</span></a><br>
<a href="https://innovium.com/2018/03/20/innovium-releases-industrys-most-advanced-switch-software-platform-for-high-performance-data-center-networking-2-2-2-2-2-2/" target="_blank"><span style="font-size:10.0pt">https://innovium.com/2018/03/20/innovium-releases-industrys-most-advanced-switch-software-platform-for-high-performance-data-center-networking-2-2-2-2-2-2/</span></a><br>
<br>
<span style="font-size:10.0pt">(Most of the new Cisco switches are powered by the Teralynx silicon, as Cisco seems to be late to this game with it's own development.)</span><br>
<br>
<span style="font-size:10.0pt">So back your question - iSCSI is not the future! NVMe and it's variants is the way to go and these new ethernet swichting products does have this in focus.</span><br>
<span style="font-size:10.0pt">Due to the performance demands of NVMe, high performance and low latency networking is required and Ethernet based RDMA — RoCE, RoCEv2 or iWARP are the leading choices.</span><br>
<br>
<span style="font-size:10.0pt">-frank-</span><br>
<br>
<span style="font-size:10.0pt">P.S. My Xmas wishlist to the IBM Spectrum Scale development team would be a "2019 HighSpeed Ethernet Networking optimization for Spectrum Scale" to make use of all these new things and options :-)
</span><br>
<br>
<span style="font-size:10.0pt">Frank Kraemer<br>
IBM Consulting IT Specialist / Client Technical Architect<br>
Am Weiher 24, 65451 Kelsterbach, Germany<br>
<a href="mailto:kraemerf@de.ibm.com" target="_blank">mailto:kraemerf@de.ibm.com</a><br>
Mobile +49171-3043699<br>
IBM Germany</span><o:p></o:p></p>
</div>
<p class="MsoNormal">_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><o:p></o:p></p>
</blockquote>
</div>
</div>
</body>
</html>