<div dir="ltr">i don't know if that works with cisco, but i use 50 an 100m cables for 40 as well as 100Gbit in my lab between 2 Mellanox switches : <div><a href="http://www.mellanox.com/products/interconnect/ethernet-active-optical-cables.php">http://www.mellanox.com/products/interconnect/ethernet-active-optical-cables.php</a><br></div><div>as paul pointed out one of the very first things one needs to do after adding an adapter is to flash the firmware to a recent level. especially of you have 2 adapters with different FW i have seen even once with higher level not work properly, so before you do anything else get them to a recent level and especially the same if its the same adapter types.</div><div><br></div><div>sven</div></div><br><div class="gmail_quote"><div dir="ltr">On Wed, Dec 20, 2017 at 10:01 PM David D Johnson <<a href="mailto:david_johnson@brown.edu">david_johnson@brown.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">We're trying to get 40 gbe connection between Mellanox switches and Cisco switches down at the other end of the machine room  <div>The BiDi part seems to be the best given about 30m  run on multimode.  However Mellanox support says it's not supported.</div><div>Want to use this to get close to IB speeds for GPFS on nodes that aren't on the IB fabric.  </div><div>Does anyone have any luck getting 40 or 100 gig at 20-30m when the switches are different brands?</div><div><br></div><div>Thanks,</div><div> -- ddj</div></div><div style="word-wrap:break-word"><div><br><div><div><blockquote type="cite"><div>On Dec 20, 2017, at 4:53 PM, Sanchez, Paul <<a href="mailto:Paul.Sanchez@deshaw.com" target="_blank">Paul.Sanchez@deshaw.com</a>> wrote:</div><br class="m_-1018774779858615118Apple-interchange-newline"><div><div class="m_-1018774779858615118WordSection1" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">We have run multiple ConnectX-4 NICs in bonded MLAG (Arista) and VPC (Cisco) switch configurations on our NSD servers.  We used to see issues with firmware versions that didn’t support the optics we wanted to use (e.g. early CX3/CX4 and Cisco 40G-BiDi).  You may also want check mstflint to see whether the firmware levels match on the MLX cards, and if you upgrade firmware in some cases a power-cycle (not reboot) can be required to finish the process.<u></u><u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">-Paul<u></u><u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><a name="m_-1018774779858615118__MailEndCompose"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></a></div><span></span><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><b><span style="font-size:11pt;font-family:Calibri,sans-serif">From:</span></b><span style="font-size:11pt;font-family:Calibri,sans-serif"><span class="m_-1018774779858615118Apple-converted-space"> </span><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a> [<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">mailto:gpfsug-discuss-bounces@spectrumscale.org</a>]<b>On Behalf Of<span class="m_-1018774779858615118Apple-converted-space"> </span></b>Andrew Beattie<br><b>Sent:</b><span class="m_-1018774779858615118Apple-converted-space"> </span>Wednesday, December 20, 2017 4:47 PM<br><b>To:</b><span class="m_-1018774779858615118Apple-converted-space"> </span><a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a><br><b>Subject:</b><span class="m_-1018774779858615118Apple-converted-space"> </span>Re: [gpfsug-discuss] more than one mlx connectx-4 adapter in samehost<u></u><u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><u></u> <u></u></div><div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif">IBM ESS building blocks can have up to 3 dual port 10GBEth, 40GB Eth, 56GB IB, 100GB IB Mlx adapater cards, because we have 2 IO nodes this is up to a total of 12 ports per building block<u></u><u></u></span></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif">so there should not be any reason for this to fail.<u></u><u></u></span></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif"> <u></u><u></u></span></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif">I regularly see a Mix of 10GB / 40GB or 10GB / IB configurations<u></u><u></u></span></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif"> <u></u><u></u></span></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif"> <u></u><u></u></span></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif"> <u></u><u></u></span></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif">Regards<u></u><u></u></span></div></div><div><div><div><div><div style="margin-top:15pt"><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><b><span style="font-family:Arial,sans-serif;color:rgb(124,124,95)">Andrew Beattie<u></u><u></u></span></b></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><b><span style="font-size:10pt;font-family:Arial,sans-serif">Software Defined Storage  - IT Specialist<u></u><u></u></span></b></div></div><div style="margin-top:7.5pt"><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><b><span style="font-size:8pt;font-family:Arial,sans-serif;color:rgb(51,102,153)">Phone:<span class="m_-1018774779858615118Apple-converted-space"> </span></span></b><span style="font-size:8pt;font-family:Arial,sans-serif">614-2133-7927<u></u><u></u></span></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><b><span style="font-size:8pt;font-family:Arial,sans-serif;color:rgb(51,102,153)">E-mail:<span class="m_-1018774779858615118Apple-converted-space"> </span></span></b><span style="font-size:8pt;font-family:Arial,sans-serif"><a href="mailto:abeattie@au1.ibm.com" style="color:purple;text-decoration:underline" target="_blank"><span style="color:rgb(85,85,85)">abeattie@au1.ibm.com</span></a><u></u><u></u></span></div></div></div></div></div></div></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif"> <u></u><u></u></span></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif"> <u></u><u></u></span></div></div><blockquote style="border-style:none none none solid;border-left-width:1.5pt;border-left-color:rgb(170,170,170);padding:0in 0in 0in 4pt;margin:5pt 0in 5pt 3.75pt"><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif">----- Original message -----<br>From: "J. Eric Wonderley" <<a href="mailto:eric.wonderley@vt.edu" style="color:purple;text-decoration:underline" target="_blank">eric.wonderley@vt.edu</a>><br>Sent by:<span class="m_-1018774779858615118Apple-converted-space"> </span><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" style="color:purple;text-decoration:underline" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a><br>To: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" style="color:purple;text-decoration:underline" target="_blank">gpfsug-discuss@spectrumscale.org</a>><br>Cc:<br>Subject: Re: [gpfsug-discuss] more than one mlx connectx-4 adapter in same host<br>Date: Thu, Dec 21, 2017 6:37 AM<br> <span class="m_-1018774779858615118Apple-converted-space"> </span><u></u><u></u></span></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif">Just plain tcpip.<u></u><u></u></span></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif"> <u></u><u></u></span></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif">We have dual port connectx4s in our nsd servers.  Upon adding a second connectx4 hba...no links go up or show "up".  I have one port on each hba configured for eth and ibv_devinfo looks sane.<u></u><u></u></span></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif"> <u></u><u></u></span></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif">I cannot find anything indicating that this should not work.  I have a ticket opened with mellanox.<u></u><u></u></span></div></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif"> <u></u><u></u></span></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif">On Wed, Dec 20, 2017 at 3:25 PM, Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] <<a href="mailto:aaron.s.knister@nasa.gov" style="color:purple;text-decoration:underline" target="_blank">aaron.s.knister@nasa.gov</a>> wrote:<span class="m_-1018774779858615118Apple-converted-space"> </span><u></u><u></u></span></div><blockquote style="border-style:none none none solid;border-left-width:1pt;border-left-color:rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in"><div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif"> <u></u><u></u></span></div></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif"><br><br>We’ve done a fair amount of VPI work but admittedly not with connectx4. Is it possible the cards are trying to talk IB rather than Eth? I figured you’re Ethernet based because of the mention of Juniper. <u></u><u></u></span></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif"> <u></u><u></u></span></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif">Are you attempting to do RoCE or just plain TCP/IP?<u></u><u></u></span></div></div><div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif"> <u></u><u></u></span></div><div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif">On December 20, 2017 at 14:40:48 EST, J. Eric Wonderley <<a href="mailto:eric.wonderley@vt.edu" style="color:purple;text-decoration:underline" target="_blank">eric.wonderley@vt.edu</a>> wrote:<u></u><u></u></span></div></div><blockquote style="border-style:none none none solid;border-left-width:1pt;border-left-color:windowtext;padding:0in 0in 0in 8pt;margin-left:0in;margin-top:5pt;margin-bottom:5pt"><div><div><div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif">Hello:<u></u><u></u></span></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif"> <u></u><u></u></span></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif">Does anyone have this type of config?<u></u><u></u></span></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif"> <u></u><u></u></span></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif">The host configuration looks sane but we seem to observe link-down on all mlx adapters no matter what we do.<u></u><u></u></span></div></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif"> <u></u><u></u></span></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif">Big picture is that we are attempting to do mc(multichassis)-lags to a core switch.  I'm somewhat fearful as to how this is implemented in the juniper switch we are about to test. <u></u><u></u></span></div></div></div></div></div></div></blockquote></div></div></div></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif"><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at<span class="m_-1018774779858615118Apple-converted-space"> </span><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=I5Dq2T7aYvC87Wp12fsz6CRLw4uo2-RVnrnpxRYfYuA&e=" style="color:purple;text-decoration:underline" target="_blank">spectrumscale.org</a><br><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=hxNNpOkwGQ9zRmTnM3FEo5hgnPSUsPG0FNqZbK6eA6Q&e=" style="color:purple;text-decoration:underline" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br> <u></u><u></u></span></div></blockquote></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10pt;font-family:'Courier New'">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=hxNNpOkwGQ9zRmTnM3FEo5hgnPSUsPG0FNqZbK6eA6Q&e=" style="color:purple;text-decoration:underline" target="_blank">https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=hxNNpOkwGQ9zRmTnM3FEo5hgnPSUsPG0FNqZbK6eA6Q&e=</a></span><span style="font-size:10.5pt;font-family:Arial,sans-serif"><u></u><u></u></span></div></div></blockquote><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><span style="font-size:10.5pt;font-family:Arial,sans-serif"> <u></u><u></u></span></div></div></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif"><u></u> <u></u></div></div><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important">_______________________________________________</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important">gpfsug-discuss mailing list</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important">gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a></span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></span></div></blockquote></div><br></div></div></div>_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote></div>