<html><head><META http-equiv="Content-Type" content="text/html;charset=utf-8"></head><body><font face="Default," helvetica="" neue,="" open="" sans,arial,sans-serif="" size="2">Walter:<div><br></div><div>Though not directly about your design, our work with NVIDIA on GPUdirect Storage and SuperPOD has shown how sensitive RDMA (IB & RoCE) to both MOFED and Firmware version compatibility can be. </div><div><br></div><div>I would suggest anyone debugging RDMA issues should look at those closely. </div><div><br></div><div>Doug</div><div><br></div><div><br>by carrier pigeon<meta name="viewport" content="initial-scale=1.1"><br><br><div id="TROriginalMessageContentWrapper" contenteditable="false"><hr>On Dec 9, 2021, 5:04:36 AM, <a href="mailto:gpfsug-discuss-request@spectrumscale.org" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="link" x-apple-data-detectors-result="1">gpfsug-discuss-request@spectrumscale.org</a> wrote:<br><br>From: <a href="mailto:gpfsug-discuss-request@spectrumscale.org" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="link" x-apple-data-detectors-result="2">gpfsug-discuss-request@spectrumscale.org</a><br>To: <a href="mailto:gpfsug-discuss@spectrumscale.org" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="link" x-apple-data-detectors-result="3">gpfsug-discuss@spectrumscale.org</a><br>Cc: <br>Date: Dec 9, 2021, 5:04:36 AM<br>Subject: [EXTERNAL] gpfsug-discuss Digest, Vol 119, Issue 5<p></p><hr><br><p></p><div id="MaaS360PIMSDKOriginalMessageId"><pre>Send gpfsug-discuss mailing list submissions to <a href="mailto:gpfsug-discuss@spectrumscale.orgTo" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="link" x-apple-data-detectors-result="5">gpfsug-discuss@spectrumscale.orgTo</a> subscribe or unsubscribe via the World Wide Web, visit <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="link" x-apple-data-detectors-result="6">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> or, via email, send a message with subject or body 'help' to <a href="mailto:gpfsug-discuss-request@spectrumscale.orgYou" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="link" x-apple-data-detectors-result="7">gpfsug-discuss-request@spectrumscale.orgYou</a> can reach the person managing the list at <a href="mailto:gpfsug-discuss-owner@spectrumscale.orgWhen" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="link" x-apple-data-detectors-result="8">gpfsug-discuss-owner@spectrumscale.orgWhen</a> replying, please edit your Subject line so it is more specificthan "Re: Contents of gpfsug-discuss digest..."</pre><br><pre>Send gpfsug-discuss mailing list submissions to <a href="mailto:gpfsug-discuss@spectrumscale.orgTo" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="link" x-apple-data-detectors-result="9">gpfsug-discuss@spectrumscale.orgTo</a> subscribe or unsubscribe via the World Wide Web, visit <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="link" x-apple-data-detectors-result="10">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> or, via email, send a message with subject or body 'help' to <a href="mailto:gpfsug-discuss-request@spectrumscale.orgYou" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="link" x-apple-data-detectors-result="11">gpfsug-discuss-request@spectrumscale.orgYou</a> can reach the person managing the list at <a href="mailto:gpfsug-discuss-owner@spectrumscale.orgWhen" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="link" x-apple-data-detectors-result="12">gpfsug-discuss-owner@spectrumscale.orgWhen</a> replying, please edit your Subject line so it is more specificthan "Re: Contents of gpfsug-discuss digest..."Today's Topics: 1. alternate path between ESS Servers for Datamigration (Walter Sklenka)</pre><br> <style><!--/* Font Definitions */@font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;}@font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;}/* Style Definitions */p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0cm; font-size:11.0pt; font-family:"Calibri",sans-serif; mso-fareast-language:EN-US;}span.EmailStyle17 {mso-style-type:personal-compose; font-family:"Calibri",sans-serif; color:windowtext;}.MsoChpDefault {mso-style-type:export-only; font-family:"Calibri",sans-serif; mso-fareast-language:EN-US;}@page WordSection1 {size:612.0pt 792.0pt; margin:70.85pt 70.85pt 2.0cm 70.85pt;}div.WordSection1 {page:WordSection1;}--></style> <div class="WordSection1"> <p class="MsoNormal">Dear spectrum scale users!</p> <p class="MsoNormal"><span>May I ask you a design question?</span></p> <p class="MsoNormal"><span>We have an IB environment which is very mixed at the moment ( connecX3 … connect-X6 with FDR , even FDR10 and with arrive of ESS5000SC7 now also HDR100 and HDR switches. We still have some big troubles in this fabric when using RDMA , a case at Mellanox and IBM is open . </span></p> <p class="MsoNormal"><span>The environment has 3 old Building blocks 2xESSGL6 and 1x GL4 , from where we want to migrate the data to ess5000 , ( mmdelvdisk +qos) </span></p> <p class="MsoNormal"><span>Due to the current problems with RDMA we though eventually we could try a workaround :</span></p> <p class="MsoNormal"><span>If you are interested there is Maybe you can find the attachment ? </span></p> <p class="MsoNormal"><span>We build 2 separate fabrics , the ess-IO servers attached to both blue and green and all other cluster members and all remote clusters only to fabric blue </span></p> <p class="MsoNormal"><span>The daemon interfaces (IPoIP) are on fabric blue</span></p> <p class="MsoNormal"><span> </span></p> <p class="MsoNormal"><span>It is the aim to setup rdma only on the ess-ioServers in the fabric green , in the blue we must use IPoIB (tcp) </span></p> <p class="MsoNormal"><span>Do you think datamigration would work between ess01,ess02,… to ess07,ess08 via RDMA ?</span></p> <p class="MsoNormal"><span>Or is it principally not possible to make a rdma network only for a subset of a cluster (though this subset would be reachable via other fabric) ?</span></p> <p class="MsoNormal"><span> </span></p> <p class="MsoNormal"><span>Thank you very much for any input !</span></p> <p class="MsoNormal"><span>Best regards walter </span></p> <p class="MsoNormal"><span> </span></p> <p class="MsoNormal"><span> </span></p> <p class="MsoNormal"><span> </span></p> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:DE-AT;;;">Mit freundlichen Grüßen<br> <b><i>Walter Sklenka</i></b><br> </span><b><i><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:gray;mso-fareast-language:DE-AT;;;">Technical Consultant</span></i></b><span style="color:black;mso-fareast-language:DE-AT;;;"> </span><span style="color:#212121;mso-fareast-language:DE-AT;;;"></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif;color:black;mso-fareast-language:DE-AT;;;"> </span><span style="color:#212121;mso-fareast-language:DE-AT;;;"></span></p> <p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:DE-AT;;;">EDV-Design Informationstechnologie GmbH<br> Giefinggasse 6/1/2, A-1210 Wien<br> Tel: <a href="tel:+43%201%2029%2022%20165-31" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="telephone" x-apple-data-detectors-result="15/2" style="color: currentcolor; text-decoration-color: rgba(128, 128, 128, 0.38);;;;">+43 1 29 22 165-31</a><br> Fax: <a href="tel:+43%201%2029%2022%20165-90" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="telephone" x-apple-data-detectors-result="15/3" style="color: currentcolor; text-decoration-color: rgba(128, 128, 128, 0.38);;;;">+43 1 29 22 165-90</a><br> E-Mail: <a href="mailto:sklenka@edv-design.at"><span style="color:#0563C1;;;">sklenka@edv-design.at</span></a></span><span style="font-size:8.0pt;color:black;mso-fareast-language:DE-AT;;;"><br> </span><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:DE-AT;;;">Internet: <a href="http://www.edv-design.at/"><span style="color:#0563C1;;;">www.edv-design.at</span></a></span></p> <p class="MsoNormal"> </p> </div></div></div></div></font><BR>
<BR>
</body></html>