<font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2"><div><div style="font-family: Verdana, Arial, Helvetica, sans-serif;"><span style="font-family: "Courier New", Courier, monospace; font-size: small;">Jonathan,</span><br></div><div style="font-family: Verdana, Arial, Helvetica, sans-serif;"><span style="font-family: "Courier New", Courier, monospace; font-size: small;"><br></span></div><div><font face="Courier New, Courier, monospace" size="2">my suspicion is that the GPFS daemon on </font><span style="font-family: "Courier New", Courier, monospace; font-size: small;">fqdn-new is not reachable via port 1191.</span></div><div><font face="Courier New, Courier, monospace" size="2">You can double check that by sending a lightweight CCR RPC to this daemon from another quorum node by attempting:</font></div><div><font face="Courier New, Courier, monospace" size="2"><br></font></div><div><font face="Courier New, Courier, monospace" size="2">mmccr echo -n </font><span style="font-family: "Courier New", Courier, monospace; font-size: small;">fqdn-new;echo $?</span></div><div><br></div><div><font face="Courier New, Courier, monospace" size="2">If this echo returns with a non-zero exit code the network settings must be verified. And even the other direction must</font></div><div><span style="font-family: "Courier New", Courier, monospace; font-size: small;">work: Node </span><font face="Courier New, Courier, monospace" size="2">fqdn-new must reach another quorum node, like (attempting on </font><span style="font-family: "Courier New", Courier, monospace; font-size: small;">fqdn-new)</span><span style="font-family: "Courier New", Courier, monospace; font-size: small;">:</span></div><div><font face="Courier New, Courier, monospace" size="2"><br></font></div><div><font face="Courier New, Courier, monospace" size="2">mmccr echo -n <QUORUM_NODE></font><span style="font-family: "Courier New", Courier, monospace; font-size: small;">;echo $?</span></div><div><br></div><font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2" style="font-family: Verdana, Arial, Helvetica, sans-serif;"><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt"><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt"><div dir="ltr"><div class="line1" style="font-size: 12px; font-family: arial, helvetica, sans-serif;">Mit freundlichen Grüßen / Kind regards</div><div class="line1" style="font-size: 12px; font-family: arial, helvetica, sans-serif;"><table border="0" cellpadding="0" cellspacing="0" style="font-family: arial, helvetica, sans-serif; border: 0px; table-layout: auto; width: 650px; font-size: 7pt;" width="680px"> <tbody> <tr> <td style="vertical-align: bottom;"> <div class="vcard" style="margin: 8px 0px;"><b class="fn n" style="font-size: 12pt;"><font color="#888888"><span class="given-name">Ralf</span> <span class="family-name">Eberhard</span></font></b><br> <span style="font-size: 9pt;">Spectrum Scale Developer</span><br> <span style="font-size: 9pt;">IBM Systems - Dept. M929</span><br> <span style="font-size: 9pt;">Mobile: +49 162 4159476</span><br> <span style="font-size: 9pt;">E-Mail: <a href="mailto:ralf.eberhard@de.ibm.com" target="_blank" rel="noopener noreferrer">ralf.eberhard@de.ibm.com</a></span></div> </td> </tr> </tbody></table><div style="font-family: arial, helvetica, sans-serif; font-size: 7pt; width: 250px;">IBM Deutschland Research & Development GmbH<br>Vorsitzender des Aufsichtsrats: Gregor Pillen<br>Geschäftsführung: Dirk Wittkopp<br>Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294<br><br><a href="https://www.ibm.com/privacy/us/en/">IBM Data Privacy Statement</a></div></div></div></div></div></font></div><br><br><div style="font-family: Verdana, Arial, Helvetica, sans-serif;"><font color="#990099"><a href="mailto:-----gpfsug-discuss-bounces@spectrumscale.org" target="_blank" rel="noopener noreferrer">-----gpfsug-discuss-bounces@spectrumscale.org</a> wrote: -----</font></div><div class="iNotesHistory" style="font-family: Verdana, Arial, Helvetica, sans-serif; padding-left: 5px;"><div style="padding-right:0px;padding-left:5px;border-left:solid black 2px;">To: <a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank" rel="noopener noreferrer">gpfsug-discuss@spectrumscale.org</a><br>From: <a href="mailto:gpfsug-discuss-request@spectrumscale.org" target="_blank" rel="noopener noreferrer">gpfsug-discuss-request@spectrumscale.org</a><br>Sent by: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank" rel="noopener noreferrer">gpfsug-discuss-bounces@spectrumscale.org</a><br>Date: 12/09/2021 05:04PM<br>Subject: [EXTERNAL] gpfsug-discuss Digest, Vol 119, Issue 7<br><br><div><font face="Courier New,Courier,monospace" size="2">Send gpfsug-discuss mailing list submissions to<br> <a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank" rel="noopener noreferrer">gpfsug-discuss@spectrumscale.org</a><br><br>To subscribe or unsubscribe via the World Wide Web, visit<br> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> <br>or, via email, send a message with subject or body 'help' to<br> <a href="mailto:gpfsug-discuss-request@spectrumscale.org" target="_blank" rel="noopener noreferrer">gpfsug-discuss-request@spectrumscale.org</a><br><br>You can reach the person managing the list at<br> <a href="mailto:gpfsug-discuss-owner@spectrumscale.org" target="_blank" rel="noopener noreferrer">gpfsug-discuss-owner@spectrumscale.org</a><br><br>When replying, please edit your Subject line so it is more specific<br>than "Re: Contents of gpfsug-discuss digest..."<br><br><br>Today's Topics:<br><br> 1. Re: alternate path between ESS Servers for Datamigration<br> (Olaf Weiser)<br> 2. Adding a quorum node (Jonathan Buzzard)<br> 3. Re: alternate path between ESS Servers for Datamigration<br> (Douglas O'flaherty)<br><br><br>----------------------------------------------------------------------<br><br>Message: 1<br>Date: Thu, 9 Dec 2021 12:04:28 +0000<br>From: "Olaf Weiser" <<a href="mailto:olaf.weiser@de.ibm.com" target="_blank" rel="noopener noreferrer">olaf.weiser@de.ibm.com</a>><br>To: <a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank" rel="noopener noreferrer">gpfsug-discuss@spectrumscale.org</a><br>Cc: <a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank" rel="noopener noreferrer">gpfsug-discuss@spectrumscale.org</a><br>Subject: Re: [gpfsug-discuss] alternate path between ESS Servers for<br> Datamigration<br>Message-ID:<br> <<a href="mailto:OF942131BB.73C8972F-ON002587A6.0041D89C-002587A6.004253B5@ibm.com" target="_blank" rel="noopener noreferrer">OF942131BB.73C8972F-ON002587A6.0041D89C-002587A6.004253B5@ibm.com</a>><br>Content-Type: text/plain; charset="us-ascii"<br><br>An HTML attachment was scrubbed...<br>URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20211209/33805b31/attachment-0001.html">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20211209/33805b31/attachment-0001.html</a> ><br><br>------------------------------<br><br>Message: 2<br>Date: Thu, 9 Dec 2021 12:36:08 +0000<br>From: Jonathan Buzzard <<a href="mailto:jonathan.buzzard@strath.ac.uk" target="_blank" rel="noopener noreferrer">jonathan.buzzard@strath.ac.uk</a>><br>To: "<a href="mailto:'gpfsug-discuss@spectrumscale.org'" target="_blank" rel="noopener noreferrer">'gpfsug-discuss@spectrumscale.org'</a>"<br> <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank" rel="noopener noreferrer">gpfsug-discuss@spectrumscale.org</a>><br>Subject: [gpfsug-discuss] Adding a quorum node<br>Message-ID: <<a href="mailto:73c81130-c120-d5f3-395f-4695e56905e1@strath.ac.uk" target="_blank" rel="noopener noreferrer">73c81130-c120-d5f3-395f-4695e56905e1@strath.ac.uk</a>><br>Content-Type: text/plain; charset=UTF-8; format=flowed<br><br><br>I am looking to replace the quorum node in our cluster. The RAID card in <br>the server we are currently using is a casualty of the RHEL8 SAS card <br>purge :-(<br><br>I have a "new" dual core server that is fully supported by RHEL8. After <br>some toing and throwing with IBM they agreed a Pentium G6400 is 70PVU a <br>core and two cores :-) That said it is currently running RHEL7 because <br>that's what the DSS-G nodes are running. The upgrade to RHEL8 is planned <br>for next year.<br><br>Anyway I have added it into the GPFS cluster all well and good and GPFS <br>is mounted just fine. However when I ran the command to make it a quorum <br>node I got the following error (sanitized to remove actual DNS names and <br>IP addresses<br><br>initialize (113, '<fqdn-new>', ('<IP address>', 1191)) failed (err 79)<br>server initialization failed (err 79)<br>mmchnode: Unexpected error from chnodes -n <br>1=<fqdn1>:1191,2<fqdn2>:1191,3=<fqdn3>:1191,113=<fqdn-new>:1191 -f 1 -P <br>1191 . Return code: 149<br>mmchnode: Unable to change the CCR quorum node configuration.<br>mmchnode: Command failed. Examine previous error messages to determine <br>cause.<br><br>fqdn-new is the new node and fqdn1/2/3 are the existing quorum nodes. I <br>want to remove fqdn3 in due course.<br><br>Anyone any idea what is going on? I thought you could change the quorum <br>nodes on the fly?<br><br><br>JAB.<br><br>-- <br>Jonathan A. Buzzard Tel: +44141-5483420<br>HPC System Administrator, ARCHIE-WeSt.<br>University of Strathclyde, John Anderson Building, Glasgow. G4 0NG<br><br><br>------------------------------<br><br>Message: 3<br>Date: Thu, 9 Dec 2021 16:04:28 +0000<br>From: "Douglas O'flaherty" <<a href="mailto:douglasof@us.ibm.com" target="_blank" rel="noopener noreferrer">douglasof@us.ibm.com</a>><br>To: "<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank" rel="noopener noreferrer">gpfsug-discuss@spectrumscale.org</a>"<br> <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank" rel="noopener noreferrer">gpfsug-discuss@spectrumscale.org</a>><br>Subject: Re: [gpfsug-discuss] alternate path between ESS Servers for<br> Datamigration<br>Message-ID:<br> <<a href="mailto:OFC5898FC6.442B7FC7-ON002587A6.00584C7A-1639065868448@ibm.com" target="_blank" rel="noopener noreferrer">OFC5898FC6.442B7FC7-ON002587A6.00584C7A-1639065868448@ibm.com</a>><br>Content-Type: text/plain; charset="utf-8"<br><br>Walter:<br> <br> Though not directly about your design, our work with NVIDIA on GPUdirect Storage and SuperPOD has shown how sensitive RDMA (IB & RoCE) to both MOFED and Firmware version compatibility can be. <br> <br> I would suggest anyone debugging RDMA issues should look at those closely. <br> <br> Doug<br> <br> by carrier pigeon<br> <br> <br> On Dec 9, 2021, 5:04:36 AM, <a href="mailto:gpfsug-discuss-request@spectrumscale.org" target="_blank" rel="noopener noreferrer">gpfsug-discuss-request@spectrumscale.org</a> wrote:<br> <br> From: <a href="mailto:gpfsug-discuss-request@spectrumscale.org" target="_blank" rel="noopener noreferrer">gpfsug-discuss-request@spectrumscale.org</a><br> To: <a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank" rel="noopener noreferrer">gpfsug-discuss@spectrumscale.org</a><br> Cc: <br> Date: Dec 9, 2021, 5:04:36 AM<br> Subject: [EXTERNAL] gpfsug-discuss Digest, Vol 119, Issue 5<br> <br> Send gpfsug-discuss mailing list submissions to <a href="mailto:gpfsug-discuss@spectrumscale.orgTo" target="_blank" rel="noopener noreferrer">gpfsug-discuss@spectrumscale.orgTo</a> subscribe or unsubscribe via the World Wide Web, visit <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> or, via email, send a message with subject or body 'help' to <a href="mailto:gpfsug-discuss-request@spectrumscale.orgYou" target="_blank" rel="noopener noreferrer">gpfsug-discuss-request@spectrumscale.orgYou</a> can reach the person managing the list at <a href="mailto:gpfsug-discuss-owner@spectrumscale.orgWhen" target="_blank" rel="noopener noreferrer">gpfsug-discuss-owner@spectrumscale.orgWhen</a> replying, please edit your Subject line so it is more specificthan "Re: Contents of gpfsug-discuss digest..."<br> <br> Send gpfsug-discuss mailing list submissions to <a href="mailto:gpfsug-discuss@spectrumscale.orgTo" target="_blank" rel="noopener noreferrer">gpfsug-discuss@spectrumscale.orgTo</a> subscribe or unsubscribe via the World Wide Web, visit <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> or, via email, send a message with subject or body 'help' to <a href="mailto:gpfsug-discuss-request@spectrumscale.orgYou" target="_blank" rel="noopener noreferrer">gpfsug-discuss-request@spectrumscale.orgYou</a> can reach the person managing the list at <a href="mailto:gpfsug-discuss-owner@spectrumscale.orgWhen" target="_blank" rel="noopener noreferrer">gpfsug-discuss-owner@spectrumscale.orgWhen</a> replying, please edit your Subject line so it is more specificthan "Re: Contents of gpfsug-discuss digest..."Today's Topics: 1. alternate path between ESS Servers for Datamigration (Walter Sklenka)<br> <br> Dear spectrum scale users! May I ask you a design question? We have an IB environment which is very mixed at the moment ( connecX3 ? connect-X6 with FDR , even FDR10 and with arrive of ESS5000SC7 now also HDR100 and HDR switches. We still have some big troubles in this fabric when using RDMA , a case at Mellanox and IBM is open . The environment has 3 old Building blocks 2xESSGL6 and 1x GL4 , from where we want to migrate the data to ess5000 , ( mmdelvdisk +qos) Due to the current problems with RDMA we though eventually we could try a workaround : If you are interested there is Maybe you can find the attachment ? We build 2 separate fabrics , the ess-IO servers attached to both blue and green and all other cluster members and all remote clusters only to fabric blue The daemon interfaces (IPoIP) are on fabric blue It is the aim to setup rdma only on the ess-ioServers in the fabric green , in the blue we must use IPoIB (tcp) Do you think datamigration would work be<br> tween ess01,ess02,? to ess07,ess08 via RDMA ? Or is it principally not possible to make a rdma network only for a subset of a cluster (though this subset would be reachable via other fabric) ? Thank you very much for any input ! Best regards walter Mit freundlichen Gr??en<br> Walter Sklenka<br> Technical Consultant EDV-Design Informationstechnologie GmbH<br> Giefinggasse 6/1/2, A-1210 Wien<br> Tel: +43 1 29 22 165-31<br> Fax: +43 1 29 22 165-90<br> E-Mail: <a href="mailto:sklenka@edv-design.at" target="_blank" rel="noopener noreferrer">sklenka@edv-design.at</a><br> Internet: <a href="http://www.edv-design.at" target="_blank" rel="noopener noreferrer">www.edv-design.at</a> <br><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20211209/f2fa4bf1/attachment.html">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20211209/f2fa4bf1/attachment.html</a> ><br><br>------------------------------<br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> <br><br><br>End of gpfsug-discuss Digest, Vol 119, Issue 7<br>**********************************************<br></font></div></div></div></font><BR>
<BR>