<div dir="ltr"><div><br></div>Yes, I've been struggelig with something similiar this week. Ganesha dying with SIGABRT -- nothing else logged. After catching a few coredumps, it has been identified as a problem with some udp-communication during mounts from solaris clients. Disabling udp as transport on the shares serverside didn't help. It was suggested to use "mount -o tcp" or whatever the solaris version of this is -- but we haven't tested this. So far the downgrade to v2.3.2 has been our workaround.<div><br></div><div>PMR: 48669,080,678</div><div><br></div><div><br></div><div> -jf</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 23, 2018 at 12:38 AM, Ray Coetzee <span dir="ltr"><<a href="mailto:coetzee.ray@gmail.com" target="_blank">coetzee.ray@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Good evening all<div><br></div><div>I'm working with IBM on a PMR where ganesha is segfaulting or causing kernel panics on one group of CES nodes.</div><div><br></div><div>We have 12 identical CES nodes split into two groups of 6 nodes each & have been running with <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">RHEL 7.3 & GPFS 5.0.0-1 since 5.0.0-1 was released.</span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div>Only one group started having issues Monday morning where ganesha would segfault and the mounts would move over to the remaining nodes.</div><div>The remaining nodes then start to fall over like dominos within minutes or hours to the point that all CES nodes are "failed" according to "mmces node list" and the VIP's are unassigned.</div><div><br></div><div>Recovering the nodes are extremely finicky and works for a few minutes or hours before segfaulting again. </div><div>Most times a complete stop of Ganesha on all nodes & then only starting it on two random nodes allow mounts to recover for a while.</div><div><br></div><div>None of the following has helped:<br></div><div>A reboot of all nodes.</div><div>Refresh CCR config file with mmsdrrestore</div><div><span style="color:rgb(84,84,84)">Remove/add CES from nodes.</span><br></div><div><span style="color:rgb(84,84,84)">Reinstall GPFS & protocol rpms</span></div><div><span style="color:rgb(84,84,84)">Update to 5.0.0-2</span></div><div><span style="color:rgb(84,84,84)">Fresh reinstall of a node</span></div><div><span style="color:rgb(84,84,84)">Network checks out with no dropped packets on either data or export networks. </span></div><div><span style="color:rgb(84,84,84)"><br></span></div><div>The only temporary fix so far has been to downrev ganesha to 2.3.2 from 2.5.3 on the affected nodes.</div><div><br></div><div>While waiting for IBM development, has anyone seen something similar maybe?</div><div class="gmail_extra"><br clear="all"><div><div class="m_8992299872510994342gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><span style="font-size:small"><font color="#888888">Kind regards</font></span><span style="font-size:small"><br><div><p>Ray Coetzee<br><br></p></div><div></div></span><br></div></div></div></div></div>
<br><div class="gmail_quote">On Sat, Apr 21, 2018 at 12:00 PM, <span dir="ltr"><<a href="mailto:gpfsug-discuss-request@spectrumscale.org" target="_blank">gpfsug-discuss-request@<wbr>spectrumscale.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Send gpfsug-discuss mailing list submissions to<br>
<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.o<wbr>rg</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/list<wbr>info/gpfsug-discuss</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:gpfsug-discuss-request@spectrumscale.org" target="_blank">gpfsug-discuss-request@spectru<wbr>mscale.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:gpfsug-discuss-owner@spectrumscale.org" target="_blank">gpfsug-discuss-owner@spectrums<wbr>cale.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of gpfsug-discuss digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: UK Meeting - tooling Spectrum Scale (Grunenberg, Renar)<br>
2. Re: UK Meeting - tooling Spectrum Scale<br>
(Simon Thompson (IT Research Support))<br>
<br>
<br>
------------------------------<wbr>------------------------------<wbr>----------<br>
<br>
Message: 1<br>
Date: Fri, 20 Apr 2018 14:01:55 +0000<br>
From: "Grunenberg, Renar" <<a href="mailto:Renar.Grunenberg@huk-coburg.de" target="_blank">Renar.Grunenberg@huk-coburg.d<wbr>e</a>><br>
To: "'<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale<wbr>.org</a>'"<br>
<<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a>><br>
Subject: Re: [gpfsug-discuss] UK Meeting - tooling Spectrum Scale<br>
Message-ID: <<a href="mailto:fb4c0ca7ece5462d96948e562803e77e@SMXRF105.msg.hukrf.de" target="_blank">fb4c0ca7ece5462d96948e562803e<wbr>77e@SMXRF105.msg.hukrf.de</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Hallo Simon,<br>
are there any reason why the link of the presentation from Yong ZY Zheng(Cognitive, ML, Hortonworks) is not linked.<br>
<br>
Renar Grunenberg<br>
Abteilung Informatik ? Betrieb<br>
<br>
HUK-COBURG<br>
Bahnhofsplatz<br>
96444 Coburg<br>
Telefon: 09561 96-44110<br>
Telefax: 09561 96-44104<br>
E-Mail: <a href="mailto:Renar.Grunenberg@huk-coburg.de" target="_blank">Renar.Grunenberg@huk-coburg.de</a><br>
Internet: <a href="http://www.huk.de" rel="noreferrer" target="_blank">www.huk.de</a><br>
______________________________<wbr>__<br>
HUK-COBURG Haftpflicht-Unterst?tzungs-Kas<wbr>se kraftfahrender Beamter Deutschlands a. G. in Coburg<br>
Reg.-Gericht Coburg HRB 100; St.-Nr. 9212/101/00021<br>
Sitz der Gesellschaft: Bahnhofsplatz, 96444 Coburg<br>
Vorsitzender des Aufsichtsrats: Prof. Dr. Heinrich R. Schradin.<br>
Vorstand: Klaus-J?rgen Heitmann (Sprecher), Stefan Gronbach, Dr. Hans Olav Her?y, Dr. J?rg Rheinl?nder (stv.), Sarah R?ssler, Daniel Thomas.<br>
______________________________<wbr>__<br>
Diese Nachricht enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen.<br>
Wenn Sie nicht der richtige Adressat sind oder diese Nachricht irrt?mlich erhalten haben,<br>
informieren Sie bitte sofort den Absender und vernichten Sie diese Nachricht.<br>
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Nachricht ist nicht gestattet.<br>
<br>
This information may contain confidential and/or privileged information.<br>
If you are not the intended recipient (or have received this information in error) please notify the<br>
sender immediately and destroy this information.<br>
Any unauthorized copying, disclosure or distribution of the material in this information is strictly forbidden.<br>
______________________________<wbr>__<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20180420/91e3d84d/attachment-0001.html" rel="noreferrer" target="_blank">http://gpfsug.org/pipermail/g<wbr>pfsug-discuss/attachments/2018<wbr>0420/91e3d84d/attachment-0001.<wbr>html</a>><br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Fri, 20 Apr 2018 14:12:11 +0000<br>
From: "Simon Thompson (IT Research Support)" <<a href="mailto:S.J.Thompson@bham.ac.uk" target="_blank">S.J.Thompson@bham.ac.uk</a>><br>
To: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a>><br>
Subject: Re: [gpfsug-discuss] UK Meeting - tooling Spectrum Scale<br>
Message-ID: <<a href="mailto:14C2312C-1B54-45E9-B867-3D9E479A52B6@bham.ac.uk" target="_blank">14C2312C-1B54-45E9-B867-3D9E4<wbr>79A52B6@bham.ac.uk</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Sorry, it was a typo from my side.<br>
<br>
The talks that are missing we are chasing for copies of the slides that we can release.<br>
<br>
Simon<br>
<br>
From: <<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectr<wbr>umscale.org</a>> on behalf of "<a href="mailto:Renar.Grunenberg@huk-coburg.de" target="_blank">Renar.Grunenberg@huk-coburg.d<wbr>e</a>" <<a href="mailto:Renar.Grunenberg@huk-coburg.de" target="_blank">Renar.Grunenberg@huk-coburg.d<wbr>e</a>><br>
Reply-To: "<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a>" <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a>><br>
Date: Friday, 20 April 2018 at 15:02<br>
To: "<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a>" <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a>><br>
Subject: Re: [gpfsug-discuss] UK Meeting - tooling Spectrum Scale<br>
<br>
Hallo Simon,<br>
are there any reason why the link of the presentation from Yong ZY Zheng(Cognitive, ML, Hortonworks) is not linked.<br>
<br>
Renar Grunenberg<br>
Abteilung Informatik ? Betrieb<br>
<br>
HUK-COBURG<br>
Bahnhofsplatz<br>
96444 Coburg<br>
Telefon:<br>
<br>
09561 96-44110<br>
<br>
Telefax:<br>
<br>
09561 96-44104<br>
<br>
E-Mail:<br>
<br>
<a href="mailto:Renar.Grunenberg@huk-coburg.de" target="_blank">Renar.Grunenberg@huk-coburg.de</a><br>
<br>
Internet:<br>
<br>
<a href="http://www.huk.de" rel="noreferrer" target="_blank">www.huk.de</a><br>
<br>
______________________________<wbr>__<br>
HUK-COBURG Haftpflicht-Unterst?tzungs-Kas<wbr>se kraftfahrender Beamter Deutschlands a. G. in Coburg<br>
Reg.-Gericht Coburg HRB 100; St.-Nr. 9212/101/00021<br>
Sitz der Gesellschaft: Bahnhofsplatz, 96444 Coburg<br>
Vorsitzender des Aufsichtsrats: Prof. Dr. Heinrich R. Schradin.<br>
Vorstand: Klaus-J?rgen Heitmann (Sprecher), Stefan Gronbach, Dr. Hans Olav Her?y, Dr. J?rg Rheinl?nder (stv.), Sarah R?ssler, Daniel Thomas.<br>
______________________________<wbr>__<br>
Diese Nachricht enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen.<br>
Wenn Sie nicht der richtige Adressat sind oder diese Nachricht irrt?mlich erhalten haben,<br>
informieren Sie bitte sofort den Absender und vernichten Sie diese Nachricht.<br>
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Nachricht ist nicht gestattet.<br>
<br>
This information may contain confidential and/or privileged information.<br>
If you are not the intended recipient (or have received this information in error) please notify the<br>
sender immediately and destroy this information.<br>
Any unauthorized copying, disclosure or distribution of the material in this information is strictly forbidden.<br>
______________________________<wbr>__<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20180420/0b8e9ffa/attachment-0001.html" rel="noreferrer" target="_blank">http://gpfsug.org/pipermail/g<wbr>pfsug-discuss/attachments/2018<wbr>0420/0b8e9ffa/attachment-0001.<wbr>html</a>><br>
<br>
------------------------------<br>
<br>
______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/list<wbr>info/gpfsug-discuss</a><br>
<br>
<br>
End of gpfsug-discuss Digest, Vol 75, Issue 34<br>
******************************<wbr>****************<br>
</blockquote></div><br></div></div>
<br>______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a><br>
<br></blockquote></div><br></div>