<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style>
<!--
@font-face
{font-family:Calibri}
@font-face
{font-family:Tahoma}
@font-face
{font-family:Consolas}
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif"}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline}
pre
{margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New","serif"}
span.HTMLPreformattedChar
{font-family:Consolas}
span.EmailStyle19
{font-family:"Calibri","sans-serif";
color:#1F497D}
.MsoChpDefault
{font-family:"Calibri","sans-serif"}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.WordSection1
{}
-->
</style>
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri","sans-serif"; color:#1F497D">This sounds like a serious problem and you should open a PMR with IBM to get direct guidance.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri","sans-serif"; color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri","sans-serif"; color:#1F497D">I normally will take a GPFS trace during a problem like this from all of the nodes that are affected or directly involved in the operation.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri","sans-serif"; color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri","sans-serif"; color:#1F497D">Hope that helps,</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri","sans-serif"; color:#1F497D">-Bryan</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri","sans-serif"; color:#1F497D"> </span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt; font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt; font-family:"Tahoma","sans-serif""> gpfsug-discuss-bounces@gpfsug.org [mailto:gpfsug-discuss-bounces@gpfsug.org]
<b>On Behalf Of </b>Sabuj Pattanayek<br>
<b>Sent:</b> Thursday, May 29, 2014 8:34 PM<br>
<b>To:</b> gpfsug main discussion list<br>
<b>Subject:</b> Re: [gpfsug-discuss] filesystem manager crashes every time mmdelsnapshot (from either the filesystem manager or some other nsd/client) is called</span></p>
<p class="MsoNormal"> </p>
<div>
<p class="MsoNormal">This is still happening in 3.5.0.18 and when a snapshot is being deleted it slows NFS read speeds to a crawl (but not gpfs and not NFS writes).</p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"> </p>
<div>
<p class="MsoNormal">On Thu, May 15, 2014 at 7:48 AM, Sabuj Pattanayek <<a href="mailto:sabujp@gmail.com" target="_blank">sabujp@gmail.com</a>> wrote:</p>
<div>
<p class="MsoNormal">Hi all,</p>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">We're running 3.5.0.17 now and it looks like the filesystem manager automatically reboots (and sometimes fails to automatically reboot) after mmdelsnapshot is called, either from the filesystem manager itself or from some other nsd/node
. It didn't start happening immediately after we updated to 17, but we never had this issue when we were at 3.5.0.11 . The error mmdelsnapshot throws at some point is :</p>
</div>
<div>
<pre>Lost connection to file system daemon.</pre>
<pre>mmdelsnapshot: An internode connection between GPFS nodes was disrupted.</pre>
<pre>mmdelsnapshot: Command failed. Examine previous error messages to</pre>
<pre>determine cause.</pre>
<pre>It also causes an mmfs generic error and or a kernel: BUG: soft lockup - CPU#15 stuck for 67s! [mmfsd:39266], the latter causes the system to not reboot itself (which is actually worse), but the former does.</pre>
<pre> </pre>
<pre>It also causes some havoc with CNFS file locking even after the filesystem manager is rebooted and has come up :</pre>
<pre> </pre>
<pre>May 15 07:10:12 mako-nsd1 sm-notify[19387]: Failed to bind RPC socket:</pre>
<pre>Address already in use</pre>
<pre> </pre>
<pre><span style="font-family:"Arial","sans-serif"">May 15 07:21:03 mako-nsd1 sm-notify[11052]: Invalid bind address or port </span></pre>
<pre>for RPC socket: Name or service not known</pre>
<pre> </pre>
<pre>Saw some snapshot related fixes in 3.5.0.18, anyone seen this behavior or know if it's fixed in 18?</pre>
<pre> </pre>
<pre>Thanks,</pre>
<pre>Sabuj</pre>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
</div>
</div>
<p class="MsoNormal"> </p>
</div>
</div>
<br>
<hr>
<font face="Arial" color="Gray" size="1"><br>
Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this
email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness
or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial
product.<br>
</font>
</body>
</html>