<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoPlainText">Hey Simon,<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">I clicked your link but I think it went to a page that is not about this RFE:<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><img width="824" height="447" id="Picture_x0020_1" src="cid:image001.png@01D2CA56.41F66270"><o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Cheers,<o:p></o:p></p>
<p class="MsoPlainText">-Bryan<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">-----Original Message-----<br>
From: gpfsug-discuss-bounces@spectrumscale.org [mailto:gpfsug-discuss-bounces@spectrumscale.org] On Behalf Of Simon Thompson (IT Research Support)<br>
Sent: Thursday, May 11, 2017 12:49 PM<br>
To: gpfsug-discuss@spectrumscale.org<br>
Subject: [gpfsug-discuss] Edge case failure mode</p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Just following up on some discussions we had at the UG this week. I<o:p></o:p></p>
<p class="MsoPlainText">mentioned a few weeks back that we were having issues with failover of<o:p></o:p></p>
<p class="MsoPlainText">NFS, and we figured a work around to our clients for this so that failover<o:p></o:p></p>
<p class="MsoPlainText">works great now (plus there is some code fixes coming down the line as<o:p></o:p></p>
<p class="MsoPlainText">well to help).<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Here's my story of fun with protocol nodes ...<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Since then we've occasionally been seeing the load average of 1 CES node<o:p></o:p></p>
<p class="MsoPlainText">rise to over 400 and then its SOOO SLOW to respond to NFS and SMB clients.<o:p></o:p></p>
<p class="MsoPlainText">A lot of digging and we found that CTDB was reporting > 80% memory used,<o:p></o:p></p>
<p class="MsoPlainText">so we tweaked the page pool down to solve this.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Great we thought ... But alas that wasn't the cause.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Just to be clear 95% of the time, the CES node is fine, I can do and ls in<o:p></o:p></p>
<p class="MsoPlainText">the mounted file-systems and all is good. When the load rises to 400, an<o:p></o:p></p>
<p class="MsoPlainText">ls takes 20-30 seconds, so they are related, but what is the initial<o:p></o:p></p>
<p class="MsoPlainText">cause? Other CES nodes are 100% fine and if we do mmces node suspend, and<o:p></o:p></p>
<p class="MsoPlainText">then resume all is well on the node (and no other CES node assumes the<o:p></o:p></p>
<p class="MsoPlainText">problem as the IP moves). Its not always the same CES IP, node or even<o:p></o:p></p>
<p class="MsoPlainText">data centre, and most of the time is looks fine.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">I logged a ticket with OCF today, and one thing they suggested was to<o:p></o:p></p>
<p class="MsoPlainText">disable NFSv3 as they've seen similar behaviour at another site. As far as<o:p></o:p></p>
<p class="MsoPlainText">I know, all my NFS clients are v4, but sure we disable v3 anyway as its<o:p></o:p></p>
<p class="MsoPlainText">not actually needed. (Both at the ganesha layer, change the default for<o:p></o:p></p>
<p class="MsoPlainText">exports and reconfigure all existing exports to v4 only for good measure).<o:p></o:p></p>
<p class="MsoPlainText">That didn't help, but certainly worth a try!<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Note that my CES cluster is multi-cluster mounting the file-systems and<o:p></o:p></p>
<p class="MsoPlainText">from the POSIX side, its fine most of the time.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">We've used the mmnetverify command to check that all is well as well. Of<o:p></o:p></p>
<p class="MsoPlainText">course this only checks the local cluster, not remote nodes, but as we<o:p></o:p></p>
<p class="MsoPlainText">aren't seeing expels and can access the FS, we assume that the GPFS layer<o:p></o:p></p>
<p class="MsoPlainText">is working fine.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">So we finally log a PMR with IBM, I catch a node in a broken state and<o:p></o:p></p>
<p class="MsoPlainText">pull a trace from it and upload that, and ask what other traces they might<o:p></o:p></p>
<p class="MsoPlainText">want (apparently there is no protocol trace for NFS in 4.2.2-3).<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Now, when we run this, I note that its doing things like mmlsfileset to<o:p></o:p></p>
<p class="MsoPlainText">the remote storage, coming from two clusters and some of this is timing<o:p></o:p></p>
<p class="MsoPlainText">out. We've already had issues with rp_filter on remote nodes causing<o:p></o:p></p>
<p class="MsoPlainText">expels, but the storage backend here has only 1 nic, and we can mount and<o:p></o:p></p>
<p class="MsoPlainText">access it all fine.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">So why doesn't mmlsfileset work to this node (I can ping it - ICMP, not<o:p></o:p></p>
<p class="MsoPlainText">GPFS ping of course), but not make "admin" calls to it. Ssh appears to<o:p></o:p></p>
<p class="MsoPlainText">work fine as well BTW to it.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">So I check on my CES and this is multi-homed and rp_filter is enabled.<o:p></o:p></p>
<p class="MsoPlainText">Setting it to a value of 2, seems to make mmlsfileset work, so yes, I'm<o:p></o:p></p>
<p class="MsoPlainText">sure I'm an edge case, but it would be REALLY REALLY helpful to get<o:p></o:p></p>
<p class="MsoPlainText">mmnetverify to work across a cluster (e.g. I say this is a remote node and<o:p></o:p></p>
<p class="MsoPlainText">here's its FQDN, can you talk to it) which would have helped with<o:p></o:p></p>
<p class="MsoPlainText">diagnosis here. I'm not entirely sure why ssh etc would work and pass<o:p></o:p></p>
<p class="MsoPlainText">rp_filter, but not GPFS traffic (in some cases apparently), but I guess<o:p></o:p></p>
<p class="MsoPlainText">its something to do with how GPFS is binding and then the kernel routing<o:p></o:p></p>
<p class="MsoPlainText">layer.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">I'm still not sure if this is my root cause as the occurrences of the high<o:p></o:p></p>
<p class="MsoPlainText">load are a bit random (anything from every hour to being stable for 2-3<o:p></o:p></p>
<p class="MsoPlainText">days), but since making the rp_filter change this afternoon, so far ...?<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">I've created an RFE for mmnetverify to be able to test across a cluster...<o:p></o:p></p>
<p class="MsoPlainText"><a href="https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=10503"><span style="color:windowtext;text-decoration:none">https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=10503</span></a><o:p></o:p></p>
<p class="MsoPlainText">0<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Simon<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">_______________________________________________<o:p></o:p></p>
<p class="MsoPlainText">gpfsug-discuss mailing list<o:p></o:p></p>
<p class="MsoPlainText">gpfsug-discuss at spectrumscale.org<o:p></o:p></p>
<p class="MsoPlainText"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><span style="color:windowtext;text-decoration:none">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></a><o:p></o:p></p>
</div>
<br>
<hr>
<font face="Arial" color="Gray" size="1"><br>
Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this
email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness
or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial
product.<br>
</font>
</body>
</html>