<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Helvetica, sans-serif;">
<div>
<div>Hi Patrick</div>
<div><br>
</div>
<div>I was going to mail you directly – but this may help spark some discussion in this area. GPFS (pardon the use of the “old school" term – You need something easier to type that Spectrum Scale) problem determination is one of those areas that is (sometimes)
more of an art than a science. IBM publishes a PD guide, and it’s a good start but doesn’t cover all the bases.</div>
<div><br>
</div>
<div>- In the GPFS log (/var/mmfs/gen/mmfslog) there are a lot of messages generated. I continue to come across ones that are not documented – or documented poorly. EVERYTHING that ends up in ANY log needs to be documented.</div>
<div>- The PD guide gives some basic things to look at for many of the error messages, but doesn’t go into alternative explanation for many errors. Example: When a node gets expelled, the PD guide tells you it’s a communication issue, when it fact in may be
related to other things like Linux network tuning. Covering all the possible causes is hard, but you can improve this.</div>
<div>- GPFS waiter information – understanding and analyzing this is key to getting to the bottom of many problems. The waiter information is not well documented. You should include at least a basic guide on how to use waiter information in determining cluster
problems. Related: Undocumented config options. You can come across some by doing “mmdiag —config”. Using some of these can help you – or get you in trouble in the long run. If I can see the option, document it.</div>
<div>- Make sure that all information I might come across online is accurate, especially on those sites managed by IBM. The Developerworks wiki has great information, but there is a lot of information out there that’s out of date or inaccurate. This leads to
confusion.</div>
<div>- The automatic deadlock detection implemented in 4.1 can be useful, but it also can be problematic in a large cluster when you get into problems. Firing off traces and taking dumps in an automated manner can cause more problems if you have a large cluster.
I ended up turning it off.</div>
<div>- GPFS doesn’t have anything setup to alert you when conditions occur that may require your attention. There are some alerting capabilities that you can customize, but something out of the box might be useful. I know there is work going on in this area.</div>
<div><br>
</div>
<div><br>
</div>
<div>mmces – I did some early testing on this but haven’t had a chance to upgrade my protocol nodes to the new level. Upgrading 1000’s of node across many cluster is – challenging :-) The newer commands are a great start. I like the ability to list out events
related to a particular protocol. </div>
<div><br>
</div>
<div>I could go on… Feel free to contact me directly for a more detailed discussion: robert.oesterlin @ nuance.com</div>
<div>
<div id="">
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<span style="font-family: Calibri; font-size: medium;"><br>
</span></div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica, sans-serif; font-size: 14px;">
<font face="Helvetica">Bob Oesterlin<br>
Sr Storage Engineer, Nuance Communications<br>
</font></div>
</div>
</div>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:12pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span><<a href="mailto:gpfsug-discuss-bounces@gpfsug.org">gpfsug-discuss-bounces@gpfsug.org</a>> on behalf of Patrick Byrne<br>
<span style="font-weight:bold">Reply-To: </span>gpfsug main discussion list<br>
<span style="font-weight:bold">Date: </span>Thursday, October 1, 2015 at 5:09 AM<br>
<span style="font-weight:bold">To: </span>"<a href="mailto:gpfsug-discuss@gpfsug.org">gpfsug-discuss@gpfsug.org</a>"<br>
<span style="font-weight:bold">Subject: </span>[gpfsug-discuss] Problem Determination<br>
</div>
<div><br>
</div>
<div>
<div>
<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt">
<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt">
<div dir="ltr">Hi all,</div>
<div dir="ltr"> </div>
<div dir="ltr">As I'm sure some of you aware, problem determination is an area that we are looking to try and make significant improvements to over the coming releases of Spectrum Scale. To help us target the areas we work to improve and make it as useful as
possible I am trying to get as much feedback as I can about different problems users have, and how people go about solving them.</div>
<div dir="ltr"> </div>
<div dir="ltr">I am interested in hearing everything from day to day annoyances to problems that have caused major frustration in trying to track down the root cause. Where possible it would be great to hear how the problems were dealt with as well, so that
others can benefit from your experience. Feel free to reply to the mailing list - maybe others have seen similar problems and could provide tips for the future - or to me directly if you'd prefer (<a href="mailto:patbyrne@uk.ibm.com">patbyrne@uk.ibm.com</a>).</div>
<div dir="ltr"> </div>
<div dir="ltr">On a related note, in 4.1.1 there was a component added that monitors the state of the various protocols that are now supported (NFS, SMB, Object). The output from this is available with the 'mmces state' and 'mmces events' CLIs and I would like
to get feedback from anyone who has had the chance make use of this. Is it useful? How could it be improved? We are looking at the possibility of extending this component to cover more than just protocols, so any feedback would be greatly appreciated.</div>
<div dir="ltr"> </div>
<div dir="ltr">Thanks in advance,</div>
<div dir="ltr"> </div>
<div dir="ltr">Patrick Byrne<br>
IBM Spectrum Scale - Development Engineer<br>
IBM Systems - Manchester Lab<br>
IBM UK Limited</div>
</div>
</div>
<br>
</div>
</div>
</span>
</body>
</html>