<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Helvetica, sans-serif;">
<div>
<div>This command is just using ssh to all the nodes and dumping the waiter information and collecting it. That means if the node is down, slow to respond, or there are a large number of nodes, it could take a while to return. In my 400-500 node clusters this
command usually take less than 10 seconds. I do prefix the command with a timeout value in case a node is hung up and ssh never returns (which it sometimes does, and that’s not the fault of GPFS) Something like this:</div>
<div><br>
</div>
<div>timeout 45s /usr/lpp/mmfs/bin/mmlsnode -N waiters –L</div>
<div><br>
</div>
<div>This means I get incomplete information, but if you don’t you end up piling up a lot of hung up commands. I would check over your cluster carefully to see if there are other issues that might cause ssh to hang up – which could impact other GPFS commands
that distribute via ssh.</div>
<div><br>
</div>
<div>Another approach would be to dump the waiters locally on each node, send node specific information to the database, and then sum it up using the graphing software.</div>
<div>
<div id="">
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<span style="font-family: Calibri; font-size: medium;"><br>
</span></div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica, sans-serif; font-size: 14px;">
<font face="Helvetica">Bob Oesterlin<br>
Sr Storage Engineer, Nuance HPC Grid<br>
</font></div>
</div>
</div>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:12pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span><<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-bounces@spectrumscale.org</a>> on behalf of Roland Pabel <<a href="mailto:dr.roland.pabel@gmail.com">dr.roland.pabel@gmail.com</a>><br>
<span style="font-weight:bold">Organization: </span>RRZK Uni Köln<br>
<span style="font-weight:bold">Reply-To: </span>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a>><br>
<span style="font-weight:bold">Date: </span>Friday, April 15, 2016 at 10:50 AM<br>
<span style="font-weight:bold">To: </span>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [gpfsug-discuss] Executing Callbacks on other Nodes<br>
</div>
<div><br>
</div>
<span style="mso-bookmark:_MailOriginalBody">
<div>
<div>
<div>Hi,</div>
<div><br>
</div>
<div>In our cluster, mmlsnode –N waiters –L takes about 25 seconds to run. So </div>
<div>running it every 30 seconds is a bit close. I'll try running it once a minute
</div>
<div>and then incorporating this into our graphing. </div>
<div><br>
</div>
<div>Maybe the command is so slow for me because a few nodes are down?</div>
<div>Is there a parameter to mmlsnode to configure the timeout?</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</span></span>
</body>
</html>