<html><body bgcolor="#FFFFFF"><p>Unfortunately, at the moment there's no safe mechanism to show the usage statistics for different NSD queues. "mmfsadm saferdump nsd" as implemented doesn't acquire locks when parsing internal data structures. Now, NSD data structures are fairly static, as much things go, so the risk of following a stale pointer and hitting a segfault isn't particularly significant. I don't think I remember ever seeing mmfsd crash with NSD dump code on the stack. That said, this isn't code that's tested and known to be safe for production use. I haven't seen a case myself where an mmfsd thread gets stuck running this dump command, either, but Bob has. If that condition ever reoccurs, I'd be interested in seeing debug data.<br><br>I agree that there's value in giving a sysadmin insight into the inner workings of the NSD server machinery, in particular the queue dynamics. mmdiag should be enhanced to allow this. That'd be a very reasonable (and doable) RFE.<br><br>yuri<br><br><img width="16" height="16" src="cid:1__=07BB0A81DFE3BB548f9e8a93df938690918c07B@" border="0" alt="Inactive hide details for "Oesterlin, Robert" ---08/17/2016 04:45:30 AM---Hi Aaron You did a perfect job of explaining a situat"><font color="#424282">"Oesterlin, Robert" ---08/17/2016 04:45:30 AM---Hi Aaron You did a perfect job of explaining a situation I've run into time after time - high latenc</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">"Oesterlin, Robert" <Robert.Oesterlin@nuance.com></font><br><font size="2" color="#5F5F5F">To: </font><font size="2">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org>, </font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">08/17/2016 04:45 AM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">Re: [gpfsug-discuss] Monitor NSD server queue?</font><br><font size="2" color="#5F5F5F">Sent by: </font><font size="2">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br><br><br><font face="Arial">Hi Aaron</font><br><font face="Arial"> </font><br><font face="Arial">You did a perfect job of explaining a situation I've run into time after time - high latency on the disk subsystem causing a backup in the NSD queues. I was doing what you suggested not to do - "mmfsadm saferdump nsd' and looking at the queues. In my case 'mmfsadm saferdump" would usually work or hang, rather than kill mmfsd. But - the hang usually resulted it a tied up thread in mmfsd, so that's no good either.</font><br><font face="Arial"> </font><br><font face="Arial">I wish I had better news - this is the only way I've found to get visibility to these queues. IBM hasn't seen fit to gives us a way to safely look at these. I personally think it's a bug that we can't safely dump these structures, as they give insight as to what's actually going on inside the NSD server. </font><br><font face="Arial"> </font><br><font face="Arial">Yuri, Sven - thoughts?</font><br><font face="Arial"> </font><br><font face="Calibri"> </font><br><font face="Arial">Bob Oesterlin<br>Sr Storage Engineer, Nuance HPC Grid<br></font><br><font face="Arial"> </font><br><font face="Arial"> </font><br><b><font size="4" face="Calibri">From: </font></b><font size="4" face="Calibri"><gpfsug-discuss-bounces@spectrumscale.org> on behalf of "Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]" <aaron.s.knister@nasa.gov></font><b><font size="4" face="Calibri"><br>Reply-To: </font></b><font size="4" face="Calibri">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></font><b><font size="4" face="Calibri"><br>Date: </font></b><font size="4" face="Calibri">Tuesday, August 16, 2016 at 8:46 PM</font><b><font size="4" face="Calibri"><br>To: </font></b><font size="4" face="Calibri">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></font><b><font size="4" face="Calibri"><br>Subject: </font></b><font size="4" face="Calibri">[EXTERNAL] [gpfsug-discuss] Monitor NSD server queue?</font><br><font size="4" face="Times New Roman"> </font><br><font size="4" face="Times New Roman">Hi Everyone, </font><br><font size="4" face="Times New Roman"> </font><br><font size="4" face="Times New Roman">We ran into a rather interesting situation over the past week. We had a job that was pounding the ever loving crap out of one of our filesystems (called dnb02) doing about 15GB/s of reads. We had other jobs experience a slowdown on a different filesystem (called dnb41) that uses entirely separate backend storage. What I can't figure out is why this other filesystem was affected. I've checked IB bandwidth and congestion, Fibre channel bandwidth and errors, Ethernet bandwidth congestion, looked at the mmpmon nsd_ds counters (including disk request wait time), and checked out the disk iowait values from collectl. I simply can't account for the slowdown on the other filesystem. The only thing I can think of is the high latency on dnb02's NSDs caused the mmfsd NSD queues to back up. </font><br><font size="4" face="Times New Roman"> </font><br><font size="4" face="Times New Roman">Here's my question-- how can I monitor the state of th NSD queues? I can't find anything in mmdiag. An mmfsadm saferdump NSD shows me the queues and their status. I'm just not sure calling saferdump NSD every 10 seconds to monitor this data is going to end well. I've seen saferdump NSD cause mmfsd to die and that's from a task we only run every 6 hours that calls saferdump NSD. </font><br><font size="4" face="Times New Roman"> </font><br><font size="4" face="Times New Roman">Any thoughts/ideas here would be great. </font><br><font size="4" face="Times New Roman"> </font><br><font size="4" face="Times New Roman">Thanks!</font><br><font size="4" face="Times New Roman"> </font><br><font size="4" face="Times New Roman">-Aaron</font><tt>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></tt><tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></tt><tt><br></tt><br><BR>
</body></html>