<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:宋体;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:__;
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:Helvetica;
color:windowtext;
font-weight:normal;
font-style:normal;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Helvetica">OK, after a bit of a delay due to a hectic travel week, here is some more information on my GPFS performance collection. At the bottom, I have links to my server and client zimon config
files and a link to my presentation at SSUG Argonne in June. I didn't actually present it but included it in case there was interest.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Helvetica"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Helvetica">I used to do a home brew system of period calls to mmpmon to collect data, sticking them into a kafka database. This was a bit cumbersome and when SS 4.2 arrived, I switched over to the
built in performance sensors (zimon) to collect the data. IBM has a "as-is" bridge between Grafana and the Zimon collector that works reasonably well - they were supposed to release it but it's been delayed - I will ask about it again and post more information
if I get it.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Helvetica"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Helvetica">My biggest struggle with the zimon configuration is the large memory requirement of the collector with large clusters (many clients, file systems, NSDs). I ended up deploying a 6 collector
federation of 16gb per collector for my larger clusters -0 even then I have to limit the number of stats and amount of time I retain it. IBM is aware of the memory issue and I believe they are looking at ways to reduce it.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Helvetica"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Helvetica">As for what specific metrics I tend to look at:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Helvetica"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Helvetica">gpfs_fis_bytes_read (written) - aggregated file system read and write stats<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Helvetica">gpfs_nsdpool_bytes_read (written) - aggregated pool stats, as I have data and metadata split<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Helvetica">gpfs_fs_tot_disk_wait_rd (wr) - NSD disk wait stats<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Helvetica"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Helvetica">These seem to make the most sense for me to get an overall sense of how things are going. I have a bunch of other more details dashboards for individual file systems and clients that
help me get details. The built-in SS GUI is pretty good for small clusters, and is getting some improvements in 4.2.1 that might make me take a closer look at it again.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Helvetica"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Helvetica">I also look at the RPC waiters stats - no present in 4.2.0 grafana, but I hear are coming in 4.2.1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Helvetica"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Helvetica">My SSUG Argonne Presentation (I didn't talk due to time constraints): http://files.gpfsug.org/presentations/2016/anl-june/SSUG_Nuance_PerfTools.pdf<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Helvetica"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Helvetica">Zimon server config file: https://www.dropbox.com/s/gvtfhhqfpsknfnh/ZIMonSensors.cfg.server?dl=0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Helvetica">Zimon client config file: https://www.dropbox.com/s/k5i6rcnaco4vxu6/ZIMonSensors.cfg.client?dl=0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Helvetica"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Calibri;color:black"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Helvetica;color:black">Bob Oesterlin<br>
Sr Storage Engineer, Nuance HPC Grid<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Helvetica"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Helvetica"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-family:Calibri;color:black">From: </span>
</b><span style="font-family:Calibri;color:black"><gpfsug-discuss-bounces@spectrumscale.org> on behalf of Brian Marshall <mimarsh2@vt.edu><br>
<b>Reply-To: </b>gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
<b>Date: </b>Wednesday, July 13, 2016 at 8:43 AM<br>
<b>To: </b>"gpfsug-discuss@spectrumscale.org" <gpfsug-discuss@spectrumscale.org><br>
<b>Subject: </b>[EXTERNAL] Re: [gpfsug-discuss] Aggregating filesystem performance (Oesterlin, Robert)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal">Robert, <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">1) Do you see any noticeable performance impact by running the performance monitoring?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">2) Can you share the zimon configuration that you use? i.e. what metrics do you find most useful?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thank you,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Brian Marshall<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>