<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" id="owaParaStyle">P {margin-top:0;margin-bottom:0;}</style>
</head>
<body fpstyle="1" ocsi="0">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">Eric, thank you very much for replying. Here is the memory configuration and current usage. Note that mmfsck is not running now. The two gss servers have the same 256GB memory and
the service node has 128GB.<br>
<br>
<br>
1. service node:<br>
total used free shared buffers cached<br>
Mem: 125 58 66 0 0 4<br>
-/+ buffers/cache: 53 71 <br>
Swap: 7 0 7 <br>
<br>
<div> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <br>
12990 root 0 -20 71.0g 43g 885m S 7.6 34.4 9306:00 mmfsd
<br>
<br>
<br>
2. gss nodes:<br>
====================================<br>
total used free shared buff/cache available<br>
Mem: 251 210 37 0 4 36<br>
Swap: 3 0 3<br>
<br>
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br>
36770 root 0 -20 0.216t 0.192t 2.667g S 48.9 78.1 75684:09 /usr/lpp/mmfs/bin/mmfsd<br>
<br>
<br>
The gss nodes' memory usage is so high because their pagepool is set to 192GB while the service node has 16GB pagepool.<br>
<br>
<br>
Mengxing<br>
<div style="font-family:Tahoma; font-size:13px">
<div><font size="1">---</font></div>
<div><font size="1">Mengxing Cheng, Ph.D. </font></div>
<div><font size="1">HPC System Administrator</font></div>
<div><font size="1">Research Computing Center</font></div>
<div><font size="1">The University of Chicago</font></div>
<div><font size="1"><br>
</font></div>
<div><font size="1">5607 S. Drexel Ave.</font></div>
<div><font size="1">Chicago, IL 60637</font></div>
<div><font size="1">email: mxcheng@uchicago.edu</font></div>
<div><font size="1">phone: (773) 702-4070</font></div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
<div style="font-family: Times New Roman; color: #000000; font-size: 16px">
<hr tabindex="-1">
<div id="divRpF926661" style="direction: ltr;"><font size="2" color="#000000" face="Tahoma"><b>From:</b> Eric Sperley [esperle@us.ibm.com]<br>
<b>Sent:</b> Thursday, March 09, 2017 3:13 PM<br>
<b>To:</b> Mengxing Cheng<br>
<b>Cc:</b> gpfsug main discussion list<br>
<b>Subject:</b> Re: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem<br>
</font><br>
</div>
<div></div>
<div>
<p>Mengxing,<br>
<br>
It is nice meeting you.<br>
<br>
I have seen a situation where the amount of RAM on a node can affect mmfsck times. Do all the nodes have the same amount of RAM, or does the slow running node have less RAM?
</p>
<p><font size="2" face="Arial">Best Regards, Eric</font></p>
<p><font size="2" face="Arial"></font><br>
<img src="cid:1__=8FBB0A4DDFE7DC3F8f9e8a93df938690918c8FB@" width="601" height="4"><br>
<table cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr valign="top">
<td width="12" valign="middle"><font size="4"> </font></td>
<td width="328" valign="middle"><font size="4"> </font></td>
<td width="245" valign="middle"><font size="4"> </font></td>
<td width="12"><img src="cid:2__=8FBB0A4DDFE7DC3F8f9e8a93df938690918c8FB@" alt="" width="1" height="1" border="0"></td>
</tr>
<tr valign="top">
<td width="12" valign="middle"><font size="4"> </font></td>
<td width="328" valign="middle"><img src="cid:3__=8FBB0A4DDFE7DC3F8f9e8a93df938690918c8FB@" width="203" height="41" align="bottom"></td>
<td width="245" valign="middle"><font size="4" color="#231F20" face="HelveticaNeue">Eric Sperley</font></td>
<td width="12" valign="middle"><font size="4"> </font></td>
</tr>
<tr valign="top">
<td width="12" valign="middle"><font size="4"> </font></td>
<td width="328" valign="middle"><font size="4"> </font></td>
<td width="245" valign="middle"><font size="2" color="#808287" face="HelveticaNeue">SDI Architect</font></td>
<td width="12" valign="middle"><font size="4"> </font></td>
</tr>
<tr valign="top">
<td width="12" valign="middle"><font size="4"> </font></td>
<td width="328" valign="middle"><i><font size="2" color="#6F7076" face="HelveticaNeue">"Carpe Diem"</font></i></td>
<td width="245" valign="middle"><font size="2" color="#4C4D53" face="HelveticaNeue">IBM Systems</font></td>
<td width="12" valign="middle"><font size="4"> </font></td>
</tr>
<tr valign="top">
<td width="12" valign="middle"><font size="4"> </font></td>
<td width="328" valign="middle"><font size="4"> </font></td>
<td width="245" valign="middle"><font size="2" color="#4C4D53" face="HelveticaNeue">esperle@us.ibm.com</font></td>
<td width="12" valign="middle"><font size="4"> </font></td>
</tr>
<tr valign="top">
<td width="12" valign="middle"><font size="4"> </font></td>
<td width="328" valign="middle"><font size="4"> </font></td>
<td width="245" valign="middle"><font size="2" color="#4C4D53" face="HelveticaNeue">+15033088721</font></td>
<td width="12" valign="middle"><font size="4"> </font></td>
</tr>
</tbody>
</table>
<br>
<br>
<br>
<img src="cid:4__=8FBB0A4DDFE7DC3F8f9e8a93df938690918c8FB@" alt="Inactive hide details for Mengxing Cheng ---03/09/2017 11:24:02 AM---Dear all, My name is Mengxing Cheng and I am a HPC system " width="16" height="16" border="0"><font color="#424282">Mengxing
Cheng ---03/09/2017 11:24:02 AM---Dear all, My name is Mengxing Cheng and I am a HPC system administrator at the University of Chicago</font><br>
<br>
<font size="2" color="#5F5F5F">From: </font><font size="2">Mengxing Cheng <mxcheng@uchicago.edu></font><br>
<font size="2" color="#5F5F5F">To: </font><font size="2">"gpfsug-discuss@spectrumscale.org" <gpfsug-discuss@spectrumscale.org></font><br>
<font size="2" color="#5F5F5F">Date: </font><font size="2">03/09/2017 11:24 AM</font><br>
<font size="2" color="#5F5F5F">Subject: </font><font size="2">[gpfsug-discuss] mmfsck runs very slowly on a small filesystem</font><br>
<font size="2" color="#5F5F5F">Sent by: </font><font size="2">gpfsug-discuss-bounces@spectrumscale.org</font><br>
</p>
<hr style="color:#8091A5" width="100%" noshade="" size="2" align="left">
<br>
<br>
<br>
<font face="Tahoma">Dear all,<br>
<br>
My name is Mengxing Cheng and I am a HPC system administrator at the University of Chicago. We have a GSS26 running gss2.5.10.3-3b and gpfs-4.2.0.3.
<br>
<br>
Recently, we run mmfsck on a relatively small filesystem with 14TB block and 73863102 inodes but it was unusually slow so as to not be able to finish in 48 hours. In contrast, mmfsck run on a filesystem with the same size and inodes but sitting on a traditional
IBM DS3512 storage took only 2 hours to complete. <br>
<br>
In particular, the mmfsck run in parallel using 3 nodes within the GSS storage cluster, we notice that one gss storage server scans inodes much slower than the other gss storage server and the quorum service node.
<br>
<br>
Has anyone experience the same mmfsck performance issue? <br>
Could anyone make recommendation to troubleshoot and improve mmfsck performance? <br>
<br>
Thank you!<br>
<br>
<br>
Mengxing</font><br>
<br>
<font face="Tahoma"></font><br>
<font size="2" face="Tahoma">---</font><br>
<font size="2" face="Tahoma">Mengxing Cheng, Ph.D. </font><br>
<font size="2" face="Tahoma">HPC System Administrator</font><br>
<font size="2" face="Tahoma">Research Computing Center</font><br>
<font size="2" face="Tahoma">The University of Chicago</font><br>
<br>
<font size="2" face="Tahoma">5607 S. Drexel Ave.</font><br>
<font size="2" face="Tahoma">Chicago, IL 60637</font><br>
<font size="2" face="Tahoma">email: mxcheng@uchicago.edu</font><br>
<font size="2" face="Tahoma">phone: (773) 702-4070</font><br>
<br>
<tt>_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org<br>
</tt><tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></tt><tt><br>
</tt><br>
<br>
<br>
</div>
</div>
</div>
</body>
</html>