<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" id="owaParaStyle">P {margin-top:0;margin-bottom:0;}</style>
</head>
<body fpstyle="1" ocsi="0">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">Achim, thank you very much!<br>
<br>
Mengxing<br>
<div><br>
<div style="font-family:Tahoma; font-size:13px">
<div><font size="1">---</font></div>
<div><font size="1">Mengxing Cheng, Ph.D. </font></div>
<div><font size="1">HPC System Administrator</font></div>
<div><font size="1">Research Computing Center</font></div>
<div><font size="1">The University of Chicago</font></div>
<div><font size="1"><br>
</font></div>
<div><font size="1">5607 S. Drexel Ave.</font></div>
<div><font size="1">Chicago, IL 60637</font></div>
<div><font size="1">email: mxcheng@uchicago.edu</font></div>
<div><font size="1">phone: (773) 702-4070</font></div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
<div style="font-family: Times New Roman; color: #000000; font-size: 16px">
<hr tabindex="-1">
<div id="divRpF433094" style="direction: ltr;"><font size="2" color="#000000" face="Tahoma"><b>From:</b> gpfsug-discuss-bounces@spectrumscale.org [gpfsug-discuss-bounces@spectrumscale.org] on behalf of Achim Rehor [Achim.Rehor@de.ibm.com]<br>
<b>Sent:</b> Friday, March 10, 2017 10:56 AM<br>
<b>To:</b> gpfsug main discussion list<br>
<b>Subject:</b> Re: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem<br>
</font><br>
</div>
<div></div>
<div><font size="2" face="sans-serif">Hi, </font><br>
<br>
<font size="2" face="sans-serif">mmfsck is highly dependent on pagepool, so my best guess would be, the the EMS node with only 16GB of pagepool is sort of a bottleneck for the slow progress.</font><br>
<font size="2" face="sans-serif">You might want to try to temporarily raise the pagepool on the service node to .. lets say 64GB, or restrict the mmfsck to run only on the storage nodes.
</font><br>
<br>
<font size="2" face="sans-serif">note: the fs mgr node will always be part of the mmfsck nodes. so if it is located on the service node, mmchmgr to a storage node first</font><br>
<br>
<font size="1" face="Arial"><br>
Mit freundlichen Grüßen / Kind regards</font>
<p><font size="2" face="Arial"><b>Achim Rehor</b></font></p>
<p>
<table style="border-collapse:collapse" width="600">
<tbody>
<tr height="8">
<td colspan="4" style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="600">
<div align="center">
<hr noshade="">
</div>
</td>
</tr>
<tr height="8">
<td colspan="4" style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="600">
<font size="1" face="Arial"> </font></td>
</tr>
<tr height="8">
<td colspan="3" style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="496">
<font size="1" face="Arial">Software Technical Support Specialist AIX/ Emea HPC Support</font></td>
<td rowspan="10" style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="103">
<div align="right"><img src="cid:_1_D975D64CD975D0CC005D0FE8C12580DF" style="border:0px solid" align="bottom"></div>
</td>
</tr>
<tr height="8">
<td colspan="3" style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="496">
<font size="1" face="Arial">IBM Certified Advanced Technical Expert - Power Systems with AIX</font></td>
</tr>
<tr height="8">
<td colspan="3" style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="496">
<font size="1" face="Arial">TSCC Software Service, Dept. 7922</font></td>
</tr>
<tr height="8">
<td colspan="3" style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="496">
<font size="1" face="Arial">Global Technology Services </font></td>
</tr>
<tr height="8">
<td colspan="3" style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="496">
<div align="center">
<hr noshade="">
</div>
</td>
</tr>
<tr height="8">
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="44">
<font size="1" color="#0060a0" face="Arial">Phone:</font></td>
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="195">
<font size="1" color="#0060a0" face="Arial">+49-7034-274-7862</font></td>
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="257">
<font size="1" color="#0060a0" face="Arial"> IBM Deutschland</font></td>
</tr>
<tr height="8">
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="44">
<font size="1" color="#0060a0" face="Arial">E-Mail:</font></td>
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="195">
<font size="1" color="#0060a0" face="Arial">Achim.Rehor@de.ibm.com</font></td>
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="257">
<font size="1" color="#0060a0" face="Arial"> Am Weiher 24</font></td>
</tr>
<tr height="8">
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="44">
<font size="1" color="#0060a0" face="Arial"> </font></td>
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="195">
<font size="1" color="#0060a0" face="Arial"> </font></td>
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="257">
<font size="1" color="#0060a0" face="Arial"> 65451 Kelsterbach</font></td>
</tr>
<tr height="8">
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="44">
<font size="1" color="#0060a0" face="Arial"> </font></td>
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="195">
<font size="1" color="#0060a0" face="Arial"> </font></td>
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="257">
<font size="1" color="#0060a0" face="Arial"> Germany</font></td>
</tr>
<tr height="8">
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="44">
<font size="1" color="#0060a0" face="Arial"> </font></td>
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="195">
<font size="1" color="#0060a0" face="Arial"> </font></td>
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="257">
<font size="1" color="#0060a0" face="Arial"> </font></td>
</tr>
<tr height="8">
<td colspan="4" style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="600">
<div align="center">
<hr noshade="">
</div>
</td>
</tr>
<tr height="8">
<td colspan="4" style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="600">
<font size="1" face="Arial"> </font></td>
</tr>
<tr height="8">
<td colspan="4" style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="600">
<font size="1" color="#a2a2a2" face="Arial">IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter
<br>
Geschäftsführung: Martina Koederitz (Vorsitzende), Reinhard Reschke, Dieter Scholz, Gregor Pillen, Ivo Koerner, Christian Noll
<br>
Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 WEEE-Reg.-Nr. DE 99369940
</font></td>
</tr>
</tbody>
</table>
<br>
</p>
<p><font size="3"> </font><br>
<br>
<br>
<br>
<font size="1" color="#5f5f5f" face="sans-serif">From: </font><font size="1" face="sans-serif">Mengxing Cheng <mxcheng@uchicago.edu></font><br>
<font size="1" color="#5f5f5f" face="sans-serif">To: </font><font size="1" face="sans-serif">Eric Sperley <esperle@us.ibm.com></font><br>
<font size="1" color="#5f5f5f" face="sans-serif">Cc: </font><font size="1" face="sans-serif">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></font><br>
<font size="1" color="#5f5f5f" face="sans-serif">Date: </font><font size="1" face="sans-serif">03/09/2017 10:24 PM</font><br>
<font size="1" color="#5f5f5f" face="sans-serif">Subject: </font><font size="1" face="sans-serif">Re: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem</font><br>
<font size="1" color="#5f5f5f" face="sans-serif">Sent by: </font><font size="1" face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br>
</p>
<hr noshade="">
<br>
<br>
<br>
<font size="2" face="Tahoma">Eric, thank you very much for replying. Here is the memory configuration and current usage. Note that mmfsck is not running now. The two gss servers have the same 256GB memory and the service node has 128GB.<br>
<br>
<br>
1. service node:<br>
total used free shared buffers cached<br>
Mem: 125 58 66 0 0 4<br>
-/+ buffers/cache: 53 71 <br>
Swap: 7 0 7 <br>
</font><br>
<font size="2" face="Tahoma"> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
<br>
12990 root 0 -20 71.0g 43g 885m S 7.6 34.4 9306:00 mmfsd
<br>
<br>
<br>
2. gss nodes:<br>
====================================<br>
total used free shared buff/cache available<br>
Mem: 251 210 37 0 4 36<br>
Swap: 3 0 3<br>
<br>
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br>
36770 root 0 -20 0.216t 0.192t 2.667g S 48.9 78.1 75684:09 /usr/lpp/mmfs/bin/mmfsd<br>
<br>
<br>
The gss nodes' memory usage is so high because their pagepool is set to 192GB while the service node has 16GB pagepool.<br>
<br>
<br>
Mengxing</font><br>
<font size="2" face="Tahoma"> </font><br>
<font size="1" face="Tahoma">---</font><br>
<font size="1" face="Tahoma">Mengxing Cheng, Ph.D. </font><br>
<font size="1" face="Tahoma">HPC System Administrator</font><br>
<font size="1" face="Tahoma">Research Computing Center</font><br>
<font size="1" face="Tahoma">The University of Chicago</font><br>
<br>
<font size="1" face="Tahoma">5607 S. Drexel Ave.</font><br>
<font size="1" face="Tahoma">Chicago, IL 60637</font><br>
<font size="1" face="Tahoma">email: mxcheng@uchicago.edu</font><br>
<font size="1" face="Tahoma">phone: (773) 702-4070</font><br>
<br>
<br>
<hr>
<br>
<font size="2" face="Tahoma"><b>From:</b> Eric Sperley [esperle@us.ibm.com]<b><br>
Sent:</b> Thursday, March 09, 2017 3:13 PM<b><br>
To:</b> Mengxing Cheng<b><br>
Cc:</b> gpfsug main discussion list<b><br>
Subject:</b> Re: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem</font><font size="3"><br>
</font><br>
<font size="3">Mengxing,<br>
<br>
It is nice meeting you.<br>
<br>
I have seen a situation where the amount of RAM on a node can affect mmfsck times. Do all the nodes have the same amount of RAM, or does the slow running node have less RAM?
</font><br>
<font size="2" face="Arial">Best Regards, Eric</font><br>
<font size="3"><br>
</font><img src="cid:_4_0B8E93D40B8E917C005D0FE8C12580DF" style="border:0px solid" width="601" height="4">
<table style="border-collapse:collapse" width="597">
<tbody>
<tr height="8">
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="12">
<font size="4"> </font></td>
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="328">
<font size="4"> </font></td>
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="245">
<font size="4"> </font></td>
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="12" valign="top">
<img src="cid:_1_0B8EB0280B8EAC0C005D0FE8C12580DF" style="border:0px solid" width="1" height="1" align="bottom"></td>
</tr>
<tr height="8">
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="12">
<font size="4"> </font></td>
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="328">
<img src="cid:_4_0B8EBB580B8EB73C005D0FE8C12580DF" style="border:0px solid" align="bottom"></td>
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="245">
<font size="4" color="#212100">Eric Sperley</font></td>
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="12">
<font size="4"> </font></td>
</tr>
<tr height="8">
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="12">
<font size="4"> </font></td>
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="328">
<font size="4"> </font></td>
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="245">
<font size="2" color="#808080">SDI Architect</font></td>
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="12">
<font size="4"> </font></td>
</tr>
<tr height="8">
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="12">
<font size="4"> </font></td>
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="328">
<font size="2" color="#808080"><i>"Carpe Diem"</i></font></td>
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="245">
<font size="2" color="#4f4f4f">IBM Systems</font></td>
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="12">
<font size="4"> </font></td>
</tr>
<tr height="8">
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="12">
<font size="4"> </font></td>
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="328">
<font size="4"> </font></td>
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="245">
<font size="2" color="#4f4f4f">esperle@us.ibm.com</font></td>
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="12">
<font size="4"> </font></td>
</tr>
<tr height="8">
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="12">
<font size="4"> </font></td>
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="328">
<font size="4"> </font></td>
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="245">
<font size="2" color="#4f4f4f">+15033088721</font></td>
<td style="border-style:none none none none; border-color:#000000; border-width:0px 0px 0px 0px; padding:0px 0px" width="12">
<font size="4"> </font></td>
</tr>
</tbody>
</table>
<br>
<font size="3"><br>
<br>
<br>
</font><img src="cid:_1_0B8DBC700B8E95FC005D0FE8C12580DF" alt="Inactive hide details for Mengxing Cheng ---03/09/2017 11:24:02 AM---Dear all, My name is Mengxing Cheng and I am a HPC system " style="border:0px solid"><font size="3" color="#424282">Mengxing
Cheng ---03/09/2017 11:24:02 AM---Dear all, My name is Mengxing Cheng and I am a HPC system administrator at the University of Chicago</font><font size="3"><br>
</font><font size="2" color="#5f5f5f"><br>
From: </font><font size="2">Mengxing Cheng <mxcheng@uchicago.edu></font><font size="2" color="#5f5f5f"><br>
To: </font><font size="2">"gpfsug-discuss@spectrumscale.org" <gpfsug-discuss@spectrumscale.org></font><font size="2" color="#5f5f5f"><br>
Date: </font><font size="2">03/09/2017 11:24 AM</font><font size="2" color="#5f5f5f"><br>
Subject: </font><font size="2">[gpfsug-discuss] mmfsck runs very slowly on a small filesystem</font><font size="2" color="#5f5f5f"><br>
Sent by: </font><font size="2">gpfsug-discuss-bounces@spectrumscale.org</font>
<p></p>
<hr noshade="">
<font size="3"><br>
<br>
</font><font size="3" face="Tahoma"><br>
Dear all,<br>
<br>
My name is Mengxing Cheng and I am a HPC system administrator at the University of Chicago. We have a GSS26 running gss2.5.10.3-3b and gpfs-4.2.0.3.
<br>
<br>
Recently, we run mmfsck on a relatively small filesystem with 14TB block and 73863102 inodes but it was unusually slow so as to not be able to finish in 48 hours. In contrast, mmfsck run on a filesystem with the same size and inodes but sitting on a traditional
IBM DS3512 storage took only 2 hours to complete. <br>
<br>
In particular, the mmfsck run in parallel using 3 nodes within the GSS storage cluster, we notice that one gss storage server scans inodes much slower than the other gss storage server and the quorum service node.
<br>
<br>
Has anyone experience the same mmfsck performance issue? <br>
Could anyone make recommendation to troubleshoot and improve mmfsck performance? <br>
<br>
Thank you!<br>
<br>
<br>
Mengxing</font><font size="3"><br>
<br>
</font><font size="2" face="Tahoma"><br>
---<br>
Mengxing Cheng, Ph.D. <br>
HPC System Administrator<br>
Research Computing Center<br>
The University of Chicago</font><font size="3"><br>
</font><font size="2" face="Tahoma"><br>
5607 S. Drexel Ave.<br>
Chicago, IL 60637<br>
email: mxcheng@uchicago.edu<br>
phone: (773) 702-4070</font><font size="3"><br>
</font><tt><font size="3"><br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org</font></tt><tt><font size="3" color="blue"><u><br>
</u></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><tt><font size="3" color="blue"><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></tt></a><font size="3"><br>
<br>
<br>
</font><tt><font size="2">_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org<br>
</font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><tt><font size="2">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size="2"><br>
</font></tt>
<p></p>
<p><br>
</p>
</div>
</div>
</div>
</body>
</html>