<font size=2 face="sans-serif">Hi, </font><br><br><font size=2 face="sans-serif">mmfsck is highly dependent on pagepool,
so my best guess would be, the the EMS node with only 16GB of pagepool
is sort of a bottleneck for the slow progress.</font><br><font size=2 face="sans-serif">You might want to try to temporarily
raise the pagepool on the service node to .. lets say 64GB, or restrict
the mmfsck to run only on the storage nodes. </font><br><br><font size=2 face="sans-serif">note: the fs mgr node will always be
part of the mmfsck nodes. so if it is located on the service node, mmchmgr
to a storage node first</font><br><br><font size=1 face="Arial"><br>Mit freundlichen Grüßen / Kind regards</font><p><font size=2 face="Arial"><b>Achim Rehor</b></font><p><table width=600 style="border-collapse:collapse;"><tr height=8><td width=600 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><div align=center><hr noshade></div><tr height=8><td width=600 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 face="Arial"> </font><tr height=8><td width=496 colspan=3 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 face="Arial">Software
Technical Support Specialist AIX/ Emea HPC Support</font><td width=103 rowspan=10 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><div align=right><img align=bottom src=cid:_1_D975D64CD975D0CC005D0FE8C12580DF style="border:0px solid;"></div><tr height=8><td width=496 colspan=3 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 face="Arial">IBM
Certified Advanced Technical Expert - Power Systems with AIX</font><tr height=8><td width=496 colspan=3 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 face="Arial">TSCC
Software Service, Dept. 7922</font><tr height=8><td width=496 colspan=3 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 face="Arial">Global
Technology Services </font><tr height=8><td width=496 colspan=3 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><div align=center><hr noshade></div><tr height=8><td width=44 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial">Phone:</font><td width=195 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial">+49-7034-274-7862</font><td width=257 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> IBM
Deutschland</font><tr height=8><td width=44 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial">E-Mail:</font><td width=195 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial">Achim.Rehor@de.ibm.com</font><td width=257 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> Am
Weiher 24</font><tr height=8><td width=44 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> </font><td width=195 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> </font><td width=257 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> 65451
Kelsterbach</font><tr height=8><td width=44 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> </font><td width=195 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> </font><td width=257 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> Germany</font><tr height=8><td width=44 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> </font><td width=195 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> </font><td width=257 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> </font><tr height=8><td width=600 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><div align=center><hr noshade></div><tr height=8><td width=600 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 face="Arial"> </font><tr height=8><td width=600 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#a2a2a2 face="Arial">IBM
Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter <br>Geschäftsführung: Martina Koederitz (Vorsitzende), Reinhard Reschke, Dieter
Scholz, Gregor Pillen, Ivo Koerner, Christian Noll <br>Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart,
HRB 14562 WEEE-Reg.-Nr. DE 99369940 </font></table><br><p><font size=3> </font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Mengxing Cheng <mxcheng@uchicago.edu></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">Eric Sperley <esperle@us.ibm.com></font><br><font size=1 color=#5f5f5f face="sans-serif">Cc:
</font><font size=1 face="sans-serif">gpfsug main discussion
list <gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">03/09/2017 10:24 PM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
mmfsck runs very slowly on a small filesystem</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=2 face="Tahoma">Eric, thank you very much for replying.
Here is the memory configuration and current usage. Note that mmfsck is
not running now. The two gss servers have the same 256GB memory and the
service node has 128GB.<br><br><br>1. service node:<br> total
used free shared
buffers cached<br>Mem: 125
58 66 0
0 4<br>-/+ buffers/cache: 53
71 <br>Swap: 7
0 7 <br></font><br><font size=2 face="Tahoma"> PID USER PR NI
VIRT RES SHR S %CPU %MEM TIME+ COMMAND
<br>12990 root 0 -20 71.0g 43g 885m S 7.6
34.4 9306:00 mmfsd
<br><br><br>2. gss nodes:<br>====================================<br> total
used free
shared buff/cache available<br>Mem: 251
210 37
0 4
36<br>Swap: 3
0 3<br><br> PID USER PR NI VIRT RES
SHR S %CPU %MEM TIME+ COMMAND<br>36770 root 0 -20 0.216t 0.192t 2.667g S 48.9
78.1 75684:09 /usr/lpp/mmfs/bin/mmfsd<br><br><br> The gss nodes' memory usage is so high because their pagepool is set to
192GB while the service node has 16GB pagepool.<br><br><br>Mengxing</font><br><font size=2 face="Tahoma"> </font><br><font size=1 face="Tahoma">---</font><br><font size=1 face="Tahoma">Mengxing Cheng, Ph.D. </font><br><font size=1 face="Tahoma">HPC System Administrator</font><br><font size=1 face="Tahoma">Research Computing Center</font><br><font size=1 face="Tahoma">The University of Chicago</font><br><br><font size=1 face="Tahoma">5607 S. Drexel Ave.</font><br><font size=1 face="Tahoma">Chicago, IL 60637</font><br><font size=1 face="Tahoma">email: mxcheng@uchicago.edu</font><br><font size=1 face="Tahoma">phone: (773) 702-4070</font><br><br><br><hr><br><font size=2 face="Tahoma"><b>From:</b> Eric Sperley [esperle@us.ibm.com]<b><br>Sent:</b> Thursday, March 09, 2017 3:13 PM<b><br>To:</b> Mengxing Cheng<b><br>Cc:</b> gpfsug main discussion list<b><br>Subject:</b> Re: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem</font><font size=3><br></font><br><font size=3>Mengxing,<br><br>It is nice meeting you.<br><br>I have seen a situation where the amount of RAM on a node can affect mmfsck
times. Do all the nodes have the same amount of RAM, or does the slow running
node have less RAM? </font><br><font size=2 face="Arial">Best Regards, Eric</font><br><font size=3><br></font><img src=cid:_4_0B8E93D40B8E917C005D0FE8C12580DF width=601 height=4 style="border:0px solid;"><table width=597 style="border-collapse:collapse;"><tr height=8><td width=12 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=4> </font><td width=328 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=4> </font><td width=245 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=4> </font><td width=12 valign=top style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><img align=bottom src=cid:_1_0B8EB0280B8EAC0C005D0FE8C12580DF width=1 height=1 style="border:0px solid;"><tr height=8><td width=12 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=4> </font><td width=328 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><img align=bottom src=cid:_4_0B8EBB580B8EB73C005D0FE8C12580DF style="border:0px solid;"><td width=245 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=4 color=#212100>Eric
Sperley</font><td width=12 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=4> </font><tr height=8><td width=12 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=4> </font><td width=328 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=4> </font><td width=245 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=2 color=#808080>SDI
Architect</font><td width=12 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=4> </font><tr height=8><td width=12 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=4> </font><td width=328 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=2 color=#808080><i>"Carpe
Diem"</i></font><td width=245 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=2 color=#4f4f4f>IBM
Systems</font><td width=12 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=4> </font><tr height=8><td width=12 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=4> </font><td width=328 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=4> </font><td width=245 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=2 color=#4f4f4f>esperle@us.ibm.com</font><td width=12 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=4> </font><tr height=8><td width=12 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=4> </font><td width=328 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=4> </font><td width=245 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=2 color=#4f4f4f>+15033088721</font><td width=12 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=4> </font></table><br><font size=3><br><br><br></font><img src=cid:_1_0B8DBC700B8E95FC005D0FE8C12580DF alt="Inactive hide details for Mengxing Cheng ---03/09/2017 11:24:02 AM---Dear all, My name is Mengxing Cheng and I am a HPC system " style="border:0px solid;"><font size=3 color=#424282>Mengxing
Cheng ---03/09/2017 11:24:02 AM---Dear all, My name is Mengxing Cheng and
I am a HPC system administrator at the University of Chicago</font><font size=3><br></font><font size=2 color=#5f5f5f><br>From: </font><font size=2>Mengxing Cheng <mxcheng@uchicago.edu></font><font size=2 color=#5f5f5f><br>To: </font><font size=2>"gpfsug-discuss@spectrumscale.org" <gpfsug-discuss@spectrumscale.org></font><font size=2 color=#5f5f5f><br>Date: </font><font size=2>03/09/2017 11:24 AM</font><font size=2 color=#5f5f5f><br>Subject: </font><font size=2>[gpfsug-discuss] mmfsck runs very slowly on
a small filesystem</font><font size=2 color=#5f5f5f><br>Sent by: </font><font size=2>gpfsug-discuss-bounces@spectrumscale.org</font><p><hr noshade><font size=3><br><br></font><font size=3 face="Tahoma"><br>Dear all,<br><br>My name is Mengxing Cheng and I am a HPC system administrator at the University
of Chicago. We have a GSS26 running gss2.5.10.3-3b and gpfs-4.2.0.3. <br><br>Recently, we run mmfsck on a relatively small filesystem with 14TB block
and 73863102 inodes but it was unusually slow so as to not be able to finish
in 48 hours. In contrast, mmfsck run on a filesystem with the same size
and inodes but sitting on a traditional IBM DS3512 storage took only 2
hours to complete. <br><br>In particular, the mmfsck run in parallel using 3 nodes within the GSS
storage cluster, we notice that one gss storage server scans inodes much
slower than the other gss storage server and the quorum service node. <br><br>Has anyone experience the same mmfsck performance issue? <br>Could anyone make recommendation to troubleshoot and improve mmfsck performance?
<br><br>Thank you!<br><br><br>Mengxing</font><font size=3><br><br></font><font size=2 face="Tahoma"><br>---<br>Mengxing Cheng, Ph.D. <br>HPC System Administrator<br>Research Computing Center<br>The University of Chicago</font><font size=3><br></font><font size=2 face="Tahoma"><br>5607 S. Drexel Ave.<br>Chicago, IL 60637<br>email: mxcheng@uchicago.edu<br>phone: (773) 702-4070</font><font size=3><br></font><tt><font size=3><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</font></tt><tt><font size=3 color=blue><u><br></u></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target=_blank><tt><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></tt></a><font size=3><br><br><br></font><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><p><p><BR>