[gpfsug-discuss] mmfsck runs very slowly on a small filesystem

[Mengxing Cheng]

Eric, thank you very much for replying. Here is the memory configuration and current usage. Note that mmfsck is not running now. The two gss servers have the same 256GB memory and the service node has 128GB.


1. service node:
                   total       used       free     shared    buffers     cached
Mem:           125         58         66          0          0          4
-/+ buffers/cache:         53         71
Swap:            7           0          7

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
12990 root       0 -20 71.0g  43g 885m S  7.6 34.4   9306:00 mmfsd


2. gss nodes:
====================================
                  total        used        free      shared  buff/cache   available
Mem:            251         210          37           0           4          36
Swap:             3           0           3

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
36770 root       0 -20  0.216t 0.192t 2.667g S  48.9 78.1  75684:09 /usr/lpp/mmfs/bin/mmfsd


 The gss nodes' memory usage is so high because their pagepool is set to 192GB while the service node has 16GB pagepool.


Mengxing

---
Mengxing Cheng, Ph.D.
HPC System Administrator
Research Computing Center
The University of Chicago

5607 S. Drexel Ave.
Chicago, IL 60637
email:    mxcheng at uchicago.edu
phone:  (773) 702-4070


________________________________
From: Eric Sperley [esperle at us.ibm.com]
Sent: Thursday, March 09, 2017 3:13 PM
To: Mengxing Cheng
Cc: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem


Mengxing,

It is nice meeting you.

I have seen a situation where the amount of RAM on a node can affect mmfsck times. Do all the nodes have the same amount of RAM, or does the slow running node have less RAM?

Best Regards, Eric



        [cid:3__=8FBB0A4DDFE7DC3F8f9e8a93df938690918c8FB@]      Eric Sperley
                SDI Architect
        "Carpe Diem"    IBM Systems
                esperle at us.ibm.com
                +15033088721



[Inactive hide details for Mengxing Cheng ---03/09/2017 11:24:02 AM---Dear all, My name is Mengxing Cheng and I am a HPC system]Mengxing Cheng ---03/09/2017 11:24:02 AM---Dear all, My name is Mengxing Cheng and I am a HPC system administrator at the University of Chicago

From: Mengxing Cheng <mxcheng at uchicago.edu>
To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Date: 03/09/2017 11:24 AM
Subject: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem
Sent by: gpfsug-discuss-bounces at spectrumscale.org

________________________________



Dear all,

My name is Mengxing Cheng and I am a HPC system administrator at the University of Chicago. We have a GSS26 running gss2.5.10.3-3b and gpfs-4.2.0.3.

Recently, we run mmfsck on a relatively small filesystem with 14TB block and 73863102 inodes but it was unusually slow so as to not be able to finish in 48 hours. In contrast, mmfsck run on a filesystem with the same size and inodes but sitting on a traditional IBM DS3512 storage took only 2 hours to complete.

In particular, the mmfsck run in parallel using 3 nodes within the GSS storage cluster, we notice that one gss storage server scans inodes much slower than the other gss storage server and the quorum service node.

Has anyone experience the same mmfsck performance issue?
Could anyone make recommendation to troubleshoot and improve mmfsck performance?

Thank you!


Mengxing


---
Mengxing Cheng, Ph.D.
HPC System Administrator
Research Computing Center
The University of Chicago

5607 S. Drexel Ave.
Chicago, IL 60637
email: mxcheng at uchicago.edu
phone: (773) 702-4070

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170309/6ad55d6b/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1C803747.gif
Type: image/gif
Size: 481 bytes
Desc: 1C803747.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170309/6ad55d6b/attachment-0008.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ecblank.gif
Type: image/gif
Size: 45 bytes
Desc: ecblank.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170309/6ad55d6b/attachment-0009.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1C166252.gif
Type: image/gif
Size: 2322 bytes
Desc: 1C166252.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170309/6ad55d6b/attachment-0010.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: graycol.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170309/6ad55d6b/attachment-0011.gif>