[gpfsug-discuss] Sub-block size not quite as expected on GPFS 5 filesystem?

Buterbaugh, Kevin L Kevin.Buterbaugh at Vanderbilt.Edu
Fri Aug 3 13:49:48 BST 2018


Hi All,

Aargh - now I really do feel like an idiot!  I had set up the stanza file over a week ago … then had to work on production issues … and completely forgot about setting the block size in the pool stanzas there.  But at least we all now know that stanza files override command line arguments to mmcrfs.

My apologies…

Kevin

On Aug 3, 2018, at 1:01 AM, Olaf Weiser <olaf.weiser at de.ibm.com<mailto:olaf.weiser at de.ibm.com>> wrote:

Can u share your stanza file ?

Von meinem iPhone gesendet

Am 02.08.2018 um 23:15 schrieb Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>>:

OK, so hold on … NOW what’s going on???  I deleted the filesystem … went to lunch … came back an hour later … recreated the filesystem with a metadata block size of 4 MB … and I STILL have a 1 MB block size in the system pool and the wrong fragment size in other pools…

Kevin

/root/gpfs
root at testnsd1# mmdelfs gpfs5
All data on the following disks of gpfs5 will be destroyed:
    test21A3nsd
    test21A4nsd
    test21B3nsd
    test21B4nsd
    test23Ansd
    test23Bnsd
    test23Cnsd
    test24Ansd
    test24Bnsd
    test24Cnsd
    test25Ansd
    test25Bnsd
    test25Cnsd
Completed deletion of file system /dev/gpfs5.
mmdelfs: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
/root/gpfs
root at testnsd1# mmcrfs gpfs5 -F ~/gpfs/gpfs5.stanza -A yes -B 4M -E yes -i 4096 -j scatter -k all -K whenpossible -m 2 -M 3 -n 32 -Q yes -r 1 -R 3 -T /gpfs5 -v yes --nofilesetdf --metadata-block-size 4M

The following disks of gpfs5 will be formatted on node testnsd3:
    test21A3nsd: size 953609 MB
    test21A4nsd: size 953609 MB
    test21B3nsd: size 953609 MB
    test21B4nsd: size 953609 MB
    test23Ansd: size 15259744 MB
    test23Bnsd: size 15259744 MB
    test23Cnsd: size 1907468 MB
    test24Ansd: size 15259744 MB
    test24Bnsd: size 15259744 MB
    test24Cnsd: size 1907468 MB
    test25Ansd: size 15259744 MB
    test25Bnsd: size 15259744 MB
    test25Cnsd: size 1907468 MB
Formatting file system ...
Disks up to size 8.29 TB can be added to storage pool system.
Disks up to size 16.60 TB can be added to storage pool raid1.
Disks up to size 132.62 TB can be added to storage pool raid6.
Creating Inode File
  12 % complete on Thu Aug  2 13:16:26 2018
  25 % complete on Thu Aug  2 13:16:31 2018
  38 % complete on Thu Aug  2 13:16:36 2018
  50 % complete on Thu Aug  2 13:16:41 2018
  62 % complete on Thu Aug  2 13:16:46 2018
  74 % complete on Thu Aug  2 13:16:52 2018
  85 % complete on Thu Aug  2 13:16:57 2018
  96 % complete on Thu Aug  2 13:17:02 2018
 100 % complete on Thu Aug  2 13:17:03 2018
Creating Allocation Maps
Creating Log Files
   3 % complete on Thu Aug  2 13:17:09 2018
  28 % complete on Thu Aug  2 13:17:15 2018
  53 % complete on Thu Aug  2 13:17:20 2018
  78 % complete on Thu Aug  2 13:17:26 2018
 100 % complete on Thu Aug  2 13:17:27 2018
Clearing Inode Allocation Map
Clearing Block Allocation Map
Formatting Allocation Map for storage pool system
  98 % complete on Thu Aug  2 13:17:34 2018
 100 % complete on Thu Aug  2 13:17:34 2018
Formatting Allocation Map for storage pool raid1
  52 % complete on Thu Aug  2 13:17:39 2018
 100 % complete on Thu Aug  2 13:17:43 2018
Formatting Allocation Map for storage pool raid6
  24 % complete on Thu Aug  2 13:17:48 2018
  50 % complete on Thu Aug  2 13:17:53 2018
  74 % complete on Thu Aug  2 13:17:58 2018
  99 % complete on Thu Aug  2 13:18:03 2018
 100 % complete on Thu Aug  2 13:18:03 2018
Completed creation of file system /dev/gpfs5.
mmcrfs: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
/root/gpfs
root at testnsd1# mmlsfs gpfs5
flag                value                    description
------------------- ------------------------ -----------------------------------
 -f                 8192                     Minimum fragment (subblock) size in bytes (system pool)
                    32768                    Minimum fragment (subblock) size in bytes (other pools)
 -i                 4096                     Inode size in bytes
 -I                 32768                    Indirect block size in bytes
 -m                 2                        Default number of metadata replicas
 -M                 3                        Maximum number of metadata replicas
 -r                 1                        Default number of data replicas
 -R                 3                        Maximum number of data replicas
 -j                 scatter                  Block allocation type
 -D                 nfs4                     File locking semantics in effect
 -k                 all                      ACL semantics in effect
 -n                 32                       Estimated number of nodes that will mount file system
 -B                 1048576                  Block size (system pool)
                    4194304                  Block size (other pools)
 -Q                 user;group;fileset       Quotas accounting enabled
                    user;group;fileset       Quotas enforced
                    none                     Default quotas enabled
 --perfileset-quota No                       Per-fileset quota enforcement
 --filesetdf        No                       Fileset df enabled?
 -V                 19.01 (5.0.1.0)          File system version
 --create-time      Thu Aug  2 13:16:47 2018 File system creation time
 -z                 No                       Is DMAPI enabled?
 -L                 33554432                 Logfile size
 -E                 Yes                      Exact mtime mount option
 -S                 relatime                 Suppress atime mount option
 -K                 whenpossible             Strict replica allocation option
 --fastea           Yes                      Fast external attributes enabled?
 --encryption       No                       Encryption enabled?
 --inode-limit      101095424                Maximum number of inodes
 --log-replicas     0                        Number of log replicas
 --is4KAligned      Yes                      is4KAligned?
 --rapid-repair     Yes                      rapidRepair enabled?
 --write-cache-threshold 0                   HAWC Threshold (max 65536)
 --subblocks-per-full-block 128              Number of subblocks per full block
 -P                 system;raid1;raid6       Disk storage pools in file system
 --file-audit-log   No                       File Audit Logging enabled?
 --maintenance-mode No                       Maintenance Mode enabled?
 -d                 test21A3nsd;test21A4nsd;test21B3nsd;test21B4nsd;test23Ansd;test23Bnsd;test23Cnsd;test24Ansd;test24Bnsd;test24Cnsd;test25Ansd;test25Bnsd;test25Cnsd  Disks in file system
 -A                 yes                      Automatic mount option
 -o                 none                     Additional mount options
 -T                 /gpfs5                   Default mount point
 --mount-priority   0                        Mount priority
/root/gpfs
root at testnsd1#


—
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633


On Aug 2, 2018, at 3:31 PM, Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>> wrote:

Hi All,

Thanks for all the responses on this, although I have the sneaking suspicion that the most significant thing that is going to come out of this thread is the knowledge that Sven has left IBM for DDN.  ;-) or :-( or :-O depending on your perspective.

Anyway … we have done some testing which has shown that a 4 MB block size is best for those workloads that use “normal” sized files.  However, we - like many similar institutions - support a mixed workload, so the 128K fragment size that comes with that is not optimal for the primarily biomedical type applications that literally create millions of very small files.  That’s why we settled on 1 MB as a compromise.

So we’re very eager to now test with GPFS 5, a 4 MB block size, and a 8K fragment size.  I’m recreating my test cluster filesystem now with that config … so 4 MB block size on the metadata only system pool, too.

Thanks to all who took the time to respond to this thread.  I hope it’s been beneficial to others as well…

Kevin

—
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633

On Aug 1, 2018, at 7:11 PM, Andrew Beattie <abeattie at au1.ibm.com<mailto:abeattie at au1.ibm.com>> wrote:

I too would second the comment about doing testing specific to your environment

We recently deployed a number of ESS building blocks into a customer site that was specifically being used for a mixed HPC workload.

We spent more than a week playing with different block sizes for both data and metadata trying to identify which variation would provide the best mix of both metadata performance and data performance.  one thing we noticed very early on is that MDtest and IOR both respond very differently as you play with both block size and subblock size.  What works for one use case may be a very poor option for another use case.

Interestingly enough it turned out that the best overall option for our particular use case was an 8MB block size with 32k sub blocks -- as that gave us good Metadata performance and good sequential data performance

which is probably why 32k sub block was the default for so many years ....
Andrew Beattie
Software Defined Storage  - IT Specialist
Phone: 614-2133-7927
E-mail: abeattie at au1.ibm.com<mailto:abeattie at au1.ibm.com>


----- Original message -----
From: "Marc A Kaplan" <makaplan at us.ibm.com<mailto:makaplan at us.ibm.com>>
Sent by: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc:
Subject: Re: [gpfsug-discuss] Sub-block size not quite as expected on GPFS 5 filesystem?
Date: Thu, Aug 2, 2018 10:01 AM

Firstly, I do suggest that you run some tests and see how much, if any, difference the settings that are available make in performance and/or storage utilization.

Secondly, as I and others have hinted at, deeper in the system, there may be additional parameters and settings.  Sometimes they are available via commands, and/or configuration settings, sometimes not.

Sometimes that's just because we didn't want to overwhelm you or ourselves with yet more "tuning knobs".

Sometimes it's because we made some component more tunable than we really needed, but did not make all the interconnected components equally or as widely tunable.
Sometimes it's because we want to save you from making ridiculous settings that would lead to problems...

OTOH, as I wrote before, if a burning requirement surfaces, things may change from release to release... Just as for so many years subblocks per block seemed forever frozen at the number 32.  Now it varies... and then the discussion shifts to why can't it be even more flexible?


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C050353d8d80b4e272ab708d5f8b70361%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636688387286266248&sdata=33lN3yKin9et0lnbjMFDVEeSDSf3rmwDQu%2BsvheTeB8%3D&reserved=0>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb821b9e8a6db4408fff308d5f80c907d%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636687655210056012&sdata=SCzz05SABDQ0vxprDYfdKGOY1VES%2Fm0tIr2kRnGlY4c%3D&reserved=0>


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C050353d8d80b4e272ab708d5f8b70361%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636688387286266248&sdata=33lN3yKin9et0lnbjMFDVEeSDSf3rmwDQu%2BsvheTeB8%3D&reserved=0>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb821b9e8a6db4408fff308d5f80c907d%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636687655210056012&sdata=SCzz05SABDQ0vxprDYfdKGOY1VES%2Fm0tIr2kRnGlY4c%3D&reserved=0<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb821b9e8a6db4408fff308d5f80c907d%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636687655210056012&sdata=SCzz05SABDQ0vxprDYfdKGOY1VES%2Fm0tIr2kRnGlY4c%3D&reserved=0>

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C89b5017f862b465a9ee908d5f9069a29%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636688729119833832&sdata=uqa0CPRESDEdVBy0XsAgqq5zmBcHttAfOJe2NGCGgdk%3D&reserved=0>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C050353d8d80b4e272ab708d5f8b70361%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636688387286266248&sdata=d1rBsXZEn1BlkmvHGKHvkk2%2FWmXAppqS5SbOQF0ZCrY%3D&reserved=0<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C050353d8d80b4e272ab708d5f8b70361%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636688387286266248&sdata=d1rBsXZEn1BlkmvHGKHvkk2%2FWmXAppqS5SbOQF0ZCrY%3D&reserved=0>


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C89b5017f862b465a9ee908d5f9069a29%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636688729119843837&sdata=0vjRu2TsZ5%2Bf84Sb7%2BTEdi8%2BmLGGpbqq%2FXNg2zfJRiw%3D&reserved=0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20180803/db9a3b8b/attachment-0002.htm>


More information about the gpfsug-discuss mailing list