[gpfsug-discuss] question on viewing block distribution across NSDs

Thu Mar 30 14:04:09 BST 2017

Hi,

Yes, you could use "mmdf" to obtain file-system "usage" across the NSDs 
(comprising the file-system).

If you want to obtain "data block distribution corresponding to a file 
across the NSDs", then there is a utility "mmgetlocation" in 
/usr/lpp/mmfs/samples/fpo that can be used to get file-data-blocks to NSD 
mapping. 

Example: 

# File-system comprises of single storage pool, all NSDs configured as 
dataAndMetadata, -m 1 -r 1, FS block-size=2MiB
# mmlsfs gpfs1b | grep 'Block size'
 -B                 2097152                  Block size

# The file-system is comprised of 10 x dataAndMetadata NSDs
# mmlsdisk gpfs1b | grep DMD | wc -l
10

# Create a sample file that is 40MiB (20 data blocks)
/mnt/sw/benchmarks/gpfsperf/gpfsperf create seq -r 2m -n 40m 
/mnt/gpfs1b/temp_dir/lf.s.1

# File size is 40 MiB
(09:52:49) c25m3n07:~ # ls -lh /mnt/gpfs1b/temp_dir/lf.s.1
-rw-r--r-- 1 root root 40M Mar 17 09:52 /mnt/gpfs1b/temp_dir/lf.s.1
(09:52:54) c25m3n07:~ # du -sh /mnt/gpfs1b/temp_dir/lf.s.1
40M     /mnt/gpfs1b/temp_dir/lf.s.1

# Verified through mmgetlocation that the file data blocks is uniformly 
striped across all the dataAndMetadata NSDs, with each NSD containing 2 
file data blocks
# In the output below, "DMD_NSDX" is name of the NSDs. 
(09:53:00) c25m3n07:~ # /usr/lpp/mmfs/samples/fpo/mmgetlocation -f 
/mnt/gpfs1b/temp_dir/lf.s.1

[FILE INFO]
------------------------------------------------------------------------

blockSize           2   MB
blockGroupFactor    1
metadataBlockSize   2M
writeAffinityDepth  0
flags:
data replication:     1 max 2
storage pool name:    system
metadata replication: 1 max 2

Chunk 0 (offset 0) is located at disks:  [ DMD_NSD09 
c25m3n07-ib,c25m3n08-ib ]
Chunk 1 (offset 2097152) is located at disks:  [ DMD_NSD10 
c25m3n08-ib,c25m3n07-ib ]
Chunk 2 (offset 4194304) is located at disks:  [ DMD_NSD01 
c25m3n07-ib,c25m3n08-ib ]
Chunk 3 (offset 6291456) is located at disks:  [ DMD_NSD02 
c25m3n08-ib,c25m3n07-ib ]
Chunk 4 (offset 8388608) is located at disks:  [ DMD_NSD03 
c25m3n07-ib,c25m3n08-ib ]
Chunk 5 (offset 10485760) is located at disks:  [ DMD_NSD04 
c25m3n08-ib,c25m3n07-ib ]
Chunk 6 (offset 12582912) is located at disks:  [ DMD_NSD05 
c25m3n07-ib,c25m3n08-ib ]
Chunk 7 (offset 14680064) is located at disks:  [ DMD_NSD06 
c25m3n08-ib,c25m3n07-ib ]
Chunk 8 (offset 16777216) is located at disks:  [ DMD_NSD07 
c25m3n07-ib,c25m3n08-ib ]
Chunk 9 (offset 18874368) is located at disks:  [ DMD_NSD08 
c25m3n08-ib,c25m3n07-ib ]
Chunk 10 (offset 20971520) is located at disks:  [ DMD_NSD09 
c25m3n07-ib,c25m3n08-ib ]
Chunk 11 (offset 23068672) is located at disks:  [ DMD_NSD10 
c25m3n08-ib,c25m3n07-ib ]
Chunk 12 (offset 25165824) is located at disks:  [ DMD_NSD01 
c25m3n07-ib,c25m3n08-ib ]
Chunk 13 (offset 27262976) is located at disks:  [ DMD_NSD02 
c25m3n08-ib,c25m3n07-ib ]
Chunk 14 (offset 29360128) is located at disks:  [ DMD_NSD03 
c25m3n07-ib,c25m3n08-ib ]
Chunk 15 (offset 31457280) is located at disks:  [ DMD_NSD04 
c25m3n08-ib,c25m3n07-ib ]
Chunk 16 (offset 33554432) is located at disks:  [ DMD_NSD05 
c25m3n07-ib,c25m3n08-ib ]
Chunk 17 (offset 35651584) is located at disks:  [ DMD_NSD06 
c25m3n08-ib,c25m3n07-ib ]
Chunk 18 (offset 37748736) is located at disks:  [ DMD_NSD07 
c25m3n07-ib,c25m3n08-ib ]
Chunk 19 (offset 39845888) is located at disks:  [ DMD_NSD08 
c25m3n08-ib,c25m3n07-ib ]

[SUMMARY INFO]
----------------------------------------------------------------------------------------------------------
Replica num  Nodename   TotalChunkst

Replica 1 :     c25m3n07-ib,c25m3n08-ib:  Total : 10
Replica 1 :     c25m3n08-ib,c25m3n07-ib:  Total : 10

Best Regards,
-Kums

From:   <Greg.Lehmann at csiro.au>
To:     <gpfsug-discuss at spectrumscale.org>
Date:   03/29/2017 08:00 PM
Subject:        Re: [gpfsug-discuss] question on viewing block 
distribution across NSDs
Sent by:        gpfsug-discuss-bounces at spectrumscale.org

I was going to keep mmdf in mind, not gpfs.snap. I will now also keep in 
mind that mmdf can have an impact as at present we have spinning disk for 
metadata. The system I am playing around on is not production yet, so I am 
safe for the moment.

Thanks again.

From: gpfsug-discuss-bounces at spectrumscale.org [
mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Knister, 
Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]
Sent: Thursday, 30 March 2017 9:55 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] question on viewing block distribution 
across NSDs

I don't necessarily think you need to run a snap prior, just the output of 
mmdf should be enough. Something to keep in mind that I should have said 
before-- an mmdf can be stressful on your system particularly if you have 
spinning disk for your metadata. We're fortunate enough to have all flash 
for our metadata and I tend to take it for granted some times :)

From: greg.lehmann at csiro.au
Sent: 3/29/17, 19:52
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] question on viewing block distribution 
across NSDs
Thanks. I don’t have a snap. I’ll keep that in mind for next time I do 
this.

From: mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of 
Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]
Sent: Thursday, 30 March 2017 9:45 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] question on viewing block distribution 
across NSDs

Hi Greg,

You could run an mmdf which will show you how full each NSD is. I'm not 
sure how to look back in time though to see the fs before the restripe. Do 
you perhaps have a gpfs.snap you took somewhat recently before the 
restripe? Maybe an internaldump in /tmp/mmfs somewhere?

From: greg.lehmann at csiro.au
Sent: 3/29/17, 19:21
To: gpfsug main discussion list
Subject: [gpfsug-discuss] question on viewing block distribution across 
NSDs
Hi All,
               I added some NSDs to an existing filesystem and ran 
mmrestripefs. I was sort of curious to see what the distribution looked 
like before and after the restripe. Is there any way of looking at it?

Cheers,

Greg._______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170330/78bcb73a/attachment-0002.htm>