<html><body><p>Hi Miroslav,<br><br>The mmdf output is very helpful. It suggests very strongly what the problem is:<br><br><tt>> ssd_5_5 80G 3 Yes No 22.31G ( 28%) 7.155G ( 9%) <br>> ssd_4_4 80G 3 Yes No 22.21G ( 28%) 7.196G ( 9%) <br>> ssd_3_3 80G 3 Yes No 22.2G ( 28%) 7.239G ( 9%) <br>> ssd_2_2 80G 3 Yes No 22.24G ( 28%) 7.146G ( 9%) <br>> ssd_1_1 80G 3 Yes No 22.29G ( 28%) 7.134G ( 9%) <br>>...</tt><br><tt>> ==================== ===================<br>> (data) 1.535P 698.9T ( 44%) 4.17T ( 0%)<br>> (metadata) 262.3T 92.96T ( 35%) 3.621T ( 1%)<br>>...</tt><br><tt>> Number of allocated inodes: 161947648<br>> Maximum number of inodes: 1342177280<br></tt><br><tt>You have 5 80G SSDs. That's not enough. Even with metadata spread across a couple dozen more SATA disks, SSDs are over 3/4 full. There's no way to accurately estimate the amount of metadata in this file system with the data at hand, but if we (very conservatively) assume that each SATA disk has only as much metadata as each SSD, i.e. ~57G, that would greatly exceed the amount of free space available on your SSDs. You need more free metadata space. Another way to look at this: you got 1.5PB of data under management. A reasonable rule-of-thumb estimate for the amount of metadata is 1-2% of the data (this is a typical ratio, but of course every file system is different, and large deviations are possible. A degenerate case is an fs containing nothing but directories, and in this case metadata usage is 100%). So you have to have at least a few TB of metadata storage. 5 80G SSDs aren't enough for an fs of this size.</tt><br><br><tt>> Could you please evaluate more on the performance overhead with <br>> having metadata<br>> on SSD+SATA? Are the read operations automatically directed to <br>> faster disks by GPFS?<br>> Is each write operation waiting for write to be finished by SATA disks?<br></tt><br><tt>Mixing disks with sharply different performance characteristics within a single storage pool is detrimental to performance. GPFS stripes blocks across all disks in a storage pool, expecting all of them to be equally suitable. If SSDs are mixed with SATA disks, the overall metadata write performance is going to be bottlenecked by SATA drives. On reads, given a choice of two replicas, GPFS V4.1.1+ picks the the replica residing on the fastest disk, but given that SSDs represent only a small fraction of your total metadata usage, this likely doesn't help a whole lot. You're on the right track in trying to shift all metadata to SSDs and away from SATA, the overall file system performance will improve as the result.</tt><br><br><tt>yuri</tt><br><br><tt>> <br>> Thank you,</tt><br><tt>> --<br>> Miroslav Bauer</tt><br><tt>> On 09/06/2016 09:06 PM, Yuri L Volobuev wrote:</tt><br><tt>> The correct way to accomplish what you're looking for (in <br>> particular, changing the fs-wide level of replication) is <br>> mmrestripefs -R. This command also takes care of moving data off <br>> disks now marked metadataOnly. <br>> <br>> The restripe job hits an error trying to move blocks of the inode <br>> file, i.e. before it gets to actual user data blocks. Note that at <br>> this point the metadata replication factor is still 2. This suggests<br>> one of two possibilities: (1) there isn't enough actual free space <br>> on the remaining metadataOnly disks, (2) there isn't enough space in<br>> some failure groups to allocate two replicas.<br>> <br>> All of this assumes you're operating within a single storage pool. <br>> If multiple storage pools are in play, there are other possibilities.<br>> <br>> 'mmdf' output would be helpful in providing more helpful advice. <br>> With the information at hand, I can only suggest trying to <br>> accomplish the task in two phases: (a) deallocated extra metadata <br>> replicas, by doing mmchfs -m 1 + mmrestripefs -R (b) move metadata <br>> off SATA disks. I do want to point out that metadata replication is <br>> a highly recommended insurance policy to have for your file system. <br>> As with other kinds of insurance, you may or may not need it, but if<br>> you do end up needing it, you'll be very glad you have it. The <br>> costs, in terms of extra metadata space and performance overhead, <br>> are very reasonable.<br>> <br>> yuri<br>> <br>> <br>> Miroslav Bauer ---09/01/2016 07:29:06 AM---Yes, failure group id is <br>> exactly what I meant :). Unfortunately, mmrestripefs with -R<br>> <br>> From: Miroslav Bauer <bauer@cesnet.cz><br>> To: gpfsug-discuss@spectrumscale.org, <br>> Date: 09/01/2016 07:29 AM<br>> Subject: Re: [gpfsug-discuss] Migration to separate metadata and data disks<br>> Sent by: gpfsug-discuss-bounces@spectrumscale.org</tt><br><tt>> <br>> <br>> <br>> Yes, failure group id is exactly what I meant :). Unfortunately, <br>> mmrestripefs with -R<br>> behaves the same as with -r. I also believed that mmrestripefs -R is the <br>> correct tool for<br>> fixing the replication settings on inodes (according to manpages), but I <br>> will try possible<br>> solutions you and Marc suggested and let you know how it went.<br>> <br>> Thank you,<br>> --<br>> Miroslav Bauer<br>> <br>> On 09/01/2016 04:02 PM, Aaron Knister wrote:<br>> > Oh! I think you've already provided the info I was looking for :) I <br>> > thought that failGroup=3 meant there were 3 failure groups within the <br>> > SSDs. I suspect that's not at all what you meant and that actually is <br>> > the failure group of all of those disks. That I think explains what's <br>> > going on-- there's only one failure group's worth of metadata-capable <br>> > disks available and as such GPFS can't place the 2nd replica for <br>> > existing files.<br>> ><br>> > Here's what I would suggest:<br>> ><br>> > - Create at least 2 failure groups within the SSDs<br>> > - Put the default metadata replication factor back to 2<br>> > - Run a restripefs -R to shuffle files around and restore the metadata <br>> > replication factor of 2 to any files created while it was set to 1<br>> ><br>> > If you're not interested in replication for metadata then perhaps all <br>> > you need to do is the mmrestripefs -R. I think that should <br>> > un-replicate the file from the SATA disks leaving the copy on the SSDs.<br>> ><br>> > Hope that helps.<br>> ><br>> > -Aaron<br>> ><br>> > On 9/1/16 9:39 AM, Aaron Knister wrote:<br>> >> By the way, I suspect the no space on device errors are because GPFS<br>> >> believes for some reason that it is unable to maintain the metadata<br>> >> replication factor of 2 that's likely set on all previously created <br>> >> inodes.<br>> >><br>> >> On 9/1/16 9:36 AM, Aaron Knister wrote:<br>> >>> I must admit, I'm curious as to the reason you're dropping the<br>> >>> replication factor from 2 down to 1. There are some serious advantages<br>> >>> we've seen to having multiple metadata replicas, as far as error<br>> >>> recovery is concerned.<br>> >>><br>> >>> Could you paste an output of mmlsdisk for the filesystem?<br>> >>><br>> >>> -Aaron<br>> >>><br>> >>> On 9/1/16 9:30 AM, Miroslav Bauer wrote:<br>> >>>> Hello,<br>> >>>><br>> >>>> I have a GPFS 3.5 filesystem (fs1) and I'm trying to migrate the<br>> >>>> filesystem metadata from state:<br>> >>>> -m = 2 (default metadata replicas)<br>> >>>> - SATA disks (dataAndMetadata, failGroup=1)<br>> >>>> - SSDs (metadataOnly, failGroup=3)<br>> >>>> to the desired state:<br>> >>>> -m = 1<br>> >>>> - SATA disks (dataOnly, failGroup=1)<br>> >>>> - SSDs (metadataOnly, failGroup=3)<br>> >>>><br>> >>>> I have done the following steps in the following order:<br>> >>>> 1) change SATA disks to dataOnly (stanza file modifies the 'usage'<br>> >>>> attribute only):<br>> >>>> # mmchdisk fs1 change -F dataOnly_disks.stanza<br>> >>>> Attention: Disk parameters were changed.<br>> >>>> Use the mmrestripefs command with the -r option to relocate data and<br>> >>>> metadata.<br>> >>>> Verifying file system configuration information ...<br>> >>>> mmchdisk: Propagating the cluster configuration data to all<br>> >>>> affected nodes. This is an asynchronous process.<br>> >>>><br>> >>>> 2) change default metadata replicas number 2->1<br>> >>>> # mmchfs fs1 -m 1<br>> >>>><br>> >>>> 3) run mmrestripefs as suggested by output of 1)<br>> >>>> # mmrestripefs fs1 -r<br>> >>>> Scanning file system metadata, phase 1 ...<br>> >>>> Error processing inodes.<br>> >>>> No space left on device<br>> >>>> mmrestripefs: Command failed. Examine previous error messages to<br>> >>>> determine cause.<br>> >>>><br>> >>>> It is, however, still possible to create new files on the filesystem.<br>> >>>> When I return one of the SATA disks as a dataAndMetadata disk, the<br>> >>>> mmrestripefs<br>> >>>> command stops complaining about No space left on device. Both df and<br>> >>>> mmdf<br>> >>>> say that there is enough space both for data (SATA) and metadata <br>> >>>> (SSDs).<br>> >>>> Does anyone have an idea why is it complaining?<br>> >>>><br>> >>>> Thanks,<br>> >>>><br>> >>>> -- <br>> >>>> Miroslav Bauer<br>> >>>><br>> >>>><br>> >>>><br>> >>>><br>> >>>> _______________________________________________<br>> >>>> gpfsug-discuss mailing list<br>> >>>> gpfsug-discuss at spectrumscale.org<br>> >>>> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>> >>>><br>> >>><br>> >><br>> ><br>> <br>> <br>> [attachment "smime.p7s" deleted by Yuri L Volobuev/Austin/IBM] <br>> _______________________________________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at spectrumscale.org<br>> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>> <br>> <br>> <br></tt><br><tt>> _______________________________________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at spectrumscale.org<br>> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br></tt><br><tt>> [attachment "smime.p7s" deleted by Yuri L Volobuev/Austin/IBM] <br>> _______________________________________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at spectrumscale.org<br>> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br></tt><BR>
</body></html>