<html><body><p>The correct way to accomplish what you're looking for (in particular, changing the fs-wide level of replication) is mmrestripefs -R. This command also takes care of moving data off disks now marked metadataOnly. <br><br>The restripe job hits an error trying to move blocks of the inode file, i.e. before it gets to actual user data blocks. Note that at this point the metadata replication factor is still 2. This suggests one of two possibilities: (1) there isn't enough actual free space on the remaining metadataOnly disks, (2) there isn't enough space in some failure groups to allocate two replicas.<br><br>All of this assumes you're operating within a single storage pool. If multiple storage pools are in play, there are other possibilities.<br><br>'mmdf' output would be helpful in providing more helpful advice. With the information at hand, I can only suggest trying to accomplish the task in two phases: (a) deallocated extra metadata replicas, by doing mmchfs -m 1 + mmrestripefs -R (b) move metadata off SATA disks. I do want to point out that metadata replication is a highly recommended insurance policy to have for your file system. As with other kinds of insurance, you may or may not need it, but if you do end up needing it, you'll be very glad you have it. The costs, in terms of extra metadata space and performance overhead, are very reasonable.<br><br>yuri<br><br><br><img width="16" height="16" src="cid:1__=07BB0AB5DFF450FE8f9e8a93df938690918c07B@" border="0" alt="Inactive hide details for Miroslav Bauer ---09/01/2016 07:29:06 AM---Yes, failure group id is exactly what I meant :). Unfortun"><font color="#424282">Miroslav Bauer ---09/01/2016 07:29:06 AM---Yes, failure group id is exactly what I meant :). Unfortunately, mmrestripefs with -R</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">Miroslav Bauer <bauer@cesnet.cz></font><br><font size="2" color="#5F5F5F">To: </font><font size="2">gpfsug-discuss@spectrumscale.org, </font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">09/01/2016 07:29 AM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">Re: [gpfsug-discuss] Migration to separate metadata and data disks</font><br><font size="2" color="#5F5F5F">Sent by: </font><font size="2">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br><br><br><tt>Yes, failure group id is exactly what I meant :). Unfortunately, <br>mmrestripefs with -R<br>behaves the same as with -r. I also believed that mmrestripefs -R is the <br>correct tool for<br>fixing the replication settings on inodes (according to manpages), but I <br>will try possible<br>solutions you and Marc suggested and let you know how it went.<br><br>Thank you,<br>--<br>Miroslav Bauer<br><br>On 09/01/2016 04:02 PM, Aaron Knister wrote:<br>> Oh! I think you've already provided the info I was looking for :) I <br>> thought that failGroup=3 meant there were 3 failure groups within the <br>> SSDs. I suspect that's not at all what you meant and that actually is <br>> the failure group of all of those disks. That I think explains what's <br>> going on-- there's only one failure group's worth of metadata-capable <br>> disks available and as such GPFS can't place the 2nd replica for <br>> existing files.<br>><br>> Here's what I would suggest:<br>><br>> - Create at least 2 failure groups within the SSDs<br>> - Put the default metadata replication factor back to 2<br>> - Run a restripefs -R to shuffle files around and restore the metadata <br>> replication factor of 2 to any files created while it was set to 1<br>><br>> If you're not interested in replication for metadata then perhaps all <br>> you need to do is the mmrestripefs -R. I think that should <br>> un-replicate the file from the SATA disks leaving the copy on the SSDs.<br>><br>> Hope that helps.<br>><br>> -Aaron<br>><br>> On 9/1/16 9:39 AM, Aaron Knister wrote:<br>>> By the way, I suspect the no space on device errors are because GPFS<br>>> believes for some reason that it is unable to maintain the metadata<br>>> replication factor of 2 that's likely set on all previously created <br>>> inodes.<br>>><br>>> On 9/1/16 9:36 AM, Aaron Knister wrote:<br>>>> I must admit, I'm curious as to the reason you're dropping the<br>>>> replication factor from 2 down to 1. There are some serious advantages<br>>>> we've seen to having multiple metadata replicas, as far as error<br>>>> recovery is concerned.<br>>>><br>>>> Could you paste an output of mmlsdisk for the filesystem?<br>>>><br>>>> -Aaron<br>>>><br>>>> On 9/1/16 9:30 AM, Miroslav Bauer wrote:<br>>>>> Hello,<br>>>>><br>>>>> I have a GPFS 3.5 filesystem (fs1) and I'm trying to migrate the<br>>>>> filesystem metadata from state:<br>>>>> -m = 2 (default metadata replicas)<br>>>>> - SATA disks (dataAndMetadata, failGroup=1)<br>>>>> - SSDs (metadataOnly, failGroup=3)<br>>>>> to the desired state:<br>>>>> -m = 1<br>>>>> - SATA disks (dataOnly, failGroup=1)<br>>>>> - SSDs (metadataOnly, failGroup=3)<br>>>>><br>>>>> I have done the following steps in the following order:<br>>>>> 1) change SATA disks to dataOnly (stanza file modifies the 'usage'<br>>>>> attribute only):<br>>>>> # mmchdisk fs1 change -F dataOnly_disks.stanza<br>>>>> Attention: Disk parameters were changed.<br>>>>> Use the mmrestripefs command with the -r option to relocate data and<br>>>>> metadata.<br>>>>> Verifying file system configuration information ...<br>>>>> mmchdisk: Propagating the cluster configuration data to all<br>>>>> affected nodes. This is an asynchronous process.<br>>>>><br>>>>> 2) change default metadata replicas number 2->1<br>>>>> # mmchfs fs1 -m 1<br>>>>><br>>>>> 3) run mmrestripefs as suggested by output of 1)<br>>>>> # mmrestripefs fs1 -r<br>>>>> Scanning file system metadata, phase 1 ...<br>>>>> Error processing inodes.<br>>>>> No space left on device<br>>>>> mmrestripefs: Command failed. Examine previous error messages to<br>>>>> determine cause.<br>>>>><br>>>>> It is, however, still possible to create new files on the filesystem.<br>>>>> When I return one of the SATA disks as a dataAndMetadata disk, the<br>>>>> mmrestripefs<br>>>>> command stops complaining about No space left on device. Both df and<br>>>>> mmdf<br>>>>> say that there is enough space both for data (SATA) and metadata <br>>>>> (SSDs).<br>>>>> Does anyone have an idea why is it complaining?<br>>>>><br>>>>> Thanks,<br>>>>><br>>>>> -- <br>>>>> Miroslav Bauer<br>>>>><br>>>>><br>>>>><br>>>>><br>>>>> _______________________________________________<br>>>>> gpfsug-discuss mailing list<br>>>>> gpfsug-discuss at spectrumscale.org<br>>>>> </tt><tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></tt><tt><br>>>>><br>>>><br>>><br>><br><br><br>[attachment "smime.p7s" deleted by Yuri L Volobuev/Austin/IBM] _______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></tt><tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></tt><tt><br></tt><br><BR>
</body></html>