<html><body><p>this triggers a few questions<br><br>1. have you tried running it only on a node that doesn't serve NFS data ? <br>2. what NFS stack are you using ? is this the kernel NFS Server as part of linux means you use cNFS ? <br><br>if the answer to 2 is yes, have you adjusted the nfsd threads in /etc/sysconfig/nfs ? the default is only 8 and if you run with the default you have a very low number of threads from the outside competing with a larger number of threads doing restripe, increasing the nfsd threads could help. you could also reduce the number of internal restripe threads to try out if that helps mitigating the impact.<br><br>to try an extreme low value set the following : <br><br>mmchconfig pitWorkerThreadsPerNode=1 -i <br><br>and retry the restripe again, to reset it back to default run<br><br>mmchconfig pitWorkerThreadsPerNode=DEFAULT -i <br><br>sven<br> <br>------------------------------------------<br>Sven Oehme <br>Scalable Storage Research <br>email: oehmes@us.ibm.com <br>Phone: +1 (408) 824-8904 <br>IBM Almaden Research Lab <br>------------------------------------------<br><br><img width="16" height="16" src="cid:1__=07BBF4E4DFEF3CDD8f9e8a93df938690918c07B@" border="0" alt="Inactive hide details for Daniel Vogel ---07/02/2015 12:12:46 AM---Sven, Yes I agree, but “using –N” to reduce the load help"><font color="#424282">Daniel Vogel ---07/02/2015 12:12:46 AM---Sven, Yes I agree, but “using –N” to reduce the load helps not really. If I use NFS, for example, as</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">Daniel Vogel <Daniel.Vogel@abcsystems.ch></font><br><font size="2" color="#5F5F5F">To: </font><font size="2">"'gpfsug main discussion list'" <gpfsug-discuss@gpfsug.org></font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">07/02/2015 12:12 AM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">Re: [gpfsug-discuss] GPFS 4.1.1 without QoS for mmrestripefs?</font><br><font size="2" color="#5F5F5F">Sent by: </font><font size="2">gpfsug-discuss-bounces@gpfsug.org</font><br><hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br><br><br><font color="#1F497D" face="Calibri">Sven,</font><br><font color="#1F497D" face="Calibri"> </font><br><font color="#1F497D" face="Calibri">Yes I agree, but “using –N” to reduce the load helps not really. If I use NFS, for example, as a ESX data store, ESX I/O latency for NFS goes very high, the VM’s hangs. By the way I use SSD PCIe cards, perfect “mirror speed” but slow I/O on NFS.</font><br><font color="#1F497D" face="Calibri">The GPFS cluster concept I use are different than GSS or traditional FC (shared storage). I use shared nothing with IB (no FPO), many GPFS nodes with NSD’s. I know the need to resync the FS with mmchdisk / mmrestripe will happen more often. The only one feature will help is QoS for the GPFS admin jobs. I hope we are not fare away from this.</font><br><font color="#1F497D" face="Calibri"> </font><br><font color="#1F497D" face="Calibri">Thanks,</font><br><font color="#1F497D" face="Calibri">Daniel</font><br><font color="#1F497D" face="Calibri"> </font><br><font color="#1F497D" face="Calibri"> </font><br><b><font face="Tahoma">Von:</font></b><font face="Tahoma"> gpfsug-discuss-bounces@gpfsug.org [</font><font face="Tahoma"><a href="mailto:gpfsug-discuss-bounces@gpfsug.org">mailto:gpfsug-discuss-bounces@gpfsug.org</a></font><font face="Tahoma">] </font><b><font face="Tahoma">Im Auftrag von </font></b><font face="Tahoma">Sven Oehme</font><b><font face="Tahoma"><br>Gesendet:</font></b><font face="Tahoma"> Mittwoch, 1. Juli 2015 16:21</font><b><font face="Tahoma"><br>An:</font></b><font face="Tahoma"> gpfsug main discussion list</font><b><font face="Tahoma"><br>Betreff:</font></b><font face="Tahoma"> Re: [gpfsug-discuss] GPFS 4.1.1 without QoS for mmrestripefs?</font><br><font size="4" face="Times New Roman"> </font><p><font size="4" face="Times New Roman">Daniel,<br><br>as you know, we can't discuss future / confidential items on a mailing list. <br>what i presented as an outlook to future releases hasn't changed from a technical standpoint, we just can't share a release date until we announce it official.<br>there are multiple ways today to limit the impact on restripe and other tasks, the best way to do this is to run the task ( using -N) on a node (or very small number of nodes) that has no performance critical role. while this is not perfect, it should limit the impact significantly. .<br><br>sven<br><br>------------------------------------------<br>Sven Oehme <br>Scalable Storage Research <br>email: </font><a href="mailto:oehmes@us.ibm.com"><u><font size="4" color="#0000FF" face="Times New Roman">oehmes@us.ibm.com</font></u></a><font size="4" face="Times New Roman"> <br>Phone: +1 (408) 824-8904 <br>IBM Almaden Research Lab <br>------------------------------------------<br><br></font><img src="cid:1__=07BBF4E4DFEF3CDD8f9e8a93df938690918c07B@" width="16" height="16" alt="Inactive hide details for Daniel Vogel ---07/01/2015 03:29:11 AM---Hi Years ago, IBM made some plan to do a implementation "QoS"><font size="4" color="#424282" face="Times New Roman">Daniel Vogel ---07/01/2015 03:29:11 AM---Hi Years ago, IBM made some plan to do a implementation "QoS for mmrestripefs, mmdeldisk...". If a "</font><font size="4" face="Times New Roman"><br></font><font color="#5F5F5F" face="Times New Roman"><br>From: </font><font face="Times New Roman">Daniel Vogel <</font><a href="mailto:Daniel.Vogel@abcsystems.ch"><u><font color="#0000FF" face="Times New Roman">Daniel.Vogel@abcsystems.ch</font></u></a><font face="Times New Roman">></font><font color="#5F5F5F" face="Times New Roman"><br>To: </font><font face="Times New Roman">"'gpfsug-discuss@gpfsug.org'" <</font><a href="mailto:gpfsug-discuss@gpfsug.org"><u><font color="#0000FF" face="Times New Roman">gpfsug-discuss@gpfsug.org</font></u></a><font face="Times New Roman">></font><font color="#5F5F5F" face="Times New Roman"><br>Date: </font><font face="Times New Roman">07/01/2015 03:29 AM</font><font color="#5F5F5F" face="Times New Roman"><br>Subject: </font><font face="Times New Roman">[gpfsug-discuss] GPFS 4.1.1 without QoS for mmrestripefs?</font><font color="#5F5F5F" face="Times New Roman"><br>Sent by: </font><a href="mailto:gpfsug-discuss-bounces@gpfsug.org"><u><font color="#0000FF" face="Times New Roman">gpfsug-discuss-bounces@gpfsug.org</font></u></a><br><hr width="100%" size="2" align="left" noshade><br><font size="4" face="Times New Roman"><br><br></font><font size="4" face="Calibri"><br>Hi</font><font size="4" face="Times New Roman"><br></font><font size="4" face="Calibri"><br>Years ago, IBM made some plan to do a implementation “QoS for mmrestripefs, mmdeldisk…”. If a “mmfsrestripe” is running, very poor performance for NFS access.<br>I opened a PMR to ask for QoS in version 4.1.1 (Spectrum Scale).</font><font size="4" face="Times New Roman"><br></font><font size="4" face="Calibri"><br>PMR 61309,113,848:</font><i><font size="4" face="Arial"><br>I discussed the question of QOS with the development team. These <br>command changes that were noticed are not meant to be used as GA code<br>which is why they are not documented. I cannot provide any further <br>information from the support perspective. </font></i><font size="4" face="Times New Roman"><br><br></font><font size="4" face="Calibri"><br>Anybody knows about QoS? The last hope was at “GPFS Workshop Stuttgart März 2015” with Sven Oehme as speaker.</font><font size="4" face="Times New Roman"><br></font><font size="4" face="Calibri"><br>Daniel Vogel<br>IT Consultant</font><font size="4" face="Times New Roman"><br></font><b><font size="4" face="Calibri"><br>ABC SYSTEMS AG</font></b><font size="4" face="Calibri"><br>Hauptsitz Zürich<br>Rütistrasse 28<br>CH - 8952 Schlieren<br>T +41 43 433 6 433<br>D +41 43 433 6 467</font><u><font size="4" color="#0000FF" face="Calibri"><br></font></u><a href="http://www.abcsystems.ch/"><u><font size="4" color="#0000FF" face="Calibri">http://www.abcsystems.ch</font></u></a><font size="4" face="Times New Roman"><br></font><b><i><font size="4" color="#1F497D" face="Calibri"><br>ABC - A</font></i></b><i><font size="4" color="#1F497D" face="Calibri">lways </font></i><b><i><font size="4" color="#1F497D" face="Calibri">B</font></i></b><i><font size="4" color="#1F497D" face="Calibri">etter </font></i><b><i><font size="4" color="#1F497D" face="Calibri">C</font></i></b><i><font size="4" color="#1F497D" face="Calibri">oncepts. </font></i><b><i><font size="4" color="#1F497D" face="Calibri">A</font></i></b><i><font size="4" color="#1F497D" face="Calibri">pproved </font></i><b><i><font size="4" color="#1F497D" face="Calibri">B</font></i></b><i><font size="4" color="#1F497D" face="Calibri">y </font></i><b><i><font size="4" color="#1F497D" face="Calibri">C</font></i></b><i><font size="4" color="#1F497D" face="Calibri">ustomers since 1981.</font></i><font face="Courier New"><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at gpfsug.org</font><u><font color="#0000FF" face="Courier New"><br></font></u><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><u><font color="#0000FF" face="Courier New">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></u></a><font size="4" face="Times New Roman"><br><br></font><tt>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at gpfsug.org<br></tt><tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></tt><tt><br></tt><br><br><BR>
</body></html>