<html><body><p><font size="2">I'm not aware this kind of defects, seems it should not. but lack of data, we don't know what happened. I suggest you can open a PMR for your issue. Thanks.</font><br><br><font size="2">Regards, The Spectrum Scale (GPFS) team<br><br>------------------------------------------------------------------------------------------------------------------<br>If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at <a href="https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479">https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479</a>. <br><br>If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. <br><br>The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team.</font><br><br><img width="16" height="16" src="cid:1__=0ABB0BDCDFB738528f9e8a93df938690918c0AB@" border="0" alt="Inactive hide details for "CAPIT, NICOLAS" ---06/27/2017 02:59:59 PM---Hello, When the node is locked up there is no waiters (""><font size="2" color="#424282">"CAPIT, NICOLAS" ---06/27/2017 02:59:59 PM---Hello, When the node is locked up there is no waiters ("mmdiad --waiters" or "mmfsadm dump waiters")</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">"CAPIT, NICOLAS" <ncapit@atos.net></font><br><font size="2" color="#5F5F5F">To: </font><font size="2">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">06/27/2017 02:59 PM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">Re: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads</font><br><font size="2" color="#5F5F5F">Sent by: </font><font size="2">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br><br><br><tt><font size="2">Hello,<br><br>When the node is locked up there is no waiters ("mmdiad --waiters" or "mmfsadm dump waiters").<br>In the GPFS log file "/var/mmfs/gen/mmfslog" there is nothing and nothing in the dmesg output or system log.<br>The "mmgetstate" command says that the node is "active".<br>The only thing is the freeze of the FS.<br><br>Best regards,<br>Nicolas Capit<br>________________________________________<br>De : gpfsug-discuss-bounces@spectrumscale.org [gpfsug-discuss-bounces@spectrumscale.org] de la part de Aaron Knister [aaron.s.knister@nasa.gov]<br>Envoyé : mardi 27 juin 2017 01:57<br>À : gpfsug-discuss@spectrumscale.org<br>Objet : Re: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads<br><br>That's a fascinating bug. When the node is locked up what does "mmdiag<br>--waiters" show from the node in question? I suspect there's more<br>low-level diagnostic data that's helpful for the gurus at IBM but I'm<br>just curious what the waiters look like.<br><br>-Aaron<br><br>On 6/26/17 3:49 AM, CAPIT, NICOLAS wrote:<br>> Hello,<br>><br>> I don't know if this behavior/bug was already reported on this ML, so in<br>> doubt.<br>><br>> Context:<br>><br>> - SpectrumScale 4.2.2-3<br>> - client node with 64 cores<br>> - OS: RHEL7.3<br>><br>> When a MPI job with 64 processes is launched on the node with 64 cores<br>> then the FS freezed (only the output log file of the MPI job is put on<br>> the GPFS; so it may be related to the 64 processes writing in a same<br>> file???).<br>><br>> strace -p 3105 # mmfsd pid stucked<br>> Process 3105 attached<br>> wait4(-1, # stucked at this point<br>><br>> strace ls /gpfs<br>> stat("/gpfs", {st_mode=S_IFDIR|0755, st_size=131072, ...}) = 0<br>> openat(AT_FDCWD, "/gpfs", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC<br>> # stucked at this point<br>><br>> I have no problem with the other nodes of 28 cores.<br>> The GPFS command mmgetstate is working and I am able to use mmshutdown<br>> to recover the node.<br>><br>><br>> If I put workerThreads=72 on the 64 core node then I am not able to<br>> reproduce the freeze and I get the right behavior.<br>><br>> Is this a known bug with a number of cores > workerThreads?<br>><br>> Best regards,<br>> --<br>> *Nicolas Capit*<br>><br>><br>> _______________________________________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at spectrumscale.org<br>> </font></tt><tt><font size="2"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><tt><font size="2"><br>><br><br>--<br>Aaron Knister<br>NASA Center for Climate Simulation (Code 606.2)<br>Goddard Space Flight Center<br>(301) 286-2776<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><tt><font size="2"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><tt><font size="2"><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><tt><font size="2"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><tt><font size="2"><br><br></font></tt><br><br><BR>
</body></html>