<font size=2 face="sans-serif">your GSS code version is very backlevel.
</font>
<br>
<br><font size=2 face="sans-serif">can you please send me the output of
mmlsrecoverygroup RGNAME -L --pdisk </font>
<br><font size=2 face="sans-serif">as well as mmlsconfig and mmlsfs all</font>
<br>
<br><font size=2 face="sans-serif">thx. Sven</font>
<br>
<br><font size=2 face="sans-serif">------------------------------------------<br>
Sven Oehme <br>
Scalable Storage Research <br>
email: oehmes@us.ibm.com <br>
Phone: +1 (408) 824-8904 <br>
IBM Almaden Research Lab <br>
------------------------------------------</font>
<br>
<br>
<br>
<br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Salvatore Di Nardo
<sdinardo@ebi.ac.uk></font>
<br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug-discuss@gpfsug.org</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">10/14/2014 08:23 AM</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
wait for permission to append to log</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@gpfsug.org</font>
<br>
<hr noshade>
<br>
<br>
<br>
<br><font size=3>On 14/10/14 15:51, Sven Oehme wrote:</font>
<br><font size=2 face="sans-serif">it means there is contention on inserting
data into the fast write log on the GSS Node, which could be config or
workload related</font><font size=3> </font><font size=2 face="sans-serif"><br>
what GSS code version are you running </font>
<br><tt><font size=3>[root@ebi5-251 ~]# mmdiag --version</font></tt><font size=3><br>
</font><tt><font size=3><br>
=== mmdiag: version ===<br>
Current GPFS build: "3.5.0-11 efix1 (888041)".<br>
Built on Jul 9 2013 at 18:03:32<br>
Running 6 days 2 hours 10 minutes 35 secs</font></tt>
<br><font size=3><br>
<br>
</font>
<br><font size=2 face="sans-serif">and how are the nodes connected with
each other (Ethernet or IB) ? </font>
<br><font size=3>ethernet. they use the same bonding (4x10Gb/s) where the
data is passing. We don't have admin dedicated network</font>
<br><tt><font size=3><br>
[root@gss03a ~]# mmlscluster </font></tt><font size=3><br>
</font><tt><font size=3><br>
GPFS cluster information<br>
========================<br>
GPFS cluster name:
GSS.ebi.ac.uk<br>
GPFS cluster id:
17987981184946329605<br>
GPFS UID domain:
GSS.ebi.ac.uk<br>
Remote shell command: /usr/bin/ssh<br>
Remote file copy command: /usr/bin/scp</font></tt><font size=3><br>
</font><tt><font size=3><br>
GPFS cluster configuration servers:<br>
-----------------------------------<br>
Primary server: gss01a.ebi.ac.uk<br>
Secondary server: gss02b.ebi.ac.uk</font></tt><font size=3><br>
</font><tt><font size=3><br>
Node Daemon node name IP address Admin
node name Designation<br>
-----------------------------------------------------------------------<br>
1 gss01a.ebi.ac.uk 10.7.28.2
gss01a.ebi.ac.uk quorum-manager<br>
2 gss01b.ebi.ac.uk 10.7.28.3
gss01b.ebi.ac.uk quorum-manager<br>
3 gss02a.ebi.ac.uk 10.7.28.67
gss02a.ebi.ac.uk quorum-manager<br>
4 gss02b.ebi.ac.uk 10.7.28.66
gss02b.ebi.ac.uk quorum-manager<br>
5 gss03a.ebi.ac.uk 10.7.28.34
gss03a.ebi.ac.uk quorum-manager<br>
6 gss03b.ebi.ac.uk 10.7.28.35
gss03b.ebi.ac.uk quorum-manager</font></tt><font size=3><br>
<br>
</font>
<br><font size=3><b>Note:</b> The 3 node "pairs" (gss01, gss02
and gss03) are in different subnet because of datacenter constraints
( They are not physically in the same row, and due to network constraints
was not possible to put them in the same subnet). The packets are routed,
but should not be a problem as there is 160Gb/s bandwidth between them.<br>
<br>
Regards,<br>
Salvatore<br>
<br>
<br>
</font>
<br><font size=2 face="sans-serif">------------------------------------------<br>
Sven Oehme <br>
Scalable Storage Research <br>
email: </font><a href=mailto:oehmes@us.ibm.com><font size=2 color=blue face="sans-serif"><u>oehmes@us.ibm.com</u></font></a><font size=2 face="sans-serif">
<br>
Phone: +1 (408) 824-8904 <br>
IBM Almaden Research Lab <br>
------------------------------------------</font><font size=3> <br>
<br>
<br>
</font><font size=1 color=#5f5f5f face="sans-serif"><br>
From: </font><font size=1 face="sans-serif">Salvatore
Di Nardo </font><a href=mailto:sdinardo@ebi.ac.uk><font size=1 color=blue face="sans-serif"><u><sdinardo@ebi.ac.uk></u></font></a><font size=3>
</font><font size=1 color=#5f5f5f face="sans-serif"><br>
To: </font><font size=1 face="sans-serif">gpfsug
main discussion list </font><a href="mailto:gpfsug-discuss@gpfsug.org"><font size=1 color=blue face="sans-serif"><u><gpfsug-discuss@gpfsug.org></u></font></a><font size=3>
</font><font size=1 color=#5f5f5f face="sans-serif"><br>
Date: </font><font size=1 face="sans-serif">10/14/2014
07:40 AM</font><font size=3> </font><font size=1 color=#5f5f5f face="sans-serif"><br>
Subject: </font><font size=1 face="sans-serif">[gpfsug-discuss]
wait for permission to append to log</font><font size=3> </font><font size=1 color=#5f5f5f face="sans-serif"><br>
Sent by: </font><a href="mailto:gpfsug-discuss-bounces@gpfsug.org"><font size=1 color=blue face="sans-serif"><u>gpfsug-discuss-bounces@gpfsug.org</u></font></a><font size=3>
<br>
</font>
<hr noshade><font size=3><br>
<br>
</font><tt><font size=2><br>
hello all,<br>
could someone explain me the meaning of those waiters?<br>
<br>
gss02b.ebi.ac.uk: 0x7F21EA8541B0 waiting 0.122786709 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA5F4EC0 waiting 0.122770807 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA9BD1A0 waiting 0.122115115 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA32FF30 waiting 0.121371877 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA6A1BA0 waiting 0.119322600 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA2E4330 waiting 0.118216774 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA72E930 waiting 0.117961594 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA6539C0 waiting 0.116133122 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA3D3490 waiting 0.116103642 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA85A060 waiting 0.115137978 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA4C84A0 waiting 0.115046631 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA229310 waiting 0.114498225 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA2AB630 waiting 0.113035120 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA83D9E0 waiting 0.112934666 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA736DC0 waiting 0.112834203 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA3A2C20 waiting 0.111498004 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA3B2250 waiting 0.111309423 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EAABDF10 waiting 0.110939219 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EAA27A00 waiting 0.110025022 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EAA8D6A0 waiting 0.109176110 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA2B3AC0 waiting 0.109025355 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA2080D0 waiting 0.108702893 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA3AC3A0 waiting 0.107691494 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EAB460E0 waiting 0.106003854 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA2093C0 waiting 0.105781682 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA6FBAE0 waiting 0.105696084 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA573E90 waiting 0.105182795 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA4191E0 waiting 0.104335963 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA82AAE0 waiting 0.104079258 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA538BB0 waiting 0.103798658 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EAA15DF0 waiting 0.102778144 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA57C320 waiting 0.100503136 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA802700 waiting 0.100499392 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EAA5F410 waiting 0.100489143 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA861200 waiting 0.100351636 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA54BAB0 waiting 0.099615942 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EAAAFBD0 waiting 0.099477387 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA657290 waiting 0.099123599 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA2BD240 waiting 0.099074074 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA205AF0 waiting 0.097532291 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA477CE0 waiting 0.097311417 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA2F9810 waiting 0.096209425 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA463AF0 waiting 0.096143868 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA8B2CB0 waiting 0.094143517 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA7D1E90 waiting 0.093156759 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EAB473D0 waiting 0.093154775 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EAB03C60 waiting 0.092952495 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA8766E0 waiting 0.092908405 seconds, <br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'<br>
<br>
Does it means that the vdisk logs are struggling?<br>
<br>
Regards,<br>
Salvatore<br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at gpfsug.org</font></tt><font size=3 color=blue><u><br>
</u></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></tt></a><tt><font size=2><br>
</font></tt><font size=3><br>
<br>
<br>
</font>
<br><tt><font size=3>_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at gpfsug.org<br>
</font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></tt></a><tt><font size=3><br>
</font></tt>
<br><tt><font size=2>_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at gpfsug.org<br>
</font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br>
</font></tt>
<br>