<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<br>
<div class="moz-cite-prefix">On 14/10/14 15:51, Sven Oehme wrote:<br>
</div>
<blockquote
cite="mid:OF8533131E.87E23F5C-ON88257D71.00514E0D-88257D71.00519716@us.ibm.com"
type="cite"><font face="sans-serif" size="2">it means there is
contention on inserting
data into the fast write log on the GSS Node, which could be
config or
workload related</font>
<br>
<font face="sans-serif" size="2">what GSS code version are you
running
</font></blockquote>
<blockquote>
<blockquote>
<blockquote><tt>[root@ebi5-251 ~]# mmdiag --version</tt><br>
<br>
<tt>=== mmdiag: version ===</tt><br>
<tt>Current GPFS build: "3.5.0-11 efix1 (888041)".</tt><br>
<tt>Built on Jul 9 2013 at 18:03:32</tt><br>
<tt>Running 6 days 2 hours 10 minutes 35 secs</tt><br>
</blockquote>
</blockquote>
</blockquote>
<br>
<br>
<br>
<blockquote
cite="mid:OF8533131E.87E23F5C-ON88257D71.00514E0D-88257D71.00519716@us.ibm.com"
type="cite"><font face="sans-serif" size="2">and how are the nodes
connected with each other (Ethernet or IB) ? </font>
<br>
</blockquote>
ethernet. they use the same bonding (4x10Gb/s) where the data is
passing. We don't have admin dedicated network<br>
<blockquote>
<blockquote>
<blockquote><br>
<tt>[root@gss03a ~]# mmlscluster </tt><br>
<br>
<tt>GPFS cluster information</tt><br>
<tt>========================</tt><br>
<tt> GPFS cluster name: GSS.ebi.ac.uk</tt><br>
<tt> GPFS cluster id: 17987981184946329605</tt><br>
<tt> GPFS UID domain: GSS.ebi.ac.uk</tt><br>
<tt> Remote shell command: /usr/bin/ssh</tt><br>
<tt> Remote file copy command: /usr/bin/scp</tt><br>
<br>
<tt>GPFS cluster configuration servers:</tt><br>
<tt>-----------------------------------</tt><br>
<tt> Primary server: gss01a.ebi.ac.uk</tt><br>
<tt> Secondary server: gss02b.ebi.ac.uk</tt><br>
<br>
<tt> Node Daemon node name IP address Admin node name
Designation</tt><br>
<tt>-----------------------------------------------------------------------</tt><br>
<tt> 1 gss01a.ebi.ac.uk 10.7.28.2 gss01a.ebi.ac.uk
quorum-manager</tt><br>
<tt> 2 gss01b.ebi.ac.uk 10.7.28.3 gss01b.ebi.ac.uk
quorum-manager</tt><br>
<tt> 3 gss02a.ebi.ac.uk 10.7.28.67 gss02a.ebi.ac.uk
quorum-manager</tt><br>
<tt> 4 gss02b.ebi.ac.uk 10.7.28.66 gss02b.ebi.ac.uk
quorum-manager</tt><br>
<tt> 5 gss03a.ebi.ac.uk 10.7.28.34 gss03a.ebi.ac.uk
quorum-manager</tt><br>
<tt> 6 gss03b.ebi.ac.uk 10.7.28.35 gss03b.ebi.ac.uk
quorum-manager</tt><br>
<br>
<br>
</blockquote>
</blockquote>
</blockquote>
<b>Note:</b> The 3 node "pairs" (gss01, gss02 and gss03) are in
different subnet because of datacenter constraints ( They are not
physically in the same row, and due to network constraints was not
possible to put them in the same subnet). The packets are routed,
but should not be a problem as there is 160Gb/s bandwidth between
them.<br>
<br>
Regards,<br>
Salvatore<br>
<br>
<br>
<br>
<blockquote
cite="mid:OF8533131E.87E23F5C-ON88257D71.00514E0D-88257D71.00519716@us.ibm.com"
type="cite"><font face="sans-serif" size="2">------------------------------------------<br>
Sven Oehme <br>
Scalable Storage Research <br>
email: <a class="moz-txt-link-abbreviated" href="mailto:oehmes@us.ibm.com">oehmes@us.ibm.com</a> <br>
Phone: +1 (408) 824-8904 <br>
IBM Almaden Research Lab <br>
------------------------------------------</font>
<br>
<br>
<br>
<br>
<font face="sans-serif" size="1" color="#5f5f5f">From:
</font><font face="sans-serif" size="1">Salvatore Di Nardo
<a class="moz-txt-link-rfc2396E" href="mailto:sdinardo@ebi.ac.uk"><sdinardo@ebi.ac.uk></a></font>
<br>
<font face="sans-serif" size="1" color="#5f5f5f">To:
</font><font face="sans-serif" size="1">gpfsug main discussion
list <a class="moz-txt-link-rfc2396E" href="mailto:gpfsug-discuss@gpfsug.org"><gpfsug-discuss@gpfsug.org></a></font>
<br>
<font face="sans-serif" size="1" color="#5f5f5f">Date:
</font><font face="sans-serif" size="1">10/14/2014 07:40 AM</font>
<br>
<font face="sans-serif" size="1" color="#5f5f5f">Subject:
</font><font face="sans-serif" size="1">[gpfsug-discuss]
wait for permission to append to log</font>
<br>
<font face="sans-serif" size="1" color="#5f5f5f">Sent by:
</font><font face="sans-serif" size="1"><a class="moz-txt-link-abbreviated" href="mailto:gpfsug-discuss-bounces@gpfsug.org">gpfsug-discuss-bounces@gpfsug.org</a></font>
<br>
<hr noshade="noshade">
<br>
<br>
<br>
<tt><font size="2">hello all,<br>
could someone explain me the meaning of those waiters?<br>
<br>
gss02b.ebi.ac.uk: 0x7F21EA8541B0 waiting 0.122786709 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA5F4EC0 waiting 0.122770807 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA9BD1A0 waiting 0.122115115 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA32FF30 waiting 0.121371877 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA6A1BA0 waiting 0.119322600 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA2E4330 waiting 0.118216774 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA72E930 waiting 0.117961594 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA6539C0 waiting 0.116133122 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA3D3490 waiting 0.116103642 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA85A060 waiting 0.115137978 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA4C84A0 waiting 0.115046631 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA229310 waiting 0.114498225 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA2AB630 waiting 0.113035120 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA83D9E0 waiting 0.112934666 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA736DC0 waiting 0.112834203 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA3A2C20 waiting 0.111498004 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA3B2250 waiting 0.111309423 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EAABDF10 waiting 0.110939219 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EAA27A00 waiting 0.110025022 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EAA8D6A0 waiting 0.109176110 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA2B3AC0 waiting 0.109025355 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA2080D0 waiting 0.108702893 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA3AC3A0 waiting 0.107691494 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EAB460E0 waiting 0.106003854 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA2093C0 waiting 0.105781682 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA6FBAE0 waiting 0.105696084 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA573E90 waiting 0.105182795 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA4191E0 waiting 0.104335963 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA82AAE0 waiting 0.104079258 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA538BB0 waiting 0.103798658 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EAA15DF0 waiting 0.102778144 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA57C320 waiting 0.100503136 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA802700 waiting 0.100499392 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EAA5F410 waiting 0.100489143 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA861200 waiting 0.100351636 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA54BAB0 waiting 0.099615942 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EAAAFBD0 waiting 0.099477387 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA657290 waiting 0.099123599 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA2BD240 waiting 0.099074074 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA205AF0 waiting 0.097532291 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA477CE0 waiting 0.097311417 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA2F9810 waiting 0.096209425 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA463AF0 waiting 0.096143868 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA8B2CB0 waiting 0.094143517 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA7D1E90 waiting 0.093156759 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EAB473D0 waiting 0.093154775 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EAB03C60 waiting 0.092952495 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
gss02b.ebi.ac.uk: 0x7F21EA8766E0 waiting 0.092908405 seconds,
<br>
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) <br>
(VdiskLogAppendCondvar), reason 'wait for permission to append
to log'<br>
<br>
Does it means that the vdisk logs are struggling?<br>
<br>
Regards,<br>
Salvatore<br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at gpfsug.org<br>
</font></tt><a moz-do-not-send="true"
href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font
size="2">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font
size="2"><br>
<br>
</font></tt>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
<a class="moz-txt-link-freetext" href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a>
</pre>
</blockquote>
<br>
</body>
</html>