<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";
mso-fareast-language:EN-US;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri","sans-serif";}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-GB link=blue vlink=purple><div class=WordSection1><p class=MsoPlainText>Hi Luke,<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>I’ve seen the same apparent grouping of nodes, I don’t believe the nodes are actually being grouped but instead the “Device Bond0:” and column headers are being re-printed to screen whenever there is a node that has the “init” status followed by a node that is “connected”. It is something I’ve noticed on many different versions of GPFS so I imagine it’s a “feature”.<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>I’ve not noticed anything but ‘0’ in the err column so I’m not sure if these correspond to error codes in the GPFS logs. If you run the command “mmfsadm dump tscomm”, you’ll see a bit more detail than the mmdiag –network shows. This suggests the sock column is number of sockets. I’ve seen the low numbers to for sent / recv using mmdiag --network, again the mmfsadm command above gives a better representation I’ve found.<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>All that being said, if you want to get in touch with us then we’ll happily open a PMR for you and find out the answer to any of your questions.<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>Kind regards,<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoNormal><b><span lang=EN-US style='color:black;mso-fareast-language:EN-GB'>Danny Metcalfe</span></b><b><span lang=EN-US style='font-size:12.0pt;color:black;mso-fareast-language:EN-GB'><o:p></o:p></span></b></p><p class=MsoNormal><span lang=EN-US style='color:black;mso-fareast-language:EN-GB'>Systems Engineer<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='color:black;mso-fareast-language:EN-GB'>OCF plc<o:p></o:p></span></p><table class=MsoNormalTable border=0 cellspacing=0 cellpadding=0 style='border-collapse:collapse'><tr><td width=182 valign=top style='width:136.5pt;padding:0cm 0cm 0cm 0cm'><p class=MsoNormal><b><span style='color:black;mso-fareast-language:EN-GB'><o:p> </o:p></span></b></p><p class=MsoNormal><b><span style='color:black;mso-fareast-language:EN-GB'>Tel:</span></b><span style='color:black;mso-fareast-language:EN-GB'> 0114 257 2200</span><span style='font-size:12.0pt;color:black;mso-fareast-language:EN-GB'><o:p></o:p></span></p></td><td width=40 valign=top style='width:30.3pt;padding:0cm 5.4pt 0cm 5.4pt'><p class=MsoNormal><span style='color:black;mso-fareast-language:EN-GB'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:black;mso-fareast-language:EN-GB'><img width=24 height=22 id="Picture_x0020_1" src="cid:image001.jpg@01CFCE04.575B8380" alt="cid:image001.jpg@01CC47B8.CE62AE70"></span><span style='font-size:12.0pt;color:black;mso-fareast-language:EN-GB'><o:p></o:p></span></p></td><td width=101 colspan=2 valign=top style='width:75.75pt;padding:0cm 5.4pt 0cm 5.4pt'><p class=MsoNormal><span style='color:black;mso-fareast-language:EN-GB'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:black;mso-fareast-language:EN-GB'><a href="http://twitter.com/ocfplc"><span style='color:black'>Twitter</span></a></span><span style='font-size:12.0pt;color:black;mso-fareast-language:EN-GB'><o:p></o:p></span></p></td></tr><tr><td width=182 valign=top style='width:136.5pt;padding:0cm 0cm 0cm 0cm'><p class=MsoNormal><b><span style='color:black;mso-fareast-language:EN-GB'>Fax:</span></b><span style='color:black;mso-fareast-language:EN-GB'> 0114 257 0022</span><span style='font-size:12.0pt;color:black;mso-fareast-language:EN-GB'><o:p></o:p></span></p></td><td width=40 valign=top style='width:30.3pt;padding:0cm 5.4pt 0cm 5.4pt'><p class=MsoNormal><span style='color:black;mso-fareast-language:EN-GB'><img border=0 width=24 height=22 id="Picture_x0020_2" src="cid:image002.jpg@01CFCE04.575B8380" alt="cid:image002.jpg@01CC47B8.CE62AE70"></span><span style='font-size:12.0pt;color:black;mso-fareast-language:EN-GB'><o:p></o:p></span></p></td><td width=101 colspan=2 valign=top style='width:75.75pt;padding:0cm 5.4pt 0cm 5.4pt'><p class=MsoNormal><span style='color:black;mso-fareast-language:EN-GB'><a href="http://blog.ocf.co.uk/"><span style='color:black'>Blog</span></a></span><span style='font-size:12.0pt;color:black;mso-fareast-language:EN-GB'><o:p></o:p></span></p></td></tr><tr><td width=182 valign=top style='width:136.5pt;padding:0cm 0cm 0cm 0cm'><p class=MsoNormal><b><span style='color:black;mso-fareast-language:EN-GB'>Mob:</span></b><span style='color:black;mso-fareast-language:EN-GB'> 07960 503404</span><span style='font-size:12.0pt;color:black;mso-fareast-language:EN-GB'><o:p></o:p></span></p></td><td width=40 valign=top style='width:30.3pt;padding:0cm 5.4pt 0cm 5.4pt'><p class=MsoNormal><span style='color:black;mso-fareast-language:EN-GB'><img border=0 width=24 height=22 id="Picture_x0020_3" src="cid:image003.jpg@01CFCE04.575B8380" alt="cid:image003.jpg@01CC47B8.CE62AE70"></span><span style='font-size:12.0pt;color:black;mso-fareast-language:EN-GB'><o:p></o:p></span></p></td><td width=101 colspan=2 valign=top style='width:75.75pt;padding:0cm 5.4pt 0cm 5.4pt'><p class=MsoNormal><span style='color:black;mso-fareast-language:EN-GB'><a href="http://www.ocf.co.uk/"><span style='color:black'>Web</span></a><o:p></o:p></span></p></td></tr><tr><td width=182 style='width:136.5pt;padding:0cm 0cm 0cm 0cm'></td><td width=40 style='width:30.3pt;padding:0cm 0cm 0cm 0cm'></td><td width=85 style='width:63.75pt;padding:0cm 0cm 0cm 0cm'></td><td width=16 style='width:12.0pt;padding:0cm 0cm 0cm 0cm'></td></tr></table><p class=MsoNormal><span style='font-size:10.0pt;color:black;mso-fareast-language:EN-GB'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;color:black;mso-fareast-language:EN-GB'>Please note, any emails relating to an OCF Support request must always be sent to <a href="mailto:support@ocf.co.uk"><span style='color:black'>support@ocf.co.uk</span></a> for a ticket number to be generated or existing support ticket to be updated. Should this not be done then OCF cannot be held responsible for requests not dealt with in a timely manner.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;color:black;mso-fareast-language:EN-GB'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;color:black;mso-fareast-language:EN-GB'>OCF plc is a company registered in England and Wales. Registered number 4132533. Registered office address: OCF plc, 5 Rotunda Business Centre, Thorncliffe Park, Chapeltown, Sheffield, S35 2PG<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;color:black;mso-fareast-language:EN-GB'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;color:black;mso-fareast-language:EN-GB'>This message is private and confidential. If you have received this message in error, please notify us immediately and remove it from your system.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;color:black;mso-fareast-language:EN-GB'><o:p> </o:p></span></p><p class=MsoPlainText><span lang=EN-US style='mso-fareast-language:EN-GB'>-----Original Message-----<br>From: gpfsug-discuss-bounces@gpfsug.org [mailto:gpfsug-discuss-bounces@gpfsug.org] On Behalf Of Luke Raimbach<br>Sent: 09 September 2014 11:24<br>To: gpfsug-discuss@gpfsug.org<br>Subject: [gpfsug-discuss] mmdiag output questions</span><o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>Hi All,<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>When tracing a problem recently (which turned out to be a NIC failure), mmdiag proved useful in tracing broken cluster connections. I have some questions about the output of mmdiag using the --network switch:<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>Occasionally I see nodes in the same cluster grouped, but in no readily identifiable way - for example, the following output has three headings "Device bon0:" with some nodes listed, but the nodes don't seem to share anything in common like status, err, ostype, etc.<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>Also, is anyone able to explain what might be seen under the err column? Do these correspond to GPFS error codes as one might see in mmfs.log.latest? What is the sock column displaying - the number of open sockets or the socket state? Lastly, the sent/recvd columns seem very low. Is there a rolling time window within which these statistics are kept in some internal mmfsd buffer?<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>Cheers.<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>=== mmdiag: network ===<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>Pending messages:<o:p></o:p></p><p class=MsoPlainText> (none)<o:p></o:p></p><p class=MsoPlainText>Inter-node communication configuration:<o:p></o:p></p><p class=MsoPlainText> tscTcpPort 1191<o:p></o:p></p><p class=MsoPlainText> my address 10.100.10.51/22 (eth0) <c0n8><o:p></o:p></p><p class=MsoPlainText> my addr list 10.200.21.1/16 (bond0)/cpdn.oerc.local 10.100.10.51/22 (eth0)<o:p></o:p></p><p class=MsoPlainText> my node number 9<o:p></o:p></p><p class=MsoPlainText>TCP Connections between nodes:<o:p></o:p></p><p class=MsoPlainText> Device bond0:<o:p></o:p></p><p class=MsoPlainText> hostname node destination status err sock sent(MB) recvd(MB) ostype<o:p></o:p></p><p class=MsoPlainText> gpfs01 <c0n0> 10.200.1.1 connected 0 32 110 110 Linux/L<o:p></o:p></p><p class=MsoPlainText> gpfs02 <c0n1> 10.200.2.1 connected 0 36 104 104 Linux/L<o:p></o:p></p><p class=MsoPlainText> linux <c0n2> 10.200.101.1 connected 0 37 0 0 Linux/L<o:p></o:p></p><p class=MsoPlainText> jupiter <c0n3> 10.200.102.1 connected 0 35 0 0 Windows/L<o:p></o:p></p><p class=MsoPlainText> cnfs0 <c0n4> 10.200.10.10 connected 0 39 0 0 Linux/L<o:p></o:p></p><p class=MsoPlainText> cnfs1 <c0n5> 10.200.10.11 init 0 -1 0 0 Linux/L<o:p></o:p></p><p class=MsoPlainText> Device bond0:<o:p></o:p></p><p class=MsoPlainText> hostname node destination status err sock sent(MB) recvd(MB) ostype<o:p></o:p></p><p class=MsoPlainText> cnfs2 <c0n6> 10.200.10.12 connected 0 33 5 5 Linux/L<o:p></o:p></p><p class=MsoPlainText> cnfs3 <c0n7> 10.200.10.13 init 0 -1 0 0 Linux/L<o:p></o:p></p><p class=MsoPlainText> cpdn-ppc02 <c0n9> 10.200.61.1 init 0 -1 0 0 Linux/L<o:p></o:p></p><p class=MsoPlainText> cpdn-ppc03 <c0n10> 10.200.62.1 init 0 -1 0 0 Linux/L<o:p></o:p></p><p class=MsoPlainText> Device bond0:<o:p></o:p></p><p class=MsoPlainText> hostname node destination status err sock sent(MB) recvd(MB) ostype<o:p></o:p></p><p class=MsoPlainText> cpdn-ppc01 <c0n11> 10.200.60.1 connected 0 38 0 0 Linux/L<o:p></o:p></p><p class=MsoPlainText>diag verbs: VERBS RDMA class not initialized<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>Conversely, the output of mmdiag --network on the file system manager node for the same cluster looks like this:<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>=== mmdiag: network ===<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>Pending messages:<o:p></o:p></p><p class=MsoPlainText> (none)<o:p></o:p></p><p class=MsoPlainText>Inter-node communication configuration:<o:p></o:p></p><p class=MsoPlainText> tscTcpPort 1191<o:p></o:p></p><p class=MsoPlainText> my address 10.100.10.21/22 (eth0) <c0n0><o:p></o:p></p><p class=MsoPlainText> my addr list 10.200.1.1/16 (bond0)/cpdn.oerc.local 10.100.10.21/22 (eth0)<o:p></o:p></p><p class=MsoPlainText> my node number 1<o:p></o:p></p><p class=MsoPlainText>TCP Connections between nodes:<o:p></o:p></p><p class=MsoPlainText> Device bond0:<o:p></o:p></p><p class=MsoPlainText> hostname node destination status err sock sent(MB) recvd(MB) ostype<o:p></o:p></p><p class=MsoPlainText> gpfs02 <c0n1> 10.200.2.1 connected 0 73 219 219 Linux/L<o:p></o:p></p><p class=MsoPlainText> linux <c0n2> 10.200.101.1 connected 0 49 180 181 Linux/L<o:p></o:p></p><p class=MsoPlainText> jupiter <c0n3> 10.200.102.1 connected 0 33 3 3 Windows/L<o:p></o:p></p><p class=MsoPlainText> cnfs0 <c0n4> 10.200.10.10 connected 0 61 3 3 Linux/L<o:p></o:p></p><p class=MsoPlainText> cnfs1 <c0n5> 10.200.10.11 connected 0 81 0 0 Linux/L<o:p></o:p></p><p class=MsoPlainText> cnfs2 <c0n6> 10.200.10.12 connected 0 64 23 23 Linux/L<o:p></o:p></p><p class=MsoPlainText> cnfs3 <c0n7> 10.200.10.13 connected 0 60 2 2 Linux/L<o:p></o:p></p><p class=MsoPlainText> tsm01 <c0n8> 10.200.21.1 connected 0 50 110 110 Linux/L<o:p></o:p></p><p class=MsoPlainText> cpdn-ppc02 <c0n9> 10.200.61.1 connected 0 63 0 0 Linux/L<o:p></o:p></p><p class=MsoPlainText> cpdn-ppc03 <c0n10> 10.200.62.1 connected 0 65 0 0 Linux/L<o:p></o:p></p><p class=MsoPlainText> cpdn-ppc01 <c0n11> 10.200.60.1 connected 0 62 94 94 Linux/L<o:p></o:p></p><p class=MsoPlainText>diag verbs: VERBS RDMA class not initialized<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>All neatly connected!<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>--<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>Luke Raimbach<o:p></o:p></p><p class=MsoPlainText>IT Manager<o:p></o:p></p><p class=MsoPlainText>Oxford e-Research Centre<o:p></o:p></p><p class=MsoPlainText>7 Keble Road,<o:p></o:p></p><p class=MsoPlainText>Oxford,<o:p></o:p></p><p class=MsoPlainText>OX1 3QG<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>+44(0)1865 610639<o:p></o:p></p><p class=MsoPlainText>_______________________________________________<o:p></o:p></p><p class=MsoPlainText>gpfsug-discuss mailing list<o:p></o:p></p><p class=MsoPlainText>gpfsug-discuss at gpfsug.org<o:p></o:p></p><p class=MsoPlainText><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><span style='color:windowtext;text-decoration:none'>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></a><o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>-----<o:p></o:p></p><p class=MsoPlainText>No virus found in this message.<o:p></o:p></p><p class=MsoPlainText>Checked by AVG - <a href="http://www.avg.com"><span style='color:windowtext;text-decoration:none'>www.avg.com</span></a><o:p></o:p></p><p class=MsoPlainText>Version: 2014.0.4765 / Virus Database: 4015/8158 - Release Date: 09/05/14<o:p></o:p></p></div></body></html>