And there you have:<br><br>[ems1-fdr,compute,gss_ppc64]<br>verbsRdmaSend yes<br><br>Try turning this off.<br><br><br> -jf<br><div class="gmail_quote"><div dir="ltr">ons. 11. jan. 2017 kl. 18.54 skrev Damir Krstic <<a href="mailto:damir.krstic@gmail.com">damir.krstic@gmail.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg">Thanks for all the suggestions. Here is our mmlsconfig file. We just purchased another GL6. During the installation of the new GL6 IBM will upgrade our existing GL6 up to the latest code levels. This will happen during the week of 23rd of Jan. <div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">I am skeptical that the upgrade is going to fix the issue. </div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">On our IO servers we are running in connected mode (please note that IB interfaces are bonded)</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[root@gssio1 ~]# cat /sys/class/net/ib0/mode</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">connected</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[root@gssio1 ~]# cat /sys/class/net/ib1/mode</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">connected</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[root@gssio1 ~]# cat /sys/class/net/ib2/mode</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">connected</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[root@gssio1 ~]# cat /sys/class/net/ib3/mode</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">connected</span></p><div class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg"><br class="gmail_msg"></span></div></div><div class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg"><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[root@gssio2 ~]# cat /sys/class/net/ib0/mode</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">connected</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[root@gssio2 ~]# cat /sys/class/net/ib1/mode</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">connected</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[root@gssio2 ~]# cat /sys/class/net/ib2/mode</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">connected</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[root@gssio2 ~]# cat /sys/class/net/ib3/mode</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">connected</span></p><div class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg"><br class="gmail_msg"></span></div><div class="gmail_msg">Our login nodes are also running connected mode as well. </div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">However, all of our compute nodes are running in datagram:</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[root@mgt ~]# psh compute cat /sys/class/net/ib0/mode</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">qnode0758: datagram</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">qnode0763: datagram</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">qnode0760: datagram</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">qnode0772: datagram</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">qnode0773: datagram</span></p><div class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">....etc.</span></div></div><div class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg"><br class="gmail_msg"></span></div><div class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">Here is our mmlsconfig:</span></div></span></div><div class="gmail_msg"><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[root@gssio1 ~]# mmlsconfig</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">Configuration data for cluster <a href="http://ess-qstorage.it.northwestern.edu" class="gmail_msg" target="_blank">ess-qstorage.it.northwestern.edu</a>:</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">----------------------------------------------------------------</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">clusterName <a href="http://ess-qstorage.it.northwestern.edu" class="gmail_msg" target="_blank">ess-qstorage.it.northwestern.edu</a></span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">clusterId 17746506346828356609</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">dmapiFileHandleSize 32</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">minReleaseLevel 4.2.0.1</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">ccrEnabled yes</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">cipherList AUTHONLY</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[gss_ppc64]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">nsdRAIDBufferPoolSizePct 80</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">maxBufferDescs 2m</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">prefetchPct 5</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">nsdRAIDTracks 128k</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">nsdRAIDSmallBufferSize 256k</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">nsdMaxWorkerThreads 3k</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">nsdMinWorkerThreads 3k</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">nsdRAIDSmallThreadRatio 2</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">nsdRAIDThreadsPerQueue 16</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">nsdRAIDEventLogToConsole all</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">nsdRAIDFastWriteFSDataLimit 256k</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">nsdRAIDFastWriteFSMetadataLimit 1M</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">nsdRAIDReconstructAggressiveness 1</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">nsdRAIDFlusherBuffersLowWatermarkPct 20</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">nsdRAIDFlusherBuffersLimitPct 80</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">nsdRAIDFlusherTracksLowWatermarkPct 20</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">nsdRAIDFlusherTracksLimitPct 80</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">nsdRAIDFlusherFWLogHighWatermarkMB 1000</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">nsdRAIDFlusherFWLogLimitMB 5000</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">nsdRAIDFlusherThreadsLowWatermark 1</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">nsdRAIDFlusherThreadsHighWatermark 512</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">nsdRAIDBlockDeviceMaxSectorsKB 8192</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">nsdRAIDBlockDeviceNrRequests 32</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">nsdRAIDBlockDeviceQueueDepth 16</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">nsdRAIDBlockDeviceScheduler deadline</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">nsdRAIDMaxTransientStale2FT 1</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">nsdRAIDMaxTransientStale3FT 1</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">nsdMultiQueue 512</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">syncWorkerThreads 256</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">nsdInlineWriteMax 32k</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">maxGeneralThreads 1280</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">maxReceiverThreads 128</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">nspdQueues 64</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[common]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">maxblocksize 16m</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[ems1-fdr,compute,gss_ppc64]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">numaMemoryInterleave yes</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[gss_ppc64]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">maxFilesToCache 12k</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[ems1-fdr,compute]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">maxFilesToCache 128k</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[ems1-fdr,compute,gss_ppc64]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">flushedDataTarget 1024</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">flushedInodeTarget 1024</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">maxFileCleaners 1024</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">maxBufferCleaners 1024</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">logBufferCount 20</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">logWrapAmountPct 2</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">logWrapThreads 128</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">maxAllocRegionsPerNode 32</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">maxBackgroundDeletionThreads 16</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">maxInodeDeallocPrefetch 128</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[gss_ppc64]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">maxMBpS 16000</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[ems1-fdr,compute]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">maxMBpS 10000</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[ems1-fdr,compute,gss_ppc64]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">worker1Threads 1024</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">worker3Threads 32</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[gss_ppc64]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">ioHistorySize 64k</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[ems1-fdr,compute]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">ioHistorySize 4k</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[gss_ppc64]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">verbsRdmaMinBytes 16k</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[ems1-fdr,compute]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">verbsRdmaMinBytes 32k</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[ems1-fdr,compute,gss_ppc64]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">verbsRdmaSend yes</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[gss_ppc64]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">verbsRdmasPerConnection 16</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[ems1-fdr,compute]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">verbsRdmasPerConnection 256</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[gss_ppc64]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">verbsRdmasPerNode 3200</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[ems1-fdr,compute]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">verbsRdmasPerNode 1024</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[ems1-fdr,compute,gss_ppc64]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">verbsSendBufferMemoryMB 1024</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">verbsRdmasPerNodeOptimize yes</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">verbsRdmaUseMultiCqThreads yes</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[ems1-fdr,compute]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">ignorePrefetchLUNCount yes</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[gss_ppc64]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">scatterBufferSize 256K</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[ems1-fdr,compute]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">scatterBufferSize 256k</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">syncIntervalStrict yes</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[ems1-fdr,compute,gss_ppc64]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">nsdClientCksumTypeLocal ck64</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">nsdClientCksumTypeRemote ck64</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[gss_ppc64]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">pagepool 72856M</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[ems1-fdr]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">pagepool 17544M</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[compute]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">pagepool 4g</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[ems1-fdr,qsched03-ib0,quser10-fdr,compute,gss_ppc64]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">verbsRdma enable</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[gss_ppc64]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">verbsPorts mlx5_0/1 mlx5_0/2 mlx5_1/1 mlx5_1/2</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[ems1-fdr]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">verbsPorts mlx5_0/1 mlx5_0/2</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[qsched03-ib0,quser10-fdr,compute]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">verbsPorts mlx4_0/1</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[common]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">autoload no</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[ems1-fdr,compute,gss_ppc64]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">maxStatCache 0</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">[common]</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">envVar MLX4_USE_MUTEX=1 MLX5_SHUT_UP_BF=1 MLX5_USE_MUTEX=1</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">deadlockOverloadThreshold 0</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">deadlockDetectionThreshold 0</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">adminMode central</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;min-height:14px" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg"></span><br class="gmail_msg"></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">File systems in cluster <a href="http://ess-qstorage.it.northwestern.edu" class="gmail_msg" target="_blank">ess-qstorage.it.northwestern.edu</a>:</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">---------------------------------------------------------</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">/dev/home</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">/dev/hpc</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">/dev/projects</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo" class="gmail_msg"><span style="font-variant-ligatures:no-common-ligatures" class="gmail_msg">/dev/tthome</span></p></div></div></div><br class="gmail_msg"><div class="gmail_quote gmail_msg"><div dir="ltr" class="gmail_msg">On Wed, Jan 11, 2017 at 9:16 AM Luis Bolinches <<a href="mailto:luis.bolinches@fi.ibm.com" class="gmail_msg" target="_blank">luis.bolinches@fi.ibm.com</a>> wrote:<br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="m_-1181131753993210372m_4020387606308923108socmaildefaultfont gmail_msg" dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt"><div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt" class="gmail_msg"><div dir="ltr" class="gmail_msg">In addition to what Olaf has said</div>
<div dir="ltr" class="gmail_msg"> </div>
<div dir="ltr" class="gmail_msg">ESS upgrades include mellanox modules upgrades in the ESS nodes. In fact, on those noes you should do not update those solo (unless support says so in your PMR), so if that's been the recommendation, I suggest you look at it.</div>
<div dir="ltr" class="gmail_msg"> </div>
<div dir="ltr" class="gmail_msg">Changelog on ESS 4.0.4 (no idea what ESS level you are running)</div>
<div dir="ltr" class="gmail_msg"> </div>
<div dir="ltr" class="gmail_msg"><pre class="gmail_msg"> c) Support of MLNX_OFED_LINUX-3.2-2.0.0.1
- Updated from MLNX_OFED_LINUX-3.1-1.0.6.1 (ESS 4.0, 4.0.1, 4.0.2)
- Updated from MLNX_OFED_LINUX-3.1-1.0.0.2 (ESS 3.5.x)
- Updated from MLNX_OFED_LINUX-2.4-1.0.2 (ESS 3.0.x)
- Support for PCIe3 LP 2-port 100 Gb EDR InfiniBand adapter x16 (FC EC3E)
- Requires System FW level FW840.20 (SV840_104)
- No changes from ESS 4.0.3</pre></div>
<div class="gmail_msg"><br class="gmail_msg">--<br class="gmail_msg">Ystävällisin terveisin / Kind regards / Saludos cordiales / Salutations<br class="gmail_msg"><br class="gmail_msg">Luis Bolinches<br class="gmail_msg">Lab Services<br class="gmail_msg"><a href="http://www-03.ibm.com/systems/services/labservices/" class="gmail_msg" target="_blank">http://www-03.ibm.com/systems/services/labservices/</a><br class="gmail_msg"><br class="gmail_msg">IBM Laajalahdentie 23 (main Entrance) Helsinki, 00330 Finland<br class="gmail_msg">Phone: <a href="tel:+358%2050%203112585" value="+358503112585" class="gmail_msg" target="_blank">+358 503112585</a><br class="gmail_msg"><br class="gmail_msg">"If you continually give you will continually have." Anonymous</div>
<div dir="ltr" class="gmail_msg"> </div>
<div dir="ltr" class="gmail_msg"> </div>
<blockquote dir="ltr" style="border-left:solid #aaaaaa 2px;margin-left:5px;padding-left:5px;direction:ltr;margin-right:0px" class="gmail_msg"></blockquote></div></div><div class="m_-1181131753993210372m_4020387606308923108socmaildefaultfont gmail_msg" dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt"><div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt" class="gmail_msg"><blockquote dir="ltr" style="border-left:solid #aaaaaa 2px;margin-left:5px;padding-left:5px;direction:ltr;margin-right:0px" class="gmail_msg">----- Original message -----<br class="gmail_msg">From: "Olaf Weiser" <<a href="mailto:olaf.weiser@de.ibm.com" class="gmail_msg" target="_blank">olaf.weiser@de.ibm.com</a>><br class="gmail_msg">Sent by: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class="gmail_msg" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a><br class="gmail_msg">To: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" class="gmail_msg" target="_blank">gpfsug-discuss@spectrumscale.org</a>><br class="gmail_msg"></blockquote></div></div><div class="m_-1181131753993210372m_4020387606308923108socmaildefaultfont gmail_msg" dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt"><div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt" class="gmail_msg"><blockquote dir="ltr" style="border-left:solid #aaaaaa 2px;margin-left:5px;padding-left:5px;direction:ltr;margin-right:0px" class="gmail_msg">Cc:<br class="gmail_msg">Subject: Re: [gpfsug-discuss] nodes being ejected out of the cluster<br class="gmail_msg">Date: Wed, Jan 11, 2017 5:03 PM<br class="gmail_msg"> <br class="gmail_msg"><font size="2" face="sans-serif" class="gmail_msg">most likely, there's smth wrong with your IB fabric ... </font><br class="gmail_msg"><font size="2" face="sans-serif" class="gmail_msg">you say, you run ~ 700 nodes ? ...</font><br class="gmail_msg"><font size="2" face="sans-serif" class="gmail_msg">Are you running with </font><font size="3" class="gmail_msg"><b class="gmail_msg">verbsRdmaSend</b></font><font size="2" face="sans-serif" class="gmail_msg">enabled ? ,if so, please consider to disable - and discuss this within the PMR </font><br class="gmail_msg"><font size="2" face="sans-serif" class="gmail_msg">another issue, you may check is - Are you running the IPoIB in connected mode or datagram ... but as I said, please discuss this within the PMR .. there are to much dependencies to discuss this here .. </font><br class="gmail_msg"><br class="gmail_msg"><br class="gmail_msg"><font size="2" face="sans-serif" class="gmail_msg">cheers</font><br class="gmail_msg"><br class="gmail_msg">
<div class="gmail_msg"><font size="2" face="sans-serif" class="gmail_msg">Mit freundlichen Grüßen / Kind regards</font><br class="gmail_msg"><br class="gmail_msg"><font size="2" face="sans-serif" class="gmail_msg"> <br class="gmail_msg">Olaf Weiser<br class="gmail_msg"><br class="gmail_msg">EMEA Storage Competence Center Mainz, German / IBM Systems, Storage Platform,<br class="gmail_msg">-------------------------------------------------------------------------------------------------------------------------------------------<br class="gmail_msg">IBM Deutschland<br class="gmail_msg">IBM Allee 1<br class="gmail_msg">71139 Ehningen<br class="gmail_msg">Phone: <a href="tel:+49%20170%205794466" value="+491705794466" class="gmail_msg" target="_blank">+49-170-579-44-66</a><br class="gmail_msg">E-Mail: <a href="mailto:olaf.weiser@de.ibm.com" class="gmail_msg" target="_blank">olaf.weiser@de.ibm.com</a><br class="gmail_msg">-------------------------------------------------------------------------------------------------------------------------------------------<br class="gmail_msg">IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter<br class="gmail_msg">Geschäftsführung: Martina Koederitz (Vorsitzende), Susanne Peter, Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Markus Koerner<br class="gmail_msg">Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 </font><br class="gmail_msg"><br class="gmail_msg"><br class="gmail_msg"><br class="gmail_msg"><font color="#5f5f5f" size="1" face="sans-serif" class="gmail_msg">From: </font><font size="1" face="sans-serif" class="gmail_msg">Damir Krstic <<a href="mailto:damir.krstic@gmail.com" class="gmail_msg" target="_blank">damir.krstic@gmail.com</a>></font><br class="gmail_msg"><font color="#5f5f5f" size="1" face="sans-serif" class="gmail_msg">To: </font><font size="1" face="sans-serif" class="gmail_msg">gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" class="gmail_msg" target="_blank">gpfsug-discuss@spectrumscale.org</a>></font><br class="gmail_msg"><font color="#5f5f5f" size="1" face="sans-serif" class="gmail_msg">Date: </font><font size="1" face="sans-serif" class="gmail_msg">01/11/2017 03:39 PM</font><br class="gmail_msg"><font color="#5f5f5f" size="1" face="sans-serif" class="gmail_msg">Subject: </font><font size="1" face="sans-serif" class="gmail_msg">[gpfsug-discuss] nodes being ejected out of the cluster</font><br class="gmail_msg"><font color="#5f5f5f" size="1" face="sans-serif" class="gmail_msg">Sent by: </font><font size="1" face="sans-serif" class="gmail_msg"><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class="gmail_msg" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a></font>
<hr noshade class="gmail_msg"><br class="gmail_msg"><br class="gmail_msg"><br class="gmail_msg"><font size="3" class="gmail_msg">We are running GPFS 4.2 on our cluster (around 700 compute nodes). Our storage (ESS GL6) is also running GPFS 4.2. Compute nodes and storage are connected via Infiniband (FDR14). At the time of implementation of ESS, we were instructed to enable RDMA in addition to IPoIB. Previously we only ran IPoIB on our GPFS3.5 cluster.</font><br class="gmail_msg"><br class="gmail_msg"><font size="3" class="gmail_msg">Every since the implementation (sometime back in July of 2016) we see a lot of compute nodes being ejected. What usually precedes the ejection are following messages:</font><br class="gmail_msg"><br class="gmail_msg"><font size="3" class="gmail_msg">Jan 11 02:03:15 quser13 mmfs: [E] VERBS RDMA rdma send error IBV_WC_RNR_RETRY_EXC_ERR to 172.41.2.5 (gssio2-fdr) on mlx4_0 port 1 fabnum 0 vendor_err 135 </font><br class="gmail_msg"><font size="3" class="gmail_msg">Jan 11 02:03:15 quser13 mmfs: [E] VERBS RDMA closed connection to 172.41.2.5 (gssio2-fdr) on mlx4_0 port 1 fabnum 0 due to send error IBV_WC_RNR_RETRY_EXC_ERR index 2</font><br class="gmail_msg"><font size="3" class="gmail_msg">Jan 11 02:03:26 quser13 mmfs: [E] VERBS RDMA rdma send error IBV_WC_RNR_RETRY_EXC_ERR to 172.41.2.5 (gssio2-fdr) on mlx4_0 port 1 fabnum 0 vendor_err 135 </font><br class="gmail_msg"><font size="3" class="gmail_msg">Jan 11 02:03:26 quser13 mmfs: [E] VERBS RDMA closed connection to 172.41.2.5 (gssio2-fdr) on mlx4_0 port 1 fabnum 0 due to send error IBV_WC_WR_FLUSH_ERR index 1</font><br class="gmail_msg"><font size="3" class="gmail_msg">Jan 11 02:03:26 quser13 mmfs: [E] VERBS RDMA rdma send error IBV_WC_RNR_RETRY_EXC_ERR to 172.41.2.5 (gssio2-fdr) on mlx4_0 port 1 fabnum 0 vendor_err 135 </font><br class="gmail_msg"><font size="3" class="gmail_msg">Jan 11 02:03:26 quser13 mmfs: [E] VERBS RDMA closed connection to 172.41.2.5 (gssio2-fdr) on mlx4_0 port 1 fabnum 0 due to send error IBV_WC_RNR_RETRY_EXC_ERR index 2</font><br class="gmail_msg"><font size="3" class="gmail_msg">Jan 11 02:06:38 quser11 mmfs: [E] VERBS RDMA rdma send error IBV_WC_RNR_RETRY_EXC_ERR to 172.41.2.5 (gssio2-fdr) on mlx4_0 port 1 fabnum 0 vendor_err 135 </font><br class="gmail_msg"><font size="3" class="gmail_msg">Jan 11 02:06:38 quser11 mmfs: [E] VERBS RDMA closed connection to 172.41.2.5 (gssio2-fdr) on mlx4_0 port 1 fabnum 0 due to send error IBV_WC_WR_FLUSH_ERR index 400</font><br class="gmail_msg"><br class="gmail_msg"><font size="3" class="gmail_msg">Even our ESS IO server sometimes ends up being ejected (case in point - yesterday morning):</font><br class="gmail_msg"><br class="gmail_msg"><font size="3" class="gmail_msg">Jan 10 11:23:42 gssio2 mmfs: [E] VERBS RDMA rdma send error IBV_WC_RNR_RETRY_EXC_ERR to 172.41.2.1 (gssio1-fdr) on mlx5_1 port 1 fabnum 0 vendor_err 135</font><br class="gmail_msg"><font size="3" class="gmail_msg">Jan 10 11:23:42 gssio2 mmfs: [E] VERBS RDMA closed connection to 172.41.2.1 (gssio1-fdr) on mlx5_1 port 1 fabnum 0 due to send error IBV_WC_RNR_RETRY_EXC_ERR index 3001</font><br class="gmail_msg"><font size="3" class="gmail_msg">Jan 10 11:23:43 gssio2 mmfs: [E] VERBS RDMA rdma send error IBV_WC_RNR_RETRY_EXC_ERR to 172.41.2.1 (gssio1-fdr) on mlx5_1 port 2 fabnum 0 vendor_err 135</font><br class="gmail_msg"><font size="3" class="gmail_msg">Jan 10 11:23:43 gssio2 mmfs: [E] VERBS RDMA closed connection to 172.41.2.1 (gssio1-fdr) on mlx5_1 port 2 fabnum 0 due to send error IBV_WC_RNR_RETRY_EXC_ERR index 2671</font><br class="gmail_msg"><font size="3" class="gmail_msg">Jan 10 11:23:43 gssio2 mmfs: [E] VERBS RDMA rdma send error IBV_WC_RNR_RETRY_EXC_ERR to 172.41.2.1 (gssio1-fdr) on mlx5_0 port 2 fabnum 0 vendor_err 135</font><br class="gmail_msg"><font size="3" class="gmail_msg">Jan 10 11:23:43 gssio2 mmfs: [E] VERBS RDMA closed connection to 172.41.2.1 (gssio1-fdr) on mlx5_0 port 2 fabnum 0 due to send error IBV_WC_RNR_RETRY_EXC_ERR index 2495</font><br class="gmail_msg"><font size="3" class="gmail_msg">Jan 10 11:23:44 gssio2 mmfs: [E] VERBS RDMA rdma send error IBV_WC_RNR_RETRY_EXC_ERR to 172.41.2.1 (gssio1-fdr) on mlx5_0 port 1 fabnum 0 vendor_err 135</font><br class="gmail_msg"><font size="3" class="gmail_msg">Jan 10 11:23:44 gssio2 mmfs: [E] VERBS RDMA closed connection to 172.41.2.1 (gssio1-fdr) on mlx5_0 port 1 fabnum 0 due to send error IBV_WC_RNR_RETRY_EXC_ERR index 3077</font><br class="gmail_msg"><font size="3" class="gmail_msg">Jan 10 11:24:11 gssio2 mmfs: [N] Node 172.41.2.1 (gssio1-fdr) lease renewal is overdue. Pinging to check if it is alive</font><br class="gmail_msg"><br class="gmail_msg"><font size="3" class="gmail_msg">I've had multiple PMRs open for this issue, and I am told that our ESS needs code level upgrades in order to fix this issue. Looking at the errors, I think the issue is Infiniband related, and I am wondering if anyone on this list has seen similar issues?</font><br class="gmail_msg"><br class="gmail_msg"><font size="3" class="gmail_msg">Thanks for your help in advance.</font><br class="gmail_msg"><br class="gmail_msg"><font size="3" class="gmail_msg">Damir</font><tt class="gmail_msg"><font size="3" face="" class="gmail_msg">_______________________________________________<br class="gmail_msg">gpfsug-discuss mailing list<br class="gmail_msg">gpfsug-discuss at <a href="http://spectrumscale.org" class="gmail_msg" target="_blank">spectrumscale.org</a></font></tt><br class="gmail_msg"><tt class="gmail_msg"><font size="3" face="" class="gmail_msg"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="gmail_msg" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><br class="gmail_msg"><br class="gmail_msg"> </div>
<div class="gmail_msg"><font size="2" face="Default Monospace,Courier New,Courier,monospace" class="gmail_msg">_______________________________________________<br class="gmail_msg">gpfsug-discuss mailing list<br class="gmail_msg">gpfsug-discuss at <a href="http://spectrumscale.org" class="gmail_msg" target="_blank">spectrumscale.org</a><br class="gmail_msg"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="gmail_msg" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></div></blockquote></div></div><div class="m_-1181131753993210372m_4020387606308923108socmaildefaultfont gmail_msg" dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt"><div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt" class="gmail_msg"><blockquote dir="ltr" style="border-left:solid #aaaaaa 2px;margin-left:5px;padding-left:5px;direction:ltr;margin-right:0px" class="gmail_msg"></blockquote>
<div dir="ltr" class="gmail_msg"> </div></div></div><br class="gmail_msg">
Ellei edellä ole toisin mainittu: / Unless stated otherwise above:<br class="gmail_msg">
Oy IBM Finland Ab<br class="gmail_msg">
PL 265, 00101 Helsinki, Finland<br class="gmail_msg">
Business ID, Y-tunnus: 0195876-3 <br class="gmail_msg">
Registered in Finland<br class="gmail_msg">
<br class="gmail_msg">
_______________________________________________<br class="gmail_msg">
gpfsug-discuss mailing list<br class="gmail_msg">
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" class="gmail_msg" target="_blank">spectrumscale.org</a><br class="gmail_msg">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" class="gmail_msg" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="gmail_msg">
</blockquote></div>
_______________________________________________<br class="gmail_msg">
gpfsug-discuss mailing list<br class="gmail_msg">
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" class="gmail_msg" target="_blank">spectrumscale.org</a><br class="gmail_msg">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" class="gmail_msg" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="gmail_msg">
</blockquote></div>