<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" id="owaParaStyle">P {margin-top:0;margin-bottom:0;}</style>
</head>
<body fpstyle="1" ocsi="0">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">
<div>Hi all,</div>
<div><br>
</div>
<div>I would like to have some hints about the following problem:<br>
</div>
<div><br>
</div>
<div>Waiting 26.6431 sec since 17:18:32, ignored, thread 38298 NSPDDiscoveryRunQueueThread: on ThCond 0x7FC98EB6A2B8 (MultiThreadWorkInstanceCond), reason 'waiting for helper threads'</div>
Waiting 2.7969 sec since 17:18:55, monitored, thread 39736 NSDThread: for I/O completion<br>
Waiting 2.8024 sec since 17:18:55, monitored, thread 39580 NSDThread: for I/O completion<br>
Waiting 3.0435 sec since 17:18:55, monitored, thread 39448 NSDThread: for I/O completion<br>
<br>
<div>I am testing a new GPFS cluster (GPFS cluster client with computing nodes remotely mounting the Storage GPFS Cluster) and I am running 65 gpfsperf commands (1 command per client in parallell) as follows:</div>
<div><br>
</div>
<div>/usr/lpp/mmfs/samples/perf/gpfsperf create seq /gpfs/home/caubet_m/gpfsperf/$(hostname).txt -fsync -n 24g -r 16m -th 8
<br>
</div>
<div><br>
</div>
<div>I am unable to reach more than 6.5GBps (Lenovo DSS G240 GPFS 5.0.2-1, on a testing a 'home' filesystem with 1MB blocksize and subblocks of 8KB). After several seconds I see many waiters for I/O completion (up to 5 seconds)</div>
<div>and also the 'waiting for helper threads' message shown above. Can somebody explain me the meaning for this message? How could I improve that?</div>
<div><br>
</div>
<div>Current config in the storage cluster is:</div>
<div><br>
</div>
<div>[root@merlindssio02 ~]# mmlsconfig <br>
Configuration data for cluster merlin.psi.ch:<br>
---------------------------------------------<br>
clusterName merlin.psi.ch<br>
clusterId 1511090979434548295<br>
autoload no<br>
dmapiFileHandleSize 32<br>
minReleaseLevel 5.0.2.0<br>
ccrEnabled yes<br>
nsdRAIDFirmwareDirectory /opt/lenovo/dss/firmware<br>
cipherList AUTHONLY<br>
maxblocksize 16m<br>
[merlindssmgt01]<br>
ignorePrefetchLUNCount yes<br>
[common]<br>
pagepool 4096M<br>
[merlindssio01,merlindssio02]<br>
pagepool 270089M<br>
[merlindssmgt01,dssg]<br>
pagepool 57684M<br>
maxBufferDescs 2m<br>
numaMemoryInterleave yes<br>
[common]<br>
prefetchPct 50<br>
[merlindssmgt01,dssg]<br>
prefetchPct 20<br>
nsdRAIDTracks 128k<br>
nsdMaxWorkerThreads 3k<br>
nsdMinWorkerThreads 3k<br>
nsdRAIDSmallThreadRatio 2<br>
nsdRAIDThreadsPerQueue 16<br>
nsdClientCksumTypeLocal ck64<br>
nsdClientCksumTypeRemote ck64<br>
nsdRAIDFlusherFWLogHighWatermarkMB 1000<br>
nsdRAIDBlockDeviceMaxSectorsKB 0<br>
nsdRAIDBlockDeviceNrRequests 0<br>
nsdRAIDBlockDeviceQueueDepth 0<br>
nsdRAIDBlockDeviceScheduler off<br>
nsdRAIDMaxPdiskQueueDepth 128<br>
nsdMultiQueue 512<br>
verbsRdma enable<br>
verbsPorts mlx5_0/1 mlx5_1/1<br>
verbsRdmaSend yes<br>
scatterBufferSize 256K<br>
maxFilesToCache 128k<br>
maxMBpS 40000<br>
workerThreads 1024<br>
nspdQueues 64<br>
[common]<br>
subnets 192.168.196.0/merlin-hpc.psi.ch;merlin.psi.ch<br>
adminMode central<br>
<br>
File systems in cluster merlin.psi.ch:<br>
--------------------------------------<br>
/dev/home<br>
/dev/t16M128K<br>
/dev/t16M16K<br>
/dev/t1M8K<br>
/dev/t4M16K<br>
/dev/t4M32K<br>
/dev/test<br>
</div>
<div><br>
</div>
<div>And for the computing cluster:</div>
<div><br>
</div>
<div>[root@merlin-c-001 ~]# mmlsconfig <br>
Configuration data for cluster merlin-hpc.psi.ch:<br>
-------------------------------------------------<br>
clusterName merlin-hpc.psi.ch<br>
clusterId 14097036579263601931<br>
autoload yes<br>
dmapiFileHandleSize 32<br>
minReleaseLevel 5.0.2.0<br>
ccrEnabled yes<br>
cipherList AUTHONLY<br>
maxblocksize 16M<br>
numaMemoryInterleave yes<br>
maxFilesToCache 128k<br>
maxMBpS 20000<br>
workerThreads 1024<br>
verbsRdma enable<br>
verbsPorts mlx5_0/1<br>
verbsRdmaSend yes<br>
scatterBufferSize 256K<br>
ignorePrefetchLUNCount yes<br>
nsdClientCksumTypeLocal ck64<br>
nsdClientCksumTypeRemote ck64<br>
pagepool 32G<br>
subnets 192.168.196.0/merlin-hpc.psi.ch;merlin.psi.ch<br>
adminMode central<br>
<br>
File systems in cluster merlin-hpc.psi.ch:<br>
------------------------------------------<br>
(none)<br>
</div>
<div><br>
</div>
<div>Thanks a lot and best regards,</div>
<div>Marc <br>
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<div class="BodyFragment"><font size="2"><span style="font-size:10pt">
<div class="PlainText">_________________________________________<br>
Paul Scherrer Institut <br>
High Performance Computing<br>
Marc Caubet Serrabou<br>
Building/Room: WHGA/019A</div>
<div class="PlainText"><font size="2"><span style="font-size:10pt">Forschungsstrasse, 111</span></font></div>
<div class="PlainText">5232 Villigen PSI<br>
Switzerland<br>
<br>
Telephone: +41 56 310 46 67<br>
E-Mail: marc.caubet@psi.ch</div>
</span></font></div>
</div>
</div>
</div>
</div>
</body>
</html>