<font size=2 face=" Arial">well - hard to say.. 60K IO may or may
not be a problem... it depends on your storage backends.. </font><br><br><font size=2 face=" Arial">check the response times to the physical
disk on the NSD server... concerning the output you provided ... check
particularly </font><br><font size=2 face=" Arial">10.1.53.5 <c0n20> and 10.1.53.7
</font><br><br><font size=2 face=" Arial">.... if they are in the same (bad/ poor)
range .. then your storage back end is in trouble or maybe just too heavily
utilized ... </font><br><font size=2 face=" Arial"> if the response times to physical
disks on the NSD server are ok... .. than maybe the network from client
<--> NSD server is somehow in trouble .. </font><br><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Aaron Knister <aaron.s.knister@nasa.gov></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif"><gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">10/15/2016 08:28 AM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
SGExceptionLogBufferFullThread waiter</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><tt><font size=2>It absolutely does, thanks Olaf!<br><br>The tasks running on these nodes are running on 63 other nodes and <br>generating ~60K iop/s of metadata writes and I *think* about the same in
<br>reads. Do you think that could be contributing to the higher waiter <br>times? I'm not sure quite what the job is up to. It's seemingly doing <br>very little data movement, the cpu %used is very low but the load is <br>rather high.<br><br>-Aaron<br><br>On 10/15/16 11:23 AM, Olaf Weiser wrote:<br>> from your file system configuration .. mmfs <dev> -L you'll
find the<br>> size of the LOG<br>> since release 4.x ..you can change it, but you need to re-mount the
FS<br>> on every client , to make the change effective ...<br>><br>> when a clients initiate writes/changes to GPFS it needs to update
its<br>> changes to the log - if this narrows a certain filling degree,
GPFS<br>> triggers so called logWrapThreads to write content to disk and so
free<br>> space<br>><br>> with your given numbers ... double digit [ms] waiter times .. you
fs<br>> get's probably slowed down.. and there's something suspect with the<br>> storage, because LOG-IOs are rather small and should not take that
long<br>><br>> to give you an example from a healthy environment... the IO times
are so<br>> small, that you usually don't see waiters for this..<br>><br>> I/O start time RW Buf type disk:sectorNum
nSec time ms tag1<br>> tag2 Disk UID
typ NSD node context thread<br>> --------------- -- ----------- ----------------- ----- -------<br>> --------- --------- ------------------ --- --------------- ---------<br>> ----------<br>> 06:23:32.358851 W logData 2:524306424
8 0.439<br>> 0 0 C0A70D08:57CF40D1 cli
192.167.20.17 LogData<br>> SGExceptionLogBufferFullThread<br>> 06:23:33.576367 W logData 1:524257280
8 0.646<br>> 0 0 C0A70D08:57CF40D0 cli
192.167.20.16 LogData<br>> SGExceptionLogBufferFullThread<br>> 06:23:32.358851 W logData 2:524306424
8 0.439<br>> 0 0 C0A70D08:57CF40D1 cli
192.167.20.17 LogData<br>> SGExceptionLogBufferFullThread<br>> 06:23:33.576367 W logData 1:524257280
8 0.646<br>> 0 0 C0A70D08:57CF40D0 cli
192.167.20.16 LogData<br>> SGExceptionLogBufferFullThread<br>> 06:23:32.212426 W iallocSeg 1:524490048
64 0.733<br>> 2 245 C0A70D08:57CF40D0 cli 192.167.20.16
Logwrap<br>> LogWrapHelperThread<br>> 06:23:32.212412 W logWrap 2:524552192
8 0.755<br>> 0 179200 C0A70D08:57CF40D1 cli 192.167.20.17
Logwrap<br>> LogWrapHelperThread<br>> 06:23:32.212432 W logWrap 2:525162760
8 0.737<br>> 0 125473 C0A70D08:57CF40D1 cli 192.167.20.17
Logwrap<br>> LogWrapHelperThread<br>> 06:23:32.212416 W iallocSeg 2:524488384
64 0.763<br>> 2 347 C0A70D08:57CF40D1 cli 192.167.20.17
Logwrap<br>> LogWrapHelperThread<br>> 06:23:32.212414 W logWrap 2:525266944
8 2.160<br>> 0 177664 C0A70D08:57CF40D1 cli 192.167.20.17
Logwrap<br>> LogWrapHelperThread<br>><br>><br>> hope this helps ..<br>><br>><br>> Mit freundlichen Grüßen / Kind regards<br>><br>><br>> Olaf Weiser<br>><br>> EMEA Storage Competence Center Mainz, German / IBM Systems, Storage<br>> Platform,<br>> -------------------------------------------------------------------------------------------------------------------------------------------<br>> IBM Deutschland<br>> IBM Allee 1<br>> 71139 Ehningen<br>> Phone: +49-170-579-44-66<br>> E-Mail: olaf.weiser@de.ibm.com<br>> -------------------------------------------------------------------------------------------------------------------------------------------<br>> IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter<br>> Geschäftsführung: Martina Koederitz (Vorsitzende), Susanne Peter,<br>> Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Markus Koerner<br>> Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht<br>> Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940<br>><br>><br>><br>> From: Aaron Knister <aaron.s.knister@nasa.gov><br>> To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>> Date: 10/15/2016 07:23 AM<br>> Subject: [gpfsug-discuss] SGExceptionLogBufferFullThread
waiter<br>> Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>> ------------------------------------------------------------------------<br>><br>><br>><br>> I've got a node that's got some curious waiters on it (see below).
Could<br>> someone explain what the "SGExceptionLogBufferFullThread"
waiter means?<br>><br>> Thanks!<br>><br>> -Aaron<br>><br>> === mmdiag: waiters ===<br>> 0x7FFFF040D600 waiting 0.038822715 seconds,<br>> SGExceptionLogBufferFullThread: on ThCond 0x7FFFDBB07628<br>> (0x7FFFDBB07628) (parallelWaitCond), reason 'wait for parallel write'<br>> for NSD I/O completion on node 10.1.53.5 <c0n20><br>> 0x7FFFE83F3D60 waiting 0.039629116 seconds, CleanBufferThread: on
ThCond<br>> 0x17B1488 (0x17B1488) (MsgRecordCondvar), reason 'RPC wait' for NSD
I/O<br>> completion on node 10.1.53.7 <c0n22><br>> 0x7FFFE8373A90 waiting 0.038921480 seconds, CleanBufferThread: on
ThCond<br>> 0x7FFFCD2B4E30 (0x7FFFCD2B4E30) (LogFileBufferDescriptorCondvar),
reason<br>> 'force wait on force active buffer write'<br>> 0x42CD9B0 waiting 0.028227004 seconds, CleanBufferThread: on ThCond<br>> 0x7FFFCD2B4E30 (0x7FFFCD2B4E30) (LogFileBufferDescriptorCondvar),
reason<br>> 'force wait for buffer write to complete'<br>> 0x7FFFE0F0EAD0 waiting 0.027864343 seconds, CleanBufferThread: on
ThCond<br>> 0x7FFFDC0EEA88 (0x7FFFDC0EEA88) (MsgRecordCondvar), reason 'RPC wait'<br>> for NSD I/O completion on node 10.1.53.7 <c0n22><br>> 0x1575560 waiting 0.028045975 seconds, RemoveHandlerThread: on ThCond<br>> 0x18020CE4E08 (0xFFFFC90020CE4E08) (LkObjCondvar), reason 'waiting
for<br>> LX lock'<br>> 0x1570560 waiting 0.038724949 seconds, CreateHandlerThread: on ThCond<br>> 0x18020CE50A0 (0xFFFFC90020CE50A0) (LkObjCondvar), reason 'waiting
for<br>> LX lock'<br>> 0x1563D60 waiting 0.073919918 seconds, RemoveHandlerThread: on ThCond<br>> 0x180235F6440 (0xFFFFC900235F6440) (LkObjCondvar), reason 'waiting
for<br>> LX lock'<br>> 0x1561560 waiting 0.054854513 seconds, RemoveHandlerThread: on ThCond<br>> 0x1802292D200 (0xFFFFC9002292D200) (LkObjCondvar), reason 'waiting
for<br>> LX lock'<br>> _______________________________________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at spectrumscale.org<br>> </font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br>><br>><br>><br>><br>><br>><br>> _______________________________________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at spectrumscale.org<br>> </font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br>><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br><br></font></tt><br><br><BR>