<span style=" font-size:10pt;font-family:sans-serif">Right for the example
from Ryan(and according to the thread name, you know that it is writing
to a file or directory), but for other cases, it may take more steps to
figure out what access to which file is causing the long waiters(i.e.,
when mmap is being used on some nodes, or token revoke pending from some
node, and etc.).</span><br><br><span style=" font-size:10pt;font-family:sans-serif">Regards, The Spectrum
Scale (GPFS) team<br><br>------------------------------------------------------------------------------------------------------------------<br>If you feel that your question can benefit other users of Spectrum
Scale (GPFS), then please post it to the public IBM developerWroks Forum
at </span><a href="https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479"><span style=" font-size:10pt;font-family:sans-serif">https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479</span></a><span style=" font-size:10pt;font-family:sans-serif">.
<br><br>If your query concerns a potential software error in Spectrum Scale (GPFS)
and you have an IBM software maintenance contract please contact 1-800-237-5511
in the United States or your local IBM Service Center in other countries.
<br><br>The forum is informally monitored as time permits and should not be used
for priority messages to the Spectrum Scale (GPFS) team.</span><br><br><br><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">From:
</span><span style=" font-size:9pt;font-family:sans-serif">Ryan
Novosielski <novosirj@rutgers.edu></span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">To:
</span><span style=" font-size:9pt;font-family:sans-serif">gpfsug
main discussion list <gpfsug-discuss@spectrumscale.org></span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Date:
</span><span style=" font-size:9pt;font-family:sans-serif">2019/10/18
09:18 AM</span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Subject:
</span><span style=" font-size:9pt;font-family:sans-serif">[EXTERNAL]
Re: [gpfsug-discuss] waiters and files causing waiters</span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Sent
by: </span><span style=" font-size:9pt;font-family:sans-serif">gpfsug-discuss-bounces@spectrumscale.org</span><br><hr noshade><br><br><br><tt><span style=" font-size:10pt">Found my notes on this; very similar
to what Behrooz was saying. <br><br>This here is from “mmfsadm dump waiters,selected_files”; as you can see
here, we’re looking at thread 29168. Apparently below, “inodeFlushHolder”
corresponds to that same thread in the case I was looking at.<br><br>You could then look up the inode with “tsfindinode -i <inode> <fsname>”,
so like for the below, "tsfindinode -i 41538053 /gpfs/cache” on our
system.<br><br>===== dump waiters ====<br>Current time 2019-05-01_13:48:26-0400<br>Waiting 0.1669 sec since 13:48:25, monitored, thread 29168 FileBlockWriteFetchHandlerThread:
on ThCond 0x7F55E40014C8 (MsgRecordCondvar), reason 'RPC wait' for quotaMsgRequestShare
on node 192.168.33.7 <c1n1><br><br>===== dump selected_files =====<br>Current time 2019-05-01_13:48:36-0400<br><br>...<br><br>OpenFile: 4E044E5B0601A8C0:000000000279D205:0000000000000000 @ 0x1806AC5EAC8<br> cach 1 ref 1 hc 2 tc 6 mtx 0x1806AC5EAF8<br> Inode: valid eff token xw @ 0x1806AC5EC70, ctMode xw seq 170823<br> lock state [ wf: 1 ] x [] flags [ ]<br> Mnode: valid eff token xw @ 0x1806AC5ECC0, ctMode xw seq 170823<br> DMAPI: invalid eff token nl @ 0x1806AC5EC20, ctMode nl seq 170821<br> SMBOpen: valid eff token (A:RMA D: ) @ 0x1806AC5EB50, ctMode (A:RMA
D: ) seq 170823<br> lock state [ M(2) D: ] x [] flags [ ]<br> SMBOpLk: valid eff token wf @ 0x1806AC5EBC0, ctMode wf Flags 0x30 (pfro+pfxw)
seq 170822<br> BR: @ 0x1806AC5ED20, ctMode nl Flags 0x10 (pfro) seq 170823<br> treeP 0x18016189C08 C btFastTrack 0 1 ranges mode RO/XW:<br> BLK [0,INF] mode XW node <403><br> Fcntl: @ 0x1806AC5ED48, ctMode nl Flags 0x30 (pfro+pfxw) seq 170823<br> treeP 0x18031A5E3F8 C btFastTrack 0 1 ranges mode RO/XW:<br> BLK [0,INF] mode XW node <403><br> inode 41538053 snap 0 USERFILE nlink 1 genNum 0x3CC2743F mode 0200100600:
-rw-------<br> tmmgr node <c1n1> (other)<br> metanode <c1n403> (me) fail+panic count -1 flags 0x0, remoteStart
0 remoteCnt 0 localCnt 177 lastFrom 65535 switchCnt 0<br> locks held in mode xw:<br> 0x1806AC5F238: 0x0-0xFFF tid 15954 gbl 0 mode xw rel 0<br> BRL nXLocksOrRelinquishes 285<br> vfsReference 1<br> dioCount 0 dioFlushNeeded 1 dioSkipCounter 0 dioReentryThreshold 0.000000<br> hasWriterInstance 1<br> inodeFlushFlag 1 inodeFlushHolder 29168 openInstCount 1<br> metadataFlushCount 2, metadataFlushWaiters 0/0, metadataCommitVersion
1<br> bufferListCount 1 bufferListChangeCount 3<br> dirty status: flushed dirtiedSyncNum 1477623<br> SMB oplock state: nWriters 1<br> indBlockDeallocLock:<br> sharedLockWord 1 exclLockWord 0 upgradeWaitingS_W 0 upgradeWaitingW_X
0<br> inodeValid 1<br> objectVersion 240<br> flushVersion 8086700 mnodeChangeCount 1<br> block size code 5 (32 subblocksPerFileBlock)<br> dataBytesPerFileBlock 4194304<br> fileSize 0 synchedFileSize 0 indirectionLevel 1<br> atime 1556732911.496160000<br> mtime 1556732911.496479000<br> ctime 1556732911.496479000<br> crtime 1556732911.496160000<br> owner uid 169589 gid 169589<br><br>> On Oct 10, 2019, at 4:43 PM, Damir Krstic <damir.krstic@gmail.com>
wrote:<br>> <br>> is it possible via some set of mmdiag --waiters or mmfsadm dump ?
to figure out which files or directories access (whether it's read or write)
is causing long-er waiters?<br>> <br>> in all my looking i have not been able to get that information out
of various diagnostic commands.<br>> <br>> thanks,<br>> damir<br>> _______________________________________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at spectrumscale.org<br>> </span></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><span style=" font-size:10pt">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></tt></a><tt><span style=" font-size:10pt"><br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></span></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><span style=" font-size:10pt">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></tt></a><tt><span style=" font-size:10pt"><br><br></span></tt><br><br><BR>