<div dir="ltr">I think it depends on the type of deadlock. For example, if hung nodes are the cause of the deadlock. I don't think there will be any files to go after. I've seen that it is possible in certain cases but no guarantees. When the deadlock is detected you can look at the internaldump that gets created on the deadlock node, for example:<div><br></div><div><font face="monospace">===== dump deadlock =====<br>Current time 2019-09-24_10:17:30-0400<br>Waiting 904.5729 sec since 10:02:25, on node aresnode7132, thread 3584968 SyncFSWorkerThread: on ThCond <font color="#ff0000">0x18042226DB8</font> (LkObjCondvar), reason 'waiting for RO lock'</font><br><div><br></div><div>Then you search in the same file for the ThCond further down. You'll most likely see that it is associated with a mutex</div><div><br></div><div><font face="monospace">===== dump condvar =====<br>Current time 2019-09-24_10:17:32-0400</font></div><div><font face="monospace">.</font></div><div><font face="monospace">.</font></div><div><font face="monospace"> 'LkObjCondvar' at <font color="#ff0000">0x18042226DB8</font> (0xFFFFC90042226DB8)<br> (mutex 'InodeCacheObjMutex' at <font color="#00ffff">0x18042226C08</font> (0xFFFFC90042226C08 PTR_OK))<br> waitCount 1 condvarEventWordP 0xFFFF880DB4AAF088</font><br></div><div><br></div><div>Then you'll search for the that mutex in the same file</div><div><br></div><div><font face="monospace">===== dump selected_files =====<br>Current time 2019-09-24_10:17:32-0400<br>Files in stripe group gpfs0:<br><br>Selected: LkObj::mostWanted: 0x18042226D80 lock_state=0x2000000000000000 xlock_state=0x0 lock_flags=0x11<br>OpenFile: 429E985A0BFE280A:000000008285ECBD:0000000000000000 @ 0x18042226BD8<br> cach 1 ref 1 hc 3 tc 6 mtx <font color="#00ffff">0x18042226C08</font><br> Inode: valid eff token xw @ 0x18042226D80, ctMode xw seq 175<br> lock state [ xw ] x [] flags [ dmn wka ] writer 39912 hasWaiters 1 0<br> Mnode: valid eff token xw @ 0x18042226DD0, ctMode xw seq 175<br> DMAPI: invalid eff token nl @ 0x18042226D30, ctMode nl seq 174<br> SMBOpen: valid eff token (A: M D: ) @ 0x18042226C60, ctMode (A: M D: ) Flags 0x30 (pfro+pfxw) seq 175<br> lock state [ (nil) D: ] x [] flags [ ] <br> SMBOpLk: valid eff token wf @ 0x18042226CD0, ctMode wf Flags 0x30 (pfro+pfxw) seq 175<br> BR: @ 0x18042226E30, ctMode nl Flags 0x10 (pfro) seq 175<br> treeP 0x18048C1EFB8 C btFastTrack 0 1 ranges mode RO/XW:<br> BLK [0,INF] mode XW node <1335><br> Fcntl: @ 0x18042226E58, ctMode nl Flags 0x30 (pfro+pfxw) seq 175<br> treeP 0x1801EBA7EE8 C btFastTrack 0 1 ranges mode RO/XW:<br> BLK [0,INF] mode XW node <1335><br> <b> inode 2189814973</b> snap 0 USERFILE nlink 1 genNum 0x2710E0CC mode 0200100644: -rw-r--r--<br></font></div><div><font face="monospace"> tmmgr node <c1n4> (other)<br> metanode <c1n1335> (me) fail+panic count -1 flags 0x0, remoteStart 0 remoteCnt 0 localCnt 0 lastFrom 65535 switchCnt 0<br> BRL nXLocksOrRelinquishes 6 <br> vfsReference 1 <br> dioCount 0 dioFlushNeeded 1 dioSkipCounter 0 dioReentryThreshold 0.000000<br> lastAllocLsn 0xB8740C5E <br> metadataFlushCount 2, metadataFlushWaiters 0/0, metadataCommitVersion 1<br> bufferListCount 1 bufferListChangeCount 1 <br> dirty status: dirty fileDirty 1 fileDirtyOrUncommitted 1 dirtiedSyncNum 81078 <br> inodeValid 1 inodeDirtyCount 5 <br> objectVersion 1 mtimeDirty 1 <br> flushVersion 8983 mnodeChangeCount 1 <br> dirtyDataBufs 1 <br> block size code 5 (32 subblocksPerFileBlock)<br> dataBytesPerFileBlock 4194304<br> fileSize 10213 synchedFileSize 0 indirectionLevel 1<br> atime 1569333733.493440000<br> mtime 1569333742.784833000<br> ctime 1569333742.784712266<br> crtime 1569333733.493440000<br><b> owner uid 6572 gid 3047</b></font><br></div><div><br></div><div>If you were lucky and all of these were found you can get the inode and the uid/gid of the owner of the file. If you happen to catch it quick enough you'll be able to find the file with lsof. Otherwise later with an ilm policy run if the file has not been deleted by the user.</div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div dir="ltr"><br><div><font face="monospace, monospace">Behrooz</font></div></div></div></div></div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Oct 10, 2019 at 7:05 PM Ryan Novosielski <<a href="mailto:novosirj@rutgers.edu">novosirj@rutgers.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="auto">
I’ll dig through my notes. I had a similar situation and an engineer taught me how to do it. It’s a bit involved though. Not something you’d bother with for something transient. <br>
<br>
<div dir="ltr"><span style="background-color:rgba(255,255,255,0)">--<br>
____<br>
|| \\UTGERS, |---------------------------*O*---------------------------<br>
||_// the State | Ryan Novosielski - <a href="mailto:novosirj@rutgers.edu" dir="ltr" target="_blank">novosirj@rutgers.edu</a><br>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus<br>
|| \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark<br>
`'</span></div>
<div dir="ltr"><br>
<blockquote type="cite">On Oct 10, 2019, at 16:44, Damir Krstic <<a href="mailto:damir.krstic@gmail.com" target="_blank">damir.krstic@gmail.com</a>> wrote:<br>
<br>
</blockquote>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">is it possible via some set of mmdiag --waiters or mmfsadm dump ? to figure out which files or directories access (whether it's read or write) is causing long-er waiters?
<div><br>
</div>
<div>in all my looking i have not been able to get that information out of various diagnostic commands.</div>
<div><br>
</div>
<div>thanks,</div>
<div>damir</div>
</div>
<span>_______________________________________________</span><br>
<span>gpfsug-discuss mailing list</span><br>
<span>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a></span><br>
<span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></span><br>
</div>
</blockquote>
</div>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote></div>