[gpfsug-discuss] waiters and files causing waiters

[Behrooz Amirzadeh]

I think it depends on the type of deadlock. For example, if hung nodes are
the cause of the deadlock. I don't think there will be any files to go
after. I've seen that it is possible in certain cases but no guarantees.
When the deadlock is detected you can look at the internaldump that gets
created on the deadlock node, for example:

===== dump deadlock =====
Current time 2019-09-24_10:17:30-0400
Waiting 904.5729 sec since 10:02:25, on node aresnode7132, thread 3584968
SyncFSWorkerThread: on ThCond 0x18042226DB8 (LkObjCondvar), reason 'waiting
for RO lock'

Then you search in the same file for the ThCond further down. You'll most
likely see that it is associated with a mutex

===== dump condvar =====
Current time 2019-09-24_10:17:32-0400
.
.
  'LkObjCondvar' at 0x18042226DB8 (0xFFFFC90042226DB8)
    (mutex 'InodeCacheObjMutex' at 0x18042226C08 (0xFFFFC90042226C08
PTR_OK))
    waitCount 1 condvarEventWordP 0xFFFF880DB4AAF088

Then you'll search for the that mutex in the same file

===== dump selected_files =====
Current time 2019-09-24_10:17:32-0400
Files in stripe group gpfs0:

Selected: LkObj::mostWanted: 0x18042226D80 lock_state=0x2000000000000000
xlock_state=0x0 lock_flags=0x11
OpenFile:  429E985A0BFE280A:000000008285ECBD:0000000000000000 @
0x18042226BD8
  cach 1 ref 1 hc 3 tc 6 mtx 0x18042226C08
  Inode: valid eff token xw @ 0x18042226D80, ctMode xw seq 175
    lock state [ xw ] x [] flags [ dmn wka ] writer 39912 hasWaiters 1 0
  Mnode: valid eff token xw @ 0x18042226DD0, ctMode xw seq 175
  DMAPI: invalid eff token nl @ 0x18042226D30, ctMode nl seq 174
  SMBOpen: valid eff token (A: M  D:   ) @ 0x18042226C60, ctMode (A: M  D:
  ) Flags 0x30 (pfro+pfxw) seq 175
    lock state [ (nil) D: ] x [] flags [ ]
  SMBOpLk: valid eff token wf @ 0x18042226CD0, ctMode wf Flags 0x30
(pfro+pfxw) seq 175
  BR: @ 0x18042226E30, ctMode nl Flags 0x10 (pfro) seq 175
    treeP 0x18048C1EFB8 C btFastTrack 0 1 ranges mode RO/XW:
    BLK [0,INF] mode XW node <1335>
  Fcntl: @ 0x18042226E58, ctMode nl Flags 0x30 (pfro+pfxw) seq 175
    treeP 0x1801EBA7EE8 C btFastTrack 0 1 ranges mode RO/XW:
    BLK [0,INF] mode XW node <1335>
 * inode 2189814973* snap 0 USERFILE nlink 1 genNum 0x2710E0CC mode
0200100644: -rw-r--r--
  tmmgr node <c1n4> (other)
  metanode <c1n1335> (me) fail+panic count -1 flags 0x0, remoteStart 0
remoteCnt 0 localCnt 0 lastFrom 65535 switchCnt 0
  BRL nXLocksOrRelinquishes 6
  vfsReference 1
  dioCount 0 dioFlushNeeded 1 dioSkipCounter 0 dioReentryThreshold 0.000000
  lastAllocLsn 0xB8740C5E
  metadataFlushCount 2, metadataFlushWaiters 0/0, metadataCommitVersion 1
  bufferListCount 1 bufferListChangeCount 1
  dirty status: dirty fileDirty 1 fileDirtyOrUncommitted 1 dirtiedSyncNum
81078
  inodeValid 1 inodeDirtyCount 5
  objectVersion 1 mtimeDirty 1
  flushVersion 8983 mnodeChangeCount 1
  dirtyDataBufs 1
  block size code 5 (32 subblocksPerFileBlock)
  dataBytesPerFileBlock 4194304
  fileSize 10213 synchedFileSize 0 indirectionLevel 1
  atime 1569333733.493440000
  mtime 1569333742.784833000
  ctime 1569333742.784712266
  crtime 1569333733.493440000
*  owner uid 6572 gid 3047*

If you were lucky and all of these were found you can get the inode and the
uid/gid of the owner of the file. If you happen to catch it quick enough
you'll be able to find the file with lsof. Otherwise later with an ilm
policy run if the file has not been deleted by the user.

Behrooz


On Thu, Oct 10, 2019 at 7:05 PM Ryan Novosielski <novosirj at rutgers.edu>
wrote:

> I’ll dig through my notes. I had a similar situation and an engineer
> taught me how to do it. It’s a bit involved though. Not something you’d
> bother with for something transient.
>
> --
> ____
> || \\UTGERS,
> |---------------------------*O*---------------------------
> ||_// the State     |         Ryan Novosielski - novosirj at rutgers.edu
> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
> ||  \\    of NJ     | Office of Advanced Research Computing - MSB C630,
> Newark
>     `'
>
> On Oct 10, 2019, at 16:44, Damir Krstic <damir.krstic at gmail.com> wrote:
>
> 
> is it possible via some set of mmdiag --waiters or mmfsadm dump ? to
> figure out which files or directories access (whether it's read or write)
> is causing long-er waiters?
>
> in all my looking i have not been able to get that information out of
> various diagnostic commands.
>
> thanks,
> damir
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20191011/2c033650/attachment-0002.htm>