<font size=2 face="sans-serif">Hi Robert,</font><br><br><font size=2 face="sans-serif">So you might have started seeing problem
after upgrading the gateway nodes to 5.0.5.2. Upgrading gateway nodes at
cache cluster to 5.0.5.6 would resolve this problem.</font><br><br><font size=2 face="sans-serif">~Venkat (vpuvvada@in.ibm.com)</font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Robert Horton <robert.horton@icr.ac.uk></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">"gpfsug-discuss@spectrumscale.org"
<gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">03/19/2021 03:13 PM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">[EXTERNAL] Re:
[gpfsug-discuss] SpectrumScale / AFM / Singularity soft lockups</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=3>Hi </font><font size=2>Venkat,</font><br><br><font size=2>Thanks for getting back to me.</font><br><br><font size=2>On the cache side we're running </font><font size=3>5.0.4-3
on the nsd servers and 5.0.5-2 everywhere else, including gateway nodes.</font><br><font size=3>The home cluster is 4.2.3-22 - unfortunately we're stuck
on 4.x due to the licensing but we're in the process of replacing that
system.</font><br><br><font size=3>The actual AFM seems to be behaving fine though so I'm
not sure that's our issue. I guess our next job is to see if we can reproduce
it in a non-AFM fileset.</font><br><br><font size=3>Rob</font><br><br><font size=3>On Fri, 2021-03-19 at 12:02 +0530, Venkateswara R Puvvada
wrote:</font><br><font size=2 color=#806210><b>CAUTION:</b></font><font size=2> This
email originated from outside of the ICR. Do not click links or open attachments
unless you recognize the sender's email address and know the content is
safe.</font><br><br><font size=2 face="sans-serif">Robert,</font><font size=3><br></font><font size=2 face="sans-serif"><br>What is the scale version ? This issue may be related to these alerts.</font><font size=3><br></font><font size=3 color=blue><u><br></u></font><a href=https://www.ibm.com/support/pages/node/6355983><font size=3 color=blue><u>https://www.ibm.com/support/pages/node/6355983</u></font></a><font size=3 color=blue><u><br></u></font><a href=https://www.ibm.com/support/pages/node/6380740><font size=2 color=blue face="sans-serif"><u>https://www.ibm.com/support/pages/node/6380740</u></font></a><font size=3><br><br>These are the recommended steps to resolve the issue, but need more details
on the scale version.<br><br>1. Stop all AFM filesets at cache using "mmafmctl device stop -j fileset"
command.<br>2. Perform rolling upgrade parallely at both cache and home clusters<br> a. All nodes on home cluster to 5.0.5.6<br> b. All gateway nodes in cache cluster to 5.0.5.6<br> 3. At home cluster, for each fileset target path, repeat below steps<br> a. Remove .afmctl file<br> mmafmlocal rm <fileset target path>/.afm/.afmctl<br> b. Enable AFM<br> mmafmconfig enable <fileset target path><br>4. Start all AFM filesets at cache using "mmafmctl device start -j
fileset" command. <br></font><font size=2 face="sans-serif"><br>~Venkat (vpuvvada@in.ibm.com)</font><font size=3><br><br><br></font><font size=1 color=#5f5f5f face="sans-serif"><br>From: </font><font size=1 face="sans-serif">Robert
Horton <robert.horton@icr.ac.uk></font><font size=1 color=#5f5f5f face="sans-serif"><br>To: </font><font size=1 face="sans-serif">"gpfsug-discuss@spectrumscale.org"
<gpfsug-discuss@spectrumscale.org></font><font size=1 color=#5f5f5f face="sans-serif"><br>Date: </font><font size=1 face="sans-serif">03/18/2021
09:17 PM</font><font size=1 color=#5f5f5f face="sans-serif"><br>Subject: </font><font size=1 face="sans-serif">[EXTERNAL]
[gpfsug-discuss] SpectrumScale / AFM / Singularity soft lockups</font><font size=1 color=#5f5f5f face="sans-serif"><br>Sent by: </font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><font size=3><br></font><hr noshade><font size=3><br><br><br>Hello,<br><br>We've recently started having an issue where processes running in a singularity
container get stuck in a soft lockup and eventually the node needs to be
forcibly rebooted. I have included a sample call trace below. Additionally,
other (non-singularity) processes on other nodes accessing the same fileset
seem to get into the same state. It's also an AFM IW fileset just to add
to the complexity ;)<br><br>Does anyone have any thoughts on what might be happening / how to proceed?
I'm not really sure if it's a GPFS issue or a Singularity / Kernel issue
- although fact it seems to spread to other nodes would seem to suggest
some GPFS involvement. It's possible the user is doing something inadvisable
with Singularity (it's difficult to work out what's happening in the Nextflow
pipeline) but even if they are it would be good to find a way of preventing
them taking nodes down. I'm assuming the AFM is unlikely to be relevant
- any views on that?<br><br>Thanks,<br>Rob<br><br> Call Trace:<br>? _Z11kSFSGetattrP15KernelOperationP13gpfsVfsData_tP10gpfsNode_tiP10cxiVattr_tP12gpfs_iattr64+0x1e4/0x5d0
[mmfs26]<br> _ZL17refreshCacheAttrsP13gpfsVfsData_tP15KernelOperationP9cxiNode_tP10pcacheAttriPcj+0x441/0x450
[mmfs26]<br> _Z21pcacheHandleCollisionP13gpfsVfsData_tP15KernelOperationP10gpfsNode_tS4_PcPvP9MMFSVInfoiP10pcacheAttriS5_10PcacheModej+0xa21/0x11b0
[mmfs26]<br> ? _ZN6ThCond6signalEv+0x82/0x190 [mmfs26]<br> ? _ZN10MemoryPool6shFreeEPv9MallocUse+0x1a5/0x2a0 [mmfs26]<br> ? _ZL14kSFSPcacheSendP13gpfsVfsData_tP15KernelOperation7FileUIDS3_PciiPPv+0x387/0x570
[mmfs26]<br> ? _ZL17pcacheNeedRefresh10PcacheModejlijj+0x206/0x230 [mmfs26]<br>_Z12pcacheLookupP13gpfsVfsData_tP15KernelOperationP10gpfsNode_tPvPcP7FilesetjjjPS5_PS4_PyPjS9_+0x1dcf/0x25c0
[mmfs26]<br>? _Z15findFilesetByIdP15KernelOperationjjPP7Filesetj+0x4f/0xa0 [mmfs26]<br> _Z10gpfsLookupP13gpfsVfsData_tPvP9cxiNode_tS1_S1_PcjPS1_PS3_PyP10cxiVattr_tPjP10ext_cred_tjS5_PiS4_SD_+0x65c/0xad0
[mmfs26]<br>gpfs_i_lookup+0x189/0x3f0 [mmfslinux]<br> ? _Z8gpfsLinkP13gpfsVfsData_tP9cxiNode_tS2_PvPcjjP10ext_cred_t+0x6e0/0x6e0
[mmfs26]<br> ? d_alloc_parallel+0x99/0x4a0<br> ? _Z33gpfsIsCifsBypassTraversalCheckingv+0xe2/0x130 [mmfs26]<br> __lookup_slow+0x97/0x150<br> lookup_slow+0x35/0x50<br> walk_component+0x1bf/0x330<br> ? _ZL12gpfsGetattrxP13gpfsVfsData_tP9cxiNode_tP10cxiVattr_tP12gpfs_iattr64i+0x147/0x390
[mmfs26]<br> path_lookupat.isra.49+0x75/0x200<br> filename_lookup.part.63+0xa0/0x170<br>? strncpy_from_user+0x4f/0x1b0<br> vfs_statx+0x73/0xe0<br> __do_sys_newlstat+0x39/0x70<br> ? syscall_trace_enter+0x1d3/0x2c0<br> ? __audit_syscall_exit+0x249/0x2a0<br> do_syscall_64+0x5b/0x1a0<br> entry_SYSCALL_64_after_hwframe+0x65/0xca</font><tt><font size=3><br>-- </font></tt><font size=3><br>Robert Horton | Research Data Storage Lead<br>The Institute of Cancer Research | 237 Fulham Road | London | SW3 6JB<br>T +44 (0)20 7153 5350 | E </font><a href=mailto:robert.horton@icr.ac.uk><font size=3 color=blue><u>robert.horton@icr.ac.uk</u></font></a><font size=3>|
W </font><a href=www.icr.ac.uk><font size=3 color=blue><u>www.icr.ac.uk</u></font></a><font size=3>|
Twitter @ICR_London<br>Facebook: </font><a href=www.facebook.com/theinstituteofcancerresearch><font size=3 color=blue><u>www.facebook.com/theinstituteofcancerresearch</u></font></a><font size=3><br><br>The Institute of Cancer Research: Royal Cancer Hospital, a charitable Company
Limited by Guarantee, Registered in England under Company No. 534147 with
its Registered Office at 123 Old Brompton Road, London SW7 3RP.<br><br>This e-mail message is confidential and for use by the addressee only.
If the message is received by anyone other than the addressee, please return
the message to the sender by replying to it and then delete the message
from your computer and network.</font><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</font></tt><font size=3 color=blue><u><br></u></font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=gHmKEtEM3EvdWRefAF0Cs8N2qXPZg5flGutpiJu_bfg&s=dnKFsINgU63_3b-7i3z3uDnxnij6iT-y8L_mmYHr8IE&e="><tt><font size=2 color=blue><u>https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=gHmKEtEM3EvdWRefAF0Cs8N2qXPZg5flGutpiJu_bfg&s=dnKFsINgU63_3b-7i3z3uDnxnij6iT-y8L_mmYHr8IE&e=</u></font></tt></a><font size=3><br><br><br></font><br><tt><font size=3>-- </font></tt><br><font size=3>Robert Horton | Research Data Storage Lead</font><br><font size=3>The Institute of Cancer Research | 237 Fulham Road | London
| SW3 6JB</font><br><font size=3>T +44 (0)20 7153 5350 | E </font><a href=mailto:robert.horton@icr.ac.uk><font size=3 color=blue><u>robert.horton@icr.ac.uk</u></font></a><font size=3>| W </font><a href=www.icr.ac.uk><font size=3>www.icr.ac.uk</font></a><font size=3>| Twitter @ICR_London</font><br><font size=3>Facebook: </font><a href=www.facebook.com/theinstituteofcancerresearch><font size=3 color=blue><u>www.facebook.com/theinstituteofcancerresearch</u></font></a><br><font size=3><br>The Institute of Cancer Research: Royal Cancer Hospital, a charitable Company
Limited by Guarantee, Registered in England under Company No. 534147 with
its Registered Office at 123 Old Brompton Road, London SW7 3RP.<br><br>This e-mail message is confidential and for use by the addressee only.
If the message is received by anyone other than the addressee, please return
the message to the sender by replying to it and then delete the message
from your computer and network.</font><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=KgYs-kXBKE5JoAaGYRiU9iIxNkJSZeicxpSTmL39_B8&s=6FodZ_EQ8VAOE_xoEkfoUzmJpaiF7bgbERvA9avLZfg&e="><tt><font size=2>https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=KgYs-kXBKE5JoAaGYRiU9iIxNkJSZeicxpSTmL39_B8&s=6FodZ_EQ8VAOE_xoEkfoUzmJpaiF7bgbERvA9avLZfg&e=</font></tt></a><tt><font size=2><br></font></tt><br><br><BR>