<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body dir="auto">
That /is/ interesting.
<div><br>
</div>
<div>I’m a little confused about how that could be playing out in a case where I’m building on -1062.9.1, building for -1062.9.1, and running on -1062.9.1. Is there something inherent in the RPM building process that hasn’t caught up, or am I misunderstanding
that change’s impact on it?<br>
<br>
<div dir="ltr"><span style="background-color: rgba(255, 255, 255, 0);">--<br>
____<br>
|| \\UTGERS, |---------------------------*O*---------------------------<br>
||_// the State | Ryan Novosielski - <a href="mailto:novosirj@rutgers.edu" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="link" x-apple-data-detectors-result="1">novosirj@rutgers.edu</a><br>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus<br>
|| \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark<br>
`'</span></div>
<div dir="ltr"><br>
<blockquote type="cite">On Jan 17, 2020, at 10:35, Felipe Knop <knop@us.ibm.com> wrote:<br>
<br>
</blockquote>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt">
<div dir="ltr">Hi Ryan,</div>
<div dir="ltr"> </div>
<div dir="ltr">Some interesting IBM-internal communication overnight. The problems seems related to a change made to LINUX_KERNEL_VERSION_VERBOSE to handle the additional digit in the kernel numbering (3.10.0-1000+) . The GPL layer expected LINUX_KERNEL_VERSION_VERBOSE
to have that extra digit, and its absence resulted in an incorrect function being compiled in, which led to the crash.</div>
<div dir="ltr"> </div>
<div dir="ltr">This, at least, seems to make sense, in terms of matching to the symptoms of the problem.</div>
<div dir="ltr"> </div>
<div dir="ltr">We are still in internal debates on whether/how update our guidelines for gplbin generation ...</div>
<div dir="ltr"> </div>
<div dir="ltr">Regards,</div>
<div dir="ltr"> </div>
<div dir="ltr"> Felipe</div>
<div dir="ltr"> </div>
<div dir="ltr">----<br>
Felipe Knop knop@us.ibm.com<br>
GPFS Development and Security<br>
IBM Systems<br>
IBM Building 008<br>
2455 South Rd, Poughkeepsie, NY 12601<br>
(845) 433-9314 T/L 293-9314<br>
</div>
<div dir="ltr"> </div>
<div dir="ltr"> </div>
<blockquote data-history-content-modified="1" data-history-expanded="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px">
----- Original message -----<br>
From: Ryan Novosielski <novosirj@rutgers.edu><br>
Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>
To: "gpfsug-discuss@spectrumscale.org" <gpfsug-discuss@spectrumscale.org><br>
Cc:<br>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Kernel BUG/panic in mm/slub.c:3772 on Spectrum Scale Data Access Edition installed via gpfs.gplbin RPM on KVM guests<br>
Date: Thu, Jan 16, 2020 4:33 PM<br>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace">-----BEGIN PGP SIGNED MESSAGE-----<br>
Hash: SHA1<br>
<br>
Hi Felipe,<br>
<br>
I either misunderstood support or convinced them to take further<br>
action. It at first looked like they were suggesting "mmbuildgpl fixed<br>
it: case closed" (I know they wanted to close the SalesForce case<br>
anyway, which would prevent communication on the issue). At this<br>
point, they've asked for a bunch more information.<br>
<br>
Support is asking similar questions re: the speculations, and I'll<br>
provide them with the relevant output ASAP, but I did confirm all of<br>
that, including that there were no stray mmfs26/tracedev kernel<br>
modules anywhere else in the relevant /lib/modules PATHs. In the<br>
original case, I built on a machine running 3.10.0-957.27.2, but<br>
pointed to the 3.10.0-1062.9.1 source code/defined the relevant<br>
portions of usr/lpp/mmfs/src/config/env.mcr. That's always worked<br>
before, and rebuilding once the build system was running<br>
3.10.0-1062.9.1 as well did not change anything either. In all cases,<br>
the GPFS version was Spectrum Scale Data Access Edition 5.0.4-1. If<br>
you build against either the wrong kernel version or the wrong GPFS<br>
version, both will appear right in the filename of the gpfs.gplbin RPM<br>
you build. Mine is called:<br>
<br>
gpfs.gplbin-3.10.0-1062.9.1.el7.x86_64-5.0.4-1.x86_64.rpm<br>
<br>
Anyway, thanks for your response; I know you might not be<br>
following/working on this directly, but I figured the extra info might<br>
be of interest.<br>
<br>
On 1/16/20 8:41 AM, Felipe Knop wrote:<br>
> Hi Ryan,<br>
><br>
> I'm aware of this ticket, and I understand that there has been<br>
> active communication with the service team on this problem.<br>
><br>
> The crash itself, as you indicate, looks like a problem that has<br>
> been fixed:<br>
><br>
> <a href="https://www.ibm.com/support/pages/ibm-spectrum-scale-gpfs-releases-423" target="_blank">
https://www.ibm.com/support/pages/ibm-spectrum-scale-gpfs-releases-423</a><br>
13-or-later-and-5022-or-later-have-issues-where-kernel-crashes-rhel76-0<br>
><br>
> The fact that the problem goes away when *mmbuildgpl* is issued<br>
> appears to point to some incompatibility with kernel levels and/or<br>
> Scale version levels. Just speculating, some possible areas may<br>
> be:<br>
><br>
><br>
> * The RPM might have been built on a version of Scale without the<br>
> fix * The RPM might have been built on a different (minor) version<br>
> of the kernel * Somehow the VM picked a "leftover" GPFS kernel<br>
> module, as opposed to the one included in gpfs.gplbin -- given<br>
> that mmfsd never complained about a missing GPL kernel module<br>
><br>
><br>
> Felipe<br>
><br>
> ---- Felipe Knop knop@us.ibm.com GPFS Development and Security IBM<br>
> Systems IBM Building 008 2455 South Rd, Poughkeepsie, NY 12601<br>
> (845) 433-9314 T/L 293-9314<br>
><br>
><br>
><br>
><br>
> ----- Original message ----- From: Ryan Novosielski<br>
> <novosirj@rutgers.edu> Sent by:<br>
> gpfsug-discuss-bounces@spectrumscale.org To: gpfsug main discussion<br>
> list <gpfsug-discuss@spectrumscale.org> Cc: Subject: [EXTERNAL]<br>
> [gpfsug-discuss] Kernel BUG/panic in mm/slub.c:3772 on Spectrum<br>
> Scale Data Access Edition installed via gpfs.gplbin RPM on KVM<br>
> guests Date: Wed, Jan 15, 2020 4:11 PM<br>
><br>
> Hi there,<br>
><br>
> I know some of the Spectrum Scale developers look at this list.<br>
> I’m having a little trouble with support on this problem.<br>
><br>
> We are seeing crashes with GPFS 5.0.4-1 Data Access Edition on KVM<br>
> guests with a portability layer that has been installed via<br>
> gpfs.gplbin RPMs that we built at our site and have used to<br>
> install GPFS all over our environment. We’ve not seen this problem<br>
> so far on any physical hosts, but have now experienced it on guests<br>
> running on number of our KVM hypervisors, across vendors and<br>
> firmware versions, etc. At one time I thought it was all happening<br>
> on systems using Mellanox virtual functions for Infiniband, but<br>
> we’ve now seen it on VMs without VFs. There may be an SELinux<br>
> interaction, but some of our hosts have it disabled outright, some<br>
> are Permissive, and some were working successfully with 5.0.2.x<br>
> GPFS.<br>
><br>
> What I’ve been instructed to try to solve this problem has been to<br>
> run “mmbuildgpl”, and it has solved the problem. I don’t consider<br>
> running "mmbuildgpl" a real solution, however. If RPMs are a<br>
> supported means of installation, it should work. Support told me<br>
> that they’d seen this solve the problem at another site as well.<br>
><br>
> Does anyone have any more information about this problem/whether<br>
> there’s a fix in the pipeline, or something that can be done to<br>
> cause this problem that we could remedy? Is there an easy place to<br>
> see a list of eFixes to see if this has come up? I know it’s very<br>
> similar to a problem that happened I believe it was after 5.0.2.2<br>
> and Linux 3.10.0-957.19.1, but that was fixed already in 5.0.3.x.<br>
><br>
> Below is a sample of the crash output:<br>
><br>
> [ 156.733477] kernel BUG at mm/slub.c:3772! [ 156.734212] invalid<br>
> opcode: 0000 [#1] SMP [ 156.735017] Modules linked in: ebtable_nat<br>
> ebtable_filter ebtable_broute bridge stp llc ebtables mmfs26(OE)<br>
> mmfslinux(OE) tracedev(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE)<br>
> iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE)<br>
> mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) ip6table_nat nf_nat_ipv6<br>
> ip6table_mangle ip6table_raw nf_conntrack_ipv6 nf_defrag_ipv6<br>
> ip6table_filter ip6_tables iptable_nat nf_nat_ipv4 nf_nat<br>
> iptable_mangle iptable_raw nf_conntrack_ipv4 nf_defrag_ipv4<br>
> xt_comment xt_multiport xt_conntrack nf_conntrack iptable_filter<br>
> iptable_security nfit libnvdimm ppdev iosf_mbi crc32_pclmul<br>
> ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper<br>
> ablk_helper sg joydev pcspkr cryptd parport_pc parport i2c_piix4<br>
> virtio_balloon knem(OE) binfmt_misc ip_tables xfs libcrc32c<br>
> mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) sr_mod cdrom ata_generic<br>
> pata_acpi virtio_console virtio_net virtio_blk crct10dif_pclmul<br>
> crct10dif_common mlx5_core(OE) mlxfw(OE) crc32c_intel ptp pps_core<br>
> devlink ata_piix serio_raw mlx_compat(OE) libata virtio_pci floppy<br>
> virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod [<br>
> 156.754814] CPU: 3 PID: 11826 Comm: request_handle* Tainted: G OE<br>
> ------------ 3.10.0-1062.9.1.el7.x86_64 #1 [ 156.756782]<br>
> Hardware name: Red Hat KVM, BIOS 1.11.0-2.el7 04/01/2014 [<br>
> 156.757978] task: ffff8aeca5bf8000 ti: ffff8ae9f7a24000 task.ti:<br>
> ffff8ae9f7a24000 [ 156.759326] RIP: 0010:[<ffffffffbbe23dec>]<br>
> [<ffffffffbbe23dec>] kfree+0x13c/0x140 [ 156.760749] RSP:<br>
> 0018:ffff8ae9f7a27278 EFLAGS: 00010246 [ 156.761717] RAX:<br>
> 001fffff00000400 RBX: ffffffffbc6974bf RCX: ffffa74dc1bcfb60 [<br>
> 156.763030] RDX: 001fffff00000000 RSI: ffff8aed90fc6500 RDI:<br>
> ffffffffbc6974bf [ 156.764321] RBP: ffff8ae9f7a27290 R08:<br>
> 0000000000000014 R09: 0000000000000003 [ 156.765612] R10:<br>
> 0000000000000048 R11: ffffdb5a82d125c0 R12: ffffa74dc4fd36c0 [<br>
> 156.766938] R13: ffffffffc0a1c562 R14: ffff8ae9f7a272f8 R15:<br>
> ffff8ae9f7a27938 [ 156.768229] FS: 00007f8ffff05700(0000)<br>
> GS:ffff8aedbfd80000(0000) knlGS:0000000000000000 [ 156.769708] CS:<br>
> 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 156.770754] CR2:<br>
> 000055963330e2b0 CR3: 0000000325ad2000 CR4: 00000000003606e0 [<br>
> 156.772076] DR0: 0000000000000000 DR1: 0000000000000000 DR2:<br>
> 0000000000000000 [ 156.773367] DR3: 0000000000000000 DR6:<br>
> 00000000fffe0ff0 DR7: 0000000000000400 [ 156.774663] Call Trace: [<br>
> 156.775154] [<ffffffffc0a1c562>]<br>
> cxiInitInodeSecurityCleanup+0x12/0x20 [mmfslinux] [ 156.776568]<br>
> [<ffffffffc0b50562>]<br>
> _Z17newInodeInitLinuxP15KernelOperationP13gpfsVfsData_tPP8OpenFilePPvP<br>
P10gpfsNode_tP7FileUIDS6_N5LkObj12LockModeEnumE+0x152/0x290<br>
><br>
><br>
[mmfs26]<br>
> [ 156.779378] [<ffffffffc0b5cdfa>]<br>
> _Z9gpfsMkdirP13gpfsVfsData_tP15KernelOperationP9cxiNode_tPPvPS4_PyS5_P<br>
cjjjP10ext_cred_t+0x46a/0x7e0<br>
><br>
><br>
[mmfs26]<br>
> [ 156.781689] [<ffffffffc0bdb928>] ?<br>
> _ZN14BaseMutexClass15releaseLockHeldEP16KernelSynchState+0x18/0x130<br>
><br>
><br>
[mmfs26]<br>
> [ 156.783565] [<ffffffffc0c3db2d>]<br>
> _ZL21pcacheHandleCacheMissP13gpfsVfsData_tP15KernelOperationP10gpfsNod<br>
e_tPvPcPyP12pCacheResp_tPS5_PS4_PjSA_j+0x4bd/0x760<br>
><br>
><br>
[mmfs26]<br>
> [ 156.786228] [<ffffffffc0c40675>]<br>
> _Z12pcacheLookupP13gpfsVfsData_tP15KernelOperationP10gpfsNode_tPvPcP7F<br>
ilesetjjjPS5_PS4_PyPjS9_+0x1ff5/0x21a0<br>
><br>
><br>
[mmfs26]<br>
> [ 156.788681] [<ffffffffc0c023ef>] ?<br>
> _Z15findFilesetByIdP15KernelOperationjjPP7Filesetj+0x4f/0xa0<br>
> [mmfs26] [ 156.790448] [<ffffffffc0b6d59c>]<br>
> _Z10gpfsLookupP13gpfsVfsData_tPvP9cxiNode_tS1_S1_PcjPS1_PS3_PyP10cxiVa<br>
ttr_tPjP10ext_cred_tjS5_PiS4_SD_+0x65c/0xad0<br>
><br>
><br>
[mmfs26]<br>
> [ 156.793032] [<ffffffffc0b8b022>] ?<br>
> _Z33gpfsIsCifsBypassTraversalCheckingv+0xe2/0x130 [mmfs26] [<br>
> 156.794588] [<ffffffffc0a36d96>] gpfs_i_lookup+0x2e6/0x5a0<br>
> [mmfslinux] [ 156.795838] [<ffffffffc0b6cf40>] ?<br>
> _Z8gpfsLinkP13gpfsVfsData_tP9cxiNode_tS2_PvPcjjP10ext_cred_t+0x6c0/0x6<br>
c0<br>
><br>
><br>
[mmfs26]<br>
> [ 156.797753] [<ffffffffbbe65d52>] ? __d_alloc+0x122/0x180 [<br>
> 156.798763] [<ffffffffbbe65e10>] ? d_alloc+0x60/0x70 [<br>
> 156.799700] [<ffffffffbbe556d3>] lookup_real+0x23/0x60 [<br>
> 156.800651] [<ffffffffbbe560f2>] __lookup_hash+0x42/0x60 [<br>
> 156.801675] [<ffffffffbc377874>] lookup_slow+0x42/0xa7 [<br>
> 156.802634] [<ffffffffbbe5ac3f>] link_path_walk+0x80f/0x8b0 [<br>
> 156.803666] [<ffffffffbbe5ae4a>] path_lookupat+0x7a/0x8b0 [<br>
> 156.804690] [<ffffffffbbdcd2fe>] ? lru_cache_add+0xe/0x10 [<br>
> 156.805690] [<ffffffffbbe24ef5>] ? kmem_cache_alloc+0x35/0x1f0 [<br>
> 156.806766] [<ffffffffbbe5c45f>] ? getname_flags+0x4f/0x1a0 [<br>
> 156.807817] [<ffffffffbbe5b6ab>] filename_lookup+0x2b/0xc0 [<br>
> 156.808834] [<ffffffffbbe5d5f7>] user_path_at_empty+0x67/0xc0 [<br>
> 156.809923] [<ffffffffbbdf3ecd>] ? handle_mm_fault+0x39d/0x9b0 [<br>
> 156.811017] [<ffffffffbbe5d661>] user_path_at+0x11/0x20 [<br>
> 156.811983] [<ffffffffbbe50343>] vfs_fstatat+0x63/0xc0 [<br>
> 156.812951] [<ffffffffbbe506fe>] SYSC_newstat+0x2e/0x60 [<br>
> 156.813931] [<ffffffffbc388a26>] ? trace_do_page_fault+0x56/0x150<br>
> [ 156.815050] [<ffffffffbbe50bbe>] SyS_newstat+0xe/0x10 [<br>
> 156.816010] [<ffffffffbc38dede>] system_call_fastpath+0x25/0x2a [<br>
> 156.817104] Code: 49 8b 03 31 f6 f6 c4 40 74 04 41 8b 73 68 4c 89<br>
> df e8 89 2f fa ff eb 84 4c 8b 58 30 48 8b 10 80 e6 80 4c 0f 44 d8<br>
> e9 28 ff ff ff <0f> 0b 66 90 0f 1f 44 00 00 55 48 89 e5 41 57 41 56<br>
> 41 55 41 54 [ 156.822192] RIP [<ffffffffbbe23dec>]<br>
> kfree+0x13c/0x140 [ 156.823180] RSP <ffff8ae9f7a27278> [<br>
> 156.823872] ---[ end trace 142960be4a4feed8 ]--- [ 156.824806]<br>
> Kernel panic - not syncing: Fatal exception [ 156.826475] Kernel<br>
> Offset: 0x3ac00000 from 0xffffffff81000000 (relocation range:<br>
> 0xffffffff80000000-0xffffffffbfffffff)<br>
><br>
> -- ____ || \\UTGERS,<br>
> |---------------------------*O*--------------------------- ||_//<br>
> the State | Ryan Novosielski - novosirj@rutgers.edu || \\<br>
> University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS<br>
> Campus || \\ of NJ | Office of Advanced Research Computing -<br>
> MSB C630, Newark `'<br>
><br>
> _______________________________________________ gpfsug-discuss<br>
> mailing list gpfsug-discuss at spectrumscale.org<br>
> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
><br>
><br>
><br>
><br>
><br>
> _______________________________________________ gpfsug-discuss<br>
> mailing list gpfsug-discuss at spectrumscale.org<br>
> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
><br>
<br>
- --<br>
____<br>
|| \\UTGERS, |----------------------*O*------------------------<br>
||_// the State | Ryan Novosielski - novosirj@rutgers.edu<br>
|| \\ University | Sr. Technologist - 973/972.0922 ~*~ RBHS Campus<br>
|| \\ of NJ | Office of Advanced Res. Comp. - MSB C630, Newark<br>
`'<br>
-----BEGIN PGP SIGNATURE-----<br>
<br>
iF0EARECAB0WIQST3OUUqPn4dxGCSm6Zv6Bp0RyxvgUCXiDWSgAKCRCZv6Bp0Ryx<br>
vpCsAKCQ2ykmeycbOVrHTGaFqb2SsU26NwCg3YyYi4Jy2d+xZjJkE6Vfht8O8gM=<br>
=9rKb<br>
-----END PGP SIGNATURE-----<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></div>
</blockquote>
<div dir="ltr"> </div>
</div>
<br>
<span>_______________________________________________</span><br>
<span>gpfsug-discuss mailing list</span><br>
<span>gpfsug-discuss at spectrumscale.org</span><br>
<span>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span><br>
</div>
</blockquote>
</div>
</body>
</html>