[gpfsug-discuss] /sbin/rmmod mmfs26 hangs on mmshutdown
Sven Oehme
oehmes at gmail.com
Thu Jul 12 14:40:15 BST 2018
if that happens it would be interesting what top reports
start top in a large resolution window (like 330x80) , press shift-H , this
will break it down per Thread, also press 1 to have a list of each cpu
individually and see if you can either spot one core on the top list with
0% idle or on the thread list on the bottom if any of the threads run at
100% core speed.
attached is a screenshot which columns to look at , this system is idle, so
nothing to see, just to show you where to look
does this machine by any chance has either large maxfilestochache or is a
token server ?
[image: image.png]
sven
On Thu, Jul 12, 2018 at 6:30 AM Billich Heinrich Rainer (PSI) <
heiner.billich at psi.ch> wrote:
> Hello Sven,
>
>
>
> Thank you. I did enable numaMemorInterleave but the issues stays.
>
>
>
> In the meantime I switched to version 5.0.0-2 just to see if it’s version
> dependent – it’s not. All gpfs filesystems are unmounted when this happens.
>
>
>
> At shutdown I often need to do a hard reset to force a reboot – o.k., I
> never waited more than 5 minutes once I saw a hang, maybe it would recover
> after some more time.
>
>
>
> ‘rmmod mmfs26’ doesn’t hang all the times, maybe at every other shutdown
> or mmstartup/mmshutdown cycle. While rmmod hangs the system seems slow,
> command like ‘ps -efH’ or ‘history’ take a long time and some mm commands
> just block, a few times the system gets completely inaccessible.
>
>
>
> I’ll reinstall the systems and move back to 4.2.3-8 and see if this is a
> stable configuration to start from an to rule out any hardware/BIOS issues.
>
>
>
> I append output from numactl -H below.
>
>
>
> Cheers,
>
>
>
> Heiner
>
>
>
> Test with 5.0.0-2
>
>
>
> [root at xbl-ces-2 ~]# numactl -H
>
> available: 2 nodes (0-1)
>
> node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 36 37 38 39 40 41
> 42 43 44 45 46 47 48 49 50 51 52 53
>
> node 0 size: 130942 MB
>
> node 0 free: 60295 MB
>
> node 1 cpus: 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 54 55
> 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71
>
> node 1 size: 131072 MB
>
> node 1 free: 60042 MB
>
> node distances:
>
> node 0 1
>
> 0: 10 21
>
> 1: 21 10
>
>
>
> [root at xbl-ces-2 ~]# mmdiag --config | grep numaM
>
> ! numaMemoryInterleave yes
>
>
>
> # cat /proc/cmdline
>
> BOOT_IMAGE=/vmlinuz-3.10.0-693.17.1.el7.x86_64
> root=/dev/mapper/vg_root-lv_root ro crashkernel=auto rd.lvm.lv=vg_root/lv_root
> console=tty0 console=ttyS0,115200 nosmap
>
>
>
>
>
> Example output of ps -efH during mmshutdown when rmmod did hang (last
> line) This is with 5.0.0-2. As I see all gpfs processe already terminated,
> just
>
>
>
> root 1 0 0 14:30 ? 00:00:10 /usr/lib/systemd/systemd
> --switched-root --system --deserialize 21
>
> root 1035 1 0 14:30 ? 00:00:02
> /usr/lib/systemd/systemd-journald
>
> root 1055 1 0 14:30 ? 00:00:00 /usr/sbin/lvmetad -f
>
> root 1072 1 0 14:30 ? 00:00:11
> /usr/lib/systemd/systemd-udevd
>
> root 1478 1 0 14:31 ? 00:00:00 /usr/sbin/sssd -i -f
>
> root 1484 1478 0 14:31 ? 00:00:00
> /usr/libexec/sssd/sssd_be --domain D.PSI.CH --uid 0 --gid 0
> --debug-to-files
>
> root 1486 1478 0 14:31 ? 00:00:00
> /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --debug-to-files
>
> root 1487 1478 0 14:31 ? 00:00:00
> /usr/libexec/sssd/sssd_pam --uid 0 --gid 0 --debug-to-files
>
> root 1479 1 0 14:31 ? 00:00:00 /usr/sbin/rasdaemon -f -r
>
> root 1482 1 0 14:31 ? 00:00:04 /usr/sbin/irqbalance
> --foreground
>
> dbus 1483 1 0 14:31 ? 00:00:00 /bin/dbus-daemon
> --system --address=systemd: --nofork --nopidfile --systemd-activation
>
> root 1496 1 0 14:31 ? 00:00:00 /usr/sbin/smartd -n -q
> never
>
> root 1498 1 0 14:31 ? 00:00:00 /usr/sbin/gssproxy -D
>
> nscd 1507 1 0 14:31 ? 00:00:01 /usr/sbin/nscd
>
> nrpe 1526 1 0 14:31 ? 00:00:00 /usr/sbin/nrpe -c
> /etc/nagios/nrpe.cfg -d
>
> root 1531 1 0 14:31 ? 00:00:00
> /usr/lib/systemd/systemd-logind
>
> root 1533 1 0 14:31 ? 00:00:00 /usr/sbin/rpc.gssd
>
> root 1803 1 0 14:31 ttyS0 00:00:00 /sbin/agetty --keep-baud
> 115200 38400 9600 ttyS0 vt220
>
> root 1804 1 0 14:31 tty1 00:00:00 /sbin/agetty --noclear
> tty1 linux
>
> root 2405 1 0 14:32 ? 00:00:00 /sbin/dhclient -q -cf
> /etc/dhcp/dhclient-ib0.conf -lf /var/lib/dhclient/dhclient--ib0.l
>
> root 2461 1 0 14:32 ? 00:00:00 /usr/sbin/sshd -D
>
> root 11561 2461 0 14:35 ? 00:00:00 sshd: root at pts/0
>
> root 11565 11561 0 14:35 pts/0 00:00:00 -bash
>
> root 16024 11565 0 14:50 pts/0 00:00:05 ps -efH
>
> root 11609 2461 0 14:35 ? 00:00:00 sshd: root at pts/1
>
> root 11644 11609 0 14:35 pts/1 00:00:00 -bash
>
> root 2718 1 0 14:32 ? 00:00:00 /usr/lpp/mmfs/bin/mmksh
> /usr/lpp/mmfs/bin/mmccrmonitor 15 0 no
>
> root 2758 1 0 14:32 ? 00:00:00
> /usr/libexec/postfix/master -w
>
> postfix 2785 2758 0 14:32 ? 00:00:00 pickup -l -t unix -u
>
> postfix 2786 2758 0 14:32 ? 00:00:00 qmgr -l -t unix -u
>
> root 3174 1 0 14:32 ? 00:00:00 /usr/sbin/crond -n
>
> ntp 3179 1 0 14:32 ? 00:00:00 /usr/sbin/ntpd -u
> ntp:ntp -g
>
> root 3915 1 3 14:32 ? 00:00:33 python
> /usr/lpp/mmfs/bin/mmsysmon.py
>
> root 13618 1 0 14:36 ? 00:00:00
> /usr/lpp/mmfs/bin/mmsdrserv 1191 10 10 /var/adm/ras/mmsdrserv.log 8192 yes
> no
>
> root 15936 1 0 14:49 pts/1 00:00:00 /usr/lpp/mmfs/bin/mmksh
> /usr/lpp/mmfs/bin/runmmfs
>
> root 15992 15936 0 14:49 pts/1 00:00:00 /sbin/rmmod mmfs26
>
>
>
> --
>
> Paul Scherrer Institut
>
> Science IT
>
> Heiner Billich
>
> WHGA 106
>
> CH 5232 Villigen PSI
>
> 056 310 36 02
>
> https://www.psi.ch
>
>
>
>
>
> *From: *<gpfsug-discuss-bounces at spectrumscale.org> on behalf of Sven
> Oehme <oehmes at gmail.com>
> *Reply-To: *gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> *Date: *Wednesday 11 July 2018 at 15:47
> *To: *gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> *Subject: *Re: [gpfsug-discuss] /sbin/rmmod mmfs26 hangs on mmshutdown
>
>
>
> Hi,
>
>
>
> what does numactl -H report ?
>
>
>
> also check if this is set to yes :
>
>
>
> root at fab3a:~# mmlsconfig numaMemoryInterleave
>
> numaMemoryInterleave yes
>
>
>
> Sven
>
>
>
> On Wed, Jul 11, 2018 at 6:40 AM Billich Heinrich Rainer (PSI) <
> heiner.billich at psi.ch> wrote:
>
> Hello,
>
>
>
> I have two nodes which hang on ‘mmshutdown’, in detail the command
> ‘/sbin/rmmod mmfs26’ hangs. I get kernel messages which I append below. I
> wonder if this looks familiar to somebody? Is it a known bug? I can avoid
> the issue if I reduce pagepool from 128G to 64G.
>
>
>
> Running ‘systemctl stop gpfs’ shows the same issue. It forcefully
> terminates after a while, but ‘rmmod’ stays stuck.
>
>
>
> Two functions cxiReleaseAndForgetPages and put_page seem to be involved,
> the first part of gpfs, the second a kernel call.
>
>
>
> The servers have 256G memory and 72 (virtual) cores each.
>
> I run 5.0.1-1 on RHEL7.4 with kernel 3.10.0-693.17.1.el7.x86_64.
>
>
>
> I can try to switch back to 5.0.0
>
>
>
> Thank you & kind regards,
>
>
>
> Heiner
>
>
>
>
>
>
>
> Jul 11 14:12:04 node-1.x.y mmremote[1641]: Unloading module mmfs26
>
> Jul 11 14:12:04 node-1.x.y mmsysmon[2440]: [E] Event raised: The Spectrum
> Scale service process not running on this node. Normal operation cannot be
> done
>
> Jul 11 14:12:04 node-1.x.y mmsysmon[2440]: [I] Event raised: The Spectrum
> Scale service process is running
>
> Jul 11 14:12:04 node-1.x.y mmsysmon[2440]: [E] Event raised: The node is
> not able to form a quorum with the other available nodes.
>
> Jul 11 14:12:38 node-1.x.y sshd[2826]: Connection closed by xxx port 52814
> [preauth]
>
>
>
> Jul 11 14:12:41 node-1.x.y kernel: NMI watchdog: BUG: soft lockup - CPU#28
> stuck for 23s! [rmmod:2695]
>
>
>
> Jul 11 14:12:41 node-1.x.y kernel: Modules linked in: mmfs26(OE-)
> mmfslinux(OE) tracedev(OE) tcp_diag inet_diag rdma_ucm(OE) ib_ucm(OE)
> rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_uverbs(OE) ib_umad(OE)
> mlx5_fpga_tools(OE) mlx5_ib(OE) mlx5_core(OE) mlxfw(OE) mlx4_en(OE)
> mlx4_ib(OE) ib_core(OE) vfat fat ext4 sb_edac edac_core intel_powerclamp
> coretemp intel_rapl iosf_mbi mbcache jbd2 kvm irqbypass crc32_pclmul
> ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd
> iTCO_wdt iTCO_vendor_support ipmi_ssif pcc_cpufreq hpilo ipmi_si sg hpwdt
> pcspkr i2c_i801 lpc_ich ipmi_devintf wmi ioatdma shpchp ipmi_msghandler
> acpi_power_meter binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc
> ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200
> i2c_algo_bit drm_kms_helper syscopyarea sysfillrect
>
> Jul 11 14:12:41 node-1.x.y kernel: sysimgblt fb_sys_fops ttm ixgbe
> mlx4_core(OE) crct10dif_pclmul mdio mlx_compat(OE) crct10dif_common drm ptp
> crc32c_intel devlink hpsa pps_core i2c_core scsi_transport_sas dca
> dm_mirror dm_region_hash dm_log dm_mod [last unloaded: tracedev]
>
> Jul 11 14:12:41 node-1.x.y kernel: CPU: 28 PID: 2695 Comm: rmmod Tainted:
> G W OEL ------------ 3.10.0-693.17.1.el7.x86_64 #1
>
> Jul 11 14:12:41 node-1.x.y kernel: Hardware name: HP ProLiant DL380
> Gen9/ProLiant DL380 Gen9, BIOS P89 01/22/2018
>
> Jul 11 14:12:41 node-1.x.y kernel: task: ffff8808c4814f10 ti:
> ffff881619778000 task.ti: ffff881619778000
>
> Jul 11 14:12:41 node-1.x.y kernel: RIP: 0010:[<ffffffff816a2970>]
> [<ffffffff816a2970>] put_compound_page+0xc3/0x174
>
> Jul 11 14:12:41 node-1.x.y kernel: RSP: 0018:ffff88161977bd50 EFLAGS:
> 00000246
>
> Jul 11 14:12:41 node-1.x.y kernel: RAX: 0000000000000283 RBX:
> 00000000fae3d201 RCX: 0000000000000284
>
> Jul 11 14:12:41 node-1.x.y kernel: RDX: 0000000000000283 RSI:
> 0000000000000246 RDI: ffffea003d478000
>
> Jul 11 14:12:41 node-1.x.y kernel: RBP: ffff88161977bd68 R08:
> ffff881ffae3d1e0 R09: 0000000180800059
>
> Jul 11 14:12:41 node-1.x.y kernel: R10: 00000000fae3d201 R11:
> ffffea007feb8f40 R12: 00000000fae3d201
>
> Jul 11 14:12:41 node-1.x.y kernel: R13: ffff88161977bd40 R14:
> 0000000000000000 R15: ffff88161977bd40
>
> Jul 11 14:12:41 node-1.x.y kernel: FS: 00007f81a1db0740(0000)
> GS:ffff883ffee80000(0000) knlGS:0000000000000000
>
> Jul 11 14:12:41 node-1.x.y kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
> 0000000080050033
>
> Jul 11 14:12:41 node-1.x.y kernel: CR2: 00007fa96e38f980 CR3:
> 0000000c36b2c000 CR4: 00000000001607e0
>
> Jul 11 14:12:41 node-1.x.y kernel: DR0: 0000000000000000 DR1:
> 0000000000000000 DR2: 0000000000000000
>
> Jul 11 14:12:41 node-1.x.y kernel: DR3: 0000000000000000 DR6:
> 00000000fffe0ff0 DR7: 0000000000000400
>
>
>
> Jul 11 14:12:41 node-1.x.y kernel: Call Trace:
>
> Jul 11 14:12:41 node-1.x.y kernel: [<ffffffff81192275>] put_page+0x45/0x50
>
> Jul 11 14:12:41 node-1.x.y kernel: [<ffffffffc08e3562>]
> cxiReleaseAndForgetPages+0xb2/0x1c0 [mmfslinux]
>
> Jul 11 14:12:41 node-1.x.y kernel: [<ffffffffc08e3ae5>]
> cxiDeallocPageList+0x45/0x110 [mmfslinux]
>
> Jul 11 14:12:41 node-1.x.y kernel: [<ffffffff811e0b02>] ?
> kmem_cache_free+0x1e2/0x200
>
> Jul 11 14:12:41 node-1.x.y kernel: [<ffffffffc08e3cda>]
> cxiFreeSharedMemory+0x12a/0x130 [mmfslinux]
>
> Jul 11 14:12:41 node-1.x.y kernel: [<ffffffffc0c70c12>]
> kxFreeAllSharedMemory+0xe2/0x160 [mmfs26]
>
> Jul 11 14:12:41 node-1.x.y kernel: [<ffffffffc0c5bd15>] mmfs+0xc85/0xca0
> [mmfs26]
>
> Jul 11 14:12:41 node-1.x.y kernel: [<ffffffffc08c8f16>]
> gpfs_clean+0x26/0x30 [mmfslinux]
>
> Jul 11 14:12:41 node-1.x.y kernel: [<ffffffffc0da5565>]
> cleanup_module+0x25/0x30 [mmfs26]
>
> Jul 11 14:12:41 node-1.x.y kernel: [<ffffffff8110044b>]
> SyS_delete_module+0x19b/0x300
>
> Jul 11 14:12:41 node-1.x.y kernel: [<ffffffff816b89fd>]
> system_call_fastpath+0x16/0x1b
>
> Jul 11 14:12:41 node-1.x.y kernel: Code: d1 00 00 00 4c 89 e7 e8 3a ff ff
> ff e9 c4 00 00 00 4c 39 e3 74 c1 41 8b 54 24 1c 85 d2 74 b8 8d 4a 01 89 d0
> f0 41 0f b1 4c 24 1c <39> c2 74 04 89 c2 eb e8 e8 f3 f0 ae ff 49 89 c5 f0
> 41 0f ba 2c
>
>
>
> Jul 11 14:13:23 node-1.x.y systemd[1]: gpfs.service stopping timed out.
> Terminating.
>
>
>
> Jul 11 14:13:27 node-1.x.y kernel: NMI watchdog: BUG: soft lockup - CPU#28
> stuck for 21s! [rmmod:2695]
>
>
>
> Jul 11 14:13:27 node-1.x.y kernel: Modules linked in: mmfs26(OE-)
> mmfslinux(OE) tracedev(OE) tcp_diag inet_diag rdma_ucm(OE) ib_ucm(OE)
> rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_uverbs(OE) ib_umad(OE)
> mlx5_fpga_tools(OE) mlx5_ib(OE) mlx5_core(OE) mlxfw(OE) mlx4_en(OE)
> mlx4_ib(OE) ib_core(OE) vfat fat ext4 sb_edac edac_core intel_powerclamp
> coretemp intel_rapl iosf_mbi mbcache jbd2 kvm irqbypass crc32_pclmul
> ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd
> iTCO_wdt iTCO_vendor_support ipmi_ssif pcc_cpufreq hpilo ipmi_si sg hpwdt
> pcspkr i2c_i801 lpc_ich ipmi_devintf wmi ioatdma shpchp ipmi_msghandler
>
> Jul 11 14:13:27 node-1.x.y kernel: INFO: rcu_sched detected stalls on
> CPUs/tasks:
>
> Jul 11 14:13:27 node-1.x.y kernel: {
>
> Jul 11 14:13:27 node-1.x.y kernel: 28
>
> Jul 11 14:13:27 node-1.x.y kernel: }
>
> Jul 11 14:13:27 node-1.x.y kernel: (detected by 17, t=60002 jiffies,
> g=267734, c=267733, q=36089)
>
> Jul 11 14:13:27 node-1.x.y kernel: Task dump for CPU 28:
>
> Jul 11 14:13:27 node-1.x.y kernel: rmmod R
>
> Jul 11 14:13:27 node-1.x.y kernel: running task
>
> Jul 11 14:13:27 node-1.x.y kernel: 0 2695 2642 0x00000008
>
> Jul 11 14:13:27 node-1.x.y kernel: Call Trace:
>
> Jul 11 14:13:27 node-1.x.y kernel:
>
> Jul 11 14:13:27 node-1.x.y kernel: [<ffffffff811dea1c>] ?
> __free_slab+0xdc/0x200
>
> Jul 11 14:13:27 node-1.x.y kernel:
>
> Jul 11 14:13:27 node-1.x.y kernel: [<ffffffff816a28ad>] ?
> __put_compound_page+0x22/0x22
>
> Jul 11 14:13:27 node-1.x.y kernel:
>
> Jul 11 14:13:27 node-1.x.y kernel: [<ffffffff81192275>] ?
> put_page+0x45/0x50
>
> Jul 11 14:13:27 node-1.x.y kernel:
>
> Jul 11 14:13:27 node-1.x.y kernel: [<ffffffffc08e3562>] ?
> cxiReleaseAndForgetPages+0xb2/0x1c0 [mmfslinux]
>
> Jul 11 14:13:27 node-1.x.y kernel:
>
> Jul 11 14:13:27 node-1.x.y kernel: [<ffffffffc08e3ae5>] ?
> cxiDeallocPageList+0x45/0x110 [mmfslinux]
>
> Jul 11 14:13:27 node-1.x.y kernel:
>
> Jul 11 14:13:27 node-1.x.y kernel: [<ffffffffc08e3cda>] ?
> cxiFreeSharedMemory+0x12a/0x130 [mmfslinux]
>
> Jul 11 14:13:27 node-1.x.y kernel:
>
> Jul 11 14:13:27 node-1.x.y kernel: [<ffffffffc0c70c12>] ?
> kxFreeAllSharedMemory+0xe2/0x160 [mmfs26]
>
> Jul 11 14:13:27 node-1.x.y kernel:
>
> Jul 11 14:13:27 node-1.x.y kernel: [<ffffffffc0c5bd15>] ?
> mmfs+0xc85/0xca0 [mmfs26]
>
> Jul 11 14:13:27 node-1.x.y kernel:
>
> Jul 11 14:13:27 node-1.x.y kernel: [<ffffffffc08c8f16>] ?
> gpfs_clean+0x26/0x30 [mmfslinux]
>
> Jul 11 14:13:27 node-1.x.y kernel:
>
> Jul 11 14:13:27 node-1.x.y kernel: [<ffffffffc0da5565>] ?
> cleanup_module+0x25/0x30 [mmfs26]
>
> Jul 11 14:13:27 node-1.x.y kernel:
>
> Jul 11 14:13:27 node-1.x.y kernel: [<ffffffff8110044b>] ?
> SyS_delete_module+0x19b/0x300
>
> Jul 11 14:13:27 node-1.x.y kernel:
>
> Jul 11 14:13:27 node-1.x.y kernel: [<ffffffff816b89fd>] ?
> system_call_fastpath+0x16/0x1b
>
> Jul 11 14:13:27 node-1.x.y kernel: acpi_power_meter
>
> Jul 11 14:13:27 node-1.x.y kernel: binfmt_misc nfsd auth_rpcgss nfs_acl
> lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif
> crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea
> sysfillrect sysimgblt fb_sys_fops ttm ixgbe mlx4_core(OE) crct10dif_pclmul
> mdio mlx_compat(OE) crct10dif_common drm ptp crc32c_intel devlink hpsa
> pps_core i2c_core scsi_transport_sas dca dm_mirror dm_region_hash dm_log
> dm_mod [last unloaded: tracedev]
>
> Jul 11 14:13:27 node-1.x.y kernel: CPU: 28 PID: 2695 Comm: rmmod Tainted:
> G W OEL ------------ 3.10.0-693.17.1.el7.x86_64 #1
>
> Jul 11 14:13:27 node-1.x.y kernel: Hardware name: HP ProLiant DL380
> Gen9/ProLiant DL380 Gen9, BIOS P89 01/22/2018
>
> Jul 11 14:13:27 node-1.x.y kernel: task: ffff8808c4814f10 ti:
> ffff881619778000 task.ti: ffff881619778000
>
> Jul 11 14:13:27 node-1.x.y kernel: RIP: 0010:[<ffffffff816a28ad>]
> [<ffffffff816a28ad>] __put_compound_page+0x22/0x22
>
> Jul 11 14:13:27 node-1.x.y kernel: RSP: 0018:ffff88161977bd70 EFLAGS:
> 00000282
>
> Jul 11 14:13:27 node-1.x.y kernel: RAX: 002fffff00008010 RBX:
> 0000000000000135 RCX: 00000000000001c1
>
> Jul 11 14:13:27 node-1.x.y kernel: RDX: ffff8814adbbf000 RSI:
> 0000000000000246 RDI: ffffea00650e7040
>
> Jul 11 14:13:27 node-1.x.y kernel: RBP: ffff88161977bd78 R08:
> ffff881ffae3df60 R09: 0000000180800052
>
> Jul 11 14:13:27 node-1.x.y kernel: R10: 00000000fae3db01 R11:
> ffffea007feb8f40 R12: ffff881ffae3df60
>
> Jul 11 14:13:27 node-1.x.y kernel: R13: 0000000180800052 R14:
> 00000000fae3db01 R15: ffffea007feb8f40
>
> Jul 11 14:13:27 node-1.x.y kernel: FS: 00007f81a1db0740(0000)
> GS:ffff883ffee80000(0000) knlGS:0000000000000000
>
> Jul 11 14:13:27 node-1.x.y kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
> 0000000080050033
>
> Jul 11 14:13:27 node-1.x.y kernel: CR2: 00007fa96e38f980 CR3:
> 0000000c36b2c000 CR4: 00000000001607e0
>
> Jul 11 14:13:27 node-1.x.y kernel: DR0: 0000000000000000 DR1:
> 0000000000000000 DR2: 0000000000000000
>
> Jul 11 14:13:27 node-1.x.y kernel: DR3: 0000000000000000 DR6:
> 00000000fffe0ff0 DR7: 0000000000000400
>
> Jul 11 14:13:27 node-1.x.y kernel: Call Trace:
>
> Jul 11 14:13:27 node-1.x.y kernel: [<ffffffff81192275>] ?
> put_page+0x45/0x50
>
> Jul 11 14:13:27 node-1.x.y kernel: [<ffffffffc08e3562>]
> cxiReleaseAndForgetPages+0xb2/0x1c0 [mmfslinux]
>
> Jul 11 14:13:27 node-1.x.y kernel: [<ffffffffc08e3ae5>]
> cxiDeallocPageList+0x45/0x110 [mmfslinux]
>
> Jul 11 14:13:27 node-1.x.y kernel: [<ffffffffc08e3cda>]
> cxiFreeSharedMemory+0x12a/0x130 [mmfslinux]
>
> Jul 11 14:13:27 node-1.x.y kernel: [<ffffffffc0c70c12>]
> kxFreeAllSharedMemory+0xe2/0x160 [mmfs26]
>
> Jul 11 14:13:27 node-1.x.y kernel: [<ffffffffc0c5bd15>] mmfs+0xc85/0xca0
> [mmfs26]
>
> Jul 11 14:13:27 node-1.x.y kernel: [<ffffffffc08c8f16>]
> gpfs_clean+0x26/0x30 [mmfslinux]
>
> Jul 11 14:13:27 node-1.x.y kernel: [<ffffffffc0da5565>]
> cleanup_module+0x25/0x30 [mmfs26]
>
> Jul 11 14:13:27 node-1.x.y kernel: [<ffffffff8110044b>]
> SyS_delete_module+0x19b/0x300
>
> Jul 11 14:13:27 node-1.x.y kernel: [<ffffffff816b89fd>]
> system_call_fastpath+0x16/0x1b
>
> Jul 11 14:13:27 node-1.x.y kernel: Code: c0 0f 95 c0 0f b6 c0 5d c3 0f 1f
> 44 00 00 55 48 89 e5 53 48 8b 07 48 89 fb a8 20 74 05 e8 0c f8 ae ff 48 89
> df ff 53 60 5b 5d c3 <0f> 1f 44 00 00 55 48 89 e5 41 55 41 54 53 48 8b 07
> 48 89 fb f6
>
>
>
> --
>
> Paul Scherrer Institut
>
> Science IT
>
> Heiner Billich
>
> WHGA 106
>
> CH 5232 Villigen PSI
>
> 056 310 36 02
>
> https://www.psi.ch
>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20180712/71701891/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 643176 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20180712/71701891/attachment-0002.png>
More information about the gpfsug-discuss
mailing list