<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hello Sven,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thank you. I did enable numaMemorInterleave but the issues stays.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In the meantime I switched to version 5.0.0-2 just to see if it’s version dependent – it’s not. All gpfs filesystems are unmounted when this happens.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">At shutdown I often need to do a hard reset to force a reboot – o.k., I never waited more than 5 minutes once I saw a hang, maybe it would recover after some more time.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">‘rmmod mmfs26’ doesn’t hang all the times, maybe at every other shutdown or mmstartup/mmshutdown cycle. While rmmod hangs the system seems slow, command like ‘ps -efH’ or ‘history’ take a long time and some mm commands just block, a few
times the system gets completely inaccessible. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’ll reinstall the systems and move back to 4.2.3-8 and see if this is a stable configuration to start from an to rule out any hardware/BIOS issues.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I append output from numactl -H below.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Cheers,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Heiner<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Test with 5.0.0-2<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[root@xbl-ces-2 ~]# numactl -H<o:p></o:p></p>
<p class="MsoNormal">available: 2 nodes (0-1)<o:p></o:p></p>
<p class="MsoNormal">node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53<o:p></o:p></p>
<p class="MsoNormal">node 0 size: 130942 MB<o:p></o:p></p>
<p class="MsoNormal">node 0 free: 60295 MB<o:p></o:p></p>
<p class="MsoNormal">node 1 cpus: 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71<o:p></o:p></p>
<p class="MsoNormal">node 1 size: 131072 MB<o:p></o:p></p>
<p class="MsoNormal">node 1 free: 60042 MB<o:p></o:p></p>
<p class="MsoNormal">node distances:<o:p></o:p></p>
<p class="MsoNormal">node 0 1<o:p></o:p></p>
<p class="MsoNormal"> 0: 10 21<o:p></o:p></p>
<p class="MsoNormal"> 1: 21 10<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[root@xbl-ces-2 ~]# mmdiag --config | grep numaM<o:p></o:p></p>
<p class="MsoNormal">! numaMemoryInterleave yes<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"># cat /proc/cmdline<o:p></o:p></p>
<p class="MsoNormal">BOOT_IMAGE=/vmlinuz-3.10.0-693.17.1.el7.x86_64 root=/dev/mapper/vg_root-lv_root ro crashkernel=auto rd.lvm.lv=vg_root/lv_root console=tty0 console=ttyS0,115200 nosmap<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Example output of ps -efH during mmshutdown when rmmod did hang (last line) This is with 5.0.0-2. As I see all gpfs processe already terminated, just
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">root 1 0 0 14:30 ? 00:00:10 /usr/lib/systemd/systemd --switched-root --system --deserialize 21<o:p></o:p></p>
<p class="MsoNormal">root 1035 1 0 14:30 ? 00:00:02 /usr/lib/systemd/systemd-journald<o:p></o:p></p>
<p class="MsoNormal">root 1055 1 0 14:30 ? 00:00:00 /usr/sbin/lvmetad -f<o:p></o:p></p>
<p class="MsoNormal">root 1072 1 0 14:30 ? 00:00:11 /usr/lib/systemd/systemd-udevd<o:p></o:p></p>
<p class="MsoNormal">root 1478 1 0 14:31 ? 00:00:00 /usr/sbin/sssd -i -f<o:p></o:p></p>
<p class="MsoNormal">root 1484 1478 0 14:31 ? 00:00:00 /usr/libexec/sssd/sssd_be --domain D.PSI.CH --uid 0 --gid 0 --debug-to-files<o:p></o:p></p>
<p class="MsoNormal">root 1486 1478 0 14:31 ? 00:00:00 /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --debug-to-files<o:p></o:p></p>
<p class="MsoNormal">root 1487 1478 0 14:31 ? 00:00:00 /usr/libexec/sssd/sssd_pam --uid 0 --gid 0 --debug-to-files<o:p></o:p></p>
<p class="MsoNormal">root 1479 1 0 14:31 ? 00:00:00 /usr/sbin/rasdaemon -f -r<o:p></o:p></p>
<p class="MsoNormal">root 1482 1 0 14:31 ? 00:00:04 /usr/sbin/irqbalance --foreground<o:p></o:p></p>
<p class="MsoNormal">dbus 1483 1 0 14:31 ? 00:00:00 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation<o:p></o:p></p>
<p class="MsoNormal">root 1496 1 0 14:31 ? 00:00:00 /usr/sbin/smartd -n -q never<o:p></o:p></p>
<p class="MsoNormal">root 1498 1 0 14:31 ? 00:00:00 /usr/sbin/gssproxy -D<o:p></o:p></p>
<p class="MsoNormal">nscd 1507 1 0 14:31 ? 00:00:01 /usr/sbin/nscd<o:p></o:p></p>
<p class="MsoNormal">nrpe 1526 1 0 14:31 ? 00:00:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d<o:p></o:p></p>
<p class="MsoNormal">root 1531 1 0 14:31 ? 00:00:00 /usr/lib/systemd/systemd-logind<o:p></o:p></p>
<p class="MsoNormal">root 1533 1 0 14:31 ? 00:00:00 /usr/sbin/rpc.gssd<o:p></o:p></p>
<p class="MsoNormal">root 1803 1 0 14:31 ttyS0 00:00:00 /sbin/agetty --keep-baud 115200 38400 9600 ttyS0 vt220<o:p></o:p></p>
<p class="MsoNormal">root 1804 1 0 14:31 tty1 00:00:00 /sbin/agetty --noclear tty1 linux<o:p></o:p></p>
<p class="MsoNormal">root 2405 1 0 14:32 ? 00:00:00 /sbin/dhclient -q -cf /etc/dhcp/dhclient-ib0.conf -lf /var/lib/dhclient/dhclient--ib0.l<o:p></o:p></p>
<p class="MsoNormal">root 2461 1 0 14:32 ? 00:00:00 /usr/sbin/sshd -D<o:p></o:p></p>
<p class="MsoNormal">root 11561 2461 0 14:35 ? 00:00:00 sshd: root@pts/0<o:p></o:p></p>
<p class="MsoNormal">root 11565 11561 0 14:35 pts/0 00:00:00 -bash<o:p></o:p></p>
<p class="MsoNormal">root 16024 11565 0 14:50 pts/0 00:00:05 ps -efH<o:p></o:p></p>
<p class="MsoNormal">root 11609 2461 0 14:35 ? 00:00:00 sshd: root@pts/1<o:p></o:p></p>
<p class="MsoNormal">root 11644 11609 0 14:35 pts/1 00:00:00 -bash<o:p></o:p></p>
<p class="MsoNormal">root 2718 1 0 14:32 ? 00:00:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15 0 no<o:p></o:p></p>
<p class="MsoNormal">root 2758 1 0 14:32 ? 00:00:00 /usr/libexec/postfix/master -w<o:p></o:p></p>
<p class="MsoNormal">postfix 2785 2758 0 14:32 ? 00:00:00 pickup -l -t unix -u<o:p></o:p></p>
<p class="MsoNormal">postfix 2786 2758 0 14:32 ? 00:00:00 qmgr -l -t unix -u<o:p></o:p></p>
<p class="MsoNormal">root 3174 1 0 14:32 ? 00:00:00 /usr/sbin/crond -n<o:p></o:p></p>
<p class="MsoNormal">ntp 3179 1 0 14:32 ? 00:00:00 /usr/sbin/ntpd -u ntp:ntp -g<o:p></o:p></p>
<p class="MsoNormal">root 3915 1 3 14:32 ? 00:00:33 python /usr/lpp/mmfs/bin/mmsysmon.py<o:p></o:p></p>
<p class="MsoNormal">root 13618 1 0 14:36 ? 00:00:00 /usr/lpp/mmfs/bin/mmsdrserv 1191 10 10 /var/adm/ras/mmsdrserv.log 8192 yes no<o:p></o:p></p>
<p class="MsoNormal">root 15936 1 0 14:49 pts/1 00:00:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/runmmfs<o:p></o:p></p>
<p class="MsoNormal">root 15992 15936 0 14:49 pts/1 00:00:00 /sbin/rmmod mmfs26<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">--<o:p></o:p></p>
<p class="MsoNormal">Paul Scherrer Institut<o:p></o:p></p>
<p class="MsoNormal">Science IT<o:p></o:p></p>
<p class="MsoNormal">Heiner Billich<o:p></o:p></p>
<p class="MsoNormal">WHGA 106<o:p></o:p></p>
<p class="MsoNormal">CH 5232 Villigen PSI<o:p></o:p></p>
<p class="MsoNormal">056 310 36 02<o:p></o:p></p>
<p class="MsoNormal">https://www.psi.ch<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black"><gpfsug-discuss-bounces@spectrumscale.org> on behalf of Sven Oehme <oehmes@gmail.com><br>
<b>Reply-To: </b>gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
<b>Date: </b>Wednesday 11 July 2018 at 15:47<br>
<b>To: </b>gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
<b>Subject: </b>Re: [gpfsug-discuss] /sbin/rmmod mmfs26 hangs on mmshutdown<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Hi, <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">what does numactl -H report ? <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">also check if this is set to yes :<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">root@fab3a:~# mmlsconfig numaMemoryInterleave<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">numaMemoryInterleave yes<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Sven<o:p></o:p></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Wed, Jul 11, 2018 at 6:40 AM Billich Heinrich Rainer (PSI) <<a href="mailto:heiner.billich@psi.ch">heiner.billich@psi.ch</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hello,<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I have two nodes which hang on ‘mmshutdown’, in detail the command ‘/sbin/rmmod mmfs26’ hangs. I get kernel messages which I append below. I wonder if this looks familiar to somebody?
Is it a known bug? I can avoid the issue if I reduce pagepool from 128G to 64G.
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Running ‘systemctl stop gpfs’ shows the same issue. It forcefully terminates after a while, but ‘rmmod’ stays stuck.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Two functions cxiReleaseAndForgetPages and put_page seem to be involved, the first part of gpfs, the second a kernel call.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">The servers have 256G memory and 72 (virtual) cores each.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I run 5.0.1-1 on RHEL7.4 with kernel 3.10.0-693.17.1.el7.x86_64.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I can try to switch back to 5.0.0<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thank you & kind regards,<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Heiner<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:04 node-1.x.y mmremote[1641]: Unloading module mmfs26<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:04 node-1.x.y mmsysmon[2440]: [E] Event raised: The Spectrum Scale service process not running on this node. Normal operation cannot be done<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:04 node-1.x.y mmsysmon[2440]: [I] Event raised: The Spectrum Scale service process is running<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:04 node-1.x.y mmsysmon[2440]: [E] Event raised: The node is not able to form a quorum with the other available nodes.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:38 node-1.x.y sshd[2826]: Connection closed by xxx port 52814 [preauth]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: NMI watchdog: BUG: soft lockup - CPU#28 stuck for 23s! [rmmod:2695]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: Modules linked in: mmfs26(OE-) mmfslinux(OE) tracedev(OE) tcp_diag inet_diag rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE)
ib_uverbs(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx5_ib(OE) mlx5_core(OE) mlxfw(OE) mlx4_en(OE) mlx4_ib(OE) ib_core(OE) vfat fat ext4 sb_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi mbcache jbd2 kvm irqbypass crc32_pclmul ghash_clmulni_intel
aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support ipmi_ssif pcc_cpufreq hpilo ipmi_si sg hpwdt pcspkr i2c_i801 lpc_ich ipmi_devintf wmi ioatdma shpchp ipmi_msghandler acpi_power_meter binfmt_misc nfsd auth_rpcgss nfs_acl
lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: sysimgblt fb_sys_fops ttm ixgbe mlx4_core(OE) crct10dif_pclmul mdio mlx_compat(OE) crct10dif_common drm ptp crc32c_intel devlink hpsa pps_core
i2c_core scsi_transport_sas dca dm_mirror dm_region_hash dm_log dm_mod [last unloaded: tracedev]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: CPU: 28 PID: 2695 Comm: rmmod Tainted: G W OEL ------------ 3.10.0-693.17.1.el7.x86_64 #1<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 01/22/2018<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: task: ffff8808c4814f10 ti: ffff881619778000 task.ti: ffff881619778000<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: RIP: 0010:[<ffffffff816a2970>] [<ffffffff816a2970>] put_compound_page+0xc3/0x174<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: RSP: 0018:ffff88161977bd50 EFLAGS: 00000246<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: RAX: 0000000000000283 RBX: 00000000fae3d201 RCX: 0000000000000284<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: RDX: 0000000000000283 RSI: 0000000000000246 RDI: ffffea003d478000<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: RBP: ffff88161977bd68 R08: ffff881ffae3d1e0 R09: 0000000180800059<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: R10: 00000000fae3d201 R11: ffffea007feb8f40 R12: 00000000fae3d201<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: R13: ffff88161977bd40 R14: 0000000000000000 R15: ffff88161977bd40<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: FS: 00007f81a1db0740(0000) GS:ffff883ffee80000(0000) knlGS:0000000000000000<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: CR2: 00007fa96e38f980 CR3: 0000000c36b2c000 CR4: 00000000001607e0<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: Call Trace:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: [<ffffffff81192275>] put_page+0x45/0x50<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: [<ffffffffc08e3562>] cxiReleaseAndForgetPages+0xb2/0x1c0 [mmfslinux]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: [<ffffffffc08e3ae5>] cxiDeallocPageList+0x45/0x110 [mmfslinux]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: [<ffffffff811e0b02>] ? kmem_cache_free+0x1e2/0x200<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: [<ffffffffc08e3cda>] cxiFreeSharedMemory+0x12a/0x130 [mmfslinux]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: [<ffffffffc0c70c12>] kxFreeAllSharedMemory+0xe2/0x160 [mmfs26]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: [<ffffffffc0c5bd15>] mmfs+0xc85/0xca0 [mmfs26]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: [<ffffffffc08c8f16>] gpfs_clean+0x26/0x30 [mmfslinux]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: [<ffffffffc0da5565>] cleanup_module+0x25/0x30 [mmfs26]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: [<ffffffff8110044b>] SyS_delete_module+0x19b/0x300<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: [<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:12:41 node-1.x.y kernel: Code: d1 00 00 00 4c 89 e7 e8 3a ff ff ff e9 c4 00 00 00 4c 39 e3 74 c1 41 8b 54 24 1c 85 d2 74 b8 8d 4a 01 89 d0 f0 41 0f b1 4c 24 1c <39> c2
74 04 89 c2 eb e8 e8 f3 f0 ae ff 49 89 c5 f0 41 0f ba 2c<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:23 node-1.x.y systemd[1]: gpfs.service stopping timed out. Terminating.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: NMI watchdog: BUG: soft lockup - CPU#28 stuck for 21s! [rmmod:2695]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: Modules linked in: mmfs26(OE-) mmfslinux(OE) tracedev(OE) tcp_diag inet_diag rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE)
ib_uverbs(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx5_ib(OE) mlx5_core(OE) mlxfw(OE) mlx4_en(OE) mlx4_ib(OE) ib_core(OE) vfat fat ext4 sb_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi mbcache jbd2 kvm irqbypass crc32_pclmul ghash_clmulni_intel
aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support ipmi_ssif pcc_cpufreq hpilo ipmi_si sg hpwdt pcspkr i2c_i801 lpc_ich ipmi_devintf wmi ioatdma shpchp ipmi_msghandler<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: INFO: rcu_sched detected stalls on CPUs/tasks:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: {<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: 28<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: }<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: (detected by 17, t=60002 jiffies, g=267734, c=267733, q=36089)<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: Task dump for CPU 28:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: rmmod R<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: running task<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: 0 2695 2642 0x00000008<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: Call Trace:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: [<ffffffff811dea1c>] ? __free_slab+0xdc/0x200<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: [<ffffffff816a28ad>] ? __put_compound_page+0x22/0x22<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: [<ffffffff81192275>] ? put_page+0x45/0x50<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: [<ffffffffc08e3562>] ? cxiReleaseAndForgetPages+0xb2/0x1c0 [mmfslinux]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: [<ffffffffc08e3ae5>] ? cxiDeallocPageList+0x45/0x110 [mmfslinux]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: [<ffffffffc08e3cda>] ? cxiFreeSharedMemory+0x12a/0x130 [mmfslinux]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: [<ffffffffc0c70c12>] ? kxFreeAllSharedMemory+0xe2/0x160 [mmfs26]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: [<ffffffffc0c5bd15>] ? mmfs+0xc85/0xca0 [mmfs26]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: [<ffffffffc08c8f16>] ? gpfs_clean+0x26/0x30 [mmfslinux]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: [<ffffffffc0da5565>] ? cleanup_module+0x25/0x30 [mmfs26]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: [<ffffffff8110044b>] ? SyS_delete_module+0x19b/0x300<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: [<ffffffff816b89fd>] ? system_call_fastpath+0x16/0x1b<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: acpi_power_meter<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper
syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ixgbe mlx4_core(OE) crct10dif_pclmul mdio mlx_compat(OE) crct10dif_common drm ptp crc32c_intel devlink hpsa pps_core i2c_core scsi_transport_sas dca dm_mirror dm_region_hash dm_log dm_mod [last unloaded: tracedev]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: CPU: 28 PID: 2695 Comm: rmmod Tainted: G W OEL ------------ 3.10.0-693.17.1.el7.x86_64 #1<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 01/22/2018<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: task: ffff8808c4814f10 ti: ffff881619778000 task.ti: ffff881619778000<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: RIP: 0010:[<ffffffff816a28ad>] [<ffffffff816a28ad>] __put_compound_page+0x22/0x22<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: RSP: 0018:ffff88161977bd70 EFLAGS: 00000282<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: RAX: 002fffff00008010 RBX: 0000000000000135 RCX: 00000000000001c1<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: RDX: ffff8814adbbf000 RSI: 0000000000000246 RDI: ffffea00650e7040<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: RBP: ffff88161977bd78 R08: ffff881ffae3df60 R09: 0000000180800052<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: R10: 00000000fae3db01 R11: ffffea007feb8f40 R12: ffff881ffae3df60<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: R13: 0000000180800052 R14: 00000000fae3db01 R15: ffffea007feb8f40<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: FS: 00007f81a1db0740(0000) GS:ffff883ffee80000(0000) knlGS:0000000000000000<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: CR2: 00007fa96e38f980 CR3: 0000000c36b2c000 CR4: 00000000001607e0<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: Call Trace:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: [<ffffffff81192275>] ? put_page+0x45/0x50<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: [<ffffffffc08e3562>] cxiReleaseAndForgetPages+0xb2/0x1c0 [mmfslinux]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: [<ffffffffc08e3ae5>] cxiDeallocPageList+0x45/0x110 [mmfslinux]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: [<ffffffffc08e3cda>] cxiFreeSharedMemory+0x12a/0x130 [mmfslinux]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: [<ffffffffc0c70c12>] kxFreeAllSharedMemory+0xe2/0x160 [mmfs26]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: [<ffffffffc0c5bd15>] mmfs+0xc85/0xca0 [mmfs26]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: [<ffffffffc08c8f16>] gpfs_clean+0x26/0x30 [mmfslinux]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: [<ffffffffc0da5565>] cleanup_module+0x25/0x30 [mmfs26]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: [<ffffffff8110044b>] SyS_delete_module+0x19b/0x300<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: [<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jul 11 14:13:27 node-1.x.y kernel: Code: c0 0f 95 c0 0f b6 c0 5d c3 0f 1f 44 00 00 55 48 89 e5 53 48 8b 07 48 89 fb a8 20 74 05 e8 0c f8 ae ff 48 89 df ff 53 60 5b 5d c3 <0f> 1f
44 00 00 55 48 89 e5 41 55 41 54 53 48 8b 07 48 89 fb f6<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">--<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Paul Scherrer Institut<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Science IT<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Heiner Billich<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">WHGA 106<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">CH 5232 Villigen PSI<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">056 310 36 02<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><a href="https://www.psi.ch" target="_blank">https://www.psi.ch</a><o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><br>
<br>
<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal">_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><o:p></o:p></p>
</blockquote>
</div>
</div>
</body>
</html>