<html><body><p><font size="2">Renar,</font><br><br><font size="2">With the change below, which is a retrofit of a change deployed in newer kernels, an inconsistency has taken place between the GPFS kernel portability layer and the kernel proper. A known result of that inconsistency is a kernel crash. One known sequence leading to the crash involves the mkdir() call.</font><br><br><font size="2">We are working on an official notification on the issue.</font><br><br><font size="2"> Felipe </font><br><br><font size="2">----<br>Felipe Knop knop@us.ibm.com<br>GPFS Development and Security<br>IBM Systems<br>IBM Building 008<br>2455 South Rd, Poughkeepsie, NY 12601<br>(845) 433-9314 T/L 293-9314<br><br></font><br><br><img width="16" height="16" src="cid:1__=8FBB0E85DFC053C58f9e8a93df938690918c8FB@" border="0" alt="Inactive hide details for "Grunenberg, Renar" ---06/11/2019 08:28:07 AM---Hallo Felipe, can you explain is this a generic Probl"><font size="2" color="#424282">"Grunenberg, Renar" ---06/11/2019 08:28:07 AM---Hallo Felipe, can you explain is this a generic Problem in rhel or only a scale related. Are there a</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">"Grunenberg, Renar" <Renar.Grunenberg@huk-coburg.de></font><br><font size="2" color="#5F5F5F">To: </font><font size="2">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">06/11/2019 08:28 AM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">[EXTERNAL] Re: [gpfsug-discuss] WG: Spectrum Scale with RHEL7.6 kernel 3.10.0-957.21.2</font><br><font size="2" color="#5F5F5F">Sent by: </font><font size="2">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br><br><br><font face="Arial">Hallo Felipe,</font><br><font face="Arial">can you explain is this a generic Problem in rhel or only a scale related. Are there any cicumstance already available? We ask redhat but have no points that this are know to them?</font><br><font face="Arial"> </font><br><font face="Arial">Regards Renar</font><br><font face="Arial"> </font><p><font face="Arial">Renar Grunenberg<br>Abteilung Informatik - Betrieb<br><br>HUK-COBURG<br>Bahnhofsplatz<br>96444 Coburg</font><table class="47a913c1-75c5-40ee-a34f-74a2f467be8dTable" border="0" cellspacing="0" cellpadding="0"><tr valign="top"><td width="56" valign="middle"><font face="Arial">Telefon:</font></td><td width="236" valign="middle"><font face="Arial">09561 96-44110</font></td></tr>
<tr valign="top"><td width="56" valign="middle"><font face="Arial">Telefax:</font></td><td width="236" valign="middle"><font face="Arial">09561 96-44104</font></td></tr>
<tr valign="top"><td width="56" valign="middle"><font face="Arial">E-Mail:</font></td><td width="236" valign="middle"><font face="Arial">Renar.Grunenberg@huk-coburg.de</font></td></tr>
<tr valign="top"><td width="56" valign="middle"><font face="Arial">Internet:</font></td><td width="236" valign="middle"><font face="Arial">www.huk.de</font></td></tr></table><hr width="100%" size="2" align="left" id="HR1"><font size="2" face="Arial">HUK-COBURG Haftpflicht-Unterstützungs-Kasse kraftfahrender Beamter Deutschlands a. G. in Coburg<br>Reg.-Gericht Coburg HRB 100; St.-Nr. 9212/101/00021<br>Sitz der Gesellschaft: Bahnhofsplatz, 96444 Coburg<br>Vorsitzender des Aufsichtsrats: Prof. Dr. Heinrich R. Schradin.<br>Vorstand: Klaus-Jürgen Heitmann (Sprecher), Stefan Gronbach, Dr. Hans Olav Herøy, Dr. Jörg Rheinländer (stv.), Sarah Rössler, Daniel Thomas.</font><font face="Arial"><br></font><hr width="100%" size="2" align="left" id="HR1_1"><font size="2" face="Arial">Diese Nachricht enthält vertrauliche und/oder rechtlich geschützte Informationen.<br>Wenn Sie nicht der richtige Adressat sind oder diese Nachricht irrtümlich erhalten haben,<br>informieren Sie bitte sofort den Absender und vernichten Sie diese Nachricht.<br>Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Nachricht ist nicht gestattet.<br><br>This information may contain confidential and/or privileged information.<br>If you are not the intended recipient (or have received this information in error) please notify the<br>sender immediately and destroy this information.<br>Any unauthorized copying, disclosure or distribution of the material in this information is strictly forbidden.</font><font face="Arial"><br></font><hr width="100%" size="2" align="left" id="HR1_2"><br><b><font face="Calibri">Von:</font></b><font face="Calibri"> gpfsug-discuss-bounces@spectrumscale.org <gpfsug-discuss-bounces@spectrumscale.org> </font><b><font face="Calibri">Im Auftrag von </font></b><font face="Calibri">Felipe Knop</font><b><font face="Calibri"><br>Gesendet:</font></b><font face="Calibri"> Montag, 10. Juni 2019 15:43</font><b><font face="Calibri"><br>An:</font></b><font face="Calibri"> gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></font><b><font face="Calibri"><br>Betreff:</font></b><font face="Calibri"> Re: [gpfsug-discuss] WG: Spectrum Scale with RHEL7.6 kernel 3.10.0-957.21.2</font><br><font face="Arial"> </font><p><font size="2" face="Arial">Renar,</font><font face="Arial"><br></font><font size="2" face="Arial"><br>Thanks. Of the changes below, it appears that</font><font face="Arial"><br><br>* security: double-free attempted in security_inode_init_security() (BZ#1702286)<br></font><font size="2" face="Arial"><br>was the one that ended up triggering the problem. Our investigations now show that RHEL kernels >= 3.10.0-957.19.1 are impacted.</font><font face="Arial"><br><br></font><font size="2" face="Arial"><br>Felipe</font><font face="Arial"><br></font><font size="2" face="Arial"><br>----<br>Felipe Knop </font><a href="mailto:knop@us.ibm.com"><u><font size="2" color="#0000FF" face="Arial">knop@us.ibm.com</font></u></a><font size="2" face="Arial"><br>GPFS Development and Security<br>IBM Systems<br>IBM Building 008<br>2455 South Rd, Poughkeepsie, NY 12601<br>(845) 433-9314 T/L 293-9314<br></font><font face="Arial"><br><br><br></font><img src="cid:1__=8FBB0E85DFC053C58f9e8a93df938690918c8FB@" width="16" height="16" alt="Inactive hide details for "Grunenberg, Renar" ---06/10/2019 08:43:27 AM---Hallo Felipe, here are the change list:"><font size="2" color="#424282" face="Arial">"Grunenberg, Renar" ---06/10/2019 08:43:27 AM---Hallo Felipe, here are the change list:</font><font face="Arial"><br></font><font size="2" color="#5F5F5F" face="Arial"><br>From: </font><font size="2" face="Arial">"Grunenberg, Renar" <</font><a href="mailto:Renar.Grunenberg@huk-coburg.de"><u><font size="2" color="#0000FF" face="Arial">Renar.Grunenberg@huk-coburg.de</font></u></a><font size="2" face="Arial">></font><font size="2" color="#5F5F5F" face="Arial"><br>To: </font><font size="2" face="Arial">"'gpfsug-discuss@spectrumscale.org'" <</font><a href="mailto:gpfsug-discuss@spectrumscale.org"><u><font size="2" color="#0000FF" face="Arial">gpfsug-discuss@spectrumscale.org</font></u></a><font size="2" face="Arial">></font><font size="2" color="#5F5F5F" face="Arial"><br>Date: </font><font size="2" face="Arial">06/10/2019 08:43 AM</font><font size="2" color="#5F5F5F" face="Arial"><br>Subject: </font><font size="2" face="Arial">[EXTERNAL] [gpfsug-discuss] WG: Spectrum Scale with RHEL7.6 kernel 3.10.0-957.21.2</font><font size="2" color="#5F5F5F" face="Arial"><br>Sent by: </font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><u><font size="2" color="#0000FF" face="Arial">gpfsug-discuss-bounces@spectrumscale.org</font></u></a><br><hr width="100%" size="2" align="left" noshade><br><font face="Arial"><br><br><br>Hallo Felipe,<br><br>here are the change list:<br>RHBA-2019:1337 kernel bug fix update<br><br><br>Summary:<br><br>Updated kernel packages that fix various bugs are now available for Red Hat Enterprise Linux 7.<br><br>The kernel packages contain the Linux kernel, the core of any Linux operating system.<br><br>This update fixes the following bugs:<br><br>* Mellanox CX-5 MAC learning with OVS H/W offload not working (BZ#1686292)<br><br>* RHEL7.4 NFS4.1 client and server repeated SEQUENCE / TEST_STATEIDs with SEQUENCE Reply has SEQ4_STATUS_RECALLABLE_STATE_REVOKED set - NFS server should return NFS4ERR_DELEG_REVOKED or NFS4ERR_BAD_STATEID for revoked delegations (BZ#1689811)<br><br>* PANIC: "BUG: unable to handle kernel paging request" in the mtip32xx mtip_init_cmd_header routine (BZ#1689929)<br><br>* The nvme cli delete-ns command hangs indefinitely. (BZ#1690519)<br><br>* drm/nouveau: nv50 - Graphics become sluggish or frozen for nvidia Pascal cards (Regression from 1584963) - Need to flush fb writes when rewinding push buffer (BZ#1690761)<br><br>* [CEE/SD] Ceph+NFS server crashed and rebooted due to CephFS kernel client issue (BZ#1692266)<br><br>* [Mellanox OVS offload] tc fails to calculate the checksum in case vlan trunk and header rewrite (BZ#1693110)<br><br>* aio O_DIRECT writes to non-page-aligned file locations on ext4 can result in the overlapped portion of the page containing zeros (BZ#1693561)<br><br>* [HP WS 7.6 bug] Audio driver does not recognize multi function audio jack microphone input (BZ#1693562)<br><br>* XFS returns ENOSPC when using extent size hint with space still available (BZ#1693796)<br><br>* OVN requires IPv6 to be enabled (BZ#1694981)<br><br>* breaks DMA API for non-GPL drivers (BZ#1695511)<br><br>* ovl_create can return positive retval and crash the host (BZ#1696292)<br><br>* ceph: append mode is broken for sync/direct write (BZ#1696595)<br><br>* Problem building module due to -EXPORT_SYMBOL_GPL/-EXPORT_SYMBOL (BZ#1697241)<br><br>* Failed to load kpatch module after install the rpm package occasionally on ppc64le (BZ#1697867)<br><br>* [Hyper-V][RHEL7] Stop suppressing PCID bit (BZ#1697940)<br><br>* Resizing an online EXT4 filesystem on a loopback device hangs (BZ#1698110)<br><br>* dm table: propagate BDI_CAP_STABLE_WRITES (BZ#1699722)<br><br>* [ESXi][RHEL7.6]After upgrade to kernel-3.10.0-957.el7, system is unable to discover newly added VMware LSI Logic SAS virtual disks without a reboot. (BZ#1699723)<br><br>* kernel: zcrypt: fix specification exception on z196 at ap probe (BZ#1700706)<br><br>* XFS: Metadata corruption detected at xfs_attr3_leaf_write_verify() (BZ#1701293)<br><br>* stime showed huge values related to wrong calculation of time deltas (L3:) (BZ#1701743)<br><br>* Kernel panic due to NULL pointer dereference at sysfs_do_create_link_sd.isra.2+0x34 while loading [ipmi_si] module using hard-coded device (BZ#1701991)<br><br>* IPv6 ECMP modulo N hashing inefficient when X^2 rt6i_nsiblings (BZ#1702282)<br><br>* security: double-free attempted in security_inode_init_security() (BZ#1702286)<br><br>* Missing wakeup leaves task stuck waiting in blk_queue_enter() (BZ#1702921)<br><br>* Satellite Capsule sync triggers several XFS corruptions (BZ#1702922)<br><br>* BUG: SELinux doesn't handle NFS crossmnt well (BZ#1702923)<br><br>* md_clear flag missing from /proc/cpuinfo on late microcode update (BZ#1712993)<br><br>* MDS mitigations are not enabled after double microcode update (BZ#1712998)<br><br>* WARNING: CPU: 0 PID: 0 at kernel/jump_label.c:90 __static_key_slow_dec+0xa6/0xb0 (BZ#1713004)<br><br>Users of kernel are advised to upgrade to these updated packages, which fix these bugs.<br><br>Full details and references:<br></font><u><font color="#0000FF" face="Arial"><br></font></u><a href="https://access.redhat.com/errata/RHBA-2019:1337?sc_cid=701600000006NHXAA2"><u><font color="#0000FF" face="Arial">https://access.redhat.com/errata/RHBA-2019:1337?sc_cid=701600000006NHXAA2</font></u></a><font face="Arial"><br><br>Revision History:<br><br>Issue Date: 2019-06-04<br>Updated: 2019-06-04<br><br>Regards Renar</font><p><font face="Arial">Renar Grunenberg<br>Abteilung Informatik - Betrieb<br><br>HUK-COBURG<br>Bahnhofsplatz<br>96444 Coburg</font><p><table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0"><tr valign="top"><td width="59" valign="middle"><font face="Arial">Telefon:</font></td><td width="236" valign="middle"><font face="Arial">09561 96-44110</font></td></tr>
<tr valign="top"><td width="59" valign="middle"><font face="Arial">Telefax:</font></td><td width="236" valign="middle"><font face="Arial">09561 96-44104</font></td></tr>
<tr valign="top"><td width="59" valign="middle"><font face="Arial">E-Mail:</font></td><td width="236" valign="middle"><a href="mailto:Renar.Grunenberg@huk-coburg.de"><u><font color="#0000FF" face="Arial">Renar.Grunenberg@huk-coburg.de</font></u></a></td></tr>
<tr valign="top"><td width="59" valign="middle"><font face="Arial">Internet:</font></td><td width="236" valign="middle"><a href="http://www.huk.de"><u><font color="#0000FF" face="Arial">www.huk.de</font></u></a></td></tr></table><hr width="100%" size="2" align="left"><br><font size="2" face="Arial">HUK-COBURG Haftpflicht-Unterstützungs-Kasse kraftfahrender Beamter Deutschlands a. G. in Coburg<br>Reg.-Gericht Coburg HRB 100; St.-Nr. 9212/101/00021<br>Sitz der Gesellschaft: Bahnhofsplatz, 96444 Coburg<br>Vorsitzender des Aufsichtsrats: Prof. Dr. Heinrich R. Schradin.<br>Vorstand: Klaus-Jürgen Heitmann (Sprecher), Stefan Gronbach, Dr. Hans Olav Herøy, Dr. Jörg Rheinländer (stv.), Sarah Rössler, Daniel Thomas.</font><br><hr width="100%" size="2" align="left"><br><font size="2" face="Arial">Diese Nachricht enthält vertrauliche und/oder rechtlich geschützte Informationen.<br>Wenn Sie nicht der richtige Adressat sind oder diese Nachricht irrtümlich erhalten haben,<br>informieren Sie bitte sofort den Absender und vernichten Sie diese Nachricht.<br>Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Nachricht ist nicht gestattet.<br><br>This information may contain confidential and/or privileged information.<br>If you are not the intended recipient (or have received this information in error) please notify the<br>sender immediately and destroy this information.<br>Any unauthorized copying, disclosure or distribution of the material in this information is strictly forbidden.</font><br><hr width="100%" size="2" align="left"><br><b><font face="Calibri"><br>Von:</font></b><font face="Calibri"> </font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><u><font color="#0000FF" face="Calibri">gpfsug-discuss-bounces@spectrumscale.org</font></u></a><font face="Calibri"> [</font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><u><font color="#0000FF" face="Calibri">mailto:gpfsug-discuss-bounces@spectrumscale.org</font></u></a><font face="Calibri">] </font><b><font face="Calibri">Im Auftrag von </font></b><font face="Calibri">Felipe Knop</font><b><font face="Calibri"><br>Gesendet:</font></b><font face="Calibri"> Montag, 10. Juni 2019 06:41</font><b><font face="Calibri"><br>An:</font></b><font face="Calibri"> gpfsug main discussion list <</font><a href="mailto:gpfsug-discuss@spectrumscale.org"><u><font color="#0000FF" face="Calibri">gpfsug-discuss@spectrumscale.org</font></u></a><font face="Calibri">></font><b><font face="Calibri"><br>Betreff:</font></b><font face="Calibri"> Re: [gpfsug-discuss] Spectrum Scale with RHEL7.6 kernel 3.10.0-957.21.2</font><p><font size="2" face="Arial">Hi,<br><br>Though we are still learning what workload results in the problem, it appears that even minimal I/O on the file system may cause the OS to crash. One pattern that we saw was 'mkdir'. There is a chance that the DR site was not yet impacted because no I/O workload has been run there. In that case, rolling back to the prior kernel level (one which has been tested before) may be advisable.<br><br>Felipe<br><br>----<br>Felipe Knop </font><a href="mailto:knop@us.ibm.com"><u><font size="2" color="#0000FF" face="Arial">knop@us.ibm.com</font></u></a><font size="2" face="Arial"><br>GPFS Development and Security<br>IBM Systems<br>IBM Building 008<br>2455 South Rd, Poughkeepsie, NY 12601<br>(845) 433-9314 T/L 293-9314</font><font face="Arial"><br><br><br><br></font><img src="cid:1__=8FBB0E85DFC053C58f9e8a93df938690918c8FB@" width="16" height="16" alt="Inactive hide details for KG ---06/09/2019 09:38:55 AM---One of my customer already upgraded their DR site. Is rollback advised"><font size="2" color="#424282" face="Arial">KG ---06/09/2019 09:38:55 AM---One of my customer already upgraded their DR site. Is rollback advised? They will be running from DR</font><font size="2" color="#5F5F5F" face="Arial"><br><br>From: </font><font size="2" face="Arial">KG <</font><a href="mailto:spectrumscale@kiranghag.com"><u><font size="2" color="#0000FF" face="Arial">spectrumscale@kiranghag.com</font></u></a><font size="2" face="Arial">></font><font size="2" color="#5F5F5F" face="Arial"><br>To: </font><font size="2" face="Arial">gpfsug main discussion list <</font><a href="mailto:gpfsug-discuss@spectrumscale.org"><u><font size="2" color="#0000FF" face="Arial">gpfsug-discuss@spectrumscale.org</font></u></a><font size="2" face="Arial">></font><font size="2" color="#5F5F5F" face="Arial"><br>Date: </font><font size="2" face="Arial">06/09/2019 09:38 AM</font><font size="2" color="#5F5F5F" face="Arial"><br>Subject: </font><font size="2" face="Arial">[EXTERNAL] Re: [gpfsug-discuss] Spectrum Scale with RHEL7.6 kernel 3.10.0-957.21.2</font><font size="2" color="#5F5F5F" face="Arial"><br>Sent by: </font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><u><font size="2" color="#0000FF" face="Arial">gpfsug-discuss-bounces@spectrumscale.org</font></u></a><br><hr width="100%" size="2" align="left" noshade><br><font face="Arial"><br><br><br><br>One of my customer already upgraded their DR site.<br><br>Is rollback advised? They will be running from DR site for a day in another week.<br><br>On Sat, Jun 8, 2019, 03:37 Felipe Knop <</font><a href="mailto:knop@us.ibm.com"><u><font color="#0000FF" face="Arial">knop@us.ibm.com</font></u></a><font face="Arial">> wrote: </font><ul><ul><ul><ul><font size="2" face="Arial">Zach,<br><br>This appears to be affecting all Scale versions, including 5.0.2 -- but only when moving to the new 3.10.0-957.21.2 kernel. (3.10.0-957 is not impacted)<br><br>Felipe<br><br>----<br>Felipe Knop </font><a href="mailto:knop@us.ibm.com" target="_blank"><u><font size="2" color="#0000FF" face="Arial">knop@us.ibm.com</font></u></a><font size="2" face="Arial"><br>GPFS Development and Security<br>IBM Systems<br>IBM Building 008<br>2455 South Rd, Poughkeepsie, NY 12601<br>(845) 433-9314 T/L 293-9314</font><font face="Arial"><br><br></font><font size="2" color="#424282" face="Arial"><br><br>Zachary Mance ---06/07/2019 05:51:37 PM---Which versions of Spectrum Scale versions are you referring to? 5.0.2-3? ---------------------------</font><font size="2" color="#5F5F5F" face="Arial"><br><br>From: </font><font size="2" face="Arial">Zachary Mance <</font><a href="mailto:zmance@ucar.edu" target="_blank"><u><font size="2" color="#0000FF" face="Arial">zmance@ucar.edu</font></u></a><font size="2" face="Arial">></font><font size="2" color="#5F5F5F" face="Arial"><br>To: </font><font size="2" face="Arial">gpfsug main discussion list <</font><a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank"><u><font size="2" color="#0000FF" face="Arial">gpfsug-discuss@spectrumscale.org</font></u></a><font size="2" face="Arial">></font><font size="2" color="#5F5F5F" face="Arial"><br>Date: </font><font size="2" face="Arial">06/07/2019 05:51 PM</font><font size="2" color="#5F5F5F" face="Arial"><br>Subject: </font><font size="2" face="Arial">[EXTERNAL] Re: [gpfsug-discuss] Spectrum Scale with RHEL7.6 kernel 3.10.0-957.21.2</font><font size="2" color="#5F5F5F" face="Arial"><br>Sent by: </font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank"><u><font size="2" color="#0000FF" face="Arial">gpfsug-discuss-bounces@spectrumscale.org</font></u></a><br><hr width="100%" size="2" align="left" noshade><br><font face="Arial"><br><br><br><br>Which versions of Spectrum Scale versions are you referring to? 5.0.2-3?</font><font size="2" face="Helvetica"><br>---------------------------------------------------------------------------------------------------------------<br>Zach Mance </font><a href="mailto:zmance@ucar.edu" target="_blank"><u><font size="2" color="#0000FF" face="Helvetica">zmance@ucar.edu</font></u></a><font size="2" face="Helvetica"> (303) 497-1883 </font><font size="2" face="Arial"><br>HPC Data Infrastructure Group</font><font size="2" face="Helvetica"> / CISL / NCAR<br>--------------------------------------------------------------------------------------------------------------- </font><font face="Arial"><br><br><br>On Fri, Jun 7, 2019 at 3:45 PM Felipe Knop <</font><a href="mailto:knop@us.ibm.com" target="_blank"><u><font color="#0000FF" face="Arial">knop@us.ibm.com</font></u></a><font face="Arial">> wrote: </font><ul><ul><ul><ul><ul><ul><ul><ul><font size="2" face="Arial">All,<br><br>There have been reported issues (including kernel crashes) on Spectrum Scale with the latest RHEL7.6 kernel 3.10.0-957.21.2. Please consider delaying upgrades to this kernel until further information is provided. <br><br>Thanks,<br><br>Felipe<br><br>----<br>Felipe Knop </font><a href="mailto:knop@us.ibm.com" target="_blank"><u><font size="2" color="#0000FF" face="Arial">knop@us.ibm.com</font></u></a><font size="2" face="Arial"><br>GPFS Development and Security<br>IBM Systems<br>IBM Building 008<br>2455 South Rd, Poughkeepsie, NY 12601<br>(845) 433-9314 T/L 293-9314</font><font face="Arial"><br><br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="http://spectrumscale.org" target="_blank"><u><font color="#0000FF" face="Arial">spectrumscale.org</font></u></a><u><font color="#0000FF" face="Arial"><br></font></u><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><u><font color="#0000FF" face="Arial">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></u></a><font size="2" face="Courier New">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="http://spectrumscale.org" target="_blank"><u><font size="2" color="#0000FF" face="Courier New">spectrumscale.org</font></u></a><u><font color="#0000FF" face="Arial"><br></font></u><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><u><font size="2" color="#0000FF" face="Courier New">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></u></a><font size="2" face="Courier New"> </font></ul></ul></ul></ul></ul></ul></ul></ul><font face="Arial"><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="http://spectrumscale.org" target="_blank"><u><font color="#0000FF" face="Arial">spectrumscale.org</font></u></a><u><font color="#0000FF" face="Arial"><br></font></u><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><u><font color="#0000FF" face="Arial">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></u></a><u><font color="#0000FF" face="Arial">[attachment "graycol.gif" deleted by Felipe Knop/Poughkeepsie/IBM] </font></u><font size="2" face="Courier New">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</font><u><font color="#0000FF" face="Arial"><br></font></u><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><u><font size="2" color="#0000FF" face="Courier New">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></u></a><font size="2" face="Courier New"> </font></ul></ul></ul></ul><font size="2" face="Courier New">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</font><u><font size="2" color="#0000FF" face="Courier New"><br></font></u><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><u><font size="2" color="#0000FF" face="Courier New">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></u></a><font size="2" face="Courier New"> </font><font face="Arial"><br><br></font><tt><font size="2">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><tt><font size="2"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><tt><font size="2"> <br></font></tt><br><br><BR>
</body></html>