<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
tt
{mso-style-priority:99;
font-family:"Courier New";}
span.gmailmsg
{mso-style-name:gmail_msg;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks Sven!
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">We recently upgrade to 4.2.2 and will see about lowering the maxFilesToCache to something more appropriate. We’re not offering NFS access as a performance solution…
but it can’t come to a crawl either!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Your help is greatly appreciated as always,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">-Bryan<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> gpfsug-discuss-bounces@spectrumscale.org [mailto:gpfsug-discuss-bounces@spectrumscale.org]
<b>On Behalf Of </b>Sven Oehme<br>
<b>Sent:</b> Friday, March 24, 2017 1:20 PM<br>
<b>To:</b> gpfsug main discussion list <gpfsug-discuss@spectrumscale.org>; IBM Spectrum Scale <scale@us.ibm.com><br>
<b>Subject:</b> Re: [gpfsug-discuss] CES node slow to respond<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">changes in ganesha management code were made in April 2016 to reduce the need for high maxfilestocache value, the ganesha daemon adjusts it allowed file cache by reading the maxfilestocache value and then reducing its allowed NOFILE value
. the code shipped with 4.2.2 release. <o:p></o:p></p>
<div>
<p class="MsoNormal">you want a high maxfilestocache value on CES nodes for various reasons. just FYI we have some customers who have MFTC set to 15 Million :-)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">sven <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Fri, Mar 24, 2017 at 11:13 AM Bryan Banister <<a href="mailto:bbanister@jumptrading.com">bbanister@jumptrading.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi Vipul,</span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hmm… interesting. We have dedicated systems running CES and nothing else, so
the only thing opening files on GPFS is ganesha. IBM Support recommended we massively increase the maxFilesToCache to fix the performance issues we were having. I could try to reproduce the problem to investigate further, but the impact to users would not
be appreciated. </span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Setting such a high value for maxFilesToCache has direct impact on the token
manager so fixing this would be very good.</span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Perhaps IBM could take a closer look at this without our involvement?</span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">-Bryan</span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span></b></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b></span><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">
Vipul Paul [mailto:<a href="mailto:vpaul@us.ibm.com" target="_blank">vpaul@us.ibm.com</a>]
<b>On Behalf Of </b>IBM Spectrum Scale</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><br>
<span class="gmailmsg"><b>Sent:</b> Friday, March 24, 2017 12:18 PM</span><br>
<span class="gmailmsg"><b>To:</b> gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>>; Bryan Banister <<a href="mailto:bbanister@jumptrading.com" target="_blank">bbanister@jumptrading.com</a>></span><br>
<span class="gmailmsg"><b>Subject:</b> Re: CES node slow to respond</span></span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Hi Bryan,</span></span><br>
<br>
<span class="gmailmsg"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Making sure Malahal's reply was received by the user group.</span></span><br>
<br>
<span class="gmailmsg"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">>> Then we noticed that the CES host had 5.4 million files open</span></span><span style="font-size:10.0pt;font-family:"Arial",sans-serif"><br>
<br>
<span class="gmailmsg">This is technically not possible with ganesha alone. A process can only open 1 million files on RHEL distro. Either we have leaks in kernel or some other processes contributing to this.</span><br>
<br>
<span class="gmailmsg">Ganesha does keep NFSv3 files open and keep open for performance. It doesn't have a good logic to close them after some inactivity. It does close them if the number is close to max open files which is configurable.</span><br>
<br>
<span class="gmailmsg">PS: kNFS does open, read/write, and then close. No caching in older versions. They did have a feature in a recent code to cache open files in NFSv3 as well.</span><br>
</span><br>
<span class="gmailmsg"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Regards, The Spectrum Scale (GPFS) team</span></span><span style="font-size:10.0pt;font-family:"Arial",sans-serif"><br>
<br>
<span class="gmailmsg">------------------------------------------------------------------------------------------------------------------</span><br>
<span class="gmailmsg">If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at
</span></span><a href="https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479" target="_blank"><span class="gmailmsg"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479</span></span></a><span class="gmailmsg"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">.
</span></span><span style="font-size:10.0pt;font-family:"Arial",sans-serif"><br>
<br>
<span class="gmailmsg">If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact <a href="tel:(800)%20237-5511" target="_blank">1-800-237-5511</a> in the United States or your
local IBM Service Center in other countries. </span><br>
<br>
<span class="gmailmsg">The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team.</span></span><br>
<br>
<br>
<br>
<span class="gmailmsg"><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">From: </span></span><span class="gmailmsg"><span style="font-size:7.5pt;font-family:"Arial",sans-serif">Bryan Banister <<a href="mailto:bbanister@jumptrading.com" target="_blank">bbanister@jumptrading.com</a>></span></span><br>
<span class="gmailmsg"><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">To: </span></span><span class="gmailmsg"><span style="font-size:7.5pt;font-family:"Arial",sans-serif">gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>></span></span><br>
<span class="gmailmsg"><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">Date: </span></span><span class="gmailmsg"><span style="font-size:7.5pt;font-family:"Arial",sans-serif">03/23/2017 08:27 AM</span></span><br>
<span class="gmailmsg"><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">Subject: </span></span><span class="gmailmsg"><span style="font-size:7.5pt;font-family:"Arial",sans-serif">Re: [gpfsug-discuss] CES node slow to respond</span></span><br>
<span class="gmailmsg"><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">Sent by: </span></span><span class="gmailmsg"><span style="font-size:7.5pt;font-family:"Arial",sans-serif"><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a></span></span><o:p></o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" noshade="" style="color:#A0A0A0" align="center">
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><br>
<br>
<br>
<span class="gmailmsg"><span style="font-size:10.0pt;font-family:"Courier New"">Anybody from IBM willing/able to give us some explanation of why ganesha is holding open so many files? Is this expected/needed/etc?</span></span><span style="font-size:10.0pt;font-family:"Courier New""><br>
<br>
<tt>Or do we have to open a PMR to get some kind of explanation?</tt><br>
<tt>-B</tt><br>
<br>
<tt>-----Original Message-----</tt><br>
<tt>From: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">
gpfsug-discuss-bounces@spectrumscale.org</a> [</tt></span><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank"><span class="gmailmsg"><span style="font-size:10.0pt;font-family:"Courier New"">mailto:gpfsug-discuss-bounces@spectrumscale.org</span></span></a><span class="gmailmsg"><span style="font-size:10.0pt;font-family:"Courier New"">]
On Behalf Of Matt Weil</span></span><span style="font-size:10.0pt;font-family:"Courier New""><br>
<tt>Sent: Thursday, March 23, 2017 10:24 AM</tt><br>
<tt>To: <a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a></tt><br>
<tt>Subject: Re: [gpfsug-discuss] CES node slow to respond</tt><br>
<br>
<tt>FYI all</tt><br>
<br>
<tt>We also ran into this after bumping maxFilesToCache.</tt><br>
<br>
<tt>Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348:</tt><br>
<tt>unexpected error:</tt><br>
<tt>Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many</tt><br>
<tt>open files in system</tt><br>
<br>
<tt>fix</tt><br>
<br>
<tt>sysctl -w fs.file-max=1000000</tt><br>
<br>
<br>
<tt>On 3/22/17 12:01 PM, Bryan Banister wrote:</tt><br>
<tt>> We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is
opened... baaaah.</tt><br>
<tt>></tt><br>
<tt>> Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this.</tt><br>
<tt>> -Bryan</tt><br>
<tt>></tt><br>
<tt>> -----Original Message-----</tt><br>
<tt>> From: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">
gpfsug-discuss-bounces@spectrumscale.org</a> [</tt></span><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank"><span class="gmailmsg"><span style="font-size:10.0pt;font-family:"Courier New"">mailto:gpfsug-discuss-bounces@spectrumscale.org</span></span></a><span class="gmailmsg"><span style="font-size:10.0pt;font-family:"Courier New"">]
On Behalf Of Matt Weil</span></span><span style="font-size:10.0pt;font-family:"Courier New""><br>
<tt>> Sent: Wednesday, March 22, 2017 11:43 AM</tt><br>
<tt>> To: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>></tt><br>
<tt>> Subject: [gpfsug-discuss] CES node slow to respond</tt><br>
<tt>></tt><br>
<tt>> All,</tt><br>
<tt>></tt><br>
<tt>> We had an indecent yesterday where one of our CES nodes slowed to a</tt><br>
<tt>> crawl. GPFS waiters showed pre fetch threads going after inodes.</tt><br>
<tt>> iohist also showed lots of inode fetching. Then we noticed that the CES</tt><br>
<tt>> host had 5.4 million files open.</tt><br>
<tt>></tt><br>
<tt>> The change I made was to set maxStatCache=DEFAULT because it is linux.</tt><br>
<tt>> And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS.</tt><br>
<tt>></tt><br>
<tt>> Is there something else we should change as well.</tt><br>
<tt>></tt><br>
<tt>> Thanks</tt><br>
<tt>></tt><br>
<tt>> Matt</tt><br>
<tt>></tt><br>
<tt>></tt><br>
<tt>> ________________________________</tt><br>
<tt>> The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any
action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.</tt><br>
<tt>> _______________________________________________</tt><br>
<tt>> gpfsug-discuss mailing list</tt><br>
<tt>> gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a></tt><br>
<tt>> </tt></span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><span class="gmailmsg"><span style="font-size:10.0pt;font-family:"Courier New"">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></span></a><span style="font-size:10.0pt;font-family:"Courier New""><br>
<tt>></tt><br>
<tt>> ________________________________</tt><br>
<tt>></tt><br>
<tt>> Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of
this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness
or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial
product.</tt><br>
<tt>> _______________________________________________</tt><br>
<tt>> gpfsug-discuss mailing list</tt><br>
<tt>> gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a></tt><br>
<tt>> </tt></span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><span class="gmailmsg"><span style="font-size:10.0pt;font-family:"Courier New"">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></span></a><span style="font-size:10.0pt;font-family:"Courier New""><br>
<br>
<br>
<tt>________________________________</tt><br>
<tt>The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any
action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.</tt><br>
<tt>_______________________________________________</tt><br>
<tt>gpfsug-discuss mailing list</tt><br>
<tt>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a></tt><br>
</span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><span class="gmailmsg"><span style="font-size:10.0pt;font-family:"Courier New"">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></span></a><span style="font-size:10.0pt;font-family:"Courier New""><br>
<br>
<tt>________________________________</tt><br>
<br>
<tt>Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of
this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness
or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial
product.</tt><br>
<tt>_______________________________________________</tt><br>
<tt>gpfsug-discuss mailing list</tt><br>
<tt>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a></tt><br>
</span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><span class="gmailmsg"><span style="font-size:10.0pt;font-family:"Courier New"">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></span></a><span style="font-size:10.0pt;font-family:"Courier New""><br>
<br>
</span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" align="center">
</div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:gray"><br>
Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this
email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness
or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial
product.</span><o:p></o:p></p>
</div>
<p class="MsoNormal">_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><o:p></o:p></p>
</blockquote>
</div>
</div>
<br>
<hr>
<font face="Arial" color="Gray" size="1"><br>
Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this
email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness
or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial
product.<br>
</font>
</body>
</html>