<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">I tried it on one node but investing in what could be up to £5000 in SSDs when we don’t know the gains isn’t something I can argue.
Not that LROC will hurt the environment but my users may not see any benefit. My cluster is the complete opposite of busy (relative to people saying they’re seeing sustained 800MB/sec throughput), I just need it stable.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Richard<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Frank Tower [mailto:frank.tower@outlook.com]
<br>
<b>Sent:</b> 24 April 2017 22:12<br>
<b>To:</b> Sobey, Richard A <r.sobey@imperial.ac.uk>; gpfsug main discussion list <gpfsug-discuss@spectrumscale.org>; Jan-Frode Myklebust <janfrode@tanso.net><br>
<b>Subject:</b> Re: [gpfsug-discuss] Protocol node recommendations<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div id="divtagdefaultwrapper">
<p><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">From what I've read from the Wiki:
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">'</span><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">The NFS protocol performance is largely dependent on the base system performance of the protocol
node hardware and network. This includes multiple factors the type and number of CPUs, the size of the main memory in the nodes, the type of disk drives used (HDD, SSD, etc.) and the disk configuration (RAID-level, replication etc.). In addition, NFS protocol
performance can be impacted by the overall load of the node (such as number of clients accessing, snapshot creation/deletion and more) and administrative tasks (for example filesystem checks or online re-striping of disk arrays).'</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black"><br>
<br>
</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">Nowadays, SSD is worst to invest.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">LROC could be an option in the future, but we need to quantify NFS/CIFS workload first.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">Are you using LROC with your GPFS installation ?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">Best,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">Frank.<o:p></o:p></span></p>
</div>
<div>
<div>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-family:"Calibri",sans-serif;color:black">
<hr size="2" width="98%" align="center">
</span></div>
<div id="divRplyFwdMsg">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> Sobey, Richard A <<a href="mailto:r.sobey@imperial.ac.uk">r.sobey@imperial.ac.uk</a>><br>
<b>Sent:</b> Monday, April 24, 2017 11:11 AM<br>
<b>To:</b> gpfsug main discussion list; Jan-Frode Myklebust<br>
<b>Subject:</b> Re: [gpfsug-discuss] Protocol node recommendations</span><span style="font-family:"Calibri",sans-serif;color:black">
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">What’s your SSD going to help with… will you implement it as a LROC device? Otherwise I can’t see the
benefit to using it to boot off.</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">
<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-bounces@spectrumscale.org</a> [<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">mailto:gpfsug-discuss-bounces@spectrumscale.org</a>]
<b>On Behalf Of </b>Frank Tower<br>
<b>Sent:</b> 23 April 2017 22:28<br>
<b>To:</b> Jan-Frode Myklebust <<a href="mailto:janfrode@tanso.net">janfrode@tanso.net</a>>;
<a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a><br>
<b>Subject:</b> Re: [gpfsug-discuss] Protocol node recommendations</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black"> <o:p></o:p></span></p>
<div id="divtagdefaultwrapper">
<p><span style="font-family:"Calibri",sans-serif;color:black">Hi,<o:p></o:p></span></p>
<p><span style="font-family:"Calibri",sans-serif;color:black"> <o:p></o:p></span></p>
<p><span style="font-family:"Calibri",sans-serif;color:black">Nice ! didn't pay attention at the revision and the spreadsheet. If someone still have a copy somewhere it could be useful, Google didn't help :(<o:p></o:p></span></p>
<p><span style="font-family:"Calibri",sans-serif;color:black"> <o:p></o:p></span></p>
<p><span style="font-family:"Calibri",sans-serif;color:black">We will follow your advise and start with 3 protocol nodes equipped with 128GB memory, 2 x 12 cores (maybe E5-2680 or E5-2670).<o:p></o:p></span></p>
<p><span style="font-family:"Calibri",sans-serif;color:black"> <o:p></o:p></span></p>
<p><span style="font-family:"Calibri",sans-serif;color:black">From what I read, NFS-Ganesha mainly depend of the hardware, Linux on a SSD should be a big plus in our case.<o:p></o:p></span></p>
<p><span style="font-family:"Calibri",sans-serif;color:black"> <o:p></o:p></span></p>
<p><span style="font-family:"Calibri",sans-serif;color:black">Best,<o:p></o:p></span></p>
<p><span style="font-family:"Calibri",sans-serif;color:black">Frank<o:p></o:p></span></p>
<p><span style="font-family:"Calibri",sans-serif;color:black"> <o:p></o:p></span></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-family:"Calibri",sans-serif;color:black">
<hr size="2" width="98%" align="center">
</span></div>
<div id="divRplyFwdMsg">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> Jan-Frode
Myklebust <<a href="mailto:janfrode@tanso.net">janfrode@tanso.net</a>><br>
<b>Sent:</b> Sunday, April 23, 2017 12:07:38 PM<br>
<b>To:</b> Frank Tower; <a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a><br>
<b>Subject:</b> Re: [gpfsug-discuss] Protocol node recommendations</span><span style="font-family:"Calibri",sans-serif;color:black">
<o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black"> <o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black">The protocol sizing tool should be available from
<a href="https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/Sizing%20Guidance%20for%20Protocol%20Node/version/70a4c7c0-a5c6-4dde-b391-8f91c542dd7d">
https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/Sizing%20Guidance%20for%20Protocol%20Node/version/70a4c7c0-a5c6-4dde-b391-8f91c542dd7d</a> , but I'm getting 404 now.<br>
<br>
I think 128GB should be enough for both protocols on same nodes, and I think your 3 node suggestion is best. Better load sharing with not dedicating subset of nodes to each protocol.<br>
<br>
<br>
<br>
-jf<o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black">lør. 22. apr. 2017 kl. 21.22 skrev Frank Tower <<a href="mailto:frank.tower@outlook.com">frank.tower@outlook.com</a>>:<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div id="m_378511725885416141divtagdefaultwrapper">
<p><span style="font-family:"Calibri",sans-serif;color:black">Hi,<o:p></o:p></span></p>
<p><span style="font-family:"Calibri",sans-serif;color:black"> <o:p></o:p></span></p>
<p><span style="font-family:"Calibri",sans-serif;color:black">Thank for the recommendations. <o:p></o:p></span></p>
<p><span style="font-family:"Calibri",sans-serif;color:black">Now we deal with the situation of: <o:p></o:p></span></p>
<p><span style="font-family:"Calibri",sans-serif;color:black"> <o:p></o:p></span></p>
<p><span style="font-family:"Calibri",sans-serif;color:black">- take 3 nodes with round robin DNS that handle both protocols<o:p></o:p></span></p>
<p><span style="font-family:"Calibri",sans-serif;color:black">- take 4 nodes, split CIFS and NFS, still use round robin DNS for CIFS and NFS services.<o:p></o:p></span></p>
<p><span style="font-family:"Calibri",sans-serif;color:black"> <o:p></o:p></span></p>
<p><span style="font-family:"Calibri",sans-serif;color:black">Regarding your recommendations, 256GB memory node could be a plus if we mix both protocols for such case.<o:p></o:p></span></p>
<p><span style="font-family:"Calibri",sans-serif;color:black"> <o:p></o:p></span></p>
<p><span style="font-family:"Calibri",sans-serif;color:black">Is the spreadsheet publicly available or do we need to ask IBM ?<o:p></o:p></span></p>
<p><span style="font-family:"Calibri",sans-serif;color:black"> <o:p></o:p></span></p>
<p><span style="font-family:"Calibri",sans-serif;color:black">Thank for your help,<o:p></o:p></span></p>
<p><span style="font-family:"Calibri",sans-serif;color:black">Frank.<o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black"> <o:p></o:p></span></p>
<div>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-family:"Calibri",sans-serif;color:black">
<hr size="2" width="98%" align="center">
</span></div>
<div id="m_378511725885416141divRplyFwdMsg">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> Jan-Frode
Myklebust <<a href="mailto:janfrode@tanso.net" target="_blank">janfrode@tanso.net</a>><br>
<b>Sent:</b> Saturday, April 22, 2017 10:50 AM<br>
<b>To:</b> <a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a><br>
<b>Subject:</b> Re: [gpfsug-discuss] Protocol node recommendations</span><span style="font-family:"Calibri",sans-serif;color:black">
<o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black"> <o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div id="m_378511725885416141divtagdefaultwrapper">
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black">That's a tiny maxFilesToCache...<br>
<br>
I would start by implementing the settings from /usr/lpp/mmfs/*/gpfsprotocolldefaul* plus a 64GB pagepool for your protocoll nodes, and leave further tuning to when you see you have issues.<br>
<br>
Regarding sizing, we have a spreadsheet somewhere where you can input some workload parameters and get an idea for how many nodes you'll need. Your node config seems fine, but one node seems too few to serve 1000+ users. We support max 3000 SMB connections/node,
and I believe the recommendation is 4000 NFS connections/node.<br>
<br>
<br>
-jf<o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black">lør. 22. apr. 2017 kl. 08.34 skrev Frank Tower <<a href="mailto:frank.tower@outlook.com" target="_blank">frank.tower@outlook.com</a>>:<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div id="m_378511725885416141m_-3610594251344932013divtagdefaultwrapper">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black">Hi,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black">We have here around 2PB GPFS (4.2.2) accessed through an HPC cluster with GPFS client on each node.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black">We will have to open GPFS to all our users over CIFS and kerberized NFS with ACL support for both protocol for around
+1000 users<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black">All users have different use case and needs: <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black">- some will do random I/O through a large set of opened files (~5k files)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black">- some will do large write with 500GB-1TB files<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black">- other will arrange sequential I/O with ~10k opened files<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black">NFS and CIFS will share the same server, so I through to use SSD drive, at least 128GB memory with 2 sockets.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black">Regarding tuning parameters, I thought at: <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black">maxFilesToCache 10000<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black">syncIntervalStrict yes<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black">workerThreads (8*core)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black">prefetchPct 40 (for now and update if needed)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black">I read the wiki 'Sizing Guidance for Protocol Node', but I was wondering if someone could share his experience/best practice regarding
hardware sizing and/or tuning parameters.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black">Thank by advance,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black">Frank<o:p></o:p></span></p>
</div>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Calibri",sans-serif;color:black">_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><o:p></o:p></span></p>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>