<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"Segoe UI Symbol";
panose-1:2 11 5 2 4 2 4 2 2 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
tt
{mso-style-priority:99;
font-family:"Courier New";}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle21
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Another possibility is to try increasing the timeouts. We used to have problems with this all of the time on clusters with thousands of nodes, but now we run
with the following settings increased from their [defaults]…<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> sqtBusyThreadTimeout [10] = 120<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> sqtCommandRetryDelay [60] = 120<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> sqtCommandTimeout [300] = 500<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">These are in the category of undocumented configurables, so you may wish to accompany this with a PMR. And you’ll need to know the secret handshake that follows
this…<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">mmchconfig: Attention: Unknown attribute specified: sqtBusyThreadTimeout. Press the ENTER key to continue.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">-Paul<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> gpfsug-discuss-bounces@spectrumscale.org <gpfsug-discuss-bounces@spectrumscale.org>
<b>On Behalf Of </b>Sven Oehme<br>
<b>Sent:</b> Thursday, February 20, 2020 17:29<br>
<b>To:</b> gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
<b>Subject:</b> Re: [gpfsug-discuss] Unkillable snapshots<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p><span style="font-size:9.0pt;font-family:"Verdana",sans-serif;color:#CC0000">This message was sent by an external party.<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Filesystem quiesce failed has nothing to do with open files.
<o:p></o:p></p>
<div>
<p class="MsoNormal">What it means is that the filesystem couldn’t flush dirty data and metadata within a defined time to take a snapshot. This can be caused by to high maxfilestocache or pagepool settings. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">To give you an simplified example (its more complex than that, but good enough to make the point) - assume you have 100 nodes, each has 16 GB pagepool and your storage system can write data out at 10 GB/sec, it will take 160 seconds to
flush all data data (assuming you did normal buffered I/O.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">If i remember correct (talking out of memory here) the default timeout is 60 seconds, given that you can’t write that fast it will always timeout under this scenario. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">There is one case where this can also happen which is a client is connected badly (flaky network or slow connection) and even your storage system is fast enough the node is too slow that it can’t de-stage within
that time while everybody else can and the storage is not the bottleneck. Other than that</span> only solutions are to a) buy faster storage or b) reduce pagepool and maxfilestocache which will reduce overall performance of the system.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Sven<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Sent from my iPad<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-bottom:12.0pt">On Feb 20, 2020, at 5:14 PM, Nathan Falk <<a href="mailto:nfalk@us.ibm.com">nfalk@us.ibm.com</a>> wrote:<o:p></o:p></p>
</blockquote>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-family:"Tahoma",sans-serif"></span><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Good point, Simon. Yes, it is a "file system quiesce" not a "fileset quiesce" so it is certainly
possible that mmfsd is unable to quiesce because there are processes keeping files open in another fileset.<br>
</span><br>
<br>
<o:p></o:p></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="0" style="width:487.5pt;border-collapse:collapse">
<tbody>
<tr style="height:6.0pt">
<td width="650" style="width:487.5pt;padding:0in 0in 0in 0in;height:6.0pt">
<p class="MsoNormal"><b><span style="font-family:"Arial",sans-serif;color:#8F8F8F">Nate Falk</span></b><span style="font-size:9.0pt;font-family:"Arial",sans-serif"><br>
IBM Spectrum Scale Level 2 Support<br>
Software Defined Infrastructure, IBM Systems</span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><span style="display:none"><o:p> </o:p></span></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="0" style="width:487.5pt;border-collapse:collapse">
<tbody>
<tr style="height:6.0pt">
<td width="650" colspan="2" style="width:487.5pt;padding:0in 0in 0in 0in;height:6.0pt">
</td>
</tr>
<tr style="height:6.0pt">
<td width="363" valign="top" style="width:272.25pt;padding:0in 0in 0in 0in;height:6.0pt">
</td>
<td width="287" valign="top" style="width:214.5pt;padding:0in 0in 0in 0in;height:6.0pt">
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><br>
<br>
<br>
<br>
<span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#5F5F5F">From: </span><span style="font-size:9.0pt;font-family:"Arial",sans-serif">Simon Thompson <<a href="mailto:S.J.Thompson@bham.ac.uk">S.J.Thompson@bham.ac.uk</a>></span><br>
<span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#5F5F5F">To: </span><span style="font-size:9.0pt;font-family:"Arial",sans-serif">gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a>></span><br>
<span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#5F5F5F">Date: </span><span style="font-size:9.0pt;font-family:"Arial",sans-serif">02/20/2020 04:39 PM</span><br>
<span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#5F5F5F">Subject: </span><span style="font-size:9.0pt;font-family:"Arial",sans-serif">[EXTERNAL] Re: [gpfsug-discuss] Unkillable snapshots</span><br>
<span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#5F5F5F">Sent by: </span><span style="font-size:9.0pt;font-family:"Arial",sans-serif"><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-bounces@spectrumscale.org</a></span><o:p></o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" noshade="" style="color:#A0A0A0" align="center">
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Calibri",sans-serif">Hi Nate,</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Calibri",sans-serif">So we're trying to clean up snapshots from the GUI ... we've found that if it fails to delete one night for whatever reason, it then doesn't go back another day and clean
up </span><span style="font-family:"Segoe UI Symbol",sans-serif">😊</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Calibri",sans-serif">But yes, essentially running this by hand to clean up.</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Calibri",sans-serif">What I have found is that lsof hangs on some of the "suspect" nodes. But if I strace it, its hanging on a process which is using a different fileset. For example, the
file-set we can't delete is:</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Calibri",sans-serif">rds-projects-b which is mounted as /rds/projects/b</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Calibri",sans-serif">But on some suspect nodes, strace lsof /rds, that hangs at a process which has open files in:</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Calibri",sans-serif">/rds/projects/g which is a different file-set.</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Calibri",sans-serif">What I'm wondering if its these hanging processes in the "g" fileset which is killing us rather than something in the "b" fileset. Looking at the "g" processes, they
look like a weather model and look to be dumping a lot of files in a shared directory, so I wonder if the mmfsd process is busy servicing that and so whilst its not got "b" locks, its just too slow to respond?</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Calibri",sans-serif">Does that sound plausible?</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Calibri",sans-serif">Thanks</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Calibri",sans-serif">Simon</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" align="center">
</div>
<p class="MsoNormal"><br>
<b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">
<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-bounces@spectrumscale.org</a> <<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-bounces@spectrumscale.org</a>> on behalf of
<a href="mailto:nfalk@us.ibm.com">nfalk@us.ibm.com</a> <<a href="mailto:nfalk@us.ibm.com">nfalk@us.ibm.com</a>><b><br>
Sent:</b> 20 February 2020 21:26:39<b><br>
To:</b> gpfsug main discussion list<b><br>
Subject:</b> Re: [gpfsug-discuss] Unkillable snapshots</span><br>
<br>
<span style="font-size:10.0pt">Hello Simon,</span><br>
<span style="font-size:10.0pt"><br>
Sadly, that "1036" is not a node ID, but just a counter.</span><br>
<span style="font-size:10.0pt"><br>
These are tricky to troubleshoot. Usually, by the time you realize it's happening and try to collect some data, things have already timed out.</span><br>
<span style="font-size:10.0pt"><br>
Since this mmdelsnapshot isn't something that's on a schedule from cron or the GUI and is a command you are running, you could try some heavy-handed data collection.</span><br>
<span style="font-size:10.0pt"><br>
You suspect a particular fileset already, so maybe have a 'mmdsh -N all lsof /path/to/fileset' ready to go in one window, and the 'mmdelsnapshot' ready to go in another window? When the mmdelsnapshot times out, you can find the nodes it was waiting on in the
file system manager mmfs.log.latest and see what matches up with the open files identified by lsof.</span><br>
<span style="font-size:10.0pt"><br>
It sounds like you already know this, but the <c0n42> type of internal node names in the log messages can be translated with 'mmfsadm dump tscomm' or also plain old 'mmdiag --network'.</span><br>
<span style="font-size:10.0pt"><br>
Thanks,</span><o:p></o:p></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="0" style="width:487.5pt;border-collapse:collapse">
<tbody>
<tr style="height:6.0pt">
<td width="650" style="width:6.75in;padding:.75pt .75pt .75pt .75pt;height:6.0pt">
<p class="MsoNormal"><b><span style="font-family:"Arial",sans-serif;color:#8F8F8F">Nate Falk</span></b><span style="font-size:9.0pt;font-family:"Arial",sans-serif"><br>
IBM Spectrum Scale Level 2 Support<br>
Software Defined Infrastructure, IBM Systems</span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><o:p> </o:p></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="0" style="width:487.5pt;border-collapse:collapse">
<tbody>
<tr style="height:6.0pt">
<td width="650" colspan="2" style="width:6.75in;padding:.75pt .75pt .75pt .75pt;height:6.0pt">
</td>
</tr>
<tr style="height:6.0pt">
<td width="362" valign="top" style="width:270.75pt;padding:.75pt .75pt .75pt .75pt;height:6.0pt">
</td>
<td width="288" valign="top" style="width:213.0pt;padding:.75pt .75pt .75pt .75pt;height:6.0pt">
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><br>
<br>
<br>
<br>
<span style="font-size:9.0pt;color:#5F5F5F"><br>
From: </span><span style="font-size:9.0pt">Simon Thompson <<a href="mailto:S.J.Thompson@bham.ac.uk">S.J.Thompson@bham.ac.uk</a>><span style="color:#5F5F5F"><br>
To: </span>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a>><span style="color:#5F5F5F"><br>
Date: </span>02/20/2020 03:14 PM<span style="color:#5F5F5F"><br>
Subject: </span>[EXTERNAL] Re: [gpfsug-discuss] Unkillable snapshots<span style="color:#5F5F5F"><br>
Sent by: </span><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-bounces@spectrumscale.org</a></span><o:p></o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" noshade="" style="color:#A0A0A0" align="center">
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Calibri",sans-serif">Hmm ... mmdiag --tokenmgr shows:</span><o:p></o:p></p>
<p class="MsoNormal"><br>
<span style="font-family:"Calibri",sans-serif"><br>
Server stats: requests 195417431 ServerSideRevokes 120140<br>
nTokens 2146923 nranges 4124507<br>
designated mnode appointed 55481 mnode thrashing detected 1036</span> <o:p>
</o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Calibri",sans-serif">So how do I convert "1036" to a node?</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Calibri",sans-serif">Simon</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" align="center">
</div>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><br>
From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">
gpfsug-discuss-bounces@spectrumscale.org</a> <<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-bounces@spectrumscale.org</a>> on behalf of Simon Thompson <<a href="mailto:S.J.Thompson@bham.ac.uk">S.J.Thompson@bham.ac.uk</a>><b><br>
Sent:</b> 20 February 2020 19:45:02<b><br>
To:</b> gpfsug main discussion list<b><br>
Subject:</b> [gpfsug-discuss] Unkillable snapshots</span><br>
<o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Calibri",sans-serif">Hi,</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Calibri",sans-serif">We have a snapshot which is stuck in the state "DeleteRequired". When deleting, it goes through the motions but eventually gives up with:</span><o:p></o:p></p>
<p class="MsoNormal"><br>
<span style="font-family:"Calibri",sans-serif"><br>
Unable to quiesce all nodes; some processes are busy or holding required resources.<br>
mmdelsnapshot: Command failed. Examine previous error messages to determine cause.</span>
<o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Calibri",sans-serif">And in the mmfslog on the FS manager there are a bunch of retries and "failure to quesce" on nodes. However in each retry its never the same set of nodes. I suspect we
have one HPC job somewhere killing us.</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Calibri",sans-serif">What's interesting is that we can delete other snapshots OK, it appears to be one particular fileset.</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Calibri",sans-serif">My old goto "mmfsadm dump tscomm" isn't showing any particular node, and waiters around just tend to point to the FS manager node.</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Calibri",sans-serif">So ... any suggestions? I'm assuming its some workload holding a lock open or some such, but tracking it down is proving elusive!</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Calibri",sans-serif">Generally the FS is also "lumpy" ... at times it feels like a wifi connection on a train using a terminal, I guess its all related though.</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Calibri",sans-serif">Thanks</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Calibri",sans-serif">Simon
</span><o:p></o:p></p>
<p class="MsoNormal"><br>
<span style="font-size:10.0pt;font-family:"Courier New""><br>
<tt>_______________________________________________</tt><br>
<tt>gpfsug-discuss mailing list</tt><br>
<tt>gpfsug-discuss at spectrumscale.org</tt></span><u><span style="color:blue"><br>
</span></u><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><span style="font-size:10.0pt">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></tt></a><br>
<br>
<br>
<br>
<tt><span style="font-size:10.0pt">_______________________________________________</span></tt><span style="font-size:10.0pt;font-family:"Courier New""><br>
<tt>gpfsug-discuss mailing list</tt><br>
<tt>gpfsug-discuss at spectrumscale.org</tt><br>
</span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><span style="font-size:10.0pt">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></tt></a><span style="font-size:10.0pt;font-family:"Courier New""><br>
</span><br>
<br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><o:p></o:p></p>
</div>
</blockquote>
</div>
</div>
</div>
</body>
</html>