<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Got my vote - thanks Robert. <br class="">
<div class="">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br class="Apple-interchange-newline">
<br class="">
</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
Gordon McPheeters</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
ALCF Storage</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
(630) 252-6430</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<a href="mailto:gmcpheeters@anl.gov" class="">gmcpheeters@anl.gov</a></div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br class="">
</div>
<br class="Apple-interchange-newline">
</div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Aug 18, 2016, at 10:00 AM, Bryan Banister <<a href="mailto:bbanister@jumptrading.com" class="">bbanister@jumptrading.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="WordSection1" style="page: WordSection1; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">Great stuff… I added my vote,<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">-Bryan<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div>
<div class="">
<div style="border-style: solid none none; border-top-color: rgb(181, 196, 223); border-top-width: 1pt; padding: 3pt 0in 0in;" class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<b class=""><span style="font-size: 10pt; font-family: Tahoma, sans-serif;" class="">From:</span></b><span style="font-size: 10pt; font-family: Tahoma, sans-serif;" class=""><span class="Apple-converted-space"> </span><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" style="color: purple; text-decoration: underline;" class="">gpfsug-discuss-bounces@spectrumscale.org</a><span class="Apple-converted-space"> </span>[<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" style="color: purple; text-decoration: underline;" class="">mailto:gpfsug-discuss-bounces@spectrumscale.org</a>]<span class="Apple-converted-space"> </span><b class="">On
Behalf Of<span class="Apple-converted-space"> </span></b>Oesterlin, Robert<br class="">
<b class="">Sent:</b><span class="Apple-converted-space"> </span>Thursday, August 18, 2016 9:47 AM<br class="">
<b class="">To:</b><span class="Apple-converted-space"> </span>gpfsug main discussion list<br class="">
<b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [gpfsug-discuss] Monitor NSD server queue?<o:p class=""></o:p></span></div>
</div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Helvetica, sans-serif;" class="">Done.<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Helvetica, sans-serif;" class="">Notification generated at: 18 Aug 2016, 10:46 AM Eastern Time (ET)<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Helvetica, sans-serif;" class="">ID: 93260<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Helvetica, sans-serif;" class="">Headline: Give sysadmin insight into the inner workings of the NSD server machinery, in particular the queue dynamics<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Helvetica, sans-serif;" class="">Submitted on: 18 Aug 2016, 10:46 AM Eastern Time (ET)<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Helvetica, sans-serif;" class="">Brand: Servers and Systems Software<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Helvetica, sans-serif;" class="">Product: Spectrum Scale (formerly known as GPFS) - Public RFEs<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Helvetica, sans-serif;" class="">Link: <a href="http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=93260" style="color: purple; text-decoration: underline;" class=""><span style="color: rgb(0, 0, 233);" class="">http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=93260</span></a><o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></span></div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 10.5pt; font-family: Helvetica, sans-serif;" class="">Bob Oesterlin<br class="">
Sr Storage Engineer, Nuance HPC Grid<br class="">
507-269-0413<o:p class=""></o:p></span></div>
</div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div>
<div style="border-style: solid none none; border-top-color: rgb(181, 196, 223); border-top-width: 1pt; padding: 3pt 0in 0in;" class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<b class=""><span style="font-family: Calibri, sans-serif;" class="">From:<span class="Apple-converted-space"> </span></span></b><span style="font-family: Calibri, sans-serif;" class=""><<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" style="color: purple; text-decoration: underline;" class="">gpfsug-discuss-bounces@spectrumscale.org</a>>
on behalf of Yuri L Volobuev <<a href="mailto:volobuev@us.ibm.com" style="color: purple; text-decoration: underline;" class="">volobuev@us.ibm.com</a>><br class="">
<b class="">Reply-To:<span class="Apple-converted-space"> </span></b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" style="color: purple; text-decoration: underline;" class="">gpfsug-discuss@spectrumscale.org</a>><br class="">
<b class="">Date:<span class="Apple-converted-space"> </span></b>Wednesday, August 17, 2016 at 3:34 PM<br class="">
<b class="">To:<span class="Apple-converted-space"> </span></b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" style="color: purple; text-decoration: underline;" class="">gpfsug-discuss@spectrumscale.org</a>><br class="">
<b class="">Subject:<span class="Apple-converted-space"> </span></b>[EXTERNAL] Re: [gpfsug-discuss] Monitor NSD server queue?<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div class="">
<p style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
Unfortunately, at the moment there's no safe mechanism to show the usage statistics for different NSD queues. "mmfsadm saferdump nsd" as implemented doesn't acquire locks when parsing internal data structures. Now, NSD data structures are fairly static, as
much things go, so the risk of following a stale pointer and hitting a segfault isn't particularly significant. I don't think I remember ever seeing mmfsd crash with NSD dump code on the stack. That said, this isn't code that's tested and known to be safe
for production use. I haven't seen a case myself where an mmfsd thread gets stuck running this dump command, either, but Bob has. If that condition ever reoccurs, I'd be interested in seeing debug data.<br class="">
<br class="">
I agree that there's value in giving a sysadmin insight into the inner workings of the NSD server machinery, in particular the queue dynamics. mmdiag should be enhanced to allow this. That'd be a very reasonable (and doable) RFE.<br class="">
<br class="">
yuri<br class="">
<br class="">
<span id="cid:image001.gif@01D1F937.54ADAA40"><image001.gif></span><span style="color: rgb(66, 66, 130);" class="">"Oesterlin, Robert" ---08/17/2016 04:45:30 AM---Hi Aaron You did a perfect job of explaining a situation I've run into time after time - high
latenc</span><br class="">
<br class="">
<span style="font-size: 10pt; color: rgb(95, 95, 95);" class="">From:<span class="Apple-converted-space"> </span></span><span style="font-size: 10pt;" class="">"Oesterlin, Robert" <<a href="mailto:Robert.Oesterlin@nuance.com" style="color: purple; text-decoration: underline;" class="">Robert.Oesterlin@nuance.com</a>></span><br class="">
<span style="font-size: 10pt; color: rgb(95, 95, 95);" class="">To:<span class="Apple-converted-space"> </span></span><span style="font-size: 10pt;" class="">gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" style="color: purple; text-decoration: underline;" class="">gpfsug-discuss@spectrumscale.org</a>>,<span class="Apple-converted-space"> </span></span><br class="">
<span style="font-size: 10pt; color: rgb(95, 95, 95);" class="">Date:<span class="Apple-converted-space"> </span></span><span style="font-size: 10pt;" class="">08/17/2016 04:45 AM</span><br class="">
<span style="font-size: 10pt; color: rgb(95, 95, 95);" class="">Subject:<span class="Apple-converted-space"> </span></span><span style="font-size: 10pt;" class="">Re: [gpfsug-discuss] Monitor NSD server queue?</span><br class="">
<span style="font-size: 10pt; color: rgb(95, 95, 95);" class="">Sent by:<span class="Apple-converted-space"> </span></span><span style="font-size: 10pt;" class=""><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" style="color: purple; text-decoration: underline;" class="">gpfsug-discuss-bounces@spectrumscale.org</a></span><o:p class=""></o:p></p>
<div class="">
<div class="MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<hr size="2" width="100%" noshade="" align="left" style="color: rgb(128, 145, 165);" class="">
</div>
</div>
<p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<br class="">
<br class="">
<br class="">
<span style="font-family: Arial, sans-serif;" class="">Hi Aaron</span><br class="">
<br class="">
<span style="font-family: Arial, sans-serif;" class="">You did a perfect job of explaining a situation I've run into time after time - high latency on the disk subsystem causing a backup in the NSD queues. I was doing what you suggested not to do - "mmfsadm
saferdump nsd' and looking at the queues. In my case 'mmfsadm saferdump" would usually work or hang, rather than kill mmfsd. But - the hang usually resulted it a tied up thread in mmfsd, so that's no good either.</span><br class="">
<br class="">
<span style="font-family: Arial, sans-serif;" class="">I wish I had better news - this is the only way I've found to get visibility to these queues. IBM hasn't seen fit to gives us a way to safely look at these. I personally think it's a bug that we can't safely
dump these structures, as they give insight as to what's actually going on inside the NSD server.<span class="Apple-converted-space"> </span></span><br class="">
<br class="">
<span style="font-family: Arial, sans-serif;" class="">Yuri, Sven - thoughts?</span><br class="">
<br class="">
<br class="">
<span style="font-family: Arial, sans-serif;" class="">Bob Oesterlin<br class="">
Sr Storage Engineer, Nuance HPC Grid<br class="">
</span><br class="">
<br class="">
<br class="">
<b class=""><span style="font-size: 13.5pt; font-family: Calibri, sans-serif;" class="">From:<span class="Apple-converted-space"> </span></span></b><span style="font-size: 13.5pt; font-family: Calibri, sans-serif;" class=""><<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" style="color: purple; text-decoration: underline;" class="">gpfsug-discuss-bounces@spectrumscale.org</a>>
on behalf of "Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]" <<a href="mailto:aaron.s.knister@nasa.gov" style="color: purple; text-decoration: underline;" class="">aaron.s.knister@nasa.gov</a>><b class=""><br class="">
Reply-To:<span class="Apple-converted-space"> </span></b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" style="color: purple; text-decoration: underline;" class="">gpfsug-discuss@spectrumscale.org</a>><b class=""><br class="">
Date:<span class="Apple-converted-space"> </span></b>Tuesday, August 16, 2016 at 8:46 PM<b class=""><br class="">
To:<span class="Apple-converted-space"> </span></b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" style="color: purple; text-decoration: underline;" class="">gpfsug-discuss@spectrumscale.org</a>><b class=""><br class="">
Subject:<span class="Apple-converted-space"> </span></b>[EXTERNAL] [gpfsug-discuss] Monitor NSD server queue?</span><br class="">
<br class="">
<span style="font-size: 13.5pt;" class="">Hi Everyone,<span class="Apple-converted-space"> </span></span><br class="">
<br class="">
<span style="font-size: 13.5pt;" class="">We ran into a rather interesting situation over the past week. We had a job that was pounding the ever loving crap out of one of our filesystems (called dnb02) doing about 15GB/s of reads. We had other jobs experience
a slowdown on a different filesystem (called dnb41) that uses entirely separate backend storage. What I can't figure out is why this other filesystem was affected. I've checked IB bandwidth and congestion, Fibre channel bandwidth and errors, Ethernet bandwidth
congestion, looked at the mmpmon nsd_ds counters (including disk request wait time), and checked out the disk iowait values from collectl. I simply can't account for the slowdown on the other filesystem. The only thing I can think of is the high latency on
dnb02's NSDs caused the mmfsd NSD queues to back up.<span class="Apple-converted-space"> </span></span><br class="">
<br class="">
<span style="font-size: 13.5pt;" class="">Here's my question-- how can I monitor the state of th NSD queues? I can't find anything in mmdiag. An mmfsadm saferdump NSD shows me the queues and their status. I'm just not sure calling saferdump NSD every 10 seconds
to monitor this data is going to end well. I've seen saferdump NSD cause mmfsd to die and that's from a task we only run every 6 hours that calls saferdump NSD.<span class="Apple-converted-space"> </span></span><br class="">
<br class="">
<span style="font-size: 13.5pt;" class="">Any thoughts/ideas here would be great.<span class="Apple-converted-space"> </span></span><br class="">
<br class="">
<span style="font-size: 13.5pt;" class="">Thanks!</span><br class="">
<br class="">
<span style="font-size: 13.5pt;" class="">-Aaron</span><tt style="font-family: 'Courier New';" class=""><span style="font-size: 10pt;" class="">_______________________________________________</span></tt><span style="font-size: 10pt; font-family: 'Courier New';" class=""><br class="">
<tt style="font-family: 'Courier New';" class="">gpfsug-discuss mailing list</tt><br class="">
<tt style="font-family: 'Courier New';" class="">gpfsug-discuss at<span class="Apple-converted-space"> </span><a href="http://spectrumscale.org/" style="color: purple; text-decoration: underline;" class="">spectrumscale.org</a></tt><br class="">
<tt style="font-family: 'Courier New';" class=""><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwMFAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=D8iCz340ioiUrtGkAFdKjfgfitPkpOr1nRkkxTRCBn0&s=ncd-C59bavCSUTkgYH1vH4ewOM12Hajhy-KhFtKZK68&e=" style="color: purple; text-decoration: underline;" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></tt></span><o:p class=""></o:p></p>
</div>
</div>
</div>
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<hr style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<font face="Arial" color="Gray" size="1" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class=""><br class="">
Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this
email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness
or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial
product.<br class="">
</font><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;" class=""></span><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;" class="">gpfsug-discuss
mailing list</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;" class="">gpfsug-discuss
at<span class="Apple-converted-space"> </span></span><a href="http://spectrumscale.org/" style="color: purple; text-decoration: underline; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">spectrumscale.org</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" style="color: purple; text-decoration: underline; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
</div>
</blockquote>
</div>
<br class="">
</body>
</html>