<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi All,
<div class=""><br class="">
</div>
<div class="">Another update on this issue as we have made significant progress today … but first let me address the two responses I received.</div>
<div class=""><br class="">
</div>
<div class="">Alex - this is a good idea and yes, we did this today. We did see some higher latencies on one storage array as compared to the others. 10-20 ms on the “good” storage arrays … 50-60 ms on the one storage array. It took us a while to be able
to do this because while the vendor provides a web management interface, that didn’t show this information. But they have an actual app that will … and the Mac and Linux versions don’t work. So we had to go scrounge up this thing called a Windows PC and
get the software installed there. ;-)</div>
<div class=""><br class="">
</div>
<div class="">Jonathan - also a good idea and yes, we also did this today. I’ll explain as part of the rest of this update.</div>
<div class=""><br class="">
</div>
<div class="">The main thing that we did today that has turned out to be most revealing is to take a list of all the NSDs in the impacted storage pool … 19 devices spread out over 7 storage arrays … and run read dd tests on all of them (the /dev/dm-XX multipath
device). 15 of them showed rates of 33 - 100+ MB/sec and the variation is almost definitely explained by the fact that they’re in production use and getting hit by varying amounts of “real” work. But 4 of them showed rates of 2-10 MB/sec and those 4 all
happen to be on storage array eon34.</div>
<div class=""><br class="">
</div>
<div class="">So, to try to rule out everything but the storage array we replaced the FC cables going from the SAN switches to the array, plugging the new cables into different ports on the SAN switches. Then we repeated the dd tests from a different NSD server,
which both eliminated the NSD server and its’ FC cables as a potential cause … and saw results virtually identical to the previous test. Therefore, we feel pretty confident that it is the storage array and have let the vendor know all of this.</div>
<div class=""><br class="">
</div>
<div class="">And there’s another piece of quite possibly relevant info … the last week in May one of the controllers in this array crashed and rebooted (it’s a active-active dual controller array) … when that happened the failover occurred … with a major glitch.
One of the LUNs essentially disappeared … more accurately, it was there, but had no size! We’ve been using this particular vendor for 15 years now and I have seen more than a couple of their controllers go bad during that time and nothing like this had ever
happened before. They were never able to adequately explain what happened there. So what I am personally suspecting has happened is that whatever caused that one LUN to go MIA has caused these issues with the other LUNs on the array. As an aside, we ended
up using mmfileid to identify the files that had blocks on the MIA LUN and restored those from tape backup.</div>
<div class=""><br class="">
</div>
<div class="">I want to thank everyone who has offered their suggestions so far. I will update the list again once we have a definitive problem determination.</div>
<div class=""><br class="">
</div>
<div class="">I hope that everyone has a great weekend. In the immortal words of the wisest man who ever lived, “I’m kinda tired … think I’ll go home now.” ;-)</div>
<div class=""><br class="">
</div>
<div class="">Kevin</div>
<div class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Jul 6, 2018, at 12:13 PM, Alex Chekholko <<a href="mailto:alex@calicolabs.com" class="">alex@calicolabs.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">
<div class="">Hi Kevin,</div>
<div class=""><br class="">
</div>
This is a bit of a "cargo cult" suggestion but one issue that I have seen is if a disk starts misbehaving a bit but does not fail, it slows down the whole raid group that it is in. And the only way to detect it is to examine the read/write latencies on the
individual disks. Does your SAN allow you to do that?
<div class=""><br class="">
</div>
<div class="">That happened to me at least twice in my life and replacing the offending individual disk solved the issue. This was on DDN, so the relevant command were something like 'show pd * counters write_lat' or similar, which showed the latency for the
I/Os for each disk. If one disk in the group is an outlier (e.g. 1s write latencies), then the whole raid array (LUN) is just waiting for that one disk.</div>
<div class=""><br class="">
</div>
<div class="">Another possibility for troubleshooting, if you have sufficient free resources: you can just suspend the problematic LUNs in GPFS, as that will remove the write load from them, while still having them service read requests and not affecting users.</div>
<div class=""><br class="">
</div>
<div class="">Regards,</div>
<div class="">Alex</div>
</div>
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="">On Fri, Jul 6, 2018 at 9:11 AM Buterbaugh, Kevin L <<a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class="">Kevin.Buterbaugh@vanderbilt.edu</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word;line-break:after-white-space" class="">Hi Jim,
<div class=""><br class="">
</div>
<div class="">
<div class="">Thank you for your response. We are taking a two-pronged approach at this point:</div>
<div class=""><br class="">
</div>
<div class="">1. While I don’t see anything wrong with our storage arrays, I have opened a ticket with the vendor (not IBM) to get them to look at things from that angle.</div>
<div class=""><br class="">
</div>
<div class="">2. Since the problem moves around from time to time, we are enhancing our monitoring script to see if we can basically go from “mmdiag —iohist” to “clients issuing those I/O requests” to “jobs running on those clients” to see if there is any
commonality there.</div>
<div class=""><br class="">
</div>
<div class="">Thanks again - much appreciated!</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">—</div>
<div class="">Kevin Buterbaugh - Senior System Administrator</div>
<div class="">Vanderbilt University - Advanced Computing Center for Research and Education</div>
<div class=""><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" target="_blank" class="">Kevin.Buterbaugh@vanderbilt.edu</a> - (615)875-9633</div>
<div class=""><br class="">
</div>
</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Jul 6, 2018, at 8:02 AM, Jim Doherty <<a href="mailto:jjdoherty@yahoo.com" target="_blank" class="">jjdoherty@yahoo.com</a>> wrote:</div>
<br class="m_-237379699367396637Apple-interchange-newline">
<div class="">
<div class="">
<div style="font-family:Helvetica Neue,Helvetica,Arial,sans-serif;font-size:13px" class="">
<div style="font-family:Helvetica Neue,Helvetica,Arial,sans-serif;font-size:13px" class="">
<div class="">You may want to get an mmtrace, but I suspect that the disk IOs are slow. The iohist is showing the time from when the start IO was issued until it was finished. Of course if you have disk IOs taking 10x too long then other IOs are going
to queue up behind it. If there are more IOs than there are NSD server threads then there are going to be IOs that are queued and waiting for a thread.
<br class="">
<div class=""><br class="">
Jim<br class="">
</div>
</div>
<div class=""></div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
</div>
<div id="m_-237379699367396637yahoo_quoted_1621782022" class="m_-237379699367396637yahoo_quoted">
<div style="font-family:'Helvetica Neue',Helvetica,Arial,sans-serif;font-size:13px;color:#26282a" class="">
<div class="">On Thursday, July 5, 2018, 9:30:30 PM EDT, Buterbaugh, Kevin L <<a href="mailto:Kevin.Buterbaugh@Vanderbilt.Edu" target="_blank" class="">Kevin.Buterbaugh@Vanderbilt.Edu</a>> wrote:
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">
<div dir="ltr" class="">Hi All,<br clear="none" class="">
<br clear="none" class="">
First off, my apologies for the delay in responding back to the list … we’ve actually been working our tails off on this one trying to collect as much data as we can on what is a very weird issue. While I’m responding to Aaron’s e-mail, I’m going to try to
address the questions raised in all the responses.<br clear="none" class="">
<br clear="none" class="">
Steve - this all started last week. You’re correct about our mixed workload. There have been no new workloads that I am aware of.<br clear="none" class="">
<br clear="none" class="">
Stephen - no, this is not an ESS. We are running GPFS 4.2.3-8.<br clear="none" class="">
<br clear="none" class="">
Aaron - no, this is not on a DDN, either.<br clear="none" class="">
<br clear="none" class="">
The hardware setup is a vanilla 8 GB FC SAN. Commodity hardware for the servers and storage. We have two SAN “stacks” and all NSD servers and storage are connected to both stacks. Linux multipathing handles path failures. 10 GbE out to the network.<br clear="none" class="">
<br clear="none" class="">
We first were alerted to this problem by one of our monitoring scripts which was designed to alert us to abnormally high I/O times, which, as I mentioned previously, in our environment has usually been caused by cache battery backup failures in the storage
array controllers (but _not_ this time). So I’m getting e-mails that in part read:<br clear="none" class="">
<br clear="none" class="">
Disk eon34Cnsd on nsd2 has a service time of 4625.083 ms.<br clear="none" class="">
Disk eon34Ensd on nsd4 has a service time of 3146.715 ms.<br clear="none" class="">
<br clear="none" class="">
The “34” tells me what storage array and the “C” or “E” tells me what LUN on that storage array. As I’ve mentioned, those two LUNs are by far and away my most frequent problem children, but here’s another report from today as well:<br clear="none" class="">
<br clear="none" class="">
Disk eon28Bnsd on nsd8 has a service time of 1119.385 ms.<br clear="none" class="">
Disk eon28Ansd on nsd7 has a service time of 1154.002 ms.<br clear="none" class="">
Disk eon31Ansd on nsd3 has a service time of 1068.987 ms.<br clear="none" class="">
Disk eon34Cnsd on nsd2 has a service time of 4991.365 ms.<br clear="none" class="">
<br clear="none" class="">
NSD server hostnames have been changed, BTW, from their real names to nsd1 - 8.<br clear="none" class="">
<br clear="none" class="">
Based on Fred’s excellent advice, we took a closer look at the “mmfsadm dump nsd” output. We wrote a Python script to pull out what we think is the most pertinent information:<br clear="none" class="">
<br clear="none" class="">
nsd1<br clear="none" class="">
29 SMALL queues, 50 requests pending, 3741 was the highest number of requests pending.<br clear="none" class="">
348 threads started, 1 threads active, 348 was the highest number of threads active.<br clear="none" class="">
29 LARGE queues, 0 requests pending, 5694 was the highest number of requests pending.<br clear="none" class="">
348 threads started, 124 threads active, 348 was the highest number of threads active.<br clear="none" class="">
nsd2<br clear="none" class="">
29 SMALL queues, 0 requests pending, 1246 was the highest number of requests pending.<br clear="none" class="">
348 threads started, 13 threads active, 348 was the highest number of threads active.<br clear="none" class="">
29 LARGE queues, 470 requests pending, 2404 was the highest number of requests pending.<br clear="none" class="">
348 threads started, 340 threads active, 348 was the highest number of threads active.<br clear="none" class="">
nsd3<br clear="none" class="">
29 SMALL queues, 108 requests pending, 1796 was the highest number of requests pending.<br clear="none" class="">
348 threads started, 0 threads active, 348 was the highest number of threads active.<br clear="none" class="">
29 LARGE queues, 35 requests pending, 3331 was the highest number of requests pending.<br clear="none" class="">
348 threads started, 4 threads active, 348 was the highest number of threads active.<br clear="none" class="">
nsd4<br clear="none" class="">
42 SMALL queues, 0 requests pending, 1529 was the highest number of requests pending.<br clear="none" class="">
504 threads started, 8 threads active, 504 was the highest number of threads active.<br clear="none" class="">
42 LARGE queues, 0 requests pending, 637 was the highest number of requests pending.<br clear="none" class="">
504 threads started, 211 threads active, 504 was the highest number of threads active.<br clear="none" class="">
nsd5<br clear="none" class="">
42 SMALL queues, 182 requests pending, 2798 was the highest number of requests pending.<br clear="none" class="">
504 threads started, 6 threads active, 504 was the highest number of threads active.<br clear="none" class="">
42 LARGE queues, 407 requests pending, 4416 was the highest number of requests pending.<br clear="none" class="">
504 threads started, 8 threads active, 504 was the highest number of threads active.<br clear="none" class="">
nsd6<br clear="none" class="">
42 SMALL queues, 0 requests pending, 1630 was the highest number of requests pending.<br clear="none" class="">
504 threads started, 0 threads active, 504 was the highest number of threads active.<br clear="none" class="">
42 LARGE queues, 0 requests pending, 148 was the highest number of requests pending.<br clear="none" class="">
504 threads started, 9 threads active, 504 was the highest number of threads active.<br clear="none" class="">
nsd7<br clear="none" class="">
42 SMALL queues, 43 requests pending, 2179 was the highest number of requests pending.<br clear="none" class="">
504 threads started, 1 threads active, 504 was the highest number of threads active.<br clear="none" class="">
42 LARGE queues, 0 requests pending, 2551 was the highest number of requests pending.<br clear="none" class="">
504 threads started, 13 threads active, 504 was the highest number of threads active.<br clear="none" class="">
nsd8<br clear="none" class="">
42 SMALL queues, 0 requests pending, 1014 was the highest number of requests pending.<br clear="none" class="">
504 threads started, 4 threads active, 504 was the highest number of threads active.<br clear="none" class="">
42 LARGE queues, 0 requests pending, 3371 was the highest number of requests pending.<br clear="none" class="">
504 threads started, 89 threads active, 504 was the highest number of threads active.<br clear="none" class="">
<br clear="none" class="">
Note that we see more “load” on the LARGE queue side of things and that nsd2 and nsd4 (the primary NSD servers for the 2 LUNs that show up most frequently in our alerts) are the heaviest loaded.<br clear="none" class="">
<br clear="none" class="">
One other thing we have noted is that our home grown RRDtool monitoring plots that are based on netstat, iostat, vmstat, etc. also show an oddity. Most of our LUNs show up as 33 - 68% utilized … but all the LUNs on eon34 (there are 4 in total) show up as 93
- 97% utilized. And another oddity there is that eon34A and eon34B rarely show up on the alert e-mails, while eon34C and eon34E show up waaaayyyyyyy more than anything else … the difference between them is that A and B are on the storage array itself and
C and E are on JBOD’s SAS-attached to the storage array (and yes, we’ve actually checked and reseated those connections).<br clear="none" class="">
<br clear="none" class="">
Another reason why I could not respond earlier today is that one of the things which I did this afternoon was to upgrade the RAM on nsd2 and nsd4 from 16 / 24 GB respectively to 64 GB each … and I then upped the pagepool on those two boxes to 40 GB. That has
not made a difference. How can I determine how much of the pagepool is actually being used, BTW? A quick Google search didn’t help me.<br clear="none" class="">
<br clear="none" class="">
So we’re trying to figure out if we have storage hardware issues causing GPFS issues or GPFS issues causing storage slowdowns. The fact that I see slowdowns most often on one storage array points in one direction, while the fact that at times I see even worse
slowdowns on multiple other arrays points the other way. The fact that some NSD servers show better stats than others in the analysis of the “mmfsadm dump nsd” output tells me … well, I don’t know what it tells me.<br clear="none" class="">
<br clear="none" class="">
I think that’s all for now. If you have read this entire very long e-mail, first off, thank you! If you’ve read it and have ideas for where I should go from here, T-H-A-N-K Y-O-U!<br clear="none" class="">
<br clear="none" class="">
Kevin<br clear="none" class="">
<br clear="none" class="">
> On Jul 4, 2018, at 7:34 AM, Aaron Knister <<a shape="rect" href="mailto:aaron.s.knister@nasa.gov" target="_blank" class="">aaron.s.knister@nasa.gov</a>> wrote:<br clear="none" class="">
> <br clear="none" class="">
> Hi Kevin,<br clear="none" class="">
> <br clear="none" class="">
> Just going out on a very weird limb here...but you're not by chance seeing this behavior on DDN hardware that runs the SFA OS are you? (e.g. SFA12K, 7K, 14K, etc.) We just started seeing some very weird and high latency on some of our SFA12ks (that have otherwise
been solid both in terms of stability and performance) but only on certain volumes and the affected volumes change. It's very bizzarre and we've been working closely with DDN to track down the root cause but we've not yet found a smoking gun. The timing and
description of your problem sounded eerily similar to what we're seeing so I'd thought I'd ask.<br clear="none" class="">
> <br clear="none" class="">
> -Aaron<br clear="none" class="">
> <br clear="none" class="">
> --<br clear="none" class="">
> Aaron Knister<br clear="none" class="">
> NASA Center for Climate Simulation (Code 606.2)<br clear="none" class="">
> Goddard Space Flight Center<br clear="none" class="">
> (301) 286-2776<br clear="none" class="">
> <br clear="none" class="">
> <br clear="none" class="">
> On Tue, 3 Jul 2018, Buterbaugh, Kevin L wrote:<br clear="none" class="">
> <br clear="none" class="">
>> Hi all,<br clear="none" class="">
>> We are experiencing some high I/O wait times (5 - 20 seconds!) on some of our NSDs as reported by “mmdiag —iohist" and are struggling to understand why. One of the<br clear="none" class="">
>> confusing things is that, while certain NSDs tend to show the problem more than others, the problem is not consistent … i.e. the problem tends to move around from<br clear="none" class="">
>> NSD to NSD (and storage array to storage array) whenever we check … which is sometimes just a few minutes apart.<br clear="none" class="">
>> In the past when I have seen “mmdiag —iohist” report high wait times like this it has *always* been hardware related. In our environment, the most common cause has<br clear="none" class="">
>> been a battery backup unit on a storage array controller going bad and the storage array switching to write straight to disk. But that’s *not* happening this time.<br clear="none" class="">
>> Is there anything within GPFS / outside of a hardware issue that I should be looking for?? Thanks!<br clear="none" class="">
>> —<br clear="none" class="">
>> Kevin Buterbaugh - Senior System Administrator<br clear="none" class="">
>> Vanderbilt University - Advanced Computing Center for Research and Education<br clear="none" class="">
>> <a shape="rect" href="mailto:Kevin.Buterbaugh@vanderbilt.edu" target="_blank" class="">
Kevin.Buterbaugh@vanderbilt.edu</a> - (615)875-9633<br clear="none" class="">
> _______________________________________________<br clear="none" class="">
> gpfsug-discuss mailing list<br clear="none" class="">
> gpfsug-discuss at <a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Caa277914313f445d702e08d5e363d347%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636664940252827256&sdata=yWdmhgLWoARwLNippWbVnOR2ztDKx4%2FhDT7DVoTxBKQ%3D&reserved=0" originalsrc="http://spectrumscale.org" shash="Iy1gwZlrUWn4kXjKlTsGMyguozyREk1BeGFwNYztDsHRC4HmY58WBphxQ/wvvZGBa4oUgZbw+sun7e4akFUHIpY8TMTLzRLPQ8WZ5CCTMQWcub9KjvPOgQatLeIyLZt/hE8iqkxJ0abLjZzqHSdFKJJpxo/IAEK68dWMDpMQgMc=" target="_blank" class="">
spectrumscale.org</a><br clear="none" class="">
> <a shape="rect" href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C9c1c75becd20479479a608d5e1ab43ec%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636663048058564742&sdata=if1uC53Y7K3D%2FMuVMskzsYqPx9qftU1ICQfP23c7bI0%3D&reserved=0" target="_blank" class="">
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C9c1c75becd20479479a608d5e1ab43ec%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636663048058564742&sdata=if1uC53Y7K3D%2FMuVMskzsYqPx9qftU1ICQfP23c7bI0%3D&reserved=0</a>
<div class="m_-237379699367396637yqt0200919239" id="m_-237379699367396637yqtfd97882">
<br clear="none" class="">
<br clear="none" class="">
_______________________________________________<br clear="none" class="">
gpfsug-discuss mailing list<br clear="none" class="">
gpfsug-discuss at <a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Caa277914313f445d702e08d5e363d347%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636664940252837265&sdata=26Nk0oAqA%2BIYHO%2BVJZ%2B26R8u9HaoAImB8IgFctjPwk0%3D&reserved=0" originalsrc="http://spectrumscale.org" shash="wiofVtMEUHz9+id1PBWOmBLtCUMILtdbfb/EuKCFLiCo1v9GZ5ZMRQ6+PeUFLL/JSTBd3ZioAvfgbJ3rBmNaDT/Rs2eHhD34799WzCzkh6ZT4AsfaRipua1UktsbEYknPHtyOfjihmjQk6pkkocrvxP5MSt1jb+LRxjQFwnFsFc=" target="_blank" class="">
spectrumscale.org</a><br clear="none" class="">
<a shape="rect" href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C331014fd459d4151432308d5e340c4fa%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636664789687056826&sdata=q8sRHafEyyBx7BgcD9scnQVzb3egxSYiCKZbKScCbA0%3D&reserved=0" target="_blank" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br clear="none" class="">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Caa277914313f445d702e08d5e363d347%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636664940252847277&sdata=dF6DIthUt7RBNJNeyeLsmICL%2F3W3impYEbPJeORiVTA%3D&reserved=0" originalsrc="http://spectrumscale.org" shash="FaLeIXE8RHuDMrPz6ymq9F/Wf7H1lNVZoELfWuTM7/sw/rur0ggGniW4msI6Xkm9affiIZWW1HcDIhJQjBX0Tz30V2M3vaYf3Wr1B/hwNMIM8Un/Vnr6Kjj0limD67/3+JMMDpUdZWst5xtGRIDSGisCyQcEzxgcjTY/3jCi+xs=" target="_blank" class="">
spectrumscale.org</a><br class="">
<a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C331014fd459d4151432308d5e340c4fa%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636664789687076842&sdata=UhjNipQdsNjxIcUB%2Ffu2qEwn7K6tIBmGWEIruxGgI4A%3D&reserved=0" target="_blank" class="">https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C331014fd459d4151432308d5e340c4fa%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636664789687076842&sdata=UhjNipQdsNjxIcUB%2Ffu2qEwn7K6tIBmGWEIruxGgI4A%3D&reserved=0</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Caa277914313f445d702e08d5e363d347%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636664940252857281&sdata=PZeTsW6wjLNS37MSiExaKT1yhiYIGFGtO9I3ZeSzhRU%3D&reserved=0" originalsrc="http://spectrumscale.org" shash="gcF44npl4u96sQYY7p9Sa8Ms+ea/ySGnnsxKJhXKuyPcAQ+UV9EOO1p1pc3VQggSBQ3yY9xdIwEGgjbnVfAeqeWQyttAoN6feKcyRp0EXIWpbvMtgJk3mks0YPRIlR9w/kfaIwWtY3FycbP9NNxjkZlAsdY7doWDvMIfNFuFJsM=" rel="noreferrer" target="_blank" class="">
spectrumscale.org</a><br class="">
<a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Caa277914313f445d702e08d5e363d347%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636664940252867293&sdata=miv%2FWmubXqp5TbfZuvhTrsmmXvHDpkRBL4RjRp6tYvY%3D&reserved=0" originalsrc="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" shash="pj/r1ySSyj5rJpP97irO0m6mRQpGmdmj45ysigydRFa+1IGc3ut4OauamVa20xSwnXBhp3Yct8a/61I3u7EyHXNppUEi9HK4Ki9LRGEHCAiuVsINdeIHlRC8Ng8+AO33XyQYsdOfCqcXwcjmiAD5WxUgG/YOczhrlmlv0VKI2y8=" rel="noreferrer" target="_blank" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="">
</blockquote>
</div>
_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class="">
<a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Caa277914313f445d702e08d5e363d347%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636664940252877301&sdata=bnjsWHwutbbKstghBrB5Y7%2FIzeX7U19vroW%2B0xA2gX8%3D&reserved=0" class="">https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Caa277914313f445d702e08d5e363d347%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636664940252877301&sdata=bnjsWHwutbbKstghBrB5Y7%2FIzeX7U19vroW%2B0xA2gX8%3D&reserved=0</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>