<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Hi Richard,
<div class=""><br class="">
</div>
<div class="">We went from GPFS 3.5.0.26 (where we also had zero problems with snapshot deletion) to GPFS 4.1.0.8 this past August and immediately hit the snapshot deletion bug (it’s some sort of race condition). It’s not pleasant … to recover we had to unmount
the affected filesystem from both clusters, which didn’t exactly make our researchers happy.</div>
<div class=""><br class="">
</div>
<div class="">But the good news is that there is an efix available for it if you’re on the 4.1.0 series and I am 99% sure that the bug has also been fixed in the last several PTF’s for the 4.1.1 series.</div>
<div class=""><br class="">
</div>
<div class="">That’s not the only bug we hit when going to 4.1.0.8 so my personal advice / opinion would be to bypass 4.1.0 and go straight to 4.1.1 or 4.2 when it comes out. We are planning on going to 4.2 as soon as feasible … it looks like it’s much more
stable plus has some new features (compression!) that we are very interested in. Again, my 2 cents worth.</div>
<div class=""><br class="">
</div>
<div class="">Kevin</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Dec 8, 2015, at 8:14 AM, Sobey, Richard A <<a href="mailto:r.sobey@imperial.ac.uk" class="">r.sobey@imperial.ac.uk</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="WordSection1" style="page: WordSection1; font-family: Helvetica; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(68, 84, 106);" class="">This may not be at all applicable to your situation, but we’re creating thousands of snapshots per day of many independent filesets. The same script(s) call mmdelsnapshot,
too. We haven’t seen any particular issues with this.<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(68, 84, 106);" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(68, 84, 106);" class="">GPFS 3.5.<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(68, 84, 106);" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(68, 84, 106);" class="">I note with intereste your bug report below about 4.1.0.x though – are you able to elaborate?<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(68, 84, 106);" class=""><o:p class=""> </o:p></span></div>
<div class="">
<div style="border-style: solid none none; border-top-color: rgb(181, 196, 223); border-top-width: 1pt; padding: 3pt 0cm 0cm;" class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<b class=""><span lang="EN-US" style="font-size: 10pt; font-family: Tahoma, sans-serif;" class="">From:</span></b><span lang="EN-US" style="font-size: 10pt; font-family: Tahoma, sans-serif;" class=""><span class="Apple-converted-space"> </span><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" style="color: purple; text-decoration: underline;" class="">gpfsug-discuss-bounces@spectrumscale.org</a><span class="Apple-converted-space"> </span>[<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" style="color: purple; text-decoration: underline;" class="">mailto:gpfsug-discuss-bounces@spectrumscale.org</a>]<span class="Apple-converted-space"> </span><b class="">On
Behalf Of<span class="Apple-converted-space"> </span></b>Buterbaugh, Kevin L<br class="">
<b class="">Sent:</b><span class="Apple-converted-space"> </span>07 December 2015 17:53<br class="">
<b class="">To:</b><span class="Apple-converted-space"> </span>gpfsug main discussion list<br class="">
<b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [gpfsug-discuss] GPFS Remote Cluster Co-existence with CTDB/NFS Re-exporting<o:p class=""></o:p></span></div>
</div>
</div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
Hi Stewart,<span class="Apple-converted-space"> </span><o:p class=""></o:p></div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
We had been running mmcrsnapshot with a ~700 node remote cluster accessing the filesystem for a couple of years now without issue.<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
However, we haven’t been running it for a little while because there is a very serious bug in GPFS 4.1.0.x relating to snapshot *deletion*. There is an efix for it and we are in the process of rolling that out, but will not try to resume snapshots until both
clusters are fully updated.<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
HTH…<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
Kevin<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="">
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
On Dec 7, 2015, at 11:23 AM, Howard, Stewart Jameson <<a href="mailto:sjhoward@iu.edu" style="color: purple; text-decoration: underline;" class="">sjhoward@iu.edu</a>> wrote:<o:p class=""></o:p></div>
</div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; background-color: white;" class="">
<span style="font-family: Calibri, sans-serif;" class="">Hi All,<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; background-color: white;" class="">
<span style="font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; background-color: white;" class="">
<span style="font-family: Calibri, sans-serif;" class="">Thanks to Doug and Kevin for the replies. In answer to Kevin's question about our choice of clustering solution for NFS: the choice was made hoping to maintain some simplicity by not using more than
one HA solution at a time. However, it seems that this choice might have introduced more wrinkles than it's ironed out.<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; background-color: white;" class="">
<span style="font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; background-color: white;" class="">
<span style="font-family: Calibri, sans-serif;" class="">An update on our situation: we have actually uncovered another clue since my last posting. One thing that this now known to be correlated *very* closely with instability in the NFS layer is running
`mmcrsnapshot`. We had noticed that flapping happened like clockwork at midnight every night. This happens to be the same time at which our crontab was running the `mmcrsnapshot` so, as an experiment, we moved the snapshot to happen at 1a.<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; background-color: white;" class="">
<span style="font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; background-color: white;" class="">
<span style="font-family: Calibri, sans-serif;" class="">After this change, the late-night flapping has moved to 1a and now happens reliably every night at that time. I saw a post on this list from 2013 stating that `mmcrsnapshot` was known to hang up the
filesystem with race conditions that result in deadlocks and am wondering if that is still a problem with the `mmcrsnapthost` command. Running the snapshots had not been an obvious problem before, but seems to have become one since we deployed ~300 additional
GPFS clients in a remote cluster configuration about a week ago.<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; background-color: white;" class="">
<span style="font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; background-color: white;" class="">
<span style="font-family: Calibri, sans-serif;" class="">Can anybody comment on the safety of running `mmcrsnapshot` with a ~300 node remote cluster accessing the filesystem?<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; background-color: white;" class="">
<span style="font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; background-color: white;" class="">
<span style="font-family: Calibri, sans-serif;" class="">Also, I would comment that this is not the only condition under which we see instability in the NFS layer. We continue to see intermittent instability through the day. The creation of a snapshot is
simply the one well-correlated condition that we've discovered so far.<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; background-color: white;" class="">
<span style="font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; background-color: white;" class="">
<span style="font-family: Calibri, sans-serif;" class="">Thanks so much to everyone for your help :)<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; background-color: white;" class="">
<span style="font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; background-color: white;" class="">
<span style="font-family: Calibri, sans-serif;" class="">Stewart<o:p class=""></o:p></span></div>
</div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-family: Calibri, sans-serif; background-color: white; background-position: initial initial; background-repeat: initial initial;" class="">_______________________________________________</span><span style="font-family: Calibri, sans-serif;" class=""><br class="">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial;" class="">gpfsug-discuss mailing list</span><br class="">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial;" class="">gpfsug-discuss at<span class="apple-converted-space"> </span></span></span><a href="http://spectrumscale.org/" style="color: purple; text-decoration: underline;" class=""><span style="font-family: Calibri, sans-serif; background-color: white; background-position: initial initial; background-repeat: initial initial;" class="">spectrumscale.org</span></a><span style="font-family: Calibri, sans-serif;" class=""><br class="">
</span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" style="color: purple; text-decoration: underline;" class=""><span style="font-family: Calibri, sans-serif; background-color: white; background-position: initial initial; background-repeat: initial initial;" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></a><o:p class=""></o:p></div>
</div>
</blockquote>
</div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div class="">
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
—<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
Kevin Buterbaugh - Senior System Administrator<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
Vanderbilt University - Advanced Computing Center for Research and Education<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" style="color: purple; text-decoration: underline;" class="">Kevin.Buterbaugh@vanderbilt.edu</a> - (615)875-9633<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""> </o:p></div>
</div>
</div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""> </o:p></div>
</div>
</div>
<span style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">gpfsug-discuss
mailing list</span><br style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">gpfsug-discuss
at<span class="Apple-converted-space"> </span></span><a href="http://spectrumscale.org/" style="color: purple; text-decoration: underline; font-family: Helvetica; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">spectrumscale.org</a><br style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" style="color: purple; text-decoration: underline; font-family: Helvetica; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></div>
</blockquote>
</div>
<br class="">
<div class="">
<div class="">
<div class="">—</div>
<div class="">Kevin Buterbaugh - Senior System Administrator</div>
<div class="">Vanderbilt University - Advanced Computing Center for Research and Education</div>
<div class=""><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class="">Kevin.Buterbaugh@vanderbilt.edu</a> - (615)875-9633</div>
<div class=""><br class="">
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</body>
</html>