<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle21
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1772704740;
mso-list-type:hybrid;
mso-list-template-ids:-1468109696 891161574 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:18.0pt;
text-indent:-18.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-font-family:Calibri;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:54.0pt;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:90.0pt;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:126.0pt;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:162.0pt;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:198.0pt;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:234.0pt;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:270.0pt;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:306.0pt;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hello Rob,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We looked at AFM years ago for DR, but after reading the bug reports, we avoided it, and also have had seen a case where it had to be removed from one customer, so we have kept things simple. Now looking again a few years later there are
still issues, <a href="https://www.ibm.com/support/pages/ibm-spectrum-scale-active-file-management-afm-issues-which-may-result-undetected-data-corruption">
IBM Spectrum Scale Active File Management (AFM) issues which may result in undetected data corruption</a>, and that was just my first google hit. We have kept it simple, and use a parallel rsync process with policy engine and can hit wire speed for copying
of millions of small files in order to have isolation between the sites at GB/s. I am not saying it is bad, just that it needs an appropriate risk/reward ratio to implement as it increases overall complexity.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Kind regards,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Dean<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>From:</b> gpfsug-discuss-bounces@spectrumscale.org <gpfsug-discuss-bounces@spectrumscale.org>
<b>On Behalf Of </b>Ryan Novosielski<br>
<b>Sent:</b> Monday, November 23, 2020 4:31 PM<br>
<b>To:</b> gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
<b>Subject:</b> Re: [gpfsug-discuss] AFM experiences?<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We use it similar to how you describe it. We now run 5.0.4.1 on the client side (I mean actual client nodes, not the home or cache clusters). Before that, we had reliability problems (failure to cache libraries of programs that were executing,
etc.). The storage clusters in our case are 5.0.3-2.3. <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We also got bit by the quotas thing. You have to set them the same on both sides, or you will have problems. It seems a little silly that they are not kept in sync by GPFS, but that’s how it is. If memory serves, the result looked like
an AFM failure (queue not being cleared), but it turned out to be that the files just could not be written at the home cluster because the user was over quota there. I also think I’ve seen load average increase due to this sort of thing, but I may be mixing
that up with another problem scenario. <br>
<br>
We monitor via Nagios which I believe monitors using mmafmctl commands. Really can’t think of a single time, apart from the other day, where the queue backed up. The instance the other day only lasted a few minutes (if you suddenly create many small files,
like installing new software, it may not catch up instantly). <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<p class="MsoNormal">#BlackLivesMatter<br>
____<br>
|| <a href="file://UTGERS">\\UTGERS</a>, |---------------------------*O*---------------------------<br>
||_// the State | Ryan Novosielski - <a href="mailto:novosirj@rutgers.edu">novosirj@rutgers.edu</a><br>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus<br>
|| \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark<br>
`'<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-bottom:12.0pt">On Nov 23, 2020, at 10:19, Robert Horton <<a href="mailto:robert.horton@icr.ac.uk">robert.horton@icr.ac.uk</a>> wrote:<o:p></o:p></p>
</blockquote>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">Hi all,<br>
<br>
We're thinking about deploying AFM and would be interested in hearing<br>
from anyone who has used it in anger - particularly independent writer.<br>
<br>
Our scenario is we have a relatively large but slow (mainly because it<br>
is stretched over two sites with a 10G link) cluster for long/medium-<br>
term storage and a smaller but faster cluster for scratch storage in<br>
our HPC system. What we're thinking of doing is using some/all of the<br>
scratch capacity as an IW cache of some/all of the main cluster, the<br>
idea to reduce the need for people to manually move data between the<br>
two.<br>
<br>
It seems to generally work as expected in a small test environment,<br>
although we have a few concerns:<br>
<br>
- Quota management on the home cluster - we need a way of ensuring<br>
people don't write data to the cache which can't be accomodated on<br>
home. Probably not insurmountable but needs a bit of thought...<br>
<br>
- It seems inodes on the cache only get freed when they are deleted on<br>
the cache cluster - not if they get deleted from the home cluster or<br>
when the blocks are evicted from the cache. Does this become an issue<br>
in time?<br>
<br>
If anyone has done anything similar I'd be interested to hear how you<br>
got on. It would be intresting to know if you created a cache fileset<br>
for each home fileset or just one for the whole lot, as well as any<br>
other pearls of wisdom you may have to offer.<br>
<br>
Thanks!<br>
Rob<br>
<br>
-- <br>
Robert Horton | Research Data Storage Lead<br>
The Institute of Cancer Research | 237 Fulham Road | London | SW3 6JB<br>
T +44 (0)20 7153 5350 | E <a href="mailto:robert.horton@icr.ac.uk">robert.horton@icr.ac.uk</a> | W
<a href="http://www.icr.ac.uk">www.icr.ac.uk</a> |<br>
Twitter @ICR_London<br>
Facebook: <a href="http://www.facebook.com/theinstituteofcancerresearch">www.facebook.com/theinstituteofcancerresearch</a><br>
<br>
The Institute of Cancer Research: Royal Cancer Hospital, a charitable Company Limited by Guarantee, Registered in England under Company No. 534147 with its Registered Office at 123 Old Brompton Road, London SW7 3RP.<br>
<br>
This e-mail message is confidential and for use by the addressee only. If the message is received by anyone other than the addressee, please return the message to the sender by replying to it and then delete the message from your computer and network.<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><o:p></o:p></p>
</div>
</blockquote>
</div>
</div>
</body>
</html>