<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
tt
{mso-style-priority:99;
font-family:"Courier New",serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">A word of caution, be careful about where you run this kind of policy scan as the sort process can consume all memory on your hosts and that could lead to issues
with the OS deciding to kill off GPFS or other similar bad things can occur. I recommend restricting the ILM policy scan to a subset of servers, no quorum nodes, and ensuring at least one NSD server is available for all NSDs in the file system(s). Watch
the memory consumption on your nodes during the sort operations to see if you need to tune that down in the mmapplypolicy options.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hope that helps,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">-Bryan<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> gpfsug-discuss-bounces@spectrumscale.org [mailto:gpfsug-discuss-bounces@spectrumscale.org]
<b>On Behalf Of </b>Frederick Stock<br>
<b>Sent:</b> Tuesday, April 11, 2017 6:54 AM<br>
<b>To:</b> gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
<b>Subject:</b> Re: [gpfsug-discuss] Policy scan against billion files for ILM/HSM<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">As Zachary noted the location of your metadata is the key and for the scanning you have planned flash is necessary. If you have the resources you may consider setting up your flash in a mirrored
RAID configuration (RAID1/RAID10) and have GPFS only keep one copy of metadata since the underlying storage is replicating it via the RAID. This should improve metadata write performance but likely has little impact on your scanning, assuming you are just
reading through the metadata.</span><br>
<br>
<span style="font-family:"Arial",sans-serif">Fred<br>
__________________________________________________<br>
Fred Stock | IBM Pittsburgh Lab | 720-430-8821<br>
<a href="mailto:stockf@us.ibm.com">stockf@us.ibm.com</a></span><br>
<br>
<br>
<br>
<span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">From: </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif">Zachary Giles <<a href="mailto:zgiles@gmail.com">zgiles@gmail.com</a>></span><br>
<span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">To: </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif">gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a>></span><br>
<span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">Date: </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif">04/11/2017 12:49 AM</span><br>
<span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">Subject: </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif">Re: [gpfsug-discuss] Policy scan against billion files for ILM/HSM</span><br>
<span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">Sent by: </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif"><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-bounces@spectrumscale.org</a></span><o:p></o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" noshade="" style="color:#A0A0A0" align="center">
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
<br>
<br>
<tt><span style="font-size:10.0pt">It's definitely doable, and these days not too hard. Flash for</span></tt><span style="font-size:10.0pt;font-family:"Courier New",serif"><br>
<tt>metadata is the key.</tt><br>
<tt>The basics of it are:</tt><br>
<tt>* Latest GPFS for performance benefits.</tt><br>
<tt>* A few 10's of TBs of flash ( or more ! ) setup in a good design..</tt><br>
<tt>lots of SAS, well balanced RAID that can consume the flash fully,</tt><br>
<tt>tuned for IOPs, and available in parallel from multiple servers.</tt><br>
<tt>* Tune up mmapplypolicy with -g somewhere-on-gpfs; --choice-algorithm</tt><br>
<tt>fast; -a, -m and -n to reasonable values ( number of cores on the</tt><br>
<tt>servers ); -A to ~1000</tt><br>
<tt>* Test first on a smaller fileset to confirm you like it. -I test</tt><br>
<tt>should work well and be around the same speed minus the migration</tt><br>
<tt>phase.</tt><br>
<tt>* Then throw ~8 well tuned Infiniband attached nodes at it using -N,</tt><br>
<tt>If they're the same as the NSD servers serving the flash, even better.</tt><br>
<br>
<tt>Should be able to do 1B in 5-30m depending on the idiosyncrasies of</tt><br>
<tt>above choices. Even 60m isn't bad and quite respectable if less gear</tt><br>
<tt>is used or if they system is busy while the policy is running.</tt><br>
<tt>Parallel metadata, it's a beautiful thing.</tt><br>
<br>
<br>
<br>
<tt>On Tue, Apr 11, 2017 at 12:29 AM, Masanori Mitsugi</tt><br>
<tt><<a href="mailto:mitsugi@linux.vnet.ibm.com">mitsugi@linux.vnet.ibm.com</a>> wrote:</tt><br>
<tt>> Hello,</tt><br>
<tt>></tt><br>
<tt>> Does anyone have experience to do mmapplypolicy against billion files for</tt><br>
<tt>> ILM/HSM?</tt><br>
<tt>></tt><br>
<tt>> Currently I'm planning/designing</tt><br>
<tt>></tt><br>
<tt>> * 1 Scale filesystem (5-10 PB)</tt><br>
<tt>> * 10-20 filesets which includes 1 billion files each</tt><br>
<tt>></tt><br>
<tt>> And our biggest concern is "How log does it take for mmapplypolicy policy</tt><br>
<tt>> scan against billion files?"</tt><br>
<tt>></tt><br>
<tt>> I know it depends on how to write the policy,</tt><br>
<tt>> but I don't have no billion files policy scan experience,</tt><br>
<tt>> so I'd like to know the order of time (min/hour/day...).</tt><br>
<tt>></tt><br>
<tt>> It would be helpful if anyone has experience of such large number of files</tt><br>
<tt>> scan and let me know any considerations or points for policy design.</tt><br>
<tt>></tt><br>
<tt>> --</tt><br>
<tt>> Masanori Mitsugi</tt><br>
<tt>> <a href="mailto:mitsugi@linux.vnet.ibm.com">mitsugi@linux.vnet.ibm.com</a></tt><br>
<tt>></tt><br>
<tt>> _______________________________________________</tt><br>
<tt>> gpfsug-discuss mailing list</tt><br>
<tt>> gpfsug-discuss at spectrumscale.org</tt><br>
<tt>> </tt></span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><span style="font-size:10.0pt">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></tt></a><span style="font-size:10.0pt;font-family:"Courier New",serif"><br>
<br>
<br>
<br>
<tt>-- </tt><br>
<tt>Zach Giles</tt><br>
<tt><a href="mailto:zgiles@gmail.com">zgiles@gmail.com</a></tt><br>
<tt>_______________________________________________</tt><br>
<tt>gpfsug-discuss mailing list</tt><br>
<tt>gpfsug-discuss at spectrumscale.org</tt><br>
</span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><span style="font-size:10.0pt">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></tt></a><span style="font-size:10.0pt;font-family:"Courier New",serif"><br>
<br>
</span><br>
<br>
<o:p></o:p></p>
</div>
<br>
<hr>
<font face="Arial" color="Gray" size="1"><br>
Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this
email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness
or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial
product.<br>
</font>
</body>
</html>