<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi All,
<div class=""><br class="">
</div>
<div class="">Thanks for all the responses on this, although I have the sneaking suspicion that the most significant thing that is going to come out of this thread is the knowledge that Sven has left IBM for DDN. ;-) or :-( or :-O depending on your perspective.</div>
<div class=""><br class="">
</div>
<div class="">Anyway … we have done some testing which has shown that a 4 MB block size is best for those workloads that use “normal” sized files. However, we - like many similar institutions - support a mixed workload, so the 128K fragment size that comes
with that is not optimal for the primarily biomedical type applications that literally create millions of very small files. That’s why we settled on 1 MB as a compromise.</div>
<div class=""><br class="">
</div>
<div class="">So we’re very eager to now test with GPFS 5, a 4 MB block size, and a 8K fragment size. I’m recreating my test cluster filesystem now with that config … so 4 MB block size on the metadata only system pool, too.</div>
<div class=""><br class="">
</div>
<div class="">Thanks to all who took the time to respond to this thread. I hope it’s been beneficial to others as well…</div>
<div class=""><br class="">
</div>
<div class="">Kevin</div>
<div class=""><br class="">
<div class="">
<div class="">—</div>
<div class="">Kevin Buterbaugh - Senior System Administrator</div>
<div class="">Vanderbilt University - Advanced Computing Center for Research and Education</div>
<div class=""><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class="">Kevin.Buterbaugh@vanderbilt.edu</a> - (615)875-9633</div>
<div class=""><br class="">
</div>
</div>
<div>
<blockquote type="cite" class="">
<div class="">On Aug 1, 2018, at 7:11 PM, Andrew Beattie <<a href="mailto:abeattie@au1.ibm.com" class="">abeattie@au1.ibm.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt">
<div dir="ltr" class="">I too would second the comment about doing testing specific to your environment</div>
<div dir="ltr" class=""> </div>
<div dir="ltr" class="">We recently deployed a number of ESS building blocks into a customer site that was specifically being used for a mixed HPC workload.</div>
<div dir="ltr" class=""> </div>
<div dir="ltr" class="">We spent more than a week playing with different block sizes for both data and metadata trying to identify which variation would provide the best mix of both metadata performance and data performance. one thing we noticed very early
on is that MDtest and IOR both respond very differently as you play with both block size and subblock size. What works for one use case may be a very poor option for another use case.</div>
<div dir="ltr" class=""> </div>
<div dir="ltr" class="">Interestingly enough it turned out that the best overall option for our particular use case was an 8MB block size with 32k sub blocks -- as that gave us good Metadata performance and good sequential data performance </div>
<div dir="ltr" class=""> </div>
<div dir="ltr" class="">which is probably why 32k sub block was the default for so many years ....</div>
<div dir="ltr" class="">
<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt">
<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt">
<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt">
<div dir="ltr" style="margin-top: 20px;" class="">
<div style="font-size: 12pt; font-weight: bold; font-family: sans-serif; color: #7C7C5F;" class="">
Andrew Beattie</div>
<div style="font-size: 10pt; font-weight: bold; font-family: sans-serif;" class="">
Software Defined Storage - IT Specialist</div>
<div style="font-size: 8pt; font-family: sans-serif; margin-top: 10px;" class="">
<div class=""><span style="font-weight: bold; color: #336699;" class="">Phone: </span>
614-2133-7927</div>
<div class=""><span style="font-weight: bold; color: #336699;" class="">E-mail: </span>
<a href="mailto:abeattie@au1.ibm.com" style="color: #555" class="">abeattie@au1.ibm.com</a></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div dir="ltr" class=""> </div>
<div dir="ltr" class=""> </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" class="">
----- Original message -----<br class="">
From: "Marc A Kaplan" <<a href="mailto:makaplan@us.ibm.com" class="">makaplan@us.ibm.com</a>><br class="">
Sent by: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class="">gpfsug-discuss-bounces@spectrumscale.org</a><br class="">
To: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" class="">gpfsug-discuss@spectrumscale.org</a>><br class="">
Cc:<br class="">
Subject: Re: [gpfsug-discuss] Sub-block size not quite as expected on GPFS 5 filesystem?<br class="">
Date: Thu, Aug 2, 2018 10:01 AM<br class="">
<br class="">
<span style=" font-size:10pt;font-family:sans-serif" class="">Firstly, I do suggest that you run some tests and see how much, if any, difference the settings that are available make in performance and/or storage utilization.</span><br class="">
<br class="">
<span style=" font-size:10pt;font-family:sans-serif" class="">Secondly, as I and others have hinted at, deeper in the system, there may be additional parameters and settings. Sometimes they are available via commands, and/or configuration settings, sometimes
not.</span><br class="">
<br class="">
<span style=" font-size:10pt;font-family:sans-serif" class="">Sometimes that's just because we didn't want to overwhelm you or ourselves with yet more "tuning knobs".</span><br class="">
<br class="">
<span style=" font-size:10pt;font-family:sans-serif" class="">Sometimes it's because we made some component more tunable than we really needed, but did not make all the interconnected components equally or as widely tunable.</span><br class="">
<span style=" font-size:10pt;font-family:sans-serif" class="">Sometimes it's because we want to save you from making ridiculous settings that would lead to problems...</span><br class="">
<br class="">
<span style=" font-size:10pt;font-family:sans-serif" class="">OTOH, as I wrote before, if a burning requirement surfaces, things may change from release to release... Just as for so many years subblocks per block seemed forever frozen at the number 32. Now
it varies... and then the discussion shifts to why can't it be even more flexible?</span><br class="">
<br class="">
<div class=""><font face="Default Monospace,Courier New,Courier,monospace" size="2" class="">_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class="">
<a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb821b9e8a6db4408fff308d5f80c907d%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636687655210056012&sdata=SCzz05SABDQ0vxprDYfdKGOY1VES%2Fm0tIr2kRnGlY4c%3D&reserved=0" originalsrc="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" shash="jNcv1QBTK1oeKa2JBcydCEunkvUKhLU34tJvhYmrJvdKc4rKgr1bv7ScNv+0cI1wR3zbRqXuiOptEtGNhE6BYs+ukrPeTyeWdZ1VhSfDXPZQXmsw0PRgvWT7LfgtWNIKVLh2C8aTI+1GrMY3GowKA0LNuv5qq7cRK8n8lkpJgQA=" target="_blank" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></div>
</blockquote>
<div dir="ltr" class=""> </div>
</div>
<br class="">
_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class="">
<a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb821b9e8a6db4408fff308d5f80c907d%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636687655210056012&sdata=SCzz05SABDQ0vxprDYfdKGOY1VES%2Fm0tIr2kRnGlY4c%3D&reserved=0" class="">https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb821b9e8a6db4408fff308d5f80c907d%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636687655210056012&sdata=SCzz05SABDQ0vxprDYfdKGOY1VES%2Fm0tIr2kRnGlY4c%3D&reserved=0</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>