<div dir="ltr">One of the parameters that you need to choose at filesystem creation time is the block allocation type.  -j {cluster|scatter} parameter to mmcrfs:  <a href="https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1ins_blkalmap.htm#ballmap">https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1ins_blkalmap.htm#ballmap</a><div><br></div><div>If you use "cluster", you will have quite high performance when the filesystem is close to empty.  If you use "scatter", the performance will stay the same no matter the filesystem utilization because blocks for a given file will always be scattered randomly. </div><div><br></div><div>Some vendors set up their GPFS filesystem using '-j cluster' and then show off their streaming write performance numbers. But the performance degrades considerably as the filesystem fills up. With "scatter", the filesystem performance is slower but stays consistent throughout its lifetime.  </div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Nov 7, 2017 at 1:19 AM, Daniel Kidger <span dir="ltr"><<a href="mailto:daniel.kidger@uk.ibm.com" target="_blank">daniel.kidger@uk.ibm.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto">I understand that this near linear performance is one of the differentiators of Spectrum Scale.<div>Others with more field experience than me might want to comment on how Lustre and other distributed filesystem perform as they approaches near full capacity.</div><div><br><div id="m_-1832164167447791796AppleMailSignature"><div style="margin-bottom:0.0001pt;line-height:normal"><span style="background-color:rgba(255,255,255,0)">Daniel</span></div><div style="margin-bottom:0.0001pt;line-height:normal"><span style="background-color:rgba(255,255,255,0)"><img alt="/spectrum_storage-banne" src="http://ausgsa.ibm.com/projects/t/tivoli_visual_design/public/2015/Spectrum-Storage/Email-signatures/Storage/spectrum_storage-banner.png" width="432.5" height="auto" style="width:601px;height:5px"></span></div><div style="margin-bottom:0.0001pt;line-height:normal"><span style="background-color:rgba(255,255,255,0)"><br> </span></div><table border="0" cellpadding="0" cellspacing="0" style="font-family:sans-serif"><tbody><tr style="word-wrap:break-word;max-width:447.5px"><td style="word-wrap:break-word;max-width:447.5px;width:201px;padding:0cm"><div style="margin-bottom:0.0001pt;line-height:normal"><img alt="Spectrum Scale Logo" src="http://ausgsa.ibm.com/projects/t/tivoli_visual_design/public/2015/Spectrum-Storage/Email-signatures/Storage/spectrum_scale-logo.png" style="width:75px;height:120px;float:left"></div><div style="margin-bottom:0.0001pt;line-height:normal"><font face=".SFUIDisplay"><span style="background-color:rgba(255,255,255,0)"> </span></font></div></td><td style="word-wrap:break-word;max-width:447.5px;width:21px;padding:0cm"><font face=".SFUIDisplay"><span style="background-color:rgba(255,255,255,0)"> </span></font></td><td style="word-wrap:break-word;max-width:447.5px;width:202px;padding:0cm"><div style="margin-bottom:0.0001pt;line-height:normal"><font face=".SFUIDisplay"><span style="background-color:rgba(255,255,255,0)"><strong>Dr Daniel Kidger</strong> <br>IBM Technical Sales Specialist<br>Software Defined Solution Sales<br><br><a dir="ltr" href="tel:+%2044-7818%20522%20266" target="_blank">+</a> <a dir="ltr" href="tel:+%2044-7818%20522%20266" target="_blank">44-(0)7818 522 266</a> <br><a dir="ltr" href="mailto:daniel.kidger@uk.ibm.com" target="_blank">daniel.kidger@uk.ibm.com</a></span></font></div></td></tr></tbody></table></div><span class=""><div><br>On 7 Nov 2017, at 00:12, Carl <<a href="mailto:mutantllama@gmail.com" target="_blank">mutantllama@gmail.com</a>> wrote:<br><br></div></span><blockquote type="cite"><div><span class=""><div dir="ltr">Thanks to all for the information. <div><br></div><div>Im happy to say that it is close to what I hoped would be the case. </div><div><br></div><div>Interesting to see the effect of the -n value. Reinforces the need to think about it and not go with the defaults.</div><div><br></div><div>Thanks again,</div><div><br></div><div>Carl.</div><div><br></div></div></span><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On 7 November 2017 at 03:18, Achim Rehor <span dir="ltr"><<a href="mailto:Achim.Rehor@de.ibm.com" target="_blank">Achim.Rehor@de.ibm.com</a>></span> wrote:<br></span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=""><font size="2" face="sans-serif">I have no practical experience on these
numbers, however, Peters experience below is matching what i learned from
Dan years ago.</font><br><br><font size="2" face="sans-serif">As long as the -n setting of the FS
(the number of nodes potentially mounting the fs) is more or less matching
the actual number of mounts, </font><br><font size="2" face="sans-serif">this 99.x % before degradation is expected.
If you are far off with that -n estimate, like having it set to 32, but
the actual number of mounts is in the thousands,</font><br><font size="2" face="sans-serif">then degradation happens earlier, since
the distribution of free blocks in the allocation maps is not matching
the actual setup as good as it could be. </font><br><br><font size="2" face="sans-serif">Naturally, this depends also on how
you do filling of the FS. If it is only a small percentage of the nodes,
doing the creates, then the distribution can </font><br><font size="2" face="sans-serif">be 'wrong' as well, and single nodes
run earlier out of allocation map space, and need to look for free blocks
elsewhere, costing RPC cycles and thus performance. </font><br><br><font size="2" face="sans-serif">Putting this in numbers seems quite
difficult ;) </font><br><br><font size="1" face="Arial"><br>Mit freundlichen Grüßen / Kind regards</font><p><font size="2" face="Arial"><b>Achim Rehor</b></font></p><p></p></span><table width="600" style="border-collapse:collapse"><tbody><tr height="8"><td width="600" colspan="4" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><div align="center"><hr noshade></div></td></tr><tr height="8"><td width="600" colspan="4" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><font size="1" face="Arial"> </font></td></tr><tr height="8"><td width="496" colspan="3" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><font size="1" face="Arial">Software
Technical Support Specialist AIX/ Emea HPC Support</font></td><td width="103" rowspan="10" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><div align="right"><_1_<wbr>D95FF418D95FEE980059980B852581<wbr>D0.gif></div></td></tr><tr height="8"><td width="496" colspan="3" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><font size="1" face="Arial">IBM
Certified Advanced Technical Expert - Power Systems with AIX</font></td></tr><tr height="8"><td width="496" colspan="3" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><font size="1" face="Arial">TSCC
Software Service, Dept. 7922</font></td></tr><tr height="8"><td width="496" colspan="3" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><font size="1" face="Arial">Global
Technology Services </font></td></tr><tr height="8"><td width="496" colspan="3" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><div align="center"><hr noshade></div></td></tr><tr height="8"><td width="44" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><font size="1" color="#0060a0" face="Arial">Phone:</font></td><td width="195" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><font size="1" color="#0060a0" face="Arial"><a href="tel:+49%207034%202747862" value="+4970342747862" target="_blank">+49-7034-274-7862</a></font></td><td width="257" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><font size="1" color="#0060a0" face="Arial"> IBM
Deutschland</font></td></tr><tr height="8"><td width="44" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><font size="1" color="#0060a0" face="Arial">E-Mail:</font></td><td width="195" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><font size="1" color="#0060a0" face="Arial"><a href="mailto:Achim.Rehor@de.ibm.com" target="_blank">Achim.Rehor@de.ibm.com</a></font></td><td width="257" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><font size="1" color="#0060a0" face="Arial"> Am
Weiher 24</font></td></tr><tr height="8"><td width="44" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><font size="1" color="#0060a0" face="Arial"> </font></td><td width="195" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><font size="1" color="#0060a0" face="Arial"> </font></td><td width="257" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><font size="1" color="#0060a0" face="Arial"> 65451
Kelsterbach</font></td></tr><tr height="8"><td width="44" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><font size="1" color="#0060a0" face="Arial"> </font></td><td width="195" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><font size="1" color="#0060a0" face="Arial"> </font></td><td width="257" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><font size="1" color="#0060a0" face="Arial"> <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3DAm-250D-2BWeiher-2B24-25C2-25A0-25C2-25A0-25C2-25A065451-250D-2BKelsterbach-25C2-25A0-25C2-25A0-25C2-25A0Germany-26entry-3Dgmail-26source-3Dg&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=zatwEi1ZS1fdfJjSR0DnoHq8vd75vVJcEyrIbaU1Hyw&s=LcAQSuZfiGE8amC7zXd_vu0W02tmvaguHINvGR6UYTs&e=" target="_blank">Germany</a></font></td></tr><tr height="8"><td width="44" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><font size="1" color="#0060a0" face="Arial"> </font></td><td width="195" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><font size="1" color="#0060a0" face="Arial"> </font></td><td width="257" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><font size="1" color="#0060a0" face="Arial"> </font></td></tr><tr height="8"><td width="600" colspan="4" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><div align="center"><hr noshade></div></td></tr><tr height="8"><td width="600" colspan="4" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><font size="1" face="Arial"> </font></td></tr><tr height="8"><td width="600" colspan="4" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><font size="1" color="#a2a2a2" face="Arial">IBM
Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter <br>Geschäftsführung: Martina Koederitz (Vorsitzende), Reinhard Reschke, Dieter
Scholz, Gregor Pillen, Ivo Koerner, Christian Noll <br>Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart,
HRB 14562 WEEE-Reg.-Nr. DE 99369940 </font></td></tr></tbody></table><div><div class="h5"><br><p><font size="3"> </font><br><br><br><br><font size="1" color="#5f5f5f" face="sans-serif">From:      
 </font><font size="1" face="sans-serif">Peter Smith <<a href="mailto:peter.smith@framestore.com" target="_blank">peter.smith@framestore.com</a>></font><span><br><font size="1" color="#5f5f5f" face="sans-serif">To:      
 </font><font size="1" face="sans-serif">gpfsug main discussion
list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a>></font><br></span><font size="1" color="#5f5f5f" face="sans-serif">Date:      
 </font><font size="1" face="sans-serif">11/06/2017 09:17 AM</font><br><font size="1" color="#5f5f5f" face="sans-serif">Subject:    
   </font><font size="1" face="sans-serif">Re: [gpfsug-discuss]
Performance of GPFS when filesystem is almost        full</font><br><font size="1" color="#5f5f5f" face="sans-serif">Sent by:    
   </font><font size="1" face="sans-serif"><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectr<wbr>umscale.org</a></font><br></p><hr noshade><span><br><br><br><tt><font size="3">Hi Carl.<br></font></tt><br><tt><font size="3">When we commissioned our system we ran an NFS stress
tool, and filled the system to the top.<br></font></tt><br><tt><font size="3">No performance degradation was seen until it was 99.7%
full.<br></font></tt><br><tt><font size="3">I believe that after this point it takes longer to
find free blocks to write to.<br></font></tt><br><tt><font size="3">YMMV.</font></tt><br><br><font size="3">On 6 November 2017 at 03:35, Carl <</font><a href="mailto:mutantllama@gmail.com" target="_blank"><font size="3" color="blue"><u>mutantllama@gmail.com</u></font></a><font size="3">>
wrote:</font><br></span><div><div class="m_-1832164167447791796h5"><font size="3">Hi Folk,</font><br><br><font size="3">Does anyone have much experience with the performance
of GPFS as it becomes close to full. In particular I am referring to split
data/meta data, where the data pool goes over 80% utilisation. </font><br><br><font size="3">How much degradation do you see above 80% usage, 90% usage?</font><br><br><font size="3">Cheers,</font><br><br><font size="3">Carl.</font><br><br><br></div></div><span><font size="3"><br>______________________________<wbr>_________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org_&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=zatwEi1ZS1fdfJjSR0DnoHq8vd75vVJcEyrIbaU1Hyw&s=ql8RA3_vpc7iOubmRTKJDfyZ78LA2tNhXA5x2oIRzNk&e=" target="_blank"><font size="3" color="blue"><u>spectrumscale.org</u></font></a><font size="3" color="blue"><u><br></u></font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=zatwEi1ZS1fdfJjSR0DnoHq8vd75vVJcEyrIbaU1Hyw&s=mnAoejImXp73wvcmpdLPXNi3JjPAmpAMDtzqIBnK7fs&e=" target="_blank"><font size="3" color="blue"><u>http://gpfsug.org/mailman/list<wbr>info/gpfsug-discuss</u></font></a><font size="3"><br></font><br><font size="3"><br><br><br>-- </font><br></span><span><font size="3" color="#4181c0"><b>Peter Smith</b> · Senior Systems Engineer<b><br>London</b> · New York · Los Angeles · Chicago · Montréal</font><font size="3"><br>T  <a href="tel:+44%2020%207344%208000" value="+442073448000" target="_blank">+44 (0)20 7344 8000</a> · M  </font><a href="tel:+44%20%280%297816%20123009" target="_blank"><font size="3" color="#0082bf">+44
(0)7816 123009</font></a><font size="3"> </font><font size="3" color="#0082bf"><i><br></i></font><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.co.uk_maps_place_19-2D23-2BWells-2BStreet-2C-2BLondon-2BW1T-2B3PQ&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=zatwEi1ZS1fdfJjSR0DnoHq8vd75vVJcEyrIbaU1Hyw&s=6acxGpabcJns6eGY7kvMNHkEl3PGz10xYjcsTVCjEiY&e=" target="_blank"><font size="3" color="#0082bf"><i>19-23
Wells Street, London W1T 3PQ</i></font></a><font size="3"> </font><font size="3" color="#0082bf"><br></font><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_framestore&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=zatwEi1ZS1fdfJjSR0DnoHq8vd75vVJcEyrIbaU1Hyw&s=Gmz8hKwZaaAM9ASBufpzpKG3_-nTjU8soMsH6j9ewAE&e=" target="_blank"><font size="3" color="#0082bf">Twitter</font></a><font size="3">· </font><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_framestore&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=zatwEi1ZS1fdfJjSR0DnoHq8vd75vVJcEyrIbaU1Hyw&s=fKTUmDeAcNoyVhMHbCLpBlJFtnLCJmToXgkHnQwup_E&e=" target="_blank"><font size="3" color="#0082bf">Facebook</font></a><font size="3">· </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.framestore.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=zatwEi1ZS1fdfJjSR0DnoHq8vd75vVJcEyrIbaU1Hyw&s=8lv8uxKsqfE4NVwEEhC7KM_poV0Tzc2lF0pyfds3t0E&e=" target="_blank"><font size="3" color="#0082bf">framestore.com</font></a><font size="3" color="blue"><u><br></u></font><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__www.framestore.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=zatwEi1ZS1fdfJjSR0DnoHq8vd75vVJcEyrIbaU1Hyw&s=Pfmf23Nhd4VeH3e8EPywrFcbegexNduC0Wi0CVlYjgI&e=" target="_blank"></a></span><span><tt><font size="2">______________________________<wbr>_________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=zatwEi1ZS1fdfJjSR0DnoHq8vd75vVJcEyrIbaU1Hyw&s=lBgAfk6EsHjN156F5mngZVzk6argsIeu8gQlu9OWH_c&e=" target="_blank">spectrumscale.org</a><br></font></tt><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=zatwEi1ZS1fdfJjSR0DnoHq8vd75vVJcEyrIbaU1Hyw&s=mnAoejImXp73wvcmpdLPXNi3JjPAmpAMDtzqIBnK7fs&e=" target="_blank"><tt><font size="2">http://gpfsug.org/mailman/list<wbr>info/gpfsug-discuss</font></tt></a><tt><font size="2"><br></font></tt><br><br><br></span><p></p><p></p><p></p><br>______________________________<wbr>_________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=zatwEi1ZS1fdfJjSR0DnoHq8vd75vVJcEyrIbaU1Hyw&s=lBgAfk6EsHjN156F5mngZVzk6argsIeu8gQlu9OWH_c&e=" rel="noreferrer" target="_blank">spectrumscale.org</a><br><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=zatwEi1ZS1fdfJjSR0DnoHq8vd75vVJcEyrIbaU1Hyw&s=mnAoejImXp73wvcmpdLPXNi3JjPAmpAMDtzqIBnK7fs&e=" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/list<wbr>info/gpfsug-discuss</a><br><br></div></div></blockquote></div><br></div></div></blockquote></div>Unless stated otherwise above:<br>
IBM United Kingdom Limited - Registered in England and Wales with number 741598. <br>
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU<br>
<br>
</div>
<br>______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a><br>
<br></blockquote></div><br></div>