<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto">My 2c ...<div>Be careful here about mixing up three different possible effects seen in filesystems<div><br></div><div>1. Performance degradation as the filesystem approaches 100% full, often due to the difficulty of finding the remaining unallocated blocks.</div><div>GPFS doesn’t noticeably suffer from this effect compared to its competitors.</div><div><br></div><div>2. Performance degradation over time as files get fragmented and so cause extra movement of the actuator arm of a HDD. (hence defrag on Windows and the idea of short stroking drives).</div><div><br></div><div>3. Performance degradation as blocks are written further from the fastest part of a hard disk drive. SSDs do not show this effect. </div><div><br></div><div><br></div><div>Benchmarks on newly formatted empty filesystems are often artificially high compared to performance after say 12 months whether or not the filesystem is near 90%+ capacity utilisation. The -j scatter option allows for more realistic performance measurement when designing for the long term usage of the filesystem. But this is due to the distributed location of the blocks not how full the filesystem is.</div><div><br></div><div><br><br><div id="AppleMailSignature"><div style="margin-bottom: 0.0001pt; line-height: normal;"><span style="background-color: rgba(255, 255, 255, 0);">Daniel</span></div><div style="margin-bottom: 0.0001pt; line-height: normal;"><span style="background-color: rgba(255, 255, 255, 0);"><img alt="/spectrum_storage-banne" src="http://ausgsa.ibm.com/projects/t/tivoli_visual_design/public/2015/Spectrum-Storage/Email-signatures/Storage/spectrum_storage-banner.png" width="432.5" height="auto" style="width: 601px; height: 5px;"></span></div><div style="margin-bottom: 0.0001pt; line-height: normal;"><span style="background-color: rgba(255, 255, 255, 0);"><br> </span></div><table border="0" cellpadding="0" cellspacing="0" style="font-family: sans-serif;"><tbody><tr style="word-wrap: break-word; max-width: 447.5px;"><td style="word-wrap: break-word; max-width: 447.5px; width: 201px; padding: 0cm;"><div style="margin-bottom: 0.0001pt; line-height: normal;"><img alt="Spectrum Scale Logo" src="http://ausgsa.ibm.com/projects/t/tivoli_visual_design/public/2015/Spectrum-Storage/Email-signatures/Storage/spectrum_scale-logo.png" style="width: 75px; height: 120px; float: left;"></div><div style="margin-bottom: 0.0001pt; line-height: normal;"><font face=".SFUIDisplay"><span style="background-color: rgba(255, 255, 255, 0);"> </span></font></div></td><td style="word-wrap: break-word; max-width: 447.5px; width: 21px; padding: 0cm;"><font face=".SFUIDisplay"><span style="background-color: rgba(255, 255, 255, 0);"> </span></font></td><td style="word-wrap: break-word; max-width: 447.5px; width: 202px; padding: 0cm;"><div style="margin-bottom: 0.0001pt; line-height: normal;"><font face=".SFUIDisplay"><span style="background-color: rgba(255, 255, 255, 0);"><strong>Dr Daniel Kidger</strong> <br>IBM Technical Sales Specialist<br>Software Defined Solution Sales<br><br><a dir="ltr" href="tel:+%2044-7818%20522%20266" x-apple-data-detectors="true" x-apple-data-detectors-type="telephone" x-apple-data-detectors-result="1">+</a> <a dir="ltr" href="tel:+%2044-7818%20522%20266" x-apple-data-detectors="true" x-apple-data-detectors-type="telephone" x-apple-data-detectors-result="1">44-(0)7818 522 266</a> <br><a dir="ltr" href="mailto:daniel.kidger@uk.ibm.com" x-apple-data-detectors="true" x-apple-data-detectors-type="link" x-apple-data-detectors-result="2">daniel.kidger@uk.ibm.com</a></span></font></div></td></tr></tbody></table></div><div><br>On 15 Nov 2017, at 11:26, Olaf Weiser <<a href="mailto:olaf.weiser@de.ibm.com">olaf.weiser@de.ibm.com</a>> wrote:<br><br></div><blockquote type="cite"><div><font size="2" face="sans-serif"> to add a comment ... .. very
simply... depending on how you allocate the physical block storage ....
if you - simply - using less physical resources when reducing the capacity
(in the same ratio) .. you get , what you see.... </font><br><br><font size="2" face="sans-serif">so you need to tell us, how you allocate
your block-storage .. (Do you using RAID controllers , where are your LUNs
coming from, are then less RAID groups involved, when reducing the capacity
?...) </font><br><br><font size="2" face="sans-serif">GPFS can be configured to give you pretty
as much as what the hardware can deliver.. if you reduce resource.. ...
you'll get less , if you enhance your hardware .. you get more... almost
regardless of the total capacity in #blocks .. </font><br><br><br><br><br><br><br><font size="1" color="#5f5f5f" face="sans-serif">From:
</font><font size="1" face="sans-serif">"Kumaran Rajaram"
<<a href="mailto:kums@us.ibm.com">kums@us.ibm.com</a>></font><br><font size="1" color="#5f5f5f" face="sans-serif">To:
</font><font size="1" face="sans-serif">gpfsug main discussion
list <<a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a>></font><br><font size="1" color="#5f5f5f" face="sans-serif">Date:
</font><font size="1" face="sans-serif">11/15/2017 11:56 AM</font><br><font size="1" color="#5f5f5f" face="sans-serif">Subject:
</font><font size="1" face="sans-serif">Re: [gpfsug-discuss]
Write performances and filesystem size</font><br><font size="1" color="#5f5f5f" face="sans-serif">Sent by:
</font><font size="1" face="sans-serif"><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-bounces@spectrumscale.org</a></font><br><hr noshade=""><br><br><br><font size="2" face="Arial">Hi,</font><font size="3" face="sans-serif"><br></font><font size="2" color="red" face="Arial"><br>>>Am I missing something? Is this an expected behaviour and someone
has an explanation for this?</font><font size="3" face="sans-serif"><br></font><font size="2" face="Arial"><br>Based on your scenario, write degradation as the file-system is populated
is possible if you had formatted the file-system with "-j cluster".
</font><font size="3" face="sans-serif"><br></font><font size="2" face="Arial"><br>For consistent file-system performance, we recommend <b>mmcrfs "-j
scatter" layoutMap.</b> Also, we need to ensure the mmcrfs
"-n" is set properly.</font><font size="3" face="sans-serif"><br></font><font size="2" face="Arial"><br>[snip from mmcrfs]</font><font size="2" color="blue" face="Arial"><i><br># mmlsfs <fs> | egrep 'Block allocation| Estimated number'<br> -j scatter
Block allocation
type<br> -n 128
Estimated
number of nodes that will mount file system</i></font><font size="2" face="Arial"><br>[/snip]</font><font size="3" face="sans-serif"><br><br></font><font size="2" face="Arial"><br>[snip from man mmcrfs]</font><font size="2" color="blue" face="Arial"><i><br> <b>layoutMap={scatter|</b></i><i> <b>cluster}</b></i><i><br> Specifies
the block allocation map type. When<br> allocating
blocks for a given file, GPFS first<br> uses a round‐robin
algorithm to spread the data<br> across all
disks in the storage pool. After a<br> disk is
selected, the location of the data<br> block on
the disk is determined by the block<br> allocation
map type<b>. If cluster is<br> specified,
GPFS attempts to allocate blocks in<br> clusters.
Blocks that belong to a particular<br> file are
kept adjacent to each other within<br> each cluster.
If scatter is specified,<br> the location
of the block is chosen randomly.</b></i></font><font size="3" face="sans-serif"><br></font><font size="2" color="blue" face="Arial"><i><br> <b> The cluster
allocation method may provide<br> better disk
performance for some disk<br> subsystems
in relatively small installations.<br> The benefits
of clustered block allocation<br> diminish
when the number of nodes in the<br> cluster
or the number of disks in a file system<br> increases,
or when the file system’s free space<br> becomes
fragmented. </b></i><i>The <b>cluster</b></i><i><br> allocation
method is the default for GPFS<br> clusters
with eight or fewer nodes and for file<br> systems
with eight or fewer disks.</i></font><font size="3" face="sans-serif"><br></font><font size="2" color="blue" face="Arial"><i><br> <b>The scatter
allocation method provides<br> more consistent
file system performance by<br> averaging
out performance variations due to<br> block location
(for many disk subsystems, the<br> location
of the data relative to the disk edge<br> has a substantial
effect on performance).</b></i><i>This<br> allocation
method is appropriate in most cases<br> and is the
default for GPFS clusters with more<br> than eight
nodes or file systems with more than<br> eight disks.</i></font><font size="3" face="sans-serif"><br></font><font size="2" color="blue" face="Arial"><i><br> The block
allocation map type cannot be changed<br> after the
storage pool has been created.</i></font><font size="3" face="sans-serif"><br><br></font><font size="2" color="blue" face="Arial"><b><i><br>-n</i></b><i> <b>NumNodes</b></i><i><br> The estimated number of nodes that will mount
the file<br> system in the local cluster and all remote
clusters.<br> This is used as a best guess for the initial
size of<br> some file system data structures. The default
is 32.<br> This value can be changed after the file system
has been<br> created but it does not change the existing
data<br> structures. Only the newly created data structure
is<br> affected by the new value. For example, new
storage<br> pool.</i></font><font size="3" face="sans-serif"><br></font><font size="2" color="blue" face="Arial"><i><br> When you create a GPFS file system, you might
want to<br> overestimate the number of nodes that will
mount the<br> file system. GPFS uses this information for
creating<br> data structures that are essential for achieving
maximum<br> parallelism in file system operations (For
more<br> information, see GPFS architecture in IBM
Spectrum<br> Scale: Concepts, Planning, and Installation
Guide ). If<br> you are sure there will never be more than
64 nodes,<br> allow the default value to be applied. If
you are<br> planning to add nodes to your system, you
should specify<br> a number larger than the default.</i></font><font size="3" face="sans-serif"><br></font><font size="2" face="Arial"><br>[/snip from man mmcrfs]</font><font size="3" face="sans-serif"><br></font><font size="2" face="Arial"><br>Regards,<br>-Kums</font><font size="3" face="sans-serif"><br><br><br><br><br></font><font size="1" color="#5f5f5f" face="sans-serif"><br>From: </font><font size="1" face="sans-serif">Ivano
Talamo <<a href="mailto:Ivano.Talamo@psi.ch">Ivano.Talamo@psi.ch</a>></font><font size="1" color="#5f5f5f" face="sans-serif"><br>To: </font><font size="1" face="sans-serif"><<a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a>></font><font size="1" color="#5f5f5f" face="sans-serif"><br>Date: </font><font size="1" face="sans-serif">11/15/2017
11:25 AM</font><font size="1" color="#5f5f5f" face="sans-serif"><br>Subject: </font><font size="1" face="sans-serif">[gpfsug-discuss]
Write performances and filesystem size</font><font size="1" color="#5f5f5f" face="sans-serif"><br>Sent by: </font><font size="1" face="sans-serif"><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-bounces@spectrumscale.org</a></font><font size="3" face="sans-serif"><br></font><hr noshade=""><font size="3" face="sans-serif"><br><br></font><tt><font size="2"><br>Hello everybody,<br><br>together with my colleagues we are actually running some tests on a new
<br>DSS G220 system and we see some unexpected behaviour.<br><br>What we actually see is that write performances (we did not test read <br>yet) decreases with the decrease of filesystem size.<br><br>I will not go into the details of the tests, but here are some numbers:<br><br>- with a filesystem using the full 1.2 PB space we get 14 GB/s as the <br>sum of the disk activity on the two IO servers;<br>- with a filesystem using half of the space we get 10 GB/s;<br>- with a filesystem using 1/4 of the space we get 5 GB/s.<br><br>We also saw that performances are not affected by the vdisks layout, ie.
<br>taking the full space with one big vdisk or 2 half-size vdisks per RG <br>gives the same performances.<br><br>To our understanding the IO should be spread evenly across all the <br>pdisks in the declustered array, and looking at iostat all disks seem to
<br>be accessed. But so there must be some other element that affects <br>performances.<br><br>Am I missing something? Is this an expected behaviour and someone has an
<br>explanation for this?<br><br>Thank you,<br>Ivano<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org">spectrumscale.org</a></font></tt><font size="3" color="blue" face="sans-serif"><u><br></u></font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=McIf98wfiVqHU8ZygezLrQ&m=py_FGl3hi9yQsby94NZdpBFPwcUU0FREyMSSvuK_10U&s=Bq1J9eIXxadn5yrjXPHmKEht0CDBwfKJNH72p--T-6s&e="><tt><font size="2" color="blue"><u>https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=McIf98wfiVqHU8ZygezLrQ&m=py_FGl3hi9yQsby94NZdpBFPwcUU0FREyMSSvuK_10U&s=Bq1J9eIXxadn5yrjXPHmKEht0CDBwfKJNH72p--T-6s&e=</u></font></tt></a><tt><font size="2"><br></font></tt><font size="3" face="sans-serif"><br><br></font><tt><font size="2">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org">spectrumscale.org</a><br></font></tt><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFJg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=Yu5Gt0RPmbb6KaS_emGivhq5C2A33w5DeecdU2aLViQ&s=K0Mz-y4oBH66YUf1syIXaQ3hxck6WjeEMsM-HNHhqAU&e="><tt><font size="2">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size="2"><br></font></tt><br><br><br>
<pre>_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at <a href="http://spectrumscale.org">spectrumscale.org</a><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=Yu5Gt0RPmbb6KaS_emGivhq5C2A33w5DeecdU2aLViQ&s=K0Mz-y4oBH66YUf1syIXaQ3hxck6WjeEMsM-HNHhqAU&e=">https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=Yu5Gt0RPmbb6KaS_emGivhq5C2A33w5DeecdU2aLViQ&s=K0Mz-y4oBH66YUf1syIXaQ3hxck6WjeEMsM-HNHhqAU&e=</a></pre><br></div></blockquote></div></div>Unless stated otherwise above:<BR>
IBM United Kingdom Limited - Registered in England and Wales with number 741598. <BR>
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU<BR>
<BR>
</body></html>