<HTML><BODY><FONT style='white-space:pre-wrap;font-family: Helvetica Neue, Helvetica, Arial, sans-serif;margin: 1em 0;'>Rjx, that makes it a bit clearer.. as your vdisk is big enough to span over all pdisks in each of your test 1/1 or 1/2 or 1/4 of capacity... should bring the same performance. .. <br> <br>You mean something about vdisk Layout. .. <br>So in your test, for the full capacity test, you use just one vdisk per RG - so 2 in total for 'data' - right? <br> <br>What about Md .. did you create separate vdisk for MD / what size then ? <br><br>Gesendet von IBM Verse</FONT><br><br><div class="domino-section" dir="ltr"><div class="domino-section-head"><span class="domino-section-title"><font color="#424282">Ivano Talamo --- Re: [gpfsug-discuss] Write performances and filesystem size --- </font></span></div><div class="domino-section-body"><br><table width="100%" border="0" cellspacing="0" cellpadding="0"><tr valign="top"><td width="1%" style="width: 96px;"><font size="2" color="#5F5F5F">Von:</font></td><td width="100%" style="width: auto;"><font size="2">"Ivano Talamo" <Ivano.Talamo@psi.ch></font></td></tr><tr valign="top"><td width="1%" style="width: 96px;"><font size="2" color="#5F5F5F">An:</font></td><td width="100%" style="width: auto;"><font size="2">"gpfsug main discussion list" <gpfsug-discuss@spectrumscale.org></font></td></tr><tr valign="top"><td width="1%" style="width: 96px;"><font size="2" color="#5F5F5F">Datum:</font></td><td width="100%" style="width: auto;"><font size="2">Do. 16.11.2017 03:49</font></td></tr><tr valign="top"><td width="1%" style="width: 96px;"><font size="2" color="#5F5F5F">Betreff:</font></td><td width="100%" style="width: auto;"><font size="2">Re: [gpfsug-discuss] Write performances and filesystem size</font></td></tr></table><hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br></div><html><body><div style="opacity: 0,87;"><pre style="white-space:pre-wrap;"><font color="#000000">Hello Olaf,<br/><br/>yes, I confirm that is the Lenovo version of the ESS GL2, so 2 <br/>enclosures/4 drawers/166 disks in total.<br/><br/>Each recovery group has one declustered array with all disks inside, so <br/>vdisks use all the physical ones, even in the case of a vdisk that is <br/>1/4 of the total size.<br/><br/>Regarding the layout allocation we used scatter.<br/><br/>The tests were done on the just created filesystem, so no close-to-full <br/>effect. And we run gpfsperf write seq.<br/><br/>Thanks,<br/>Ivano<br/><br/><br/>Il 16/11/17 04:42, Olaf Weiser ha scritto:<br/><!-- -->> Sure... as long we assume that really all physical disk are used .. the<br/><!-- -->> fact that was told 1/2 or 1/4 might turn out that one / two complet<br/><!-- -->> enclosures 're eliminated ... ? ..that s why I was asking for more<br/><!-- -->> details ..<br/><!-- -->><br/><!-- -->> I dont see this degration in my environments. . as long the vdisks are<br/><!-- -->> big enough to span over all pdisks ( which should be the case for<br/><!-- -->> capacity in a range of TB ) ... the performance stays the same<br/><!-- -->><br/><!-- -->> Gesendet von IBM Verse<br/><!-- -->><br/><!-- -->> Jan-Frode Myklebust --- Re: [gpfsug-discuss] Write performances and<br/><!-- -->> filesystem size ---<br/><!-- -->><br/><!-- -->> Von: "Jan-Frode Myklebust" <!-- --><janfrode@tanso.net<!-- -->><br/><!-- -->> An: "gpfsug main discussion list" <!-- --><gpfsug-discuss@spectrumscale.org<!-- -->><br/><!-- -->> Datum: Mi. 15.11.2017 21:35<br/><!-- -->> Betreff: Re: [gpfsug-discuss] Write performances and filesystem size<br/><!-- -->><br/><!-- -->> ------------------------------------------------------------------------<br/><!-- -->><br/><!-- -->> Olaf, this looks like a Lenovo «ESS GLxS» version. Should be using same<br/><!-- -->> number of spindles for any size filesystem, so I would also expect them<br/><!-- -->> to perform the same.<br/><!-- -->><br/><!-- -->><br/><!-- -->><br/><!-- -->> -jf<br/><!-- -->><br/><!-- -->><br/><!-- -->> ons. 15. nov. 2017 kl. 11:26 skrev Olaf Weiser <!-- --><olaf.weiser@de.ibm.com<br/><!-- -->> <!-- --><mailto:olaf.weiser@de.ibm.com<!-- -->><!-- -->>:<br/><!-- -->><br/><!-- -->> to add a comment ... .. very simply... depending on how you<br/><!-- -->> allocate the physical block storage .... if you - simply - using<br/><!-- -->> less physical resources when reducing the capacity (in the same<br/><!-- -->> ratio) .. you get , what you see....<br/><!-- -->><br/><!-- -->> so you need to tell us, how you allocate your block-storage .. (Do<br/><!-- -->> you using RAID controllers , where are your LUNs coming from, are<br/><!-- -->> then less RAID groups involved, when reducing the capacity ?...)<br/><!-- -->><br/><!-- -->> GPFS can be configured to give you pretty as much as what the<br/><!-- -->> hardware can deliver.. if you reduce resource.. ... you'll get less<br/><!-- -->> , if you enhance your hardware .. you get more... almost regardless<br/><!-- -->> of the total capacity in #blocks ..<br/><!-- -->><br/><!-- -->><br/><!-- -->><br/><!-- -->><br/><!-- -->><br/><!-- -->><br/><!-- -->> From: "Kumaran Rajaram" <!-- --><kums@us.ibm.com<br/><!-- -->> <!-- --><mailto:kums@us.ibm.com<!-- -->><!-- -->><br/><!-- -->> To: gpfsug main discussion list<br/><!-- -->> <!-- --><gpfsug-discuss@spectrumscale.org<br/><!-- -->> <!-- --><mailto:gpfsug-discuss@spectrumscale.org<!-- -->><!-- -->><br/><!-- -->> Date: 11/15/2017 11:56 AM<br/><!-- -->> Subject: Re: [gpfsug-discuss] Write performances and<br/><!-- -->> filesystem size<br/><!-- -->> Sent by: gpfsug-discuss-bounces@spectrumscale.org<br/><!-- -->> <!-- --><mailto:gpfsug-discuss-bounces@spectrumscale.org<!-- -->><br/><!-- -->> ------------------------------------------------------------------------<br/><!-- -->><br/><!-- -->><br/><!-- -->><br/><!-- -->> Hi,<br/><!-- -->><br/><!-- -->> <!-- -->><!-- -->>Am I missing something? Is this an expected behaviour and someone<br/><!-- -->> has an explanation for this?<br/><!-- -->><br/><!-- -->> Based on your scenario, write degradation as the file-system is<br/><!-- -->> populated is possible if you had formatted the file-system with "-j<br/><!-- -->> cluster".<br/><!-- -->><br/><!-- -->> For consistent file-system performance, we recommend *mmcrfs "-j<br/><!-- -->> scatter" layoutMap.* Also, we need to ensure the mmcrfs "-n" is<br/><!-- -->> set properly.<br/><!-- -->><br/><!-- -->> [snip from mmcrfs]/<br/><!-- -->> # mmlsfs <!-- --><fs<!-- -->> | egrep 'Block allocation| Estimated number'<br/><!-- -->> -j scatter Block allocation type<br/><!-- -->> -n 128 Estimated number of<br/><!-- -->> nodes that will mount file system/<br/><!-- -->> [/snip]<br/><!-- -->><br/><!-- -->><br/><!-- -->> [snip from man mmcrfs]/<br/><!-- -->> *layoutMap={scatter|*//*cluster}*//<br/><!-- -->> Specifies the block allocation map type. When<br/><!-- -->> allocating blocks for a given file, GPFS first<br/><!-- -->> uses a round‐robin algorithm to spread the data<br/><!-- -->> across all disks in the storage pool. After a<br/><!-- -->> disk is selected, the location of the data<br/><!-- -->> block on the disk is determined by the block<br/><!-- -->> allocation map type*. If cluster is<br/><!-- -->> specified, GPFS attempts to allocate blocks in<br/><!-- -->> clusters. Blocks that belong to a particular<br/><!-- -->> file are kept adjacent to each other within<br/><!-- -->> each cluster. If scatter is specified,<br/><!-- -->> the location of the block is chosen randomly.*/<br/><!-- -->> /<br/><!-- -->> * The cluster allocation method may provide<br/><!-- -->> better disk performance for some disk<br/><!-- -->> subsystems in relatively small installations.<br/><!-- -->> The benefits of clustered block allocation<br/><!-- -->> diminish when the number of nodes in the<br/><!-- -->> cluster or the number of disks in a file system<br/><!-- -->> increases, or when the file system’s free space<br/><!-- -->> becomes fragmented. *//The *cluster*//<br/><!-- -->> allocation method is the default for GPFS<br/><!-- -->> clusters with eight or fewer nodes and for file<br/><!-- -->> systems with eight or fewer disks./<br/><!-- -->> /<br/><!-- -->> *The scatter allocation method provides<br/><!-- -->> more consistent file system performance by<br/><!-- -->> averaging out performance variations due to<br/><!-- -->> block location (for many disk subsystems, the<br/><!-- -->> location of the data relative to the disk edge<br/><!-- -->> has a substantial effect on performance).*//This<br/><!-- -->> allocation method is appropriate in most cases<br/><!-- -->> and is the default for GPFS clusters with more<br/><!-- -->> than eight nodes or file systems with more than<br/><!-- -->> eight disks./<br/><!-- -->> /<br/><!-- -->> The block allocation map type cannot be changed<br/><!-- -->> after the storage pool has been created./<br/><!-- -->><br/><!-- -->> */<br/><!-- -->> -n/*/*NumNodes*//<br/><!-- -->> The estimated number of nodes that will mount the file<br/><!-- -->> system in the local cluster and all remote clusters.<br/><!-- -->> This is used as a best guess for the initial size of<br/><!-- -->> some file system data structures. The default is 32.<br/><!-- -->> This value can be changed after the file system has been<br/><!-- -->> created but it does not change the existing data<br/><!-- -->> structures. Only the newly created data structure is<br/><!-- -->> affected by the new value. For example, new storage<br/><!-- -->> pool./<br/><!-- -->> /<br/><!-- -->> When you create a GPFS file system, you might want to<br/><!-- -->> overestimate the number of nodes that will mount the<br/><!-- -->> file system. GPFS uses this information for creating<br/><!-- -->> data structures that are essential for achieving maximum<br/><!-- -->> parallelism in file system operations (For more<br/><!-- -->> information, see GPFS architecture in IBM Spectrum<br/><!-- -->> Scale: Concepts, Planning, and Installation Guide ). If<br/><!-- -->> you are sure there will never be more than 64 nodes,<br/><!-- -->> allow the default value to be applied. If you are<br/><!-- -->> planning to add nodes to your system, you should specify<br/><!-- -->> a number larger than the default./<br/><!-- -->><br/><!-- -->> [/snip from man mmcrfs]<br/><!-- -->><br/><!-- -->> Regards,<br/><!-- -->> -Kums<br/><!-- -->><br/><!-- -->><br/><!-- -->><br/><!-- -->><br/><!-- -->><br/><!-- -->> From: Ivano Talamo <!-- --><Ivano.Talamo@psi.ch<br/><!-- -->> <!-- --><mailto:Ivano.Talamo@psi.ch<!-- -->><!-- -->><br/><!-- -->> To: <!-- --><gpfsug-discuss@spectrumscale.org<br/><!-- -->> <!-- --><mailto:gpfsug-discuss@spectrumscale.org<!-- -->><!-- -->><br/><!-- -->> Date: 11/15/2017 11:25 AM<br/><!-- -->> Subject: [gpfsug-discuss] Write performances and filesystem size<br/><!-- -->> Sent by: gpfsug-discuss-bounces@spectrumscale.org<br/><!-- -->> <!-- --><mailto:gpfsug-discuss-bounces@spectrumscale.org<!-- -->><br/><!-- -->> ------------------------------------------------------------------------<br/><!-- -->><br/><!-- -->><br/><!-- -->><br/><!-- -->> Hello everybody,<br/><!-- -->><br/><!-- -->> together with my colleagues we are actually running some tests on a new<br/><!-- -->> DSS G220 system and we see some unexpected behaviour.<br/><!-- -->><br/><!-- -->> What we actually see is that write performances (we did not test read<br/><!-- -->> yet) decreases with the decrease of filesystem size.<br/><!-- -->><br/><!-- -->> I will not go into the details of the tests, but here are some numbers:<br/><!-- -->><br/><!-- -->> - with a filesystem using the full 1.2 PB space we get 14 GB/s as the<br/><!-- -->> sum of the disk activity on the two IO servers;<br/><!-- -->> - with a filesystem using half of the space we get 10 GB/s;<br/><!-- -->> - with a filesystem using 1/4 of the space we get 5 GB/s.<br/><!-- -->><br/><!-- -->> We also saw that performances are not affected by the vdisks layout,<br/><!-- -->> ie.<br/><!-- -->> taking the full space with one big vdisk or 2 half-size vdisks per RG<br/><!-- -->> gives the same performances.<br/><!-- -->><br/><!-- -->> To our understanding the IO should be spread evenly across all the<br/><!-- -->> pdisks in the declustered array, and looking at iostat all disks<br/><!-- -->> seem to<br/><!-- -->> be accessed. But so there must be some other element that affects<br/><!-- -->> performances.<br/><!-- -->><br/><!-- -->> Am I missing something? Is this an expected behaviour and someone<br/><!-- -->> has an<br/><!-- -->> explanation for this?<br/><!-- -->><br/><!-- -->> Thank you,<br/><!-- -->> Ivano<br/><!-- -->> _______________________________________________<br/><!-- -->> gpfsug-discuss mailing list<br/><!-- -->> gpfsug-discuss at spectrumscale.org <!-- --><http://spectrumscale.org<!-- -->>_<br/><!-- -->> __https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=McIf98wfiVqHU8ZygezLrQ&m=py_FGl3hi9yQsby94NZdpBFPwcUU0FREyMSSvuK_10U&s=Bq1J9eIXxadn5yrjXPHmKEht0CDBwfKJNH72p--T-6s&e=_<br/><!-- -->><br/><!-- -->><br/><!-- -->> _______________________________________________<br/><!-- -->> gpfsug-discuss mailing list<br/><!-- -->> gpfsug-discuss at spectrumscale.org <!-- --><http://spectrumscale.org<!-- -->><br/><!-- -->> http://gpfsug.org/mailman/listinfo/gpfsug-discuss<br/><!-- -->><br/><!-- -->><br/><!-- -->><br/><!-- -->> _______________________________________________<br/><!-- -->> gpfsug-discuss mailing list<br/><!-- -->> gpfsug-discuss at spectrumscale.org <!-- --><http://spectrumscale.org<!-- -->><br/><!-- -->> http://gpfsug.org/mailman/listinfo/gpfsug-discuss<br/><!-- -->><br/><!-- -->><br/><!-- -->><br/><!-- -->><br/><!-- -->> _______________________________________________<br/><!-- -->> gpfsug-discuss mailing list<br/><!-- -->> gpfsug-discuss at spectrumscale.org<br/><!-- -->> http://gpfsug.org/mailman/listinfo/gpfsug-discuss<br/><!-- -->><br/>_______________________________________________<br/>gpfsug-discuss mailing list<br/>gpfsug-discuss at spectrumscale.org<br/>http://gpfsug.org/mailman/listinfo/gpfsug-discuss<br/></font></pre></div><BR>
</body></html></BODY></HTML>