<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">Michal,<div><br></div><div>I think you need to revise your testing method. Let me explain.</div><div><br></div><div>Based on my understandings:</div><div><br></div><div><p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 13px; line-height: normal; font-family: "Helvetica Neue"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;">3 FE servers and one storage system</p><p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 13px; line-height: normal; font-family: "Helvetica Neue"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><br></p><p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 13px; line-height: normal; font-family: "Helvetica Neue"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><ul class="MailOutline"><li>~4500 MiB/s from 8 RAID groups using XFS (one XFS per one RAID group) and parallel fio test.</li><li> one FS accross all 8 RAID groups and we observed performance drop down to ~3300</li></ul></p><p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 13px; line-height: normal; font-family: "Helvetica Neue"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><br></p><p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 13px; line-height: normal; font-family: "Helvetica Neue"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;">The test you are running is a non-clustered fs versus a clustered fs.</p><p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 13px; line-height: normal; font-family: "Helvetica Neue"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><br></p><p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 13px; line-height: normal; font-family: "Helvetica Neue"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;">XFS,</p><p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 13px; line-height: normal; font-family: "Helvetica Neue"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><br></p><p style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><ul class="MailOutline"><li><font face="Helvetica Neue" size="2">8 XFS filesystems.</font></li><li><font face="Helvetica Neue" size="2">Each FS has it own Array and independent Meta, not shared between nodes</font></li><li><font face="Helvetica Neue" size="2">Array will see sequential IO for each array and will be able to aggregate IO’s and prefetch on write.</font></li><li><font face="Helvetica Neue" size="2">No lock traffic between nodes</font></li><li><font face="Helvetica Neue" size="2">Didn’t mention for the FIO runs is this one node or the three nodes with fs’s spread across?</font></li></ul><div><font face="Helvetica Neue" size="2"><br></font></div><div><font face="Helvetica Neue" size="2">Clusterd,</font></div><div><font face="Helvetica Neue" size="2"><br></font></div><div><ul class="MailOutline"><li><font face="Helvetica Neue" size="2">1 Filesystem (fs0) In this case</font></li><li><font face="Helvetica Neue" size="2">Parallel Filsystem with shared Meta and access</font></li><li><font face="Helvetica Neue" size="2">Lock and Meta traffic across nodes</font></li><li><font face="Helvetica Neue" size="2">GPFS Stripes across NSD, 8 in this case. </font></li><li><font face="Helvetica Neue" size="2">Each FIO stream will in a less sequential stream at the array level</font></li><li><font face="Helvetica Neue" size="2">The LBA will be spread causing the array to work harder</font></li><li><font face="Helvetica Neue" size="2">Array logics will not see this a sequential and delivery a much lower performance from a sequential point of view.</font></li></ul></div></p><p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 13px; line-height: normal; font-family: "Helvetica Neue"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"> </p><p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 13px; line-height: normal; font-family: "Helvetica Neue"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><br></p><p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 13px; line-height: normal; font-family: "Helvetica Neue"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><br></p><p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 13px; line-height: normal; font-family: "Helvetica Neue"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;">What to do,</p><p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 13px; line-height: normal; font-family: "Helvetica Neue"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><br></p><p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 13px; line-height: normal; font-family: "Helvetica Neue"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;">Try </p><p style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><ol class="MailOutline"><li><font face="Helvetica Neue" size="2">8 FS with your FIO test like XFS test</font></li><li><font face="Helvetica Neue" size="2">1 FS 1 Array and matching 1 FIO ( then x8 result)</font></li></ol><div><font face="Helvetica Neue" size="2"><br></font></div><div><font face="Helvetica Neue" size="2"><br></font></div></p><p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 13px; line-height: normal; font-family: "Helvetica Neue"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><br></p><p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 13px; line-height: normal; font-family: "Helvetica Neue"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;">PS: You haven’t mention the type of array used? Sometimes the following is important.</p><p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 13px; line-height: normal; font-family: "Helvetica Neue"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><br></p><p style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><ul class="MailOutline"><li><font face="Helvetica Neue" size="2">Disable prefetch at the array. This causes the array to sometimes over work it backend due to incorrectly fetching data that is never used causing xtra io and cache displacement. Ie GPFS aggressively prefetches which triggers the array to do further prefetch and both are not used.</font></li></ul></p><p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 13px; line-height: normal; font-family: "Helvetica Neue"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><br></p></div><div>Dale<br><div><br><blockquote type="cite"><div>On 9 Feb 2024, at 6:30 am, Alec <anacreo@gmail.com> wrote:</div><br class="Apple-interchange-newline"><div><div dir="auto">This won't affect your current issue but if you're doing a lot of large sequential IO you may want to consider setting prefetchPct to 40 to 60 percent instead of default 20%. In our environment that has measurable impact, but we have a lot less ram than you do in the pagepool (8g versus 64g).<div dir="auto"><br></div><div dir="auto">Also do you have a dedicated meta pool? If not that could be a source of contention. Highly recommend a small pinnable LUN or two as a dedicated meta pool.</div><div dir="auto"><br></div><div dir="auto">Alec</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Feb 8, 2024, 7:01 AM Michal Hruška <<a href="mailto:Michal.Hruska@mcomputers.cz">Michal.Hruska@mcomputers.cz</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="CS" link="#0563C1" vlink="#954F72">
<div class="m_-6093307153041181205WordSection1"><p class="MsoNormal">@Aaron<u></u><u></u></p><p class="MsoNormal">Yes, I can confirm that 2MB blocks are transfered over.<br>
<br>
<u></u><u></u></p><p class="MsoNormal">@ Jan-Frode<u></u><u></u></p><p class="MsoNormal">We tried to change multiple parameters, but if you know the best combination for sequential IO, please let me know.<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">#mmlsconfig<u></u><u></u></p><p class="MsoNormal">autoload no<u></u><u></u></p><p class="MsoNormal">dmapiFileHandleSize 32<u></u><u></u></p><p class="MsoNormal">minReleaseLevel 5.1.9.0<u></u><u></u></p><p class="MsoNormal">tscCmdAllowRemoteConnections no<u></u><u></u></p><p class="MsoNormal">ccrEnabled yes<u></u><u></u></p><p class="MsoNormal">cipherList AUTHONLY<u></u><u></u></p><p class="MsoNormal">sdrNotifyAuthEnabled yes<u></u><u></u></p><p class="MsoNormal">pagepool 64G<u></u><u></u></p><p class="MsoNormal">maxblocksize 16384K<u></u><u></u></p><p class="MsoNormal">maxMBpS 40000<u></u><u></u></p><p class="MsoNormal">maxReceiverThreads 32<u></u><u></u></p><p class="MsoNormal">nsdMaxWorkerThreads 512<u></u><u></u></p><p class="MsoNormal">nsdMinWorkerThreads 8<u></u><u></u></p><p class="MsoNormal">nsdMultiQueue 256<u></u><u></u></p><p class="MsoNormal">nsdSmallThreadRatio 0<u></u><u></u></p><p class="MsoNormal">nsdThreadsPerQueue 3<u></u><u></u></p><p class="MsoNormal">prefetchAggressiveness 2<u></u><u></u></p><p class="MsoNormal">adminMode central<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">/dev/fs0<br>
<br>
<u></u><u></u></p><p class="MsoNormal">@Uwe<u></u><u></u></p><p class="MsoNormal">Using iohist we found out that gpfs is overloading one dm-device (it took about 500ms to finish IOs). We replaced the „problematic“ dm-device (as we have enough drives to play with) for new one but the overloading issue just jumped to another
dm-device.<br>
We believe that this behaviour is caused by the gpfs but we are unable to locate the root cause of it.<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p>
<div>
<div><p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Verdana",sans-serif;color:#595959">Best,<br>
Michal<u></u><u></u></span></p>
</div>
</div><p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://gpfsug.org/" rel="noreferrer noreferrer" target="_blank">gpfsug.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org" rel="noreferrer noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org</a><br>
</blockquote></div>
_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at gpfsug.org<br>http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org<br></div></blockquote></div><br></div></body></html>