<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:8.0pt;
margin-left:36.0pt;
mso-add-space:auto;
line-height:106%;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
p.MsoListParagraphCxSpFirst, li.MsoListParagraphCxSpFirst, div.MsoListParagraphCxSpFirst
{mso-style-priority:34;
mso-style-type:export-only;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
mso-add-space:auto;
line-height:106%;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
p.MsoListParagraphCxSpMiddle, li.MsoListParagraphCxSpMiddle, div.MsoListParagraphCxSpMiddle
{mso-style-priority:34;
mso-style-type:export-only;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
mso-add-space:auto;
line-height:106%;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
p.MsoListParagraphCxSpLast, li.MsoListParagraphCxSpLast, div.MsoListParagraphCxSpLast
{mso-style-priority:34;
mso-style-type:export-only;
margin-top:0cm;
margin-right:0cm;
margin-bottom:8.0pt;
margin-left:36.0pt;
mso-add-space:auto;
line-height:106%;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.StylE-mailovZprvy18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.StylE-mailovZprvy19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:738140146;
mso-list-type:hybrid;
mso-list-template-ids:1307371486 2006719920 67436569 67436571 67436559 67436569 67436571 67436559 67436569 67436571;}
@list l0:level1
{mso-level-text:"%1\.\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1
{mso-list-id:1551377003;
mso-list-type:hybrid;
mso-list-template-ids:814090328 721344946 67436569 67436571 67436559 67436569 67436571 67436559 67436569 67436571;}
@list l1:level1
{mso-level-text:"%1\.\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="CS" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Dear friends,</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Thank you all for your time and thoughts/ideas!<br>
<br>
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">The main goal for sharing our test results comparing XFS and GPFS was to show, that the storage subsystem is able to do better if the I/O is provided in different way. We were not trying to compare XFS and GPFS directly,
we understand that there will be some performance drop using GPFS (compared to “raw” performance) but we are just surprised by the ~20-25% performance drop.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">We tried to change multiple suggested parameters but we got no performance gain. As there was no change we tried to do more troubleshooting using different configurations.<br>
<br>
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">To better understand what we tried I have to describe our environment a bit more:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Our underlying storage system is IBM FS7300 (each controller has 384 GB cache). There are 8 DRAIDs (8+2+1). Each DRAID has its own pool and each pool has one Volume (LUN). Every FE server (we have 3 of them) is connected
directly to this storage using two 32 GFC connections. 3 client servers and FE servers are connected to LAN switch using 100GbE connection.
<br>
<br>
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Testing results (metadata are located on NVMe SSD DRAID):<o:p></o:p></span></p>
<ol style="margin-top:0cm" start="1" type="1">
<li class="MsoListParagraphCxSpFirst" style="margin-left:0cm;mso-add-space:auto;mso-list:l1 level1 lfo1">
<span lang="EN-US">We used second - identical storage to test the performance but we are getting almost the same results compared to first storage. In iohist we can see that one LUN (dm-device) is probably overloaded as IO time is high – from 300 to 500 ms.
<o:p></o:p></span></li><li class="MsoListParagraphCxSpMiddle" style="margin-left:0cm;mso-add-space:auto;mso-list:l1 level1 lfo1">
<span lang="EN-US">Using both storage systems together in one big FS (GPFS): always is only one dm-device slow (according to iohist output) but the “problematic” dm-device changes in time.<o:p></o:p></span></li><li class="MsoListParagraphCxSpMiddle" style="margin-left:0cm;mso-add-space:auto;mso-list:l1 level1 lfo1">
<span lang="EN-US">During out tests we also tried synchronous fio test but we observed significant performance drop.<o:p></o:p></span></li><li class="MsoListParagraphCxSpLast" style="margin-left:0cm;mso-add-space:auto;mso-list:l1 level1 lfo1">
<span lang="EN-US">We tried to compare single LUN performance GPFS against XFS: GPFS 435MB/s compared to XFS 485MB/s. From single server. The drop is not so significant but when we added more LUNs to the comparison the performance drop was more painful.<o:p></o:p></span></li></ol>
<p class="MsoNormal"><span lang="EN-US">For this testing “session” we were able to gather data by Storage Insights to check storage performance:<o:p></o:p></span></p>
<ol style="margin-top:0cm" start="1" type="1">
<li class="MsoListParagraphCxSpFirst" style="margin-left:0cm;mso-add-space:auto;mso-list:l0 level1 lfo2">
<span lang="EN-US">There is no problematic HDD – the worst latency seen is 42ms from all 176 drives in two storage systems. Average latency is 15ms.<o:p></o:p></span></li><li class="MsoListParagraphCxSpMiddle" style="margin-left:0cm;mso-add-space:auto;mso-list:l0 level1 lfo2">
<span lang="EN-US">CPU usage was at 25% max.<o:p></o:p></span></li><li class="MsoListParagraphCxSpMiddle" style="margin-left:0cm;mso-add-space:auto;mso-list:l0 level1 lfo2">
<span lang="EN-US">“Problematic” DRAID latency – average is 16ms the worst is 430ms. I can not tell if there was the same peak in latency during XFS tests but I think that no (or not so bad) – as the XFS is able to perform better than GPFS.
<o:p></o:p></span></li><li class="MsoListParagraphCxSpLast" style="margin-left:0cm;mso-add-space:auto;mso-list:l0 level1 lfo2">
<span lang="EN-US">During our tests the write cache for all pools was fully allocated. Both for XFS and GPFS tests. Which is expected state as the cache is much faster than HDDs and it should help organize writes before they are forwarded to RAID groups.<o:p></o:p></span></li></ol>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Do you see some other possible problems we missed?
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I do not want to leave it behind “unfinished” but I am out of ideas.
</span><span lang="EN-US" style="font-family:"Segoe UI Emoji",sans-serif">😊</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Best,<br>
Michal<br>
<br>
<o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="mso-fareast-language:CS">From:</span></b><span style="mso-fareast-language:CS"> Michal Hruška
<br>
<b>Sent:</b> Thursday, February 8, 2024 3:59 PM<br>
<b>To:</b> 'gpfsug-discuss@gpfsug.org' <gpfsug-discuss@gpfsug.org><br>
<b>Subject:</b> Re: [gpfsug-discuss] sequential I/O write - performance<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">@Aaron<o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">Yes, I can confirm that 2MB blocks are transfered over.<o:p></o:p></p>
<p class="MsoNormal">@ Jan-Frode<o:p></o:p></p>
<p class="MsoNormal">We tried to change multiple parameters, but if you know the best combination for sequential IO, please let me know.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">#mmlsconfig<o:p></o:p></p>
<p class="MsoNormal">autoload no<o:p></o:p></p>
<p class="MsoNormal">dmapiFileHandleSize 32<o:p></o:p></p>
<p class="MsoNormal">minReleaseLevel 5.1.9.0<o:p></o:p></p>
<p class="MsoNormal">tscCmdAllowRemoteConnections no<o:p></o:p></p>
<p class="MsoNormal">ccrEnabled yes<o:p></o:p></p>
<p class="MsoNormal">cipherList AUTHONLY<o:p></o:p></p>
<p class="MsoNormal">sdrNotifyAuthEnabled yes<o:p></o:p></p>
<p class="MsoNormal">pagepool 64G<o:p></o:p></p>
<p class="MsoNormal">maxblocksize 16384K<o:p></o:p></p>
<p class="MsoNormal">maxMBpS 40000<o:p></o:p></p>
<p class="MsoNormal">maxReceiverThreads 32<o:p></o:p></p>
<p class="MsoNormal">nsdMaxWorkerThreads 512<o:p></o:p></p>
<p class="MsoNormal">nsdMinWorkerThreads 8<o:p></o:p></p>
<p class="MsoNormal">nsdMultiQueue 256<o:p></o:p></p>
<p class="MsoNormal">nsdSmallThreadRatio 0<o:p></o:p></p>
<p class="MsoNormal">nsdThreadsPerQueue 3<o:p></o:p></p>
<p class="MsoNormal">prefetchAggressiveness 2<o:p></o:p></p>
<p class="MsoNormal">adminMode central<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">/dev/fs0<o:p></o:p></p>
<p class="MsoNormal">@Uwe<o:p></o:p></p>
<p class="MsoNormal">Using iohist we found out that gpfs is overloading one dm-device (it took about 500ms to finish IOs). We replaced the „problematic“ dm-device (as we have enough drives to play with) for new one but the overloading issue just jumped to another
dm-device.<br>
We believe that this behaviour is caused by the gpfs but we are unable to locate the root cause of it.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Verdana",sans-serif;color:#595959;mso-fareast-language:CS">Best,<br>
Michal<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>