<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hi, <br>
</p>
<p>the test bench is gpfsperf running on up to 12 clients with
1...64 threads doing sequential reads and writes , file size per
gpfsperf process is 12TB (with 6TB I saw caching effects in
particular for large thread numbers ...) <br>
</p>
<p>As I wrote initially: GPFS is issuing nothing but 8MiB IOs to the
data disks, as expected in that case. <br>
</p>
<p>Interesting thing though: <br>
</p>
<p>I have rebooted the suspicious node. Now, it does not issue
smaller IOs than the others, but -- unbelievable -- larger ones
(up to about 4.7MiB). This is still harmful as also that size is
incompatible with full stripe writes on the storage ( 8+2 disk
groups, i.e. logically RAID6)<br>
</p>
<p>Currently, I draw this information from the storage boxes; I have
not yet checked iostat data for that benchmark test after the
reboot (before, when IO sizes were smaller, we saw that both in
iostat and in the perf data retrieved from the storage
controllers).</p>
<p><br>
</p>
<p>And: we have a separate data pool , hence dataOnly NSDs, I am
just talking about these ... <br>
</p>
<p><br>
</p>
<p>As for "Are you sure that Linux OS is configured the same on all
4 NSD servers?." - of course there are not two boxes identical in
the world. I have actually not installed those machines, and, yes,
i also considered reinstalling them (or at least the disturbing
one).</p>
<p>However, I do not have reason to assume or expect a difference,
the supplier has just implemented these systems recently from
scratch. <br>
</p>
<p><br>
</p>
<p>In the current situation (i.e. with IOs bit larger than 4MiB)
setting max_sectors_kB to 4096 might do the trick, but as I do not
know the cause for that behaviour it might well start to issue IOs
smaller than 4MiB again at some point, so that is not a nice
solution.<br>
</p>
<p><br>
</p>
<p>Thanks</p>
<p>Uwe<br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 23.02.22 22:20, Andrew Beattie
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:OF72C54091.85E779B6-ON002587F2.00753444-1645651211450@ibm.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
Alex,
<div><br>
</div>
<div>Metadata will be 4Kib </div>
<div><br>
</div>
<div>Depending on the filesystem version you will also have
subblocks to consider V4 filesystems have 1/32 subblocks, V5
filesystems have 1/1024 subblocks (assuming metadata and data
block size is the same)</div>
<div><br>
My first question would be is “ Are you sure that Linux OS is
configured the same on all 4 NSD servers?.</div>
<div><br>
</div>
<div>My second question would be do you know what your average
file size is if most of your files are smaller than your
filesystem block size, then you are always going to be
performing writes using groups of subblocks rather than a full
block writes.</div>
<div><br>
</div>
<div>Regards, </div>
<div><br>
</div>
<div>Andrew<br>
<div dir="ltr"><br>
</div>
<div dir="ltr"><br>
<blockquote type="cite">On 24 Feb 2022, at 04:39, Alex
Chekholko <a class="moz-txt-link-rfc2396E" href="mailto:alex@calicolabs.com"><alex@calicolabs.com></a> wrote:<br>
<br>
</blockquote>
</div>
<blockquote type="cite">
<div dir="ltr">
<!-- BaNnErBlUrFlE-HeAdEr-start -->
<meta name="viewport" content="width=device-width,
initial-scale=1.0">
<style>.pfptBannerTableMSO { padding: 0px 12px 5px 12px; width: 100%;border-radius:4px;border-top:4px solid #90a4ae;background-color:#d0d8dc; }.pfptTitleMSO { color:#000000 !important;font-family: 'Arial', sans-serif !important;font-weight:bold !important;font-size:14px !important; }.pfptSubtitleMSO { font-size:12px !important; font-family: 'Arial', sans-serif !important; }.pfptButtonMSO { mso-padding-alt: 7.5px; padding: 7.5px; text-decoration: none; font-family: 'Arial', sans-serif !important; font-size: 14px; line-height: 40px; border-radius:2px; }.pfptPrimaryButtonMSO {
border: 1.5px solid #666666; color: #000000;
}.pfptBanner {
margin: 15px 14px 30px 14px;
padding: 8px 16px 8px 16px;
border-radius: 4px;
min-width: 200px;
background-color: #d0d8dc;
border-top: 4px solid #90a4ae;
}.pfptBannerTitle {
color: #000000;
font-family: 'Arial', sans-serif;
font-size: 14px;
font-weight: bold;
line-height: 18px;
display: block;
}.pfptBannerSubtitle {
color: #000000;
font-weight: normal;
font-family: 'Arial', sans-serif;
font-size: 12px;
line-height: 18px;
margin-top: 2px;
display: block;
}.pfptButton {
display: inline-block;
font-family: 'Arial', sans-serif;
font-size: 14px;
font-weight: normal;
border-radius: 2px;
padding: 7.5px 16px;
margin: 3px 0 3px 16px;
white-space: nowrap;
width: fit-content;
}.pfptPrimaryButton {
border: 1px solid #666666;
}.pfptPrimaryButton:hover, .pfptPrimaryButton:focus {
background-color: #b4c1c7;
}.pfptPrimaryButton:active {
background-color: #90a4ae;
}.pfptMessageContainer {
display: inline-block;
margin: 0px 0px 1px 0px;
max-width: 600px;
}.pfptButtonGroup {
float: right;
margin: 0px 0px 0px 16px;
text-align: right;
width: fit-content;
}.pfptPreheader { display:none !important; visibility:hidden; mso-hide:all; font-size:1px; line-height:1px; max-height:0px; max-width:0px; opacity:0; overflow:hidden; }</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
<!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<span class="pfptPreheader" style="display:none
!important;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;max-width:0px;opacity:0;overflow:hidden;">
Hi, Metadata I/Os will always be smaller than the usual
data block size, right? Which version of GPFS? Regards,
Alex On Wed, Feb 23, 2022 at 10:26 AM Uwe Falke
<a class="moz-txt-link-rfc2396E" href="mailto:uwe.falke@kit.edu"><uwe.falke@kit.edu></a> wrote: Dear all, sorry for
asking a question which seems
</span><!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<span style="display:none
!important;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;max-width:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</span>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 3px 0px 16px 0px;"><tr><td>
<table class="pfptBannerTableMSO" border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%;border-radius:4px;border-top:4px solid #90a4ae;background-color:#d0d8dc;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td><span class="pfptTitleMSO" style="color:#000000 !important;font-family: 'Arial', sans-serif;font-weight:bold !important;font-size:14px !important;">
This Message Is From an External Sender
</span></td></tr>
<tr><td><span class="pfptSubtitleMSO" style="color:#000000 !important;font-weight:normal !important;font-family: 'Arial', sans-serif; font-size:12px !important;">
This message came from outside your organization.
</span></td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<!--[if !((ie)|(mso))]-->
<div class="pfptBanner" style="margin:16px 0px 16px 0px;
padding:8px 16px 8px 16px; border-radius: 4px; min-width:
200px;background-color: #d0d8dc; border-top: 4px solid
#90a4ae;">
<div class="pfptMessageContainer" style="display:
inline-block; margin: 0px 0px 1px 0px; max-width:
600px;">
<div class="pfptBannerTitle" style="color:#000000
!important;font-family: 'Arial', sans-serif
!important;font-weight:bold !important;font-size:14px
!important;line-height:18px;display:block;"> This
Message Is From an External Sender </div>
<div class="pfptBannerSubtitle" style="color:#000000
!important;font-weight:normal !important;font-family:
'Arial', sans-serif !important;font-size:12px
!important;line-height:18px;margin-top:2px;display:block">This
message came from outside your organization. </div>
</div>
</div>
<!--[endif]-->
<span style="display:none
!important;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;max-width:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd<br>
</span><!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<div dir="ltr">Hi,
<div><br>
</div>
<div>Metadata I/Os will always be smaller than the usual
data block size, right?</div>
<div>Which version of GPFS?</div>
<div><br>
</div>
<div>Regards,</div>
<div>Alex</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Feb 23, 2022 at
10:26 AM Uwe Falke <<a
href="mailto:uwe.falke@kit.edu" moz-do-not-send="true"
class="moz-txt-link-freetext">uwe.falke@kit.edu</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">Dear all,<br>
<br>
sorry for asking a question which seems not directly
GPFS related:<br>
<br>
In a setup with 4 NSD servers (old-style, with storage
controllers in <br>
the back end), 12 clients and 10 Seagate storage
systems, I do see in <br>
benchmark tests that just one of the NSD servers does
send smaller IO <br>
requests to the storage than the other 3 (that is, both
reads and <br>
writes are smaller).<br>
<br>
The NSD servers form 2 pairs, each pair is connected to
5 seagate boxes <br>
( one server to the controllers A, the other one to
controllers B of the <br>
Seagates, resp.).<br>
<br>
All 4 NSD servers are set up similarly:<br>
<br>
kernel: 3.10.0-1160.el7.x86_64 #1 SMP<br>
<br>
HBA: Broadcom / LSI Fusion-MPT 12GSAS/PCIe Secure
SAS38xx<br>
<br>
driver : mpt3sas 31.100.01.00<br>
<br>
max_sectors_kb=8192 (max_hw_sectors_kb=16383 , not
16384, as limited by <br>
mpt3sas) for all sd devices and all multipath (dm)
devices built on top.<br>
<br>
scheduler: deadline<br>
<br>
multipath (actually we do have 3 paths to each volume,
so there is some <br>
asymmetry, but that should not affect the IOs, shouldn't
it?, and if it <br>
did we would see the same effect in both pairs of NSD
servers, but we do <br>
not).<br>
<br>
All 4 storage systems are also configured the same way
(2 disk groups / <br>
pools / declustered arrays, one managed by ctrl A, one
by ctrl B, and <br>
8 volumes out of each; makes altogether 2 x 8 x 10 = 160
NSDs).<br>
<br>
<br>
GPFS BS is 8MiB , according to iohistory (mmdiag) we do
see clean IO <br>
requests of 16384 disk blocks (i.e. 8192kiB) from GPFS.<br>
<br>
The first question I have - but that is not my main one:
I do see, both <br>
in iostat and on the storage systems, that the default
IO requests are <br>
about 4MiB, not 8MiB as I'd expect from above settings
(max_sectors_kb <br>
is really in terms of kiB, not sectors, cf. <br>
<a
href="https://www.kernel.org/doc/Documentation/block/queue-sysfs.txt"
rel="noreferrer" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">https://www.kernel.org/doc/Documentation/block/queue-sysfs.txt</a>).<br>
<br>
But what puzzles me even more: one of the server
compiles IOs even <br>
smaller, varying between 3.2MiB and 3.6MiB mostly - both
for reads and <br>
writes ... I just cannot see why.<br>
<br>
I have to suspect that this will (in writing to the
storage) cause <br>
incomplete stripe writes on our erasure-coded volumes
(8+2p)(as long as <br>
the controller is not able to re-coalesce the data
properly; and it <br>
seems it cannot do it completely at least)<br>
<br>
<br>
If someone of you has seen that already and/or knows a
potential <br>
explanation I'd be glad to learn about.<br>
<br>
<br>
And if some of you wonder: yes, I (was) moved away from
IBM and am now <br>
at KIT.<br>
<br>
Many thanks in advance<br>
<br>
Uwe<br>
<br>
<br>
-- <br>
Karlsruhe Institute of Technology (KIT)<br>
Steinbuch Centre for Computing (SCC)<br>
Scientific Data Management (SDM)<br>
<br>
Uwe Falke<br>
<br>
Hermann-von-Helmholtz-Platz 1, Building 442, Room 187<br>
D-76344 Eggenstein-Leopoldshafen<br>
<br>
Tel: +49 721 608 28024<br>
Email: <a href="mailto:uwe.falke@kit.edu"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">uwe.falke@kit.edu</a><br>
<a href="http://www.scc.kit.edu" rel="noreferrer"
target="_blank" moz-do-not-send="true">www.scc.kit.edu</a><br>
<br>
Registered office:<br>
Kaiserstraße 12, 76131 Karlsruhe, Germany<br>
<br>
KIT – The Research University in the Helmholtz
Association<br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org"
rel="noreferrer" target="_blank"
moz-do-not-send="true">spectrumscale.org</a><br>
<a
href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"
rel="noreferrer" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
<br>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
<a class="moz-txt-link-freetext" href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a>
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
Karlsruhe Institute of Technology (KIT)
Steinbuch Centre for Computing (SCC)
Scientific Data Management (SDM)
Uwe Falke
Hermann-von-Helmholtz-Platz 1, Building 442, Room 187
D-76344 Eggenstein-Leopoldshafen
Tel: +49 721 608 28024
Email: <a class="moz-txt-link-abbreviated" href="mailto:uwe.falke@kit.edu">uwe.falke@kit.edu</a>
<a class="moz-txt-link-abbreviated" href="http://www.scc.kit.edu">www.scc.kit.edu</a>
Registered office:
Kaiserstraße 12, 76131 Karlsruhe, Germany
KIT – The Research University in the Helmholtz Association
</pre>
</body>
</html>