<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Vladis/Kums/Fred/Kevin/Stephen,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks so much for your insights, thoughts, and pointers! - Certainly increased my knowledge and understanding of potential culprits to watch for...</div>
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
So we finally discovered the root issue to this problem: An unattended TSM restore exercise profusely writing to a single file, over and over again into the GBs!!..I'm opening up a ticket with TSM support to learn how to mitigate this in the future.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
But with the RAID 6 writing costs Vladis explained, it now makes sense why the write IO was badly affected...</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Excerpt from output file:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="">
<p style="margin: 0px;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">--- User Action is Required ---</span></p>
<p style="margin: 0px;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">File '/gpfs1/X/Y/Z/fileABC' is write protected</span></p>
<p style="margin: 0px; min-height: 24px;"><br>
</p>
<p style="margin: 0px;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Select an appropriate action</span></p>
<p style="margin: 0px;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">1. Force an overwrite for this object</span></p>
<p style="margin: 0px;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">2. Force an overwrite on all objects that are write protected</span></p>
<p style="margin: 0px;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">3. Skip this object</span></p>
<p style="margin: 0px;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">4. Skip all objects that are write protected</span></p>
<p style="margin: 0px;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">A. Abort this operation</span></p>
<p style="margin: 0px;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Action [1,2,3,4,A] : The only valid responses are characters from this set: [1, 2, 3, 4, A]</span></p>
<span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255); display: inline !important">Action [1,2,3,4,A] : The only valid responses are characters from this set: [1, 2, 3, 4, A]</span></div>
<div style=""><span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255); display: inline !important">Action [1,2,3,4,A] : The only valid responses are characters from this set: [1, 2, 3, 4, A]</span></div>
<div style=""><span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255); display: inline !important">Action [1,2,3,4,A] : The only valid responses are characters from this set: [1, 2, 3, 4, A]</span></div>
<div style=""><span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255); display: inline !important">...</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div></div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Arial,Helvetica,sans-serif">
<p style="margin-top:0px; margin-bottom:0px"></p>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<span style="font-family:Arial,sans-serif; font-size:13.3333px">Thanks,</span><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif"><br>
</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif">
<p style="color:rgb(32,31,30); text-align:start; background-color:rgb(255,255,255); font-size:11pt; font-family:Calibri,sans-serif; margin:0px">
<b><span style="margin:0px; font-size:10pt; font-family:Arial,sans-serif; color:black">Oluwasijibomi (Siji) Saula</span></b></p>
<p style="color:rgb(32,31,30); text-align:start; background-color:rgb(255,255,255); font-size:11pt; font-family:Calibri,sans-serif; margin:0px">
<span style="margin:0px; font-size:10pt; font-family:Arial,sans-serif; color:black">HPC Systems Administrator / Information Technology</span><span style="margin:0px; font-family:"Times New Roman",serif; color:black"></span></p>
<p style="color:rgb(32,31,30); text-align:start; background-color:rgb(255,255,255); font-size:11pt; font-family:Calibri,sans-serif; margin:0px">
<span style="margin:0px; font-size:10pt; font-family:Arial,sans-serif; color:black"> </span><span style="margin:0px; font-family:"Times New Roman",serif; color:black"></span></p>
<p style="color:rgb(32,31,30); text-align:start; background-color:rgb(255,255,255); font-size:11pt; font-family:Calibri,sans-serif; margin:0px">
<span style="margin:0px; font-size:10pt; font-family:Arial,sans-serif; color:black"><span style="font-family:Arial,sans-serif; background-color:rgb(255,255,255); display:inline!important">Research 2 Building</span> 220B / Fargo ND 58108-6050</span></p>
<p style="color:rgb(32,31,30); text-align:start; background-color:rgb(255,255,255); font-size:11pt; font-family:Calibri,sans-serif; margin:0px">
<span style="margin:0px; font-size:10pt; font-family:Arial,sans-serif; color:black">p: 701.231.7749 /<span> </span><a href="http://www.ndsu.edu/" target="_blank" rel="noopener noreferrer" style="margin:0px"><span style="margin:0px; color:purple">www.ndsu.edu</span></a></span><span style="margin:0px; font-family:"Times New Roman",serif; color:black"></span></p>
<p style="color:rgb(32,31,30); text-align:start; background-color:rgb(255,255,255); font-size:11pt; font-family:Calibri,sans-serif; margin:0px">
<span style="margin:0px; font-size:10pt; font-family:Arial,sans-serif; color:black"> </span><span style="margin:0px; font-family:"Times New Roman",serif; color:black"></span></p>
<p style="color:rgb(32,31,30); text-align:start; background-color:rgb(255,255,255); font-size:11pt; font-family:Calibri,sans-serif; margin:0px">
<span style="margin:0px; color:rgb(31,73,125)"><img class="EmojiInsert" style="margin:0px; width:140.99pt; height:26.24pt" data-outlook-trace="F:0|T:1" src="cid:image001.gif@01D57DE0.91C300C0"></span></p>
</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<br>
</div>
<br>
<p style="margin-top:0px; margin-bottom:0px"></p>
</div>
</div>
</div>
<div id="appendonsend"></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> gpfsug-discuss-bounces@spectrumscale.org <gpfsug-discuss-bounces@spectrumscale.org> on behalf of gpfsug-discuss-request@spectrumscale.org
<gpfsug-discuss-request@spectrumscale.org><br>
<b>Sent:</b> Friday, June 5, 2020 6:00 AM<br>
<b>To:</b> gpfsug-discuss@spectrumscale.org <gpfsug-discuss@spectrumscale.org><br>
<b>Subject:</b> gpfsug-discuss Digest, Vol 101, Issue 12</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt">
<div class="PlainText">Send gpfsug-discuss mailing list submissions to<br>
gpfsug-discuss@spectrumscale.org<br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
or, via email, send a message with subject or body 'help' to<br>
gpfsug-discuss-request@spectrumscale.org<br>
<br>
You can reach the person managing the list at<br>
gpfsug-discuss-owner@spectrumscale.org<br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of gpfsug-discuss digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: Client Latency and High NSD Server Load Average<br>
(Valdis Kl=?utf-8?Q?=c4=93?=tnieks)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Thu, 04 Jun 2020 21:17:08 -0400<br>
From: "Valdis Kl=?utf-8?Q?=c4=93?=tnieks" <valdis.kletnieks@vt.edu><br>
To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
Subject: Re: [gpfsug-discuss] Client Latency and High NSD Server Load<br>
Average<br>
Message-ID: <309214.1591319828@turing-police><br>
Content-Type: text/plain; charset="us-ascii"<br>
<br>
On Thu, 04 Jun 2020 15:33:18 -0000, "Saula, Oluwasijibomi" said:<br>
<br>
> However, I still can't understand why write IO operations are 5x more latent<br>
> than ready operations to the same class of disks.<br>
<br>
Two things that may be biting you:<br>
<br>
First, on a RAID 5 or 6 LUN, most of the time you only need to do 2 physical<br>
reads (data and parity block). To do a write, you have to read the old parity<br>
block, compute the new value, and write the data block and new parity block.<br>
This is often called the "RAID write penalty".<br>
<br>
Second, if a read size is smaller than the physical block size, the storage array can read<br>
a block, and return only the fragment needed. But on a write, it has to read<br>
the whole block, splice in the new data, and write back the block - a RMW (read<br>
modify write) cycle.<br>
-------------- next part --------------<br>
A non-text attachment was scrubbed...<br>
Name: not available<br>
Type: application/pgp-signature<br>
Size: 832 bytes<br>
Desc: not available<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200604/da016913/attachment-0001.sig">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200604/da016913/attachment-0001.sig</a>><br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
<br>
<br>
End of gpfsug-discuss Digest, Vol 101, Issue 12<br>
***********************************************<br>
</div>
</span></font></div>
</body>
</html>