<div dir="ltr"><div dir="ltr"><br><div>One thing to check: Storwize/SVC code will *always* guess wrong on prefetching for GPFS. You can see this with having a lot higher read data throughput on mdisk vs. on on vdisks in the webui. To fix it, disable cache_prefetch with "chsystem -cache_prefetch off".</div><div><br></div><div>This being a global setting, you probably only should set it if the system is only used for GPFS. </div><div><br></div><div><br></div><div>   -jf</div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 28, 2021 at 5:58 PM Saula, Oluwasijibomi <<a href="mailto:oluwasijibomi.saula@ndsu.edu">oluwasijibomi.saula@ndsu.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">




<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi Folks,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
So, we are experiencing some very long IO waiters in our GPFS cluster:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-weight:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">#<span> 
</span>mmdiag --waiters<span> </span></span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-weight:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139);min-height:20px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-weight:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">=== mmdiag: waiters ===</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-weight:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">Waiting 17.3823 sec since 10:41:01, monitored, thread 21761 NSDThread: for I/O completion</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-weight:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">Waiting 16.6140 sec since 10:41:02, monitored, thread 21730 NSDThread: for I/O completion</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-weight:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">Waiting 15.3004 sec since 10:41:03, monitored, thread 21763 NSDThread: for I/O completion</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-weight:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;background-color:rgb(254,244,139)">
<span style="font-variant-ligatures:no-common-ligatures">Waiting 15.2013 sec since 10:41:03, monitored, thread 22175 <br>
</span></p>
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
However, GPFS support is pointing to our <span style="background-color:rgb(255,255,255);display:inline">IBM Storwize V5030</span> disk system as the source of latency. Unfortunately, we don't have paid support for the system so we are polling for
 anyone who might be able to assist.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Does anyone by chance have any experience with IBM Storwize V5030 or possess a<span style="background-color:rgb(255,255,255);display:inline"> problem determination guide for the V5030?</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
We've briefly reviewed the V5030 management portal, but we still haven't identified a cause for the increased latencies (i.e. read ~129ms, write ~198ms).
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Granted, we have some heavy client workloads, yet we seem to experience this drastic drop in performance every couple of months, probably exacerbated by heavy IO demands.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Any assistance would be much appreciated.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div id="gmail-m_4007529768240072585Signature">
<div>
<div></div>
<div id="gmail-m_4007529768240072585divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif">
<p style="margin-top:0px;margin-bottom:0px"></p>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
<span style="font-family:Arial,sans-serif;font-size:13.3333px">Thanks,</span><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif"><br>
</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif">
<p style="color:rgb(32,31,30);text-align:start;background-color:rgb(255,255,255);font-size:11pt;font-family:Calibri,sans-serif;margin:0px">
<b><span style="margin:0px;font-size:10pt;font-family:Arial,sans-serif;color:black">Oluwasijibomi (Siji) Saula</span></b></p>
<p style="color:rgb(32,31,30);text-align:start;background-color:rgb(255,255,255);font-size:11pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:10pt;font-family:Arial,sans-serif;color:black">HPC Systems Administrator  /  Information Technology</span><span style="margin:0px;font-family:"Times New Roman",serif;color:black"></span></p>
<p style="color:rgb(32,31,30);text-align:start;background-color:rgb(255,255,255);font-size:11pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:10pt;font-family:Arial,sans-serif;color:black"> </span><span style="margin:0px;font-family:"Times New Roman",serif;color:black"></span></p>
<p style="color:rgb(32,31,30);text-align:start;background-color:rgb(255,255,255);font-size:11pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:10pt;font-family:Arial,sans-serif;color:black"><span style="font-family:Arial,sans-serif;background-color:rgb(255,255,255);display:inline">Research 2 Building</span> 220B / Fargo ND 58108-6050</span></p>
<p style="color:rgb(32,31,30);text-align:start;background-color:rgb(255,255,255);font-size:11pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:10pt;font-family:Arial,sans-serif;color:black">p: 701.231.7749 /<span> </span><a href="http://www.ndsu.edu/" rel="noopener noreferrer" style="margin:0px" target="_blank"><span style="margin:0px;color:purple">www.ndsu.edu</span></a></span><span style="margin:0px;font-family:"Times New Roman",serif;color:black"></span></p>
<p style="color:rgb(32,31,30);text-align:start;background-color:rgb(255,255,255);font-size:11pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;font-size:10pt;font-family:Arial,sans-serif;color:black"> </span><span style="margin:0px;font-family:"Times New Roman",serif;color:black"></span></p>
<p style="color:rgb(32,31,30);text-align:start;background-color:rgb(255,255,255);font-size:11pt;font-family:Calibri,sans-serif;margin:0px">
<span style="margin:0px;color:rgb(31,73,125)"><img style="margin: 0px; width: 140.99pt; height: 26.24pt;"></span></p>
</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
<br>
</div>
<br>
<p style="margin-top:0px;margin-bottom:0px"></p>
</div>
</div>
</div>
</div>
</div>

_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote></div>