<html><body><p><font size="2">You dont state whether your running GPFS or ESS and which level.</font><br><font size="2">One thing you can check, is whether the SES and enclosure drivers are being loaded.</font><br><font size="2">The lsmod command will show if they are.</font><br><font size="2">These drivers were found to cause SCSI IO hangs in Linux RH7.3 and 7.4.</font><br><font size="2">If they are being loaded, you can blacklist and unload them with no impact to ESS/GNR</font><br><font size="2">By default these drivers are blacklisted in ESS.</font><br><br><font size="2"> Stephen Tee</font><br><font size="2"> ESS Storage Development</font><br><font size="2"> IBM Systems and Technology</font><br><font size="2"> Austin, TX</font><br><font size="2"> 512-963-7177</font><br><br><br><img width="16" height="16" src="cid:1__=8FBB0853DF9DE3BE8f9e8a93df938690918c8FB@" border="0" alt="Inactive hide details for Steve Crusan ---07/03/2018 05:08:28 PM---Kevin, While this is happening, are you able to grab lat"><font size="2" color="#424282">Steve Crusan ---07/03/2018 05:08:28 PM---Kevin, While this is happening, are you able to grab latency stats per LUN (hardware vendor agno</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">Steve Crusan <scrusan@ddn.com></font><br><font size="2" color="#5F5F5F">To: </font><font size="2">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">07/03/2018 05:08 PM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">Re: [gpfsug-discuss] High I/O wait times</font><br><font size="2" color="#5F5F5F">Sent by: </font><font size="2">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br><br><br><tt><font size="2">Kevin,<br><br> While this is happening, are you able to grab latency stats per LUN (hardware vendor agnostic) to see if there are any outliers? Also, when looking at the mmdiag output, are both reads and writes affected? Depending on the storage hardware, your writes might be hitting cache, so maybe this problem is being exasperated by many small reads (that are too random to be coalesced, take advantage of drive NCQ, etc). <br><br> The other response about the nsd threads is also a good start, but if the I/O waits shift between different NSD servers and across hardware vendors, my assumption would be that you are hitting a bottleneck somewhere, but what you are seeing is symptoms of I/O backlog, which can manifest at any number of places. This could be something as low level as a few slow drives.<br><br> Have you just started noticing this behavior? Any new applications on your system? Going by your institution, you're probably supposing a wide variety of codes, so if these problems just started happening, its possible that someone changed their code, or decided to run new scientific packages.<br><br>-Steve<br>________________________________________<br>From: gpfsug-discuss-bounces@spectrumscale.org [gpfsug-discuss-bounces@spectrumscale.org] on behalf of Buterbaugh, Kevin L [Kevin.Buterbaugh@Vanderbilt.Edu]<br>Sent: Tuesday, July 03, 2018 11:43 AM<br>To: gpfsug main discussion list<br>Subject: [gpfsug-discuss] High I/O wait times<br><br>Hi all,<br>not <br>We are experiencing some high I/O wait times (5 - 20 seconds!) on some of our NSDs as reported by “mmdiag —iohist" and are struggling to understand why. One of the confusing things is that, while certain NSDs tend to show the problem more than others, the problem is consistent … i.e. the problem tends to move around from NSD to NSD (and storage array to storage array) whenever we check … which is sometimes just a few minutes apart.<br><br>In the past when I have seen “mmdiag —iohist” report high wait times like this it has *always* been hardware related. In our environment, the most common cause has been a battery backup unit on a storage array controller going bad and the storage array switching to write straight to disk. But that’s *not* happening this time.<br><br>Is there anything within GPFS / outside of a hardware issue that I should be looking for?? Thanks!<br><br>—<br>Kevin Buterbaugh - Senior System Administrator<br>Vanderbilt University - Advanced Computing Center for Research and Education<br>Kevin.Buterbaugh@vanderbilt.edu<</font></tt><tt><font size="2"><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu">mailto:Kevin.Buterbaugh@vanderbilt.edu</a></font></tt><tt><font size="2">> - (615)875-9633<br><br><br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><tt><font size="2"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><tt><font size="2"><br><br></font></tt><br><br><BR>
</body></html>