<html><body><p>Hi Paul,<br><br>Yes, GPFS certainly needs to behave better in this situation. We are currently working on proper support for running on newer hardware that supports Superuser Mode Access Prevention (SMAP) instructions. I believe those are new to Broadwell CPUs, but there's some confusing info out there, I'm not positive what the deal is with Haswell. For the time being, booting with the "nosmap" kernel parameter is the workaround, but you're absolutely correct, the code needs to fail more gracefully when SMAP is enabled. We'll fix that.<br><br>The current FAQ structure is, without question, suboptimal. We're looking for a better format to present this information, along the lines of more modern approaches like a structured Knowledge Base. The problem is recognized, on our end, but we've been having hard time making forward progress on this.<br><br>yuri<br><br><img width="16" height="16" src="cid:1__=07BBF558DFF719D38f9e8a93df938690918c07B@" border="0" alt="Inactive hide details for "Sanchez, Paul" ---06/03/2016 06:38:27 AM---After some puzzling debugging on our new Broadwell server"><font color="#424282">"Sanchez, Paul" ---06/03/2016 06:38:27 AM---After some puzzling debugging on our new Broadwell servers, all of which slowly became brick-like up</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">"Sanchez, Paul" <Paul.Sanchez@deshaw.com></font><br><font size="2" color="#5F5F5F">To: </font><font size="2">"gpfsug main discussion list (gpfsug-discuss@spectrumscale.org)" <gpfsug-discuss@spectrumscale.org>, </font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">06/03/2016 06:38 AM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">[gpfsug-discuss] nosmap parameter for RHEL7 x86_64 on Haswell/Broadwell?</font><br><font size="2" color="#5F5F5F">Sent by: </font><font size="2">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br><br><br><font face="Calibri">After some puzzling debugging on our new Broadwell servers, all of which slowly became brick-like upon after getting stuck starting GPFS, we discovered that this was already a known issue in the FAQ. Adding “nosmap” to the kernel command line in grub prevents SMAP from seeing the kernel-userspace memory interactions of GPFS as a reason to slowly grind all cores to a standstill, apparently spinning on stuck locks(?). (Big thanks go to RedHat for turning us on to the answer when we opened a case.)</font><br><font face="Calibri"> </font><br><font face="Calibri">From </font><a href="https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html"><u><font color="#0000FF" face="Calibri">https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html</font></u></a><font face="Calibri">, section 3.2:</font><br><font face="Calibri"> </font><br><i><font face="Calibri">Note: In order for IBM Spectrum Scale on RHEL 7 to run on the Haswell processor </font></i><li type="disc"><i><font face="Calibri">Disable the Supervisor Mode Access Prevention (smap) kernel parameter</font></i><li type="disc"><i><font face="Calibri">Reboot the RHEL 7 node before using GPFS</font></i><br><font face="Calibri"> </font><br><font face="Calibri"> </font><br><font face="Calibri">Some observations worth noting:</font><br><font face="Calibri"> </font><br>1. <font face="Calibri">We’ve been running for a year with Haswell processors and have hundreds of Haswell RHEL7 nodes which do not exhibit this problem. So maybe this only really affects Broadwell CPUs?</font><br>2. <font face="Calibri">It would be very nice for SpectrumScale to take a peek at /proc/cpuinfo and /proc/cmdline before starting up, and refuse to break the host when it has affected processors and kernel without “nosmap”. Instead, an error message describing the fix would have made my day.</font><br>3. <font face="Calibri">I’m going to have to start using a script to diff the FAQ for these gotchas, unless anyone knows of a better way to subscribe just to updates to this doc. </font><br><font face="Calibri"> </font><br><font face="Calibri">Thanks,</font><br><font face="Calibri">Paul Sanchez</font><br><font face="Calibri"> </font><tt>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></tt><tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></tt><tt><br></tt><br><BR>
</body></html>