<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Simon, Kevin<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thankyou for your responses.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Simon, indeed we do see this behavior on AFM filesets which have an ‘old’ view – and we can watch the AFM fileset change as the information is updated.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In this case, this filesystem is not involved with AFM.<o:p></o:p></p>
<p class="MsoNormal">I Changed the locking semantics from NFSv4 to Posix and the report is that this has solved the problem.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> gpfsug-discuss-bounces@spectrumscale.org [mailto:gpfsug-discuss-bounces@spectrumscale.org]
<b>On Behalf Of </b>Buterbaugh, Kevin L<br>
<b>Sent:</b> Wednesday, February 14, 2018 9:54 PM<br>
<b>To:</b> gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
<b>Subject:</b> Re: [gpfsug-discuss] Odd d????????? permissions<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi John, <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We had a similar incident happen just a week or so ago here, although in our case it was that certain files within a directory showed up with the question marks, while others didn’t.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The problem was simply that the node had been run out of RAM and the GPFS daemon couldn’t allocate memory. Killing the offending process(es) and restarting GPFS fixed the issue.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We saw hundreds of messages like:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">2018-02-07_16:35:13.267-0600: [E] Failed to allocate 92274688 bytes in memory pool, err -1<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">In the GPFS log when this was happening. HTHAL…<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Kevin<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">—<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Kevin Buterbaugh - Senior System Administrator<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Vanderbilt University - Advanced Computing Center for Research and Education<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu">Kevin.Buterbaugh@vanderbilt.edu</a> - (615)875-9633<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Feb 14, 2018, at 12:38 PM, Simon Thompson (IT Research Support) <<a href="mailto:S.J.Thompson@bham.ac.uk">S.J.Thompson@bham.ac.uk</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Is it an AFM cache? We see this sort of behaviour occasionally where the cache has an "old" view of the directory. Doing an ls, it evidently goes back to home but by then you already have weird stuff. The next ls is usually fine.<br>
<br>
Simon<br>
________________________________________<br>
From: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-bounces@spectrumscale.org</a> [<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-bounces@spectrumscale.org</a>] on behalf of
<a href="mailto:john.hearns@asml.com">john.hearns@asml.com</a> [<a href="mailto:john.hearns@asml.com">john.hearns@asml.com</a>]<br>
Sent: 14 February 2018 09:00<br>
To: gpfsug main discussion list<br>
Subject: [gpfsug-discuss] Odd d????????? permissions<br>
<br>
I am sure this is a known behavior and I am going to feel very foolish in a few minutes…<br>
<br>
We often see this behavior on a GPFS filesystem.<br>
I log into a client.<br>
<br>
[jhearns@pn715 test]$ ls -la ../<br>
ls: cannot access ../..: Permission denied<br>
total 160<br>
drwx------ 4 jhearns root 4096 Feb 14 09:46 .<br>
d????????? ? ? ? ? ? ..<br>
drwxr-xr-x 2 jhearns users 4096 Feb 9 11:13 gpfsperf<br>
-rw-r--r-- 1 jhearns users 27336 Feb 9 22:24 iozone.out<br>
-rw-r--r-- 1 jhearns users 6083 Feb 9 10:55 IozoneResults.py<br>
-rw-r--r-- 1 jhearns users 22959 Feb 9 11:17 iozone.txt<br>
-rw-r--r-- 1 jhearns users 2977 Feb 9 10:55 iozone.txtvi<br>
-rwxr-xr-x 1 jhearns users 102 Feb 9 10:55 run-iozone.sh<br>
drwxr-xr-x 2 jhearns users 4096 Feb 14 09:46 test<br>
-r-x------ 1 jhearns users 51504 Feb 9 11:02 tsqosperf<br>
<br>
This behavior changes after a certain number of minutes, and the .. directory looks normal.<br>
<br>
For information this filesystem has nfsv4 file locking semantics and ACL semantics set to all<br>
<br>
-- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated
otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your
own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be
liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt.<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=01%7C01%7Cjohn.hearns%40asml.com%7C3b04bafdd99d48fb4b8008d573ed1bf8%7Caf73baa8f5944eb2a39d93e96cad61fc%7C1&sdata=b7aBB656yJB0E9w%2FiW%2B%2FPSVxP%2BBdDd%2BYfja8NB9geJc%3D&reserved=0">
spectrumscale.org</a><br>
<a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C9df4b4d88544447ac29608d573da2d51%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636542303262503651&sdata=v6pnBIEvu6lyP3mGkkRX7hSj58H8vvkUl6R%2FCsq6gmc%3D&reserved=0">https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C9df4b4d88544447ac29608d573da2d51%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636542303262503651&sdata=v6pnBIEvu6lyP3mGkkRX7hSj58H8vvkUl6R%2FCsq6gmc%3D&reserved=0</a><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
-- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated
otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your
own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be
liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt.
</body>
</html>