<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi John,
<div class=""><br class="">
</div>
<div class="">We had a similar incident happen just a week or so ago here, although in our case it was that certain files within a directory showed up with the question marks, while others didn’t.</div>
<div class=""><br class="">
</div>
<div class="">The problem was simply that the node had been run out of RAM and the GPFS daemon couldn’t allocate memory. Killing the offending process(es) and restarting GPFS fixed the issue.</div>
<div class=""><br class="">
</div>
<div class="">We saw hundreds of messages like:</div>
<div class=""><br class="">
</div>
<div class="">2018-02-07_16:35:13.267-0600: [E] Failed to allocate 92274688 bytes in memory pool, err -1</div>
<div class=""><br class="">
</div>
<div class="">In the GPFS log when this was happening. HTHAL…</div>
<div class=""><br class="">
</div>
<div class="">Kevin</div>
<div class=""><br class="">
<div class="">
<div class="">—</div>
<div class="">Kevin Buterbaugh - Senior System Administrator</div>
<div class="">Vanderbilt University - Advanced Computing Center for Research and Education</div>
<div class=""><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class="">Kevin.Buterbaugh@vanderbilt.edu</a> - (615)875-9633</div>
<div class=""><br class="">
</div>
</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Feb 14, 2018, at 12:38 PM, Simon Thompson (IT Research Support) <<a href="mailto:S.J.Thompson@bham.ac.uk" class="">S.J.Thompson@bham.ac.uk</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">Is it an AFM cache? We see this sort of behaviour occasionally where the cache has an "old" view of the directory. Doing an ls, it evidently goes back to home but by then you already have weird stuff. The next ls is usually fine.<br class="">
<br class="">
Simon<br class="">
________________________________________<br class="">
From: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class="">gpfsug-discuss-bounces@spectrumscale.org</a> [<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class="">gpfsug-discuss-bounces@spectrumscale.org</a>] on behalf of
<a href="mailto:john.hearns@asml.com" class="">john.hearns@asml.com</a> [<a href="mailto:john.hearns@asml.com" class="">john.hearns@asml.com</a>]<br class="">
Sent: 14 February 2018 09:00<br class="">
To: gpfsug main discussion list<br class="">
Subject: [gpfsug-discuss] Odd d????????? permissions<br class="">
<br class="">
I am sure this is a known behavior and I am going to feel very foolish in a few minutes…<br class="">
<br class="">
We often see this behavior on a GPFS filesystem.<br class="">
I log into a client.<br class="">
<br class="">
[jhearns@pn715 test]$ ls -la ../<br class="">
ls: cannot access ../..: Permission denied<br class="">
total 160<br class="">
drwx------ 4 jhearns root 4096 Feb 14 09:46 .<br class="">
d????????? ? ? ? ? ? ..<br class="">
drwxr-xr-x 2 jhearns users 4096 Feb 9 11:13 gpfsperf<br class="">
-rw-r--r-- 1 jhearns users 27336 Feb 9 22:24 iozone.out<br class="">
-rw-r--r-- 1 jhearns users 6083 Feb 9 10:55 IozoneResults.py<br class="">
-rw-r--r-- 1 jhearns users 22959 Feb 9 11:17 iozone.txt<br class="">
-rw-r--r-- 1 jhearns users 2977 Feb 9 10:55 iozone.txtvi<br class="">
-rwxr-xr-x 1 jhearns users 102 Feb 9 10:55 run-iozone.sh<br class="">
drwxr-xr-x 2 jhearns users 4096 Feb 14 09:46 test<br class="">
-r-x------ 1 jhearns users 51504 Feb 9 11:02 tsqosperf<br class="">
<br class="">
This behavior changes after a certain number of minutes, and the .. directory looks normal.<br class="">
<br class="">
For information this filesystem has nfsv4 file locking semantics and ACL semantics set to all<br class="">
<br class="">
-- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated
otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your
own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be
liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt.<br class="">
_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class="">
<a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C9df4b4d88544447ac29608d573da2d51%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636542303262503651&sdata=v6pnBIEvu6lyP3mGkkRX7hSj58H8vvkUl6R%2FCsq6gmc%3D&reserved=0" class="">https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C9df4b4d88544447ac29608d573da2d51%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636542303262503651&sdata=v6pnBIEvu6lyP3mGkkRX7hSj58H8vvkUl6R%2FCsq6gmc%3D&reserved=0</a><br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>