<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Lucida Grande";
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Oformaterad text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.E-postmall19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.OformateradtextChar
{mso-style-name:"Oformaterad text Char";
mso-style-priority:99;
mso-style-link:"Oformaterad text";
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-GB link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>Yes, this is repeating.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>We’ve ascertained that it has nothing to do at all with file operations on the GPFS side.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>Randomly throughout the filesystem mounted via NFS, ls or file access will give<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>”<o:p></o:p></span></p><p class=MsoPlainText>> ls: reading directory /gpfs/filessystem/test/testdir: Invalid argument<span style='color:#1F497D'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>“<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>Trying again later might work on that folder, but might fail somewhere else.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>We have tried exporting the same filesystem via a standard kernel NFS instead of the CES Ganesha-NFS, and then the problem doesn’t exist.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>So it is definitely related to the Ganesha NFS server, or its interaction with the file system.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>Will see if I can get a tcpdump of the issue.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>Regards,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>Andreas<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:black'>_____________________________________________<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Lucida Grande",serif;color:black'><img width=121 height=65 style='width:1.2604in;height:.677in' id="Bild_x0020_2" src="cid:image001.png@01D48338.535C1D60" alt="cid:part1.08040705.03090509@maxiv.lu.se"><br></span><b><span style='font-size:10.0pt;font-family:"Calibri",sans-serif;color:black'> <br>Andreas Mattsson<br></span></b><span style='font-size:8.0pt;font-family:"Calibri",sans-serif;color:black'>Systems Engineer</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal><span style='color:black'><o:p> </o:p></span></p><p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-size:8.0pt;font-family:"Calibri",sans-serif;color:black'>MAX IV Laboratory<br>Lund University<br></span><span style='font-size:8.5pt;color:black'>P.O. Box 118, SE-221 00 Lund, Sweden<br>Visiting address: Fotongatan 2, 224 84 Lund<br></span><span style='font-size:8.0pt;font-family:"Calibri",sans-serif;color:black'>Mobile: +46 706 64 95 44<br></span><a href="mailto:andreas.mattsson@maxiv.lu.se"><span style='font-size:8.0pt;font-family:"Calibri",sans-serif'>andreas.mattsson@maxiv.lu.se</span></a><span style='font-size:8.0pt;font-family:"Calibri",sans-serif;color:black'><br></span><a href="http://www.maxiv.se/"><span style='font-size:8.0pt;font-family:"Calibri",sans-serif'>www.maxiv.se</span></a><span style='font-size:8.0pt;font-family:"Calibri",sans-serif;color:black'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal style='margin-left:36.0pt'><b><span lang=SV style='font-size:11.0pt;font-family:"Calibri",sans-serif'>Från:</span></b><span lang=SV style='font-size:11.0pt;font-family:"Calibri",sans-serif'> gpfsug-discuss-bounces@spectrumscale.org <gpfsug-discuss-bounces@spectrumscale.org> <b>För </b>Malahal R Naineni<br><b>Skickat:</b> den 22 november 2018 11:32<br><b>Till:</b> gpfsug-discuss@spectrumscale.org<br><b>Kopia:</b> gpfsug-discuss@spectrumscale.org<br><b>Ämne:</b> Re: [gpfsug-discuss] Filesystem access issues via CES NFS<o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt'><span lang=SV><o:p> </o:p></span></p><div><div><p class=MsoNormal style='margin-left:36.0pt'><span style='font-size:10.5pt;font-family:"Arial",sans-serif'>We have seen empty lists (ls showing nothing). If this repeats, please take tcpdump from the client and we will investigate.<o:p></o:p></span></p></div><div><p class=MsoNormal style='margin-left:36.0pt'><span style='font-size:10.5pt;font-family:"Arial",sans-serif'> <o:p></o:p></span></p></div><div><p class=MsoNormal style='margin-left:36.0pt'><span style='font-size:10.5pt;font-family:"Arial",sans-serif'>Regards, Malahal.<o:p></o:p></span></p></div><div><p class=MsoNormal style='margin-left:36.0pt'><span style='font-size:10.5pt;font-family:"Arial",sans-serif'> <o:p></o:p></span></p></div><blockquote style='border:none;border-left:solid #AAAAAA 1.5pt;padding:0cm 0cm 0cm 4.0pt;margin-left:3.75pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt'><p class=MsoNormal style='margin-left:36.0pt'><span style='font-size:10.5pt;font-family:"Arial",sans-serif'>----- Original message -----<br>From: Andreas Mattsson <andreas.mattsson@maxiv.lu.se><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: "gpfsug-discuss@spectrumscale.org" <gpfsug-discuss@spectrumscale.org><br>Cc:<br>Subject: [gpfsug-discuss] Filesystem access issues via CES NFS<br>Date: Tue, Nov 20, 2018 8:47 PM<br> <o:p></o:p></span></p><div><p style='margin-left:36.0pt'><span style='font-size:10.5pt;font-family:"Arial",sans-serif'>On one of our clusters, from time to time if users try to access files or folders via the direct full path over NFS, the NFS-client gets invalid information from the server.<o:p></o:p></span></p><p style='margin-left:36.0pt'><span style='font-size:10.5pt;font-family:"Arial",sans-serif'>For instance, if I run “ls /gpfs/filesystem/test/test2/test3” over NFS-mount, result is just full of ????????<o:p></o:p></span></p><p style='margin-left:36.0pt'><span style='font-size:10.5pt;font-family:"Arial",sans-serif'>If I recurse through the path once, for instance by ls’ing or cd’ing through the folders one at a time or running ls –R, I can then access directly via the full path afterwards.<o:p></o:p></span></p><p style='margin-left:36.0pt'><span style='font-size:10.5pt;font-family:"Arial",sans-serif'>This seem to be intermittent, and I haven’t found how to reliably recreate the issue.<o:p></o:p></span></p><p style='margin-left:36.0pt'><span style='font-size:10.5pt;font-family:"Arial",sans-serif'>Possibly, it can be connected to creating or changing files or folders via a GPFS mount, and then accessing them through NFS, but it doesn’t happen consistently.<o:p></o:p></span></p><p style='margin-left:36.0pt'><span style='font-size:10.5pt;font-family:"Arial",sans-serif'> <o:p></o:p></span></p><p style='margin-left:36.0pt'><span style='font-size:10.5pt;font-family:"Arial",sans-serif'>Is this a known behaviour or bug, and does anyone know how to fix the issue?<o:p></o:p></span></p><p style='margin-left:36.0pt'><span style='font-size:10.5pt;font-family:"Arial",sans-serif'>These NSD-servers currently run Scale 4.2.2.3, while the CES is on 5.0.1.1.<o:p></o:p></span></p><p style='margin-left:36.0pt'><span style='font-size:10.5pt;font-family:"Arial",sans-serif'>GPFS clients run Scale 5.0.1.1, and NFS clients run CentOS 7.5.<o:p></o:p></span></p><p style='margin-left:36.0pt'><span style='font-size:10.5pt;font-family:"Arial",sans-serif'> <o:p></o:p></span></p><p style='margin-left:36.0pt'><span style='font-size:10.5pt;font-family:"Arial",sans-serif'>Regards,<o:p></o:p></span></p><p style='margin-left:36.0pt'><span style='font-size:10.5pt;font-family:"Arial",sans-serif'>Andreas Mattsson<o:p></o:p></span></p><p style='margin-left:36.0pt'><span style='color:black'>_____________________________________________</span><span style='font-size:10.5pt;font-family:"Arial",sans-serif'><o:p></o:p></span></p><p style='margin-left:36.0pt'><span style='font-size:10.5pt;font-family:"Lucida Grande",serif;color:black'><img border=0 width=121 height=65 style='width:1.2604in;height:.677in' id="_x0036_b0a11de-89d6-4863-b8e8-188f1e1c5a0c" src="cid:image001.png@01D48338.535C1D60" alt="cid:part1.08040705.03090509@maxiv.lu.se"></span><span style='font-size:10.5pt;font-family:"Arial",sans-serif'><br></span><b><span style='font-size:10.0pt;font-family:"Arial",sans-serif;color:black'> <br>Andreas Mattsson</span></b><span style='font-size:10.5pt;font-family:"Arial",sans-serif'><br></span><span style='font-size:8.0pt;font-family:"Arial",sans-serif;color:black'>Systems Engineer</span><span style='font-size:10.5pt;font-family:"Arial",sans-serif'><o:p></o:p></span></p><p style='margin-left:36.0pt'><span style='color:black'> </span><span style='font-size:10.5pt;font-family:"Arial",sans-serif'><o:p></o:p></span></p><p style='mso-margin-top-alt:5.0pt;margin-right:0cm;margin-bottom:12.0pt;margin-left:36.0pt'><span style='font-size:8.0pt;font-family:"Arial",sans-serif;color:black'>MAX IV Laboratory<br>Lund University</span><span style='font-size:10.5pt;font-family:"Arial",sans-serif'><br></span><span style='font-size:8.5pt;color:black'>P.O. Box 118, SE-221 00 Lund, Sweden<br>Visiting address: Fotongatan 2, 224 84 Lund</span><span style='font-size:10.5pt;font-family:"Arial",sans-serif'><br></span><span style='font-size:8.0pt;font-family:"Arial",sans-serif;color:black'>Mobile: +46 706 64 95 44</span><span style='font-size:10.5pt;font-family:"Arial",sans-serif'><br></span><a href="mailto:andreas.mattsson@maxiv.lu.se" target="_blank"><span style='font-size:8.0pt;font-family:"Arial",sans-serif'>andreas.mattsson@maxiv.lu.se</span></a><span style='font-size:10.5pt;font-family:"Arial",sans-serif'><br></span><a href="http://www.maxiv.se/" target="_blank"><span style='font-size:8.0pt;font-family:"Arial",sans-serif'>www.maxiv.se</span></a><span style='font-size:10.5pt;font-family:"Arial",sans-serif'><o:p></o:p></span></p><p style='margin-left:36.0pt'><span style='font-size:10.5pt;font-family:"Arial",sans-serif'> <o:p></o:p></span></p></div><div id=MIMEAttachInfoDiv><p class=MsoNormal style='margin-left:36.0pt'><span style='font-size:10.5pt;font-family:"Arial",sans-serif'> <o:p></o:p></span></p></div><div><p class=MsoNormal style='margin-left:36.0pt'><span style='font-size:10.0pt;font-family:"Courier New"'>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><span style='font-size:10.0pt;font-family:"Courier New"'>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></a><span style='font-size:10.5pt;font-family:"Arial",sans-serif'><o:p></o:p></span></p></div></blockquote><div><p class=MsoNormal style='margin-left:36.0pt'><span style='font-size:10.5pt;font-family:"Arial",sans-serif'> <o:p></o:p></span></p></div></div><p class=MsoNormal style='margin-left:36.0pt'><o:p> </o:p></p></div></body></html>