<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p><tt>Dear Malahal,</tt></p>
<p><tt>thanks for the answer. Concerning SSSD, we are also using it,
should we use 5.0.2-PTF3? We would like to avoid using 5.0.2.2,
as it has issues with recent RHEL 7.6 kernels [*] and we are
impacted: do you suggest to use 5.0.3.3?</tt></p>
<p><tt>cheers</tt></p>
<p><tt>leo<br>
</tt></p>
<p><tt><br>
</tt></p>
<p><tt>[*] </tt><a
href="https://www.ibm.com/support/pages/ibm-spectrum-scale-gpfs-releases-42313-or-later-and-5022-or-later-have-issues-where-kernel-crashes-rhel76-0">https://www.ibm.com/support/pages/ibm-spectrum-scale-gpfs-releases-42313-or-later-and-5022-or-later-have-issues-where-kernel-crashes-rhel76-0</a></p>
<pre class="moz-signature" cols="72">Paul Scherrer Institut
Dr. Leonardo Sala
Group Leader High Performance Computing
Deputy Section Head Science IT
Science IT
WHGA/106
5232 Villigen PSI
Switzerland
Phone: +41 56 310 3369
<a class="moz-txt-link-abbreviated" href="mailto:leonardo.sala@psi.ch">leonardo.sala@psi.ch</a>
<a class="moz-txt-link-abbreviated" href="http://www.psi.ch">www.psi.ch</a></pre>
<div class="moz-cite-prefix">On 03.10.19 19:15, Malahal R Naineni
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:OF13182575.836A6F6A-ON00258488.005D37F7-00258488.005EC3B2@notes.na.collabserv.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div class="socmaildefaultfont" dir="ltr"
style="font-family:Arial, Helvetica, sans-serif;font-size:10pt">
<div dir="ltr">
<div>>> @Malahal: Looks like you have written the
netgroup caching code, feel free to ask for further details
if required.</div>
<div> </div>
<div>Hi Ulrich, Ganesha uses innetgr() call for netgroup
information and sssd has too many issues in its
implementation. Redhat said that they are going to fix sssd
synchronization issues in RHEL8. It is in my plate to
serialize innergr() call in Ganesha to match kernel NFS
server usage! I expect the sssd issue to give EACCESS/EPERM
kind of issue but not EINVAL though.</div>
<div> </div>
<div>If you are using sssd, you must be getting into a sssd
issue. Ganesha has a host-ip cache fix in 5.0.2 PTF3. Please
make sure you use ganesha version V2.5.3-ibm030.01 if you
are using netgroups (shipped with 5.0.2 PTF3 but can be used
with Scale 5.0.1 or later)</div>
<div> </div>
<div>Regards, Malahal.</div>
</div>
<div dir="ltr"> </div>
<blockquote data-history-content-modified="1"
data-history-expanded="1" dir="ltr" style="border-left:solid
#aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr;
margin-right:0px">----- Original message -----<br>
From: Ulrich Sibiller <a class="moz-txt-link-rfc2396E" href="mailto:u.sibiller@science-computing.de"><u.sibiller@science-computing.de></a><br>
Sent by: <a class="moz-txt-link-abbreviated" href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-bounces@spectrumscale.org</a><br>
To: <a class="moz-txt-link-abbreviated" href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a><br>
Cc:<br>
Subject: Re: [gpfsug-discuss] Filesystem access issues via CES
NFS<br>
Date: Thu, Dec 13, 2018 7:32 PM<br>
<div><font size="2" face="Default Monospace,Courier
New,Courier,monospace">On 23.11.2018 14:41, Andreas
Mattsson wrote:<br>
> Yes, this is repeating.<br>
><br>
> We’ve ascertained that it has nothing to do at all
with file operations on the GPFS side.<br>
><br>
> Randomly throughout the filesystem mounted via NFS,
ls or file access will give<br>
><br>
> ”<br>
><br>
> > ls: reading directory
/gpfs/filessystem/test/testdir: Invalid argument<br>
><br>
> “<br>
><br>
> Trying again later might work on that folder, but
might fail somewhere else.<br>
><br>
> We have tried exporting the same filesystem via a
standard kernel NFS instead of the CES<br>
> Ganesha-NFS, and then the problem doesn’t exist.<br>
><br>
> So it is definitely related to the Ganesha NFS
server, or its interaction with the file system.<br>
> > Will see if I can get a tcpdump of the issue.<br>
<br>
We see this, too. We cannot trigger it. Fortunately I have
managed to capture some logs with<br>
debugging enabled. I have now dug into the ganesha 2.5.3
code and I think the netgroup caching is<br>
the culprit.<br>
<br>
Here some FULL_DEBUG output:<br>
2018-12-13 11:53:41 : epoch 0009008d : server1 :
gpfs.ganesha.nfsd-258762[work-250]<br>
export_check_access :EXPORT :M_DBG :Check for address
1.2.3.4 for export id 1 path /gpfsexport<br>
2018-12-13 11:53:41 : epoch 0009008d : server1 :
gpfs.ganesha.nfsd-258762[work-250] client_match<br>
:EXPORT :M_DBG :Match V4: 0xcf7fe0 NETGROUP_CLIENT:
netgroup1 (options=421021e2root_squash , RWrw,<br>
3--, ---, TCP, ----, Manage_Gids , -- Deleg, anon_uid=
-2, anon_gid= -2, sys)<br>
2018-12-13 11:53:41 : epoch 0009008d : server1 :
gpfs.ganesha.nfsd-258762[work-250] nfs_ip_name_get<br>
:DISP :F_DBG :Cache get hit for 1.2.3.4->client1.domain<br>
2018-12-13 11:53:41 : epoch 0009008d : server1 :
gpfs.ganesha.nfsd-258762[work-250] client_match<br>
:EXPORT :M_DBG :Match V4: 0xcfe320 NETGROUP_CLIENT:
netgroup2 (options=421021e2root_squash , RWrw,<br>
3--, ---, TCP, ----, Manage_Gids , -- Deleg, anon_uid=
-2, anon_gid= -2, sys)<br>
2018-12-13 11:53:41 : epoch 0009008d : server1 :
gpfs.ganesha.nfsd-258762[work-250] nfs_ip_name_get<br>
:DISP :F_DBG :Cache get hit for 1.2.3.4->client1.domain<br>
2018-12-13 11:53:41 : epoch 0009008d : server1 :
gpfs.ganesha.nfsd-258762[work-250] client_match<br>
:EXPORT :M_DBG :Match V4: 0xcfe380 NETGROUP_CLIENT:
netgroup3 (options=421021e2root_squash , RWrw,<br>
3--, ---, TCP, ----, Manage_Gids , -- Deleg, anon_uid=
-2, anon_gid= -2, sys)<br>
2018-12-13 11:53:41 : epoch 0009008d : server1 :
gpfs.ganesha.nfsd-258762[work-250] nfs_ip_name_get<br>
:DISP :F_DBG :Cache get hit for 1.2.3.4->client1.domain<br>
2018-12-13 11:53:41 : epoch 0009008d : server1 :
gpfs.ganesha.nfsd-258762[work-250]<br>
export_check_access :EXPORT :M_DBG :EXPORT
(options=03303002 , , ,<br>
, , -- Deleg, ,
)<br>
2018-12-13 11:53:41 : epoch 0009008d : server1 :
gpfs.ganesha.nfsd-258762[work-250]<br>
export_check_access :EXPORT :M_DBG :EXPORT_DEFAULTS
(options=42102002root_squash , ----, 3--, ---,<br>
TCP, ----, Manage_Gids , , anon_uid= -2,
anon_gid= -2, sys)<br>
2018-12-13 11:53:41 : epoch 0009008d : server1 :
gpfs.ganesha.nfsd-258762[work-250]<br>
export_check_access :EXPORT :M_DBG :default options
(options=03303002root_squash , ----, 34-, UDP,<br>
TCP, ----, No Manage_Gids, -- Deleg, anon_uid= -2,
anon_gid= -2, none, sys)<br>
2018-12-13 11:53:41 : epoch 0009008d : server1 :
gpfs.ganesha.nfsd-258762[work-250]<br>
export_check_access :EXPORT :M_DBG :Final options
(options=42102002root_squash , ----, 3--, ---,<br>
TCP, ----, Manage_Gids , -- Deleg, anon_uid= -2,
anon_gid= -2, sys)<br>
2018-12-13 11:53:41 : epoch 0009008d : server1 :
gpfs.ganesha.nfsd-258762[work-250] nfs_rpc_execute<br>
:DISP :INFO :DISP: INFO: Client ::ffff:1.2.3.4 is not
allowed to access Export_Id 1 /gpfsexport,<br>
vers=3, proc=18<br>
<br>
The client "client1" is definitely a member of the
"netgroup1". But the NETGROUP_CLIENT lookups for<br>
"netgroup2" and "netgroup3" can only happen if the
netgroup caching code reports that "client1" is<br>
NOT a member of "netgroup1".<br>
<br>
I have also opened a support case at IBM for this.<br>
<br>
@Malahal: Looks like you have written the netgroup caching
code, feel free to ask for further<br>
details if required.<br>
<br>
Kind regards,<br>
<br>
Ulrich Sibiller<br>
<br>
--<br>
Dipl.-Inf. Ulrich Sibiller science + computing
ag<br>
System Administration Hagellocher Weg
73<br>
72070 Tuebingen,
Germany<br>
<a
href="https://atos.net/de/deutschland/sc"
target="_blank" moz-do-not-send="true">https://atos.net/de/deutschland/sc</a><br>
--<br>
Science + Computing AG<br>
Vorstandsvorsitzender/Chairman of the board of management:<br>
Dr. Martin Matzke<br>
Vorstand/Board of Management:<br>
Matthias Schempp, Sabine Hohenstein<br>
Vorsitzender des Aufsichtsrats/<br>
Chairman of the Supervisory Board:<br>
Philippe Miltin<br>
Aufsichtsrat/Supervisory Board:<br>
Martin Wibbe, Ursula Morgenstern<br>
Sitz/Registered Office: Tuebingen<br>
Registergericht/Registration Court: Stuttgart<br>
Registernummer/Commercial Register No.: HRB 382196<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org<br>
<a
href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"
target="_blank" moz-do-not-send="true">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font><br>
</div>
</blockquote>
<div dir="ltr"> </div>
</div>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
<a class="moz-txt-link-freetext" href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a>
</pre>
</blockquote>
</body>
</html>