<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" >Hi</div>
<div dir="ltr" > </div>
<div dir="ltr" >The doc is right. The testing acceptance would be a testing environment if you really want end to end testing. Assuming you have the CES nodes on different Spectrum Scale cluster, nothing stops you from have another Spectrum Scale cluster for the new CES code level (different IPs for serving)</div>
<div dir="ltr" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" ><div style="font-size: 12pt; font-weight: bold; font-family: sans-serif; color: #7C7C5F;" ><font size="3" face="Arial" >--<br>Ystävällisin terveisin / Kind regards / Saludos cordiales / Salutations</font><br><font size="3" face="Arial" >Luis Bolinches</font><br><font size="3" face="Arial" >Certified Consultant IT Specialist</font><br><font size="3" face="Arial" >Mobile Phone: +358503112585</font><br><a href="https://www.youracclaim.com/user/luis-bolinches" rel="noopener" target="_blank"><b><font size="3" face="Arial" >https://www.youracclaim.com/user/luis-bolinches</font></b></a><br><br><b><font size="3" face="Arial" color="#7C7C5F" >"If you always give you will always have" -- Anonymous</font></b></div>
<div style="font-size: 8pt; font-family: sans-serif; margin-top: 10px;" ><div> </div></div></div></div></div></div></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" data-history-expanded="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: "Talamo Ivano Giuseppe (PSI)" <Ivano.Talamo@psi.ch><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Cc:<br>Subject: [EXTERNAL] Re: [gpfsug-discuss] Filesystem access issues via CES NFS<br>Date: Wed, Oct 23, 2019 12:49 PM<br>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace" >Dear all,<br><br>We are actually in the process of upgrading our CES cluster to 5.0.3-3 but we have doubts about how to proceed.<br>Considering that the CES cluster is in production and heavily used, our plan is to add a new node with 5.0.3-3 to the cluster that is currently 5.0.2.1.<br><br>And we would like to proceed in a cautious way, so that the new node would not take any IP and just one day per week (when we will declare to be “at risk”) we would move some IPs to it. After some weeks of tests if we would see no problem we would upgrade the rest of the cluster.<br><br>But reading these doc [1] it seems that we cannot have multiple GPFS/SMB version in the same cluster. So in that case we could not have a testing/acceptance phase but could only make the full blind jump. Can someone confirm or negate this?<br><br>Thanks,<br>Ivano<br><br>[1] <a href="https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.2/com.ibm.spectrum.scale.v5r02.doc/bl1ins_updatingsmb.htm" target="_blank">https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.2/com.ibm.spectrum.scale.v5r02.doc/bl1ins_updatingsmb.htm</a><br><br>On 04.10.19, 12:55, "gpfsug-discuss-bounces@spectrumscale.org on behalf of Malahal R Naineni" <gpfsug-discuss-bounces@spectrumscale.org on behalf of mnaineni@in.ibm.com> wrote:<br><br> You can use 5.0.3.3 . There is no fix for the sssd issue yet though. I will work with Ganesha upstream community pretty soon.<br> <br> Regards, Malahal.<br> <br> ----- Original message -----<br> From: Leonardo Sala <leonardo.sala@psi.ch><br> To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org>, "Malahal R Naineni" <mnaineni@in.ibm.com>, <u.sibiller@science-computing.de><br> Cc:<br> Subject: [EXTERNAL] Re: [gpfsug-discuss] Filesystem access issues via CES NFS<br> Date: Fri, Oct 4, 2019 12:02 PM<br> <br> Dear Malahal,<br> thanks for the answer. Concerning SSSD, we are also using it, should we use 5.0.2-PTF3? We would like to avoid using 5.0.2.2, as it has issues with recent RHEL 7.6 kernels [*] and we are impacted: do you suggest to use 5.0.3.3?<br> cheers<br> leo<br> <br> [*] <a href="https://www.ibm.com/support/pages/ibm-spectrum-scale-gpfs-releases-42313-or-later-and-5022-or-later-have-issues-where-kernel-crashes-rhel76-0" target="_blank">https://www.ibm.com/support/pages/ibm-spectrum-scale-gpfs-releases-42313-or-later-and-5022-or-later-have-issues-where-kernel-crashes-rhel76-0</a><br> Paul Scherrer Institut<br> Dr. Leonardo Sala<br> Group Leader High Performance Computing<br> Deputy Section Head Science IT<br> Science IT<br> WHGA/106<br> 5232 Villigen PSI<br> Switzerland<br> <br> Phone: +41 56 310 3369<br> leonardo.sala@psi.ch<br> www.psi.ch <<a href="http://www.psi.ch" target="_blank">http://www.psi.ch</a> ><br> On 03.10.19 19:15, Malahal R Naineni wrote:<br> >> @Malahal: Looks like you have written the netgroup caching code, feel free to ask for further details if required.<br> <br> Hi Ulrich, Ganesha uses innetgr() call for netgroup information and sssd has too many issues in its implementation. Redhat said that they are going to fix sssd synchronization issues in RHEL8. It is in my plate to serialize innergr() call in Ganesha to<br> match kernel NFS server usage! I expect the sssd issue to give EACCESS/EPERM kind of issue but not EINVAL though.<br> <br> If you are using sssd, you must be getting into a sssd issue. Ganesha has a host-ip cache fix in 5.0.2 PTF3. Please make sure you use ganesha version V2.5.3-ibm030.01 if you are using netgroups (shipped with 5.0.2 PTF3 but can be used with Scale 5.0.1<br> or later)<br> <br> Regards, Malahal.<br> <br> <br> <br> ----- Original message -----<br> From: Ulrich Sibiller<br> <u.sibiller@science-computing.de> <<a href="mailto:u.sibiller@science-computing.de" target="_blank">mailto:u.sibiller@science-computing.de</a>><br> Sent by:<br> gpfsug-discuss-bounces@spectrumscale.org <<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">mailto:gpfsug-discuss-bounces@spectrumscale.org</a>><br> To: gpfsug-discuss@spectrumscale.org<br> Cc:<br> Subject: Re: [gpfsug-discuss] Filesystem access issues via CES NFS<br> Date: Thu, Dec 13, 2018 7:32 PM<br> <br> On 23.11.2018 14:41, Andreas Mattsson wrote:<br> > Yes, this is repeating.<br> ><br> > We’ve ascertained that it has nothing to do at all with file operations on the GPFS side.<br> ><br> > Randomly throughout the filesystem mounted via NFS, ls or file access will give<br> ><br> > ”<br> ><br> > > ls: reading directory /gpfs/filessystem/test/testdir: Invalid argument<br> ><br> > “<br> ><br> > Trying again later might work on that folder, but might fail somewhere else.<br> ><br> > We have tried exporting the same filesystem via a standard kernel NFS instead of the CES<br> > Ganesha-NFS, and then the problem doesn’t exist.<br> ><br> > So it is definitely related to the Ganesha NFS server, or its interaction with the file system.<br> > > Will see if I can get a tcpdump of the issue.<br> <br> We see this, too. We cannot trigger it. Fortunately I have managed to capture some logs with<br> debugging enabled. I have now dug into the ganesha 2.5.3 code and I think the netgroup caching is<br> the culprit.<br> <br> Here some FULL_DEBUG output:<br> 2018-12-13 11:53:41 : epoch 0009008d : server1 : gpfs.ganesha.nfsd-258762[work-250]<br> export_check_access :EXPORT :M_DBG :Check for address 1.2.3.4 for export id 1 path /gpfsexport<br> 2018-12-13 11:53:41 : epoch 0009008d : server1 : gpfs.ganesha.nfsd-258762[work-250] client_match<br> :EXPORT :M_DBG :Match V4: 0xcf7fe0 NETGROUP_CLIENT: netgroup1 (options=421021e2root_squash , RWrw,<br> 3--, ---, TCP, ----, Manage_Gids , -- Deleg, anon_uid= -2, anon_gid= -2, sys)<br> 2018-12-13 11:53:41 : epoch 0009008d : server1 : gpfs.ganesha.nfsd-258762[work-250] nfs_ip_name_get<br> :DISP :F_DBG :Cache get hit for 1.2.3.4->client1.domain<br> 2018-12-13 11:53:41 : epoch 0009008d : server1 : gpfs.ganesha.nfsd-258762[work-250] client_match<br> :EXPORT :M_DBG :Match V4: 0xcfe320 NETGROUP_CLIENT: netgroup2 (options=421021e2root_squash , RWrw,<br> 3--, ---, TCP, ----, Manage_Gids , -- Deleg, anon_uid= -2, anon_gid= -2, sys)<br> 2018-12-13 11:53:41 : epoch 0009008d : server1 : gpfs.ganesha.nfsd-258762[work-250] nfs_ip_name_get<br> :DISP :F_DBG :Cache get hit for 1.2.3.4->client1.domain<br> 2018-12-13 11:53:41 : epoch 0009008d : server1 : gpfs.ganesha.nfsd-258762[work-250] client_match<br> :EXPORT :M_DBG :Match V4: 0xcfe380 NETGROUP_CLIENT: netgroup3 (options=421021e2root_squash , RWrw,<br> 3--, ---, TCP, ----, Manage_Gids , -- Deleg, anon_uid= -2, anon_gid= -2, sys)<br> 2018-12-13 11:53:41 : epoch 0009008d : server1 : gpfs.ganesha.nfsd-258762[work-250] nfs_ip_name_get<br> :DISP :F_DBG :Cache get hit for 1.2.3.4->client1.domain<br> 2018-12-13 11:53:41 : epoch 0009008d : server1 : gpfs.ganesha.nfsd-258762[work-250]<br> export_check_access :EXPORT :M_DBG :EXPORT (options=03303002 , , ,<br> , , -- Deleg, , )<br> 2018-12-13 11:53:41 : epoch 0009008d : server1 : gpfs.ganesha.nfsd-258762[work-250]<br> export_check_access :EXPORT :M_DBG :EXPORT_DEFAULTS (options=42102002root_squash , ----, 3--, ---,<br> TCP, ----, Manage_Gids , , anon_uid= -2, anon_gid= -2, sys)<br> 2018-12-13 11:53:41 : epoch 0009008d : server1 : gpfs.ganesha.nfsd-258762[work-250]<br> export_check_access :EXPORT :M_DBG :default options (options=03303002root_squash , ----, 34-, UDP,<br> TCP, ----, No Manage_Gids, -- Deleg, anon_uid= -2, anon_gid= -2, none, sys)<br> 2018-12-13 11:53:41 : epoch 0009008d : server1 : gpfs.ganesha.nfsd-258762[work-250]<br> export_check_access :EXPORT :M_DBG :Final options (options=42102002root_squash , ----, 3--, ---,<br> TCP, ----, Manage_Gids , -- Deleg, anon_uid= -2, anon_gid= -2, sys)<br> 2018-12-13 11:53:41 : epoch 0009008d : server1 : gpfs.ganesha.nfsd-258762[work-250] nfs_rpc_execute<br> :DISP :INFO :DISP: INFO: Client ::ffff:1.2.3.4 is not allowed to access Export_Id 1 /gpfsexport,<br> vers=3, proc=18<br> <br> The client "client1" is definitely a member of the "netgroup1". But the NETGROUP_CLIENT lookups for<br> "netgroup2" and "netgroup3" can only happen if the netgroup caching code reports that "client1" is<br> NOT a member of "netgroup1".<br> <br> I have also opened a support case at IBM for this.<br> <br> @Malahal: Looks like you have written the netgroup caching code, feel free to ask for further<br> details if required.<br> <br> Kind regards,<br> <br> Ulrich Sibiller<br> <br> --<br> Dipl.-Inf. Ulrich Sibiller science + computing ag<br> System Administration Hagellocher Weg 73<br> 72070 Tuebingen, Germany<br> <br> <a href="https://atos.net/de/deutschland/sc" target="_blank">https://atos.net/de/deutschland/sc</a> <<a href="https://atos.net/de/deutschland/sc" target="_blank">https://atos.net/de/deutschland/sc</a> ><br> --<br> Science + Computing AG<br> Vorstandsvorsitzender/Chairman of the board of management:<br> Dr. Martin Matzke<br> Vorstand/Board of Management:<br> Matthias Schempp, Sabine Hohenstein<br> Vorsitzender des Aufsichtsrats/<br> Chairman of the Supervisory Board:<br> Philippe Miltin<br> Aufsichtsrat/Supervisory Board:<br> Martin Wibbe, Ursula Morgenstern<br> Sitz/Registered Office: Tuebingen<br> Registergericht/Registration Court: Stuttgart<br> Registernummer/Commercial Register No.: HRB 382196<br> _______________________________________________<br> gpfsug-discuss mailing list<br> gpfsug-discuss at spectrumscale.org<br> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> <br> <br> <br> <br> <br> <br> _______________________________________________<br> gpfsug-discuss mailing list<br> gpfsug-discuss at spectrumscale.org<br> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> <br> <br> <br> <br> <br> <br> <br> <br> <br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> </font><br> </div></blockquote>
<div dir="ltr" > </div></div><BR>
Ellei edellä ole toisin mainittu: / Unless stated otherwise above:<BR>
Oy IBM Finland Ab<BR>
PL 265, 00101 Helsinki, Finland<BR>
Business ID, Y-tunnus: 0195876-3 <BR>
Registered in Finland<BR>
<BR>