<html><body><p><font size="2">All,</font><br><br><font size="2">For at least one of the instances reported by this group, a PMR has been opened, and a fix is being developed. For folks that are getting affected by the problem: Please contact the service team to confirm your problem is the same as the one previously reported, and for an outlook for the availability of the fix.</font><br><br><font size="2">Thanks,</font><br><br><font size="2"> Felipe</font><br><br><font size="2">----<br>Felipe Knop knop@us.ibm.com<br>GPFS Development and Security<br>IBM Systems<br>IBM Building 008<br>2455 South Rd, Poughkeepsie, NY 12601<br>(845) 433-9314 T/L 293-9314<br><br></font><br><br><img width="16" height="16" src="cid:1__=8FBB08BCDFDA24DD8f9e8a93df938690918c8FB@" border="0" alt="Inactive hide details for Bryan Banister ---02/08/2018 04:11:49 PM---It may be related to this issue of using root squashed fil"><font size="2" color="#424282">Bryan Banister ---02/08/2018 04:11:49 PM---It may be related to this issue of using root squashed file system option, here are some edited comm</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">Bryan Banister <bbanister@jumptrading.com></font><br><font size="2" color="#5F5F5F">To: </font><font size="2">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org>, "Loic Tortay" <tortay@cc.in2p3.fr></font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">02/08/2018 04:11 PM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">Re: [gpfsug-discuss] mounts taking longer in 4.2 vs 4.1?</font><br><font size="2" color="#5F5F5F">Sent by: </font><font size="2">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br><br><br><font face="Calibri">It may be related to this issue of using root squashed file system option, here are some edited comments from my colleague who stumbled upon this while chatting with a friend at a CUG:</font><br><font face="Calibri"> </font><br><font face="Calibri">" Something I learned last week: apparently the libmount code from util-linux (used by /bin/mount) will call utimensat() on new mountpoints if access() fails (for example, on root-squashed filesystems). This is done "just to be sure" that the filesystem is really read-only. This operation can be quite expensive and (anecdotally) may cause huge slowdowns when mounting root-squashed parallel filesystems on thousands of clients.</font><br><font face="Calibri"> </font><br><font face="Calibri">Here is the relevant code:</font><br><font face="Calibri"> </font><br><font face="Calibri"><a href="https://github.com/karelzak/util-linux/blame/1ea4e7bd8d9d0f0ef317558c627e6fa069950e8d/libmount/src/utils.c#L222">https://github.com/karelzak/util-linux/blame/1ea4e7bd8d9d0f0ef317558c627e6fa069950e8d/libmount/src/utils.c#L222</a></font><br><font face="Calibri"> </font><br><font face="Calibri">This code has been in util-linux for years.</font><br><font face="Calibri"> </font><br><font face="Calibri">It's not clear exactly what the impact is in our environment, but this certainly can't be helping, especially since we've grown the size of the cluster considerably. Mounting GPFS has recently really become a slow and disruptive operation – if you try to mount many clients at once, the FS will hang for a considerable period of time.</font><br><font face="Calibri"> </font><br><font face="Calibri">The timing varies, but here is one example from an isolated mounting operation:</font><br><font face="Calibri"> </font><br><font face="Calibri">12:09:11.222513 mount("<gpfs_fs>", "<mount_point>", "gpfs", MS_MGC_VAL, "dev=<gpfs_cluster>"...) = 0 <1.590217></font><br><font face="Calibri">12:09:12.812777 access("<mount_point>", W_OK) = -1 EACCES (Permission denied) <0.000022></font><br><font face="Calibri">12:09:12.812841 utimensat(AT_FDCWD, "<mount_point>", \{UTIME_NOW, \{93824994378048, 1073741822}}, 0) = -1 EPERM (Operation not permitted) <2.993689></font><br><font face="Calibri">Here, the utimensat() took ~3 seconds, almost twice as long as the mount operation! I also suspect it will slow down other clients trying to mount the filesystem since the sgmgr has to process this write attempt to the mountpoint.</font><br><font face="Calibri"> </font><br><font face="Calibri">(Hilariously, it still returns the "wrong" answer, because this filesystem is not read-only, just squashed.)</font><br><font face="Calibri"> </font><br><font face="Calibri">As of today, the person who originally brought the issue to my attention at CUG has raised it for discussion on the util-linux mailing list.</font><br><font face="Calibri"><a href="https://marc.info/?l=util-linux-ng&m=151075932824688&w=2">https://marc.info/?l=util-linux-ng&m=151075932824688&w=2</a></font><br><font face="Calibri">"</font><br><font face="Calibri"> </font><br><font face="Calibri">We ended up putting facls on our mountpoints like such, which hacked around this stupidity:</font><br><font size="2" face="Courier New">for fs in gpfs_mnt_point ; do</font><br><font size="2" face="Courier New"> chmod 1755 $fs</font><br><font size="2" face="Courier New"> setfacl -m u:99:rwx $fs # 99 is the "nobody" uid to which root is mapped--see "mmauth" output</font><br><font size="2" face="Courier New">done</font><br><font face="Calibri"> </font><br><font face="Calibri">Hope that helps,</font><br><font face="Calibri">-Bryan</font><br><font face="Calibri"> </font><br><font face="Calibri">-----Original Message-----<br>From: gpfsug-discuss-bounces@spectrumscale.org [</font><font face="Calibri"><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">mailto:gpfsug-discuss-bounces@spectrumscale.org</a></font><font face="Calibri">] On Behalf Of Aaron Knister<br>Sent: Thursday, February 08, 2018 2:23 PM<br>To: Loic Tortay <tortay@cc.in2p3.fr><br>Cc: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Subject: Re: [gpfsug-discuss] mounts taking longer in 4.2 vs 4.1?</font><br><font face="Calibri"> </font><br><font face="Calibri">Note: External Email</font><br><font face="Calibri">-------------------------------------------------</font><br><font face="Calibri"> </font><br><font face="Calibri">Hi Loic,</font><br><font face="Calibri"> </font><br><font face="Calibri">Thank you for that information!</font><br><font face="Calibri"> </font><br><font face="Calibri">I have two follow up questions--</font><br><font face="Calibri">1. Are you using ccr?</font><br><font face="Calibri">2. Do you happen to have mmsdrserv disabled in your environment? (e.g. what's the output of "mmlsconfig mmsdrservPort" on your cluster?).</font><br><font face="Calibri"> </font><br><font face="Calibri">-Aaron</font><br><font face="Calibri"> </font><br><font face="Calibri">On Thu, 8 Feb 2018, Loic Tortay wrote:</font><br><font face="Calibri"> </font><br><font face="Calibri">> On 07/02/2018 22:28, Aaron Knister wrote:</font><br><font face="Calibri">>> I noticed something curious after migrating some nodes from 4.1 to 4.2</font><br><font face="Calibri">>> which is that mounts now can take foorrreeevverrr. It seems to boil down</font><br><font face="Calibri">>> to the point in the mount process where getEFOptions is called.</font><br><font face="Calibri">>> </font><br><font face="Calibri">>> To highlight the difference--</font><br><font face="Calibri">>> </font><br><font face="Calibri">> [...]</font><br><font face="Calibri">>> </font><br><font face="Calibri">> Hello,</font><br><font face="Calibri">> I have had this (or a very similar) issue after migrating from 4.1.1.8 to </font><br><font face="Calibri">> 4.2.3. There are 37 filesystems in our main cluster, which made the problem </font><br><font face="Calibri">> really noticeable.</font><br><font face="Calibri">> </font><br><font face="Calibri">> A PMR has been opened. I have tested the fixes included in 4.2.3.7, (which, </font><br><font face="Calibri">> I'm told, should be released today) actually resolve my problems (APAR </font><br><font face="Calibri">> IJ03192 & IJ03235).</font><br><font face="Calibri">> </font><br><font face="Calibri">> </font><br><font face="Calibri">> Loïc.</font><br><font face="Calibri">> -- </font><br><font face="Calibri">> | Loïc Tortay <</font><a href="mailto:tortay@cc.in2p3.fr"><font color="#0563C1" face="Calibri">tortay@cc.in2p3.fr</font></a><font face="Calibri">> - IN2P3 Computing Centre |</font><br><font face="Calibri">> </font><p><br><hr width="100%" size="2" align="left"><font size="2" color="#808080" face="Arial"><br>Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.</font><tt><font size="2">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><tt><font size="2"><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=oNT2koCZX0xmWlSlLblR9Q&m=C0S8WTufrOCvXbHUegB8zS9jk_1SLczALa-4aVEubu4&s=VTWKI-xcUiJ_LeMhJ-xOPmnz0Zm9IspKsU3bsxA4BNo&e=">https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=oNT2koCZX0xmWlSlLblR9Q&m=C0S8WTufrOCvXbHUegB8zS9jk_1SLczALa-4aVEubu4&s=VTWKI-xcUiJ_LeMhJ-xOPmnz0Zm9IspKsU3bsxA4BNo&e=</a></font></tt><tt><font size="2"><br></font></tt><p><p><BR>
</body></html>