[gpfsug-discuss] How to do multiple mounts via GPFS

Tue Feb 22 22:07:47 GMT 2022

I'd love to see your fstab to see how you're doing that bind mount.
Do you use systemd?
What cluster manager are you using?

On 2/22/22 15:21, Maloney, J.D. wrote:
>
> Our Puppet/Ansible GPFS modules/playbooks handle this sequencing for 
> us (we use bind mounts for things like u, projects, and scratch 
> also).  Like Skylar mentioned page pool allocation, quorum checking, 
> and cluster arbitration have to come before a mount of the FS so that 
> time you mentioned doesn’t seem totally off to me.  We just make the 
> creation of the bind mounts dependent on the actual GPFS mount 
> occurring in the configuration management tooling which has worked out 
> well for us in that regard.
>
> Best,
>
> J.D. Maloney
>
> Sr. HPC Storage Engineer | Storage Enabling Technologies Group
>
> National Center for Supercomputing Applications (NCSA)
>
> *From: *gpfsug-discuss-bounces at spectrumscale.org 
> <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Skylar 
> Thompson <skylar2 at uw.edu>
> *Date: *Tuesday, February 22, 2022 at 2:13 PM
> *To: *gpfsug-discuss at spectrumscale.org <gpfsug-discuss at spectrumscale.org>
> *Subject: *Re: [gpfsug-discuss] How to do multiple mounts via GPFS
>
> The problem might be that the service indicates success when mmstartup
> returns rather than when the mount is actually active (requires quorum
> checking, arbitration, etc.). A couple tricks I can think of would be 
> using
> ConditionPathIsMountPoint from systemd.unit[1], or maybe adding a
> callback[2] that triggers on the mount condition for your filesystem that
> makes the bind mount rather than systemd.
>
> [1] 
> https://urldefense.com/v3/__https://www.freedesktop.org/software/systemd/man/systemd.unit.html*ConditionPathIsMountPoint=__;Iw!!DZ3fjg!vTT86FG0CVqF6KsdQdq6n66YOYiOPr6K2MrdTnqc2vnVduE1uhiO8VJcWTqzv4xJQwzZ$ 
> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2Fwww.freedesktop.org%2Fsoftware%2Fsystemd%2Fman%2Fsystemd.unit.html*ConditionPathIsMountPoint%3D__%3BIw!!DZ3fjg!vTT86FG0CVqF6KsdQdq6n66YOYiOPr6K2MrdTnqc2vnVduE1uhiO8VJcWTqzv4xJQwzZ%24&data=04%7C01%7Cjcantrell1%40gsu.edu%7C94e150c57b034356a37008d9f645b5e9%7C515ad73d8d5e4169895c9789dc742a70%7C0%7C0%7C637811602709016503%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=OiVg7qD5RPLMX3TbinijYhxPCGTsluZdORvidpye%2FNM%3D&reserved=0> 
>
> [2] 
> https://urldefense.com/v3/__https://www.ibm.com/docs/en/spectrum-scale/5.1.2?topic=reference-mmaddcallback-command__;!!DZ3fjg!vTT86FG0CVqF6KsdQdq6n66YOYiOPr6K2MrdTnqc2vnVduE1uhiO8VJcWTqzv3f90Gia$ 
> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2Fwww.ibm.com%2Fdocs%2Fen%2Fspectrum-scale%2F5.1.2%3Ftopic%3Dreference-mmaddcallback-command__%3B!!DZ3fjg!vTT86FG0CVqF6KsdQdq6n66YOYiOPr6K2MrdTnqc2vnVduE1uhiO8VJcWTqzv3f90Gia%24&data=04%7C01%7Cjcantrell1%40gsu.edu%7C94e150c57b034356a37008d9f645b5e9%7C515ad73d8d5e4169895c9789dc742a70%7C0%7C0%7C637811602709016503%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=mGxwgwT8LEPoZc1Mh1%2FDaroBQSoWGv6Np9KtoAh7ms8%3D&reserved=0> 
>
>
> These are both on our todo list for improving our own GPFS mounting as we
> have problems with our job scheduler not starting reliably on reboot, but
> for us we can have Puppet start it on the next run so it just means nodes
> might not return to service for 30 minutes or so.
>
> On Tue, Feb 22, 2022 at 03:05:58PM -0500, Justin Cantrell wrote:
> > This is how we're currently solving this problem, with systemd timer and
> > mount. None of the requires seem to work with gpfs since it starts 
> so late.
> > I would like a better solution.
> >
> > Is it normal for gpfs to start so late?? I think it doesn't mount until
> > after the gpfs.service starts, and even then it's 20-30 seconds.
> >
> >
> > On 2/22/22 14:42, Skylar Thompson wrote:
> > > Like Tina, we're doing bind mounts in autofs. I forgot that there 
> might be
> > > a race condition if you're doing it in fstab. If you're on system 
> with systemd,
> > > another option might be to do this directly with systemd.mount 
> rather than
> > > let the fstab generator make the systemd.mount units:
> > >
> > > 
> https://urldefense.com/v3/__https://nam11.safelinks.protection.outlook.com/?url=https*3A*2F*2Fwww.freedesktop.org*2Fsoftware*2Fsystemd*2Fman*2Fsystemd.mount.html&data=04*7C01*7Cjcantrell1*40gsu.edu*7C2a65cd0ddefd48cb81a308d9f63bb840*7C515ad73d8d5e4169895c9789dc742a70*7C0*7C0*7C637811559082622923*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000&sdata=*2BWWD7cCNSMeJEYwELldYT3pLdXVX3AxJj7gqZQCqUv4*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJSUl!!DZ3fjg!vTT86FG0CVqF6KsdQdq6n66YOYiOPr6K2MrdTnqc2vnVduE1uhiO8VJcWTqzv0tqF9rU$ 
> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2Fnam11.safelinks.protection.outlook.com%2F%3Furl%3Dhttps*3A*2F*2Fwww.freedesktop.org*2Fsoftware*2Fsystemd*2Fman*2Fsystemd.mount.html%26data%3D04*7C01*7Cjcantrell1*40gsu.edu*7C2a65cd0ddefd48cb81a308d9f63bb840*7C515ad73d8d5e4169895c9789dc742a70*7C0*7C0*7C637811559082622923*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000%26sdata%3D*2BWWD7cCNSMeJEYwELldYT3pLdXVX3AxJj7gqZQCqUv4*3D%26reserved%3D0__%3BJSUlJSUlJSUlJSUlJSUlJSUlJSUl!!DZ3fjg!vTT86FG0CVqF6KsdQdq6n66YOYiOPr6K2MrdTnqc2vnVduE1uhiO8VJcWTqzv0tqF9rU%24&data=04%7C01%7Cjcantrell1%40gsu.edu%7C94e150c57b034356a37008d9f645b5e9%7C515ad73d8d5e4169895c9789dc742a70%7C0%7C0%7C637811602709016503%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=7XXSUyYlMFqvXDCZwfhOM0%2F2%2BRDwsTZVR4kVxl1Pr0s%3D&reserved=0> 
>
> > >
> > > You could then set RequiresMountFor=gpfs1.mount in the bind mount 
> unit.
> > >
> > > On Tue, Feb 22, 2022 at 02:23:53PM -0500, Justin Cantrell wrote:
> > > > I tried a bind mount, but perhaps I'm doing it wrong. The system 
> fails
> > > > to boot because gpfs doesn't start until too late in the boot 
> process.
> > > > In fact, the system boots and the gpfs1 partition isn't 
> available for a
> > > > good 20-30 seconds.
> > > >
> > > > /gfs1/home??? /home??? none???? bind
> > > > I've tried adding mount options of x-systemd-requires=gpfs1, noauto.
> > > > The noauto lets it boot, but the mount is never mounted 
> properly. Doing
> > > > a manual mount -a mounts it.
> > > >
> > > > On 2/22/22 12:37, Skylar Thompson wrote:
> > > > > Assuming this is on Linux, you ought to be able to use bind 
> mounts for
> > > > > that, something like this in fstab or equivalent:
> > > > >
> > > > > /home /gpfs1/home bind defaults 0 0
> > > > >
> > > > > On Tue, Feb 22, 2022 at 12:24:09PM -0500, Justin Cantrell wrote:
> > > > > > We're trying to mount multiple mounts at boot up via gpfs.
> > > > > > We can mount the main gpfs mount /gpfs1, but would like to 
> mount things
> > > > > > like:
> > > > > > /home /gpfs1/home
> > > > > > /other /gpfs1/other
> > > > > > /stuff /gpfs1/stuff
> > > > > >
> > > > > > But adding that to fstab doesn't work, because from what I 
> understand,
> > > > > > that's not how gpfs works with mounts.
> > > > > > What's the standard way to accomplish something like this?
> > > > > > We've used systemd timers/mounts to accomplish it, but 
> that's not ideal.
> > > > > > Is there a way to do this natively with gpfs or does this 
> have to be done
> > > > > > through symlinks or gpfs over nfs?
> > > > _______________________________________________
> > > > gpfsug-discuss mailing list
> > > > gpfsug-discuss at spectrumscale.org
> > > > 
> https://urldefense.com/v3/__https://nam11.safelinks.protection.outlook.com/?url=http*3A*2F*2Fgpfsug.org*2Fmailman*2Flistinfo*2Fgpfsug-discuss&data=04*7C01*7Cjcantrell1*40gsu.edu*7C2a65cd0ddefd48cb81a308d9f63bb840*7C515ad73d8d5e4169895c9789dc742a70*7C0*7C0*7C637811559082622923*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000&sdata=F4oXAT0zdY*2BS1mR784ZGghUt0G*2F6Ofu36MfJ9WnPsPM*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJSUl!!DZ3fjg!vTT86FG0CVqF6KsdQdq6n66YOYiOPr6K2MrdTnqc2vnVduE1uhiO8VJcWTqzv5uX7C9S$ 
> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2Fnam11.safelinks.protection.outlook.com%2F%3Furl%3Dhttp*3A*2F*2Fgpfsug.org*2Fmailman*2Flistinfo*2Fgpfsug-discuss%26data%3D04*7C01*7Cjcantrell1*40gsu.edu*7C2a65cd0ddefd48cb81a308d9f63bb840*7C515ad73d8d5e4169895c9789dc742a70*7C0*7C0*7C637811559082622923*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000%26sdata%3DF4oXAT0zdY*2BS1mR784ZGghUt0G*2F6Ofu36MfJ9WnPsPM*3D%26reserved%3D0__%3BJSUlJSUlJSUlJSUlJSUlJSUlJSUl!!DZ3fjg!vTT86FG0CVqF6KsdQdq6n66YOYiOPr6K2MrdTnqc2vnVduE1uhiO8VJcWTqzv5uX7C9S%24&data=04%7C01%7Cjcantrell1%40gsu.edu%7C94e150c57b034356a37008d9f645b5e9%7C515ad73d8d5e4169895c9789dc742a70%7C0%7C0%7C637811602709016503%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=3g5BnkVPqUP25iX87eaSxI%2BKEZwgyv6BGQ5r3cRfrj0%3D&reserved=0> 
>
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> > 
> https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss__;!!DZ3fjg!vTT86FG0CVqF6KsdQdq6n66YOYiOPr6K2MrdTnqc2vnVduE1uhiO8VJcWTqzv34vkiw2$ 
> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__http%3A%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss__%3B!!DZ3fjg!vTT86FG0CVqF6KsdQdq6n66YOYiOPr6K2MrdTnqc2vnVduE1uhiO8VJcWTqzv34vkiw2%24&data=04%7C01%7Cjcantrell1%40gsu.edu%7C94e150c57b034356a37008d9f645b5e9%7C515ad73d8d5e4169895c9789dc742a70%7C0%7C0%7C637811602709016503%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=fVvoPrvQI%2BDCor7NMvdhmpTTZGd4WJIn0OEjSpNVZtg%3D&reserved=0> 
>
>
> -- 
> -- Skylar Thompson (skylar2 at u.washington.edu)
> -- Genome Sciences Department (UW Medicine), System Administrator
> -- Foege Building S046, (206)-685-7354
> -- Pronouns: He/Him/His
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss__;!!DZ3fjg!vTT86FG0CVqF6KsdQdq6n66YOYiOPr6K2MrdTnqc2vnVduE1uhiO8VJcWTqzv34vkiw2$ 
> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__http%3A%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss__%3B!!DZ3fjg!vTT86FG0CVqF6KsdQdq6n66YOYiOPr6K2MrdTnqc2vnVduE1uhiO8VJcWTqzv34vkiw2%24&data=04%7C01%7Cjcantrell1%40gsu.edu%7C94e150c57b034356a37008d9f645b5e9%7C515ad73d8d5e4169895c9789dc742a70%7C0%7C0%7C637811602709016503%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=fVvoPrvQI%2BDCor7NMvdhmpTTZGd4WJIn0OEjSpNVZtg%3D&reserved=0> 
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20220222/14234b72/attachment-0002.htm>