[gpfsug-discuss] Reexporting GPFS via NFS on VM host

[Christopher J. Walker]

On 14/08/15 18:54, Dean Hildebrand wrote:
> Thanks for the replies Simon...
>
> Chris, are you using -v to give the container access to the nfs subdir
> (and hence to a gpfs subdir) (and hence achieve a level of
> multi-tenancy)?

-v option to what?

> Even without containers, I wonder if this could allow
> users to run their own VMs as root as well...and preventing them from
> becoming root on gpfs...


>
> I'd love for you to share your experience (mgmt and perf) with this
> architecture once you get it up and running.

A quick and dirty test:

 From a VM:
-bash-4.1$ time dd if=/dev/zero of=cjwtestfile2 bs=1M count=10240
real 0m20.411s 0m22.137s 0m21.431s 0m21.730s 0m22.056s 0m21.759s
user 0m0.005s  0m0.007s  0m0.006s  0m0.003s  0m0.002s  0m0.004s
sys  0m11.710s 0m10.615s 0m10.399s 0m10.474s 0m10.682s 0m9.965s

 From the underlying hypervisor.

real 0m11.138s 0m9.813s 0m9.761s 0m9.793s 0m9.773s 0m9.723s
user 0m0.006s  0m0.013s 0m0.009s 0m0.008s 0m0.008s 0m0.009s
sys  0m5.447s  0m5.529s 0m5.802s 0m5.580s 0m6.190s 0m5.516s

So there's a factor of just over 2 slowdown.

As it's still 500MB/s, I think it's good enough for now.

The machine has a 10Gbit/s network connection and both hypervisor and VM 
are running SL6.

> Some side benefits of this
> architecture that we have been thinking about as well  is that it allows
> both the containers and VMs to be somewhat ephemeral, while the gpfs
> continues to run in the hypervisor...

Indeed. This is another advantage.

If we were running Debian, it would be possible to export part of a 
filesystem to a VM. Which would presumably work. In redhat derived OSs 
(we are currently using Scientific Linux), I don't believe it is - hence 
exporting via NFS.

>
> To ensure VMotion works relatively smoothly, just ensure each VM is
> given a hostname to mount that always routes back to the localhost nfs
> server on each machine...and I think things should work relatively
> smoothly.  Note you'll need to maintain the same set of nfs exports
> across the entire cluster as well, so taht when a VM moves to another
> machine it immediately has an available export to mount.

Yes, we are doing this.

Simon alludes to potential problems at the NFS layer on live migration. 
Otherwise, yes indeed the setup should be fine.  I'm not familiar enough 
with the details of NFS - but I have heard NFS described as "a stateless 
filesystem with state". It's the stateful bits I'm concerned about.

Chris

>
> Dean Hildebrand
> IBM Almaden Research Center
>
>
> Inactive hide details for "Simon Thompson (Research Computing - IT
> Services)" ---08/13/2015 07:33:16 AM--->I've set up a couple"Simon
> Thompson (Research Computing - IT Services)" ---08/13/2015 07:33:16
> AM--->I've set up a couple of VM hosts to export some of its GPFS
> filesystem >via NFS to machines on that
>
> From: "Simon Thompson (Research Computing - IT Services)"
> <S.J.Thompson at bham.ac.uk>
> To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
> Date: 08/13/2015 07:33 AM
> Subject: Re: [gpfsug-discuss] Reexporting GPFS via NFS on VM host
> Sent by: gpfsug-discuss-bounces at gpfsug.org
>
> ------------------------------------------------------------------------
>
>
>
>
>  >I've set up a couple of VM hosts to export some of its GPFS filesystem
>  >via NFS to machines on that VM host[1,2].
>
> Provided all your sockets no the VM host are licensed.
>
>  >Is live migration of VMs likely to work?
>  >
>  >Live migration isn't a hard requirement, but if it will work, it could
>  >make our life easier.
>
> Live migration using a GPFS file-system on the hypervisor node should work
> (subject to the usual caveats of live migration).
>
> Whether live migration and your VM instances would still be able to NFS
> mount (assuming loopback address?) if they moved to a different
> hypervisor, pass, you might get weird NFS locks. And if they are still
> mounting from the original VM host, then you are not doing what the FAQ
> says you can do.
>
> Simon
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>