[gpfsug-discuss] GPFS and both Samba and NFS
Chair
chair at gpfsug.org
Mon Dec 16 16:05:26 GMT 2013
Hi Sven,
Many thanks for taking the time to write back in detail.
This is exactly the sort of discussion the user group is aimed at - I.E.
technical discussion outside of the GPFS Developer Forum.
I heartily encourage other IBMers to get involved.
Regards,
Jez
On 16/12/13 15:31, Sven Oehme wrote:
> Jez,
>
> the other replies to my email where kind of unexpected as i was indeed
> intending to help, i will continue to provide details and try to
> answer serious questions and comments.
>
> also as you point out there is no secret master plan to spread
> misinformation, the topic is simply complicated and there are reasons
> that are hard to explain why there is no full official support outside
> the current shipping IBM products and i will not dive into discussions
> around it.
>
> a few words of clarification, on the controversial point, compile ctdb
> yes/no.
> while its technically correct, that you don't have to recompile ctdb
> and it has no code that links/depends in any form on GPFS (samba
> does), you still have to compile one if the one shipped with your
> distro is out of date and while i didn't explicitly wrote it this way,
> this is what i intended to say.
>
> the version RH ships is very old and since has gotten many fixes which
> i highly recommend people to use unless the distros will pickup more
> recent version like RHEL7 will do. i probably wouldn't recommend a
> daily git build, but rather the packaged versions that are hosted on
> ftp.samba.org like :
> _https://ftp.samba.org/pub/ctdb/packages/redhat/RHEL6/_
>
> so the proposed order of things would be to install gpfs, pull the src
> package of ctdb, compile and install ctdb and the devel packages, then
> pull a recent samba src package , install all the dependencies and
> build samba on this same host with gpfs and ctdb packages already
> installed. the resulting rpm's should contain the proper code to
> continue.
>
> after you got your versions compiled, installed and the basics of ctdb
> running, you should use the following smb.conf as a starting point :
>
> [global]
> netbios name = cluster
> fileid:mapping = fsname
> gpfs:sharemodes = yes
> gpfs:leases = yes
> gpfs:dfreequota = yes
> gpfs:hsm = yes
> syncops:onmeta = no
> kernel oplocks = no
> level2 oplocks = yes
> notify:inotify = no
> vfs objects = shadow_copy2 syncops gpfs fileid
> shadow:snapdir = .snapshots
> shadow:fixinodes = yes
> shadow:snapdirseverywhere = yes
> shadow:sort = desc
> wide links = no
> async smb echo handler = yes
> smbd:backgroundqueue = False
> use sendfile = no
> strict locking = yes
> posix locking = yes
> force unknown acl user = yes
> nfs4:mode = simple
> nfs4:chown = yes
> nfs4:acedup = merge
> nfs4:sidmap = /etc/samba/sidmap.tdb
> gpfs:winattr = yes
> store dos attributes = yes
> map readonly = yes
> map archive = yes
> map system = yes
> map hidden = yes
> ea support = yes
> dmapi support = no
> unix extensions = no
> socket options = TCP_NODELAY SO_KEEPALIVE TCP_KEEPCNT=4
> TCP_KEEPIDLE=240 TCP_KEEPINTVL=15
> strict allocate = yes
> gpfs:prealloc = yes
>
>
> if you don't configure using the registry you need to maintain the
> smb.conf file on all your nodes and i am not diving into how to setup
> the registry, but for the people who care, michael adam's presentation
> is a good starting point
> _http://www.samba.org/~obnox/presentations/linux-kongress-2008/lk2008-obnox.pdf_
> <http://www.samba.org/%7Eobnox/presentations/linux-kongress-2008/lk2008-obnox.pdf>
>
> also depending on your authentication/idmap setup , there are multiple
> changes/additions that needs to be made.
>
> on the gpfs side you should set the following config parameters :
>
> cifsBypassShareLocksOnRename=yes
> syncSambaMetadataOps=yes
> cifsBypassTraversalChecking=yes
> allowSambaCaseInsensitiveLookup=no
>
> your filesystem should have the following settings configured :
>
> -D nfs4 File locking semantics in effect
> -k nfs4 ACL semantics in effect
> -o nfssync Additional mount options
> --fastea Yes Fast external attributes enabled?
> -S relatime Suppress atime mount option
>
> the -S is a performance optimization, but you need to check if your
> backup/av or other software can deal with it, it essentially reduces
> the frequency of atime updates to once every 24 hours which is the new
> default on nfs mounts and others as well.
>
> a lot of the configuration parameters above are also above and beyond
> locking and data integrity, they are for better windows compatibility
> and should be checked if applicable for your environment.
>
> i would also recommend to run on GPFS 3.5 TL3 or newer to get the
> proper GPFS level of fixes for this type of configurations.
>
> i would like to repeat that i don't write this email to encourage
> people to all go start installing/ configuring samba on top of GPFS as
> i pointed out that you are kind on your own unless you can read source
> code and/or have somebody who does and is able to help as soon as you
> run into a problem.
> the main point of sharing this information is to clarify a lot of
> misinformation out there and provide the people who already have
> setups that are incorrect the information to at least not run into
> data corruption issues do to wrong configuration.
>
> Sven
>
>
>
>
> From: Chair <chair at gpfsug.org>
> To: gpfsug-discuss at gpfsug.org
> Date: 12/16/2013 03:31 AM
> Subject: Re: [gpfsug-discuss] GPFS and both Samba and NFS
> Sent by: gpfsug-discuss-bounces at gpfsug.org
> ------------------------------------------------------------------------
>
>
>
> Allo
>
> Just jumping in here a minute:
>
> > It is unworthy of an IBM employee to spread such inaccurate
> misinformation.
>
> Whilst this may be inaccurate - I very, very, much doubt that IBM or
> their employees have a secret master plan to spread misinformation (!)
> In the spirit of this group, let's work together to technically look at
> such issues.
>
> Sven, if that is the case, perhaps you could crib the lines of code /
> show your methodology that supports your views / experience.
>
>
> Regards,
>
> Jez
> --
> UG Chair
>
>
>
> On 16/12/13 11:21, Jonathan Buzzard wrote:
> > On Fri, 2013-12-13 at 10:14 -0800, Sven Oehme wrote:
> >
> > [SNIP]
> >
> >> the only way to get something working (don't get confused with
> >> officially Supported) is to recompile the CTDB src packages AND the
> >> Samba src packages on a node that has GPFS already installed. also the
> >> inclusion of CTDB into Samba will not address this, its just a more
> >> convenient packaging.
> >>
> >> Only if the build happens on such a node things like the vfs modules
> >> for GPFS are build and included in the package.
> >>
> > That is a factually inaccurate statement. There is nothing in CTDB that
> > is GPFS specific. Trust me I have examined the code closely to determine
> > if this is the case. So unless this has changed recently you are flat
> > out wrong.
> >
> > Consequently there is no requirement whatsoever to rebuild CTDB to get
> > the vfs_gpfs module. In addition there is also no requirement to
> > actually have GPFS installed to build the vfs_gpfs module either. What
> > you need to have is the GPFS GPL header files and nothing else. As it is
> > a loadable VFS module linking takes place at load time not compile time.
> >
> > It is unworthy of an IBM employee to spread such inaccurate
> > misinformation.
> >
> > [SNIP]
> >
> >> said all this the binaries alone are only part of the Solution, after
> >> you have the correct packages, you need to properly configuration the
> >> system and setting all the right options (on GPFS as well as on CTDB
> >> and smbd.conf), which unfortunate are very System configuration
> >> specific, as otherwise you still can end up with data corruption if
> >> not set right.
> > Indeed. However I know not only what those options are, but also what
> > they do despite IBM's refusal to tell us anything about them.
> >
> > I would also point out that there are sites that where running Samba on
> > top of GPFS for many years before IBM began offering their
> > SONAS/Storwize Unifed products.
> >
> >
> > JAB.
> >
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131216/19e7bfde/attachment-0003.htm>
More information about the gpfsug-discuss
mailing list