<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hi Sven,<br>
<br>
Many thanks for taking the time to write back in detail.<br>
<br>
This is exactly the sort of discussion the user group is aimed at
- I.E. technical discussion outside of the GPFS Developer Forum.<br>
I heartily encourage other IBMers to get involved.<br>
<br>
Regards,<br>
<br>
Jez<br>
<br>
On 16/12/13 15:31, Sven Oehme wrote:<br>
</div>
<blockquote
cite="mid:OFAB1BB05D.C824E4FE-ON88257C43.004E4293-88257C43.00555132@us.ibm.com"
type="cite"><font face="Arial" size="2">Jez, </font>
<br>
<br>
<font face="Arial" size="2">the other replies to my email where
kind
of unexpected as i was indeed intending to help, i will continue
to provide
details and try to answer serious questions and comments. </font>
<br>
<br>
<font face="Arial" size="2">also as you point out there is no
secret
master plan to spread misinformation, the topic is simply
complicated and
there are reasons that are hard to explain why there is no full
official
support outside the current shipping IBM products and i will not
dive into
discussions around it. </font>
<br>
<br>
<font face="Arial" size="2">a few words of clarification, on the
controversial
point, compile ctdb yes/no. </font>
<br>
<font face="Arial" size="2">while its technically correct, that
you don't
have to recompile ctdb and it has no code that links/depends in
any form
on GPFS (samba does), you still have to compile one if the one
shipped
with your distro is out of date and while i didn't explicitly
wrote it
this way, this is what i intended to say. </font>
<br>
<br>
<font face="Arial" size="2">the version RH ships is very old and
since
has gotten many fixes which i highly recommend people to use
unless the
distros will pickup more recent version like RHEL7 will do. i
probably
wouldn't recommend a daily git build, but rather the packaged
versions
that are hosted on <a class="moz-txt-link-abbreviated" href="ftp://ftp.samba.org">ftp.samba.org</a> like : </font><a
moz-do-not-send="true"
href="https://ftp.samba.org/pub/ctdb/packages/redhat/RHEL6/"><font
color="blue" face="Arial" size="2"><u>https://ftp.samba.org/pub/ctdb/packages/redhat/RHEL6/</u></font></a><font
face="Arial" size="2">
</font>
<br>
<br>
<font face="Arial" size="2">so the proposed order of things would
be
to install gpfs, pull the src package of ctdb, compile and
install ctdb
and the devel packages, then pull a recent samba src package ,
install
all the dependencies and build samba on this same host with gpfs
and ctdb
packages already installed. the resulting rpm's should contain
the proper
code to continue.</font>
<br>
<br>
<font face="Arial" size="2">after you got your versions compiled,
installed
and the basics of ctdb running, you should use the following
smb.conf as
a starting point : </font>
<br>
<br>
<font face="Arial" size="2">[global]</font>
<br>
<font face="Arial" size="2"> netbios
name = cluster</font>
<br>
<font face="Arial" size="2"> fileid:mapping
= fsname</font>
<br>
<font face="Arial" size="2"> gpfs:sharemodes
= yes</font>
<br>
<font face="Arial" size="2"> gpfs:leases
= yes</font>
<br>
<font face="Arial" size="2"> gpfs:dfreequota
= yes</font>
<br>
<font face="Arial" size="2"> gpfs:hsm
= yes</font>
<br>
<font face="Arial" size="2"> syncops:onmeta
= no</font>
<br>
<font face="Arial" size="2"> kernel
oplocks = no</font>
<br>
<font face="Arial" size="2"> level2
oplocks = yes</font>
<br>
<font face="Arial" size="2"> notify:inotify
= no</font>
<br>
<font face="Arial" size="2"> vfs objects
= shadow_copy2 syncops gpfs fileid</font>
<br>
<font face="Arial" size="2"> shadow:snapdir
= .snapshots</font>
<br>
<font face="Arial" size="2"> shadow:fixinodes
= yes</font>
<br>
<font face="Arial" size="2"> shadow:snapdirseverywhere
= yes</font>
<br>
<font face="Arial" size="2"> shadow:sort
= desc</font>
<br>
<font face="Arial" size="2"> wide
links = no</font>
<br>
<font face="Arial" size="2"> async
smb echo handler = yes</font>
<br>
<font face="Arial" size="2"> smbd:backgroundqueue
= False</font>
<br>
<font face="Arial" size="2"> use sendfile
= no</font>
<br>
<font face="Arial" size="2"> strict
locking = yes</font>
<br>
<font face="Arial" size="2"> posix
locking = yes</font>
<br>
<font face="Arial" size="2"> force
unknown acl user = yes</font>
<br>
<font face="Arial" size="2"> nfs4:mode
= simple</font>
<br>
<font face="Arial" size="2"> nfs4:chown
= yes</font>
<br>
<font face="Arial" size="2"> nfs4:acedup
= merge</font>
<br>
<font face="Arial" size="2"> nfs4:sidmap
= /etc/samba/sidmap.tdb</font>
<br>
<font face="Arial" size="2"> gpfs:winattr
= yes</font>
<br>
<font face="Arial" size="2"> store
dos attributes = yes</font>
<br>
<font face="Arial" size="2"> map readonly
= yes</font>
<br>
<font face="Arial" size="2"> map archive
= yes</font>
<br>
<font face="Arial" size="2"> map system
= yes</font>
<br>
<font face="Arial" size="2"> map hidden
= yes</font>
<br>
<font face="Arial" size="2"> ea support
= yes</font>
<br>
<font face="Arial" size="2"> dmapi
support = no</font>
<br>
<font face="Arial" size="2"> unix
extensions = no</font>
<br>
<font face="Arial" size="2"> socket
options = TCP_NODELAY SO_KEEPALIVE TCP_KEEPCNT=4
TCP_KEEPIDLE=240 TCP_KEEPINTVL=15</font>
<br>
<font face="Arial" size="2"> strict
allocate = yes</font>
<br>
<font face="Arial" size="2"> gpfs:prealloc
= yes</font>
<br>
<br>
<br>
<font face="Arial" size="2">if you don't configure using the
registry
you need to maintain the smb.conf file on all your nodes and i
am not diving
into how to setup the registry, but for the people who care,
michael adam's
presentation is a good starting point </font><a
moz-do-not-send="true"
href="http://www.samba.org/%7Eobnox/presentations/linux-kongress-2008/lk2008-obnox.pdf"><font
color="blue" face="Arial" size="2"><u>http://www.samba.org/~obnox/presentations/linux-kongress-2008/lk2008-obnox.pdf</u></font></a><font
face="Arial" size="2">
</font>
<br>
<br>
<font face="Arial" size="2">also depending on your
authentication/idmap
setup , there are multiple changes/additions that needs to be
made.</font>
<br>
<br>
<font face="Arial" size="2">on the gpfs side you should set the
following
config parameters : </font>
<br>
<br>
<font face="Arial" size="2">cifsBypassShareLocksOnRename=yes</font>
<br>
<font face="Arial" size="2">syncSambaMetadataOps=yes</font>
<br>
<font face="Arial" size="2">cifsBypassTraversalChecking=yes</font>
<br>
<font face="Arial" size="2">allowSambaCaseInsensitiveLookup=no</font>
<br>
<br>
<font face="Arial" size="2">your filesystem should have the
following
settings configured :</font>
<br>
<br>
<font face="Arial" size="2"> -D
nfs4
File locking semantics in effect</font>
<br>
<font face="Arial" size="2"> -k
nfs4
ACL semantics in effect</font>
<br>
<font face="Arial" size="2"> -o
nfssync
Additional mount options</font>
<br>
<font face="Arial" size="2"> --fastea
Yes
Fast external attributes enabled?</font>
<br>
<font face="Arial" size="2"> -S
relatime
Suppress atime mount option</font>
<br>
<br>
<font face="Arial" size="2">the -S is a performance optimization,
but
you need to check if your backup/av or other software can deal
with it,
it essentially reduces the frequency of atime updates to once
every 24
hours which is the new default on nfs mounts and others as well.
</font>
<br>
<br>
<font face="Arial" size="2">a lot of the configuration parameters
above
are also above and beyond locking and data integrity, they are
for better
windows compatibility and should be checked if applicable for
your environment.</font>
<br>
<br>
<font face="Arial" size="2">i would also recommend to run on GPFS
3.5
TL3 or newer to get the proper GPFS level of fixes for this type
of configurations.
</font>
<br>
<br>
<font face="Arial" size="2">i would like to repeat that i don't
write
this email to encourage people to all go start installing/
configuring
samba on top of GPFS as i pointed out that you are kind on your
own unless
you can read source code and/or have somebody who does and is
able to help
as soon as you run into a problem. </font>
<br>
<font face="Arial" size="2">the main point of sharing this
information
is to clarify a lot of misinformation out there and provide the
people
who already have setups that are incorrect the information to at
least
not run into data corruption issues do to wrong configuration. </font>
<br>
<br>
<font face="Arial" size="2">Sven</font>
<br>
<br>
<br>
<br>
<br>
<font color="#5f5f5f" face="Arial" size="2">From:
</font><font face="Arial" size="2">Chair
<a class="moz-txt-link-rfc2396E" href="mailto:chair@gpfsug.org"><chair@gpfsug.org></a></font>
<br>
<font color="#5f5f5f" face="Arial" size="2">To: </font><font
face="Arial" size="2"><a class="moz-txt-link-abbreviated" href="mailto:gpfsug-discuss@gpfsug.org">gpfsug-discuss@gpfsug.org</a></font>
<br>
<font color="#5f5f5f" face="Arial" size="2">Date:
</font><font face="Arial" size="2">12/16/2013 03:31 AM</font>
<br>
<font color="#5f5f5f" face="Arial" size="2">Subject:
</font><font face="Arial" size="2">Re: [gpfsug-discuss] GPFS
and both Samba and NFS</font>
<br>
<font color="#5f5f5f" face="Arial" size="2">Sent by:
</font><font face="Arial" size="2"><a class="moz-txt-link-abbreviated" href="mailto:gpfsug-discuss-bounces@gpfsug.org">gpfsug-discuss-bounces@gpfsug.org</a></font>
<br>
<hr noshade="noshade">
<br>
<br>
<br>
<font face="Arial" size="2">Allo<br>
<br>
Just jumping in here a minute:<br>
<br>
> It is unworthy of an IBM employee to spread such inaccurate
misinformation.<br>
<br>
Whilst this may be inaccurate - I very, very, much doubt that
IBM or <br>
their employees have a secret master plan to spread
misinformation (!)<br>
In the spirit of this group, let's work together to technically
look at
<br>
such issues.<br>
<br>
Sven, if that is the case, perhaps you could crib the lines of
code / <br>
show your methodology that supports your views / experience.<br>
<br>
<br>
Regards,<br>
<br>
Jez<br>
--<br>
UG Chair<br>
<br>
<br>
<br>
On 16/12/13 11:21, Jonathan Buzzard wrote:<br>
> On Fri, 2013-12-13 at 10:14 -0800, Sven Oehme wrote:<br>
><br>
> [SNIP]<br>
><br>
>> the only way to get something working (don't get
confused with<br>
>> officially Supported) is to recompile the CTDB src
packages AND
the<br>
>> Samba src packages on a node that has GPFS already
installed.
also the<br>
>> inclusion of CTDB into Samba will not address this, its
just a
more<br>
>> convenient packaging.<br>
>><br>
>> Only if the build happens on such a node things like
the vfs modules<br>
>> for GPFS are build and included in the package.<br>
>><br>
> That is a factually inaccurate statement. There is nothing
in CTDB
that<br>
> is GPFS specific. Trust me I have examined the code closely
to determine<br>
> if this is the case. So unless this has changed recently
you are flat<br>
> out wrong.<br>
><br>
> Consequently there is no requirement whatsoever to rebuild
CTDB to
get<br>
> the vfs_gpfs module. In addition there is also no
requirement to<br>
> actually have GPFS installed to build the vfs_gpfs module
either.
What<br>
> you need to have is the GPFS GPL header files and nothing
else. As
it is<br>
> a loadable VFS module linking takes place at load time not
compile
time.<br>
><br>
> It is unworthy of an IBM employee to spread such inaccurate<br>
> misinformation.<br>
><br>
> [SNIP]<br>
><br>
>> said all this the binaries alone are only part of the
Solution,
after<br>
>> you have the correct packages, you need to properly
configuration
the<br>
>> system and setting all the right options (on GPFS as
well as on
CTDB<br>
>> and smbd.conf), which unfortunate are very System
configuration<br>
>> specific, as otherwise you still can end up with data
corruption
if<br>
>> not set right.<br>
> Indeed. However I know not only what those options are, but
also what<br>
> they do despite IBM's refusal to tell us anything about
them.<br>
><br>
> I would also point out that there are sites that where
running Samba
on<br>
> top of GPFS for many years before IBM began offering their<br>
> SONAS/Storwize Unifed products.<br>
><br>
><br>
> JAB.<br>
><br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at gpfsug.org<br>
</font><a moz-do-not-send="true"
href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><font
face="Arial" size="2">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></a><font
face="Arial" size="2"><br>
<br>
</font>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
<a class="moz-txt-link-freetext" href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a>
</pre>
</blockquote>
<br>
</body>
</html>