From sabujp at gmail.com  Thu Mar 13 00:07:22 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Wed, 12 Mar 2014 19:07:22 -0500
Subject: [gpfsug-discuss] question about why unix extensions = no is
 recommended when using samba + gpfs?
Message-ID: <CAEeMGHvH84GDv6yEnPgKtNwOhyvXLkbvV2XGtf3vHpJgSuu=Lg@mail.gmail.com>

Hi all,

I was wondering why here :

https://www.mail-archive.com/gpfsug-discuss at gpfsug.org/msg00121.html

...and several other forums, wikis, etc it's recommended to use :

unix extensions = no

for samba setups with GPFS? This disables the ability for linux/unix samba
clients to see the actual mode bits on files.

Thanks,
Sabuj
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140312/e07bdc29/attachment.htm>

From jonathan at buzzard.me.uk  Thu Mar 13 10:09:21 2014
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Thu, 13 Mar 2014 10:09:21 +0000
Subject: [gpfsug-discuss] question about why unix extensions = no is
 recommended when using samba + gpfs?
In-Reply-To: <CAEeMGHvH84GDv6yEnPgKtNwOhyvXLkbvV2XGtf3vHpJgSuu=Lg@mail.gmail.com>
References: <CAEeMGHvH84GDv6yEnPgKtNwOhyvXLkbvV2XGtf3vHpJgSuu=Lg@mail.gmail.com>
Message-ID: <1394705361.16270.9.camel@buzzard.phy.strath.ac.uk>

On Wed, 2014-03-12 at 19:07 -0500, Sabuj Pattanayek wrote:
> Hi all,
> 
> 
> I was wondering why here :
> 
> 
> https://www.mail-archive.com/gpfsug-discuss at gpfsug.org/msg00121.html
> 
> 
> ...and several other forums, wikis, etc it's recommended to use :
> 
> 
> unix extensions = no 
> 
> 
> for samba setups with GPFS? This disables the ability for linux/unix
> samba clients to see the actual mode bits on files.
> 

Because it messes horribly with NFSv4 ACL's, and MacOSX clients in
particular do "bad things" using Unix extensions that break group
shares. Therefore unless you absolutely need it which most people don't
then disabling it is a sensible choice to avoid wasting hours of your
time trying to work out why everything is broken.


JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.



From sabujp at gmail.com  Thu Mar 13 12:45:01 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Thu, 13 Mar 2014 07:45:01 -0500
Subject: [gpfsug-discuss] question about why unix extensions = no is
 recommended when using samba + gpfs?
In-Reply-To: <1394705361.16270.9.camel@buzzard.phy.strath.ac.uk>
References: <CAEeMGHvH84GDv6yEnPgKtNwOhyvXLkbvV2XGtf3vHpJgSuu=Lg@mail.gmail.com>
	<1394705361.16270.9.camel@buzzard.phy.strath.ac.uk>
Message-ID: <CAEeMGHu4EaiEqa6=qJjHL+4GnkOnQeqh+h5azTHB-1QuqUiTMw@mail.gmail.com>

Hi,

We tried the nfsv4 acl route and it didn't work out so well for us when
files/directories would get promoted to nfsv4 acl's (but maybe I'll revisit
it when I get a chance), I had unix extensions turned off at that time.
We're using for our main template share :

vfs objects = shadow_copy2 gpfs acl_xattr fileid
gpfs:acl = no

to pass acl's to acl_xattr and it seems to work ok and tivoli is able to
backup the security.NTACL extended attribute and restore it without
problems. It'll end up using posix ACLs assigning default acl's for the
users/groups that are assigned to the files/dirs . All of it breaks umask
and other things though, which isn't that big of a deal with samba's
ability to force modes for particular shares.

Regarding unix extensions, it seems there are problems either way (or
perhaps were?), but the problems may be "more" severe if unix extensions
are turned off?

http://wiki.phys.ethz.ch/readme/mac_samba

I'll need to do some more testing with the latest OSX in that case since it
looks like many of these posts were written years ago. We are also running
the latest stable samba 4.1.x from sernet. But it's good to know that unix
extensions = no is not because of some requirement in GPFS.

Thanks,
Sabuj





On Thu, Mar 13, 2014 at 5:09 AM, Jonathan Buzzard <jonathan at buzzard.me.uk>wrote:

> On Wed, 2014-03-12 at 19:07 -0500, Sabuj Pattanayek wrote:
> > Hi all,
> >
> >
> > I was wondering why here :
> >
> >
> > https://www.mail-archive.com/gpfsug-discuss at gpfsug.org/msg00121.html
> >
> >
> > ...and several other forums, wikis, etc it's recommended to use :
> >
> >
> > unix extensions = no
> >
> >
> > for samba setups with GPFS? This disables the ability for linux/unix
> > samba clients to see the actual mode bits on files.
> >
>
> Because it messes horribly with NFSv4 ACL's, and MacOSX clients in
> particular do "bad things" using Unix extensions that break group
> shares. Therefore unless you absolutely need it which most people don't
> then disabling it is a sensible choice to avoid wasting hours of your
> time trying to work out why everything is broken.
>
>
> JAB.
>
> --
> Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
> Fife, United Kingdom.
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140313/31f7efc9/attachment.htm>

From jonathan at buzzard.me.uk  Thu Mar 13 13:08:51 2014
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Thu, 13 Mar 2014 13:08:51 +0000
Subject: [gpfsug-discuss] question about why unix extensions = no is
 recommended when using samba + gpfs?
In-Reply-To: <CAEeMGHu4EaiEqa6=qJjHL+4GnkOnQeqh+h5azTHB-1QuqUiTMw@mail.gmail.com>
References: <CAEeMGHvH84GDv6yEnPgKtNwOhyvXLkbvV2XGtf3vHpJgSuu=Lg@mail.gmail.com>
	<1394705361.16270.9.camel@buzzard.phy.strath.ac.uk>
	<CAEeMGHu4EaiEqa6=qJjHL+4GnkOnQeqh+h5azTHB-1QuqUiTMw@mail.gmail.com>
Message-ID: <1394716131.2612.24.camel@buzzard.phy.strath.ac.uk>

On Thu, 2014-03-13 at 07:45 -0500, Sabuj Pattanayek wrote:
> Hi,
> 
> 
> We tried the nfsv4 acl route and it didn't work out so well for us
> when files/directories would get promoted to nfsv4 acl's (but maybe
> I'll revisit it when I get a chance), I had unix extensions turned off
> at that time. We're using for our main template share :
> 
> 
> vfs objects = shadow_copy2 gpfs acl_xattr fileid
> 
> gpfs:acl = no
> 

That is an "unsual" way to run an Samba/GPFS file server so don't expect
much help from anyone. The vast majority use GPFS in NFSv4 ACL only mode
with the Samba GPFS VFS module doing all the mapping through to the
NFSv4 ACLs to do all the rich permission.

> 
> to pass acl's to acl_xattr and it seems to work ok and tivoli is able
> to backup the security.NTACL extended attribute and restore it without
> problems. It'll end up using posix ACLs assigning default acl's for
> the users/groups that are assigned to the files/dirs . All of it
> breaks umask and other things though, which isn't that big of a deal
> with samba's ability to force modes for particular shares.
> 

That is a "wacked out" setup if you ask me. 

> 
> Regarding unix extensions, it seems there are problems either way (or
> perhaps were?), but the problems may be "more" severe if unix
> extensions are turned off?
> 
> http://wiki.phys.ethz.ch/readme/mac_samba
> 

Yeah changing Unix permissions on a SMB share does not work. Rather like
a real Windows file server don't you think?

Besides which on a group share with Unix extensions on the MacOS X
client (or at least did) goes and changes the permission and ownership
on the file and breaks a "group" share. That is one where a group of
people have equal read/write access to a shared area. Working group
shares are a million times more important that the odd power user being
able to muck about with file permissions.

Note this only effects MacOS because Linux has the tools to see and
manipulate SMB ACL's properly.

> 
> I'll need to do some more testing with the latest OSX in that case
> since it looks like many of these posts were written years ago. We are
> also running the latest stable samba 4.1.x from sernet. But it's good
> to know that unix extensions = no is not because of some requirement
> in GPFS.
> 

It also does not play well with case insensitive = yes in my experience.


JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.



From chair at gpfsug.org  Tue Mar 18 15:43:50 2014
From: chair at gpfsug.org (Jez Tucker (Chair))
Date: Tue, 18 Mar 2014 15:43:50 +0000
Subject: [gpfsug-discuss] GPFS UG10 - Finalised Agenda - 29th April 2014
Message-ID: <532869B6.90706@gpfsug.org>

Hello all,

   We've finalised the agenda for this year's GPFS User Group.

The day encompasses technical presentations (no sales) on the new 
version of GPFS and tweakage.
Come along and meet the GPFS developers, users and technical pros.
Waffle, discuss, opinionate, eat, drink.

The event will be held at: IBM Southbank Client Centre, London, UK 
<http://www-05.ibm.com/uk/clientcentre/about-us.html>
It is free to attend.

Please register via email to secretary at gpfsug.org 
<mailto:secretary at gpfsug.org>
*
*More updates will be provided on http://www.gpfsug.org*

*See you there.

Jez

*
GPFS User Group Meeting: 29th April 2014**
**Agenda**

**10:30 Arrivals and refreshments**

**11:00 Introductions and committee updates*
     Jez Tucker, Group Chair & Claire Robson, Group Secretary

*11:05 GPFS 4.1*
     Scott Fadden, IBM GPFS Technical Marketing

*12:15 GPFS Performance (Basics)*
     Sven Oehme, IBM Scalable Storage Research

*13:00 Lunch**(provided)

**13:45 GPFS Performance (Advanced) *
     Sven Oehme, IBM Scalable Storage Research

*14:45 Refreshments break**

**15:05 Numascale*

*15:35 User Case Study*

*16:00 Group discussion: Questions & Committee matters*
     Led by Jez Tucker, Group Chairperson

*16:05 Close*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140318/66ca96d2/attachment.htm>

From gtsai at slac.stanford.edu  Thu Mar 20 22:21:58 2014
From: gtsai at slac.stanford.edu (Grace Tsai)
Date: Thu, 20 Mar 2014 15:21:58 -0700
Subject: [gpfsug-discuss] Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?
Message-ID: <532B6A06.3000207@slac.stanford.edu>

Hi,

Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?

Thanks.

Grace


From sabujp at gmail.com  Fri Mar 21 01:38:33 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Thu, 20 Mar 2014 20:38:33 -0500
Subject: [gpfsug-discuss] Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?
In-Reply-To: <532B6A06.3000207@slac.stanford.edu>
References: <532B6A06.3000207@slac.stanford.edu>
Message-ID: <CAEeMGHtFHY=8oJu-K2kQopK8LEAUB9DFTL6i++DrszOvvh3N1A@mail.gmail.com>

We're using tsm 7.1 with gpfs 3.5.0.11. At some point we do want to enable
the HSM features but haven't had time to properly configure/set them up
yet. I had dmapi enabled on GPFS but was never able to bring it up with
dmapi enabled. Everything wasn't properly configured at the time and we
were missing some pieces (not my post, but same issue) :

https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014622591

I'd say that we are having less than optimal performance with TSM however.
We're only able to pull about 4TB a day. It took us 20 days to backup 80TB
for our initial full. Using rsync/tar piped to tar would probably have
taken less than a week. We tried various methods, e.g. using a fast
intermediate diskpool, going simultaneously to our 6 LTO6 tape drives, etc
but each "cp" (tsm client) process that TSM would use seemed to be very
slow. We tweaked just about every setting to optimize performance but to
really no avail. When going to the disk pool this is what should have
happened :

GPFS => relatively fast random I/O (on par with rsync/tar piped to tar) tsm
disk cache => large sequential I/O's for each disk pool volume => tape

this is what really happened

GPFS => slow random I/O => tsm disk pool cache => slow random I/O => tape

so instead we did :

GPFS => slow random I/O (TSM) => tape

..but was the same speed as going through the tsm disk pool cache. We
closely monitored the network, disk, memory, cpu, on the tsm server and
none of the hardware or capabilities of the server were the bottleneck, it
was all in TSM.

If anyone has seen this sort of behavior and has some pointers/hints at
improving performance I'd be glad to hear it.

Thanks,
Sabuj


On Thu, Mar 20, 2014 at 5:21 PM, Grace Tsai <gtsai at slac.stanford.edu> wrote:

> Hi,
>
> Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?
>
> Thanks.
>
> Grace
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140320/83143bf7/attachment.htm>

From oehmes at us.ibm.com  Fri Mar 21 21:39:57 2014
From: oehmes at us.ibm.com (Sven Oehme)
Date: Fri, 21 Mar 2014 14:39:57 -0700
Subject: [gpfsug-discuss] Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?
In-Reply-To: <CAEeMGHtFHY=8oJu-K2kQopK8LEAUB9DFTL6i++DrszOvvh3N1A@mail.gmail.com>
References: <532B6A06.3000207@slac.stanford.edu>
	<CAEeMGHtFHY=8oJu-K2kQopK8LEAUB9DFTL6i++DrszOvvh3N1A@mail.gmail.com>
Message-ID: <OF014D6BE3.885B54C0-ON88257CA2.0070C7EC-88257CA2.0077038F@us.ibm.com>

Hi,

there are various ways to speed up backup of data from GPFS to TSM (which 
btw is the same problem for most other backup solutions as well), but 
first one needs to find out what the problem is as it can be multiple 
completely independent issues and each of them needs a different solution, 
so let me explain the different issues and also what you can do about 
them.

the very first challenge is to find what data has changed. the way TSM 
does this is by crawling trough your filesystem, looking at mtime on each 
file to find out which file has changed. think about a ls -Rl on your 
filesystem root. this can, depending on how many files you have, take days 
in a large scale environment (think 100's of millions  of files). there is 
very little one can speed up on this process the way its done. all you can 
do is put metadata on faster disks (e.g. SSD) and that will improve the 
speed of this 'scan phase'. an alternative is to not do this scan at all 
with the TSM client, but instead let GPFS find out for TSM what files have 
changed and then share this information with TSM . the function/command in 
GPFS to do so is called mmbackup : 

https://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=%2Fcom.ibm.cluster.gpfs.v3r5.gpfs100.doc%2Fbl1adm_backupusingmmbackup.htm

it essentially traverses the GPFS metadata sequentially and in parallel 
across all nodes and filters out files that needs to be backed up. 
in several customer environments i was called to assist with issues like 
this, this change alone did speed up the backup process multiple orders of 
magnitudes. we had a few customers where this change reduced the scan time 
from days down to minutes. its not always this big, but its usually the 
largest chunk of the issue. 

the 2nd challenge is if you have to backup a very large number (millions) 
of very small (<32k) files. 
the main issue here is that for each file TSM issues a random i/o to GPFS, 
one at a time, so your throughput directly correlates with size of the 
files and latency for a single file read operation. if you are not on 3.5 
TL3 and/or your files don't fit into the inode its actually even 2 random 
i/os that are issued as you need to read the metadata followed by the data 
block for the file. 
in this scenario you can only do 2 things : 

1. parallelism - mmbackup again starts multiple processes in parallel to 
speed up this phase of the backup 
2. use a 'helper' process to prefetch data for a single TSM client so all 
data comes out of cache and the latency for the random reads is eliminated 
to increase throughput.

without any of this seeing only a few low MB/sec is not uncommon for 
customers, but with the changes above you are able to backup very large 
quantities of data. 

hope this helps. Sven

------------------------------------------
Sven Oehme 
Scalable Storage Research 
email: oehmes at us.ibm.com 
IBM Almaden Research Lab 
------------------------------------------



From:   Sabuj Pattanayek <sabujp at gmail.com>
To:     gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Date:   03/20/2014 06:39 PM
Subject:        Re: [gpfsug-discuss] Is TSM/HSM 7.1 compatible with GPFS 
3.5.0.12 ?
Sent by:        gpfsug-discuss-bounces at gpfsug.org



We're using tsm 7.1 with gpfs 3.5.0.11. At some point we do want to enable 
the HSM features but haven't had time to properly configure/set them up 
yet. I had dmapi enabled on GPFS but was never able to bring it up with 
dmapi enabled. Everything wasn't properly configured at the time and we 
were missing some pieces (not my post, but same issue) :

https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014622591

I'd say that we are having less than optimal performance with TSM however. 
We're only able to pull about 4TB a day. It took us 20 days to backup 80TB 
for our initial full. Using rsync/tar piped to tar would probably have 
taken less than a week. We tried various methods, e.g. using a fast 
intermediate diskpool, going simultaneously to our 6 LTO6 tape drives, etc 
but each "cp" (tsm client) process that TSM would use seemed to be very 
slow. We tweaked just about every setting to optimize performance but to 
really no avail. When going to the disk pool this is what should have 
happened :

GPFS => relatively fast random I/O (on par with rsync/tar piped to tar) 
tsm disk cache => large sequential I/O's for each disk pool volume => tape

this is what really happened

GPFS => slow random I/O => tsm disk pool cache => slow random I/O => tape

so instead we did :

GPFS => slow random I/O (TSM) => tape

..but was the same speed as going through the tsm disk pool cache. We 
closely monitored the network, disk, memory, cpu, on the tsm server and 
none of the hardware or capabilities of the server were the bottleneck, it 
was all in TSM.

If anyone has seen this sort of behavior and has some pointers/hints at 
improving performance I'd be glad to hear it.

Thanks,
Sabuj


On Thu, Mar 20, 2014 at 5:21 PM, Grace Tsai <gtsai at slac.stanford.edu> 
wrote:
Hi,

Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?

Thanks.

Grace
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140321/d1581e61/attachment.htm>

From sabujp at gmail.com  Fri Mar 21 22:16:10 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Fri, 21 Mar 2014 17:16:10 -0500
Subject: [gpfsug-discuss] Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?
In-Reply-To: <OF014D6BE3.885B54C0-ON88257CA2.0070C7EC-88257CA2.0077038F@us.ibm.com>
References: <532B6A06.3000207@slac.stanford.edu>
	<CAEeMGHtFHY=8oJu-K2kQopK8LEAUB9DFTL6i++DrszOvvh3N1A@mail.gmail.com>
	<OF014D6BE3.885B54C0-ON88257CA2.0070C7EC-88257CA2.0077038F@us.ibm.com>
Message-ID: <CAEeMGHt63xJFLsvzDk46jZxcmH_9=E+4twtcKJnGzUr6mGPvpQ@mail.gmail.com>

> the very first challenge is to find what data has changed. the way TSM
> does this is by crawling trough your filesystem, looking at mtime on each
> file to find out which file has changed. think about a ls -Rl on your
> filesystem root. this


The mmbackup mmapplypolicy phase is not a problem. It's going to take as
long as it's going to take. We're using 5 RAID1 SAS SSD NSD's for metadata
and it takes ~1 hour to do the traversal through 157 million files.


>
> the 2nd challenge is if you have to backup a very large number (millions)
> of very small (<32k) files.
> the main issue here is that for each file TSM issues a random i/o to GPFS,
> one at a time, so your throughput directly correlates with size of the
> files and latency for a single file read operation. if you are not on 3.5
> TL3 and/or your files don't fit into the inode its actually even 2 random
> i/os that are issued as you need to read the metadata followed by the data
> block for the file.
> in this scenario you can only do 2 things :


The problem here is why is a single rsync or tar | tar process orders of
magnitude faster than a single tsm client at pulling data off of GPFS into
the same backup system's disk (e.g. disk pool)? It's not a problem with
GPFS, it's a problem with TSM itself. We tried various things, e.g. :

1) changed commmethod to sharedmem
2) increase txnbytelimit to 10G
3) increased movesizethresh to the same as txnbytelimit (10G)
4) increase diskbufsize to 1023kb
5) increased txngroupmax to 65000
6) increased movesizethresh to 10240

the next problem is that one would expect backups to tape to do straight
sequential I/O to tape, in the case of putting the files to the disk pool
before moving them to tape, it did the same random I/O to tape even with
8GB disk pool chunks. We haven't tried the file pool option yet, but we've
been told that it'll do the same thing. If I'm tar'ing or dd'ing large
files to tape that's the most efficient, why doesn't TSM do something
similar?


> 1. parallelism - mmbackup again starts multiple processes in parallel to
> speed up this phase of the backup


..use multiple clients. This would help, but again I'm trying to get a
single tsm client to be on par with a single "cp" process.

Thanks,
Sabuj
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140321/9a73f570/attachment.htm>

From sabujp at gmail.com  Fri Mar 21 22:19:38 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Fri, 21 Mar 2014 17:19:38 -0500
Subject: [gpfsug-discuss] Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?
In-Reply-To: <CAEeMGHt63xJFLsvzDk46jZxcmH_9=E+4twtcKJnGzUr6mGPvpQ@mail.gmail.com>
References: <532B6A06.3000207@slac.stanford.edu>
	<CAEeMGHtFHY=8oJu-K2kQopK8LEAUB9DFTL6i++DrszOvvh3N1A@mail.gmail.com>
	<OF014D6BE3.885B54C0-ON88257CA2.0070C7EC-88257CA2.0077038F@us.ibm.com>
	<CAEeMGHt63xJFLsvzDk46jZxcmH_9=E+4twtcKJnGzUr6mGPvpQ@mail.gmail.com>
Message-ID: <CAEeMGHu3YmKq5F2fGiPDEkxOvSxOTQP-v+kSr7jguXJ0aHS2_g@mail.gmail.com>

It's also not a problem with the TSM database, which is sitting on a RAID10
of 4 SSDs. I'll watch dstat and it'll basically do a large 500MB/s write to
the LUN every 30 mins - hour and then just sit idle waiting for more data.


On Fri, Mar 21, 2014 at 5:16 PM, Sabuj Pattanayek <sabujp at gmail.com> wrote:

>
> the very first challenge is to find what data has changed. the way TSM
>> does this is by crawling trough your filesystem, looking at mtime on each
>> file to find out which file has changed. think about a ls -Rl on your
>> filesystem root. this
>
>
> The mmbackup mmapplypolicy phase is not a problem. It's going to take as
> long as it's going to take. We're using 5 RAID1 SAS SSD NSD's for metadata
> and it takes ~1 hour to do the traversal through 157 million files.
>
>
>>
>> the 2nd challenge is if you have to backup a very large number (millions)
>> of very small (<32k) files.
>> the main issue here is that for each file TSM issues a random i/o to
>> GPFS, one at a time, so your throughput directly correlates with size of
>> the files and latency for a single file read operation. if you are not on
>> 3.5 TL3 and/or your files don't fit into the inode its actually even 2
>> random i/os that are issued as you need to read the metadata followed by
>> the data block for the file.
>> in this scenario you can only do 2 things :
>
>
> The problem here is why is a single rsync or tar | tar process orders of
> magnitude faster than a single tsm client at pulling data off of GPFS into
> the same backup system's disk (e.g. disk pool)? It's not a problem with
> GPFS, it's a problem with TSM itself. We tried various things, e.g. :
>
> 1) changed commmethod to sharedmem
> 2) increase txnbytelimit to 10G
> 3) increased movesizethresh to the same as txnbytelimit (10G)
> 4) increase diskbufsize to 1023kb
> 5) increased txngroupmax to 65000
> 6) increased movesizethresh to 10240
>
> the next problem is that one would expect backups to tape to do straight
> sequential I/O to tape, in the case of putting the files to the disk pool
> before moving them to tape, it did the same random I/O to tape even with
> 8GB disk pool chunks. We haven't tried the file pool option yet, but we've
> been told that it'll do the same thing. If I'm tar'ing or dd'ing large
> files to tape that's the most efficient, why doesn't TSM do something
> similar?
>
>
>> 1. parallelism - mmbackup again starts multiple processes in parallel to
>> speed up this phase of the backup
>
>
> ..use multiple clients. This would help, but again I'm trying to get a
> single tsm client to be on par with a single "cp" process.
>
> Thanks,
> Sabuj
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140321/dcfd38d0/attachment.htm>

From oehmes at us.ibm.com  Fri Mar 21 22:48:50 2014
From: oehmes at us.ibm.com (Sven Oehme)
Date: Fri, 21 Mar 2014 15:48:50 -0700
Subject: [gpfsug-discuss] Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?
In-Reply-To: <CAEeMGHt63xJFLsvzDk46jZxcmH_9=E+4twtcKJnGzUr6mGPvpQ@mail.gmail.com>
References: <532B6A06.3000207@slac.stanford.edu>	<CAEeMGHtFHY=8oJu-K2kQopK8LEAUB9DFTL6i++DrszOvvh3N1A@mail.gmail.com>
	<OF014D6BE3.885B54C0-ON88257CA2.0070C7EC-88257CA2.0077038F@us.ibm.com>
	<CAEeMGHt63xJFLsvzDk46jZxcmH_9=E+4twtcKJnGzUr6mGPvpQ@mail.gmail.com>
Message-ID: <OFC63E3472.65E21691-ON88257CA2.007B27D1-88257CA2.007D51FB@us.ibm.com>

Hi,

> the very first challenge is to find what data has changed. the way 
> TSM does this is by crawling trough your filesystem, looking at 
> mtime on each file to find out which file has changed. think about a
> ls -Rl on your filesystem root. this 
> 
> The mmbackup mmapplypolicy phase is not a problem. It's going to 
> take as long as it's going to take. We're using 5 RAID1 SAS SSD 
> NSD's for metadata and it takes ~1 hour to do the traversal through 
> 157 million files.
>  

that sounds too long, if you want i could take a look at your gpfs config 
and make suggestions on how to further improve this. 
if you want me to do that please send me a email with the output of 
mmlscluster,mmlsconfig,mmlsnsd and mmlsfs all to oehmes at us.ibm.com 

> 
> the 2nd challenge is if you have to backup a very large number 
> (millions) of very small (<32k) files. 
> the main issue here is that for each file TSM issues a random i/o to
> GPFS, one at a time, so your throughput directly correlates with 
> size of the files and latency for a single file read operation. if 
> you are not on 3.5 TL3 and/or your files don't fit into the inode 
> its actually even 2 random i/os that are issued as you need to read 
> the metadata followed by the data block for the file. 
> in this scenario you can only do 2 things : 
> 
> The problem here is why is a single rsync or tar | tar process 
> orders of magnitude faster than a single tsm client at pulling data 
> off of GPFS into the same backup system's disk (e.g. disk pool)? 
> It's not a problem with GPFS, it's a problem with TSM itself. 
> We tried various things, e.g. :
> 
> 1) changed commmethod to sharedmem
> 2) increase txnbytelimit to 10G
> 3) increased movesizethresh to the same as txnbytelimit (10G)
> 4) increase diskbufsize to 1023kb
> 5) increased txngroupmax to 65000
> 6) increased movesizethresh to 10240
> 
> the next problem is that one would expect backups to tape to do 
> straight sequential I/O to tape, in the case of putting the files to
> the disk pool before moving them to tape, it did the same random I/O
> to tape even with 8GB disk pool chunks. We haven't tried the file 
> pool option yet, but we've been told that it'll do the same thing. 

i assume you refer to using raw disks behind TSM ? i never used them, in 
fact the fastest TSM Storage you can build for TSM is to use a GPFS 
filesystem and put files into the filesystem and create a pool with 
sequential file volumes for TSM. if you match the gpfs blocksize with the 
raid blocksize you get very high throughputs to the TSM pool, i have 
customers who see >600 MB/sec backup throughput to a single TSM Server in 
production. 
> If I'm tar'ing or dd'ing large files to tape that's the most 
> efficient, why doesn't TSM do something similar?

i can't tell you much about the tape part of TSM but i can help you speed 
up the ingest into a TSM Server leveraging a Filesystem pool if you want.

> 
> 
> 1. parallelism - mmbackup again starts multiple processes in 
> parallel to speed up this phase of the backup 
> 
> ..use multiple clients. This would help, but again I'm trying to get
> a single tsm client to be on par with a single "cp" process.

cp uses buffered write operation and therefore will not sync the data to 
the disk, which is why its faster. TSM guarantees each write operation to 
to be committed to stable storage before it ever returns to the client, 
this is most likely the difference you see. 

don't get me wrong, i don't try to defend the way TSM works, i just know 
that there are several ways to make it fast and i am happy to help with 
that :-)

if you need a single TSM client to run faster, then option 2 (helper 
process) would fix that. if you want to try something out send me a email 
and i can help you with that. 

> 
> Thanks,
> Sabuj_______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140321/5747d594/attachment.htm>

From sabujp at gmail.com  Thu Mar 13 00:07:22 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Wed, 12 Mar 2014 19:07:22 -0500
Subject: [gpfsug-discuss] question about why unix extensions = no is
 recommended when using samba + gpfs?
Message-ID: <CAEeMGHvH84GDv6yEnPgKtNwOhyvXLkbvV2XGtf3vHpJgSuu=Lg@mail.gmail.com>

Hi all,

I was wondering why here :

https://www.mail-archive.com/gpfsug-discuss at gpfsug.org/msg00121.html

...and several other forums, wikis, etc it's recommended to use :

unix extensions = no

for samba setups with GPFS? This disables the ability for linux/unix samba
clients to see the actual mode bits on files.

Thanks,
Sabuj
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140312/e07bdc29/attachment-0001.htm>

From jonathan at buzzard.me.uk  Thu Mar 13 10:09:21 2014
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Thu, 13 Mar 2014 10:09:21 +0000
Subject: [gpfsug-discuss] question about why unix extensions = no is
 recommended when using samba + gpfs?
In-Reply-To: <CAEeMGHvH84GDv6yEnPgKtNwOhyvXLkbvV2XGtf3vHpJgSuu=Lg@mail.gmail.com>
References: <CAEeMGHvH84GDv6yEnPgKtNwOhyvXLkbvV2XGtf3vHpJgSuu=Lg@mail.gmail.com>
Message-ID: <1394705361.16270.9.camel@buzzard.phy.strath.ac.uk>

On Wed, 2014-03-12 at 19:07 -0500, Sabuj Pattanayek wrote:
> Hi all,
> 
> 
> I was wondering why here :
> 
> 
> https://www.mail-archive.com/gpfsug-discuss at gpfsug.org/msg00121.html
> 
> 
> ...and several other forums, wikis, etc it's recommended to use :
> 
> 
> unix extensions = no 
> 
> 
> for samba setups with GPFS? This disables the ability for linux/unix
> samba clients to see the actual mode bits on files.
> 

Because it messes horribly with NFSv4 ACL's, and MacOSX clients in
particular do "bad things" using Unix extensions that break group
shares. Therefore unless you absolutely need it which most people don't
then disabling it is a sensible choice to avoid wasting hours of your
time trying to work out why everything is broken.


JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.



From sabujp at gmail.com  Thu Mar 13 12:45:01 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Thu, 13 Mar 2014 07:45:01 -0500
Subject: [gpfsug-discuss] question about why unix extensions = no is
 recommended when using samba + gpfs?
In-Reply-To: <1394705361.16270.9.camel@buzzard.phy.strath.ac.uk>
References: <CAEeMGHvH84GDv6yEnPgKtNwOhyvXLkbvV2XGtf3vHpJgSuu=Lg@mail.gmail.com>
	<1394705361.16270.9.camel@buzzard.phy.strath.ac.uk>
Message-ID: <CAEeMGHu4EaiEqa6=qJjHL+4GnkOnQeqh+h5azTHB-1QuqUiTMw@mail.gmail.com>

Hi,

We tried the nfsv4 acl route and it didn't work out so well for us when
files/directories would get promoted to nfsv4 acl's (but maybe I'll revisit
it when I get a chance), I had unix extensions turned off at that time.
We're using for our main template share :

vfs objects = shadow_copy2 gpfs acl_xattr fileid
gpfs:acl = no

to pass acl's to acl_xattr and it seems to work ok and tivoli is able to
backup the security.NTACL extended attribute and restore it without
problems. It'll end up using posix ACLs assigning default acl's for the
users/groups that are assigned to the files/dirs . All of it breaks umask
and other things though, which isn't that big of a deal with samba's
ability to force modes for particular shares.

Regarding unix extensions, it seems there are problems either way (or
perhaps were?), but the problems may be "more" severe if unix extensions
are turned off?

http://wiki.phys.ethz.ch/readme/mac_samba

I'll need to do some more testing with the latest OSX in that case since it
looks like many of these posts were written years ago. We are also running
the latest stable samba 4.1.x from sernet. But it's good to know that unix
extensions = no is not because of some requirement in GPFS.

Thanks,
Sabuj





On Thu, Mar 13, 2014 at 5:09 AM, Jonathan Buzzard <jonathan at buzzard.me.uk>wrote:

> On Wed, 2014-03-12 at 19:07 -0500, Sabuj Pattanayek wrote:
> > Hi all,
> >
> >
> > I was wondering why here :
> >
> >
> > https://www.mail-archive.com/gpfsug-discuss at gpfsug.org/msg00121.html
> >
> >
> > ...and several other forums, wikis, etc it's recommended to use :
> >
> >
> > unix extensions = no
> >
> >
> > for samba setups with GPFS? This disables the ability for linux/unix
> > samba clients to see the actual mode bits on files.
> >
>
> Because it messes horribly with NFSv4 ACL's, and MacOSX clients in
> particular do "bad things" using Unix extensions that break group
> shares. Therefore unless you absolutely need it which most people don't
> then disabling it is a sensible choice to avoid wasting hours of your
> time trying to work out why everything is broken.
>
>
> JAB.
>
> --
> Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
> Fife, United Kingdom.
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140313/31f7efc9/attachment-0001.htm>

From jonathan at buzzard.me.uk  Thu Mar 13 13:08:51 2014
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Thu, 13 Mar 2014 13:08:51 +0000
Subject: [gpfsug-discuss] question about why unix extensions = no is
 recommended when using samba + gpfs?
In-Reply-To: <CAEeMGHu4EaiEqa6=qJjHL+4GnkOnQeqh+h5azTHB-1QuqUiTMw@mail.gmail.com>
References: <CAEeMGHvH84GDv6yEnPgKtNwOhyvXLkbvV2XGtf3vHpJgSuu=Lg@mail.gmail.com>
	<1394705361.16270.9.camel@buzzard.phy.strath.ac.uk>
	<CAEeMGHu4EaiEqa6=qJjHL+4GnkOnQeqh+h5azTHB-1QuqUiTMw@mail.gmail.com>
Message-ID: <1394716131.2612.24.camel@buzzard.phy.strath.ac.uk>

On Thu, 2014-03-13 at 07:45 -0500, Sabuj Pattanayek wrote:
> Hi,
> 
> 
> We tried the nfsv4 acl route and it didn't work out so well for us
> when files/directories would get promoted to nfsv4 acl's (but maybe
> I'll revisit it when I get a chance), I had unix extensions turned off
> at that time. We're using for our main template share :
> 
> 
> vfs objects = shadow_copy2 gpfs acl_xattr fileid
> 
> gpfs:acl = no
> 

That is an "unsual" way to run an Samba/GPFS file server so don't expect
much help from anyone. The vast majority use GPFS in NFSv4 ACL only mode
with the Samba GPFS VFS module doing all the mapping through to the
NFSv4 ACLs to do all the rich permission.

> 
> to pass acl's to acl_xattr and it seems to work ok and tivoli is able
> to backup the security.NTACL extended attribute and restore it without
> problems. It'll end up using posix ACLs assigning default acl's for
> the users/groups that are assigned to the files/dirs . All of it
> breaks umask and other things though, which isn't that big of a deal
> with samba's ability to force modes for particular shares.
> 

That is a "wacked out" setup if you ask me. 

> 
> Regarding unix extensions, it seems there are problems either way (or
> perhaps were?), but the problems may be "more" severe if unix
> extensions are turned off?
> 
> http://wiki.phys.ethz.ch/readme/mac_samba
> 

Yeah changing Unix permissions on a SMB share does not work. Rather like
a real Windows file server don't you think?

Besides which on a group share with Unix extensions on the MacOS X
client (or at least did) goes and changes the permission and ownership
on the file and breaks a "group" share. That is one where a group of
people have equal read/write access to a shared area. Working group
shares are a million times more important that the odd power user being
able to muck about with file permissions.

Note this only effects MacOS because Linux has the tools to see and
manipulate SMB ACL's properly.

> 
> I'll need to do some more testing with the latest OSX in that case
> since it looks like many of these posts were written years ago. We are
> also running the latest stable samba 4.1.x from sernet. But it's good
> to know that unix extensions = no is not because of some requirement
> in GPFS.
> 

It also does not play well with case insensitive = yes in my experience.


JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.



From chair at gpfsug.org  Tue Mar 18 15:43:50 2014
From: chair at gpfsug.org (Jez Tucker (Chair))
Date: Tue, 18 Mar 2014 15:43:50 +0000
Subject: [gpfsug-discuss] GPFS UG10 - Finalised Agenda - 29th April 2014
Message-ID: <532869B6.90706@gpfsug.org>

Hello all,

   We've finalised the agenda for this year's GPFS User Group.

The day encompasses technical presentations (no sales) on the new 
version of GPFS and tweakage.
Come along and meet the GPFS developers, users and technical pros.
Waffle, discuss, opinionate, eat, drink.

The event will be held at: IBM Southbank Client Centre, London, UK 
<http://www-05.ibm.com/uk/clientcentre/about-us.html>
It is free to attend.

Please register via email to secretary at gpfsug.org 
<mailto:secretary at gpfsug.org>
*
*More updates will be provided on http://www.gpfsug.org*

*See you there.

Jez

*
GPFS User Group Meeting: 29th April 2014**
**Agenda**

**10:30 Arrivals and refreshments**

**11:00 Introductions and committee updates*
     Jez Tucker, Group Chair & Claire Robson, Group Secretary

*11:05 GPFS 4.1*
     Scott Fadden, IBM GPFS Technical Marketing

*12:15 GPFS Performance (Basics)*
     Sven Oehme, IBM Scalable Storage Research

*13:00 Lunch**(provided)

**13:45 GPFS Performance (Advanced) *
     Sven Oehme, IBM Scalable Storage Research

*14:45 Refreshments break**

**15:05 Numascale*

*15:35 User Case Study*

*16:00 Group discussion: Questions & Committee matters*
     Led by Jez Tucker, Group Chairperson

*16:05 Close*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140318/66ca96d2/attachment-0001.htm>

From gtsai at slac.stanford.edu  Thu Mar 20 22:21:58 2014
From: gtsai at slac.stanford.edu (Grace Tsai)
Date: Thu, 20 Mar 2014 15:21:58 -0700
Subject: [gpfsug-discuss] Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?
Message-ID: <532B6A06.3000207@slac.stanford.edu>

Hi,

Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?

Thanks.

Grace


From sabujp at gmail.com  Fri Mar 21 01:38:33 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Thu, 20 Mar 2014 20:38:33 -0500
Subject: [gpfsug-discuss] Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?
In-Reply-To: <532B6A06.3000207@slac.stanford.edu>
References: <532B6A06.3000207@slac.stanford.edu>
Message-ID: <CAEeMGHtFHY=8oJu-K2kQopK8LEAUB9DFTL6i++DrszOvvh3N1A@mail.gmail.com>

We're using tsm 7.1 with gpfs 3.5.0.11. At some point we do want to enable
the HSM features but haven't had time to properly configure/set them up
yet. I had dmapi enabled on GPFS but was never able to bring it up with
dmapi enabled. Everything wasn't properly configured at the time and we
were missing some pieces (not my post, but same issue) :

https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014622591

I'd say that we are having less than optimal performance with TSM however.
We're only able to pull about 4TB a day. It took us 20 days to backup 80TB
for our initial full. Using rsync/tar piped to tar would probably have
taken less than a week. We tried various methods, e.g. using a fast
intermediate diskpool, going simultaneously to our 6 LTO6 tape drives, etc
but each "cp" (tsm client) process that TSM would use seemed to be very
slow. We tweaked just about every setting to optimize performance but to
really no avail. When going to the disk pool this is what should have
happened :

GPFS => relatively fast random I/O (on par with rsync/tar piped to tar) tsm
disk cache => large sequential I/O's for each disk pool volume => tape

this is what really happened

GPFS => slow random I/O => tsm disk pool cache => slow random I/O => tape

so instead we did :

GPFS => slow random I/O (TSM) => tape

..but was the same speed as going through the tsm disk pool cache. We
closely monitored the network, disk, memory, cpu, on the tsm server and
none of the hardware or capabilities of the server were the bottleneck, it
was all in TSM.

If anyone has seen this sort of behavior and has some pointers/hints at
improving performance I'd be glad to hear it.

Thanks,
Sabuj


On Thu, Mar 20, 2014 at 5:21 PM, Grace Tsai <gtsai at slac.stanford.edu> wrote:

> Hi,
>
> Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?
>
> Thanks.
>
> Grace
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140320/83143bf7/attachment-0001.htm>

From oehmes at us.ibm.com  Fri Mar 21 21:39:57 2014
From: oehmes at us.ibm.com (Sven Oehme)
Date: Fri, 21 Mar 2014 14:39:57 -0700
Subject: [gpfsug-discuss] Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?
In-Reply-To: <CAEeMGHtFHY=8oJu-K2kQopK8LEAUB9DFTL6i++DrszOvvh3N1A@mail.gmail.com>
References: <532B6A06.3000207@slac.stanford.edu>
	<CAEeMGHtFHY=8oJu-K2kQopK8LEAUB9DFTL6i++DrszOvvh3N1A@mail.gmail.com>
Message-ID: <OF014D6BE3.885B54C0-ON88257CA2.0070C7EC-88257CA2.0077038F@us.ibm.com>

Hi,

there are various ways to speed up backup of data from GPFS to TSM (which 
btw is the same problem for most other backup solutions as well), but 
first one needs to find out what the problem is as it can be multiple 
completely independent issues and each of them needs a different solution, 
so let me explain the different issues and also what you can do about 
them.

the very first challenge is to find what data has changed. the way TSM 
does this is by crawling trough your filesystem, looking at mtime on each 
file to find out which file has changed. think about a ls -Rl on your 
filesystem root. this can, depending on how many files you have, take days 
in a large scale environment (think 100's of millions  of files). there is 
very little one can speed up on this process the way its done. all you can 
do is put metadata on faster disks (e.g. SSD) and that will improve the 
speed of this 'scan phase'. an alternative is to not do this scan at all 
with the TSM client, but instead let GPFS find out for TSM what files have 
changed and then share this information with TSM . the function/command in 
GPFS to do so is called mmbackup : 

https://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=%2Fcom.ibm.cluster.gpfs.v3r5.gpfs100.doc%2Fbl1adm_backupusingmmbackup.htm

it essentially traverses the GPFS metadata sequentially and in parallel 
across all nodes and filters out files that needs to be backed up. 
in several customer environments i was called to assist with issues like 
this, this change alone did speed up the backup process multiple orders of 
magnitudes. we had a few customers where this change reduced the scan time 
from days down to minutes. its not always this big, but its usually the 
largest chunk of the issue. 

the 2nd challenge is if you have to backup a very large number (millions) 
of very small (<32k) files. 
the main issue here is that for each file TSM issues a random i/o to GPFS, 
one at a time, so your throughput directly correlates with size of the 
files and latency for a single file read operation. if you are not on 3.5 
TL3 and/or your files don't fit into the inode its actually even 2 random 
i/os that are issued as you need to read the metadata followed by the data 
block for the file. 
in this scenario you can only do 2 things : 

1. parallelism - mmbackup again starts multiple processes in parallel to 
speed up this phase of the backup 
2. use a 'helper' process to prefetch data for a single TSM client so all 
data comes out of cache and the latency for the random reads is eliminated 
to increase throughput.

without any of this seeing only a few low MB/sec is not uncommon for 
customers, but with the changes above you are able to backup very large 
quantities of data. 

hope this helps. Sven

------------------------------------------
Sven Oehme 
Scalable Storage Research 
email: oehmes at us.ibm.com 
IBM Almaden Research Lab 
------------------------------------------



From:   Sabuj Pattanayek <sabujp at gmail.com>
To:     gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Date:   03/20/2014 06:39 PM
Subject:        Re: [gpfsug-discuss] Is TSM/HSM 7.1 compatible with GPFS 
3.5.0.12 ?
Sent by:        gpfsug-discuss-bounces at gpfsug.org



We're using tsm 7.1 with gpfs 3.5.0.11. At some point we do want to enable 
the HSM features but haven't had time to properly configure/set them up 
yet. I had dmapi enabled on GPFS but was never able to bring it up with 
dmapi enabled. Everything wasn't properly configured at the time and we 
were missing some pieces (not my post, but same issue) :

https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014622591

I'd say that we are having less than optimal performance with TSM however. 
We're only able to pull about 4TB a day. It took us 20 days to backup 80TB 
for our initial full. Using rsync/tar piped to tar would probably have 
taken less than a week. We tried various methods, e.g. using a fast 
intermediate diskpool, going simultaneously to our 6 LTO6 tape drives, etc 
but each "cp" (tsm client) process that TSM would use seemed to be very 
slow. We tweaked just about every setting to optimize performance but to 
really no avail. When going to the disk pool this is what should have 
happened :

GPFS => relatively fast random I/O (on par with rsync/tar piped to tar) 
tsm disk cache => large sequential I/O's for each disk pool volume => tape

this is what really happened

GPFS => slow random I/O => tsm disk pool cache => slow random I/O => tape

so instead we did :

GPFS => slow random I/O (TSM) => tape

..but was the same speed as going through the tsm disk pool cache. We 
closely monitored the network, disk, memory, cpu, on the tsm server and 
none of the hardware or capabilities of the server were the bottleneck, it 
was all in TSM.

If anyone has seen this sort of behavior and has some pointers/hints at 
improving performance I'd be glad to hear it.

Thanks,
Sabuj


On Thu, Mar 20, 2014 at 5:21 PM, Grace Tsai <gtsai at slac.stanford.edu> 
wrote:
Hi,

Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?

Thanks.

Grace
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140321/d1581e61/attachment-0001.htm>

From sabujp at gmail.com  Fri Mar 21 22:16:10 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Fri, 21 Mar 2014 17:16:10 -0500
Subject: [gpfsug-discuss] Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?
In-Reply-To: <OF014D6BE3.885B54C0-ON88257CA2.0070C7EC-88257CA2.0077038F@us.ibm.com>
References: <532B6A06.3000207@slac.stanford.edu>
	<CAEeMGHtFHY=8oJu-K2kQopK8LEAUB9DFTL6i++DrszOvvh3N1A@mail.gmail.com>
	<OF014D6BE3.885B54C0-ON88257CA2.0070C7EC-88257CA2.0077038F@us.ibm.com>
Message-ID: <CAEeMGHt63xJFLsvzDk46jZxcmH_9=E+4twtcKJnGzUr6mGPvpQ@mail.gmail.com>

> the very first challenge is to find what data has changed. the way TSM
> does this is by crawling trough your filesystem, looking at mtime on each
> file to find out which file has changed. think about a ls -Rl on your
> filesystem root. this


The mmbackup mmapplypolicy phase is not a problem. It's going to take as
long as it's going to take. We're using 5 RAID1 SAS SSD NSD's for metadata
and it takes ~1 hour to do the traversal through 157 million files.


>
> the 2nd challenge is if you have to backup a very large number (millions)
> of very small (<32k) files.
> the main issue here is that for each file TSM issues a random i/o to GPFS,
> one at a time, so your throughput directly correlates with size of the
> files and latency for a single file read operation. if you are not on 3.5
> TL3 and/or your files don't fit into the inode its actually even 2 random
> i/os that are issued as you need to read the metadata followed by the data
> block for the file.
> in this scenario you can only do 2 things :


The problem here is why is a single rsync or tar | tar process orders of
magnitude faster than a single tsm client at pulling data off of GPFS into
the same backup system's disk (e.g. disk pool)? It's not a problem with
GPFS, it's a problem with TSM itself. We tried various things, e.g. :

1) changed commmethod to sharedmem
2) increase txnbytelimit to 10G
3) increased movesizethresh to the same as txnbytelimit (10G)
4) increase diskbufsize to 1023kb
5) increased txngroupmax to 65000
6) increased movesizethresh to 10240

the next problem is that one would expect backups to tape to do straight
sequential I/O to tape, in the case of putting the files to the disk pool
before moving them to tape, it did the same random I/O to tape even with
8GB disk pool chunks. We haven't tried the file pool option yet, but we've
been told that it'll do the same thing. If I'm tar'ing or dd'ing large
files to tape that's the most efficient, why doesn't TSM do something
similar?


> 1. parallelism - mmbackup again starts multiple processes in parallel to
> speed up this phase of the backup


..use multiple clients. This would help, but again I'm trying to get a
single tsm client to be on par with a single "cp" process.

Thanks,
Sabuj
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140321/9a73f570/attachment-0001.htm>

From sabujp at gmail.com  Fri Mar 21 22:19:38 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Fri, 21 Mar 2014 17:19:38 -0500
Subject: [gpfsug-discuss] Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?
In-Reply-To: <CAEeMGHt63xJFLsvzDk46jZxcmH_9=E+4twtcKJnGzUr6mGPvpQ@mail.gmail.com>
References: <532B6A06.3000207@slac.stanford.edu>
	<CAEeMGHtFHY=8oJu-K2kQopK8LEAUB9DFTL6i++DrszOvvh3N1A@mail.gmail.com>
	<OF014D6BE3.885B54C0-ON88257CA2.0070C7EC-88257CA2.0077038F@us.ibm.com>
	<CAEeMGHt63xJFLsvzDk46jZxcmH_9=E+4twtcKJnGzUr6mGPvpQ@mail.gmail.com>
Message-ID: <CAEeMGHu3YmKq5F2fGiPDEkxOvSxOTQP-v+kSr7jguXJ0aHS2_g@mail.gmail.com>

It's also not a problem with the TSM database, which is sitting on a RAID10
of 4 SSDs. I'll watch dstat and it'll basically do a large 500MB/s write to
the LUN every 30 mins - hour and then just sit idle waiting for more data.


On Fri, Mar 21, 2014 at 5:16 PM, Sabuj Pattanayek <sabujp at gmail.com> wrote:

>
> the very first challenge is to find what data has changed. the way TSM
>> does this is by crawling trough your filesystem, looking at mtime on each
>> file to find out which file has changed. think about a ls -Rl on your
>> filesystem root. this
>
>
> The mmbackup mmapplypolicy phase is not a problem. It's going to take as
> long as it's going to take. We're using 5 RAID1 SAS SSD NSD's for metadata
> and it takes ~1 hour to do the traversal through 157 million files.
>
>
>>
>> the 2nd challenge is if you have to backup a very large number (millions)
>> of very small (<32k) files.
>> the main issue here is that for each file TSM issues a random i/o to
>> GPFS, one at a time, so your throughput directly correlates with size of
>> the files and latency for a single file read operation. if you are not on
>> 3.5 TL3 and/or your files don't fit into the inode its actually even 2
>> random i/os that are issued as you need to read the metadata followed by
>> the data block for the file.
>> in this scenario you can only do 2 things :
>
>
> The problem here is why is a single rsync or tar | tar process orders of
> magnitude faster than a single tsm client at pulling data off of GPFS into
> the same backup system's disk (e.g. disk pool)? It's not a problem with
> GPFS, it's a problem with TSM itself. We tried various things, e.g. :
>
> 1) changed commmethod to sharedmem
> 2) increase txnbytelimit to 10G
> 3) increased movesizethresh to the same as txnbytelimit (10G)
> 4) increase diskbufsize to 1023kb
> 5) increased txngroupmax to 65000
> 6) increased movesizethresh to 10240
>
> the next problem is that one would expect backups to tape to do straight
> sequential I/O to tape, in the case of putting the files to the disk pool
> before moving them to tape, it did the same random I/O to tape even with
> 8GB disk pool chunks. We haven't tried the file pool option yet, but we've
> been told that it'll do the same thing. If I'm tar'ing or dd'ing large
> files to tape that's the most efficient, why doesn't TSM do something
> similar?
>
>
>> 1. parallelism - mmbackup again starts multiple processes in parallel to
>> speed up this phase of the backup
>
>
> ..use multiple clients. This would help, but again I'm trying to get a
> single tsm client to be on par with a single "cp" process.
>
> Thanks,
> Sabuj
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140321/dcfd38d0/attachment-0001.htm>

From oehmes at us.ibm.com  Fri Mar 21 22:48:50 2014
From: oehmes at us.ibm.com (Sven Oehme)
Date: Fri, 21 Mar 2014 15:48:50 -0700
Subject: [gpfsug-discuss] Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?
In-Reply-To: <CAEeMGHt63xJFLsvzDk46jZxcmH_9=E+4twtcKJnGzUr6mGPvpQ@mail.gmail.com>
References: <532B6A06.3000207@slac.stanford.edu>	<CAEeMGHtFHY=8oJu-K2kQopK8LEAUB9DFTL6i++DrszOvvh3N1A@mail.gmail.com>
	<OF014D6BE3.885B54C0-ON88257CA2.0070C7EC-88257CA2.0077038F@us.ibm.com>
	<CAEeMGHt63xJFLsvzDk46jZxcmH_9=E+4twtcKJnGzUr6mGPvpQ@mail.gmail.com>
Message-ID: <OFC63E3472.65E21691-ON88257CA2.007B27D1-88257CA2.007D51FB@us.ibm.com>

Hi,

> the very first challenge is to find what data has changed. the way 
> TSM does this is by crawling trough your filesystem, looking at 
> mtime on each file to find out which file has changed. think about a
> ls -Rl on your filesystem root. this 
> 
> The mmbackup mmapplypolicy phase is not a problem. It's going to 
> take as long as it's going to take. We're using 5 RAID1 SAS SSD 
> NSD's for metadata and it takes ~1 hour to do the traversal through 
> 157 million files.
>  

that sounds too long, if you want i could take a look at your gpfs config 
and make suggestions on how to further improve this. 
if you want me to do that please send me a email with the output of 
mmlscluster,mmlsconfig,mmlsnsd and mmlsfs all to oehmes at us.ibm.com 

> 
> the 2nd challenge is if you have to backup a very large number 
> (millions) of very small (<32k) files. 
> the main issue here is that for each file TSM issues a random i/o to
> GPFS, one at a time, so your throughput directly correlates with 
> size of the files and latency for a single file read operation. if 
> you are not on 3.5 TL3 and/or your files don't fit into the inode 
> its actually even 2 random i/os that are issued as you need to read 
> the metadata followed by the data block for the file. 
> in this scenario you can only do 2 things : 
> 
> The problem here is why is a single rsync or tar | tar process 
> orders of magnitude faster than a single tsm client at pulling data 
> off of GPFS into the same backup system's disk (e.g. disk pool)? 
> It's not a problem with GPFS, it's a problem with TSM itself. 
> We tried various things, e.g. :
> 
> 1) changed commmethod to sharedmem
> 2) increase txnbytelimit to 10G
> 3) increased movesizethresh to the same as txnbytelimit (10G)
> 4) increase diskbufsize to 1023kb
> 5) increased txngroupmax to 65000
> 6) increased movesizethresh to 10240
> 
> the next problem is that one would expect backups to tape to do 
> straight sequential I/O to tape, in the case of putting the files to
> the disk pool before moving them to tape, it did the same random I/O
> to tape even with 8GB disk pool chunks. We haven't tried the file 
> pool option yet, but we've been told that it'll do the same thing. 

i assume you refer to using raw disks behind TSM ? i never used them, in 
fact the fastest TSM Storage you can build for TSM is to use a GPFS 
filesystem and put files into the filesystem and create a pool with 
sequential file volumes for TSM. if you match the gpfs blocksize with the 
raid blocksize you get very high throughputs to the TSM pool, i have 
customers who see >600 MB/sec backup throughput to a single TSM Server in 
production. 
> If I'm tar'ing or dd'ing large files to tape that's the most 
> efficient, why doesn't TSM do something similar?

i can't tell you much about the tape part of TSM but i can help you speed 
up the ingest into a TSM Server leveraging a Filesystem pool if you want.

> 
> 
> 1. parallelism - mmbackup again starts multiple processes in 
> parallel to speed up this phase of the backup 
> 
> ..use multiple clients. This would help, but again I'm trying to get
> a single tsm client to be on par with a single "cp" process.

cp uses buffered write operation and therefore will not sync the data to 
the disk, which is why its faster. TSM guarantees each write operation to 
to be committed to stable storage before it ever returns to the client, 
this is most likely the difference you see. 

don't get me wrong, i don't try to defend the way TSM works, i just know 
that there are several ways to make it fast and i am happy to help with 
that :-)

if you need a single TSM client to run faster, then option 2 (helper 
process) would fix that. if you want to try something out send me a email 
and i can help you with that. 

> 
> Thanks,
> Sabuj_______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140321/5747d594/attachment-0001.htm>

From sabujp at gmail.com  Thu Mar 13 00:07:22 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Wed, 12 Mar 2014 19:07:22 -0500
Subject: [gpfsug-discuss] question about why unix extensions = no is
 recommended when using samba + gpfs?
Message-ID: <CAEeMGHvH84GDv6yEnPgKtNwOhyvXLkbvV2XGtf3vHpJgSuu=Lg@mail.gmail.com>

Hi all,

I was wondering why here :

https://www.mail-archive.com/gpfsug-discuss at gpfsug.org/msg00121.html

...and several other forums, wikis, etc it's recommended to use :

unix extensions = no

for samba setups with GPFS? This disables the ability for linux/unix samba
clients to see the actual mode bits on files.

Thanks,
Sabuj
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140312/e07bdc29/attachment-0002.htm>

From jonathan at buzzard.me.uk  Thu Mar 13 10:09:21 2014
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Thu, 13 Mar 2014 10:09:21 +0000
Subject: [gpfsug-discuss] question about why unix extensions = no is
 recommended when using samba + gpfs?
In-Reply-To: <CAEeMGHvH84GDv6yEnPgKtNwOhyvXLkbvV2XGtf3vHpJgSuu=Lg@mail.gmail.com>
References: <CAEeMGHvH84GDv6yEnPgKtNwOhyvXLkbvV2XGtf3vHpJgSuu=Lg@mail.gmail.com>
Message-ID: <1394705361.16270.9.camel@buzzard.phy.strath.ac.uk>

On Wed, 2014-03-12 at 19:07 -0500, Sabuj Pattanayek wrote:
> Hi all,
> 
> 
> I was wondering why here :
> 
> 
> https://www.mail-archive.com/gpfsug-discuss at gpfsug.org/msg00121.html
> 
> 
> ...and several other forums, wikis, etc it's recommended to use :
> 
> 
> unix extensions = no 
> 
> 
> for samba setups with GPFS? This disables the ability for linux/unix
> samba clients to see the actual mode bits on files.
> 

Because it messes horribly with NFSv4 ACL's, and MacOSX clients in
particular do "bad things" using Unix extensions that break group
shares. Therefore unless you absolutely need it which most people don't
then disabling it is a sensible choice to avoid wasting hours of your
time trying to work out why everything is broken.


JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.



From sabujp at gmail.com  Thu Mar 13 12:45:01 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Thu, 13 Mar 2014 07:45:01 -0500
Subject: [gpfsug-discuss] question about why unix extensions = no is
 recommended when using samba + gpfs?
In-Reply-To: <1394705361.16270.9.camel@buzzard.phy.strath.ac.uk>
References: <CAEeMGHvH84GDv6yEnPgKtNwOhyvXLkbvV2XGtf3vHpJgSuu=Lg@mail.gmail.com>
	<1394705361.16270.9.camel@buzzard.phy.strath.ac.uk>
Message-ID: <CAEeMGHu4EaiEqa6=qJjHL+4GnkOnQeqh+h5azTHB-1QuqUiTMw@mail.gmail.com>

Hi,

We tried the nfsv4 acl route and it didn't work out so well for us when
files/directories would get promoted to nfsv4 acl's (but maybe I'll revisit
it when I get a chance), I had unix extensions turned off at that time.
We're using for our main template share :

vfs objects = shadow_copy2 gpfs acl_xattr fileid
gpfs:acl = no

to pass acl's to acl_xattr and it seems to work ok and tivoli is able to
backup the security.NTACL extended attribute and restore it without
problems. It'll end up using posix ACLs assigning default acl's for the
users/groups that are assigned to the files/dirs . All of it breaks umask
and other things though, which isn't that big of a deal with samba's
ability to force modes for particular shares.

Regarding unix extensions, it seems there are problems either way (or
perhaps were?), but the problems may be "more" severe if unix extensions
are turned off?

http://wiki.phys.ethz.ch/readme/mac_samba

I'll need to do some more testing with the latest OSX in that case since it
looks like many of these posts were written years ago. We are also running
the latest stable samba 4.1.x from sernet. But it's good to know that unix
extensions = no is not because of some requirement in GPFS.

Thanks,
Sabuj





On Thu, Mar 13, 2014 at 5:09 AM, Jonathan Buzzard <jonathan at buzzard.me.uk>wrote:

> On Wed, 2014-03-12 at 19:07 -0500, Sabuj Pattanayek wrote:
> > Hi all,
> >
> >
> > I was wondering why here :
> >
> >
> > https://www.mail-archive.com/gpfsug-discuss at gpfsug.org/msg00121.html
> >
> >
> > ...and several other forums, wikis, etc it's recommended to use :
> >
> >
> > unix extensions = no
> >
> >
> > for samba setups with GPFS? This disables the ability for linux/unix
> > samba clients to see the actual mode bits on files.
> >
>
> Because it messes horribly with NFSv4 ACL's, and MacOSX clients in
> particular do "bad things" using Unix extensions that break group
> shares. Therefore unless you absolutely need it which most people don't
> then disabling it is a sensible choice to avoid wasting hours of your
> time trying to work out why everything is broken.
>
>
> JAB.
>
> --
> Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
> Fife, United Kingdom.
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140313/31f7efc9/attachment-0002.htm>

From jonathan at buzzard.me.uk  Thu Mar 13 13:08:51 2014
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Thu, 13 Mar 2014 13:08:51 +0000
Subject: [gpfsug-discuss] question about why unix extensions = no is
 recommended when using samba + gpfs?
In-Reply-To: <CAEeMGHu4EaiEqa6=qJjHL+4GnkOnQeqh+h5azTHB-1QuqUiTMw@mail.gmail.com>
References: <CAEeMGHvH84GDv6yEnPgKtNwOhyvXLkbvV2XGtf3vHpJgSuu=Lg@mail.gmail.com>
	<1394705361.16270.9.camel@buzzard.phy.strath.ac.uk>
	<CAEeMGHu4EaiEqa6=qJjHL+4GnkOnQeqh+h5azTHB-1QuqUiTMw@mail.gmail.com>
Message-ID: <1394716131.2612.24.camel@buzzard.phy.strath.ac.uk>

On Thu, 2014-03-13 at 07:45 -0500, Sabuj Pattanayek wrote:
> Hi,
> 
> 
> We tried the nfsv4 acl route and it didn't work out so well for us
> when files/directories would get promoted to nfsv4 acl's (but maybe
> I'll revisit it when I get a chance), I had unix extensions turned off
> at that time. We're using for our main template share :
> 
> 
> vfs objects = shadow_copy2 gpfs acl_xattr fileid
> 
> gpfs:acl = no
> 

That is an "unsual" way to run an Samba/GPFS file server so don't expect
much help from anyone. The vast majority use GPFS in NFSv4 ACL only mode
with the Samba GPFS VFS module doing all the mapping through to the
NFSv4 ACLs to do all the rich permission.

> 
> to pass acl's to acl_xattr and it seems to work ok and tivoli is able
> to backup the security.NTACL extended attribute and restore it without
> problems. It'll end up using posix ACLs assigning default acl's for
> the users/groups that are assigned to the files/dirs . All of it
> breaks umask and other things though, which isn't that big of a deal
> with samba's ability to force modes for particular shares.
> 

That is a "wacked out" setup if you ask me. 

> 
> Regarding unix extensions, it seems there are problems either way (or
> perhaps were?), but the problems may be "more" severe if unix
> extensions are turned off?
> 
> http://wiki.phys.ethz.ch/readme/mac_samba
> 

Yeah changing Unix permissions on a SMB share does not work. Rather like
a real Windows file server don't you think?

Besides which on a group share with Unix extensions on the MacOS X
client (or at least did) goes and changes the permission and ownership
on the file and breaks a "group" share. That is one where a group of
people have equal read/write access to a shared area. Working group
shares are a million times more important that the odd power user being
able to muck about with file permissions.

Note this only effects MacOS because Linux has the tools to see and
manipulate SMB ACL's properly.

> 
> I'll need to do some more testing with the latest OSX in that case
> since it looks like many of these posts were written years ago. We are
> also running the latest stable samba 4.1.x from sernet. But it's good
> to know that unix extensions = no is not because of some requirement
> in GPFS.
> 

It also does not play well with case insensitive = yes in my experience.


JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.



From chair at gpfsug.org  Tue Mar 18 15:43:50 2014
From: chair at gpfsug.org (Jez Tucker (Chair))
Date: Tue, 18 Mar 2014 15:43:50 +0000
Subject: [gpfsug-discuss] GPFS UG10 - Finalised Agenda - 29th April 2014
Message-ID: <532869B6.90706@gpfsug.org>

Hello all,

   We've finalised the agenda for this year's GPFS User Group.

The day encompasses technical presentations (no sales) on the new 
version of GPFS and tweakage.
Come along and meet the GPFS developers, users and technical pros.
Waffle, discuss, opinionate, eat, drink.

The event will be held at: IBM Southbank Client Centre, London, UK 
<http://www-05.ibm.com/uk/clientcentre/about-us.html>
It is free to attend.

Please register via email to secretary at gpfsug.org 
<mailto:secretary at gpfsug.org>
*
*More updates will be provided on http://www.gpfsug.org*

*See you there.

Jez

*
GPFS User Group Meeting: 29th April 2014**
**Agenda**

**10:30 Arrivals and refreshments**

**11:00 Introductions and committee updates*
     Jez Tucker, Group Chair & Claire Robson, Group Secretary

*11:05 GPFS 4.1*
     Scott Fadden, IBM GPFS Technical Marketing

*12:15 GPFS Performance (Basics)*
     Sven Oehme, IBM Scalable Storage Research

*13:00 Lunch**(provided)

**13:45 GPFS Performance (Advanced) *
     Sven Oehme, IBM Scalable Storage Research

*14:45 Refreshments break**

**15:05 Numascale*

*15:35 User Case Study*

*16:00 Group discussion: Questions & Committee matters*
     Led by Jez Tucker, Group Chairperson

*16:05 Close*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140318/66ca96d2/attachment-0002.htm>

From gtsai at slac.stanford.edu  Thu Mar 20 22:21:58 2014
From: gtsai at slac.stanford.edu (Grace Tsai)
Date: Thu, 20 Mar 2014 15:21:58 -0700
Subject: [gpfsug-discuss] Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?
Message-ID: <532B6A06.3000207@slac.stanford.edu>

Hi,

Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?

Thanks.

Grace


From sabujp at gmail.com  Fri Mar 21 01:38:33 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Thu, 20 Mar 2014 20:38:33 -0500
Subject: [gpfsug-discuss] Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?
In-Reply-To: <532B6A06.3000207@slac.stanford.edu>
References: <532B6A06.3000207@slac.stanford.edu>
Message-ID: <CAEeMGHtFHY=8oJu-K2kQopK8LEAUB9DFTL6i++DrszOvvh3N1A@mail.gmail.com>

We're using tsm 7.1 with gpfs 3.5.0.11. At some point we do want to enable
the HSM features but haven't had time to properly configure/set them up
yet. I had dmapi enabled on GPFS but was never able to bring it up with
dmapi enabled. Everything wasn't properly configured at the time and we
were missing some pieces (not my post, but same issue) :

https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014622591

I'd say that we are having less than optimal performance with TSM however.
We're only able to pull about 4TB a day. It took us 20 days to backup 80TB
for our initial full. Using rsync/tar piped to tar would probably have
taken less than a week. We tried various methods, e.g. using a fast
intermediate diskpool, going simultaneously to our 6 LTO6 tape drives, etc
but each "cp" (tsm client) process that TSM would use seemed to be very
slow. We tweaked just about every setting to optimize performance but to
really no avail. When going to the disk pool this is what should have
happened :

GPFS => relatively fast random I/O (on par with rsync/tar piped to tar) tsm
disk cache => large sequential I/O's for each disk pool volume => tape

this is what really happened

GPFS => slow random I/O => tsm disk pool cache => slow random I/O => tape

so instead we did :

GPFS => slow random I/O (TSM) => tape

..but was the same speed as going through the tsm disk pool cache. We
closely monitored the network, disk, memory, cpu, on the tsm server and
none of the hardware or capabilities of the server were the bottleneck, it
was all in TSM.

If anyone has seen this sort of behavior and has some pointers/hints at
improving performance I'd be glad to hear it.

Thanks,
Sabuj


On Thu, Mar 20, 2014 at 5:21 PM, Grace Tsai <gtsai at slac.stanford.edu> wrote:

> Hi,
>
> Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?
>
> Thanks.
>
> Grace
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140320/83143bf7/attachment-0002.htm>

From oehmes at us.ibm.com  Fri Mar 21 21:39:57 2014
From: oehmes at us.ibm.com (Sven Oehme)
Date: Fri, 21 Mar 2014 14:39:57 -0700
Subject: [gpfsug-discuss] Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?
In-Reply-To: <CAEeMGHtFHY=8oJu-K2kQopK8LEAUB9DFTL6i++DrszOvvh3N1A@mail.gmail.com>
References: <532B6A06.3000207@slac.stanford.edu>
	<CAEeMGHtFHY=8oJu-K2kQopK8LEAUB9DFTL6i++DrszOvvh3N1A@mail.gmail.com>
Message-ID: <OF014D6BE3.885B54C0-ON88257CA2.0070C7EC-88257CA2.0077038F@us.ibm.com>

Hi,

there are various ways to speed up backup of data from GPFS to TSM (which 
btw is the same problem for most other backup solutions as well), but 
first one needs to find out what the problem is as it can be multiple 
completely independent issues and each of them needs a different solution, 
so let me explain the different issues and also what you can do about 
them.

the very first challenge is to find what data has changed. the way TSM 
does this is by crawling trough your filesystem, looking at mtime on each 
file to find out which file has changed. think about a ls -Rl on your 
filesystem root. this can, depending on how many files you have, take days 
in a large scale environment (think 100's of millions  of files). there is 
very little one can speed up on this process the way its done. all you can 
do is put metadata on faster disks (e.g. SSD) and that will improve the 
speed of this 'scan phase'. an alternative is to not do this scan at all 
with the TSM client, but instead let GPFS find out for TSM what files have 
changed and then share this information with TSM . the function/command in 
GPFS to do so is called mmbackup : 

https://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=%2Fcom.ibm.cluster.gpfs.v3r5.gpfs100.doc%2Fbl1adm_backupusingmmbackup.htm

it essentially traverses the GPFS metadata sequentially and in parallel 
across all nodes and filters out files that needs to be backed up. 
in several customer environments i was called to assist with issues like 
this, this change alone did speed up the backup process multiple orders of 
magnitudes. we had a few customers where this change reduced the scan time 
from days down to minutes. its not always this big, but its usually the 
largest chunk of the issue. 

the 2nd challenge is if you have to backup a very large number (millions) 
of very small (<32k) files. 
the main issue here is that for each file TSM issues a random i/o to GPFS, 
one at a time, so your throughput directly correlates with size of the 
files and latency for a single file read operation. if you are not on 3.5 
TL3 and/or your files don't fit into the inode its actually even 2 random 
i/os that are issued as you need to read the metadata followed by the data 
block for the file. 
in this scenario you can only do 2 things : 

1. parallelism - mmbackup again starts multiple processes in parallel to 
speed up this phase of the backup 
2. use a 'helper' process to prefetch data for a single TSM client so all 
data comes out of cache and the latency for the random reads is eliminated 
to increase throughput.

without any of this seeing only a few low MB/sec is not uncommon for 
customers, but with the changes above you are able to backup very large 
quantities of data. 

hope this helps. Sven

------------------------------------------
Sven Oehme 
Scalable Storage Research 
email: oehmes at us.ibm.com 
IBM Almaden Research Lab 
------------------------------------------



From:   Sabuj Pattanayek <sabujp at gmail.com>
To:     gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Date:   03/20/2014 06:39 PM
Subject:        Re: [gpfsug-discuss] Is TSM/HSM 7.1 compatible with GPFS 
3.5.0.12 ?
Sent by:        gpfsug-discuss-bounces at gpfsug.org



We're using tsm 7.1 with gpfs 3.5.0.11. At some point we do want to enable 
the HSM features but haven't had time to properly configure/set them up 
yet. I had dmapi enabled on GPFS but was never able to bring it up with 
dmapi enabled. Everything wasn't properly configured at the time and we 
were missing some pieces (not my post, but same issue) :

https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014622591

I'd say that we are having less than optimal performance with TSM however. 
We're only able to pull about 4TB a day. It took us 20 days to backup 80TB 
for our initial full. Using rsync/tar piped to tar would probably have 
taken less than a week. We tried various methods, e.g. using a fast 
intermediate diskpool, going simultaneously to our 6 LTO6 tape drives, etc 
but each "cp" (tsm client) process that TSM would use seemed to be very 
slow. We tweaked just about every setting to optimize performance but to 
really no avail. When going to the disk pool this is what should have 
happened :

GPFS => relatively fast random I/O (on par with rsync/tar piped to tar) 
tsm disk cache => large sequential I/O's for each disk pool volume => tape

this is what really happened

GPFS => slow random I/O => tsm disk pool cache => slow random I/O => tape

so instead we did :

GPFS => slow random I/O (TSM) => tape

..but was the same speed as going through the tsm disk pool cache. We 
closely monitored the network, disk, memory, cpu, on the tsm server and 
none of the hardware or capabilities of the server were the bottleneck, it 
was all in TSM.

If anyone has seen this sort of behavior and has some pointers/hints at 
improving performance I'd be glad to hear it.

Thanks,
Sabuj


On Thu, Mar 20, 2014 at 5:21 PM, Grace Tsai <gtsai at slac.stanford.edu> 
wrote:
Hi,

Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?

Thanks.

Grace
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140321/d1581e61/attachment-0002.htm>

From sabujp at gmail.com  Fri Mar 21 22:16:10 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Fri, 21 Mar 2014 17:16:10 -0500
Subject: [gpfsug-discuss] Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?
In-Reply-To: <OF014D6BE3.885B54C0-ON88257CA2.0070C7EC-88257CA2.0077038F@us.ibm.com>
References: <532B6A06.3000207@slac.stanford.edu>
	<CAEeMGHtFHY=8oJu-K2kQopK8LEAUB9DFTL6i++DrszOvvh3N1A@mail.gmail.com>
	<OF014D6BE3.885B54C0-ON88257CA2.0070C7EC-88257CA2.0077038F@us.ibm.com>
Message-ID: <CAEeMGHt63xJFLsvzDk46jZxcmH_9=E+4twtcKJnGzUr6mGPvpQ@mail.gmail.com>

> the very first challenge is to find what data has changed. the way TSM
> does this is by crawling trough your filesystem, looking at mtime on each
> file to find out which file has changed. think about a ls -Rl on your
> filesystem root. this


The mmbackup mmapplypolicy phase is not a problem. It's going to take as
long as it's going to take. We're using 5 RAID1 SAS SSD NSD's for metadata
and it takes ~1 hour to do the traversal through 157 million files.


>
> the 2nd challenge is if you have to backup a very large number (millions)
> of very small (<32k) files.
> the main issue here is that for each file TSM issues a random i/o to GPFS,
> one at a time, so your throughput directly correlates with size of the
> files and latency for a single file read operation. if you are not on 3.5
> TL3 and/or your files don't fit into the inode its actually even 2 random
> i/os that are issued as you need to read the metadata followed by the data
> block for the file.
> in this scenario you can only do 2 things :


The problem here is why is a single rsync or tar | tar process orders of
magnitude faster than a single tsm client at pulling data off of GPFS into
the same backup system's disk (e.g. disk pool)? It's not a problem with
GPFS, it's a problem with TSM itself. We tried various things, e.g. :

1) changed commmethod to sharedmem
2) increase txnbytelimit to 10G
3) increased movesizethresh to the same as txnbytelimit (10G)
4) increase diskbufsize to 1023kb
5) increased txngroupmax to 65000
6) increased movesizethresh to 10240

the next problem is that one would expect backups to tape to do straight
sequential I/O to tape, in the case of putting the files to the disk pool
before moving them to tape, it did the same random I/O to tape even with
8GB disk pool chunks. We haven't tried the file pool option yet, but we've
been told that it'll do the same thing. If I'm tar'ing or dd'ing large
files to tape that's the most efficient, why doesn't TSM do something
similar?


> 1. parallelism - mmbackup again starts multiple processes in parallel to
> speed up this phase of the backup


..use multiple clients. This would help, but again I'm trying to get a
single tsm client to be on par with a single "cp" process.

Thanks,
Sabuj
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140321/9a73f570/attachment-0002.htm>

From sabujp at gmail.com  Fri Mar 21 22:19:38 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Fri, 21 Mar 2014 17:19:38 -0500
Subject: [gpfsug-discuss] Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?
In-Reply-To: <CAEeMGHt63xJFLsvzDk46jZxcmH_9=E+4twtcKJnGzUr6mGPvpQ@mail.gmail.com>
References: <532B6A06.3000207@slac.stanford.edu>
	<CAEeMGHtFHY=8oJu-K2kQopK8LEAUB9DFTL6i++DrszOvvh3N1A@mail.gmail.com>
	<OF014D6BE3.885B54C0-ON88257CA2.0070C7EC-88257CA2.0077038F@us.ibm.com>
	<CAEeMGHt63xJFLsvzDk46jZxcmH_9=E+4twtcKJnGzUr6mGPvpQ@mail.gmail.com>
Message-ID: <CAEeMGHu3YmKq5F2fGiPDEkxOvSxOTQP-v+kSr7jguXJ0aHS2_g@mail.gmail.com>

It's also not a problem with the TSM database, which is sitting on a RAID10
of 4 SSDs. I'll watch dstat and it'll basically do a large 500MB/s write to
the LUN every 30 mins - hour and then just sit idle waiting for more data.


On Fri, Mar 21, 2014 at 5:16 PM, Sabuj Pattanayek <sabujp at gmail.com> wrote:

>
> the very first challenge is to find what data has changed. the way TSM
>> does this is by crawling trough your filesystem, looking at mtime on each
>> file to find out which file has changed. think about a ls -Rl on your
>> filesystem root. this
>
>
> The mmbackup mmapplypolicy phase is not a problem. It's going to take as
> long as it's going to take. We're using 5 RAID1 SAS SSD NSD's for metadata
> and it takes ~1 hour to do the traversal through 157 million files.
>
>
>>
>> the 2nd challenge is if you have to backup a very large number (millions)
>> of very small (<32k) files.
>> the main issue here is that for each file TSM issues a random i/o to
>> GPFS, one at a time, so your throughput directly correlates with size of
>> the files and latency for a single file read operation. if you are not on
>> 3.5 TL3 and/or your files don't fit into the inode its actually even 2
>> random i/os that are issued as you need to read the metadata followed by
>> the data block for the file.
>> in this scenario you can only do 2 things :
>
>
> The problem here is why is a single rsync or tar | tar process orders of
> magnitude faster than a single tsm client at pulling data off of GPFS into
> the same backup system's disk (e.g. disk pool)? It's not a problem with
> GPFS, it's a problem with TSM itself. We tried various things, e.g. :
>
> 1) changed commmethod to sharedmem
> 2) increase txnbytelimit to 10G
> 3) increased movesizethresh to the same as txnbytelimit (10G)
> 4) increase diskbufsize to 1023kb
> 5) increased txngroupmax to 65000
> 6) increased movesizethresh to 10240
>
> the next problem is that one would expect backups to tape to do straight
> sequential I/O to tape, in the case of putting the files to the disk pool
> before moving them to tape, it did the same random I/O to tape even with
> 8GB disk pool chunks. We haven't tried the file pool option yet, but we've
> been told that it'll do the same thing. If I'm tar'ing or dd'ing large
> files to tape that's the most efficient, why doesn't TSM do something
> similar?
>
>
>> 1. parallelism - mmbackup again starts multiple processes in parallel to
>> speed up this phase of the backup
>
>
> ..use multiple clients. This would help, but again I'm trying to get a
> single tsm client to be on par with a single "cp" process.
>
> Thanks,
> Sabuj
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140321/dcfd38d0/attachment-0002.htm>

From oehmes at us.ibm.com  Fri Mar 21 22:48:50 2014
From: oehmes at us.ibm.com (Sven Oehme)
Date: Fri, 21 Mar 2014 15:48:50 -0700
Subject: [gpfsug-discuss] Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?
In-Reply-To: <CAEeMGHt63xJFLsvzDk46jZxcmH_9=E+4twtcKJnGzUr6mGPvpQ@mail.gmail.com>
References: <532B6A06.3000207@slac.stanford.edu>	<CAEeMGHtFHY=8oJu-K2kQopK8LEAUB9DFTL6i++DrszOvvh3N1A@mail.gmail.com>
	<OF014D6BE3.885B54C0-ON88257CA2.0070C7EC-88257CA2.0077038F@us.ibm.com>
	<CAEeMGHt63xJFLsvzDk46jZxcmH_9=E+4twtcKJnGzUr6mGPvpQ@mail.gmail.com>
Message-ID: <OFC63E3472.65E21691-ON88257CA2.007B27D1-88257CA2.007D51FB@us.ibm.com>

Hi,

> the very first challenge is to find what data has changed. the way 
> TSM does this is by crawling trough your filesystem, looking at 
> mtime on each file to find out which file has changed. think about a
> ls -Rl on your filesystem root. this 
> 
> The mmbackup mmapplypolicy phase is not a problem. It's going to 
> take as long as it's going to take. We're using 5 RAID1 SAS SSD 
> NSD's for metadata and it takes ~1 hour to do the traversal through 
> 157 million files.
>  

that sounds too long, if you want i could take a look at your gpfs config 
and make suggestions on how to further improve this. 
if you want me to do that please send me a email with the output of 
mmlscluster,mmlsconfig,mmlsnsd and mmlsfs all to oehmes at us.ibm.com 

> 
> the 2nd challenge is if you have to backup a very large number 
> (millions) of very small (<32k) files. 
> the main issue here is that for each file TSM issues a random i/o to
> GPFS, one at a time, so your throughput directly correlates with 
> size of the files and latency for a single file read operation. if 
> you are not on 3.5 TL3 and/or your files don't fit into the inode 
> its actually even 2 random i/os that are issued as you need to read 
> the metadata followed by the data block for the file. 
> in this scenario you can only do 2 things : 
> 
> The problem here is why is a single rsync or tar | tar process 
> orders of magnitude faster than a single tsm client at pulling data 
> off of GPFS into the same backup system's disk (e.g. disk pool)? 
> It's not a problem with GPFS, it's a problem with TSM itself. 
> We tried various things, e.g. :
> 
> 1) changed commmethod to sharedmem
> 2) increase txnbytelimit to 10G
> 3) increased movesizethresh to the same as txnbytelimit (10G)
> 4) increase diskbufsize to 1023kb
> 5) increased txngroupmax to 65000
> 6) increased movesizethresh to 10240
> 
> the next problem is that one would expect backups to tape to do 
> straight sequential I/O to tape, in the case of putting the files to
> the disk pool before moving them to tape, it did the same random I/O
> to tape even with 8GB disk pool chunks. We haven't tried the file 
> pool option yet, but we've been told that it'll do the same thing. 

i assume you refer to using raw disks behind TSM ? i never used them, in 
fact the fastest TSM Storage you can build for TSM is to use a GPFS 
filesystem and put files into the filesystem and create a pool with 
sequential file volumes for TSM. if you match the gpfs blocksize with the 
raid blocksize you get very high throughputs to the TSM pool, i have 
customers who see >600 MB/sec backup throughput to a single TSM Server in 
production. 
> If I'm tar'ing or dd'ing large files to tape that's the most 
> efficient, why doesn't TSM do something similar?

i can't tell you much about the tape part of TSM but i can help you speed 
up the ingest into a TSM Server leveraging a Filesystem pool if you want.

> 
> 
> 1. parallelism - mmbackup again starts multiple processes in 
> parallel to speed up this phase of the backup 
> 
> ..use multiple clients. This would help, but again I'm trying to get
> a single tsm client to be on par with a single "cp" process.

cp uses buffered write operation and therefore will not sync the data to 
the disk, which is why its faster. TSM guarantees each write operation to 
to be committed to stable storage before it ever returns to the client, 
this is most likely the difference you see. 

don't get me wrong, i don't try to defend the way TSM works, i just know 
that there are several ways to make it fast and i am happy to help with 
that :-)

if you need a single TSM client to run faster, then option 2 (helper 
process) would fix that. if you want to try something out send me a email 
and i can help you with that. 

> 
> Thanks,
> Sabuj_______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140321/5747d594/attachment-0002.htm>

From sabujp at gmail.com  Thu Mar 13 00:07:22 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Wed, 12 Mar 2014 19:07:22 -0500
Subject: [gpfsug-discuss] question about why unix extensions = no is
 recommended when using samba + gpfs?
Message-ID: <CAEeMGHvH84GDv6yEnPgKtNwOhyvXLkbvV2XGtf3vHpJgSuu=Lg@mail.gmail.com>

Hi all,

I was wondering why here :

https://www.mail-archive.com/gpfsug-discuss at gpfsug.org/msg00121.html

...and several other forums, wikis, etc it's recommended to use :

unix extensions = no

for samba setups with GPFS? This disables the ability for linux/unix samba
clients to see the actual mode bits on files.

Thanks,
Sabuj
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140312/e07bdc29/attachment-0003.htm>

From jonathan at buzzard.me.uk  Thu Mar 13 10:09:21 2014
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Thu, 13 Mar 2014 10:09:21 +0000
Subject: [gpfsug-discuss] question about why unix extensions = no is
 recommended when using samba + gpfs?
In-Reply-To: <CAEeMGHvH84GDv6yEnPgKtNwOhyvXLkbvV2XGtf3vHpJgSuu=Lg@mail.gmail.com>
References: <CAEeMGHvH84GDv6yEnPgKtNwOhyvXLkbvV2XGtf3vHpJgSuu=Lg@mail.gmail.com>
Message-ID: <1394705361.16270.9.camel@buzzard.phy.strath.ac.uk>

On Wed, 2014-03-12 at 19:07 -0500, Sabuj Pattanayek wrote:
> Hi all,
> 
> 
> I was wondering why here :
> 
> 
> https://www.mail-archive.com/gpfsug-discuss at gpfsug.org/msg00121.html
> 
> 
> ...and several other forums, wikis, etc it's recommended to use :
> 
> 
> unix extensions = no 
> 
> 
> for samba setups with GPFS? This disables the ability for linux/unix
> samba clients to see the actual mode bits on files.
> 

Because it messes horribly with NFSv4 ACL's, and MacOSX clients in
particular do "bad things" using Unix extensions that break group
shares. Therefore unless you absolutely need it which most people don't
then disabling it is a sensible choice to avoid wasting hours of your
time trying to work out why everything is broken.


JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.



From sabujp at gmail.com  Thu Mar 13 12:45:01 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Thu, 13 Mar 2014 07:45:01 -0500
Subject: [gpfsug-discuss] question about why unix extensions = no is
 recommended when using samba + gpfs?
In-Reply-To: <1394705361.16270.9.camel@buzzard.phy.strath.ac.uk>
References: <CAEeMGHvH84GDv6yEnPgKtNwOhyvXLkbvV2XGtf3vHpJgSuu=Lg@mail.gmail.com>
	<1394705361.16270.9.camel@buzzard.phy.strath.ac.uk>
Message-ID: <CAEeMGHu4EaiEqa6=qJjHL+4GnkOnQeqh+h5azTHB-1QuqUiTMw@mail.gmail.com>

Hi,

We tried the nfsv4 acl route and it didn't work out so well for us when
files/directories would get promoted to nfsv4 acl's (but maybe I'll revisit
it when I get a chance), I had unix extensions turned off at that time.
We're using for our main template share :

vfs objects = shadow_copy2 gpfs acl_xattr fileid
gpfs:acl = no

to pass acl's to acl_xattr and it seems to work ok and tivoli is able to
backup the security.NTACL extended attribute and restore it without
problems. It'll end up using posix ACLs assigning default acl's for the
users/groups that are assigned to the files/dirs . All of it breaks umask
and other things though, which isn't that big of a deal with samba's
ability to force modes for particular shares.

Regarding unix extensions, it seems there are problems either way (or
perhaps were?), but the problems may be "more" severe if unix extensions
are turned off?

http://wiki.phys.ethz.ch/readme/mac_samba

I'll need to do some more testing with the latest OSX in that case since it
looks like many of these posts were written years ago. We are also running
the latest stable samba 4.1.x from sernet. But it's good to know that unix
extensions = no is not because of some requirement in GPFS.

Thanks,
Sabuj





On Thu, Mar 13, 2014 at 5:09 AM, Jonathan Buzzard <jonathan at buzzard.me.uk>wrote:

> On Wed, 2014-03-12 at 19:07 -0500, Sabuj Pattanayek wrote:
> > Hi all,
> >
> >
> > I was wondering why here :
> >
> >
> > https://www.mail-archive.com/gpfsug-discuss at gpfsug.org/msg00121.html
> >
> >
> > ...and several other forums, wikis, etc it's recommended to use :
> >
> >
> > unix extensions = no
> >
> >
> > for samba setups with GPFS? This disables the ability for linux/unix
> > samba clients to see the actual mode bits on files.
> >
>
> Because it messes horribly with NFSv4 ACL's, and MacOSX clients in
> particular do "bad things" using Unix extensions that break group
> shares. Therefore unless you absolutely need it which most people don't
> then disabling it is a sensible choice to avoid wasting hours of your
> time trying to work out why everything is broken.
>
>
> JAB.
>
> --
> Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
> Fife, United Kingdom.
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140313/31f7efc9/attachment-0003.htm>

From jonathan at buzzard.me.uk  Thu Mar 13 13:08:51 2014
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Thu, 13 Mar 2014 13:08:51 +0000
Subject: [gpfsug-discuss] question about why unix extensions = no is
 recommended when using samba + gpfs?
In-Reply-To: <CAEeMGHu4EaiEqa6=qJjHL+4GnkOnQeqh+h5azTHB-1QuqUiTMw@mail.gmail.com>
References: <CAEeMGHvH84GDv6yEnPgKtNwOhyvXLkbvV2XGtf3vHpJgSuu=Lg@mail.gmail.com>
	<1394705361.16270.9.camel@buzzard.phy.strath.ac.uk>
	<CAEeMGHu4EaiEqa6=qJjHL+4GnkOnQeqh+h5azTHB-1QuqUiTMw@mail.gmail.com>
Message-ID: <1394716131.2612.24.camel@buzzard.phy.strath.ac.uk>

On Thu, 2014-03-13 at 07:45 -0500, Sabuj Pattanayek wrote:
> Hi,
> 
> 
> We tried the nfsv4 acl route and it didn't work out so well for us
> when files/directories would get promoted to nfsv4 acl's (but maybe
> I'll revisit it when I get a chance), I had unix extensions turned off
> at that time. We're using for our main template share :
> 
> 
> vfs objects = shadow_copy2 gpfs acl_xattr fileid
> 
> gpfs:acl = no
> 

That is an "unsual" way to run an Samba/GPFS file server so don't expect
much help from anyone. The vast majority use GPFS in NFSv4 ACL only mode
with the Samba GPFS VFS module doing all the mapping through to the
NFSv4 ACLs to do all the rich permission.

> 
> to pass acl's to acl_xattr and it seems to work ok and tivoli is able
> to backup the security.NTACL extended attribute and restore it without
> problems. It'll end up using posix ACLs assigning default acl's for
> the users/groups that are assigned to the files/dirs . All of it
> breaks umask and other things though, which isn't that big of a deal
> with samba's ability to force modes for particular shares.
> 

That is a "wacked out" setup if you ask me. 

> 
> Regarding unix extensions, it seems there are problems either way (or
> perhaps were?), but the problems may be "more" severe if unix
> extensions are turned off?
> 
> http://wiki.phys.ethz.ch/readme/mac_samba
> 

Yeah changing Unix permissions on a SMB share does not work. Rather like
a real Windows file server don't you think?

Besides which on a group share with Unix extensions on the MacOS X
client (or at least did) goes and changes the permission and ownership
on the file and breaks a "group" share. That is one where a group of
people have equal read/write access to a shared area. Working group
shares are a million times more important that the odd power user being
able to muck about with file permissions.

Note this only effects MacOS because Linux has the tools to see and
manipulate SMB ACL's properly.

> 
> I'll need to do some more testing with the latest OSX in that case
> since it looks like many of these posts were written years ago. We are
> also running the latest stable samba 4.1.x from sernet. But it's good
> to know that unix extensions = no is not because of some requirement
> in GPFS.
> 

It also does not play well with case insensitive = yes in my experience.


JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.



From chair at gpfsug.org  Tue Mar 18 15:43:50 2014
From: chair at gpfsug.org (Jez Tucker (Chair))
Date: Tue, 18 Mar 2014 15:43:50 +0000
Subject: [gpfsug-discuss] GPFS UG10 - Finalised Agenda - 29th April 2014
Message-ID: <532869B6.90706@gpfsug.org>

Hello all,

   We've finalised the agenda for this year's GPFS User Group.

The day encompasses technical presentations (no sales) on the new 
version of GPFS and tweakage.
Come along and meet the GPFS developers, users and technical pros.
Waffle, discuss, opinionate, eat, drink.

The event will be held at: IBM Southbank Client Centre, London, UK 
<http://www-05.ibm.com/uk/clientcentre/about-us.html>
It is free to attend.

Please register via email to secretary at gpfsug.org 
<mailto:secretary at gpfsug.org>
*
*More updates will be provided on http://www.gpfsug.org*

*See you there.

Jez

*
GPFS User Group Meeting: 29th April 2014**
**Agenda**

**10:30 Arrivals and refreshments**

**11:00 Introductions and committee updates*
     Jez Tucker, Group Chair & Claire Robson, Group Secretary

*11:05 GPFS 4.1*
     Scott Fadden, IBM GPFS Technical Marketing

*12:15 GPFS Performance (Basics)*
     Sven Oehme, IBM Scalable Storage Research

*13:00 Lunch**(provided)

**13:45 GPFS Performance (Advanced) *
     Sven Oehme, IBM Scalable Storage Research

*14:45 Refreshments break**

**15:05 Numascale*

*15:35 User Case Study*

*16:00 Group discussion: Questions & Committee matters*
     Led by Jez Tucker, Group Chairperson

*16:05 Close*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140318/66ca96d2/attachment-0003.htm>

From gtsai at slac.stanford.edu  Thu Mar 20 22:21:58 2014
From: gtsai at slac.stanford.edu (Grace Tsai)
Date: Thu, 20 Mar 2014 15:21:58 -0700
Subject: [gpfsug-discuss] Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?
Message-ID: <532B6A06.3000207@slac.stanford.edu>

Hi,

Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?

Thanks.

Grace


From sabujp at gmail.com  Fri Mar 21 01:38:33 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Thu, 20 Mar 2014 20:38:33 -0500
Subject: [gpfsug-discuss] Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?
In-Reply-To: <532B6A06.3000207@slac.stanford.edu>
References: <532B6A06.3000207@slac.stanford.edu>
Message-ID: <CAEeMGHtFHY=8oJu-K2kQopK8LEAUB9DFTL6i++DrszOvvh3N1A@mail.gmail.com>

We're using tsm 7.1 with gpfs 3.5.0.11. At some point we do want to enable
the HSM features but haven't had time to properly configure/set them up
yet. I had dmapi enabled on GPFS but was never able to bring it up with
dmapi enabled. Everything wasn't properly configured at the time and we
were missing some pieces (not my post, but same issue) :

https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014622591

I'd say that we are having less than optimal performance with TSM however.
We're only able to pull about 4TB a day. It took us 20 days to backup 80TB
for our initial full. Using rsync/tar piped to tar would probably have
taken less than a week. We tried various methods, e.g. using a fast
intermediate diskpool, going simultaneously to our 6 LTO6 tape drives, etc
but each "cp" (tsm client) process that TSM would use seemed to be very
slow. We tweaked just about every setting to optimize performance but to
really no avail. When going to the disk pool this is what should have
happened :

GPFS => relatively fast random I/O (on par with rsync/tar piped to tar) tsm
disk cache => large sequential I/O's for each disk pool volume => tape

this is what really happened

GPFS => slow random I/O => tsm disk pool cache => slow random I/O => tape

so instead we did :

GPFS => slow random I/O (TSM) => tape

..but was the same speed as going through the tsm disk pool cache. We
closely monitored the network, disk, memory, cpu, on the tsm server and
none of the hardware or capabilities of the server were the bottleneck, it
was all in TSM.

If anyone has seen this sort of behavior and has some pointers/hints at
improving performance I'd be glad to hear it.

Thanks,
Sabuj


On Thu, Mar 20, 2014 at 5:21 PM, Grace Tsai <gtsai at slac.stanford.edu> wrote:

> Hi,
>
> Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?
>
> Thanks.
>
> Grace
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140320/83143bf7/attachment-0003.htm>

From oehmes at us.ibm.com  Fri Mar 21 21:39:57 2014
From: oehmes at us.ibm.com (Sven Oehme)
Date: Fri, 21 Mar 2014 14:39:57 -0700
Subject: [gpfsug-discuss] Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?
In-Reply-To: <CAEeMGHtFHY=8oJu-K2kQopK8LEAUB9DFTL6i++DrszOvvh3N1A@mail.gmail.com>
References: <532B6A06.3000207@slac.stanford.edu>
	<CAEeMGHtFHY=8oJu-K2kQopK8LEAUB9DFTL6i++DrszOvvh3N1A@mail.gmail.com>
Message-ID: <OF014D6BE3.885B54C0-ON88257CA2.0070C7EC-88257CA2.0077038F@us.ibm.com>

Hi,

there are various ways to speed up backup of data from GPFS to TSM (which 
btw is the same problem for most other backup solutions as well), but 
first one needs to find out what the problem is as it can be multiple 
completely independent issues and each of them needs a different solution, 
so let me explain the different issues and also what you can do about 
them.

the very first challenge is to find what data has changed. the way TSM 
does this is by crawling trough your filesystem, looking at mtime on each 
file to find out which file has changed. think about a ls -Rl on your 
filesystem root. this can, depending on how many files you have, take days 
in a large scale environment (think 100's of millions  of files). there is 
very little one can speed up on this process the way its done. all you can 
do is put metadata on faster disks (e.g. SSD) and that will improve the 
speed of this 'scan phase'. an alternative is to not do this scan at all 
with the TSM client, but instead let GPFS find out for TSM what files have 
changed and then share this information with TSM . the function/command in 
GPFS to do so is called mmbackup : 

https://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=%2Fcom.ibm.cluster.gpfs.v3r5.gpfs100.doc%2Fbl1adm_backupusingmmbackup.htm

it essentially traverses the GPFS metadata sequentially and in parallel 
across all nodes and filters out files that needs to be backed up. 
in several customer environments i was called to assist with issues like 
this, this change alone did speed up the backup process multiple orders of 
magnitudes. we had a few customers where this change reduced the scan time 
from days down to minutes. its not always this big, but its usually the 
largest chunk of the issue. 

the 2nd challenge is if you have to backup a very large number (millions) 
of very small (<32k) files. 
the main issue here is that for each file TSM issues a random i/o to GPFS, 
one at a time, so your throughput directly correlates with size of the 
files and latency for a single file read operation. if you are not on 3.5 
TL3 and/or your files don't fit into the inode its actually even 2 random 
i/os that are issued as you need to read the metadata followed by the data 
block for the file. 
in this scenario you can only do 2 things : 

1. parallelism - mmbackup again starts multiple processes in parallel to 
speed up this phase of the backup 
2. use a 'helper' process to prefetch data for a single TSM client so all 
data comes out of cache and the latency for the random reads is eliminated 
to increase throughput.

without any of this seeing only a few low MB/sec is not uncommon for 
customers, but with the changes above you are able to backup very large 
quantities of data. 

hope this helps. Sven

------------------------------------------
Sven Oehme 
Scalable Storage Research 
email: oehmes at us.ibm.com 
IBM Almaden Research Lab 
------------------------------------------



From:   Sabuj Pattanayek <sabujp at gmail.com>
To:     gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Date:   03/20/2014 06:39 PM
Subject:        Re: [gpfsug-discuss] Is TSM/HSM 7.1 compatible with GPFS 
3.5.0.12 ?
Sent by:        gpfsug-discuss-bounces at gpfsug.org



We're using tsm 7.1 with gpfs 3.5.0.11. At some point we do want to enable 
the HSM features but haven't had time to properly configure/set them up 
yet. I had dmapi enabled on GPFS but was never able to bring it up with 
dmapi enabled. Everything wasn't properly configured at the time and we 
were missing some pieces (not my post, but same issue) :

https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014622591

I'd say that we are having less than optimal performance with TSM however. 
We're only able to pull about 4TB a day. It took us 20 days to backup 80TB 
for our initial full. Using rsync/tar piped to tar would probably have 
taken less than a week. We tried various methods, e.g. using a fast 
intermediate diskpool, going simultaneously to our 6 LTO6 tape drives, etc 
but each "cp" (tsm client) process that TSM would use seemed to be very 
slow. We tweaked just about every setting to optimize performance but to 
really no avail. When going to the disk pool this is what should have 
happened :

GPFS => relatively fast random I/O (on par with rsync/tar piped to tar) 
tsm disk cache => large sequential I/O's for each disk pool volume => tape

this is what really happened

GPFS => slow random I/O => tsm disk pool cache => slow random I/O => tape

so instead we did :

GPFS => slow random I/O (TSM) => tape

..but was the same speed as going through the tsm disk pool cache. We 
closely monitored the network, disk, memory, cpu, on the tsm server and 
none of the hardware or capabilities of the server were the bottleneck, it 
was all in TSM.

If anyone has seen this sort of behavior and has some pointers/hints at 
improving performance I'd be glad to hear it.

Thanks,
Sabuj


On Thu, Mar 20, 2014 at 5:21 PM, Grace Tsai <gtsai at slac.stanford.edu> 
wrote:
Hi,

Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?

Thanks.

Grace
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140321/d1581e61/attachment-0003.htm>

From sabujp at gmail.com  Fri Mar 21 22:16:10 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Fri, 21 Mar 2014 17:16:10 -0500
Subject: [gpfsug-discuss] Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?
In-Reply-To: <OF014D6BE3.885B54C0-ON88257CA2.0070C7EC-88257CA2.0077038F@us.ibm.com>
References: <532B6A06.3000207@slac.stanford.edu>
	<CAEeMGHtFHY=8oJu-K2kQopK8LEAUB9DFTL6i++DrszOvvh3N1A@mail.gmail.com>
	<OF014D6BE3.885B54C0-ON88257CA2.0070C7EC-88257CA2.0077038F@us.ibm.com>
Message-ID: <CAEeMGHt63xJFLsvzDk46jZxcmH_9=E+4twtcKJnGzUr6mGPvpQ@mail.gmail.com>

> the very first challenge is to find what data has changed. the way TSM
> does this is by crawling trough your filesystem, looking at mtime on each
> file to find out which file has changed. think about a ls -Rl on your
> filesystem root. this


The mmbackup mmapplypolicy phase is not a problem. It's going to take as
long as it's going to take. We're using 5 RAID1 SAS SSD NSD's for metadata
and it takes ~1 hour to do the traversal through 157 million files.


>
> the 2nd challenge is if you have to backup a very large number (millions)
> of very small (<32k) files.
> the main issue here is that for each file TSM issues a random i/o to GPFS,
> one at a time, so your throughput directly correlates with size of the
> files and latency for a single file read operation. if you are not on 3.5
> TL3 and/or your files don't fit into the inode its actually even 2 random
> i/os that are issued as you need to read the metadata followed by the data
> block for the file.
> in this scenario you can only do 2 things :


The problem here is why is a single rsync or tar | tar process orders of
magnitude faster than a single tsm client at pulling data off of GPFS into
the same backup system's disk (e.g. disk pool)? It's not a problem with
GPFS, it's a problem with TSM itself. We tried various things, e.g. :

1) changed commmethod to sharedmem
2) increase txnbytelimit to 10G
3) increased movesizethresh to the same as txnbytelimit (10G)
4) increase diskbufsize to 1023kb
5) increased txngroupmax to 65000
6) increased movesizethresh to 10240

the next problem is that one would expect backups to tape to do straight
sequential I/O to tape, in the case of putting the files to the disk pool
before moving them to tape, it did the same random I/O to tape even with
8GB disk pool chunks. We haven't tried the file pool option yet, but we've
been told that it'll do the same thing. If I'm tar'ing or dd'ing large
files to tape that's the most efficient, why doesn't TSM do something
similar?


> 1. parallelism - mmbackup again starts multiple processes in parallel to
> speed up this phase of the backup


..use multiple clients. This would help, but again I'm trying to get a
single tsm client to be on par with a single "cp" process.

Thanks,
Sabuj
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140321/9a73f570/attachment-0003.htm>

From sabujp at gmail.com  Fri Mar 21 22:19:38 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Fri, 21 Mar 2014 17:19:38 -0500
Subject: [gpfsug-discuss] Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?
In-Reply-To: <CAEeMGHt63xJFLsvzDk46jZxcmH_9=E+4twtcKJnGzUr6mGPvpQ@mail.gmail.com>
References: <532B6A06.3000207@slac.stanford.edu>
	<CAEeMGHtFHY=8oJu-K2kQopK8LEAUB9DFTL6i++DrszOvvh3N1A@mail.gmail.com>
	<OF014D6BE3.885B54C0-ON88257CA2.0070C7EC-88257CA2.0077038F@us.ibm.com>
	<CAEeMGHt63xJFLsvzDk46jZxcmH_9=E+4twtcKJnGzUr6mGPvpQ@mail.gmail.com>
Message-ID: <CAEeMGHu3YmKq5F2fGiPDEkxOvSxOTQP-v+kSr7jguXJ0aHS2_g@mail.gmail.com>

It's also not a problem with the TSM database, which is sitting on a RAID10
of 4 SSDs. I'll watch dstat and it'll basically do a large 500MB/s write to
the LUN every 30 mins - hour and then just sit idle waiting for more data.


On Fri, Mar 21, 2014 at 5:16 PM, Sabuj Pattanayek <sabujp at gmail.com> wrote:

>
> the very first challenge is to find what data has changed. the way TSM
>> does this is by crawling trough your filesystem, looking at mtime on each
>> file to find out which file has changed. think about a ls -Rl on your
>> filesystem root. this
>
>
> The mmbackup mmapplypolicy phase is not a problem. It's going to take as
> long as it's going to take. We're using 5 RAID1 SAS SSD NSD's for metadata
> and it takes ~1 hour to do the traversal through 157 million files.
>
>
>>
>> the 2nd challenge is if you have to backup a very large number (millions)
>> of very small (<32k) files.
>> the main issue here is that for each file TSM issues a random i/o to
>> GPFS, one at a time, so your throughput directly correlates with size of
>> the files and latency for a single file read operation. if you are not on
>> 3.5 TL3 and/or your files don't fit into the inode its actually even 2
>> random i/os that are issued as you need to read the metadata followed by
>> the data block for the file.
>> in this scenario you can only do 2 things :
>
>
> The problem here is why is a single rsync or tar | tar process orders of
> magnitude faster than a single tsm client at pulling data off of GPFS into
> the same backup system's disk (e.g. disk pool)? It's not a problem with
> GPFS, it's a problem with TSM itself. We tried various things, e.g. :
>
> 1) changed commmethod to sharedmem
> 2) increase txnbytelimit to 10G
> 3) increased movesizethresh to the same as txnbytelimit (10G)
> 4) increase diskbufsize to 1023kb
> 5) increased txngroupmax to 65000
> 6) increased movesizethresh to 10240
>
> the next problem is that one would expect backups to tape to do straight
> sequential I/O to tape, in the case of putting the files to the disk pool
> before moving them to tape, it did the same random I/O to tape even with
> 8GB disk pool chunks. We haven't tried the file pool option yet, but we've
> been told that it'll do the same thing. If I'm tar'ing or dd'ing large
> files to tape that's the most efficient, why doesn't TSM do something
> similar?
>
>
>> 1. parallelism - mmbackup again starts multiple processes in parallel to
>> speed up this phase of the backup
>
>
> ..use multiple clients. This would help, but again I'm trying to get a
> single tsm client to be on par with a single "cp" process.
>
> Thanks,
> Sabuj
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140321/dcfd38d0/attachment-0003.htm>

From oehmes at us.ibm.com  Fri Mar 21 22:48:50 2014
From: oehmes at us.ibm.com (Sven Oehme)
Date: Fri, 21 Mar 2014 15:48:50 -0700
Subject: [gpfsug-discuss] Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?
In-Reply-To: <CAEeMGHt63xJFLsvzDk46jZxcmH_9=E+4twtcKJnGzUr6mGPvpQ@mail.gmail.com>
References: <532B6A06.3000207@slac.stanford.edu>	<CAEeMGHtFHY=8oJu-K2kQopK8LEAUB9DFTL6i++DrszOvvh3N1A@mail.gmail.com>
	<OF014D6BE3.885B54C0-ON88257CA2.0070C7EC-88257CA2.0077038F@us.ibm.com>
	<CAEeMGHt63xJFLsvzDk46jZxcmH_9=E+4twtcKJnGzUr6mGPvpQ@mail.gmail.com>
Message-ID: <OFC63E3472.65E21691-ON88257CA2.007B27D1-88257CA2.007D51FB@us.ibm.com>

Hi,

> the very first challenge is to find what data has changed. the way 
> TSM does this is by crawling trough your filesystem, looking at 
> mtime on each file to find out which file has changed. think about a
> ls -Rl on your filesystem root. this 
> 
> The mmbackup mmapplypolicy phase is not a problem. It's going to 
> take as long as it's going to take. We're using 5 RAID1 SAS SSD 
> NSD's for metadata and it takes ~1 hour to do the traversal through 
> 157 million files.
>  

that sounds too long, if you want i could take a look at your gpfs config 
and make suggestions on how to further improve this. 
if you want me to do that please send me a email with the output of 
mmlscluster,mmlsconfig,mmlsnsd and mmlsfs all to oehmes at us.ibm.com 

> 
> the 2nd challenge is if you have to backup a very large number 
> (millions) of very small (<32k) files. 
> the main issue here is that for each file TSM issues a random i/o to
> GPFS, one at a time, so your throughput directly correlates with 
> size of the files and latency for a single file read operation. if 
> you are not on 3.5 TL3 and/or your files don't fit into the inode 
> its actually even 2 random i/os that are issued as you need to read 
> the metadata followed by the data block for the file. 
> in this scenario you can only do 2 things : 
> 
> The problem here is why is a single rsync or tar | tar process 
> orders of magnitude faster than a single tsm client at pulling data 
> off of GPFS into the same backup system's disk (e.g. disk pool)? 
> It's not a problem with GPFS, it's a problem with TSM itself. 
> We tried various things, e.g. :
> 
> 1) changed commmethod to sharedmem
> 2) increase txnbytelimit to 10G
> 3) increased movesizethresh to the same as txnbytelimit (10G)
> 4) increase diskbufsize to 1023kb
> 5) increased txngroupmax to 65000
> 6) increased movesizethresh to 10240
> 
> the next problem is that one would expect backups to tape to do 
> straight sequential I/O to tape, in the case of putting the files to
> the disk pool before moving them to tape, it did the same random I/O
> to tape even with 8GB disk pool chunks. We haven't tried the file 
> pool option yet, but we've been told that it'll do the same thing. 

i assume you refer to using raw disks behind TSM ? i never used them, in 
fact the fastest TSM Storage you can build for TSM is to use a GPFS 
filesystem and put files into the filesystem and create a pool with 
sequential file volumes for TSM. if you match the gpfs blocksize with the 
raid blocksize you get very high throughputs to the TSM pool, i have 
customers who see >600 MB/sec backup throughput to a single TSM Server in 
production. 
> If I'm tar'ing or dd'ing large files to tape that's the most 
> efficient, why doesn't TSM do something similar?

i can't tell you much about the tape part of TSM but i can help you speed 
up the ingest into a TSM Server leveraging a Filesystem pool if you want.

> 
> 
> 1. parallelism - mmbackup again starts multiple processes in 
> parallel to speed up this phase of the backup 
> 
> ..use multiple clients. This would help, but again I'm trying to get
> a single tsm client to be on par with a single "cp" process.

cp uses buffered write operation and therefore will not sync the data to 
the disk, which is why its faster. TSM guarantees each write operation to 
to be committed to stable storage before it ever returns to the client, 
this is most likely the difference you see. 

don't get me wrong, i don't try to defend the way TSM works, i just know 
that there are several ways to make it fast and i am happy to help with 
that :-)

if you need a single TSM client to run faster, then option 2 (helper 
process) would fix that. if you want to try something out send me a email 
and i can help you with that. 

> 
> Thanks,
> Sabuj_______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140321/5747d594/attachment-0003.htm>