From jfosburg at mdanderson.org  Wed Oct  2 21:44:55 2013
From: jfosburg at mdanderson.org (Fosburgh,Jonathan)
Date: Wed, 2 Oct 2013 20:44:55 +0000
Subject: [gpfsug-discuss] Copying ACLs from outside sources
Message-ID: <CE71EFF4.200B4%jfosburg@mdanderson.org>

Hello, I'm new to this list, as well as GPFS.

We have new (installed over the last 6 weeks) GPFS cluster that we are currently trying to get into service.  The NSDs are running on Red Hat (6 NSDs), the storage is DCS3700 directly attached via SAS.  We are running GPFS 3.5.0.12.


Some of the data that will be placed on this cluster will be existing data coming from other sources, one of which is our NAS (Isilon, supporting both NFS and CIFS).  We have tried a few different ways of copying the data.  First we tried to rsync directly between an NSD and one of the Isilon nodes.  Unfortunately, it appears the rsync doesn't work with the ACL format used on the Isilon.  We have also tried using various methods of copying data via CIFS, but we continue to have no luck getting ACLs to copy.  We know that, for instance, copying data via CIFS into the Isilon using robocopy will preserve the ACLs, and we were hoping that would work for getting data onto GPFS as well.  However, the ACLs are being stripped and replaced with whatever ACL is present on GPFS.  Does anyone have experience doing something like that who can give me some pointers on what my configuration should look like? I assume there are samba and/or GPFS parameters that need to be tweaked in order to make this work, especially since this functionality would have to be present in SONAS.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131002/a1ef2992/attachment.htm>

From pete at realisestudio.com  Thu Oct  3 09:09:07 2013
From: pete at realisestudio.com (Pete Smith)
Date: Thu, 3 Oct 2013 09:09:07 +0100
Subject: [gpfsug-discuss] Copying ACLs from outside sources
In-Reply-To: <CE71EFF4.200B4%jfosburg@mdanderson.org>
References: <CE71EFF4.200B4%jfosburg@mdanderson.org>
Message-ID: <CAM9ZKkjHcSK9RgmLJKL=QedA=TN+6NhZZhHtWgpyLB=Za7S+EQ@mail.gmail.com>

Hi, and welcome, Jonathan.

Which combination of -A and -X rsync options did you use?

Pete

On 2 October 2013 21:44, Fosburgh,Jonathan <jfosburg at mdanderson.org> wrote:
> Hello, I'm new to this list, as well as GPFS.
>
> We have new (installed over the last 6 weeks) GPFS cluster that we are
> currently trying to get into service.  The NSDs are running on Red Hat (6
> NSDs), the storage is DCS3700 directly attached via SAS.  We are running
> GPFS 3.5.0.12.
>
>
> Some of the data that will be placed on this cluster will be existing data
> coming from other sources, one of which is our NAS (Isilon, supporting both
> NFS and CIFS).  We have tried a few different ways of copying the data.
> First we tried to rsync directly between an NSD and one of the Isilon nodes.
> Unfortunately, it appears the rsync doesn't work with the ACL format used on
> the Isilon.  We have also tried using various methods of copying data via
> CIFS, but we continue to have no luck getting ACLs to copy.  We know that,
> for instance, copying data via CIFS into the Isilon using robocopy will
> preserve the ACLs, and we were hoping that would work for getting data onto
> GPFS as well.  However, the ACLs are being stripped and replaced with
> whatever ACL is present on GPFS.  Does anyone have experience doing
> something like that who can give me some pointers on what my configuration
> should look like? I assume there are samba and/or GPFS parameters that need
> to be tweaked in order to make this work, especially since this
> functionality would have to be present in SONAS.
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>



-- 
Pete Smith
DevOp/System Administrator
Realise Studio
12/13 Poland Street, London W1F 8QB
T. +44 (0)20 7165 9644

realisestudio.com


From orlando.richards at ed.ac.uk  Thu Oct  3 09:17:47 2013
From: orlando.richards at ed.ac.uk (Orlando Richards)
Date: Thu, 03 Oct 2013 09:17:47 +0100
Subject: [gpfsug-discuss] Copying ACLs from outside sources
In-Reply-To: <CAM9ZKkjHcSK9RgmLJKL=QedA=TN+6NhZZhHtWgpyLB=Za7S+EQ@mail.gmail.com>
References: <CE71EFF4.200B4%jfosburg@mdanderson.org>
	<CAM9ZKkjHcSK9RgmLJKL=QedA=TN+6NhZZhHtWgpyLB=Za7S+EQ@mail.gmail.com>
Message-ID: <524D282B.905@ed.ac.uk>

Hi Jonathan,

Might be a bit of a long shot this - but for copying NFSv4 ACLs between 
GPFS filesystems using rsync, I had to patch rsync with some code 
released by IBM at some point. We've got that patched rsync hosted on 
the gpfsug github:

   https://github.com/gpfsug/gpfsug-tools/tree/master/bin/rsync

Since you're not doing gpfs -> gpfs, it might be no use - but you never 
know!

Oh - and, of course, make sure your GPFS filesystem is configured to 
support NFSv4 ACLs.

Here's our smb.conf settings related to GPFS, ACLs and other bits and bobs:

vfs objects = shadow_copy2, fileid, gpfs
clustering = yes
fileid:mapping = fsname
gpfs:sharemodes = No
gpfs:getrealfilename = no
nfs4: mode = simple
nfs4: chown = yes
nfs4: acedup = merge
force unknown acl user = yes
acl group control = no
map acl inherit = yes
inherit acls = no
dos filemode = no
map readonly = no
map hidden = no
map archive = no
map readonly = no
ea support = yes
store dos attributes = yes
gpfs:winattr = yes
max protocol = SMB2
shadow:fixinodes = yes
shadow:snapdir = .snapshots
shadow:basedir = /exports/nas
use sendfile = yes
wide links = no
unix extensions = no

Hope that helps,
Orlando

--
Orlando

On 03/10/13 09:09, Pete Smith wrote:
> Hi, and welcome, Jonathan.
>
> Which combination of -A and -X rsync options did you use?
>
> Pete
>
> On 2 October 2013 21:44, Fosburgh,Jonathan <jfosburg at mdanderson.org> wrote:
>> Hello, I'm new to this list, as well as GPFS.
>>
>> We have new (installed over the last 6 weeks) GPFS cluster that we are
>> currently trying to get into service.  The NSDs are running on Red Hat (6
>> NSDs), the storage is DCS3700 directly attached via SAS.  We are running
>> GPFS 3.5.0.12.
>>
>>
>> Some of the data that will be placed on this cluster will be existing data
>> coming from other sources, one of which is our NAS (Isilon, supporting both
>> NFS and CIFS).  We have tried a few different ways of copying the data.
>> First we tried to rsync directly between an NSD and one of the Isilon nodes.
>> Unfortunately, it appears the rsync doesn't work with the ACL format used on
>> the Isilon.  We have also tried using various methods of copying data via
>> CIFS, but we continue to have no luck getting ACLs to copy.  We know that,
>> for instance, copying data via CIFS into the Isilon using robocopy will
>> preserve the ACLs, and we were hoping that would work for getting data onto
>> GPFS as well.  However, the ACLs are being stripped and replaced with
>> whatever ACL is present on GPFS.  Does anyone have experience doing
>> something like that who can give me some pointers on what my configuration
>> should look like? I assume there are samba and/or GPFS parameters that need
>> to be tweaked in order to make this work, especially since this
>> functionality would have to be present in SONAS.
>>
>>
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at gpfsug.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>
>
>
>


-- 
             --
    Dr Orlando Richards
   Information Services
IT Infrastructure Division
        Unix Section
     Tel: 0131 650 4994
   skype: orlando.richards

The University of Edinburgh is a charitable body, registered in 
Scotland, with registration number SC005336.


From jonathan at buzzard.me.uk  Thu Oct  3 10:23:02 2013
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Thu, 03 Oct 2013 10:23:02 +0100
Subject: [gpfsug-discuss] Copying ACLs from outside sources
In-Reply-To: <CE71EFF4.200B4%jfosburg@mdanderson.org>
References: <CE71EFF4.200B4%jfosburg@mdanderson.org>
Message-ID: <1380792182.19439.18.camel@buzzard.phy.strath.ac.uk>

On Wed, 2013-10-02 at 20:44 +0000, Fosburgh,Jonathan wrote: 
> Hello, I'm new to this list, as well as GPFS.  
> 
> 
> We have new (installed over the last 6 weeks) GPFS cluster that we are
> currently trying to get into service.  The NSDs are running on Red Hat
> (6 NSDs), the storage is DCS3700 directly attached via SAS.  We are
> running GPFS 3.5.0.12.
> 
> Some of the data that will be placed on this cluster will be existing
> data coming from other sources, one of which is our NAS (Isilon,
> supporting both NFS and CIFS).  We have tried a few different ways of
> copying the data.  First we tried to rsync directly between an NSD and
> one of the Isilon nodes.  Unfortunately, it appears the rsync doesn't
> work with the ACL format used on the Isilon.  We have also tried using
> various methods of copying data via CIFS, but we continue to have no
> luck getting ACLs to copy.  We know that, for instance, copying data
> via CIFS into the Isilon using robocopy will preserve the ACLs, and we
> were hoping that would work for getting data onto GPFS as well.
> However, the ACLs are being stripped and replaced with whatever ACL is
> present on GPFS.  Does anyone have experience doing something like
> that who can give me some pointers on what my configuration should
> look like? I assume there are samba and/or GPFS parameters that need
> to be tweaked in order to make this work, especially since this
> functionality would have to be present in SONAS.
> 

I guess the first thing to do is confirm that you have functioning
CIFS's ACL's in your Samba GPFS file server. Specifically you would need
to have NFSv4 ACL's enabled in the file system and the gpfs VFS module
loaded in Samba. Note having mixed ACL modes did not work very well for
me, so best to turn off Posix ACL's and only do NFSv4 ACL's.

These are the samba options I use in combination with GPFS to make the
server as windows like as possible.

# general options
vfs objects = fileid gpfs
        unix extensions = no
        mangled names = no
        case sensitive = no

# store DOS attributes in extended attributes (vfs_gpfs then stores them
in the file system)
ea support = yes
store dos attributes = yes
map readonly = no
map archive = no
map system = no

# the ctdb clustering and GPFS stuff
clustering = yes
ctdbd socket = /tmp/ctdb.socket
        fileid : algorithm = fsname
        gpfs : sharemodes = yes
        gpfs : winattr = yes
        force unknown acl user = yes
        nfs4 : mode = special
        nfs4 : chown = no
        nfs4 : acedup = merge

The best way to make sure it is working is point a Windows client at the
file server and manipulate the ACL's by hand.

Note that if you are going for that full make my Samba/GPFS file server
look as close as possible to a pucker MS Windows server, you might want
to consider setting the following GPFS options

     cifsBypassShareLocksOnRename
     cifsBypassTraversalChecking
     allowWriteWithDeleteChild

All fairly self explanatory, and make GPFS follow Windows schematics
more closely, though they are "undocumented". There are a couple of
other options as allowSambaCaseInsensitiveLookup and
syncSambaMetadataOps but I have not determined exactly what they do...

There is also there is an undocumented option for ACL's on mmchfs 

      mmchfs test -k samba

Even shows up in the output of mmlsfs. Not entirely sure what samba
ACL's are mind you...

Note that IBM have alluded at various times that there these "magic"
options exist and are used in SONAS etc.

Anyway getting back to ACL's robocopy will move them over, but in my
experience copying from a Windows file server the results where not what
I wanted. I ended up with extra ACL's for administrators and bizzare
file ownership issues. Some of that might be because the files had
originally been on a Netware file server which died, and where hastily
moved to a Windows server. I gave up in the end and manually unpicked
what the ACL's where doing for the migration to the GPFS file server.
However I only had 20GB of data...

You should also find that xcopy will move them over as well.

A possible alternative if you have access to an AIX box would be to
NFSv4 mount the Isilon on the AIX box and mount GPFS as well and just
use cp.


JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.




From jfosburg at mdanderson.org  Wed Oct  9 18:31:45 2013
From: jfosburg at mdanderson.org (Fosburgh,Jonathan)
Date: Wed, 9 Oct 2013 17:31:45 +0000
Subject: [gpfsug-discuss] Copying ACLs from outside sources
In-Reply-To: <1380792182.19439.18.camel@buzzard.phy.strath.ac.uk>
Message-ID: <CE7AFC34.21539%jfosburg@mdanderson.org>



On 10/3/13 4:23 AM, "Jonathan Buzzard" <jonathan at buzzard.me.uk> wrote:

>
>> 
>
>I guess the first thing to do is confirm that you have functioning
>CIFS's ACL's in your Samba GPFS file server. Specifically you would need
>to have NFSv4 ACL's enabled in the file system and the gpfs VFS module
>loaded in Samba. Note having mixed ACL modes did not work very well for
>me, so best to turn off Posix ACL's and only do NFSv4 ACL's.
>
Thank you for the response. We have been able to make some headway on
this, but I am still bumping up on some issues.  It looks as if, when an
NFSv4 ACL is already present on the parent directory tree, copying acls
via robocopy works. However, if there is a posix ACL present, then the
ACLs are still not copied correctly.  I was interested in setting the ACL
mode to nfs4 only (currently set to all), however, I have some
reservations about this.

The NFS server is Red Hat, and I need to be able to server version 3 and
version 4, plus we will have native GPFS clients and the CIFS clients.
What are the ramifications, in this setup, of changing the ACL type? What
would happen to any existing ACLs?



From jonathan at buzzard.me.uk  Wed Oct  9 20:54:53 2013
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Wed, 09 Oct 2013 20:54:53 +0100
Subject: [gpfsug-discuss] Copying ACLs from outside sources
In-Reply-To: <CE7AFC34.21539%jfosburg@mdanderson.org>
References: <CE7AFC34.21539%jfosburg@mdanderson.org>
Message-ID: <5255B48D.1070401@buzzard.me.uk>

On 09/10/13 18:31, Fosburgh,Jonathan wrote:
>
>
> On 10/3/13 4:23 AM, "Jonathan Buzzard" <jonathan at buzzard.me.uk> wrote:
>
>>
>>>
>>
>> I guess the first thing to do is confirm that you have functioning
>> CIFS's ACL's in your Samba GPFS file server. Specifically you would need
>> to have NFSv4 ACL's enabled in the file system and the gpfs VFS module
>> loaded in Samba. Note having mixed ACL modes did not work very well for
>> me, so best to turn off Posix ACL's and only do NFSv4 ACL's.
>>
> Thank you for the response. We have been able to make some headway on
> this, but I am still bumping up on some issues.  It looks as if, when an
> NFSv4 ACL is already present on the parent directory tree, copying acls
> via robocopy works. However, if there is a posix ACL present, then the
> ACLs are still not copied correctly.  I was interested in setting the ACL
> mode to nfs4 only (currently set to all), however, I have some
> reservations about this.
>

Like I said in my experience having both Posix and NFSv4 ACL's active at 
the same time did not work properly as you have found out. You have to 
pick one and stick with it. Clearly if you have an existing system with 
both ACL types on then you have problems.

> The NFS server is Red Hat, and I need to be able to server version 3 and
> version 4, plus we will have native GPFS clients and the CIFS clients.
> What are the ramifications, in this setup, of changing the ACL type? What
> would happen to any existing ACLs?

My guess is the Posix ACL's either get lost or are "converted" into 
NFSv4 ACL's. I would try this out on your "development/test" GPFS setup 
first and work out a strategy for moving forward.

The biggest problem is that the IBM provided tools for manipulating the 
GPFS ACL's from within Linux are awful.


JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.


From crobson at ocf.co.uk  Tue Oct 22 17:27:24 2013
From: crobson at ocf.co.uk (Claire Robson)
Date: Tue, 22 Oct 2013 17:27:24 +0100
Subject: [gpfsug-discuss] SC'13 World Wide User Forum
Message-ID: <A42128435E851644B9B011BB824F6C816E24AEDFA7@MAIL.ocf.local>

Dear Members,

For anyone who is attending Supercomputing next month, there is a Worldwide GPFS User Forum taking place on Monday 18th November from 1-5pm. The GPFS User Forum is a worldwide gathering for current and new users of GPFS. Various topics will be discussed by current clients and IBM teams, including discussions on deployments, GPFS Storage Server, roadmaps and more!

https://www-950.ibm.com/events/wwe/grp/grp017.nsf/v17_enrollall?openform&seminar=D7QQ56ES&locale=en_US

Agenda
1:00       Welcome and Formalities              Andrew Dean, GPFS User Group and Dave McDonnell, IBM
1:15       ANL's GPFS GSS Storage System William Allcock, Argonne Leadership Computing Facility
1:45       GPFS Update Scott Fadden, IBM
2:30       Break Refreshments
2:45       GSS Update Matt Drahzal, IBM
3:30       NCAR's GLADE GPFS Environment Pamela Gillman, NCAR Computational Information Sciences Laboratory
4:00       GPFS Roadmap (NDA Required)    Krystal Rothaup, IBM (Session Moved to Centennial D for SPXXL Joint NDA Session)

Enjoy the event of you are able to make it. I hope to be in touch again soon with a date for the next UK meeting - watch this space!

Kind regards,

Claire Robson
GPFS User Group Secretary

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131022/2ccf16ee/attachment.htm>

From oester at gmail.com  Tue Oct 22 18:19:20 2013
From: oester at gmail.com (Bob Oesterlin)
Date: Tue, 22 Oct 2013 12:19:20 -0500
Subject: [gpfsug-discuss] SC'13 World Wide User Forum
In-Reply-To: <A42128435E851644B9B011BB824F6C816E24AEDFA7@MAIL.ocf.local>
References: <A42128435E851644B9B011BB824F6C816E24AEDFA7@MAIL.ocf.local>
Message-ID: <CAMNdFvB8_1WJvuZsyhbntU53s5oN9t3Kkx7H2UW83v3O0EQVbQ@mail.gmail.com>

Where is this event being held?


Bob Oesterlin
Nuance Communications



On Tue, Oct 22, 2013 at 11:27 AM, Claire Robson <crobson at ocf.co.uk> wrote:

> Dear Members,****
>
> ** **
>
> For anyone who is attending Supercomputing next month, there is a
> Worldwide GPFS User Forum taking place on Monday 18th November from
> 1-5pm. The GPFS User Forum is a worldwide gathering for current and new
> users of GPFS. Various topics will be discussed by current clients and IBM
> teams, including discussions on deployments, GPFS Storage Server, roadmaps
> and more! ****
>
> ** **
>
>
> https://www-950.ibm.com/events/wwe/grp/grp017.nsf/v17_enrollall?openform&seminar=D7QQ56ES&locale=en_US
> ****
>
> ** **
>
> Agenda****
>
> 1:00       Welcome and Formalities              Andrew Dean, GPFS User
> Group and Dave McDonnell, IBM ****
>
> 1:15       ANL's GPFS GSS Storage System William Allcock, Argonne
> Leadership Computing Facility ****
>
> 1:45       GPFS Update Scott Fadden, IBM****
>
> 2:30       Break Refreshments****
>
> 2:45       GSS Update Matt Drahzal, IBM****
>
> 3:30       NCAR's GLADE GPFS Environment Pamela Gillman, NCAR
> Computational Information Sciences Laboratory****
>
> 4:00       GPFS Roadmap (NDA Required)    Krystal Rothaup, IBM (Session
> Moved to Centennial D for SPXXL Joint NDA Session)****
>
> ** **
>
> Enjoy the event of you are able to make it. I hope to be in touch again
> soon with a date for the next UK meeting ? watch this space!****
>
> ** **
>
> Kind regards,****
>
> ** **
>
> *Claire Robson*
>
> GPFS User Group Secretary****
>
> ** **
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131022/13013ddd/attachment.htm>

From sfadden at us.ibm.com  Tue Oct 22 20:41:24 2013
From: sfadden at us.ibm.com (Scott Fadden)
Date: Tue, 22 Oct 2013 12:41:24 -0700
Subject: [gpfsug-discuss] SC'13 World Wide User Forum
In-Reply-To: <CAMNdFvB8_1WJvuZsyhbntU53s5oN9t3Kkx7H2UW83v3O0EQVbQ@mail.gmail.com>
References: <A42128435E851644B9B011BB824F6C816E24AEDFA7@MAIL.ocf.local>
	<CAMNdFvB8_1WJvuZsyhbntU53s5oN9t3Kkx7H2UW83v3O0EQVbQ@mail.gmail.com>
Message-ID: <OF0E566F5E.8E3EA689-ON88257C0C.006C2731-88257C0C.006C2938@us.ibm.com>



Hyatt Regency
650 15th Street
Denver, Colorado
United States


Scott Fadden
GPFS Technical Marketing
Phone: (503) 880-5833
sfadden at us.ibm.com
http://www.ibm.com/systems/gpfs


|------------>
| From:      |
|------------>
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
  |Bob Oesterlin <oester at gmail.com>                                                                                                               |
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| To:        |
|------------>
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
  |gpfsug main discussion list <gpfsug-discuss at gpfsug.org>,                                                                                       |
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Date:      |
|------------>
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
  |10/22/2013 10:28 AM                                                                                                                            |
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Subject:   |
|------------>
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
  |Re: [gpfsug-discuss] SC'13 World Wide User Forum                                                                                               |
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Sent by:   |
|------------>
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
  |gpfsug-discuss-bounces at gpfsug.org                                                                                                              |
  >-----------------------------------------------------------------------------------------------------------------------------------------------|





Where is this event being held?


Bob Oesterlin
Nuance Communications



On Tue, Oct 22, 2013 at 11:27 AM, Claire Robson <crobson at ocf.co.uk> wrote:
  Dear Members,





  For anyone who is attending Supercomputing next month, there is a
  Worldwide GPFS User Forum taking place on Monday 18th November from
  1-5pm. The GPFS User Forum is a worldwide gathering for current and new
  users of GPFS. Various topics will be discussed by current clients and
  IBM teams, including discussions on deployments, GPFS Storage Server,
  roadmaps and more!





  https://www-950.ibm.com/events/wwe/grp/grp017.nsf/v17_enrollall?openform&seminar=D7QQ56ES&locale=en_US





  Agenda


  1:00 ????? Welcome and Formalities????????????? Andrew Dean, GPFS User
  Group and Dave McDonnell, IBM


  1:15?????? ANL's GPFS GSS Storage System William Allcock, Argonne
  Leadership Computing Facility


  1:45?????? GPFS Update Scott Fadden, IBM


  2:30?????? Break Refreshments


  2:45?????? GSS Update Matt Drahzal, IBM


  3:30?????? NCAR's GLADE GPFS Environment Pamela Gillman, NCAR
  Computational Information Sciences Laboratory


  4:00?????? GPFS Roadmap (NDA Required)??? Krystal Rothaup, IBM (Session
  Moved to Centennial D for SPXXL Joint NDA Session)





  Enjoy the event of you are able to make it. I hope to be in touch again
  soon with a date for the next UK meeting ? watch this space!





  Kind regards,





  Claire Robson


  GPFS User Group Secretary




  _______________________________________________
  gpfsug-discuss mailing list
  gpfsug-discuss at gpfsug.org
  http://gpfsug.org/mailman/listinfo/gpfsug-discuss

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131022/140a00b7/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131022/140a00b7/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ecblank.gif
Type: image/gif
Size: 45 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131022/140a00b7/attachment-0001.gif>

From crobson at ocf.co.uk  Wed Oct 23 09:04:32 2013
From: crobson at ocf.co.uk (Claire Robson)
Date: Wed, 23 Oct 2013 09:04:32 +0100
Subject: [gpfsug-discuss] SC'13 World Wide User Forum
In-Reply-To: <CAMNdFvB8_1WJvuZsyhbntU53s5oN9t3Kkx7H2UW83v3O0EQVbQ@mail.gmail.com>
References: <A42128435E851644B9B011BB824F6C816E24AEDFA7@MAIL.ocf.local>
	<CAMNdFvB8_1WJvuZsyhbntU53s5oN9t3Kkx7H2UW83v3O0EQVbQ@mail.gmail.com>
Message-ID: <A42128435E851644B9B011BB824F6C816E24AEDFB8@MAIL.ocf.local>

Supercomputing is taking place at the Colorado Convention Center in Denver, Colorado, USA from 17-22nd November 2013.

Thanks,
Claire

From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Bob Oesterlin
Sent: 22 October 2013 18:19
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] SC'13 World Wide User Forum

Where is this event being held?


Bob Oesterlin
Nuance Communications


On Tue, Oct 22, 2013 at 11:27 AM, Claire Robson <crobson at ocf.co.uk<mailto:crobson at ocf.co.uk>> wrote:
Dear Members,

For anyone who is attending Supercomputing next month, there is a Worldwide GPFS User Forum taking place on Monday 18th November from 1-5pm. The GPFS User Forum is a worldwide gathering for current and new users of GPFS. Various topics will be discussed by current clients and IBM teams, including discussions on deployments, GPFS Storage Server, roadmaps and more!

https://www-950.ibm.com/events/wwe/grp/grp017.nsf/v17_enrollall?openform&seminar=D7QQ56ES&locale=en_US

Agenda
1:00       Welcome and Formalities              Andrew Dean, GPFS User Group and Dave McDonnell, IBM
1:15       ANL's GPFS GSS Storage System William Allcock, Argonne Leadership Computing Facility
1:45       GPFS Update Scott Fadden, IBM
2:30       Break Refreshments
2:45       GSS Update Matt Drahzal, IBM
3:30       NCAR's GLADE GPFS Environment Pamela Gillman, NCAR Computational Information Sciences Laboratory
4:00       GPFS Roadmap (NDA Required)    Krystal Rothaup, IBM (Session Moved to Centennial D for SPXXL Joint NDA Session)

Enjoy the event of you are able to make it. I hope to be in touch again soon with a date for the next UK meeting - watch this space!

Kind regards,

Claire Robson
GPFS User Group Secretary


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org<http://gpfsug.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131023/083b2225/attachment.htm>

From chair at gpfsug.org  Fri Oct 25 17:09:16 2013
From: chair at gpfsug.org (Chair GPFS UG)
Date: Fri, 25 Oct 2013 17:09:16 +0100
Subject: [gpfsug-discuss] Friday MPI fun?
Message-ID: <CA+M26=T7eeSMWhV_GFFvEkaKnDHNxF9OXqHbg4DvfVhnWN8CfA@mail.gmail.com>

Allo all,

  I'm attempting to cheat.  As per-usual, cheating takes more time than
'Doing It Properly' - but it is vastly more fun.

So without setting up Grid or Moab etc, I need to pin processes to a cpu.
 I.E. on Linux: taskset blah blah.
I could write a small housekeeping script which RR new spawned processes
across CPUs using taskset, but I was wondering if OpenMPI could be a good
way to go.

So:

I have a non-MPI application X.
Which is spawned and forked by a parent process into its own process group.
This can occur at any time, however there will only ever be a maximium N of
appl X.

Using mpirun it appears that you can set off parallel instances of a
non-MPI application:

mpirun -np 4 applicationX

However, really what I'm requiring to do is say:
  Max slots = N (which I can define in the mpi hostfile).

mpirun -np 1 applicationX
mpirun -np 1 applicationX (started at a random future time)
mpirun -np 1 applicationX (started at a random future time)
mpirun -np 1 applicationX (started at a random future time)

Each being automatically pinned to a CPU, but I'm fairly convinced this is
not the way MPI works.
Would it do what I'm after?

Does anyone know of a better way?

Jez
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131025/37f6a4c9/attachment.htm>

From zgiles at gmail.com  Sat Oct 26 05:13:37 2013
From: zgiles at gmail.com (Zachary Giles)
Date: Sat, 26 Oct 2013 00:13:37 -0400
Subject: [gpfsug-discuss] Friday MPI fun?
In-Reply-To: <CA+M26=T7eeSMWhV_GFFvEkaKnDHNxF9OXqHbg4DvfVhnWN8CfA@mail.gmail.com>
References: <CA+M26=T7eeSMWhV_GFFvEkaKnDHNxF9OXqHbg4DvfVhnWN8CfA@mail.gmail.com>
Message-ID: <CAMYZk=esiDC4fq3DnqngyttsvvYCSBNOO2vZBsF=uJdT=fcU2Q@mail.gmail.com>

You could do it that way, if you told MPI to bind each task (MPI task, or
non-MPI task in your case) to a processor. By default, most MPI's don't
bind to a specific processor unless the scheduler (you don't have one) or
something else tells it to. OpenMPI, for example, I think, has something
like --bind-to-cpu, or --bind-to-socket. Often a scheduler or c-groups via
a scheduler will pass-in a list of cpu's to bind to, but you could do it
manually with the appropriate flags and files for your MPI distro.

You could also probably get away with just raw c-groups depending on your
distro. You would probably touch a few files and echo a few lines to files
to make the right groups then "cgrun <app>" to get things in to those
groups. That works pretty well for binding, found decent results with that,
I recommend it.


On Fri, Oct 25, 2013 at 12:09 PM, Chair GPFS UG <chair at gpfsug.org> wrote:

> Allo all,
>
>   I'm attempting to cheat.  As per-usual, cheating takes more time than
> 'Doing It Properly' - but it is vastly more fun.
>
> So without setting up Grid or Moab etc, I need to pin processes to a cpu.
>  I.E. on Linux: taskset blah blah.
> I could write a small housekeeping script which RR new spawned processes
> across CPUs using taskset, but I was wondering if OpenMPI could be a good
> way to go.
>
> So:
>
> I have a non-MPI application X.
> Which is spawned and forked by a parent process into its own process group.
> This can occur at any time, however there will only ever be a maximium N
> of appl X.
>
> Using mpirun it appears that you can set off parallel instances of a
> non-MPI application:
>
> mpirun -np 4 applicationX
>
> However, really what I'm requiring to do is say:
>   Max slots = N (which I can define in the mpi hostfile).
>
> mpirun -np 1 applicationX
> mpirun -np 1 applicationX (started at a random future time)
> mpirun -np 1 applicationX (started at a random future time)
> mpirun -np 1 applicationX (started at a random future time)
>
> Each being automatically pinned to a CPU, but I'm fairly convinced this is
> not the way MPI works.
> Would it do what I'm after?
>
> Does anyone know of a better way?
>
> Jez
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>


-- 
Zach Giles
zgiles at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131026/217d585e/attachment.htm>

From richard.lefebvre at calculquebec.ca  Mon Oct 28 15:31:07 2013
From: richard.lefebvre at calculquebec.ca (Richard Lefebvre)
Date: Mon, 28 Oct 2013 11:31:07 -0400
Subject: [gpfsug-discuss] Moving/copying files from one file system to
	another
Message-ID: <526E833B.6040402@calculquebec.ca>

Hi,

I'm looking to copy/move files from one gpfs file system to another in
some efficient way. Initially I was thinking to not create another file
system and just add the nsd to the current file system and gradually
remove the old nsd. That would have been the easiest with little impact
to the users. But I wanted to switch from a scatter allocation to a
cluster allocation (too improve performance) and I cannot to that if I
just add/remove nsd. I'm hopping for a better solution then a plain
rsync which would probably take a few days (~210TB).

Is there a gpfs solution to this?

Also, another question, under what condition a scatter allocation better
then cluster allocation. We currently have a cluster of 650 nodes all
accessing the same 230TB gpfs file system.

Richard


From seanlee at tw.ibm.com  Wed Oct 30 12:30:26 2013
From: seanlee at tw.ibm.com (Sean S Lee)
Date: Wed, 30 Oct 2013 20:30:26 +0800
Subject: [gpfsug-discuss] Moving/copying files from one file system to
	another
In-Reply-To: <mailman.1.1383134402.20340.gpfsug-discuss@gpfsug.org>
References: <mailman.1.1383134402.20340.gpfsug-discuss@gpfsug.org>
Message-ID: <OF9B62D9AF.094922A5-ON48257C14.00438E14-48257C14.0044C452@tw.ibm.com>


Hi Richard,

==========

The cluster allocation method may provide better disk performance for some
disk subsystems in
relatively small installations. The benefits of clustered block allocation
diminish when the number
of nodes in the cluster or the number of disks in a file system increases,
or when the file system's
free space becomes fragmented. The cluster allocation method is the default
for GPFS clusters
with eight or fewer nodes and for file systems with eight or fewer disks.

The scatter allocation method provides more consistent file system
performance by averaging out
performance variations due to block location (for many disk subsystems, the
location of the data
relative to the disk edge has a substantial effect on performance). This
allocation method is
appropriate in most cases and is the default for GPFS clusters with more
than eight nodes or file
systems with more than eight disks.

Source: GPFS Administration and Programming Reference
==========

Based on above it seems the scatter allocation would be more suitable for
your environment (more nodes, more disks), no?

The Web forum has few threads on GPFS copying, e.g. take a look at this:
https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014934284&ps=25

I don't know if there's a better way or not.

If you don't change the allocation type, then you don't have to do anything
but add the disks and perhaps restripe.

Regards
Sean
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131030/5c77c624/attachment.htm>

From jonathan at buzzard.me.uk  Wed Oct 30 12:47:07 2013
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Wed, 30 Oct 2013 12:47:07 +0000
Subject: [gpfsug-discuss] Moving/copying files from one file system to
 another
In-Reply-To: <526E833B.6040402@calculquebec.ca>
References: <526E833B.6040402@calculquebec.ca>
Message-ID: <1383137227.2921.4.camel@buzzard.phy.strath.ac.uk>

On Mon, 2013-10-28 at 11:31 -0400, Richard Lefebvre wrote:

[SNIP]

> Also, another question, under what condition a scatter allocation better
> then cluster allocation. We currently have a cluster of 650 nodes all
> accessing the same 230TB gpfs file system.
> 

Scatter allocation is better in almost all circumstances. Basically by
scattering the files to all corners you don't get hotspots where just a
small subset of the disks are being hammered by lots of accesses to a
handful of files, while the rest of the disks sit idle.

JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.



From chekh at stanford.edu  Wed Oct 30 17:56:46 2013
From: chekh at stanford.edu (Alex Chekholko)
Date: Wed, 30 Oct 2013 10:56:46 -0700
Subject: [gpfsug-discuss] Moving/copying files from one file system to
 another
In-Reply-To: <1383137227.2921.4.camel@buzzard.phy.strath.ac.uk>
References: <526E833B.6040402@calculquebec.ca>
	<1383137227.2921.4.camel@buzzard.phy.strath.ac.uk>
Message-ID: <5271485E.80405@stanford.edu>

On 10/30/13, 5:47 AM, Jonathan Buzzard wrote:
> On Mon, 2013-10-28 at 11:31 -0400, Richard Lefebvre wrote:
>
> [SNIP]
>
>> Also, another question, under what condition a scatter allocation better
>> then cluster allocation. We currently have a cluster of 650 nodes all
>> accessing the same 230TB gpfs file system.
>>
>
> Scatter allocation is better in almost all circumstances. Basically by
> scattering the files to all corners you don't get hotspots where just a
> small subset of the disks are being hammered by lots of accesses to a
> handful of files, while the rest of the disks sit idle.
>

If you do benchmarks with only a few threads, you will see higher 
performance with 'cluster' allocation.  So if your workload is only a 
few clients accessing the FS in a mostly streaming way, you'd see better 
performance from 'cluster'.

With 650 nodes, even if each client is doing streaming reads, at the 
filesystem level that would all be interleaved and thus be random reads. 
  But it's tough to do a big enough benchmark to show the difference in 
performance.

I had a tough time convincing people to use 'scatter' instead of 
'cluster' even though I think the documentation is clear about the 
difference, and even gives you the sizing parameters ( greater than 8 
disks or 8 NSDs?  use 'scatter').

We use 'scatter' now.

Regards
-- 
chekh at stanford.edu


From TOMP at il.ibm.com  Wed Oct 30 18:31:01 2013
From: TOMP at il.ibm.com (Tomer Perry)
Date: Wed, 30 Oct 2013 20:31:01 +0200
Subject: [gpfsug-discuss] Moving/copying files from one file system to
	another
In-Reply-To: <5271485E.80405@stanford.edu>
References: <526E833B.6040402@calculquebec.ca>	<1383137227.2921.4.camel@buzzard.phy.strath.ac.uk>
	<5271485E.80405@stanford.edu>
Message-ID: <OF39DC62FE.1C8ABCF9-ONC2257C14.00657D38-C2257C14.0065BC4C@il.ibm.com>

Hi,

cluster will also be there for some time, as soon as one start deleting 
and creating file - it will kind of shift to be scattered...
So, while its good for benchmarks on "clean and new" system - cluster is 
somewhat meaningless ( and was hidden for some time because of that).

As far as it goes for copying from old to new system - AFM can be 
considered as well ( will make the transition easier). You can prefetch ( 
using mmafmctl) and even work in LU mode in order to get the data from the 
old FS to the new one without pushing changes back.

hth,
Tomer.




From:   Alex Chekholko <chekh at stanford.edu>
To:     gpfsug-discuss at gpfsug.org, 
Date:   10/30/2013 07:57 PM
Subject:        Re: [gpfsug-discuss] Moving/copying files from one file 
system to another
Sent by:        gpfsug-discuss-bounces at gpfsug.org



On 10/30/13, 5:47 AM, Jonathan Buzzard wrote:
> On Mon, 2013-10-28 at 11:31 -0400, Richard Lefebvre wrote:
>
> [SNIP]
>
>> Also, another question, under what condition a scatter allocation 
better
>> then cluster allocation. We currently have a cluster of 650 nodes all
>> accessing the same 230TB gpfs file system.
>>
>
> Scatter allocation is better in almost all circumstances. Basically by
> scattering the files to all corners you don't get hotspots where just a
> small subset of the disks are being hammered by lots of accesses to a
> handful of files, while the rest of the disks sit idle.
>

If you do benchmarks with only a few threads, you will see higher 
performance with 'cluster' allocation.  So if your workload is only a 
few clients accessing the FS in a mostly streaming way, you'd see better 
performance from 'cluster'.

With 650 nodes, even if each client is doing streaming reads, at the 
filesystem level that would all be interleaved and thus be random reads. 
  But it's tough to do a big enough benchmark to show the difference in 
performance.

I had a tough time convincing people to use 'scatter' instead of 
'cluster' even though I think the documentation is clear about the 
difference, and even gives you the sizing parameters ( greater than 8 
disks or 8 NSDs?  use 'scatter').

We use 'scatter' now.

Regards
-- 
chekh at stanford.edu
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131030/d130f587/attachment.htm>

From jfosburg at mdanderson.org  Wed Oct  2 21:44:55 2013
From: jfosburg at mdanderson.org (Fosburgh,Jonathan)
Date: Wed, 2 Oct 2013 20:44:55 +0000
Subject: [gpfsug-discuss] Copying ACLs from outside sources
Message-ID: <CE71EFF4.200B4%jfosburg@mdanderson.org>

Hello, I'm new to this list, as well as GPFS.

We have new (installed over the last 6 weeks) GPFS cluster that we are currently trying to get into service.  The NSDs are running on Red Hat (6 NSDs), the storage is DCS3700 directly attached via SAS.  We are running GPFS 3.5.0.12.


Some of the data that will be placed on this cluster will be existing data coming from other sources, one of which is our NAS (Isilon, supporting both NFS and CIFS).  We have tried a few different ways of copying the data.  First we tried to rsync directly between an NSD and one of the Isilon nodes.  Unfortunately, it appears the rsync doesn't work with the ACL format used on the Isilon.  We have also tried using various methods of copying data via CIFS, but we continue to have no luck getting ACLs to copy.  We know that, for instance, copying data via CIFS into the Isilon using robocopy will preserve the ACLs, and we were hoping that would work for getting data onto GPFS as well.  However, the ACLs are being stripped and replaced with whatever ACL is present on GPFS.  Does anyone have experience doing something like that who can give me some pointers on what my configuration should look like? I assume there are samba and/or GPFS parameters that need to be tweaked in order to make this work, especially since this functionality would have to be present in SONAS.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131002/a1ef2992/attachment-0001.htm>

From pete at realisestudio.com  Thu Oct  3 09:09:07 2013
From: pete at realisestudio.com (Pete Smith)
Date: Thu, 3 Oct 2013 09:09:07 +0100
Subject: [gpfsug-discuss] Copying ACLs from outside sources
In-Reply-To: <CE71EFF4.200B4%jfosburg@mdanderson.org>
References: <CE71EFF4.200B4%jfosburg@mdanderson.org>
Message-ID: <CAM9ZKkjHcSK9RgmLJKL=QedA=TN+6NhZZhHtWgpyLB=Za7S+EQ@mail.gmail.com>

Hi, and welcome, Jonathan.

Which combination of -A and -X rsync options did you use?

Pete

On 2 October 2013 21:44, Fosburgh,Jonathan <jfosburg at mdanderson.org> wrote:
> Hello, I'm new to this list, as well as GPFS.
>
> We have new (installed over the last 6 weeks) GPFS cluster that we are
> currently trying to get into service.  The NSDs are running on Red Hat (6
> NSDs), the storage is DCS3700 directly attached via SAS.  We are running
> GPFS 3.5.0.12.
>
>
> Some of the data that will be placed on this cluster will be existing data
> coming from other sources, one of which is our NAS (Isilon, supporting both
> NFS and CIFS).  We have tried a few different ways of copying the data.
> First we tried to rsync directly between an NSD and one of the Isilon nodes.
> Unfortunately, it appears the rsync doesn't work with the ACL format used on
> the Isilon.  We have also tried using various methods of copying data via
> CIFS, but we continue to have no luck getting ACLs to copy.  We know that,
> for instance, copying data via CIFS into the Isilon using robocopy will
> preserve the ACLs, and we were hoping that would work for getting data onto
> GPFS as well.  However, the ACLs are being stripped and replaced with
> whatever ACL is present on GPFS.  Does anyone have experience doing
> something like that who can give me some pointers on what my configuration
> should look like? I assume there are samba and/or GPFS parameters that need
> to be tweaked in order to make this work, especially since this
> functionality would have to be present in SONAS.
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>



-- 
Pete Smith
DevOp/System Administrator
Realise Studio
12/13 Poland Street, London W1F 8QB
T. +44 (0)20 7165 9644

realisestudio.com


From orlando.richards at ed.ac.uk  Thu Oct  3 09:17:47 2013
From: orlando.richards at ed.ac.uk (Orlando Richards)
Date: Thu, 03 Oct 2013 09:17:47 +0100
Subject: [gpfsug-discuss] Copying ACLs from outside sources
In-Reply-To: <CAM9ZKkjHcSK9RgmLJKL=QedA=TN+6NhZZhHtWgpyLB=Za7S+EQ@mail.gmail.com>
References: <CE71EFF4.200B4%jfosburg@mdanderson.org>
	<CAM9ZKkjHcSK9RgmLJKL=QedA=TN+6NhZZhHtWgpyLB=Za7S+EQ@mail.gmail.com>
Message-ID: <524D282B.905@ed.ac.uk>

Hi Jonathan,

Might be a bit of a long shot this - but for copying NFSv4 ACLs between 
GPFS filesystems using rsync, I had to patch rsync with some code 
released by IBM at some point. We've got that patched rsync hosted on 
the gpfsug github:

   https://github.com/gpfsug/gpfsug-tools/tree/master/bin/rsync

Since you're not doing gpfs -> gpfs, it might be no use - but you never 
know!

Oh - and, of course, make sure your GPFS filesystem is configured to 
support NFSv4 ACLs.

Here's our smb.conf settings related to GPFS, ACLs and other bits and bobs:

vfs objects = shadow_copy2, fileid, gpfs
clustering = yes
fileid:mapping = fsname
gpfs:sharemodes = No
gpfs:getrealfilename = no
nfs4: mode = simple
nfs4: chown = yes
nfs4: acedup = merge
force unknown acl user = yes
acl group control = no
map acl inherit = yes
inherit acls = no
dos filemode = no
map readonly = no
map hidden = no
map archive = no
map readonly = no
ea support = yes
store dos attributes = yes
gpfs:winattr = yes
max protocol = SMB2
shadow:fixinodes = yes
shadow:snapdir = .snapshots
shadow:basedir = /exports/nas
use sendfile = yes
wide links = no
unix extensions = no

Hope that helps,
Orlando

--
Orlando

On 03/10/13 09:09, Pete Smith wrote:
> Hi, and welcome, Jonathan.
>
> Which combination of -A and -X rsync options did you use?
>
> Pete
>
> On 2 October 2013 21:44, Fosburgh,Jonathan <jfosburg at mdanderson.org> wrote:
>> Hello, I'm new to this list, as well as GPFS.
>>
>> We have new (installed over the last 6 weeks) GPFS cluster that we are
>> currently trying to get into service.  The NSDs are running on Red Hat (6
>> NSDs), the storage is DCS3700 directly attached via SAS.  We are running
>> GPFS 3.5.0.12.
>>
>>
>> Some of the data that will be placed on this cluster will be existing data
>> coming from other sources, one of which is our NAS (Isilon, supporting both
>> NFS and CIFS).  We have tried a few different ways of copying the data.
>> First we tried to rsync directly between an NSD and one of the Isilon nodes.
>> Unfortunately, it appears the rsync doesn't work with the ACL format used on
>> the Isilon.  We have also tried using various methods of copying data via
>> CIFS, but we continue to have no luck getting ACLs to copy.  We know that,
>> for instance, copying data via CIFS into the Isilon using robocopy will
>> preserve the ACLs, and we were hoping that would work for getting data onto
>> GPFS as well.  However, the ACLs are being stripped and replaced with
>> whatever ACL is present on GPFS.  Does anyone have experience doing
>> something like that who can give me some pointers on what my configuration
>> should look like? I assume there are samba and/or GPFS parameters that need
>> to be tweaked in order to make this work, especially since this
>> functionality would have to be present in SONAS.
>>
>>
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at gpfsug.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>
>
>
>


-- 
             --
    Dr Orlando Richards
   Information Services
IT Infrastructure Division
        Unix Section
     Tel: 0131 650 4994
   skype: orlando.richards

The University of Edinburgh is a charitable body, registered in 
Scotland, with registration number SC005336.


From jonathan at buzzard.me.uk  Thu Oct  3 10:23:02 2013
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Thu, 03 Oct 2013 10:23:02 +0100
Subject: [gpfsug-discuss] Copying ACLs from outside sources
In-Reply-To: <CE71EFF4.200B4%jfosburg@mdanderson.org>
References: <CE71EFF4.200B4%jfosburg@mdanderson.org>
Message-ID: <1380792182.19439.18.camel@buzzard.phy.strath.ac.uk>

On Wed, 2013-10-02 at 20:44 +0000, Fosburgh,Jonathan wrote: 
> Hello, I'm new to this list, as well as GPFS.  
> 
> 
> We have new (installed over the last 6 weeks) GPFS cluster that we are
> currently trying to get into service.  The NSDs are running on Red Hat
> (6 NSDs), the storage is DCS3700 directly attached via SAS.  We are
> running GPFS 3.5.0.12.
> 
> Some of the data that will be placed on this cluster will be existing
> data coming from other sources, one of which is our NAS (Isilon,
> supporting both NFS and CIFS).  We have tried a few different ways of
> copying the data.  First we tried to rsync directly between an NSD and
> one of the Isilon nodes.  Unfortunately, it appears the rsync doesn't
> work with the ACL format used on the Isilon.  We have also tried using
> various methods of copying data via CIFS, but we continue to have no
> luck getting ACLs to copy.  We know that, for instance, copying data
> via CIFS into the Isilon using robocopy will preserve the ACLs, and we
> were hoping that would work for getting data onto GPFS as well.
> However, the ACLs are being stripped and replaced with whatever ACL is
> present on GPFS.  Does anyone have experience doing something like
> that who can give me some pointers on what my configuration should
> look like? I assume there are samba and/or GPFS parameters that need
> to be tweaked in order to make this work, especially since this
> functionality would have to be present in SONAS.
> 

I guess the first thing to do is confirm that you have functioning
CIFS's ACL's in your Samba GPFS file server. Specifically you would need
to have NFSv4 ACL's enabled in the file system and the gpfs VFS module
loaded in Samba. Note having mixed ACL modes did not work very well for
me, so best to turn off Posix ACL's and only do NFSv4 ACL's.

These are the samba options I use in combination with GPFS to make the
server as windows like as possible.

# general options
vfs objects = fileid gpfs
        unix extensions = no
        mangled names = no
        case sensitive = no

# store DOS attributes in extended attributes (vfs_gpfs then stores them
in the file system)
ea support = yes
store dos attributes = yes
map readonly = no
map archive = no
map system = no

# the ctdb clustering and GPFS stuff
clustering = yes
ctdbd socket = /tmp/ctdb.socket
        fileid : algorithm = fsname
        gpfs : sharemodes = yes
        gpfs : winattr = yes
        force unknown acl user = yes
        nfs4 : mode = special
        nfs4 : chown = no
        nfs4 : acedup = merge

The best way to make sure it is working is point a Windows client at the
file server and manipulate the ACL's by hand.

Note that if you are going for that full make my Samba/GPFS file server
look as close as possible to a pucker MS Windows server, you might want
to consider setting the following GPFS options

     cifsBypassShareLocksOnRename
     cifsBypassTraversalChecking
     allowWriteWithDeleteChild

All fairly self explanatory, and make GPFS follow Windows schematics
more closely, though they are "undocumented". There are a couple of
other options as allowSambaCaseInsensitiveLookup and
syncSambaMetadataOps but I have not determined exactly what they do...

There is also there is an undocumented option for ACL's on mmchfs 

      mmchfs test -k samba

Even shows up in the output of mmlsfs. Not entirely sure what samba
ACL's are mind you...

Note that IBM have alluded at various times that there these "magic"
options exist and are used in SONAS etc.

Anyway getting back to ACL's robocopy will move them over, but in my
experience copying from a Windows file server the results where not what
I wanted. I ended up with extra ACL's for administrators and bizzare
file ownership issues. Some of that might be because the files had
originally been on a Netware file server which died, and where hastily
moved to a Windows server. I gave up in the end and manually unpicked
what the ACL's where doing for the migration to the GPFS file server.
However I only had 20GB of data...

You should also find that xcopy will move them over as well.

A possible alternative if you have access to an AIX box would be to
NFSv4 mount the Isilon on the AIX box and mount GPFS as well and just
use cp.


JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.




From jfosburg at mdanderson.org  Wed Oct  9 18:31:45 2013
From: jfosburg at mdanderson.org (Fosburgh,Jonathan)
Date: Wed, 9 Oct 2013 17:31:45 +0000
Subject: [gpfsug-discuss] Copying ACLs from outside sources
In-Reply-To: <1380792182.19439.18.camel@buzzard.phy.strath.ac.uk>
Message-ID: <CE7AFC34.21539%jfosburg@mdanderson.org>



On 10/3/13 4:23 AM, "Jonathan Buzzard" <jonathan at buzzard.me.uk> wrote:

>
>> 
>
>I guess the first thing to do is confirm that you have functioning
>CIFS's ACL's in your Samba GPFS file server. Specifically you would need
>to have NFSv4 ACL's enabled in the file system and the gpfs VFS module
>loaded in Samba. Note having mixed ACL modes did not work very well for
>me, so best to turn off Posix ACL's and only do NFSv4 ACL's.
>
Thank you for the response. We have been able to make some headway on
this, but I am still bumping up on some issues.  It looks as if, when an
NFSv4 ACL is already present on the parent directory tree, copying acls
via robocopy works. However, if there is a posix ACL present, then the
ACLs are still not copied correctly.  I was interested in setting the ACL
mode to nfs4 only (currently set to all), however, I have some
reservations about this.

The NFS server is Red Hat, and I need to be able to server version 3 and
version 4, plus we will have native GPFS clients and the CIFS clients.
What are the ramifications, in this setup, of changing the ACL type? What
would happen to any existing ACLs?



From jonathan at buzzard.me.uk  Wed Oct  9 20:54:53 2013
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Wed, 09 Oct 2013 20:54:53 +0100
Subject: [gpfsug-discuss] Copying ACLs from outside sources
In-Reply-To: <CE7AFC34.21539%jfosburg@mdanderson.org>
References: <CE7AFC34.21539%jfosburg@mdanderson.org>
Message-ID: <5255B48D.1070401@buzzard.me.uk>

On 09/10/13 18:31, Fosburgh,Jonathan wrote:
>
>
> On 10/3/13 4:23 AM, "Jonathan Buzzard" <jonathan at buzzard.me.uk> wrote:
>
>>
>>>
>>
>> I guess the first thing to do is confirm that you have functioning
>> CIFS's ACL's in your Samba GPFS file server. Specifically you would need
>> to have NFSv4 ACL's enabled in the file system and the gpfs VFS module
>> loaded in Samba. Note having mixed ACL modes did not work very well for
>> me, so best to turn off Posix ACL's and only do NFSv4 ACL's.
>>
> Thank you for the response. We have been able to make some headway on
> this, but I am still bumping up on some issues.  It looks as if, when an
> NFSv4 ACL is already present on the parent directory tree, copying acls
> via robocopy works. However, if there is a posix ACL present, then the
> ACLs are still not copied correctly.  I was interested in setting the ACL
> mode to nfs4 only (currently set to all), however, I have some
> reservations about this.
>

Like I said in my experience having both Posix and NFSv4 ACL's active at 
the same time did not work properly as you have found out. You have to 
pick one and stick with it. Clearly if you have an existing system with 
both ACL types on then you have problems.

> The NFS server is Red Hat, and I need to be able to server version 3 and
> version 4, plus we will have native GPFS clients and the CIFS clients.
> What are the ramifications, in this setup, of changing the ACL type? What
> would happen to any existing ACLs?

My guess is the Posix ACL's either get lost or are "converted" into 
NFSv4 ACL's. I would try this out on your "development/test" GPFS setup 
first and work out a strategy for moving forward.

The biggest problem is that the IBM provided tools for manipulating the 
GPFS ACL's from within Linux are awful.


JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.


From crobson at ocf.co.uk  Tue Oct 22 17:27:24 2013
From: crobson at ocf.co.uk (Claire Robson)
Date: Tue, 22 Oct 2013 17:27:24 +0100
Subject: [gpfsug-discuss] SC'13 World Wide User Forum
Message-ID: <A42128435E851644B9B011BB824F6C816E24AEDFA7@MAIL.ocf.local>

Dear Members,

For anyone who is attending Supercomputing next month, there is a Worldwide GPFS User Forum taking place on Monday 18th November from 1-5pm. The GPFS User Forum is a worldwide gathering for current and new users of GPFS. Various topics will be discussed by current clients and IBM teams, including discussions on deployments, GPFS Storage Server, roadmaps and more!

https://www-950.ibm.com/events/wwe/grp/grp017.nsf/v17_enrollall?openform&seminar=D7QQ56ES&locale=en_US

Agenda
1:00       Welcome and Formalities              Andrew Dean, GPFS User Group and Dave McDonnell, IBM
1:15       ANL's GPFS GSS Storage System William Allcock, Argonne Leadership Computing Facility
1:45       GPFS Update Scott Fadden, IBM
2:30       Break Refreshments
2:45       GSS Update Matt Drahzal, IBM
3:30       NCAR's GLADE GPFS Environment Pamela Gillman, NCAR Computational Information Sciences Laboratory
4:00       GPFS Roadmap (NDA Required)    Krystal Rothaup, IBM (Session Moved to Centennial D for SPXXL Joint NDA Session)

Enjoy the event of you are able to make it. I hope to be in touch again soon with a date for the next UK meeting - watch this space!

Kind regards,

Claire Robson
GPFS User Group Secretary

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131022/2ccf16ee/attachment-0001.htm>

From oester at gmail.com  Tue Oct 22 18:19:20 2013
From: oester at gmail.com (Bob Oesterlin)
Date: Tue, 22 Oct 2013 12:19:20 -0500
Subject: [gpfsug-discuss] SC'13 World Wide User Forum
In-Reply-To: <A42128435E851644B9B011BB824F6C816E24AEDFA7@MAIL.ocf.local>
References: <A42128435E851644B9B011BB824F6C816E24AEDFA7@MAIL.ocf.local>
Message-ID: <CAMNdFvB8_1WJvuZsyhbntU53s5oN9t3Kkx7H2UW83v3O0EQVbQ@mail.gmail.com>

Where is this event being held?


Bob Oesterlin
Nuance Communications



On Tue, Oct 22, 2013 at 11:27 AM, Claire Robson <crobson at ocf.co.uk> wrote:

> Dear Members,****
>
> ** **
>
> For anyone who is attending Supercomputing next month, there is a
> Worldwide GPFS User Forum taking place on Monday 18th November from
> 1-5pm. The GPFS User Forum is a worldwide gathering for current and new
> users of GPFS. Various topics will be discussed by current clients and IBM
> teams, including discussions on deployments, GPFS Storage Server, roadmaps
> and more! ****
>
> ** **
>
>
> https://www-950.ibm.com/events/wwe/grp/grp017.nsf/v17_enrollall?openform&seminar=D7QQ56ES&locale=en_US
> ****
>
> ** **
>
> Agenda****
>
> 1:00       Welcome and Formalities              Andrew Dean, GPFS User
> Group and Dave McDonnell, IBM ****
>
> 1:15       ANL's GPFS GSS Storage System William Allcock, Argonne
> Leadership Computing Facility ****
>
> 1:45       GPFS Update Scott Fadden, IBM****
>
> 2:30       Break Refreshments****
>
> 2:45       GSS Update Matt Drahzal, IBM****
>
> 3:30       NCAR's GLADE GPFS Environment Pamela Gillman, NCAR
> Computational Information Sciences Laboratory****
>
> 4:00       GPFS Roadmap (NDA Required)    Krystal Rothaup, IBM (Session
> Moved to Centennial D for SPXXL Joint NDA Session)****
>
> ** **
>
> Enjoy the event of you are able to make it. I hope to be in touch again
> soon with a date for the next UK meeting ? watch this space!****
>
> ** **
>
> Kind regards,****
>
> ** **
>
> *Claire Robson*
>
> GPFS User Group Secretary****
>
> ** **
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131022/13013ddd/attachment-0001.htm>

From sfadden at us.ibm.com  Tue Oct 22 20:41:24 2013
From: sfadden at us.ibm.com (Scott Fadden)
Date: Tue, 22 Oct 2013 12:41:24 -0700
Subject: [gpfsug-discuss] SC'13 World Wide User Forum
In-Reply-To: <CAMNdFvB8_1WJvuZsyhbntU53s5oN9t3Kkx7H2UW83v3O0EQVbQ@mail.gmail.com>
References: <A42128435E851644B9B011BB824F6C816E24AEDFA7@MAIL.ocf.local>
	<CAMNdFvB8_1WJvuZsyhbntU53s5oN9t3Kkx7H2UW83v3O0EQVbQ@mail.gmail.com>
Message-ID: <OF0E566F5E.8E3EA689-ON88257C0C.006C2731-88257C0C.006C2938@us.ibm.com>



Hyatt Regency
650 15th Street
Denver, Colorado
United States


Scott Fadden
GPFS Technical Marketing
Phone: (503) 880-5833
sfadden at us.ibm.com
http://www.ibm.com/systems/gpfs


|------------>
| From:      |
|------------>
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
  |Bob Oesterlin <oester at gmail.com>                                                                                                               |
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| To:        |
|------------>
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
  |gpfsug main discussion list <gpfsug-discuss at gpfsug.org>,                                                                                       |
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Date:      |
|------------>
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
  |10/22/2013 10:28 AM                                                                                                                            |
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Subject:   |
|------------>
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
  |Re: [gpfsug-discuss] SC'13 World Wide User Forum                                                                                               |
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Sent by:   |
|------------>
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
  |gpfsug-discuss-bounces at gpfsug.org                                                                                                              |
  >-----------------------------------------------------------------------------------------------------------------------------------------------|





Where is this event being held?


Bob Oesterlin
Nuance Communications



On Tue, Oct 22, 2013 at 11:27 AM, Claire Robson <crobson at ocf.co.uk> wrote:
  Dear Members,





  For anyone who is attending Supercomputing next month, there is a
  Worldwide GPFS User Forum taking place on Monday 18th November from
  1-5pm. The GPFS User Forum is a worldwide gathering for current and new
  users of GPFS. Various topics will be discussed by current clients and
  IBM teams, including discussions on deployments, GPFS Storage Server,
  roadmaps and more!





  https://www-950.ibm.com/events/wwe/grp/grp017.nsf/v17_enrollall?openform&seminar=D7QQ56ES&locale=en_US





  Agenda


  1:00 ????? Welcome and Formalities????????????? Andrew Dean, GPFS User
  Group and Dave McDonnell, IBM


  1:15?????? ANL's GPFS GSS Storage System William Allcock, Argonne
  Leadership Computing Facility


  1:45?????? GPFS Update Scott Fadden, IBM


  2:30?????? Break Refreshments


  2:45?????? GSS Update Matt Drahzal, IBM


  3:30?????? NCAR's GLADE GPFS Environment Pamela Gillman, NCAR
  Computational Information Sciences Laboratory


  4:00?????? GPFS Roadmap (NDA Required)??? Krystal Rothaup, IBM (Session
  Moved to Centennial D for SPXXL Joint NDA Session)





  Enjoy the event of you are able to make it. I hope to be in touch again
  soon with a date for the next UK meeting ? watch this space!





  Kind regards,





  Claire Robson


  GPFS User Group Secretary




  _______________________________________________
  gpfsug-discuss mailing list
  gpfsug-discuss at gpfsug.org
  http://gpfsug.org/mailman/listinfo/gpfsug-discuss

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131022/140a00b7/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131022/140a00b7/attachment-0002.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ecblank.gif
Type: image/gif
Size: 45 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131022/140a00b7/attachment-0003.gif>

From crobson at ocf.co.uk  Wed Oct 23 09:04:32 2013
From: crobson at ocf.co.uk (Claire Robson)
Date: Wed, 23 Oct 2013 09:04:32 +0100
Subject: [gpfsug-discuss] SC'13 World Wide User Forum
In-Reply-To: <CAMNdFvB8_1WJvuZsyhbntU53s5oN9t3Kkx7H2UW83v3O0EQVbQ@mail.gmail.com>
References: <A42128435E851644B9B011BB824F6C816E24AEDFA7@MAIL.ocf.local>
	<CAMNdFvB8_1WJvuZsyhbntU53s5oN9t3Kkx7H2UW83v3O0EQVbQ@mail.gmail.com>
Message-ID: <A42128435E851644B9B011BB824F6C816E24AEDFB8@MAIL.ocf.local>

Supercomputing is taking place at the Colorado Convention Center in Denver, Colorado, USA from 17-22nd November 2013.

Thanks,
Claire

From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Bob Oesterlin
Sent: 22 October 2013 18:19
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] SC'13 World Wide User Forum

Where is this event being held?


Bob Oesterlin
Nuance Communications


On Tue, Oct 22, 2013 at 11:27 AM, Claire Robson <crobson at ocf.co.uk<mailto:crobson at ocf.co.uk>> wrote:
Dear Members,

For anyone who is attending Supercomputing next month, there is a Worldwide GPFS User Forum taking place on Monday 18th November from 1-5pm. The GPFS User Forum is a worldwide gathering for current and new users of GPFS. Various topics will be discussed by current clients and IBM teams, including discussions on deployments, GPFS Storage Server, roadmaps and more!

https://www-950.ibm.com/events/wwe/grp/grp017.nsf/v17_enrollall?openform&seminar=D7QQ56ES&locale=en_US

Agenda
1:00       Welcome and Formalities              Andrew Dean, GPFS User Group and Dave McDonnell, IBM
1:15       ANL's GPFS GSS Storage System William Allcock, Argonne Leadership Computing Facility
1:45       GPFS Update Scott Fadden, IBM
2:30       Break Refreshments
2:45       GSS Update Matt Drahzal, IBM
3:30       NCAR's GLADE GPFS Environment Pamela Gillman, NCAR Computational Information Sciences Laboratory
4:00       GPFS Roadmap (NDA Required)    Krystal Rothaup, IBM (Session Moved to Centennial D for SPXXL Joint NDA Session)

Enjoy the event of you are able to make it. I hope to be in touch again soon with a date for the next UK meeting - watch this space!

Kind regards,

Claire Robson
GPFS User Group Secretary


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org<http://gpfsug.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131023/083b2225/attachment-0001.htm>

From chair at gpfsug.org  Fri Oct 25 17:09:16 2013
From: chair at gpfsug.org (Chair GPFS UG)
Date: Fri, 25 Oct 2013 17:09:16 +0100
Subject: [gpfsug-discuss] Friday MPI fun?
Message-ID: <CA+M26=T7eeSMWhV_GFFvEkaKnDHNxF9OXqHbg4DvfVhnWN8CfA@mail.gmail.com>

Allo all,

  I'm attempting to cheat.  As per-usual, cheating takes more time than
'Doing It Properly' - but it is vastly more fun.

So without setting up Grid or Moab etc, I need to pin processes to a cpu.
 I.E. on Linux: taskset blah blah.
I could write a small housekeeping script which RR new spawned processes
across CPUs using taskset, but I was wondering if OpenMPI could be a good
way to go.

So:

I have a non-MPI application X.
Which is spawned and forked by a parent process into its own process group.
This can occur at any time, however there will only ever be a maximium N of
appl X.

Using mpirun it appears that you can set off parallel instances of a
non-MPI application:

mpirun -np 4 applicationX

However, really what I'm requiring to do is say:
  Max slots = N (which I can define in the mpi hostfile).

mpirun -np 1 applicationX
mpirun -np 1 applicationX (started at a random future time)
mpirun -np 1 applicationX (started at a random future time)
mpirun -np 1 applicationX (started at a random future time)

Each being automatically pinned to a CPU, but I'm fairly convinced this is
not the way MPI works.
Would it do what I'm after?

Does anyone know of a better way?

Jez
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131025/37f6a4c9/attachment-0001.htm>

From zgiles at gmail.com  Sat Oct 26 05:13:37 2013
From: zgiles at gmail.com (Zachary Giles)
Date: Sat, 26 Oct 2013 00:13:37 -0400
Subject: [gpfsug-discuss] Friday MPI fun?
In-Reply-To: <CA+M26=T7eeSMWhV_GFFvEkaKnDHNxF9OXqHbg4DvfVhnWN8CfA@mail.gmail.com>
References: <CA+M26=T7eeSMWhV_GFFvEkaKnDHNxF9OXqHbg4DvfVhnWN8CfA@mail.gmail.com>
Message-ID: <CAMYZk=esiDC4fq3DnqngyttsvvYCSBNOO2vZBsF=uJdT=fcU2Q@mail.gmail.com>

You could do it that way, if you told MPI to bind each task (MPI task, or
non-MPI task in your case) to a processor. By default, most MPI's don't
bind to a specific processor unless the scheduler (you don't have one) or
something else tells it to. OpenMPI, for example, I think, has something
like --bind-to-cpu, or --bind-to-socket. Often a scheduler or c-groups via
a scheduler will pass-in a list of cpu's to bind to, but you could do it
manually with the appropriate flags and files for your MPI distro.

You could also probably get away with just raw c-groups depending on your
distro. You would probably touch a few files and echo a few lines to files
to make the right groups then "cgrun <app>" to get things in to those
groups. That works pretty well for binding, found decent results with that,
I recommend it.


On Fri, Oct 25, 2013 at 12:09 PM, Chair GPFS UG <chair at gpfsug.org> wrote:

> Allo all,
>
>   I'm attempting to cheat.  As per-usual, cheating takes more time than
> 'Doing It Properly' - but it is vastly more fun.
>
> So without setting up Grid or Moab etc, I need to pin processes to a cpu.
>  I.E. on Linux: taskset blah blah.
> I could write a small housekeeping script which RR new spawned processes
> across CPUs using taskset, but I was wondering if OpenMPI could be a good
> way to go.
>
> So:
>
> I have a non-MPI application X.
> Which is spawned and forked by a parent process into its own process group.
> This can occur at any time, however there will only ever be a maximium N
> of appl X.
>
> Using mpirun it appears that you can set off parallel instances of a
> non-MPI application:
>
> mpirun -np 4 applicationX
>
> However, really what I'm requiring to do is say:
>   Max slots = N (which I can define in the mpi hostfile).
>
> mpirun -np 1 applicationX
> mpirun -np 1 applicationX (started at a random future time)
> mpirun -np 1 applicationX (started at a random future time)
> mpirun -np 1 applicationX (started at a random future time)
>
> Each being automatically pinned to a CPU, but I'm fairly convinced this is
> not the way MPI works.
> Would it do what I'm after?
>
> Does anyone know of a better way?
>
> Jez
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>


-- 
Zach Giles
zgiles at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131026/217d585e/attachment-0001.htm>

From richard.lefebvre at calculquebec.ca  Mon Oct 28 15:31:07 2013
From: richard.lefebvre at calculquebec.ca (Richard Lefebvre)
Date: Mon, 28 Oct 2013 11:31:07 -0400
Subject: [gpfsug-discuss] Moving/copying files from one file system to
	another
Message-ID: <526E833B.6040402@calculquebec.ca>

Hi,

I'm looking to copy/move files from one gpfs file system to another in
some efficient way. Initially I was thinking to not create another file
system and just add the nsd to the current file system and gradually
remove the old nsd. That would have been the easiest with little impact
to the users. But I wanted to switch from a scatter allocation to a
cluster allocation (too improve performance) and I cannot to that if I
just add/remove nsd. I'm hopping for a better solution then a plain
rsync which would probably take a few days (~210TB).

Is there a gpfs solution to this?

Also, another question, under what condition a scatter allocation better
then cluster allocation. We currently have a cluster of 650 nodes all
accessing the same 230TB gpfs file system.

Richard


From seanlee at tw.ibm.com  Wed Oct 30 12:30:26 2013
From: seanlee at tw.ibm.com (Sean S Lee)
Date: Wed, 30 Oct 2013 20:30:26 +0800
Subject: [gpfsug-discuss] Moving/copying files from one file system to
	another
In-Reply-To: <mailman.1.1383134402.20340.gpfsug-discuss@gpfsug.org>
References: <mailman.1.1383134402.20340.gpfsug-discuss@gpfsug.org>
Message-ID: <OF9B62D9AF.094922A5-ON48257C14.00438E14-48257C14.0044C452@tw.ibm.com>


Hi Richard,

==========

The cluster allocation method may provide better disk performance for some
disk subsystems in
relatively small installations. The benefits of clustered block allocation
diminish when the number
of nodes in the cluster or the number of disks in a file system increases,
or when the file system's
free space becomes fragmented. The cluster allocation method is the default
for GPFS clusters
with eight or fewer nodes and for file systems with eight or fewer disks.

The scatter allocation method provides more consistent file system
performance by averaging out
performance variations due to block location (for many disk subsystems, the
location of the data
relative to the disk edge has a substantial effect on performance). This
allocation method is
appropriate in most cases and is the default for GPFS clusters with more
than eight nodes or file
systems with more than eight disks.

Source: GPFS Administration and Programming Reference
==========

Based on above it seems the scatter allocation would be more suitable for
your environment (more nodes, more disks), no?

The Web forum has few threads on GPFS copying, e.g. take a look at this:
https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014934284&ps=25

I don't know if there's a better way or not.

If you don't change the allocation type, then you don't have to do anything
but add the disks and perhaps restripe.

Regards
Sean
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131030/5c77c624/attachment-0001.htm>

From jonathan at buzzard.me.uk  Wed Oct 30 12:47:07 2013
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Wed, 30 Oct 2013 12:47:07 +0000
Subject: [gpfsug-discuss] Moving/copying files from one file system to
 another
In-Reply-To: <526E833B.6040402@calculquebec.ca>
References: <526E833B.6040402@calculquebec.ca>
Message-ID: <1383137227.2921.4.camel@buzzard.phy.strath.ac.uk>

On Mon, 2013-10-28 at 11:31 -0400, Richard Lefebvre wrote:

[SNIP]

> Also, another question, under what condition a scatter allocation better
> then cluster allocation. We currently have a cluster of 650 nodes all
> accessing the same 230TB gpfs file system.
> 

Scatter allocation is better in almost all circumstances. Basically by
scattering the files to all corners you don't get hotspots where just a
small subset of the disks are being hammered by lots of accesses to a
handful of files, while the rest of the disks sit idle.

JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.



From chekh at stanford.edu  Wed Oct 30 17:56:46 2013
From: chekh at stanford.edu (Alex Chekholko)
Date: Wed, 30 Oct 2013 10:56:46 -0700
Subject: [gpfsug-discuss] Moving/copying files from one file system to
 another
In-Reply-To: <1383137227.2921.4.camel@buzzard.phy.strath.ac.uk>
References: <526E833B.6040402@calculquebec.ca>
	<1383137227.2921.4.camel@buzzard.phy.strath.ac.uk>
Message-ID: <5271485E.80405@stanford.edu>

On 10/30/13, 5:47 AM, Jonathan Buzzard wrote:
> On Mon, 2013-10-28 at 11:31 -0400, Richard Lefebvre wrote:
>
> [SNIP]
>
>> Also, another question, under what condition a scatter allocation better
>> then cluster allocation. We currently have a cluster of 650 nodes all
>> accessing the same 230TB gpfs file system.
>>
>
> Scatter allocation is better in almost all circumstances. Basically by
> scattering the files to all corners you don't get hotspots where just a
> small subset of the disks are being hammered by lots of accesses to a
> handful of files, while the rest of the disks sit idle.
>

If you do benchmarks with only a few threads, you will see higher 
performance with 'cluster' allocation.  So if your workload is only a 
few clients accessing the FS in a mostly streaming way, you'd see better 
performance from 'cluster'.

With 650 nodes, even if each client is doing streaming reads, at the 
filesystem level that would all be interleaved and thus be random reads. 
  But it's tough to do a big enough benchmark to show the difference in 
performance.

I had a tough time convincing people to use 'scatter' instead of 
'cluster' even though I think the documentation is clear about the 
difference, and even gives you the sizing parameters ( greater than 8 
disks or 8 NSDs?  use 'scatter').

We use 'scatter' now.

Regards
-- 
chekh at stanford.edu


From TOMP at il.ibm.com  Wed Oct 30 18:31:01 2013
From: TOMP at il.ibm.com (Tomer Perry)
Date: Wed, 30 Oct 2013 20:31:01 +0200
Subject: [gpfsug-discuss] Moving/copying files from one file system to
	another
In-Reply-To: <5271485E.80405@stanford.edu>
References: <526E833B.6040402@calculquebec.ca>	<1383137227.2921.4.camel@buzzard.phy.strath.ac.uk>
	<5271485E.80405@stanford.edu>
Message-ID: <OF39DC62FE.1C8ABCF9-ONC2257C14.00657D38-C2257C14.0065BC4C@il.ibm.com>

Hi,

cluster will also be there for some time, as soon as one start deleting 
and creating file - it will kind of shift to be scattered...
So, while its good for benchmarks on "clean and new" system - cluster is 
somewhat meaningless ( and was hidden for some time because of that).

As far as it goes for copying from old to new system - AFM can be 
considered as well ( will make the transition easier). You can prefetch ( 
using mmafmctl) and even work in LU mode in order to get the data from the 
old FS to the new one without pushing changes back.

hth,
Tomer.




From:   Alex Chekholko <chekh at stanford.edu>
To:     gpfsug-discuss at gpfsug.org, 
Date:   10/30/2013 07:57 PM
Subject:        Re: [gpfsug-discuss] Moving/copying files from one file 
system to another
Sent by:        gpfsug-discuss-bounces at gpfsug.org



On 10/30/13, 5:47 AM, Jonathan Buzzard wrote:
> On Mon, 2013-10-28 at 11:31 -0400, Richard Lefebvre wrote:
>
> [SNIP]
>
>> Also, another question, under what condition a scatter allocation 
better
>> then cluster allocation. We currently have a cluster of 650 nodes all
>> accessing the same 230TB gpfs file system.
>>
>
> Scatter allocation is better in almost all circumstances. Basically by
> scattering the files to all corners you don't get hotspots where just a
> small subset of the disks are being hammered by lots of accesses to a
> handful of files, while the rest of the disks sit idle.
>

If you do benchmarks with only a few threads, you will see higher 
performance with 'cluster' allocation.  So if your workload is only a 
few clients accessing the FS in a mostly streaming way, you'd see better 
performance from 'cluster'.

With 650 nodes, even if each client is doing streaming reads, at the 
filesystem level that would all be interleaved and thus be random reads. 
  But it's tough to do a big enough benchmark to show the difference in 
performance.

I had a tough time convincing people to use 'scatter' instead of 
'cluster' even though I think the documentation is clear about the 
difference, and even gives you the sizing parameters ( greater than 8 
disks or 8 NSDs?  use 'scatter').

We use 'scatter' now.

Regards
-- 
chekh at stanford.edu
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131030/d130f587/attachment-0001.htm>

From jfosburg at mdanderson.org  Wed Oct  2 21:44:55 2013
From: jfosburg at mdanderson.org (Fosburgh,Jonathan)
Date: Wed, 2 Oct 2013 20:44:55 +0000
Subject: [gpfsug-discuss] Copying ACLs from outside sources
Message-ID: <CE71EFF4.200B4%jfosburg@mdanderson.org>

Hello, I'm new to this list, as well as GPFS.

We have new (installed over the last 6 weeks) GPFS cluster that we are currently trying to get into service.  The NSDs are running on Red Hat (6 NSDs), the storage is DCS3700 directly attached via SAS.  We are running GPFS 3.5.0.12.


Some of the data that will be placed on this cluster will be existing data coming from other sources, one of which is our NAS (Isilon, supporting both NFS and CIFS).  We have tried a few different ways of copying the data.  First we tried to rsync directly between an NSD and one of the Isilon nodes.  Unfortunately, it appears the rsync doesn't work with the ACL format used on the Isilon.  We have also tried using various methods of copying data via CIFS, but we continue to have no luck getting ACLs to copy.  We know that, for instance, copying data via CIFS into the Isilon using robocopy will preserve the ACLs, and we were hoping that would work for getting data onto GPFS as well.  However, the ACLs are being stripped and replaced with whatever ACL is present on GPFS.  Does anyone have experience doing something like that who can give me some pointers on what my configuration should look like? I assume there are samba and/or GPFS parameters that need to be tweaked in order to make this work, especially since this functionality would have to be present in SONAS.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131002/a1ef2992/attachment-0002.htm>

From pete at realisestudio.com  Thu Oct  3 09:09:07 2013
From: pete at realisestudio.com (Pete Smith)
Date: Thu, 3 Oct 2013 09:09:07 +0100
Subject: [gpfsug-discuss] Copying ACLs from outside sources
In-Reply-To: <CE71EFF4.200B4%jfosburg@mdanderson.org>
References: <CE71EFF4.200B4%jfosburg@mdanderson.org>
Message-ID: <CAM9ZKkjHcSK9RgmLJKL=QedA=TN+6NhZZhHtWgpyLB=Za7S+EQ@mail.gmail.com>

Hi, and welcome, Jonathan.

Which combination of -A and -X rsync options did you use?

Pete

On 2 October 2013 21:44, Fosburgh,Jonathan <jfosburg at mdanderson.org> wrote:
> Hello, I'm new to this list, as well as GPFS.
>
> We have new (installed over the last 6 weeks) GPFS cluster that we are
> currently trying to get into service.  The NSDs are running on Red Hat (6
> NSDs), the storage is DCS3700 directly attached via SAS.  We are running
> GPFS 3.5.0.12.
>
>
> Some of the data that will be placed on this cluster will be existing data
> coming from other sources, one of which is our NAS (Isilon, supporting both
> NFS and CIFS).  We have tried a few different ways of copying the data.
> First we tried to rsync directly between an NSD and one of the Isilon nodes.
> Unfortunately, it appears the rsync doesn't work with the ACL format used on
> the Isilon.  We have also tried using various methods of copying data via
> CIFS, but we continue to have no luck getting ACLs to copy.  We know that,
> for instance, copying data via CIFS into the Isilon using robocopy will
> preserve the ACLs, and we were hoping that would work for getting data onto
> GPFS as well.  However, the ACLs are being stripped and replaced with
> whatever ACL is present on GPFS.  Does anyone have experience doing
> something like that who can give me some pointers on what my configuration
> should look like? I assume there are samba and/or GPFS parameters that need
> to be tweaked in order to make this work, especially since this
> functionality would have to be present in SONAS.
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>



-- 
Pete Smith
DevOp/System Administrator
Realise Studio
12/13 Poland Street, London W1F 8QB
T. +44 (0)20 7165 9644

realisestudio.com


From orlando.richards at ed.ac.uk  Thu Oct  3 09:17:47 2013
From: orlando.richards at ed.ac.uk (Orlando Richards)
Date: Thu, 03 Oct 2013 09:17:47 +0100
Subject: [gpfsug-discuss] Copying ACLs from outside sources
In-Reply-To: <CAM9ZKkjHcSK9RgmLJKL=QedA=TN+6NhZZhHtWgpyLB=Za7S+EQ@mail.gmail.com>
References: <CE71EFF4.200B4%jfosburg@mdanderson.org>
	<CAM9ZKkjHcSK9RgmLJKL=QedA=TN+6NhZZhHtWgpyLB=Za7S+EQ@mail.gmail.com>
Message-ID: <524D282B.905@ed.ac.uk>

Hi Jonathan,

Might be a bit of a long shot this - but for copying NFSv4 ACLs between 
GPFS filesystems using rsync, I had to patch rsync with some code 
released by IBM at some point. We've got that patched rsync hosted on 
the gpfsug github:

   https://github.com/gpfsug/gpfsug-tools/tree/master/bin/rsync

Since you're not doing gpfs -> gpfs, it might be no use - but you never 
know!

Oh - and, of course, make sure your GPFS filesystem is configured to 
support NFSv4 ACLs.

Here's our smb.conf settings related to GPFS, ACLs and other bits and bobs:

vfs objects = shadow_copy2, fileid, gpfs
clustering = yes
fileid:mapping = fsname
gpfs:sharemodes = No
gpfs:getrealfilename = no
nfs4: mode = simple
nfs4: chown = yes
nfs4: acedup = merge
force unknown acl user = yes
acl group control = no
map acl inherit = yes
inherit acls = no
dos filemode = no
map readonly = no
map hidden = no
map archive = no
map readonly = no
ea support = yes
store dos attributes = yes
gpfs:winattr = yes
max protocol = SMB2
shadow:fixinodes = yes
shadow:snapdir = .snapshots
shadow:basedir = /exports/nas
use sendfile = yes
wide links = no
unix extensions = no

Hope that helps,
Orlando

--
Orlando

On 03/10/13 09:09, Pete Smith wrote:
> Hi, and welcome, Jonathan.
>
> Which combination of -A and -X rsync options did you use?
>
> Pete
>
> On 2 October 2013 21:44, Fosburgh,Jonathan <jfosburg at mdanderson.org> wrote:
>> Hello, I'm new to this list, as well as GPFS.
>>
>> We have new (installed over the last 6 weeks) GPFS cluster that we are
>> currently trying to get into service.  The NSDs are running on Red Hat (6
>> NSDs), the storage is DCS3700 directly attached via SAS.  We are running
>> GPFS 3.5.0.12.
>>
>>
>> Some of the data that will be placed on this cluster will be existing data
>> coming from other sources, one of which is our NAS (Isilon, supporting both
>> NFS and CIFS).  We have tried a few different ways of copying the data.
>> First we tried to rsync directly between an NSD and one of the Isilon nodes.
>> Unfortunately, it appears the rsync doesn't work with the ACL format used on
>> the Isilon.  We have also tried using various methods of copying data via
>> CIFS, but we continue to have no luck getting ACLs to copy.  We know that,
>> for instance, copying data via CIFS into the Isilon using robocopy will
>> preserve the ACLs, and we were hoping that would work for getting data onto
>> GPFS as well.  However, the ACLs are being stripped and replaced with
>> whatever ACL is present on GPFS.  Does anyone have experience doing
>> something like that who can give me some pointers on what my configuration
>> should look like? I assume there are samba and/or GPFS parameters that need
>> to be tweaked in order to make this work, especially since this
>> functionality would have to be present in SONAS.
>>
>>
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at gpfsug.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>
>
>
>


-- 
             --
    Dr Orlando Richards
   Information Services
IT Infrastructure Division
        Unix Section
     Tel: 0131 650 4994
   skype: orlando.richards

The University of Edinburgh is a charitable body, registered in 
Scotland, with registration number SC005336.


From jonathan at buzzard.me.uk  Thu Oct  3 10:23:02 2013
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Thu, 03 Oct 2013 10:23:02 +0100
Subject: [gpfsug-discuss] Copying ACLs from outside sources
In-Reply-To: <CE71EFF4.200B4%jfosburg@mdanderson.org>
References: <CE71EFF4.200B4%jfosburg@mdanderson.org>
Message-ID: <1380792182.19439.18.camel@buzzard.phy.strath.ac.uk>

On Wed, 2013-10-02 at 20:44 +0000, Fosburgh,Jonathan wrote: 
> Hello, I'm new to this list, as well as GPFS.  
> 
> 
> We have new (installed over the last 6 weeks) GPFS cluster that we are
> currently trying to get into service.  The NSDs are running on Red Hat
> (6 NSDs), the storage is DCS3700 directly attached via SAS.  We are
> running GPFS 3.5.0.12.
> 
> Some of the data that will be placed on this cluster will be existing
> data coming from other sources, one of which is our NAS (Isilon,
> supporting both NFS and CIFS).  We have tried a few different ways of
> copying the data.  First we tried to rsync directly between an NSD and
> one of the Isilon nodes.  Unfortunately, it appears the rsync doesn't
> work with the ACL format used on the Isilon.  We have also tried using
> various methods of copying data via CIFS, but we continue to have no
> luck getting ACLs to copy.  We know that, for instance, copying data
> via CIFS into the Isilon using robocopy will preserve the ACLs, and we
> were hoping that would work for getting data onto GPFS as well.
> However, the ACLs are being stripped and replaced with whatever ACL is
> present on GPFS.  Does anyone have experience doing something like
> that who can give me some pointers on what my configuration should
> look like? I assume there are samba and/or GPFS parameters that need
> to be tweaked in order to make this work, especially since this
> functionality would have to be present in SONAS.
> 

I guess the first thing to do is confirm that you have functioning
CIFS's ACL's in your Samba GPFS file server. Specifically you would need
to have NFSv4 ACL's enabled in the file system and the gpfs VFS module
loaded in Samba. Note having mixed ACL modes did not work very well for
me, so best to turn off Posix ACL's and only do NFSv4 ACL's.

These are the samba options I use in combination with GPFS to make the
server as windows like as possible.

# general options
vfs objects = fileid gpfs
        unix extensions = no
        mangled names = no
        case sensitive = no

# store DOS attributes in extended attributes (vfs_gpfs then stores them
in the file system)
ea support = yes
store dos attributes = yes
map readonly = no
map archive = no
map system = no

# the ctdb clustering and GPFS stuff
clustering = yes
ctdbd socket = /tmp/ctdb.socket
        fileid : algorithm = fsname
        gpfs : sharemodes = yes
        gpfs : winattr = yes
        force unknown acl user = yes
        nfs4 : mode = special
        nfs4 : chown = no
        nfs4 : acedup = merge

The best way to make sure it is working is point a Windows client at the
file server and manipulate the ACL's by hand.

Note that if you are going for that full make my Samba/GPFS file server
look as close as possible to a pucker MS Windows server, you might want
to consider setting the following GPFS options

     cifsBypassShareLocksOnRename
     cifsBypassTraversalChecking
     allowWriteWithDeleteChild

All fairly self explanatory, and make GPFS follow Windows schematics
more closely, though they are "undocumented". There are a couple of
other options as allowSambaCaseInsensitiveLookup and
syncSambaMetadataOps but I have not determined exactly what they do...

There is also there is an undocumented option for ACL's on mmchfs 

      mmchfs test -k samba

Even shows up in the output of mmlsfs. Not entirely sure what samba
ACL's are mind you...

Note that IBM have alluded at various times that there these "magic"
options exist and are used in SONAS etc.

Anyway getting back to ACL's robocopy will move them over, but in my
experience copying from a Windows file server the results where not what
I wanted. I ended up with extra ACL's for administrators and bizzare
file ownership issues. Some of that might be because the files had
originally been on a Netware file server which died, and where hastily
moved to a Windows server. I gave up in the end and manually unpicked
what the ACL's where doing for the migration to the GPFS file server.
However I only had 20GB of data...

You should also find that xcopy will move them over as well.

A possible alternative if you have access to an AIX box would be to
NFSv4 mount the Isilon on the AIX box and mount GPFS as well and just
use cp.


JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.




From jfosburg at mdanderson.org  Wed Oct  9 18:31:45 2013
From: jfosburg at mdanderson.org (Fosburgh,Jonathan)
Date: Wed, 9 Oct 2013 17:31:45 +0000
Subject: [gpfsug-discuss] Copying ACLs from outside sources
In-Reply-To: <1380792182.19439.18.camel@buzzard.phy.strath.ac.uk>
Message-ID: <CE7AFC34.21539%jfosburg@mdanderson.org>



On 10/3/13 4:23 AM, "Jonathan Buzzard" <jonathan at buzzard.me.uk> wrote:

>
>> 
>
>I guess the first thing to do is confirm that you have functioning
>CIFS's ACL's in your Samba GPFS file server. Specifically you would need
>to have NFSv4 ACL's enabled in the file system and the gpfs VFS module
>loaded in Samba. Note having mixed ACL modes did not work very well for
>me, so best to turn off Posix ACL's and only do NFSv4 ACL's.
>
Thank you for the response. We have been able to make some headway on
this, but I am still bumping up on some issues.  It looks as if, when an
NFSv4 ACL is already present on the parent directory tree, copying acls
via robocopy works. However, if there is a posix ACL present, then the
ACLs are still not copied correctly.  I was interested in setting the ACL
mode to nfs4 only (currently set to all), however, I have some
reservations about this.

The NFS server is Red Hat, and I need to be able to server version 3 and
version 4, plus we will have native GPFS clients and the CIFS clients.
What are the ramifications, in this setup, of changing the ACL type? What
would happen to any existing ACLs?



From jonathan at buzzard.me.uk  Wed Oct  9 20:54:53 2013
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Wed, 09 Oct 2013 20:54:53 +0100
Subject: [gpfsug-discuss] Copying ACLs from outside sources
In-Reply-To: <CE7AFC34.21539%jfosburg@mdanderson.org>
References: <CE7AFC34.21539%jfosburg@mdanderson.org>
Message-ID: <5255B48D.1070401@buzzard.me.uk>

On 09/10/13 18:31, Fosburgh,Jonathan wrote:
>
>
> On 10/3/13 4:23 AM, "Jonathan Buzzard" <jonathan at buzzard.me.uk> wrote:
>
>>
>>>
>>
>> I guess the first thing to do is confirm that you have functioning
>> CIFS's ACL's in your Samba GPFS file server. Specifically you would need
>> to have NFSv4 ACL's enabled in the file system and the gpfs VFS module
>> loaded in Samba. Note having mixed ACL modes did not work very well for
>> me, so best to turn off Posix ACL's and only do NFSv4 ACL's.
>>
> Thank you for the response. We have been able to make some headway on
> this, but I am still bumping up on some issues.  It looks as if, when an
> NFSv4 ACL is already present on the parent directory tree, copying acls
> via robocopy works. However, if there is a posix ACL present, then the
> ACLs are still not copied correctly.  I was interested in setting the ACL
> mode to nfs4 only (currently set to all), however, I have some
> reservations about this.
>

Like I said in my experience having both Posix and NFSv4 ACL's active at 
the same time did not work properly as you have found out. You have to 
pick one and stick with it. Clearly if you have an existing system with 
both ACL types on then you have problems.

> The NFS server is Red Hat, and I need to be able to server version 3 and
> version 4, plus we will have native GPFS clients and the CIFS clients.
> What are the ramifications, in this setup, of changing the ACL type? What
> would happen to any existing ACLs?

My guess is the Posix ACL's either get lost or are "converted" into 
NFSv4 ACL's. I would try this out on your "development/test" GPFS setup 
first and work out a strategy for moving forward.

The biggest problem is that the IBM provided tools for manipulating the 
GPFS ACL's from within Linux are awful.


JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.


From crobson at ocf.co.uk  Tue Oct 22 17:27:24 2013
From: crobson at ocf.co.uk (Claire Robson)
Date: Tue, 22 Oct 2013 17:27:24 +0100
Subject: [gpfsug-discuss] SC'13 World Wide User Forum
Message-ID: <A42128435E851644B9B011BB824F6C816E24AEDFA7@MAIL.ocf.local>

Dear Members,

For anyone who is attending Supercomputing next month, there is a Worldwide GPFS User Forum taking place on Monday 18th November from 1-5pm. The GPFS User Forum is a worldwide gathering for current and new users of GPFS. Various topics will be discussed by current clients and IBM teams, including discussions on deployments, GPFS Storage Server, roadmaps and more!

https://www-950.ibm.com/events/wwe/grp/grp017.nsf/v17_enrollall?openform&seminar=D7QQ56ES&locale=en_US

Agenda
1:00       Welcome and Formalities              Andrew Dean, GPFS User Group and Dave McDonnell, IBM
1:15       ANL's GPFS GSS Storage System William Allcock, Argonne Leadership Computing Facility
1:45       GPFS Update Scott Fadden, IBM
2:30       Break Refreshments
2:45       GSS Update Matt Drahzal, IBM
3:30       NCAR's GLADE GPFS Environment Pamela Gillman, NCAR Computational Information Sciences Laboratory
4:00       GPFS Roadmap (NDA Required)    Krystal Rothaup, IBM (Session Moved to Centennial D for SPXXL Joint NDA Session)

Enjoy the event of you are able to make it. I hope to be in touch again soon with a date for the next UK meeting - watch this space!

Kind regards,

Claire Robson
GPFS User Group Secretary

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131022/2ccf16ee/attachment-0002.htm>

From oester at gmail.com  Tue Oct 22 18:19:20 2013
From: oester at gmail.com (Bob Oesterlin)
Date: Tue, 22 Oct 2013 12:19:20 -0500
Subject: [gpfsug-discuss] SC'13 World Wide User Forum
In-Reply-To: <A42128435E851644B9B011BB824F6C816E24AEDFA7@MAIL.ocf.local>
References: <A42128435E851644B9B011BB824F6C816E24AEDFA7@MAIL.ocf.local>
Message-ID: <CAMNdFvB8_1WJvuZsyhbntU53s5oN9t3Kkx7H2UW83v3O0EQVbQ@mail.gmail.com>

Where is this event being held?


Bob Oesterlin
Nuance Communications



On Tue, Oct 22, 2013 at 11:27 AM, Claire Robson <crobson at ocf.co.uk> wrote:

> Dear Members,****
>
> ** **
>
> For anyone who is attending Supercomputing next month, there is a
> Worldwide GPFS User Forum taking place on Monday 18th November from
> 1-5pm. The GPFS User Forum is a worldwide gathering for current and new
> users of GPFS. Various topics will be discussed by current clients and IBM
> teams, including discussions on deployments, GPFS Storage Server, roadmaps
> and more! ****
>
> ** **
>
>
> https://www-950.ibm.com/events/wwe/grp/grp017.nsf/v17_enrollall?openform&seminar=D7QQ56ES&locale=en_US
> ****
>
> ** **
>
> Agenda****
>
> 1:00       Welcome and Formalities              Andrew Dean, GPFS User
> Group and Dave McDonnell, IBM ****
>
> 1:15       ANL's GPFS GSS Storage System William Allcock, Argonne
> Leadership Computing Facility ****
>
> 1:45       GPFS Update Scott Fadden, IBM****
>
> 2:30       Break Refreshments****
>
> 2:45       GSS Update Matt Drahzal, IBM****
>
> 3:30       NCAR's GLADE GPFS Environment Pamela Gillman, NCAR
> Computational Information Sciences Laboratory****
>
> 4:00       GPFS Roadmap (NDA Required)    Krystal Rothaup, IBM (Session
> Moved to Centennial D for SPXXL Joint NDA Session)****
>
> ** **
>
> Enjoy the event of you are able to make it. I hope to be in touch again
> soon with a date for the next UK meeting ? watch this space!****
>
> ** **
>
> Kind regards,****
>
> ** **
>
> *Claire Robson*
>
> GPFS User Group Secretary****
>
> ** **
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131022/13013ddd/attachment-0002.htm>

From sfadden at us.ibm.com  Tue Oct 22 20:41:24 2013
From: sfadden at us.ibm.com (Scott Fadden)
Date: Tue, 22 Oct 2013 12:41:24 -0700
Subject: [gpfsug-discuss] SC'13 World Wide User Forum
In-Reply-To: <CAMNdFvB8_1WJvuZsyhbntU53s5oN9t3Kkx7H2UW83v3O0EQVbQ@mail.gmail.com>
References: <A42128435E851644B9B011BB824F6C816E24AEDFA7@MAIL.ocf.local>
	<CAMNdFvB8_1WJvuZsyhbntU53s5oN9t3Kkx7H2UW83v3O0EQVbQ@mail.gmail.com>
Message-ID: <OF0E566F5E.8E3EA689-ON88257C0C.006C2731-88257C0C.006C2938@us.ibm.com>



Hyatt Regency
650 15th Street
Denver, Colorado
United States


Scott Fadden
GPFS Technical Marketing
Phone: (503) 880-5833
sfadden at us.ibm.com
http://www.ibm.com/systems/gpfs


|------------>
| From:      |
|------------>
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
  |Bob Oesterlin <oester at gmail.com>                                                                                                               |
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| To:        |
|------------>
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
  |gpfsug main discussion list <gpfsug-discuss at gpfsug.org>,                                                                                       |
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Date:      |
|------------>
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
  |10/22/2013 10:28 AM                                                                                                                            |
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Subject:   |
|------------>
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
  |Re: [gpfsug-discuss] SC'13 World Wide User Forum                                                                                               |
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Sent by:   |
|------------>
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
  |gpfsug-discuss-bounces at gpfsug.org                                                                                                              |
  >-----------------------------------------------------------------------------------------------------------------------------------------------|





Where is this event being held?


Bob Oesterlin
Nuance Communications



On Tue, Oct 22, 2013 at 11:27 AM, Claire Robson <crobson at ocf.co.uk> wrote:
  Dear Members,





  For anyone who is attending Supercomputing next month, there is a
  Worldwide GPFS User Forum taking place on Monday 18th November from
  1-5pm. The GPFS User Forum is a worldwide gathering for current and new
  users of GPFS. Various topics will be discussed by current clients and
  IBM teams, including discussions on deployments, GPFS Storage Server,
  roadmaps and more!





  https://www-950.ibm.com/events/wwe/grp/grp017.nsf/v17_enrollall?openform&seminar=D7QQ56ES&locale=en_US





  Agenda


  1:00 ????? Welcome and Formalities????????????? Andrew Dean, GPFS User
  Group and Dave McDonnell, IBM


  1:15?????? ANL's GPFS GSS Storage System William Allcock, Argonne
  Leadership Computing Facility


  1:45?????? GPFS Update Scott Fadden, IBM


  2:30?????? Break Refreshments


  2:45?????? GSS Update Matt Drahzal, IBM


  3:30?????? NCAR's GLADE GPFS Environment Pamela Gillman, NCAR
  Computational Information Sciences Laboratory


  4:00?????? GPFS Roadmap (NDA Required)??? Krystal Rothaup, IBM (Session
  Moved to Centennial D for SPXXL Joint NDA Session)





  Enjoy the event of you are able to make it. I hope to be in touch again
  soon with a date for the next UK meeting ? watch this space!





  Kind regards,





  Claire Robson


  GPFS User Group Secretary




  _______________________________________________
  gpfsug-discuss mailing list
  gpfsug-discuss at gpfsug.org
  http://gpfsug.org/mailman/listinfo/gpfsug-discuss

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131022/140a00b7/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131022/140a00b7/attachment-0004.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ecblank.gif
Type: image/gif
Size: 45 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131022/140a00b7/attachment-0005.gif>

From crobson at ocf.co.uk  Wed Oct 23 09:04:32 2013
From: crobson at ocf.co.uk (Claire Robson)
Date: Wed, 23 Oct 2013 09:04:32 +0100
Subject: [gpfsug-discuss] SC'13 World Wide User Forum
In-Reply-To: <CAMNdFvB8_1WJvuZsyhbntU53s5oN9t3Kkx7H2UW83v3O0EQVbQ@mail.gmail.com>
References: <A42128435E851644B9B011BB824F6C816E24AEDFA7@MAIL.ocf.local>
	<CAMNdFvB8_1WJvuZsyhbntU53s5oN9t3Kkx7H2UW83v3O0EQVbQ@mail.gmail.com>
Message-ID: <A42128435E851644B9B011BB824F6C816E24AEDFB8@MAIL.ocf.local>

Supercomputing is taking place at the Colorado Convention Center in Denver, Colorado, USA from 17-22nd November 2013.

Thanks,
Claire

From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Bob Oesterlin
Sent: 22 October 2013 18:19
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] SC'13 World Wide User Forum

Where is this event being held?


Bob Oesterlin
Nuance Communications


On Tue, Oct 22, 2013 at 11:27 AM, Claire Robson <crobson at ocf.co.uk<mailto:crobson at ocf.co.uk>> wrote:
Dear Members,

For anyone who is attending Supercomputing next month, there is a Worldwide GPFS User Forum taking place on Monday 18th November from 1-5pm. The GPFS User Forum is a worldwide gathering for current and new users of GPFS. Various topics will be discussed by current clients and IBM teams, including discussions on deployments, GPFS Storage Server, roadmaps and more!

https://www-950.ibm.com/events/wwe/grp/grp017.nsf/v17_enrollall?openform&seminar=D7QQ56ES&locale=en_US

Agenda
1:00       Welcome and Formalities              Andrew Dean, GPFS User Group and Dave McDonnell, IBM
1:15       ANL's GPFS GSS Storage System William Allcock, Argonne Leadership Computing Facility
1:45       GPFS Update Scott Fadden, IBM
2:30       Break Refreshments
2:45       GSS Update Matt Drahzal, IBM
3:30       NCAR's GLADE GPFS Environment Pamela Gillman, NCAR Computational Information Sciences Laboratory
4:00       GPFS Roadmap (NDA Required)    Krystal Rothaup, IBM (Session Moved to Centennial D for SPXXL Joint NDA Session)

Enjoy the event of you are able to make it. I hope to be in touch again soon with a date for the next UK meeting - watch this space!

Kind regards,

Claire Robson
GPFS User Group Secretary


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org<http://gpfsug.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131023/083b2225/attachment-0002.htm>

From chair at gpfsug.org  Fri Oct 25 17:09:16 2013
From: chair at gpfsug.org (Chair GPFS UG)
Date: Fri, 25 Oct 2013 17:09:16 +0100
Subject: [gpfsug-discuss] Friday MPI fun?
Message-ID: <CA+M26=T7eeSMWhV_GFFvEkaKnDHNxF9OXqHbg4DvfVhnWN8CfA@mail.gmail.com>

Allo all,

  I'm attempting to cheat.  As per-usual, cheating takes more time than
'Doing It Properly' - but it is vastly more fun.

So without setting up Grid or Moab etc, I need to pin processes to a cpu.
 I.E. on Linux: taskset blah blah.
I could write a small housekeeping script which RR new spawned processes
across CPUs using taskset, but I was wondering if OpenMPI could be a good
way to go.

So:

I have a non-MPI application X.
Which is spawned and forked by a parent process into its own process group.
This can occur at any time, however there will only ever be a maximium N of
appl X.

Using mpirun it appears that you can set off parallel instances of a
non-MPI application:

mpirun -np 4 applicationX

However, really what I'm requiring to do is say:
  Max slots = N (which I can define in the mpi hostfile).

mpirun -np 1 applicationX
mpirun -np 1 applicationX (started at a random future time)
mpirun -np 1 applicationX (started at a random future time)
mpirun -np 1 applicationX (started at a random future time)

Each being automatically pinned to a CPU, but I'm fairly convinced this is
not the way MPI works.
Would it do what I'm after?

Does anyone know of a better way?

Jez
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131025/37f6a4c9/attachment-0002.htm>

From zgiles at gmail.com  Sat Oct 26 05:13:37 2013
From: zgiles at gmail.com (Zachary Giles)
Date: Sat, 26 Oct 2013 00:13:37 -0400
Subject: [gpfsug-discuss] Friday MPI fun?
In-Reply-To: <CA+M26=T7eeSMWhV_GFFvEkaKnDHNxF9OXqHbg4DvfVhnWN8CfA@mail.gmail.com>
References: <CA+M26=T7eeSMWhV_GFFvEkaKnDHNxF9OXqHbg4DvfVhnWN8CfA@mail.gmail.com>
Message-ID: <CAMYZk=esiDC4fq3DnqngyttsvvYCSBNOO2vZBsF=uJdT=fcU2Q@mail.gmail.com>

You could do it that way, if you told MPI to bind each task (MPI task, or
non-MPI task in your case) to a processor. By default, most MPI's don't
bind to a specific processor unless the scheduler (you don't have one) or
something else tells it to. OpenMPI, for example, I think, has something
like --bind-to-cpu, or --bind-to-socket. Often a scheduler or c-groups via
a scheduler will pass-in a list of cpu's to bind to, but you could do it
manually with the appropriate flags and files for your MPI distro.

You could also probably get away with just raw c-groups depending on your
distro. You would probably touch a few files and echo a few lines to files
to make the right groups then "cgrun <app>" to get things in to those
groups. That works pretty well for binding, found decent results with that,
I recommend it.


On Fri, Oct 25, 2013 at 12:09 PM, Chair GPFS UG <chair at gpfsug.org> wrote:

> Allo all,
>
>   I'm attempting to cheat.  As per-usual, cheating takes more time than
> 'Doing It Properly' - but it is vastly more fun.
>
> So without setting up Grid or Moab etc, I need to pin processes to a cpu.
>  I.E. on Linux: taskset blah blah.
> I could write a small housekeeping script which RR new spawned processes
> across CPUs using taskset, but I was wondering if OpenMPI could be a good
> way to go.
>
> So:
>
> I have a non-MPI application X.
> Which is spawned and forked by a parent process into its own process group.
> This can occur at any time, however there will only ever be a maximium N
> of appl X.
>
> Using mpirun it appears that you can set off parallel instances of a
> non-MPI application:
>
> mpirun -np 4 applicationX
>
> However, really what I'm requiring to do is say:
>   Max slots = N (which I can define in the mpi hostfile).
>
> mpirun -np 1 applicationX
> mpirun -np 1 applicationX (started at a random future time)
> mpirun -np 1 applicationX (started at a random future time)
> mpirun -np 1 applicationX (started at a random future time)
>
> Each being automatically pinned to a CPU, but I'm fairly convinced this is
> not the way MPI works.
> Would it do what I'm after?
>
> Does anyone know of a better way?
>
> Jez
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>


-- 
Zach Giles
zgiles at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131026/217d585e/attachment-0002.htm>

From richard.lefebvre at calculquebec.ca  Mon Oct 28 15:31:07 2013
From: richard.lefebvre at calculquebec.ca (Richard Lefebvre)
Date: Mon, 28 Oct 2013 11:31:07 -0400
Subject: [gpfsug-discuss] Moving/copying files from one file system to
	another
Message-ID: <526E833B.6040402@calculquebec.ca>

Hi,

I'm looking to copy/move files from one gpfs file system to another in
some efficient way. Initially I was thinking to not create another file
system and just add the nsd to the current file system and gradually
remove the old nsd. That would have been the easiest with little impact
to the users. But I wanted to switch from a scatter allocation to a
cluster allocation (too improve performance) and I cannot to that if I
just add/remove nsd. I'm hopping for a better solution then a plain
rsync which would probably take a few days (~210TB).

Is there a gpfs solution to this?

Also, another question, under what condition a scatter allocation better
then cluster allocation. We currently have a cluster of 650 nodes all
accessing the same 230TB gpfs file system.

Richard


From seanlee at tw.ibm.com  Wed Oct 30 12:30:26 2013
From: seanlee at tw.ibm.com (Sean S Lee)
Date: Wed, 30 Oct 2013 20:30:26 +0800
Subject: [gpfsug-discuss] Moving/copying files from one file system to
	another
In-Reply-To: <mailman.1.1383134402.20340.gpfsug-discuss@gpfsug.org>
References: <mailman.1.1383134402.20340.gpfsug-discuss@gpfsug.org>
Message-ID: <OF9B62D9AF.094922A5-ON48257C14.00438E14-48257C14.0044C452@tw.ibm.com>


Hi Richard,

==========

The cluster allocation method may provide better disk performance for some
disk subsystems in
relatively small installations. The benefits of clustered block allocation
diminish when the number
of nodes in the cluster or the number of disks in a file system increases,
or when the file system's
free space becomes fragmented. The cluster allocation method is the default
for GPFS clusters
with eight or fewer nodes and for file systems with eight or fewer disks.

The scatter allocation method provides more consistent file system
performance by averaging out
performance variations due to block location (for many disk subsystems, the
location of the data
relative to the disk edge has a substantial effect on performance). This
allocation method is
appropriate in most cases and is the default for GPFS clusters with more
than eight nodes or file
systems with more than eight disks.

Source: GPFS Administration and Programming Reference
==========

Based on above it seems the scatter allocation would be more suitable for
your environment (more nodes, more disks), no?

The Web forum has few threads on GPFS copying, e.g. take a look at this:
https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014934284&ps=25

I don't know if there's a better way or not.

If you don't change the allocation type, then you don't have to do anything
but add the disks and perhaps restripe.

Regards
Sean
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131030/5c77c624/attachment-0002.htm>

From jonathan at buzzard.me.uk  Wed Oct 30 12:47:07 2013
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Wed, 30 Oct 2013 12:47:07 +0000
Subject: [gpfsug-discuss] Moving/copying files from one file system to
 another
In-Reply-To: <526E833B.6040402@calculquebec.ca>
References: <526E833B.6040402@calculquebec.ca>
Message-ID: <1383137227.2921.4.camel@buzzard.phy.strath.ac.uk>

On Mon, 2013-10-28 at 11:31 -0400, Richard Lefebvre wrote:

[SNIP]

> Also, another question, under what condition a scatter allocation better
> then cluster allocation. We currently have a cluster of 650 nodes all
> accessing the same 230TB gpfs file system.
> 

Scatter allocation is better in almost all circumstances. Basically by
scattering the files to all corners you don't get hotspots where just a
small subset of the disks are being hammered by lots of accesses to a
handful of files, while the rest of the disks sit idle.

JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.



From chekh at stanford.edu  Wed Oct 30 17:56:46 2013
From: chekh at stanford.edu (Alex Chekholko)
Date: Wed, 30 Oct 2013 10:56:46 -0700
Subject: [gpfsug-discuss] Moving/copying files from one file system to
 another
In-Reply-To: <1383137227.2921.4.camel@buzzard.phy.strath.ac.uk>
References: <526E833B.6040402@calculquebec.ca>
	<1383137227.2921.4.camel@buzzard.phy.strath.ac.uk>
Message-ID: <5271485E.80405@stanford.edu>

On 10/30/13, 5:47 AM, Jonathan Buzzard wrote:
> On Mon, 2013-10-28 at 11:31 -0400, Richard Lefebvre wrote:
>
> [SNIP]
>
>> Also, another question, under what condition a scatter allocation better
>> then cluster allocation. We currently have a cluster of 650 nodes all
>> accessing the same 230TB gpfs file system.
>>
>
> Scatter allocation is better in almost all circumstances. Basically by
> scattering the files to all corners you don't get hotspots where just a
> small subset of the disks are being hammered by lots of accesses to a
> handful of files, while the rest of the disks sit idle.
>

If you do benchmarks with only a few threads, you will see higher 
performance with 'cluster' allocation.  So if your workload is only a 
few clients accessing the FS in a mostly streaming way, you'd see better 
performance from 'cluster'.

With 650 nodes, even if each client is doing streaming reads, at the 
filesystem level that would all be interleaved and thus be random reads. 
  But it's tough to do a big enough benchmark to show the difference in 
performance.

I had a tough time convincing people to use 'scatter' instead of 
'cluster' even though I think the documentation is clear about the 
difference, and even gives you the sizing parameters ( greater than 8 
disks or 8 NSDs?  use 'scatter').

We use 'scatter' now.

Regards
-- 
chekh at stanford.edu


From TOMP at il.ibm.com  Wed Oct 30 18:31:01 2013
From: TOMP at il.ibm.com (Tomer Perry)
Date: Wed, 30 Oct 2013 20:31:01 +0200
Subject: [gpfsug-discuss] Moving/copying files from one file system to
	another
In-Reply-To: <5271485E.80405@stanford.edu>
References: <526E833B.6040402@calculquebec.ca>	<1383137227.2921.4.camel@buzzard.phy.strath.ac.uk>
	<5271485E.80405@stanford.edu>
Message-ID: <OF39DC62FE.1C8ABCF9-ONC2257C14.00657D38-C2257C14.0065BC4C@il.ibm.com>

Hi,

cluster will also be there for some time, as soon as one start deleting 
and creating file - it will kind of shift to be scattered...
So, while its good for benchmarks on "clean and new" system - cluster is 
somewhat meaningless ( and was hidden for some time because of that).

As far as it goes for copying from old to new system - AFM can be 
considered as well ( will make the transition easier). You can prefetch ( 
using mmafmctl) and even work in LU mode in order to get the data from the 
old FS to the new one without pushing changes back.

hth,
Tomer.




From:   Alex Chekholko <chekh at stanford.edu>
To:     gpfsug-discuss at gpfsug.org, 
Date:   10/30/2013 07:57 PM
Subject:        Re: [gpfsug-discuss] Moving/copying files from one file 
system to another
Sent by:        gpfsug-discuss-bounces at gpfsug.org



On 10/30/13, 5:47 AM, Jonathan Buzzard wrote:
> On Mon, 2013-10-28 at 11:31 -0400, Richard Lefebvre wrote:
>
> [SNIP]
>
>> Also, another question, under what condition a scatter allocation 
better
>> then cluster allocation. We currently have a cluster of 650 nodes all
>> accessing the same 230TB gpfs file system.
>>
>
> Scatter allocation is better in almost all circumstances. Basically by
> scattering the files to all corners you don't get hotspots where just a
> small subset of the disks are being hammered by lots of accesses to a
> handful of files, while the rest of the disks sit idle.
>

If you do benchmarks with only a few threads, you will see higher 
performance with 'cluster' allocation.  So if your workload is only a 
few clients accessing the FS in a mostly streaming way, you'd see better 
performance from 'cluster'.

With 650 nodes, even if each client is doing streaming reads, at the 
filesystem level that would all be interleaved and thus be random reads. 
  But it's tough to do a big enough benchmark to show the difference in 
performance.

I had a tough time convincing people to use 'scatter' instead of 
'cluster' even though I think the documentation is clear about the 
difference, and even gives you the sizing parameters ( greater than 8 
disks or 8 NSDs?  use 'scatter').

We use 'scatter' now.

Regards
-- 
chekh at stanford.edu
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131030/d130f587/attachment-0002.htm>

From jfosburg at mdanderson.org  Wed Oct  2 21:44:55 2013
From: jfosburg at mdanderson.org (Fosburgh,Jonathan)
Date: Wed, 2 Oct 2013 20:44:55 +0000
Subject: [gpfsug-discuss] Copying ACLs from outside sources
Message-ID: <CE71EFF4.200B4%jfosburg@mdanderson.org>

Hello, I'm new to this list, as well as GPFS.

We have new (installed over the last 6 weeks) GPFS cluster that we are currently trying to get into service.  The NSDs are running on Red Hat (6 NSDs), the storage is DCS3700 directly attached via SAS.  We are running GPFS 3.5.0.12.


Some of the data that will be placed on this cluster will be existing data coming from other sources, one of which is our NAS (Isilon, supporting both NFS and CIFS).  We have tried a few different ways of copying the data.  First we tried to rsync directly between an NSD and one of the Isilon nodes.  Unfortunately, it appears the rsync doesn't work with the ACL format used on the Isilon.  We have also tried using various methods of copying data via CIFS, but we continue to have no luck getting ACLs to copy.  We know that, for instance, copying data via CIFS into the Isilon using robocopy will preserve the ACLs, and we were hoping that would work for getting data onto GPFS as well.  However, the ACLs are being stripped and replaced with whatever ACL is present on GPFS.  Does anyone have experience doing something like that who can give me some pointers on what my configuration should look like? I assume there are samba and/or GPFS parameters that need to be tweaked in order to make this work, especially since this functionality would have to be present in SONAS.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131002/a1ef2992/attachment-0003.htm>

From pete at realisestudio.com  Thu Oct  3 09:09:07 2013
From: pete at realisestudio.com (Pete Smith)
Date: Thu, 3 Oct 2013 09:09:07 +0100
Subject: [gpfsug-discuss] Copying ACLs from outside sources
In-Reply-To: <CE71EFF4.200B4%jfosburg@mdanderson.org>
References: <CE71EFF4.200B4%jfosburg@mdanderson.org>
Message-ID: <CAM9ZKkjHcSK9RgmLJKL=QedA=TN+6NhZZhHtWgpyLB=Za7S+EQ@mail.gmail.com>

Hi, and welcome, Jonathan.

Which combination of -A and -X rsync options did you use?

Pete

On 2 October 2013 21:44, Fosburgh,Jonathan <jfosburg at mdanderson.org> wrote:
> Hello, I'm new to this list, as well as GPFS.
>
> We have new (installed over the last 6 weeks) GPFS cluster that we are
> currently trying to get into service.  The NSDs are running on Red Hat (6
> NSDs), the storage is DCS3700 directly attached via SAS.  We are running
> GPFS 3.5.0.12.
>
>
> Some of the data that will be placed on this cluster will be existing data
> coming from other sources, one of which is our NAS (Isilon, supporting both
> NFS and CIFS).  We have tried a few different ways of copying the data.
> First we tried to rsync directly between an NSD and one of the Isilon nodes.
> Unfortunately, it appears the rsync doesn't work with the ACL format used on
> the Isilon.  We have also tried using various methods of copying data via
> CIFS, but we continue to have no luck getting ACLs to copy.  We know that,
> for instance, copying data via CIFS into the Isilon using robocopy will
> preserve the ACLs, and we were hoping that would work for getting data onto
> GPFS as well.  However, the ACLs are being stripped and replaced with
> whatever ACL is present on GPFS.  Does anyone have experience doing
> something like that who can give me some pointers on what my configuration
> should look like? I assume there are samba and/or GPFS parameters that need
> to be tweaked in order to make this work, especially since this
> functionality would have to be present in SONAS.
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>



-- 
Pete Smith
DevOp/System Administrator
Realise Studio
12/13 Poland Street, London W1F 8QB
T. +44 (0)20 7165 9644

realisestudio.com


From orlando.richards at ed.ac.uk  Thu Oct  3 09:17:47 2013
From: orlando.richards at ed.ac.uk (Orlando Richards)
Date: Thu, 03 Oct 2013 09:17:47 +0100
Subject: [gpfsug-discuss] Copying ACLs from outside sources
In-Reply-To: <CAM9ZKkjHcSK9RgmLJKL=QedA=TN+6NhZZhHtWgpyLB=Za7S+EQ@mail.gmail.com>
References: <CE71EFF4.200B4%jfosburg@mdanderson.org>
	<CAM9ZKkjHcSK9RgmLJKL=QedA=TN+6NhZZhHtWgpyLB=Za7S+EQ@mail.gmail.com>
Message-ID: <524D282B.905@ed.ac.uk>

Hi Jonathan,

Might be a bit of a long shot this - but for copying NFSv4 ACLs between 
GPFS filesystems using rsync, I had to patch rsync with some code 
released by IBM at some point. We've got that patched rsync hosted on 
the gpfsug github:

   https://github.com/gpfsug/gpfsug-tools/tree/master/bin/rsync

Since you're not doing gpfs -> gpfs, it might be no use - but you never 
know!

Oh - and, of course, make sure your GPFS filesystem is configured to 
support NFSv4 ACLs.

Here's our smb.conf settings related to GPFS, ACLs and other bits and bobs:

vfs objects = shadow_copy2, fileid, gpfs
clustering = yes
fileid:mapping = fsname
gpfs:sharemodes = No
gpfs:getrealfilename = no
nfs4: mode = simple
nfs4: chown = yes
nfs4: acedup = merge
force unknown acl user = yes
acl group control = no
map acl inherit = yes
inherit acls = no
dos filemode = no
map readonly = no
map hidden = no
map archive = no
map readonly = no
ea support = yes
store dos attributes = yes
gpfs:winattr = yes
max protocol = SMB2
shadow:fixinodes = yes
shadow:snapdir = .snapshots
shadow:basedir = /exports/nas
use sendfile = yes
wide links = no
unix extensions = no

Hope that helps,
Orlando

--
Orlando

On 03/10/13 09:09, Pete Smith wrote:
> Hi, and welcome, Jonathan.
>
> Which combination of -A and -X rsync options did you use?
>
> Pete
>
> On 2 October 2013 21:44, Fosburgh,Jonathan <jfosburg at mdanderson.org> wrote:
>> Hello, I'm new to this list, as well as GPFS.
>>
>> We have new (installed over the last 6 weeks) GPFS cluster that we are
>> currently trying to get into service.  The NSDs are running on Red Hat (6
>> NSDs), the storage is DCS3700 directly attached via SAS.  We are running
>> GPFS 3.5.0.12.
>>
>>
>> Some of the data that will be placed on this cluster will be existing data
>> coming from other sources, one of which is our NAS (Isilon, supporting both
>> NFS and CIFS).  We have tried a few different ways of copying the data.
>> First we tried to rsync directly between an NSD and one of the Isilon nodes.
>> Unfortunately, it appears the rsync doesn't work with the ACL format used on
>> the Isilon.  We have also tried using various methods of copying data via
>> CIFS, but we continue to have no luck getting ACLs to copy.  We know that,
>> for instance, copying data via CIFS into the Isilon using robocopy will
>> preserve the ACLs, and we were hoping that would work for getting data onto
>> GPFS as well.  However, the ACLs are being stripped and replaced with
>> whatever ACL is present on GPFS.  Does anyone have experience doing
>> something like that who can give me some pointers on what my configuration
>> should look like? I assume there are samba and/or GPFS parameters that need
>> to be tweaked in order to make this work, especially since this
>> functionality would have to be present in SONAS.
>>
>>
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at gpfsug.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>
>
>
>


-- 
             --
    Dr Orlando Richards
   Information Services
IT Infrastructure Division
        Unix Section
     Tel: 0131 650 4994
   skype: orlando.richards

The University of Edinburgh is a charitable body, registered in 
Scotland, with registration number SC005336.


From jonathan at buzzard.me.uk  Thu Oct  3 10:23:02 2013
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Thu, 03 Oct 2013 10:23:02 +0100
Subject: [gpfsug-discuss] Copying ACLs from outside sources
In-Reply-To: <CE71EFF4.200B4%jfosburg@mdanderson.org>
References: <CE71EFF4.200B4%jfosburg@mdanderson.org>
Message-ID: <1380792182.19439.18.camel@buzzard.phy.strath.ac.uk>

On Wed, 2013-10-02 at 20:44 +0000, Fosburgh,Jonathan wrote: 
> Hello, I'm new to this list, as well as GPFS.  
> 
> 
> We have new (installed over the last 6 weeks) GPFS cluster that we are
> currently trying to get into service.  The NSDs are running on Red Hat
> (6 NSDs), the storage is DCS3700 directly attached via SAS.  We are
> running GPFS 3.5.0.12.
> 
> Some of the data that will be placed on this cluster will be existing
> data coming from other sources, one of which is our NAS (Isilon,
> supporting both NFS and CIFS).  We have tried a few different ways of
> copying the data.  First we tried to rsync directly between an NSD and
> one of the Isilon nodes.  Unfortunately, it appears the rsync doesn't
> work with the ACL format used on the Isilon.  We have also tried using
> various methods of copying data via CIFS, but we continue to have no
> luck getting ACLs to copy.  We know that, for instance, copying data
> via CIFS into the Isilon using robocopy will preserve the ACLs, and we
> were hoping that would work for getting data onto GPFS as well.
> However, the ACLs are being stripped and replaced with whatever ACL is
> present on GPFS.  Does anyone have experience doing something like
> that who can give me some pointers on what my configuration should
> look like? I assume there are samba and/or GPFS parameters that need
> to be tweaked in order to make this work, especially since this
> functionality would have to be present in SONAS.
> 

I guess the first thing to do is confirm that you have functioning
CIFS's ACL's in your Samba GPFS file server. Specifically you would need
to have NFSv4 ACL's enabled in the file system and the gpfs VFS module
loaded in Samba. Note having mixed ACL modes did not work very well for
me, so best to turn off Posix ACL's and only do NFSv4 ACL's.

These are the samba options I use in combination with GPFS to make the
server as windows like as possible.

# general options
vfs objects = fileid gpfs
        unix extensions = no
        mangled names = no
        case sensitive = no

# store DOS attributes in extended attributes (vfs_gpfs then stores them
in the file system)
ea support = yes
store dos attributes = yes
map readonly = no
map archive = no
map system = no

# the ctdb clustering and GPFS stuff
clustering = yes
ctdbd socket = /tmp/ctdb.socket
        fileid : algorithm = fsname
        gpfs : sharemodes = yes
        gpfs : winattr = yes
        force unknown acl user = yes
        nfs4 : mode = special
        nfs4 : chown = no
        nfs4 : acedup = merge

The best way to make sure it is working is point a Windows client at the
file server and manipulate the ACL's by hand.

Note that if you are going for that full make my Samba/GPFS file server
look as close as possible to a pucker MS Windows server, you might want
to consider setting the following GPFS options

     cifsBypassShareLocksOnRename
     cifsBypassTraversalChecking
     allowWriteWithDeleteChild

All fairly self explanatory, and make GPFS follow Windows schematics
more closely, though they are "undocumented". There are a couple of
other options as allowSambaCaseInsensitiveLookup and
syncSambaMetadataOps but I have not determined exactly what they do...

There is also there is an undocumented option for ACL's on mmchfs 

      mmchfs test -k samba

Even shows up in the output of mmlsfs. Not entirely sure what samba
ACL's are mind you...

Note that IBM have alluded at various times that there these "magic"
options exist and are used in SONAS etc.

Anyway getting back to ACL's robocopy will move them over, but in my
experience copying from a Windows file server the results where not what
I wanted. I ended up with extra ACL's for administrators and bizzare
file ownership issues. Some of that might be because the files had
originally been on a Netware file server which died, and where hastily
moved to a Windows server. I gave up in the end and manually unpicked
what the ACL's where doing for the migration to the GPFS file server.
However I only had 20GB of data...

You should also find that xcopy will move them over as well.

A possible alternative if you have access to an AIX box would be to
NFSv4 mount the Isilon on the AIX box and mount GPFS as well and just
use cp.


JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.




From jfosburg at mdanderson.org  Wed Oct  9 18:31:45 2013
From: jfosburg at mdanderson.org (Fosburgh,Jonathan)
Date: Wed, 9 Oct 2013 17:31:45 +0000
Subject: [gpfsug-discuss] Copying ACLs from outside sources
In-Reply-To: <1380792182.19439.18.camel@buzzard.phy.strath.ac.uk>
Message-ID: <CE7AFC34.21539%jfosburg@mdanderson.org>



On 10/3/13 4:23 AM, "Jonathan Buzzard" <jonathan at buzzard.me.uk> wrote:

>
>> 
>
>I guess the first thing to do is confirm that you have functioning
>CIFS's ACL's in your Samba GPFS file server. Specifically you would need
>to have NFSv4 ACL's enabled in the file system and the gpfs VFS module
>loaded in Samba. Note having mixed ACL modes did not work very well for
>me, so best to turn off Posix ACL's and only do NFSv4 ACL's.
>
Thank you for the response. We have been able to make some headway on
this, but I am still bumping up on some issues.  It looks as if, when an
NFSv4 ACL is already present on the parent directory tree, copying acls
via robocopy works. However, if there is a posix ACL present, then the
ACLs are still not copied correctly.  I was interested in setting the ACL
mode to nfs4 only (currently set to all), however, I have some
reservations about this.

The NFS server is Red Hat, and I need to be able to server version 3 and
version 4, plus we will have native GPFS clients and the CIFS clients.
What are the ramifications, in this setup, of changing the ACL type? What
would happen to any existing ACLs?



From jonathan at buzzard.me.uk  Wed Oct  9 20:54:53 2013
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Wed, 09 Oct 2013 20:54:53 +0100
Subject: [gpfsug-discuss] Copying ACLs from outside sources
In-Reply-To: <CE7AFC34.21539%jfosburg@mdanderson.org>
References: <CE7AFC34.21539%jfosburg@mdanderson.org>
Message-ID: <5255B48D.1070401@buzzard.me.uk>

On 09/10/13 18:31, Fosburgh,Jonathan wrote:
>
>
> On 10/3/13 4:23 AM, "Jonathan Buzzard" <jonathan at buzzard.me.uk> wrote:
>
>>
>>>
>>
>> I guess the first thing to do is confirm that you have functioning
>> CIFS's ACL's in your Samba GPFS file server. Specifically you would need
>> to have NFSv4 ACL's enabled in the file system and the gpfs VFS module
>> loaded in Samba. Note having mixed ACL modes did not work very well for
>> me, so best to turn off Posix ACL's and only do NFSv4 ACL's.
>>
> Thank you for the response. We have been able to make some headway on
> this, but I am still bumping up on some issues.  It looks as if, when an
> NFSv4 ACL is already present on the parent directory tree, copying acls
> via robocopy works. However, if there is a posix ACL present, then the
> ACLs are still not copied correctly.  I was interested in setting the ACL
> mode to nfs4 only (currently set to all), however, I have some
> reservations about this.
>

Like I said in my experience having both Posix and NFSv4 ACL's active at 
the same time did not work properly as you have found out. You have to 
pick one and stick with it. Clearly if you have an existing system with 
both ACL types on then you have problems.

> The NFS server is Red Hat, and I need to be able to server version 3 and
> version 4, plus we will have native GPFS clients and the CIFS clients.
> What are the ramifications, in this setup, of changing the ACL type? What
> would happen to any existing ACLs?

My guess is the Posix ACL's either get lost or are "converted" into 
NFSv4 ACL's. I would try this out on your "development/test" GPFS setup 
first and work out a strategy for moving forward.

The biggest problem is that the IBM provided tools for manipulating the 
GPFS ACL's from within Linux are awful.


JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.


From crobson at ocf.co.uk  Tue Oct 22 17:27:24 2013
From: crobson at ocf.co.uk (Claire Robson)
Date: Tue, 22 Oct 2013 17:27:24 +0100
Subject: [gpfsug-discuss] SC'13 World Wide User Forum
Message-ID: <A42128435E851644B9B011BB824F6C816E24AEDFA7@MAIL.ocf.local>

Dear Members,

For anyone who is attending Supercomputing next month, there is a Worldwide GPFS User Forum taking place on Monday 18th November from 1-5pm. The GPFS User Forum is a worldwide gathering for current and new users of GPFS. Various topics will be discussed by current clients and IBM teams, including discussions on deployments, GPFS Storage Server, roadmaps and more!

https://www-950.ibm.com/events/wwe/grp/grp017.nsf/v17_enrollall?openform&seminar=D7QQ56ES&locale=en_US

Agenda
1:00       Welcome and Formalities              Andrew Dean, GPFS User Group and Dave McDonnell, IBM
1:15       ANL's GPFS GSS Storage System William Allcock, Argonne Leadership Computing Facility
1:45       GPFS Update Scott Fadden, IBM
2:30       Break Refreshments
2:45       GSS Update Matt Drahzal, IBM
3:30       NCAR's GLADE GPFS Environment Pamela Gillman, NCAR Computational Information Sciences Laboratory
4:00       GPFS Roadmap (NDA Required)    Krystal Rothaup, IBM (Session Moved to Centennial D for SPXXL Joint NDA Session)

Enjoy the event of you are able to make it. I hope to be in touch again soon with a date for the next UK meeting - watch this space!

Kind regards,

Claire Robson
GPFS User Group Secretary

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131022/2ccf16ee/attachment-0003.htm>

From oester at gmail.com  Tue Oct 22 18:19:20 2013
From: oester at gmail.com (Bob Oesterlin)
Date: Tue, 22 Oct 2013 12:19:20 -0500
Subject: [gpfsug-discuss] SC'13 World Wide User Forum
In-Reply-To: <A42128435E851644B9B011BB824F6C816E24AEDFA7@MAIL.ocf.local>
References: <A42128435E851644B9B011BB824F6C816E24AEDFA7@MAIL.ocf.local>
Message-ID: <CAMNdFvB8_1WJvuZsyhbntU53s5oN9t3Kkx7H2UW83v3O0EQVbQ@mail.gmail.com>

Where is this event being held?


Bob Oesterlin
Nuance Communications



On Tue, Oct 22, 2013 at 11:27 AM, Claire Robson <crobson at ocf.co.uk> wrote:

> Dear Members,****
>
> ** **
>
> For anyone who is attending Supercomputing next month, there is a
> Worldwide GPFS User Forum taking place on Monday 18th November from
> 1-5pm. The GPFS User Forum is a worldwide gathering for current and new
> users of GPFS. Various topics will be discussed by current clients and IBM
> teams, including discussions on deployments, GPFS Storage Server, roadmaps
> and more! ****
>
> ** **
>
>
> https://www-950.ibm.com/events/wwe/grp/grp017.nsf/v17_enrollall?openform&seminar=D7QQ56ES&locale=en_US
> ****
>
> ** **
>
> Agenda****
>
> 1:00       Welcome and Formalities              Andrew Dean, GPFS User
> Group and Dave McDonnell, IBM ****
>
> 1:15       ANL's GPFS GSS Storage System William Allcock, Argonne
> Leadership Computing Facility ****
>
> 1:45       GPFS Update Scott Fadden, IBM****
>
> 2:30       Break Refreshments****
>
> 2:45       GSS Update Matt Drahzal, IBM****
>
> 3:30       NCAR's GLADE GPFS Environment Pamela Gillman, NCAR
> Computational Information Sciences Laboratory****
>
> 4:00       GPFS Roadmap (NDA Required)    Krystal Rothaup, IBM (Session
> Moved to Centennial D for SPXXL Joint NDA Session)****
>
> ** **
>
> Enjoy the event of you are able to make it. I hope to be in touch again
> soon with a date for the next UK meeting ? watch this space!****
>
> ** **
>
> Kind regards,****
>
> ** **
>
> *Claire Robson*
>
> GPFS User Group Secretary****
>
> ** **
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131022/13013ddd/attachment-0003.htm>

From sfadden at us.ibm.com  Tue Oct 22 20:41:24 2013
From: sfadden at us.ibm.com (Scott Fadden)
Date: Tue, 22 Oct 2013 12:41:24 -0700
Subject: [gpfsug-discuss] SC'13 World Wide User Forum
In-Reply-To: <CAMNdFvB8_1WJvuZsyhbntU53s5oN9t3Kkx7H2UW83v3O0EQVbQ@mail.gmail.com>
References: <A42128435E851644B9B011BB824F6C816E24AEDFA7@MAIL.ocf.local>
	<CAMNdFvB8_1WJvuZsyhbntU53s5oN9t3Kkx7H2UW83v3O0EQVbQ@mail.gmail.com>
Message-ID: <OF0E566F5E.8E3EA689-ON88257C0C.006C2731-88257C0C.006C2938@us.ibm.com>



Hyatt Regency
650 15th Street
Denver, Colorado
United States


Scott Fadden
GPFS Technical Marketing
Phone: (503) 880-5833
sfadden at us.ibm.com
http://www.ibm.com/systems/gpfs


|------------>
| From:      |
|------------>
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
  |Bob Oesterlin <oester at gmail.com>                                                                                                               |
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| To:        |
|------------>
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
  |gpfsug main discussion list <gpfsug-discuss at gpfsug.org>,                                                                                       |
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Date:      |
|------------>
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
  |10/22/2013 10:28 AM                                                                                                                            |
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Subject:   |
|------------>
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
  |Re: [gpfsug-discuss] SC'13 World Wide User Forum                                                                                               |
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Sent by:   |
|------------>
  >-----------------------------------------------------------------------------------------------------------------------------------------------|
  |gpfsug-discuss-bounces at gpfsug.org                                                                                                              |
  >-----------------------------------------------------------------------------------------------------------------------------------------------|





Where is this event being held?


Bob Oesterlin
Nuance Communications



On Tue, Oct 22, 2013 at 11:27 AM, Claire Robson <crobson at ocf.co.uk> wrote:
  Dear Members,





  For anyone who is attending Supercomputing next month, there is a
  Worldwide GPFS User Forum taking place on Monday 18th November from
  1-5pm. The GPFS User Forum is a worldwide gathering for current and new
  users of GPFS. Various topics will be discussed by current clients and
  IBM teams, including discussions on deployments, GPFS Storage Server,
  roadmaps and more!





  https://www-950.ibm.com/events/wwe/grp/grp017.nsf/v17_enrollall?openform&seminar=D7QQ56ES&locale=en_US





  Agenda


  1:00 ????? Welcome and Formalities????????????? Andrew Dean, GPFS User
  Group and Dave McDonnell, IBM


  1:15?????? ANL's GPFS GSS Storage System William Allcock, Argonne
  Leadership Computing Facility


  1:45?????? GPFS Update Scott Fadden, IBM


  2:30?????? Break Refreshments


  2:45?????? GSS Update Matt Drahzal, IBM


  3:30?????? NCAR's GLADE GPFS Environment Pamela Gillman, NCAR
  Computational Information Sciences Laboratory


  4:00?????? GPFS Roadmap (NDA Required)??? Krystal Rothaup, IBM (Session
  Moved to Centennial D for SPXXL Joint NDA Session)





  Enjoy the event of you are able to make it. I hope to be in touch again
  soon with a date for the next UK meeting ? watch this space!





  Kind regards,





  Claire Robson


  GPFS User Group Secretary




  _______________________________________________
  gpfsug-discuss mailing list
  gpfsug-discuss at gpfsug.org
  http://gpfsug.org/mailman/listinfo/gpfsug-discuss

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131022/140a00b7/attachment-0003.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131022/140a00b7/attachment-0006.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ecblank.gif
Type: image/gif
Size: 45 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131022/140a00b7/attachment-0007.gif>

From crobson at ocf.co.uk  Wed Oct 23 09:04:32 2013
From: crobson at ocf.co.uk (Claire Robson)
Date: Wed, 23 Oct 2013 09:04:32 +0100
Subject: [gpfsug-discuss] SC'13 World Wide User Forum
In-Reply-To: <CAMNdFvB8_1WJvuZsyhbntU53s5oN9t3Kkx7H2UW83v3O0EQVbQ@mail.gmail.com>
References: <A42128435E851644B9B011BB824F6C816E24AEDFA7@MAIL.ocf.local>
	<CAMNdFvB8_1WJvuZsyhbntU53s5oN9t3Kkx7H2UW83v3O0EQVbQ@mail.gmail.com>
Message-ID: <A42128435E851644B9B011BB824F6C816E24AEDFB8@MAIL.ocf.local>

Supercomputing is taking place at the Colorado Convention Center in Denver, Colorado, USA from 17-22nd November 2013.

Thanks,
Claire

From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Bob Oesterlin
Sent: 22 October 2013 18:19
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] SC'13 World Wide User Forum

Where is this event being held?


Bob Oesterlin
Nuance Communications


On Tue, Oct 22, 2013 at 11:27 AM, Claire Robson <crobson at ocf.co.uk<mailto:crobson at ocf.co.uk>> wrote:
Dear Members,

For anyone who is attending Supercomputing next month, there is a Worldwide GPFS User Forum taking place on Monday 18th November from 1-5pm. The GPFS User Forum is a worldwide gathering for current and new users of GPFS. Various topics will be discussed by current clients and IBM teams, including discussions on deployments, GPFS Storage Server, roadmaps and more!

https://www-950.ibm.com/events/wwe/grp/grp017.nsf/v17_enrollall?openform&seminar=D7QQ56ES&locale=en_US

Agenda
1:00       Welcome and Formalities              Andrew Dean, GPFS User Group and Dave McDonnell, IBM
1:15       ANL's GPFS GSS Storage System William Allcock, Argonne Leadership Computing Facility
1:45       GPFS Update Scott Fadden, IBM
2:30       Break Refreshments
2:45       GSS Update Matt Drahzal, IBM
3:30       NCAR's GLADE GPFS Environment Pamela Gillman, NCAR Computational Information Sciences Laboratory
4:00       GPFS Roadmap (NDA Required)    Krystal Rothaup, IBM (Session Moved to Centennial D for SPXXL Joint NDA Session)

Enjoy the event of you are able to make it. I hope to be in touch again soon with a date for the next UK meeting - watch this space!

Kind regards,

Claire Robson
GPFS User Group Secretary


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org<http://gpfsug.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131023/083b2225/attachment-0003.htm>

From chair at gpfsug.org  Fri Oct 25 17:09:16 2013
From: chair at gpfsug.org (Chair GPFS UG)
Date: Fri, 25 Oct 2013 17:09:16 +0100
Subject: [gpfsug-discuss] Friday MPI fun?
Message-ID: <CA+M26=T7eeSMWhV_GFFvEkaKnDHNxF9OXqHbg4DvfVhnWN8CfA@mail.gmail.com>

Allo all,

  I'm attempting to cheat.  As per-usual, cheating takes more time than
'Doing It Properly' - but it is vastly more fun.

So without setting up Grid or Moab etc, I need to pin processes to a cpu.
 I.E. on Linux: taskset blah blah.
I could write a small housekeeping script which RR new spawned processes
across CPUs using taskset, but I was wondering if OpenMPI could be a good
way to go.

So:

I have a non-MPI application X.
Which is spawned and forked by a parent process into its own process group.
This can occur at any time, however there will only ever be a maximium N of
appl X.

Using mpirun it appears that you can set off parallel instances of a
non-MPI application:

mpirun -np 4 applicationX

However, really what I'm requiring to do is say:
  Max slots = N (which I can define in the mpi hostfile).

mpirun -np 1 applicationX
mpirun -np 1 applicationX (started at a random future time)
mpirun -np 1 applicationX (started at a random future time)
mpirun -np 1 applicationX (started at a random future time)

Each being automatically pinned to a CPU, but I'm fairly convinced this is
not the way MPI works.
Would it do what I'm after?

Does anyone know of a better way?

Jez
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131025/37f6a4c9/attachment-0003.htm>

From zgiles at gmail.com  Sat Oct 26 05:13:37 2013
From: zgiles at gmail.com (Zachary Giles)
Date: Sat, 26 Oct 2013 00:13:37 -0400
Subject: [gpfsug-discuss] Friday MPI fun?
In-Reply-To: <CA+M26=T7eeSMWhV_GFFvEkaKnDHNxF9OXqHbg4DvfVhnWN8CfA@mail.gmail.com>
References: <CA+M26=T7eeSMWhV_GFFvEkaKnDHNxF9OXqHbg4DvfVhnWN8CfA@mail.gmail.com>
Message-ID: <CAMYZk=esiDC4fq3DnqngyttsvvYCSBNOO2vZBsF=uJdT=fcU2Q@mail.gmail.com>

You could do it that way, if you told MPI to bind each task (MPI task, or
non-MPI task in your case) to a processor. By default, most MPI's don't
bind to a specific processor unless the scheduler (you don't have one) or
something else tells it to. OpenMPI, for example, I think, has something
like --bind-to-cpu, or --bind-to-socket. Often a scheduler or c-groups via
a scheduler will pass-in a list of cpu's to bind to, but you could do it
manually with the appropriate flags and files for your MPI distro.

You could also probably get away with just raw c-groups depending on your
distro. You would probably touch a few files and echo a few lines to files
to make the right groups then "cgrun <app>" to get things in to those
groups. That works pretty well for binding, found decent results with that,
I recommend it.


On Fri, Oct 25, 2013 at 12:09 PM, Chair GPFS UG <chair at gpfsug.org> wrote:

> Allo all,
>
>   I'm attempting to cheat.  As per-usual, cheating takes more time than
> 'Doing It Properly' - but it is vastly more fun.
>
> So without setting up Grid or Moab etc, I need to pin processes to a cpu.
>  I.E. on Linux: taskset blah blah.
> I could write a small housekeeping script which RR new spawned processes
> across CPUs using taskset, but I was wondering if OpenMPI could be a good
> way to go.
>
> So:
>
> I have a non-MPI application X.
> Which is spawned and forked by a parent process into its own process group.
> This can occur at any time, however there will only ever be a maximium N
> of appl X.
>
> Using mpirun it appears that you can set off parallel instances of a
> non-MPI application:
>
> mpirun -np 4 applicationX
>
> However, really what I'm requiring to do is say:
>   Max slots = N (which I can define in the mpi hostfile).
>
> mpirun -np 1 applicationX
> mpirun -np 1 applicationX (started at a random future time)
> mpirun -np 1 applicationX (started at a random future time)
> mpirun -np 1 applicationX (started at a random future time)
>
> Each being automatically pinned to a CPU, but I'm fairly convinced this is
> not the way MPI works.
> Would it do what I'm after?
>
> Does anyone know of a better way?
>
> Jez
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>


-- 
Zach Giles
zgiles at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131026/217d585e/attachment-0003.htm>

From richard.lefebvre at calculquebec.ca  Mon Oct 28 15:31:07 2013
From: richard.lefebvre at calculquebec.ca (Richard Lefebvre)
Date: Mon, 28 Oct 2013 11:31:07 -0400
Subject: [gpfsug-discuss] Moving/copying files from one file system to
	another
Message-ID: <526E833B.6040402@calculquebec.ca>

Hi,

I'm looking to copy/move files from one gpfs file system to another in
some efficient way. Initially I was thinking to not create another file
system and just add the nsd to the current file system and gradually
remove the old nsd. That would have been the easiest with little impact
to the users. But I wanted to switch from a scatter allocation to a
cluster allocation (too improve performance) and I cannot to that if I
just add/remove nsd. I'm hopping for a better solution then a plain
rsync which would probably take a few days (~210TB).

Is there a gpfs solution to this?

Also, another question, under what condition a scatter allocation better
then cluster allocation. We currently have a cluster of 650 nodes all
accessing the same 230TB gpfs file system.

Richard


From seanlee at tw.ibm.com  Wed Oct 30 12:30:26 2013
From: seanlee at tw.ibm.com (Sean S Lee)
Date: Wed, 30 Oct 2013 20:30:26 +0800
Subject: [gpfsug-discuss] Moving/copying files from one file system to
	another
In-Reply-To: <mailman.1.1383134402.20340.gpfsug-discuss@gpfsug.org>
References: <mailman.1.1383134402.20340.gpfsug-discuss@gpfsug.org>
Message-ID: <OF9B62D9AF.094922A5-ON48257C14.00438E14-48257C14.0044C452@tw.ibm.com>


Hi Richard,

==========

The cluster allocation method may provide better disk performance for some
disk subsystems in
relatively small installations. The benefits of clustered block allocation
diminish when the number
of nodes in the cluster or the number of disks in a file system increases,
or when the file system's
free space becomes fragmented. The cluster allocation method is the default
for GPFS clusters
with eight or fewer nodes and for file systems with eight or fewer disks.

The scatter allocation method provides more consistent file system
performance by averaging out
performance variations due to block location (for many disk subsystems, the
location of the data
relative to the disk edge has a substantial effect on performance). This
allocation method is
appropriate in most cases and is the default for GPFS clusters with more
than eight nodes or file
systems with more than eight disks.

Source: GPFS Administration and Programming Reference
==========

Based on above it seems the scatter allocation would be more suitable for
your environment (more nodes, more disks), no?

The Web forum has few threads on GPFS copying, e.g. take a look at this:
https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014934284&ps=25

I don't know if there's a better way or not.

If you don't change the allocation type, then you don't have to do anything
but add the disks and perhaps restripe.

Regards
Sean
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131030/5c77c624/attachment-0003.htm>

From jonathan at buzzard.me.uk  Wed Oct 30 12:47:07 2013
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Wed, 30 Oct 2013 12:47:07 +0000
Subject: [gpfsug-discuss] Moving/copying files from one file system to
 another
In-Reply-To: <526E833B.6040402@calculquebec.ca>
References: <526E833B.6040402@calculquebec.ca>
Message-ID: <1383137227.2921.4.camel@buzzard.phy.strath.ac.uk>

On Mon, 2013-10-28 at 11:31 -0400, Richard Lefebvre wrote:

[SNIP]

> Also, another question, under what condition a scatter allocation better
> then cluster allocation. We currently have a cluster of 650 nodes all
> accessing the same 230TB gpfs file system.
> 

Scatter allocation is better in almost all circumstances. Basically by
scattering the files to all corners you don't get hotspots where just a
small subset of the disks are being hammered by lots of accesses to a
handful of files, while the rest of the disks sit idle.

JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.



From chekh at stanford.edu  Wed Oct 30 17:56:46 2013
From: chekh at stanford.edu (Alex Chekholko)
Date: Wed, 30 Oct 2013 10:56:46 -0700
Subject: [gpfsug-discuss] Moving/copying files from one file system to
 another
In-Reply-To: <1383137227.2921.4.camel@buzzard.phy.strath.ac.uk>
References: <526E833B.6040402@calculquebec.ca>
	<1383137227.2921.4.camel@buzzard.phy.strath.ac.uk>
Message-ID: <5271485E.80405@stanford.edu>

On 10/30/13, 5:47 AM, Jonathan Buzzard wrote:
> On Mon, 2013-10-28 at 11:31 -0400, Richard Lefebvre wrote:
>
> [SNIP]
>
>> Also, another question, under what condition a scatter allocation better
>> then cluster allocation. We currently have a cluster of 650 nodes all
>> accessing the same 230TB gpfs file system.
>>
>
> Scatter allocation is better in almost all circumstances. Basically by
> scattering the files to all corners you don't get hotspots where just a
> small subset of the disks are being hammered by lots of accesses to a
> handful of files, while the rest of the disks sit idle.
>

If you do benchmarks with only a few threads, you will see higher 
performance with 'cluster' allocation.  So if your workload is only a 
few clients accessing the FS in a mostly streaming way, you'd see better 
performance from 'cluster'.

With 650 nodes, even if each client is doing streaming reads, at the 
filesystem level that would all be interleaved and thus be random reads. 
  But it's tough to do a big enough benchmark to show the difference in 
performance.

I had a tough time convincing people to use 'scatter' instead of 
'cluster' even though I think the documentation is clear about the 
difference, and even gives you the sizing parameters ( greater than 8 
disks or 8 NSDs?  use 'scatter').

We use 'scatter' now.

Regards
-- 
chekh at stanford.edu


From TOMP at il.ibm.com  Wed Oct 30 18:31:01 2013
From: TOMP at il.ibm.com (Tomer Perry)
Date: Wed, 30 Oct 2013 20:31:01 +0200
Subject: [gpfsug-discuss] Moving/copying files from one file system to
	another
In-Reply-To: <5271485E.80405@stanford.edu>
References: <526E833B.6040402@calculquebec.ca>	<1383137227.2921.4.camel@buzzard.phy.strath.ac.uk>
	<5271485E.80405@stanford.edu>
Message-ID: <OF39DC62FE.1C8ABCF9-ONC2257C14.00657D38-C2257C14.0065BC4C@il.ibm.com>

Hi,

cluster will also be there for some time, as soon as one start deleting 
and creating file - it will kind of shift to be scattered...
So, while its good for benchmarks on "clean and new" system - cluster is 
somewhat meaningless ( and was hidden for some time because of that).

As far as it goes for copying from old to new system - AFM can be 
considered as well ( will make the transition easier). You can prefetch ( 
using mmafmctl) and even work in LU mode in order to get the data from the 
old FS to the new one without pushing changes back.

hth,
Tomer.




From:   Alex Chekholko <chekh at stanford.edu>
To:     gpfsug-discuss at gpfsug.org, 
Date:   10/30/2013 07:57 PM
Subject:        Re: [gpfsug-discuss] Moving/copying files from one file 
system to another
Sent by:        gpfsug-discuss-bounces at gpfsug.org



On 10/30/13, 5:47 AM, Jonathan Buzzard wrote:
> On Mon, 2013-10-28 at 11:31 -0400, Richard Lefebvre wrote:
>
> [SNIP]
>
>> Also, another question, under what condition a scatter allocation 
better
>> then cluster allocation. We currently have a cluster of 650 nodes all
>> accessing the same 230TB gpfs file system.
>>
>
> Scatter allocation is better in almost all circumstances. Basically by
> scattering the files to all corners you don't get hotspots where just a
> small subset of the disks are being hammered by lots of accesses to a
> handful of files, while the rest of the disks sit idle.
>

If you do benchmarks with only a few threads, you will see higher 
performance with 'cluster' allocation.  So if your workload is only a 
few clients accessing the FS in a mostly streaming way, you'd see better 
performance from 'cluster'.

With 650 nodes, even if each client is doing streaming reads, at the 
filesystem level that would all be interleaved and thus be random reads. 
  But it's tough to do a big enough benchmark to show the difference in 
performance.

I had a tough time convincing people to use 'scatter' instead of 
'cluster' even though I think the documentation is clear about the 
difference, and even gives you the sizing parameters ( greater than 8 
disks or 8 NSDs?  use 'scatter').

We use 'scatter' now.

Regards
-- 
chekh at stanford.edu
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131030/d130f587/attachment-0003.htm>