From TROPPENS at de.ibm.com  Tue May  3 13:05:51 2016
From: TROPPENS at de.ibm.com (Ulf Troppens)
Date: Tue, 3 May 2016 14:05:51 +0200
Subject: [gpfsug-discuss] GPFS/Spectrum Scale Upcoming US Events - Save
 the Dates - Registration for New York May 26
In-Reply-To: <9489DBA2-1F12-4B05-A968-5D4855FBEA40@gpfsug.org>
References: <4192D644-E3AB-4B7B-AF7A-96D3D617FA7B@gpfsug.org>
	<9489DBA2-1F12-4B05-A968-5D4855FBEA40@gpfsug.org>
Message-ID: <201605031206.u43C5wss002725@d06av09.portsmouth.uk.ibm.com>

Registration for the first event in New York is up as well - the agenda
still needs some touch.

Please register here and select 'May 26 GPFS Day Registration', if you
don't attend SPXXL:
https://www.spxxl.org/?q=New-York-City-2016
https://www.eventbrite.com/e/spxxlscicomp-2016-summer-meeting-registration-24444020724


--
IBM Spectrum Scale Development - Client Engagements & Solutions Delivery
Consulting IT Specialist
Author "Storage Networks Explained"

IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Gesch?ftsf?hrung: Dirk Wittkopp
Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart,
HRB 243294




From:	GPFS UG USA Principal <usa-principal at gpfsug.org>
To:	gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Date:	28.04.2016 23:44
Subject:	Re: [gpfsug-discuss] GPFS/Spectrum Scale Upcoming US Events -
            Save	the Dates
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



All, the registration page for the second event listed below at Argonne
National Lab on June 10th is now up. An updated agenda is also at this
site.

Please register here: https://www.regonline.com/Spectrumscalemeeting

We look forward to seeing some of you at these upcoming events.

Feel free to send suggestions for future events in your area.

Cheers,
-Kristy


      On Apr 4, 2016, at 4:52 PM, GPFS UG USA Principal <
      usa-principal at gpfsug.org> wrote:

      Hello all,

      We?d like to announce two upcoming US GPFS/Spectrum Scale Events. One
      on the east coast, one in the midwest.

      1) May 26th (full day event): GPFS/Spectrum Scale Day at the SPXXL
      conference in NYC https://www.spxxl.org/?q=New-York-City-2016
      Developers and Engineers from IBM will be at the meeting to cover
      topics, and open dialogue between IBM and customers will be
      encouraged. More details about the agenda, hotel and other logistics
      will come later this month.

      Tentative Agenda:
      ? 10 reasons for upgrading from GPFS 3.5 to Spectrum Scale 4.2.1
      ? Enhancements for CORAL from IBM
      ? Panel discussion with customers, topic TBD
      ? AFM and integration with Spectrum Protect
      ? Best practices for GPFS or Spectrum Scale Tuning.
      ? At least one site update

      Location:
      New York Academy of Medicine
      1216 Fifth Avenue
      New York, NY 10029

      ??

      2) June 10th (full day event): GPFS/Spectrum Scale Users Group
      Meeting at Argonne National Lab (ANL). Thanks to Argonne for hosting
      this event. Developers and Engineers from IBM will be at the meeting
      to cover topics, and open dialogue between IBM and customers will be
      encouraged, as usual no marketing pitches!

      Location: Argonne National Lab more details and final agenda will
      come later.

      Tentative Agenda:

      9:00a-12:30p
      9-9:30a - Opening Remarks
      9:30-10a Deep Dive - Update on ESS
      10a-11a Deep Dive - Problem Determination (Presentation 30 min/Panel
      30 min?)
      11-11:30 Break
      11:30a-Noon - Deep Dive - Protect & Scale integration
      Noon-12:30p HDFS/Hadoop

      12:30 - 1:30p Lunch

      1:30p-5:00p
      1:30 - 2:00p IBM AFM Update
      2:00-2:30p ANL: AFM as a burst buffer
      2:30-3:00p ANL: GHI (GPFS HPSS Integration)
      3:00-3:30p Break
      3:30p - 4:00p LANL: ? or other site preso
      4:00-4:30p Nuance: GPFS Performance Sensors Deployment Experiences
      4:30p -5:00p Closing comments and Open Forum for Questions

      5:00 - ?
      Beer hunting?

      ??

      We hope you can attend one or both of these events.

      Best,
      Kristy Kallback-Rose & Bob Oesterlin
      GPFS Users Group - USA Chapter - Principal & Co-principal
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160503/c00a2018/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160503/c00a2018/attachment.gif>

From A.K.Ghumra at bham.ac.uk  Tue May  3 14:45:23 2016
From: A.K.Ghumra at bham.ac.uk (Aslam Ghumra (IT Services, Facilities Management))
Date: Tue, 3 May 2016 13:45:23 +0000
Subject: [gpfsug-discuss] SMB access speed
Message-ID: <D50747684476FC4EAA59F1BE20FD0557D7AD34BC@EX12.adf.bham.ac.uk>

Hi Yaron, 

Local hdd to hdd copy speed is approx. 323.15MB/s, not sure if its's a different hdd or different partition, hopefully will get that information later.

Yup, gui is not an option and hopefully the researcher will use command line.

Regards, Aslam
Research Computing Team
DDI: +44 (121) 414 5877 | Skype: JanitorX | Twitter: @aslamghumra | a.k.ghumra at bham.ac.uk | intranet.birmingham.ac.uk/bear


-----Original Message-----
From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of gpfsug-discuss-request at spectrumscale.org
Sent: 30 April 2016 12:00

Today's Topics:

   1. Re: SMB access speed (Yaron Daniel)


----------------------------------------------------------------------

Message: 1
Date: Sat, 30 Apr 2016 08:17:28 +0300
From: "Yaron Daniel" <YARD at il.ibm.com>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] SMB access speed
Message-ID:
	<201604300517.u3U5HcbY022432 at d06av12.portsmouth.uk.ibm.com>
Content-Type: text/plain; charset="iso-8859-1"

Hi

It could be that GUI use in the "background" default command which use smb v1.

Regard copy files from GPFS to Local HDD, it might be related to the local HDD settings.
What is the speed transfer between the local HHD ? Cache Settings and so..

 
Regards
 


 
 
Yaron Daniel
 94 Em Ha'Moshavot Rd

Server, Storage and Data Services - Team Leader  Petach Tiqva, 49527 Global Technology Services  Israel
Phone:
+972-3-916-5672
 
 
Fax:
+972-3-916-5672
 
 
Mobile:
+972-52-8395593
 
 
e-mail:
yard at il.ibm.com
 
 
IBM Israel
 
 
 
 

 



From:   "Aslam Ghumra (IT Services, Facilities Management)" 
<A.K.Ghumra at bham.ac.uk>
To:     "gpfsug-discuss at spectrumscale.org" 
<gpfsug-discuss at spectrumscale.org>
Date:   04/29/2016 07:07 PM
Subject:        [gpfsug-discuss] SMB access speed
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Many thanks Yaron, after the change to disable encryption we were able to 
increase the speed via Ubuntu of copying files from the local desktop to 
our gpfs filestore with average speeds of 60Mbps.
 
We also tried changing the mount from vers=3.0 to vers=2.1, which gave 
similar figures
 
However, using the Ubuntu gui ( Unity ) the speed drops down to 7Mbps, 
however, we?re not concerned as the user will use rsync / cp. 
 
The other issue is copying data from gpfs filestore to the local HDD, 
which resulted in 4Mbps.
 
Aslam Ghumra
Research Data Management
____________________________
IT Services
Elms Road Data Centre
Building G5
Edgbaston
Birmingham B15 2TT
T: 0121 414 5877
F; 0121 414 3952
Skype : JanitorX
Twitter : @aslamghumra
http://intranet.bham.ac.uk/bear
 _______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160430/875e5aa0/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 1851 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160430/875e5aa0/attachment-0001.gif>

------------------------------

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


End of gpfsug-discuss Digest, Vol 51, Issue 57
**********************************************


From Kevin.Buterbaugh at Vanderbilt.Edu  Tue May  3 21:20:15 2016
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Tue, 3 May 2016 20:20:15 +0000
Subject: [gpfsug-discuss] CNFS and multiple IP addresses
Message-ID: <94CBCF33-0B97-440E-8E51-F4A51E9333FB@vanderbilt.edu>

Hi All,

I?ve looked in the documentation and on the GPFS Wiki for the answer to this question but haven?t found it.  Please feel free to point me to docs / web pages if I?ve missed something.

One of the GPFS clusters I administer uses CNFS, which I did not set up on this cluster ? and the person who did is now an employee of another organization.  The cluster in question runs GPFS 4.1.0.x, although I doubt that matters.

There are 4 CNFS servers, each of which has 2 IP addresses listed under the CNFS IP address list when running ?mmlscluster ?cnfs?.  I want to delete one of those two IP addresses for each server due to a planned network reorganization.  How do I do that?  Or do I need to completely delete the server from CNFS and add it back in with just the one IP address I want to keep?

Thanks in advance?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160503/76735cf0/attachment.htm>

From Kevin.Buterbaugh at Vanderbilt.Edu  Tue May  3 21:27:12 2016
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Tue, 3 May 2016 20:27:12 +0000
Subject: [gpfsug-discuss] Data replication and fileset quotas
Message-ID: <AEE0A7CC-E792-462F-9BD0-C6DF9C551E41@vanderbilt.edu>

Hi again all,

I have another question on a completely different topic and therefore I decided it was better to send two separate e-mails.

For a number of years now we have had a GPFS filesystem where we use filesets and set fileset quotas.  Data replication is set to one.  I understand how that all works.

We are creating another GPFS filesystem where we intend to also use filesets and fileset quotas, but set data replication to two.  Based on my experience with data replication on a filesystem that doesn?t use filesets, I am expecting that setting data replication to two means that I will need to double the quota for each fileset (i.e. if a group has bought 5 TB of space I?ll need to set their fileset quota to 10 TB) but haven?t found where that is explicitly documented.  Is that correct?

Thanks again, all?

Kevin

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160503/c1274617/attachment.htm>

From bbanister at jumptrading.com  Tue May  3 21:31:53 2016
From: bbanister at jumptrading.com (Bryan Banister)
Date: Tue, 3 May 2016 20:31:53 +0000
Subject: [gpfsug-discuss] CNFS and multiple IP addresses
In-Reply-To: <94CBCF33-0B97-440E-8E51-F4A51E9333FB@vanderbilt.edu>
References: <94CBCF33-0B97-440E-8E51-F4A51E9333FB@vanderbilt.edu>
Message-ID: <21BC488F0AEA2245B2C3E83FC0B33DBB060CA609@CHI-EXCHANGEW1.w2k.jumptrading.com>

Hi Kevin,

Look at the man page for mmchnode, you?ll need to use the ?--cnfs-interface=<ip_address_list>? option.

BTW, I get the cleanest looking man pages for GPFS commands by using the ?-E ascii? option.

You may also have a Round-Robin DNS record setup that includes the current IP addresses that you will likely want to update, not forgetting about DNS caching timeouts that you may need to let expire before no client attempts to mount from an old address.

Hope that helps!
-Bryan

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Buterbaugh, Kevin L
Sent: Tuesday, May 03, 2016 3:20 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] CNFS and multiple IP addresses

Hi All,

I?ve looked in the documentation and on the GPFS Wiki for the answer to this question but haven?t found it.  Please feel free to point me to docs / web pages if I?ve missed something.

One of the GPFS clusters I administer uses CNFS, which I did not set up on this cluster ? and the person who did is now an employee of another organization.  The cluster in question runs GPFS 4.1.0.x, although I doubt that matters.

There are 4 CNFS servers, each of which has 2 IP addresses listed under the CNFS IP address list when running ?mmlscluster ?cnfs?.  I want to delete one of those two IP addresses for each server due to a planned network reorganization.  How do I do that?  Or do I need to completely delete the server from CNFS and add it back in with just the one IP address I want to keep?

Thanks in advance?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633




________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160503/5ffdbf58/attachment.htm>

From bbanister at jumptrading.com  Tue May  3 21:33:24 2016
From: bbanister at jumptrading.com (Bryan Banister)
Date: Tue, 3 May 2016 20:33:24 +0000
Subject: [gpfsug-discuss] Data replication and fileset quotas
In-Reply-To: <AEE0A7CC-E792-462F-9BD0-C6DF9C551E41@vanderbilt.edu>
References: <AEE0A7CC-E792-462F-9BD0-C6DF9C551E41@vanderbilt.edu>
Message-ID: <21BC488F0AEA2245B2C3E83FC0B33DBB060CA62F@CHI-EXCHANGEW1.w2k.jumptrading.com>

Yes, I believe that is the case due to the fact that every file is replicated therefore taking twice as much space within the file system.
-Bryan

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Buterbaugh, Kevin L
Sent: Tuesday, May 03, 2016 3:27 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] Data replication and fileset quotas

Hi again all,

I have another question on a completely different topic and therefore I decided it was better to send two separate e-mails.

For a number of years now we have had a GPFS filesystem where we use filesets and set fileset quotas.  Data replication is set to one.  I understand how that all works.

We are creating another GPFS filesystem where we intend to also use filesets and fileset quotas, but set data replication to two.  Based on my experience with data replication on a filesystem that doesn?t use filesets, I am expecting that setting data replication to two means that I will need to double the quota for each fileset (i.e. if a group has bought 5 TB of space I?ll need to set their fileset quota to 10 TB) but haven?t found where that is explicitly documented.  Is that correct?

Thanks again, all?

Kevin

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633




________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160503/67d1cbad/attachment.htm>

From Kevin.Buterbaugh at Vanderbilt.Edu  Tue May  3 21:51:48 2016
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Tue, 3 May 2016 20:51:48 +0000
Subject: [gpfsug-discuss] CNFS and multiple IP addresses
In-Reply-To: <21BC488F0AEA2245B2C3E83FC0B33DBB060CA609@CHI-EXCHANGEW1.w2k.jumptrading.com>
References: <94CBCF33-0B97-440E-8E51-F4A51E9333FB@vanderbilt.edu>
	<21BC488F0AEA2245B2C3E83FC0B33DBB060CA609@CHI-EXCHANGEW1.w2k.jumptrading.com>
Message-ID: <FBB84DD9-9B12-4607-9D98-E7DD87C7C145@vanderbilt.edu>

Hi Bryan,

Thanks for the response.

I had looked at the man page for mmchnode previously, but was misinterpreting the wording.  I was under the impression that the ??cnfs-interface? option could only be used to add IP addresses or delete a node from CNFS with the ?DEFAULT? keyword.

Your explanation makes sense - thanks for clarifying?

Kevin

On May 3, 2016, at 3:31 PM, Bryan Banister <bbanister at jumptrading.com<mailto:bbanister at jumptrading.com>> wrote:

Hi Kevin,

Look at the man page for mmchnode, you?ll need to use the ?--cnfs-interface=<ip_address_list>? option.

BTW, I get the cleanest looking man pages for GPFS commands by using the ?-E ascii? option.

You may also have a Round-Robin DNS record setup that includes the current IP addresses that you will likely want to update, not forgetting about DNS caching timeouts that you may need to let expire before no client attempts to mount from an old address.

Hope that helps!
-Bryan

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Buterbaugh, Kevin L
Sent: Tuesday, May 03, 2016 3:20 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: [gpfsug-discuss] CNFS and multiple IP addresses

Hi All,

I?ve looked in the documentation and on the GPFS Wiki for the answer to this question but haven?t found it.  Please feel free to point me to docs / web pages if I?ve missed something.

One of the GPFS clusters I administer uses CNFS, which I did not set up on this cluster ? and the person who did is now an employee of another organization.  The cluster in question runs GPFS 4.1.0.x, although I doubt that matters.

There are 4 CNFS servers, each of which has 2 IP addresses listed under the CNFS IP address list when running ?mmlscluster ?cnfs?.  I want to delete one of those two IP addresses for each server due to a planned network reorganization.  How do I do that?  Or do I need to completely delete the server from CNFS and add it back in with just the one IP address I want to keep?

Thanks in advance?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633




________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160503/4af7fe04/attachment.htm>

From S.J.Thompson at bham.ac.uk  Wed May  4 08:11:18 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Wed, 4 May 2016 07:11:18 +0000
Subject: [gpfsug-discuss] Data replication and fileset quotas
In-Reply-To: <21BC488F0AEA2245B2C3E83FC0B33DBB060CA62F@CHI-EXCHANGEW1.w2k.jumptrading.com>
References: <21BC488F0AEA2245B2C3E83FC0B33DBB060CA62F@CHI-EXCHANGEW1.w2k.jumptrading.com>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F576E8B6@EX13.adf.bham.ac.uk>




Yes, this correct (I think there was some discussion on the mailing list a little while back).

The quota is the space used on disk, so if you run compression its the compressed size, tape hsm don't count until you recall them.

I assume mcstore is the same, I.e. Only counts on recall, but I haven't tested this.

Simon

--

Simon Thompson

Research Computing Team, IT Services

+44 121 415 8675

-----Original Message-----
From: Bryan Banister [bbanister at jumptrading.com<mailto:bbanister at jumptrading.com>]
Sent: Tuesday, May 03, 2016 09:33 PM GMT Standard Time
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Data replication and fileset quotas

Yes, I believe that is the case due to the fact that every file is replicated therefore taking twice as much space within the file system.
-Bryan

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Buterbaugh, Kevin L
Sent: Tuesday, May 03, 2016 3:27 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] Data replication and fileset quotas

Hi again all,

I have another question on a completely different topic and therefore I decided it was better to send two separate e-mails.

For a number of years now we have had a GPFS filesystem where we use filesets and set fileset quotas.  Data replication is set to one.  I understand how that all works.

We are creating another GPFS filesystem where we intend to also use filesets and fileset quotas, but set data replication to two.  Based on my experience with data replication on a filesystem that doesn?t use filesets, I am expecting that setting data replication to two means that I will need to double the quota for each fileset (i.e. if a group has bought 5 TB of space I?ll need to set their fileset quota to 10 TB) but haven?t found where that is explicitly documented.  Is that correct?

Thanks again, all?

Kevin

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633




________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160504/34c11a8d/attachment.htm>

From Robert.Oesterlin at nuance.com  Wed May  4 12:18:54 2016
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Wed, 4 May 2016 11:18:54 +0000
Subject: [gpfsug-discuss] Data replication and fileset quotas
Message-ID: <1C31AA0D-666C-4CB7-A54D-99CFC5AC8FAE@nuance.com>

From the documentation:

"When setting quota limits for a file system, replication within the file system should be considered. GPFS quota management takes replication into account when reporting on and determining if quota limits have been exceeded for both block and file usage. In a file system that has either type of replication set to a value of two, the values reported on by both the mmlsquota command and the mmrepquota command are double the value reported by the ls command."

Bob Oesterlin
Sr Storage Engineer, Nuance HPC Grid
507-269-0413


From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of "Simon Thompson (Research Computing - IT Services)" <S.J.Thompson at bham.ac.uk<mailto:S.J.Thompson at bham.ac.uk>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, May 4, 2016 at 2:11 AM
To: 'gpfsug main discussion list' <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Data replication and fileset quotas




Yes, this correct (I think there was some discussion on the mailing list a little while back).

The quota is the space used on disk, so if you run compression its the compressed size, tape hsm don't count until you recall them.

I assume mcstore is the same, I.e. Only counts on recall, but I haven't tested this.

Simon

--

Simon Thompson

Research Computing Team, IT Services

+44 121 415 8675

-----Original Message-----
From: Bryan Banister [bbanister at jumptrading.com<mailto:bbanister at jumptrading.com>]
Sent: Tuesday, May 03, 2016 09:33 PM GMT Standard Time
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Data replication and fileset quotas

Yes, I believe that is the case due to the fact that every file is replicated therefore taking twice as much space within the file system.
-Bryan

From:gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Buterbaugh, Kevin L
Sent: Tuesday, May 03, 2016 3:27 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: [gpfsug-discuss] Data replication and fileset quotas

Hi again all,

I have another question on a completely different topic and therefore I decided it was better to send two separate e-mails.

For a number of years now we have had a GPFS filesystem where we use filesets and set fileset quotas.  Data replication is set to one.  I understand how that all works.

We are creating another GPFS filesystem where we intend to also use filesets and fileset quotas, but set data replication to two.  Based on my experience with data replication on a filesystem that doesn?t use filesets, I am expecting that setting data replication to two means that I will need to double the quota for each fileset (i.e. if a group has bought 5 TB of space I?ll need to set their fileset quota to 10 TB) but haven?t found where that is explicitly documented.  Is that correct?

Thanks again, all?

Kevin

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633




________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160504/99bdbc87/attachment.htm>

From daniel.kidger at uk.ibm.com  Wed May  4 12:36:06 2016
From: daniel.kidger at uk.ibm.com (Daniel Kidger)
Date: Wed, 4 May 2016 11:36:06 +0000
Subject: [gpfsug-discuss] Data replication and fileset quotas
In-Reply-To: <1C31AA0D-666C-4CB7-A54D-99CFC5AC8FAE@nuance.com>
References: <1C31AA0D-666C-4CB7-A54D-99CFC5AC8FAE@nuance.com>
Message-ID: <201605041136.u44BaBZT029213@d06av06.portsmouth.uk.ibm.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160504/597bfe75/attachment.htm>

From D.R.Barker at exeter.ac.uk  Wed May  4 13:38:23 2016
From: D.R.Barker at exeter.ac.uk (Barker, David)
Date: Wed, 4 May 2016 12:38:23 +0000
Subject: [gpfsug-discuss] Data replication and fileset quotas
In-Reply-To: <201605041136.u44BaBZT029213@d06av06.portsmouth.uk.ibm.com>
References: <1C31AA0D-666C-4CB7-A54D-99CFC5AC8FAE@nuance.com>
	<201605041136.u44BaBZT029213@d06av06.portsmouth.uk.ibm.com>
Message-ID: <DB5PR03MB1366BB56598980F182E8B18CE67B0@DB5PR03MB1366.eurprd03.prod.outlook.com>

Hi Daniel,

It?s funny you mention compression & quotas ? it?s something I?ve hit on a.n.other system. The supplier ended up providing different ways of presenting quotas as a mount option. Options are:


-          Quotas are tracked using the post compression/dedupe actually used space. (i.e. users see the benefit/drawback of compression)

-          Quotas are tracked using the pre compression/dedupe filesize. (i.e. the sysadmin sees the benefit; users are unaware what?s happening underneath).

Would something similar be possible in GPFS land?

David

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Daniel Kidger
Sent: 04 May 2016 12:36
To: gpfsug-discuss at spectrumscale.org
Cc: gpfsug-discuss at spectrumscale.org
Subject: Re: [gpfsug-discuss] Data replication and fileset quotas

As Simon says, Quota measures used blocks in the filesystem.
Hence users can and should have behaviour that keeps within these limits.

GPFS Replication though is a system-administrator level concept - to protect data access in the case of power outages or though gross hardware failures. So as such should be transparent to the end users.

Unless users are enabled to choose 1 or 2 (or 3) way replication of their own files dependent on their importance (eg 1 copy for scratch files) then imho replication should not be measured in quota reporting.

On a related note, compression is great new feature, but it may confuse users if they delete some older but big 100GB files then try and recreate them only to find they can't because their quota is now exceeded (as compression is not at file creation but driven later by policies.

Thoughts?
Daniel






Dr Daniel Kidger
IBM Technical Sales Specialist
Software Defined Solution Sales

+44-07818 522 266
daniel.kidger at uk.ibm.com<mailto:daniel.kidger at uk.ibm.com>







----- Original message -----
From: "Oesterlin, Robert" <Robert.Oesterlin at nuance.com<mailto:Robert.Oesterlin at nuance.com>>
Sent by: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc:
Subject: Re: [gpfsug-discuss] Data replication and fileset quotas
Date: Wed, May 4, 2016 12:19 PM

From the documentation:

"When setting quota limits for a file system, replication within the file system should be considered. GPFS quota management takes replication into account when reporting on and determining if quota limits have been exceeded for both block and file usage. In a file system that has either type of replication set to a value of two, the values reported on by both the mmlsquota command and the mmrepquota command are double the value reported by the ls command."

Bob Oesterlin
Sr Storage Engineer, Nuance HPC Grid
507-269-0413


From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of "Simon Thompson (Research Computing - IT Services)" <S.J.Thompson at bham.ac.uk<mailto:S.J.Thompson at bham.ac.uk>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, May 4, 2016 at 2:11 AM
To: 'gpfsug main discussion list' <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Data replication and fileset quotas




Yes, this correct (I think there was some discussion on the mailing list a little while back).

The quota is the space used on disk, so if you run compression its the compressed size, tape hsm don't count until you recall them.

I assume mcstore is the same, I.e. Only counts on recall, but I haven't tested this.

Simon

--

Simon Thompson

Research Computing Team, IT Services

+44 121 415 8675

-----Original Message-----
From: Bryan Banister [bbanister at jumptrading.com<mailto:bbanister at jumptrading.com>]
Sent: Tuesday, May 03, 2016 09:33 PM GMT Standard Time
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Data replication and fileset quotas

Yes, I believe that is the case due to the fact that every file is replicated therefore taking twice as much space within the file system.

-Bryan



From:gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Buterbaugh, Kevin L
Sent: Tuesday, May 03, 2016 3:27 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: [gpfsug-discuss] Data replication and fileset quotas



Hi again all,



I have another question on a completely different topic and therefore I decided it was better to send two separate e-mails.



For a number of years now we have had a GPFS filesystem where we use filesets and set fileset quotas.  Data replication is set to one.  I understand how that all works.



We are creating another GPFS filesystem where we intend to also use filesets and fileset quotas, but set data replication to two.  Based on my experience with data replication on a filesystem that doesn?t use filesets, I am expecting that setting data replication to two means that I will need to double the quota for each fileset (i.e. if a group has bought 5 TB of space I?ll need to set their fileset quota to 10 TB) but haven?t found where that is explicitly documented.  Is that correct?



Thanks again, all?



Kevin



?

Kevin Buterbaugh - Senior System Administrator

Vanderbilt University - Advanced Computing Center for Research and Education

Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633







________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160504/46e3773a/attachment.htm>

From r.sobey at imperial.ac.uk  Wed May  4 16:32:32 2016
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Wed, 4 May 2016 15:32:32 +0000
Subject: [gpfsug-discuss] TSM and re-linked filesets
Message-ID: <AMSPR06MB4053E885E19DC5F12F46F74DF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>

This is more a TSM question than GPFS but it involves GPFS to a point so I hope I'm ok asking this.

I've unlinked a fileset and relinked it at a different junction point. Will this cause TSM to see a "new" fileset and back up all its content? This is very undesireable so if I need to put it back where it belongs then I need to do this before tonight.

GPFS 3.5 and TSM 7.1.

Does anyone know?

Cheers

Richard

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160504/7399359a/attachment.htm>

From r.sobey at imperial.ac.uk  Wed May  4 16:35:29 2016
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Wed, 4 May 2016 15:35:29 +0000
Subject: [gpfsug-discuss] TSM and re-linked filesets
In-Reply-To: <AMSPR06MB4053E885E19DC5F12F46F74DF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
References: <AMSPR06MB4053E885E19DC5F12F46F74DF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
Message-ID: <AMSPR06MB4053EB49583DA7B3623505EDF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>

A quick Google has given me the answer I dreaded:

"Attention: If you are using the TSM Backup-Archive client you must use caution when you unlink filesets that contain data backed up by TSM. TSM tracks files by pathname and does not track filesets. As a result, when you unlink a fileset, it appears to TSM that you deleted the contents of the fileset. Therefore, the TSM Backup-Archive client inactivates the data on the TSM server which may result in the loss of backup data during the expiration process."

So I need to relink it to the original place I think. Oh well. Not a disaster.

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A
Sent: 04 May 2016 16:33
To: 'gpfsug-discuss at spectrumscale.org' <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] TSM and re-linked filesets

This is more a TSM question than GPFS but it involves GPFS to a point so I hope I'm ok asking this.

I've unlinked a fileset and relinked it at a different junction point. Will this cause TSM to see a "new" fileset and back up all its content? This is very undesireable so if I need to put it back where it belongs then I need to do this before tonight.

GPFS 3.5 and TSM 7.1.

Does anyone know?

Cheers

Richard

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160504/f6c35dd3/attachment.htm>

From S.J.Thompson at bham.ac.uk  Wed May  4 16:59:27 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Wed, 4 May 2016 15:59:27 +0000
Subject: [gpfsug-discuss] TSM and re-linked filesets
In-Reply-To: <AMSPR06MB4053EB49583DA7B3623505EDF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
References: <AMSPR06MB4053EB49583DA7B3623505EDF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F576F2A9@EX13.adf.bham.ac.uk>

Could you do something really nasty in the tsm db2 database to update the paths?

-----Original Message-----
From: Sobey, Richard A [r.sobey at imperial.ac.uk<mailto:r.sobey at imperial.ac.uk>]
Sent: Wednesday, May 04, 2016 04:35 PM GMT Standard Time
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] TSM and re-linked filesets

A quick Google has given me the answer I dreaded:

?Attention: If you are using the TSM Backup-Archive client you must use caution when you unlink filesets that contain data backed up by TSM. TSM tracks files by pathname and does not track filesets. As a result, when you unlink a fileset, it appears to TSM that you deleted the contents of the fileset. Therefore, the TSM Backup-Archive client inactivates the data on the TSM server which may result in the loss of backup data during the expiration process.?

So I need to relink it to the original place I think. Oh well. Not a disaster.

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A
Sent: 04 May 2016 16:33
To: 'gpfsug-discuss at spectrumscale.org' <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] TSM and re-linked filesets

This is more a TSM question than GPFS but it involves GPFS to a point so I hope I?m ok asking this.

I?ve unlinked a fileset and relinked it at a different junction point. Will this cause TSM to see a ?new? fileset and back up all its content? This is very undesireable so if I need to put it back where it belongs then I need to do this before tonight.

GPFS 3.5 and TSM 7.1.

Does anyone know?

Cheers

Richard

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160504/7874b433/attachment.htm>

From makaplan at us.ibm.com  Wed May  4 17:14:44 2016
From: makaplan at us.ibm.com (Marc A Kaplan)
Date: Wed, 4 May 2016 12:14:44 -0400
Subject: [gpfsug-discuss] TSM and re-linked filesets
In-Reply-To: <AMSPR06MB4053EB49583DA7B3623505EDF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
References: <AMSPR06MB4053E885E19DC5F12F46F74DF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
	<AMSPR06MB4053EB49583DA7B3623505EDF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
Message-ID: <201605041614.u44GEnbI031747@d03av03.boulder.ibm.com>

I think you found your answer:  TSM tracks files by pathname. 

So... if a file had path /w/x/y/z on Monday.  But was moved to /w/x/q/p on 
Tuesday, how would TSM "know" it was the same file...?
It wouldn't! To TSM it seems you've deleted the first and created the 
second.

Technically there are some other possibilities, and some backup systems 
may use them, but NOT TSM:

1) Record the inode number and generation number and/or creation 
timestamp.   Within a given Posix-ish file system, that uniquely 
identifies the file.

2) Record a strong (cryptographic quality) checksum (hash) of the contents 
of the file.  If two files have the same checksum (hash) then the odds are
we can use the same backup data for both and don't have to make an extra 
copy in the backup system.    To make the odds really, really "long" you 
want to
take into account the "birthday paradox" and use lots and lots of bits. 
Long odds can also be compared to the probability of losing a file due to 
a bug
or an IO error or accident or disaster...

For example SHA-256, might be strong and long enough for you to believe 
in.
Backup is not generally a cryptographic game, so perhaps you should not 
worry much about some evil doer purposely trying to confound your backup 
system.
OTOH - if you have users who are adversaries, all backing up into the same 
system... In theory one might "destroy" another's backup.

This save transmission and storage of duplicates, but of course the backup 
system has to read the contents of each suspected new file and compute the 
hash...



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160504/b5aca332/attachment.htm>

From laurence at qsplace.co.uk  Wed May  4 18:49:31 2016
From: laurence at qsplace.co.uk (Laurence Horrors-Barlow)
Date: Wed, 4 May 2016 18:49:31 +0100
Subject: [gpfsug-discuss] TSM and re-linked filesets
In-Reply-To: <AMSPR06MB4053EB49583DA7B3623505EDF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
References: <AMSPR06MB4053E885E19DC5F12F46F74DF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
	<AMSPR06MB4053EB49583DA7B3623505EDF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
Message-ID: <7DCCBF95-92CF-4473-8BAB-F845C5408A46@qsplace.co.uk>

Hi Richard,

That's correct, TSM will only look at the path and compare files it doesn't understand the file set. Once the path has changed it will then start expiring the missing files.

What are you trying to achieve here as you might be able to update the expiration or fudge the DB.

-- Lauz

Sent from my iPad

> On 4 May 2016, at 16:35, Sobey, Richard A <r.sobey at imperial.ac.uk> wrote:
> 
> A quick Google has given me the answer I dreaded:
>  
> ?Attention: If you are using the TSM Backup-Archive client you must use caution when you unlink filesets that contain data backed up by TSM. TSM tracks files by pathname and does not track filesets. As a result, when you unlink a fileset, it appears to TSM that you deleted the contents of the fileset. Therefore, the TSM Backup-Archive client inactivates the data on the TSM server which may result in the loss of backup data during the expiration process.?
>  
> So I need to relink it to the original place I think. Oh well. Not a disaster.
>  
> From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A
> Sent: 04 May 2016 16:33
> To: 'gpfsug-discuss at spectrumscale.org' <gpfsug-discuss at spectrumscale.org>
> Subject: [gpfsug-discuss] TSM and re-linked filesets
>  
> This is more a TSM question than GPFS but it involves GPFS to a point so I hope I?m ok asking this.
>  
> I?ve unlinked a fileset and relinked it at a different junction point. Will this cause TSM to see a ?new? fileset and back up all its content? This is very undesireable so if I need to put it back where it belongs then I need to do this before tonight.
>  
> GPFS 3.5 and TSM 7.1.
>  
> Does anyone know?
>  
> Cheers
>  
> Richard
>  
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160504/4e625d03/attachment.htm>

From service at metamodul.com  Wed May  4 19:15:29 2016
From: service at metamodul.com (service at metamodul.com)
Date: Wed, 04 May 2016 20:15:29 +0200
Subject: [gpfsug-discuss] TSM and re-linked filesets
Message-ID: <tl3ri9d2ub9oacdlv1vloej2.1462385341788@email.android.com>

Have thought about the use of a submount ?
Meaning you link your fileset to the new directory and mount that dir on the old dir or you do not unlink at all but submount the old directory at the new directory.


Von Samsung Mobile gesendet

<div>-------- Urspr?ngliche Nachricht --------</div><div>Von: "Sobey, Richard A" <r.sobey at imperial.ac.uk> </div><div>Datum:2016.05.04  17:32  (GMT+01:00) </div><div>An: "'gpfsug-discuss at spectrumscale.org'" <gpfsug-discuss at spectrumscale.org> </div><div>Betreff: [gpfsug-discuss] TSM and re-linked filesets </div><div>
</div>This is more a TSM question than GPFS but it involves GPFS to a point so I hope I?m ok asking this.
 
I?ve unlinked a fileset and relinked it at a different junction point. Will this cause TSM to see a ?new? fileset and back up all its content? This is very undesireable so if I need to put it back where it belongs then I need to do this before tonight.
 
GPFS 3.5 and TSM 7.1.
 
Does anyone know?
 
Cheers
 
Richard
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160504/9a4c0a0d/attachment.htm>

From r.sobey at imperial.ac.uk  Thu May  5 09:25:02 2016
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Thu, 5 May 2016 08:25:02 +0000
Subject: [gpfsug-discuss] TSM and re-linked filesets
In-Reply-To: <7DCCBF95-92CF-4473-8BAB-F845C5408A46@qsplace.co.uk>
References: <AMSPR06MB4053E885E19DC5F12F46F74DF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
	<AMSPR06MB4053EB49583DA7B3623505EDF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
	<7DCCBF95-92CF-4473-8BAB-F845C5408A46@qsplace.co.uk>
Message-ID: <AMSPR06MB405B89D8C47300501E4A447DF7C0@AMSPR06MB405.eurprd06.prod.outlook.com>

Hi Lauz

When we want to remove filesets we link them in an area that isn?t exported by CIFS or NFS. This allows us a grace period where the users most definitely cannot access the files before we delete them permanently.

Normally it?s not a problem as most of our filesets are only a few hundred GB, but this particular one contains 80TB of data (160TB replicated) and I don?t fancy it getting backed up again!

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Laurence Horrors-Barlow
Sent: 04 May 2016 18:50
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] TSM and re-linked filesets

Hi Richard,

That's correct, TSM will only look at the path and compare files it doesn't understand the file set. Once the path has changed it will then start expiring the missing files.

What are you trying to achieve here as you might be able to update the expiration or fudge the DB.

-- Lauz

Sent from my iPad

On 4 May 2016, at 16:35, Sobey, Richard A <r.sobey at imperial.ac.uk<mailto:r.sobey at imperial.ac.uk>> wrote:
A quick Google has given me the answer I dreaded:

?Attention: If you are using the TSM Backup-Archive client you must use caution when you unlink filesets that contain data backed up by TSM. TSM tracks files by pathname and does not track filesets. As a result, when you unlink a fileset, it appears to TSM that you deleted the contents of the fileset. Therefore, the TSM Backup-Archive client inactivates the data on the TSM server which may result in the loss of backup data during the expiration process.?

So I need to relink it to the original place I think. Oh well. Not a disaster.

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A
Sent: 04 May 2016 16:33
To: 'gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>' <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: [gpfsug-discuss] TSM and re-linked filesets

This is more a TSM question than GPFS but it involves GPFS to a point so I hope I?m ok asking this.

I?ve unlinked a fileset and relinked it at a different junction point. Will this cause TSM to see a ?new? fileset and back up all its content? This is very undesireable so if I need to put it back where it belongs then I need to do this before tonight.

GPFS 3.5 and TSM 7.1.

Does anyone know?

Cheers

Richard

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/f8406628/attachment.htm>

From r.sobey at imperial.ac.uk  Thu May  5 09:25:49 2016
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Thu, 5 May 2016 08:25:49 +0000
Subject: [gpfsug-discuss] TSM and re-linked filesets
In-Reply-To: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F576F2A9@EX13.adf.bham.ac.uk>
References: <AMSPR06MB4053EB49583DA7B3623505EDF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
	<CF45EE16DEF2FE4B9AA7FF2B6EE26545F576F2A9@EX13.adf.bham.ac.uk>
Message-ID: <AMSPR06MB40548B61A9C4917DBA25F2BDF7C0@AMSPR06MB405.eurprd06.prod.outlook.com>

Err, I think you?ve already answered the question with the word ?nasty? heh.

Thanks for all the suggestions folks.

Richard

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (Research Computing - IT Services)
Sent: 04 May 2016 16:59
To: 'gpfsug main discussion list' <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] TSM and re-linked filesets

Could you do something really nasty in the tsm db2 database to update the paths?

-----Original Message-----
From: Sobey, Richard A [r.sobey at imperial.ac.uk<mailto:r.sobey at imperial.ac.uk>]
Sent: Wednesday, May 04, 2016 04:35 PM GMT Standard Time
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] TSM and re-linked filesets


A quick Google has given me the answer I dreaded:

?Attention: If you are using the TSM Backup-Archive client you must use caution when you unlink filesets that contain data backed up by TSM. TSM tracks files by pathname and does not track filesets. As a result, when you unlink a fileset, it appears to TSM that you deleted the contents of the fileset. Therefore, the TSM Backup-Archive client inactivates the data on the TSM server which may result in the loss of backup data during the expiration process.?

So I need to relink it to the original place I think. Oh well. Not a disaster.

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A
Sent: 04 May 2016 16:33
To: 'gpfsug-discuss at spectrumscale.org' <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: [gpfsug-discuss] TSM and re-linked filesets

This is more a TSM question than GPFS but it involves GPFS to a point so I hope I?m ok asking this.

I?ve unlinked a fileset and relinked it at a different junction point. Will this cause TSM to see a ?new? fileset and back up all its content? This is very undesireable so if I need to put it back where it belongs then I need to do this before tonight.

GPFS 3.5 and TSM 7.1.

Does anyone know?

Cheers

Richard

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/284f5a55/attachment.htm>

From stijn.deweirdt at ugent.be  Thu May  5 09:33:45 2016
From: stijn.deweirdt at ugent.be (Stijn De Weirdt)
Date: Thu, 5 May 2016 10:33:45 +0200
Subject: [gpfsug-discuss] GPFS VM passthrough
Message-ID: <572B0569.1000201@ugent.be>

hi all,


we are examining the possibility to give VMs access to GPFS mounted on
the hypervisors. (use cases are readonly or read-mostly access to the
data on gpfs, the hypervisors have IB, the vms not yet; and we have no
idea how to handle possible explosion of gpfs client licenses if we
mount gpfs inside the VMs ;)

has anyone experience with using the 9p filesystem passthrough and gpfs
with qemu/kvm?

many thanks,

stijn


From laurence at qsplace.co.uk  Thu May  5 09:48:25 2016
From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow)
Date: Thu, 05 May 2016 09:48:25 +0100
Subject: [gpfsug-discuss] GPFS VM passthrough
In-Reply-To: <572B0569.1000201@ugent.be>
References: <572B0569.1000201@ugent.be>
Message-ID: <FF599E3E-FFEE-46B3-B2F1-BB1E0C882B6A@qsplace.co.uk>

I have a few KVM systems that use 9p to pass GPFS through and it does just work; although I've not really benchmarked or pushed it yet.

Regarding licencing, its my understanding that the guest inherits the hosts licence i.e. if you have a client license on the host all vms are covered by the same license (iirc there is a core/socket caveat) So that is worth exploring further.

-- Lauz

On 5 May 2016 09:33:45 BST, Stijn De Weirdt <stijn.deweirdt at ugent.be> wrote:
>hi all,
>
>
>we are examining the possibility to give VMs access to GPFS mounted on
>the hypervisors. (use cases are readonly or read-mostly access to the
>data on gpfs, the hypervisors have IB, the vms not yet; and we have no
>idea how to handle possible explosion of gpfs client licenses if we
>mount gpfs inside the VMs ;)
>
>has anyone experience with using the 9p filesystem passthrough and gpfs
>with qemu/kvm?
>
>many thanks,
>
>stijn
>_______________________________________________
>gpfsug-discuss mailing list
>gpfsug-discuss at spectrumscale.org
>http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/6f03a920/attachment.htm>

From laurence at qsplace.co.uk  Thu May  5 09:48:25 2016
From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow)
Date: Thu, 05 May 2016 09:48:25 +0100
Subject: [gpfsug-discuss] GPFS VM passthrough
In-Reply-To: <572B0569.1000201@ugent.be>
References: <572B0569.1000201@ugent.be>
Message-ID: <FF599E3E-FFEE-46B3-B2F1-BB1E0C882B6A@qsplace.co.uk>

I have a few KVM systems that use 9p to pass GPFS through and it does just work; although I've not really benchmarked or pushed it yet.

Regarding licencing, its my understanding that the guest inherits the hosts licence i.e. if you have a client license on the host all vms are covered by the same license (iirc there is a core/socket caveat) So that is worth exploring further.

-- Lauz

On 5 May 2016 09:33:45 BST, Stijn De Weirdt <stijn.deweirdt at ugent.be> wrote:
>hi all,
>
>
>we are examining the possibility to give VMs access to GPFS mounted on
>the hypervisors. (use cases are readonly or read-mostly access to the
>data on gpfs, the hypervisors have IB, the vms not yet; and we have no
>idea how to handle possible explosion of gpfs client licenses if we
>mount gpfs inside the VMs ;)
>
>has anyone experience with using the 9p filesystem passthrough and gpfs
>with qemu/kvm?
>
>many thanks,
>
>stijn
>_______________________________________________
>gpfsug-discuss mailing list
>gpfsug-discuss at spectrumscale.org
>http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/6f03a920/attachment-0001.htm>

From daniel.kidger at uk.ibm.com  Thu May  5 09:52:22 2016
From: daniel.kidger at uk.ibm.com (Daniel Kidger)
Date: Thu, 5 May 2016 08:52:22 +0000
Subject: [gpfsug-discuss] GPFS VM passthrough
In-Reply-To: <572B0569.1000201@ugent.be>
Message-ID: <201605050852.u458qSQn011217@d06av04.portsmouth.uk.ibm.com>

Fortunately the licences issue is simple. 
The rules state that the sum of licences of all VMs on a node is capped at the number of sockets that server has.
So if you have 9 VMs using a mixture of different core counts each then you do not need more than 2 client licences if all hosted on a standard two socket server.

Daniel

Sent from IBM Verse


   Stijn De Weirdt --- [gpfsug-discuss] GPFS VM passthrough --- 
    From:"Stijn De Weirdt" <stijn.deweirdt at ugent.be>To:"gpfsug main discussion list" <gpfsug-discuss at gpfsug.org>Date:Thu, 5 May 2016 09:34Subject:[gpfsug-discuss] GPFS VM passthrough
  
    hi all,we are examining the possibility to give VMs access to GPFS mounted onthe hypervisors. (use cases are readonly or read-mostly access to thedata on gpfs, the hypervisors have IB, the vms not yet; and we have noidea how to handle possible explosion of gpfs client licenses if wemount gpfs inside the VMs ;)has anyone experience with using the 9p filesystem passthrough and gpfswith qemu/kvm?many thanks,stijn_______________________________________________gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discussUnless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/6fc5b68f/attachment.htm>

From makaplan at us.ibm.com  Thu May  5 16:26:33 2016
From: makaplan at us.ibm.com (Marc A Kaplan)
Date: Thu, 5 May 2016 11:26:33 -0400
Subject: [gpfsug-discuss] TSM and re-linked filesets
In-Reply-To: <AMSPR06MB40548B61A9C4917DBA25F2BDF7C0@AMSPR06MB405.eurprd06.prod.outlook.com>
References: <AMSPR06MB4053EB49583DA7B3623505EDF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
	<CF45EE16DEF2FE4B9AA7FF2B6EE26545F576F2A9@EX13.adf.bham.ac.uk>
	<AMSPR06MB40548B61A9C4917DBA25F2BDF7C0@AMSPR06MB405.eurprd06.prod.outlook.com>
Message-ID: <201605051526.u45FQgxg012902@d03av02.boulder.ibm.com>

My little suggestion is to put it back where it was.  And then add a 
symlink from the desired (new) path to the TSM-needs-this actual path.



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/36637b8b/attachment.htm>

From syi at ca.ibm.com  Thu May  5 22:09:49 2016
From: syi at ca.ibm.com (Yi Sun)
Date: Thu, 5 May 2016 17:09:49 -0400
Subject: [gpfsug-discuss] SPXXL meeting in NYC
Message-ID: <201605052109.u45L9vJ6015648@d03av01.boulder.ibm.com>





Hi Kristy and Bob,


For SPXXL meeting, if only interested in GPFS/Spectrum Scale day, is it
still nessary to make registration to SPXXL meeting?


Regards,


Yi Sun


syi at ca.ibm.com



>
>
> Message: 1
> Date: Mon, 4 Apr 2016 16:52:37 -0400
> From: GPFS UG USA Principal <usa-principal at gpfsug.org>
> To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
> Subject: [gpfsug-discuss] GPFS/Spectrum Scale Upcoming US Events -
> 		 		  Save the		 		  Dates
> Message-ID: <4192D644-E3AB-4B7B-AF7A-96D3D617FA7B at gpfsug.org>
> Content-Type: text/plain; charset="utf-8"
>
> Hello all,
>
> 		 		  We?d like to announce two upcoming US
GPFS/Spectrum Scale
> Events. One on the east coast, one in the midwest.
>
> 1) May 26th (full day event): GPFS/Spectrum Scale Day at the SPXXL
> conference in NYC https://www.spxxl.org/?q=New-York-City-2016 <
> https://www.spxxl.org/?q=New-York-City-2016> Developers and Engineers
from
> IBM will be at the meeting to cover topics, and open dialogue between IBM
> and customers will be encouraged. More details about the agenda, hotel
and
> other logistics will come later this month.
>
> Tentative Agenda:
> 		 		  ? 10 reasons for upgrading from GPFS 3.5 to
Spectrum Scale
> 4.2.1
> 		 		  ? Enhancements for CORAL from IBM
> 		 		  ? Panel discussion with customers, topic TBD
> 		 		  ? AFM and integration with Spectrum Protect
> 		 		  ? Best practices for GPFS or Spectrum Scale
Tuning.
> 		 		  ? At least one site update
>
> Location:
> New York Academy of Medicine
> 1216 Fifth Avenue
> New York, NY 10029
>
> ??
>
> 2) June 10th (full day event): GPFS/Spectrum Scale Users Group Meeting at
> Argonne National Lab (ANL). Thanks to Argonne for hosting this event.
> Developers and Engineers from IBM will be at the meeting to cover topics,
> and open dialogue between IBM and customers will be encouraged, as usual
no
> marketing pitches!
>
> Location: Argonne National Lab more details and final agenda will come
> later.
>
> Tentative Agenda:
>
> 9:00a-12:30p
> 9-9:30a - Opening Remarks
> 9:30-10a Deep Dive - Update on ESS
> 10a-11a Deep Dive - Problem Determination (Presentation 30 min/Panel 30
> min?)
> 11-11:30 Break
> 11:30a-Noon - Deep Dive - Protect & Scale integration
> Noon-12:30p HDFS/Hadoop
>
> 12:30 - 1:30p Lunch
>
> 1:30p-5:00p
> 1:30 - 2:00p IBM AFM Update
> 2:00-2:30p ANL: AFM as a burst buffer
> 2:30-3:00p ANL: GHI (GPFS HPSS Integration)
> 3:00-3:30p Break
> 3:30p - 4:00p LANL: ? or other site preso
> 4:00-4:30p Nuance: GPFS Performance Sensors Deployment Experiences
> 4:30p -5:00p Closing comments and Open Forum for Questions
>
> 5:00 - ?
> Beer hunting?
>
> ??
>
> We hope you can attend one or both of these events.
>
> Best,
> Kristy Kallback-Rose & Bob Oesterlin
> GPFS Users Group - USA Chapter - Principal & Co-principal
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
>
http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160404/2e191559/attachment-0001.html

> >
>
> ------------------------------
>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/d03f7737/attachment.htm>

From kallbac at iu.edu  Thu May  5 22:31:43 2016
From: kallbac at iu.edu (Kallback-Rose, Kristy A)
Date: Thu, 5 May 2016 17:31:43 -0400
Subject: [gpfsug-discuss] SPXXL meeting in NYC
In-Reply-To: <201605052109.u45L9vJ6015648@d03av01.boulder.ibm.com>
References: <201605052109.u45L9vJ6015648@d03av01.boulder.ibm.com>
Message-ID: <106AD35B-EC40-45D6-A4C9-42F5C86600CB@iu.edu>

Yes, you should register so they know how many will attend and have seating/food. There is a $0 option for the GPFS day. Thanks for asking! If you have another question, let us know.

-Kristy

> On May 5, 2016, at 5:09 PM, Yi Sun <syi at ca.ibm.com> wrote:
> 
> Hi Kristy and Bob,
> 
> For SPXXL meeting, if only interested in GPFS/Spectrum Scale day, is it still nessary to make registration to SPXXL meeting?
> 
> Regards,
> 
> Yi Sun
> 
> syi at ca.ibm.com
> 
> 
> >
> >
> > Message: 1
> > Date: Mon, 4 Apr 2016 16:52:37 -0400
> > From: GPFS UG USA Principal <usa-principal at gpfsug.org>
> > To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
> > Subject: [gpfsug-discuss] GPFS/Spectrum Scale Upcoming US Events -
> >  Save the  Dates
> > Message-ID: <4192D644-E3AB-4B7B-AF7A-96D3D617FA7B at gpfsug.org>
> > Content-Type: text/plain; charset="utf-8"
> >
> > Hello all,
> >
> >  We?d like to announce two upcoming US GPFS/Spectrum Scale
> > Events. One on the east coast, one in the midwest.
> >
> > 1) May 26th (full day event): GPFS/Spectrum Scale Day at the SPXXL
> > conference in NYC https://www.spxxl.org/?q=New-York-City-2016 <https://www.spxxl.org/?q=New-York-City-2016> <
> > https://www.spxxl.org/?q=New-York-City-2016 <https://www.spxxl.org/?q=New-York-City-2016>> Developers and Engineers from
> > IBM will be at the meeting to cover topics, and open dialogue between IBM
> > and customers will be encouraged. More details about the agenda, hotel and
> > other logistics will come later this month.
> >
> > Tentative Agenda:
> >  ? 10 reasons for upgrading from GPFS 3.5 to Spectrum Scale
> > 4.2.1
> >  ? Enhancements for CORAL from IBM
> >  ? Panel discussion with customers, topic TBD
> >  ? AFM and integration with Spectrum Protect
> >  ? Best practices for GPFS or Spectrum Scale Tuning.
> >  ? At least one site update
> >
> > Location:
> > New York Academy of Medicine
> > 1216 Fifth Avenue
> > New York, NY 10029
> >
> > ??
> >
> > 2) June 10th (full day event): GPFS/Spectrum Scale Users Group Meeting at
> > Argonne National Lab (ANL). Thanks to Argonne for hosting this event.
> > Developers and Engineers from IBM will be at the meeting to cover topics,
> > and open dialogue between IBM and customers will be encouraged, as usual no
> > marketing pitches!
> >
> > Location: Argonne National Lab more details and final agenda will come
> > later.
> >
> > Tentative Agenda:
> >
> > 9:00a-12:30p
> > 9-9:30a - Opening Remarks
> > 9:30-10a Deep Dive - Update on ESS
> > 10a-11a Deep Dive - Problem Determination (Presentation 30 min/Panel 30
> > min?)
> > 11-11:30 Break
> > 11:30a-Noon - Deep Dive - Protect & Scale integration
> > Noon-12:30p HDFS/Hadoop
> >
> > 12:30 - 1:30p Lunch
> >
> > 1:30p-5:00p
> > 1:30 - 2:00p IBM AFM Update
> > 2:00-2:30p ANL: AFM as a burst buffer
> > 2:30-3:00p ANL: GHI (GPFS HPSS Integration)
> > 3:00-3:30p Break
> > 3:30p - 4:00p LANL: ? or other site preso
> > 4:00-4:30p Nuance: GPFS Performance Sensors Deployment Experiences
> > 4:30p -5:00p Closing comments and Open Forum for Questions
> >
> > 5:00 - ?
> > Beer hunting?
> >
> > ??
> >
> > We hope you can attend one or both of these events.
> >
> > Best,
> > Kristy Kallback-Rose & Bob Oesterlin
> > GPFS Users Group - USA Chapter - Principal & Co-principal
> > -------------- next part --------------
> > An HTML attachment was scrubbed...
> > URL: <
> > http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160404/2e191559/attachment-0001.html <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160404/2e191559/attachment-0001.html>
> > >
> >
> > ------------------------------
> >
> 
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/5ebe30d2/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/5ebe30d2/attachment.sig>

From syi at ca.ibm.com  Thu May  5 22:36:05 2016
From: syi at ca.ibm.com (Yi Sun)
Date: Thu, 5 May 2016 17:36:05 -0400
Subject: [gpfsug-discuss] SPXXL meeting in NYC
Message-ID: <201605052136.u45LaEjv002729@d03av03.boulder.ibm.com>



Thank you Kristy.

=============================================================================================================
Date: Thu, 5 May 2016 17:31:43 -0400
From: "Kallback-Rose, Kristy A" <kallbac at iu.edu>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] SPXXL meeting in NYC
Message-ID: <106AD35B-EC40-45D6-A4C9-42F5C86600CB at iu.edu>
Content-Type: text/plain; charset="us-ascii"

Yes, you should register so they know how many will attend and have
seating/food. There is a $0 option for the GPFS day. Thanks for asking! If
you have another question, let us know.

-Kristy

> On May 5, 2016, at 5:09 PM, Yi Sun <syi at ca.ibm.com> wrote:
>
> Hi Kristy and Bob,
>
> For SPXXL meeting, if only interested in GPFS/Spectrum Scale day, is it
still nessary to make registration to SPXXL meeting?
>
> Regards,
>
> Yi Sun
>
> syi at ca.ibm.com




Regards,



Yi (Y.) Sun
Technical Account Manager
IBM SDI
IBM Systems


                                                                                      
                                                                                      
                                                                                      
 Phone: 1-905-316-2681                                                          IBM   
 E-mail: syi at ca.ibm.com                                                               
                                                           3600 Steeles Avenue East   
                                                                Markham, ON L3R 9Z7   
                                                                             Canada   
                                                                                      
                                                                                      
                                                                                      





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/2fb39b0e/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0C466617.gif
Type: image/gif
Size: 2022 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/2fb39b0e/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ecblank.gif
Type: image/gif
Size: 45 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/2fb39b0e/attachment-0001.gif>

From TROPPENS at de.ibm.com  Tue May 10 13:00:16 2016
From: TROPPENS at de.ibm.com (Ulf Troppens)
Date: Tue, 10 May 2016 14:00:16 +0200
Subject: [gpfsug-discuss] GPFS/Spectrum Scale Upcoming US Events - Save
 the Dates - Agenda for New York May 26
In-Reply-To: <201605031206.u43C5wss002725@d06av09.portsmouth.uk.ibm.com>
References: <4192D644-E3AB-4B7B-AF7A-96D3D617FA7B@gpfsug.org><9489DBA2-1F12-4B05-A968-5D4855FBEA40@gpfsug.org>
	<201605031206.u43C5wss002725@d06av09.portsmouth.uk.ibm.com>
Message-ID: <201605101100.u4AB0Od3024998@d06av10.portsmouth.uk.ibm.com>

The agenda for the meeting in New York is now available here:
https://www.spxxl.org/?q=New-York-City-2016

For your convenience - here are the session titles:
- Top 10 reasons for upgrading from GPFS 3.5 to Spectrum Scale 4.2.1
- Panel Discussion: My favorite tool for monitoring/reporting
- Problem Determination
- User presentation : TBD
- Spectrum Scale in Life Sciences
- HPSS Update
- Best practices for GPFS or Spectrum Scale Tuning
- Enhancements for CORAL

Please use the link below for registration.

--
IBM Spectrum Scale Development - Client Engagements & Solutions Delivery
Consulting IT Specialist
Author "Storage Networks Explained"

IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Gesch?ftsf?hrung: Dirk Wittkopp
Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart,
HRB 243294




From:	Ulf Troppens/Germany/IBM at IBMDE
To:	gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:	03.05.2016 14:06
Subject:	Re: [gpfsug-discuss] GPFS/Spectrum Scale Upcoming US Events -
            Save the Dates - Registration for New York May 26
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



Registration for the first event in New York is up as well - the agenda
still needs some touch.

Please register here and select 'May 26 GPFS Day Registration', if you
don't attend SPXXL:
https://www.spxxl.org/?q=New-York-City-2016
https://www.eventbrite.com/e/spxxlscicomp-2016-summer-meeting-registration-24444020724



--
IBM Spectrum Scale Development - Client Engagements & Solutions Delivery
Consulting IT Specialist
Author "Storage Networks Explained"

IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Gesch?ftsf?hrung: Dirk Wittkopp
Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart,
HRB 243294


Inactive hide details for GPFS UG USA Principal ---28.04.2016
23:44:50---All, the registration page for the second event listedGPFS UG
USA Principal ---28.04.2016 23:44:50---All, the registration page for the
second event listed below at Argonne National Lab on June 10th is

From: GPFS UG USA Principal <usa-principal at gpfsug.org>
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Date: 28.04.2016 23:44
Subject: Re: [gpfsug-discuss] GPFS/Spectrum Scale Upcoming US Events - Save
the Dates
Sent by: gpfsug-discuss-bounces at spectrumscale.org



All, the registration page for the second event listed below at Argonne
National Lab on June 10th is now up. An updated agenda is also at this
site.

Please register here: https://www.regonline.com/Spectrumscalemeeting

We look forward to seeing some of you at these upcoming events.

Feel free to send suggestions for future events in your area.

Cheers,
-Kristy

            On Apr 4, 2016, at 4:52 PM, GPFS UG USA Principal <
            usa-principal at gpfsug.org> wrote:

            Hello all,

            We?d like to announce two upcoming US GPFS/Spectrum Scale
            Events. One on the east coast, one in the midwest.

            1) May 26th (full day event): GPFS/Spectrum Scale Day at the
            SPXXL conference in NYC
            https://www.spxxl.org/?q=New-York-City-2016 Developers and
            Engineers from IBM will be at the meeting to cover topics, and
            open dialogue between IBM and customers will be encouraged.
            More details about the agenda, hotel and other logistics will
            come later this month.

            Tentative Agenda:
            ? 10 reasons for upgrading from GPFS 3.5 to Spectrum Scale
            4.2.1
            ? Enhancements for CORAL from IBM
            ? Panel discussion with customers, topic TBD
            ? AFM and integration with Spectrum Protect
            ? Best practices for GPFS or Spectrum Scale Tuning.
            ? At least one site update

            Location:
            New York Academy of Medicine
            1216 Fifth Avenue
            New York, NY 10029

            ??

            2) June 10th (full day event): GPFS/Spectrum Scale Users Group
            Meeting at Argonne National Lab (ANL). Thanks to Argonne for
            hosting this event. Developers and Engineers from IBM will be
            at the meeting to cover topics, and open dialogue between IBM
            and customers will be encouraged, as usual no marketing
            pitches!

            Location: Argonne National Lab more details and final agenda
            will come later.

            Tentative Agenda:

            9:00a-12:30p
            9-9:30a - Opening Remarks
            9:30-10a Deep Dive - Update on ESS
            10a-11a Deep Dive - Problem Determination (Presentation 30
            min/Panel 30 min?)
            11-11:30 Break
            11:30a-Noon - Deep Dive - Protect & Scale integration
            Noon-12:30p HDFS/Hadoop

            12:30 - 1:30p Lunch

            1:30p-5:00p
            1:30 - 2:00p IBM AFM Update
            2:00-2:30p ANL: AFM as a burst buffer
            2:30-3:00p ANL: GHI (GPFS HPSS Integration)
            3:00-3:30p Break
            3:30p - 4:00p LANL: ? or other site preso
            4:00-4:30p Nuance: GPFS Performance Sensors Deployment
            Experiences
            4:30p -5:00p Closing comments and Open Forum for Questions

            5:00 - ?
            Beer hunting?

            ??

            We hope you can attend one or both of these events.

            Best,
            Kristy Kallback-Rose & Bob Oesterlin
            GPFS Users Group - USA Chapter - Principal & Co-principal


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160510/2d3525b5/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160510/2d3525b5/attachment.gif>

From Mark.Bush at siriuscom.com  Wed May 11 03:02:30 2016
From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com)
Date: Wed, 11 May 2016 02:02:30 +0000
Subject: [gpfsug-discuss] Demo
Message-ID: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>

I?m tasked with coming up with a demo of Spectrum Scale for our customers.  Something that shows a little bit of most of the features of Scale.  Does anyone have some ideas of ?must see? features that might make for a good demo?

My thoughts are something like:

Overview of architecture
NSD Servers, Protocol Servers, Quorum, etc
Show adding a Node to the mix as a client and a server
Show creating a filesystem (and a fileset)
Show copying some files from a client (linux client with GPFS or through a filesystem thru protocol nodes)
Show GUI
Show Snapshot

Any other thoughts?


Mark R. Bush | Solutions Architect
Mobile: 210.237.8415 | mark.bush at siriuscom.com
Sirius Computer Solutions | www.siriuscom.com<http://www.siriuscom.com>
10100 Reunion Place, Suite 500, San Antonio, TX 78216


This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you.

Sirius Computer Solutions<http://www.siriuscom.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160511/b9bb4704/attachment.htm>

From pinto at scinet.utoronto.ca  Wed May 11 03:35:19 2016
From: pinto at scinet.utoronto.ca (Jaime Pinto)
Date: Tue, 10 May 2016 22:35:19 -0400
Subject: [gpfsug-discuss] Demo
In-Reply-To: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
References: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
Message-ID: <20160510223519.11041xrqkfoeqbfr@support.scinet.utoronto.ca>

Make a clear distinction between the hardware and the software layers  
to start with.
Then between server infrastructure and client infrastructure (and  
licensing requirements)

NSD Servers, Protocol Servers, disks, LUNs, HW raid if any,  
configuration of the storage building blocks, NSD fail-over, etc

hardware raid vs. software raid vs, appliances

File system managers, cluster manager, clients addition/setup, FSmgr  
fail-over, Quorum, etc
File system creation, clustered vs. scattered, FS attributes,  
blocksize, capacity expansion/shrinkage, etc.

Fileset creation, junction linking, mounting, exportfs, protocol nodes.

storage cluster vs. compute cluster
snapshot, quota, (backup?), .....

For me there are areas/subjects that belong together, and a logic in  
the unveiling of features.

I would be interested in the end result

Thanks
Jaime


Quoting "Mark.Bush at siriuscom.com" <Mark.Bush at siriuscom.com>:

> I?m tasked with coming up with a demo of Spectrum Scale for our   
> customers.  Something that shows a little bit of most of the   
> features of Scale.  Does anyone have some ideas of ?must see?   
> features that might make for a good demo?
>
> My thoughts are something like:
>
> Overview of architecture
> NSD Servers, Protocol Servers, Quorum, etc
> Show adding a Node to the mix as a client and a server
> Show creating a filesystem (and a fileset)
> Show copying some files from a client (linux client with GPFS or   
> through a filesystem thru protocol nodes)
> Show GUI
> Show Snapshot
>
> Any other thoughts?
>
>
> Mark R. Bush | Solutions Architect
> Mobile: 210.237.8415 | mark.bush at siriuscom.com
> Sirius Computer Solutions | www.siriuscom.com<http://www.siriuscom.com>
> 10100 Reunion Place, Suite 500, San Antonio, TX 78216
>
>
> This message (including any attachments) is intended only for the   
> use of the individual or entity to which it is addressed and may   
> contain information that is non-public, proprietary, privileged,   
> confidential, and exempt from disclosure under applicable law. If   
> you are not the intended recipient, you are hereby notified that any  
>  use, dissemination, distribution, or copying of this communication   
> is strictly prohibited. This message may be viewed by parties at   
> Sirius Computer Solutions other than those named in the message   
> header. This message does not contain an official representation of   
> Sirius Computer Solutions. If you have received this communication   
> in error, notify Sirius Computer Solutions immediately and (i)   
> destroy this message if a facsimile or (ii) delete this message   
> immediately if this is an electronic communication. Thank you.
>
> Sirius Computer Solutions<http://www.siriuscom.com>
>




---
Jaime Pinto
SciNet HPC Consortium  - Compute/Calcul Canada
www.scinet.utoronto.ca - www.computecanada.org
University of Toronto
256 McCaul Street, Room 235
Toronto, ON, M5T1W5
P: 416-978-2755
C: 416-505-1477

----------------------------------------------------------------
This message was sent using IMP at SciNet Consortium, University of Toronto.



From sfadden at us.ibm.com  Wed May 11 06:51:38 2016
From: sfadden at us.ibm.com (Scott Fadden)
Date: Tue, 10 May 2016 22:51:38 -0700
Subject: [gpfsug-discuss] Demo
In-Reply-To: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
References: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
Message-ID: <201605110551.u4B5phC8017323@d03av02.boulder.ibm.com>

I developed a demo that shows how the file heat metric can be used to move 
"cool" data from a Flash tier to spinning disk. As visuals it displays a 
speedometer for transaction rate, a chart showing capacity used in each 
tier and total cost of storing the data ($/GiB). 

For a simpler demo maybe the GUI with live performance, maybe a policy on 
lots of files (use a list policy) or something like fileset level 
snapshots.


Scott Fadden
Spectrum Scale - Technical Marketing 
Phone: (503) 880-5833 
sfadden at us.ibm.com
http://www.ibm.com/systems/storage/spectrum/scale



From:   "Mark.Bush at siriuscom.com" <Mark.Bush at siriuscom.com>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   05/10/2016 07:02 PM
Subject:        [gpfsug-discuss] Demo
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



I?m tasked with coming up with a demo of Spectrum Scale for our customers. 
 Something that shows a little bit of most of the features of Scale.  Does 
anyone have some ideas of ?must see? features that might make for a good 
demo?

My thoughts are something like: 

Overview of architecture
NSD Servers, Protocol Servers, Quorum, etc
Show adding a Node to the mix as a client and a server
Show creating a filesystem (and a fileset)
Show copying some files from a client (linux client with GPFS or through a 
filesystem thru protocol nodes)
Show GUI
Show Snapshot

Any other thoughts?


Mark R. Bush | Solutions Architect
Mobile: 210.237.8415 | mark.bush at siriuscom.com
Sirius Computer Solutions | www.siriuscom.com
10100 Reunion Place, Suite 500, San Antonio, TX 78216 

This message (including any attachments) is intended only for the use of 
the individual or entity to which it is addressed and may contain 
information that is non-public, proprietary, privileged, confidential, and 
exempt from disclosure under applicable law. If you are not the intended 
recipient, you are hereby notified that any use, dissemination, 
distribution, or copying of this communication is strictly prohibited. 
This message may be viewed by parties at Sirius Computer Solutions other 
than those named in the message header. This message does not contain an 
official representation of Sirius Computer Solutions. If you have received 
this communication in error, notify Sirius Computer Solutions immediately 
and (i) destroy this message if a facsimile or (ii) delete this message 
immediately if this is an electronic communication. Thank you. 
Sirius Computer Solutions _______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160510/af957fe6/attachment.htm>

From Mikal.Sande at altibox.no  Wed May 11 08:10:02 2016
From: Mikal.Sande at altibox.no (Sande, Mikal)
Date: Wed, 11 May 2016 07:10:02 +0000
Subject: [gpfsug-discuss] Demo
In-Reply-To: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
References: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
Message-ID: <84C0B9AB4719954985704BC46DF8602A9F7E480B@prdmbx01.lyse.no>

Apart from the usefulness of parallel access to the same LUNs, one feature I value for metadata heavy workloads is being able to split data and metadata. When we moved metadata from spindles to flash LUNs on one of our clusters it was a day vs. night kind of change in behavior in the cluster.

Just my two cents, I know exactly how to demo the split between data and metadata. Maybe when creating the filesystem? Or maybe afterwards and show how restripe is used to perform such a change?


Best regards,
Mikal Sande
________________________________
From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Mark.Bush at siriuscom.com [Mark.Bush at siriuscom.com]
Sent: Wednesday, May 11, 2016 4:02 AM
To: gpfsug main discussion list
Subject: [gpfsug-discuss] Demo

I?m tasked with coming up with a demo of Spectrum Scale for our customers.  Something that shows a little bit of most of the features of Scale.  Does anyone have some ideas of ?must see? features that might make for a good demo?

My thoughts are something like:

Overview of architecture
NSD Servers, Protocol Servers, Quorum, etc
Show adding a Node to the mix as a client and a server
Show creating a filesystem (and a fileset)
Show copying some files from a client (linux client with GPFS or through a filesystem thru protocol nodes)
Show GUI
Show Snapshot

Any other thoughts?


Mark R. Bush | Solutions Architect
Mobile: 210.237.8415 | mark.bush at siriuscom.com
Sirius Computer Solutions | www.siriuscom.com<http://www.siriuscom.com>
10100 Reunion Place, Suite 500, San Antonio, TX 78216


This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you.

Sirius Computer Solutions<http://www.siriuscom.com>


From D.R.Barker at exeter.ac.uk  Wed May 11 10:53:15 2016
From: D.R.Barker at exeter.ac.uk (Barker, David)
Date: Wed, 11 May 2016 09:53:15 +0000
Subject: [gpfsug-discuss] UK May Meeting
Message-ID: <A4DBAD9F-EE30-4A0E-AD59-51BFCB18F4DB@exeter.ac.uk>

Hi Simon - sorry to have to cancel at the last minute, but I won?t be able to attend now. Feel free to give my space to someone else. :-/

Best regards,
Dave

-- 
David Barker
01392 723900
www.exeter.ac.uk
Laver Building, North Park Road, EXETER, EX4 4QE

 
 









On 11/04/2016, 10:37, "gpfsug-discuss-bounces at spectrumscale.org on behalf of GPFS UG Chair (Simon Thompson)" <gpfsug-discuss-bounces at spectrumscale.org on behalf of chair at spectrumscale.org> wrote:

>
>Hi All,
>
>We are down to our last few places for the May user group meeting, if you
>are planning to come along, please do register:
>
>The draft agenda and registration for the day is at:
>http://www.eventbrite.com/e/spectrum-scale-gpfs-uk-user-group-spring-2016-t
>ickets-21724951916
>
>If you have registered and aren't able to attend now, please do let us
>know so that we can free the slot for other members of the group.
>
>We also have 1 slot left on the agenda for a user talk, so if you have an
>interesting deployment or plans and are able to speak, please let me know!
>
>Thanks
>
>Simon
>
>
>_______________________________________________
>gpfsug-discuss mailing list
>gpfsug-discuss at spectrumscale.org
>http://gpfsug.org/mailman/listinfo/gpfsug-discuss

From Robert.Oesterlin at nuance.com  Wed May 11 17:05:12 2016
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Wed, 11 May 2016 16:05:12 +0000
Subject: [gpfsug-discuss] Reminder: Spectrum Scale/GPFS UG meeting,
	June 10th at Argonne Lab
Message-ID: <AB6F0D83-4CBA-4651-BE78-384276359819@nuance.com>

A Reminder that there will be a user group meeting on June 10th at Argonne Labs, near Chicago, IL. A great agenda and a great opportunity to network with other users.

Details: http://spectrumscale.org/spectrum-scale-user-group-usa-meeting-june-10th/

Registration link:https://www.regonline.com/Spectrumscalemeeting

Registration deadline is May 27th!

Bob Oesterlin
GPFS UG Co-Principal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160511/e5e5c3f7/attachment.htm>

From janfrode at tanso.net  Wed May 11 18:04:37 2016
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Wed, 11 May 2016 17:04:37 +0000
Subject: [gpfsug-discuss] Demo
In-Reply-To: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
References: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
Message-ID: <CAHwPathGPijoRstXc6GzBNJxfWiOF8kuNJFbJUui1xH-fJzFdA@mail.gmail.com>

Some suggestions for stuff I find demo-interesting..

Show snapshots accessible as previous version trough SMB/Windows Explorer.
Show offline files with X indicator in Explorer, if you can do HSM (maybe
trough via the cloud gateway beta).

Show object storage trough cyberduck, or similar clients.

Failover nfs/smb.

Mixed windows/linux cluster accessing same filesystem.

mmbackup doing backups of million of files in no time.



-jf

ons. 11. mai 2016 kl. 04.02 skrev Mark.Bush at siriuscom.com <
Mark.Bush at siriuscom.com>:

> I?m tasked with coming up with a demo of Spectrum Scale for our
> customers.  Something that shows a little bit of most of the features of
> Scale.  Does anyone have some ideas of ?must see? features that might make
> for a good demo?
>
> My thoughts are something like:
>
> Overview of architecture
> NSD Servers, Protocol Servers, Quorum, etc
> Show adding a Node to the mix as a client and a server
> Show creating a filesystem (and a fileset)
> Show copying some files from a client (linux client with GPFS or through a
> filesystem thru protocol nodes)
> Show GUI
> Show Snapshot
>
> Any other thoughts?
>
>
> Mark R. Bush | Solutions Architect
> Mobile: 210.237.8415 | mark.bush at siriuscom.com
> Sirius Computer Solutions | www.siriuscom.com
> 10100 Reunion Place, Suite 500, San Antonio, TX 78216
>
> This message (including any attachments) is intended only for the use of
> the individual or entity to which it is addressed and may contain
> information that is non-public, proprietary, privileged, confidential, and
> exempt from disclosure under applicable law. If you are not the intended
> recipient, you are hereby notified that any use, dissemination,
> distribution, or copying of this communication is strictly prohibited. This
> message may be viewed by parties at Sirius Computer Solutions other than
> those named in the message header. This message does not contain an
> official representation of Sirius Computer Solutions. If you have received
> this communication in error, notify Sirius Computer Solutions immediately
> and (i) destroy this message if a facsimile or (ii) delete this message
> immediately if this is an electronic communication. Thank you.
> Sirius Computer Solutions <http://www.siriuscom.com>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160511/5f4d6e64/attachment.htm>

From YARD at il.ibm.com  Wed May 11 20:49:08 2016
From: YARD at il.ibm.com (Yaron Daniel)
Date: Wed, 11 May 2016 22:49:08 +0300
Subject: [gpfsug-discuss] Demo
In-Reply-To: <201605110551.u4B5phC8017323@d03av02.boulder.ibm.com>
References: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
	<201605110551.u4B5phC8017323@d03av02.boulder.ibm.com>
Message-ID: <201605111949.u4BJnF8L015520@d06av02.portsmouth.uk.ibm.com>

Hi

I think what is hot:

- Compression
- Mixed access to same files via CIFS/NFS/GPFS client
- AFM DR capabilities

 
Regards
 


 
 
Yaron Daniel
 94 Em Ha'Moshavot Rd

Server, Storage and Data Services - Team Leader  
 Petach Tiqva, 49527
Global Technology Services
 Israel
Phone:
+972-3-916-5672
 
 
Fax:
+972-3-916-5672
 
 
Mobile:
+972-52-8395593
 
 
e-mail:
yard at il.ibm.com
 
 
IBM Israel
 
 
 
 

 



From:   "Scott Fadden" <sfadden at us.ibm.com>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   05/11/2016 08:51 AM
Subject:        Re: [gpfsug-discuss] Demo
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



I developed a demo that shows how the file heat metric can be used to move 
"cool" data from a Flash tier to spinning disk. As visuals it displays a 
speedometer for transaction rate, a chart showing capacity used in each 
tier and total cost of storing the data ($/GiB). 

For a simpler demo maybe the GUI with live performance, maybe a policy on 
lots of files (use a list policy) or something like fileset level 
snapshots.


Scott Fadden
Spectrum Scale - Technical Marketing 
Phone: (503) 880-5833 
sfadden at us.ibm.com
http://www.ibm.com/systems/storage/spectrum/scale



From:        "Mark.Bush at siriuscom.com" <Mark.Bush at siriuscom.com>
To:        gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:        05/10/2016 07:02 PM
Subject:        [gpfsug-discuss] Demo
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



I?m tasked with coming up with a demo of Spectrum Scale for our customers. 
 Something that shows a little bit of most of the features of Scale.  Does 
anyone have some ideas of ?must see? features that might make for a good 
demo?

My thoughts are something like: 

Overview of architecture
NSD Servers, Protocol Servers, Quorum, etc
Show adding a Node to the mix as a client and a server
Show creating a filesystem (and a fileset)
Show copying some files from a client (linux client with GPFS or through a 
filesystem thru protocol nodes)
Show GUI
Show Snapshot

Any other thoughts?


Mark R. Bush| Solutions Architect
Mobile: 210.237.8415 | mark.bush at siriuscom.com
Sirius Computer Solutions | www.siriuscom.com
10100 Reunion Place, Suite 500, San Antonio, TX 78216 
This message (including any attachments) is intended only for the use of 
the individual or entity to which it is addressed and may contain 
information that is non-public, proprietary, privileged, confidential, and 
exempt from disclosure under applicable law. If you are not the intended 
recipient, you are hereby notified that any use, dissemination, 
distribution, or copying of this communication is strictly prohibited. 
This message may be viewed by parties at Sirius Computer Solutions other 
than those named in the message header. This message does not contain an 
official representation of Sirius Computer Solutions. If you have received 
this communication in error, notify Sirius Computer Solutions immediately 
and (i) destroy this message if a facsimile or (ii) delete this message 
immediately if this is an electronic communication. Thank you. 
Sirius Computer Solutions _______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160511/8d1a26c5/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 1851 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160511/8d1a26c5/attachment.gif>

From pgalgali at us.ibm.com  Wed May 11 21:53:25 2016
From: pgalgali at us.ibm.com (Pallavi V Galgali)
Date: Wed, 11 May 2016 15:53:25 -0500
Subject: [gpfsug-discuss] Demo
In-Reply-To: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
References: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
Message-ID: <201605112053.u4BKrWeI005237@d01av04.pok.ibm.com>

Pls explore Spectrum Scale trial VM and its attached documentation -
http://www-03.ibm.com/systems/storage/spectrum/scale/trial.html
This comes with documentation ( Explore guide and Advanced user guide) that
has some interesting scenarios that can be used for demo purpose.

                                                                                     
   Thanks & Regards,                                                                 
                                                                                     
   Pallavi Galgali                                                                   
   Manager, Client Adoption and Solutions Delivery                                   
   IBM Spectrum Scale Development                                                    
   Cell: +1 914 433 9882                                                             
   11501 Burnet Road Austin TX 78758                                                 
                                                                                     








From:	"Mark.Bush at siriuscom.com" <Mark.Bush at siriuscom.com>
To:	gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:	05/10/2016 09:02 PM
Subject:	[gpfsug-discuss] Demo
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



I?m tasked with coming up with a demo of Spectrum Scale for our customers.
Something that shows a little bit of most of the features of Scale.  Does
anyone have some ideas of ?must see? features that might make for a good
demo?

My thoughts are something like:

Overview of architecture
NSD Servers, Protocol Servers, Quorum, etc
Show adding a Node to the mix as a client and a server
Show creating a filesystem (and a fileset)
Show copying some files from a client (linux client with GPFS or through a
filesystem thru protocol nodes)
Show GUI
Show Snapshot

Any other thoughts?


Mark R. Bush | Solutions Architect
Mobile: 210.237.8415 | mark.bush at siriuscom.com
Sirius Computer Solutions | www.siriuscom.com
10100 Reunion Place, Suite 500, San Antonio, TX 78216



This message (including any attachments) is intended only for the use of
the individual or entity to which it is addressed and may contain
information that is non-public, proprietary, privileged, confidential, and
exempt from disclosure under applicable law. If you are not the intended
recipient, you are hereby notified that any use, dissemination,
distribution, or copying of this communication is strictly prohibited. This
message may be viewed by parties at Sirius Computer Solutions other than
those named in the message header. This message does not contain an
official representation of Sirius Computer Solutions. If you have received
this communication in error, notify Sirius Computer Solutions immediately
and (i) destroy this message if a facsimile or (ii) delete this message
immediately if this is an electronic communication. Thank you.


Sirius Computer Solutions _______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160511/694b75a9/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160511/694b75a9/attachment.gif>

From douglasof at us.ibm.com  Fri May 13 16:01:55 2016
From: douglasof at us.ibm.com (Douglas O'flaherty)
Date: Fri, 13 May 2016 11:01:55 -0400
Subject: [gpfsug-discuss] ISC U/G Agenda
Message-ID: <201605131502.u4DF23Bi020162@d03av05.boulder.ibm.com>

Greetings:

IBM is happy to announce the agenda for the joint IBM Spectrum Scale and 
Platform Computing User Group Agenda at ISC. We will finish on time to 
attend the opening reception. 

As with other user group meetings, the agenda includes user stories, 
updates on IBM Spectrum Scale & Platform LSF, and access to IBM experts 
and your peers. 

Please join us! 

To attend, please email Fabian.Beckmann at de.ibm.com so we can have an 
accurate count of attendees. 


Monday June 20, 2016  - 14:30-18:00 - Conference Room Konstant

14:30-14:40 [10 min] Welcome (Douglas o'Flaherty, IBM)
14:40-15:00 [20 min] Ten Reasons to Upgrade from GPFS 3.4 to Spectrum 
Scale 4.2 (Olaf Weiser, IBM)
15:00-15:30 [30 min] Shared Storage with in-memory latency: EMC DSSD D5 
and IBM Spectrum Scale (Stefan Radtke, EMC)
15:30-16:00 [30 min] Workload scheduling and data management in a private 
cloud (Uwe Sommer, Airbus)
16:00-16:30 [30 min] Spectrum Scale site report (To be confirmed by 
customer)
16:30-17:00 [30 min] What's new in Platform LSF 10.1 & storage integration 
(Bill McMillan, IBM)
17:00-17:30 [30 min] What's new in Spectrum Scale 4.2.1 (Mathias Dietz, 
IBM)
17:30-18:00 [30 min] CORAL enhancements for Spectrum Scale (Sven Oehme, 
IBM)

Looking forward to seeing you there!


doug

PS: IBMers can register their clients at this IBM link: 
https://w3-9033.ibm.com/events/ast/schedule/16isc.nsf 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160513/bba0eeae/attachment.htm>

From billowen at us.ibm.com  Sat May 14 13:05:14 2016
From: billowen at us.ibm.com (Bill Owen)
Date: Sat, 14 May 2016 05:05:14 -0700
Subject: [gpfsug-discuss] OpenStack & Spectrum Scale Usage Survey
Message-ID: <201605141205.u4EC5KBm000338@d03av04.boulder.ibm.com>


We in Spectrum Scale development are very interested in learning about your
current & planned usage of OpenStack with Spectrum Scale, as well as plans
for deploying containers in your Spectrum Scale environment.

We would like to use this information to help shape our road map in this
area over the next 18-24 months.

Please take 10 minutes to answer the questions in this short survey:
https://www.surveygizmo.com/s3/2774614/IBMSpectrumScale-OpenStackUsageSurvey

Thank you,
Bill Owen
billowen at us.ibm.com
Spectrum Scale Object Storage
520-799-4829
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160514/be7206f9/attachment.htm>

From S.J.Thompson at bham.ac.uk  Mon May 16 09:44:05 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Mon, 16 May 2016 08:44:05 +0000
Subject: [gpfsug-discuss] Flash for metadata
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F577FBC6@EX13.adf.bham.ac.uk>

Hi all

I'm currently about to deploy some flash systems (FS-900) to take our metadata for one of our Spectrum Scale file-systems. We use 4k inodes for this file-system.

On the FS-900, when creating a volume, we get a choice of 512b or 4k sectors. Does anyone have a suggestion on this? On a spinning disk RAID array, I get what a strip size is and we'd use multiples of the stripe where possible for the block size.

Is the sector size on the FS-900 the smallest chunk that can be written out? And so would it make sense to use 4k sectors for 4k GPFS inodes right?

Related. In the deployment guide for the FS-900 arrays, if one is using a file-system (e.g. XFS) on the array the guide talks about aligning the partitions with with array, when GPFS is used on a storage array, does it automatically work out the sector alignment?


Thanks

Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160516/2909e86d/attachment.htm>

From Paul.Sanchez at deshaw.com  Mon May 16 10:28:35 2016
From: Paul.Sanchez at deshaw.com (Sanchez, Paul)
Date: Mon, 16 May 2016 09:28:35 +0000
Subject: [gpfsug-discuss] Flash for metadata
In-Reply-To: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F577FBC6@EX13.adf.bham.ac.uk>
References: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F577FBC6@EX13.adf.bham.ac.uk>
Message-ID: <49a3ab3b493a49e59177c6c25dd34d23@mbxtoa1.winmail.deshaw.com>

Simon,

I can't speak specifically to the FS-900, since I generally use Dell Compellent and NetApp EF arrays for flash today.  But I also take care to ensure that the system pool metadata NSDs use RAID10 (or equivalent) rather than RAID5/6.  For small 4K metadata writes, on a 4K blocksize LUN this could result in optimal writes.  For larger native blocksize LUNs, it eliminates the parity calculation for read-modify-write operations.

Thanks,
Paul

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (Research Computing - IT Services)
Sent: Monday, May 16, 2016 9:44 AM
To: gpfsug-discuss at spectrumscale.org
Subject: [gpfsug-discuss] Flash for metadata

Hi all

I'm currently about to deploy some flash systems (FS-900) to take our metadata for one of our Spectrum Scale file-systems. We use 4k inodes for this file-system.

On the FS-900, when creating a volume, we get a choice of 512b or 4k sectors. Does anyone have a suggestion on this? On a spinning disk RAID array, I get what a strip size is and we'd use multiples of the stripe where possible for the block size.

Is the sector size on the FS-900 the smallest chunk that can be written out? And so would it make sense to use 4k sectors for 4k GPFS inodes right?

Related. In the deployment guide for the FS-900 arrays, if one is using a file-system (e.g. XFS) on the array the guide talks about aligning the partitions with with array, when GPFS is used on a storage array, does it automatically work out the sector alignment?


Thanks

Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160516/f4964333/attachment.htm>

From S.J.Thompson at bham.ac.uk  Mon May 16 10:48:12 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Mon, 16 May 2016 09:48:12 +0000
Subject: [gpfsug-discuss] Flash for metadata
In-Reply-To: <49a3ab3b493a49e59177c6c25dd34d23@mbxtoa1.winmail.deshaw.com>
References: <49a3ab3b493a49e59177c6c25dd34d23@mbxtoa1.winmail.deshaw.com>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F577FC47@EX13.adf.bham.ac.uk>


Hi Paul,

Thanks, interestingly the FS-900 only does raid5. I get no choice about that, I do get a sector size choice when I create a lun on it...

I was also wondering how gpfs would do a metadata update. Would it be smart enough to know the sector size was 512b and only modify that, or would it always do a read modify write on the whole 4k inode?

Simon

-----Original Message-----
From: Sanchez, Paul [Paul.Sanchez at deshaw.com<mailto:Paul.Sanchez at deshaw.com>]
Sent: Monday, May 16, 2016 10:28 AM GMT Standard Time
To: gpfsug-discuss at spectrumscale.org
Subject: Re: [gpfsug-discuss] Flash for metadata

Simon,

I can?t speak specifically to the FS-900, since I generally use Dell Compellent and NetApp EF arrays for flash today.  But I also take care to ensure that the system pool metadata NSDs use RAID10 (or equivalent) rather than RAID5/6.  For small 4K metadata writes, on a 4K blocksize LUN this could result in optimal writes.  For larger native blocksize LUNs, it eliminates the parity calculation for read-modify-write operations.

Thanks,
Paul

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (Research Computing - IT Services)
Sent: Monday, May 16, 2016 9:44 AM
To: gpfsug-discuss at spectrumscale.org
Subject: [gpfsug-discuss] Flash for metadata

Hi all

I'm currently about to deploy some flash systems (FS-900) to take our metadata for one of our Spectrum Scale file-systems. We use 4k inodes for this file-system.

On the FS-900, when creating a volume, we get a choice of 512b or 4k sectors. Does anyone have a suggestion on this? On a spinning disk RAID array, I get what a strip size is and we'd use multiples of the stripe where possible for the block size.

Is the sector size on the FS-900 the smallest chunk that can be written out? And so would it make sense to use 4k sectors for 4k GPFS inodes right?

Related. In the deployment guide for the FS-900 arrays, if one is using a file-system (e.g. XFS) on the array the guide talks about aligning the partitions with with array, when GPFS is used on a storage array, does it automatically work out the sector alignment?


Thanks

Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160516/9dbf385b/attachment.htm>

From dacalder at co.ibm.com  Tue May 17 00:14:53 2016
From: dacalder at co.ibm.com (Danny Alexander Calderon Rodriguez)
Date: Mon, 16 May 2016 23:14:53 +0000
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 52,
	Issue 20 Flash for metadata
In-Reply-To: <mailman.3639.1463392096.1371.gpfsug-discuss@spectrumscale.org>
References: <mailman.3639.1463392096.1371.gpfsug-discuss@spectrumscale.org>
Message-ID: <201605162315.u4GNDqVo032100@mx0b-001b2d01.pphosted.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160516/14aea31a/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Image.14634403158690.gif
Type: image/gif
Size: 360 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160516/14aea31a/attachment.gif>

From Luke.Raimbach at crick.ac.uk  Tue May 17 09:03:24 2016
From: Luke.Raimbach at crick.ac.uk (Luke Raimbach)
Date: Tue, 17 May 2016 08:03:24 +0000
Subject: [gpfsug-discuss] Feature Request: Clones Inheriting XATTRS
Message-ID: <AMSPR03MB020C6EAB8A9965F03517B16B0480@AMSPR03MB020.eurprd03.prod.outlook.com>

HI All,

Where do I send feature requests?

I'd like the mmclone command to have a switch which allows user defined attributes to be passed to file clones. I can understand why the "gpfs." defined extended attributes don't get passed along, because they might hold special information like how a DMAPI agent has fiddled with the data, which won't necessarily hold true for clones.

However, what would be really useful is something like "--xattr" switch for mmclone, which allows "user." defined attributes to be inherited by clones.

We are intending to use file cloning as an internal data sharing mechanism (to reduce physical disk space usage). Some scientific groups may wish to pass along all the extended attributes (e.g. instrument type, serial number, experiment ID, etc.) to those they share data with, and others may wish to strip out those attributes.

Where shall I send this one?

Cheers,
Luke.

Luke Raimbach?
Senior HPC Data and Storage Systems Engineer,
The Francis Crick Institute,
Gibbs Building,
215 Euston Road,
London NW1 2BE.

E: luke.raimbach at crick.ac.uk
W: www.crick.ac.uk


The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no. 06885462, with its registered office at 215 Euston Road, London NW1 2BE.

From samuel.guest at noaa.gov  Tue May 17 18:31:11 2016
From: samuel.guest at noaa.gov (Sam Guest)
Date: Tue, 17 May 2016 13:31:11 -0400
Subject: [gpfsug-discuss] New UG Subscriber
Message-ID: <0ca901d1b061$e7fd2ba0$b7f782e0$@noaa.gov>

Our organization is implementing a ESS/GPFS integrated solution via RedHat 7
on Power8 and IBM DCS3700 hardware to support the Comprehensive Large
Array-data Stewardship System (CLASS) for NOAA.gov. The ESS/GPFS solution
will be replacing a legacy shared file system solution from another storage
vendor. Our office is located in the Washington, DC metro area. The CLASS
application is expected to process over 500k files and approximately 30 TB
of data per day in the near future. I am new to the ESS/GPFS environment and
have an extensive background as a AIX and UNIX systems administrator. I am
excited to have the opportunity to be a part of such a dynamic technology
and look forward to learning, sharing, and networking with the group.

Sincerely,

Sam Guest, NOAA Affiliate
Systems Engineer
CSRA, DGP Team
NESDIS/OSD/GSD/CLASS
7855 Walker Drive, Suite 200, Greenbelt, MD 20770 
Desk: 240-542-1117 Cell : 301-706-1168 Fax : 301-474-4074





From jonathan at buzzard.me.uk  Tue May 17 21:22:39 2016
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Tue, 17 May 2016 21:22:39 +0100
Subject: [gpfsug-discuss] Flash for metadata
In-Reply-To: <49a3ab3b493a49e59177c6c25dd34d23@mbxtoa1.winmail.deshaw.com>
References: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F577FBC6@EX13.adf.bham.ac.uk>
	<49a3ab3b493a49e59177c6c25dd34d23@mbxtoa1.winmail.deshaw.com>
Message-ID: <573B7D8F.7010106@buzzard.me.uk>

On 16/05/16 10:28, Sanchez, Paul wrote:
> Simon,
>
> I can?t speak specifically to the FS-900, since I generally use Dell
> Compellent and NetApp EF arrays for flash today.  But I also take care
> to ensure that the system pool metadata NSDs use RAID10 (or equivalent)
> rather than RAID5/6.  For small 4K metadata writes, on a 4K blocksize
> LUN this could result in optimal writes.  For larger native blocksize
> LUNs, it eliminates the parity calculation for read-modify-write
> operations.
>

General point would it not make more sense to make multiple RAID1's and 
set replication of the metadata than make a RAID10? Preferably with the 
RAID1's on more than one storage array?

Always gives me a warm fuzzy feeling to have my metadata replicated even 
if the data is not.

JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.


From makaplan at us.ibm.com  Wed May 18 00:26:58 2016
From: makaplan at us.ibm.com (Marc A Kaplan)
Date: Tue, 17 May 2016 19:26:58 -0400
Subject: [gpfsug-discuss] Flash for metadata
In-Reply-To: <573B7D8F.7010106@buzzard.me.uk>
References: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F577FBC6@EX13.adf.bham.ac.uk><49a3ab3b493a49e59177c6c25dd34d23@mbxtoa1.winmail.deshaw.com>
	<573B7D8F.7010106@buzzard.me.uk>
Message-ID: <201605172327.u4HNRYAv023425@d03av05.boulder.ibm.com>

I think there are two points here:

A) RAID striping is probably a "loser" for GPFS metadata. 
B) RAID mirroring for your metadata may or may not be faster and/or more 
reliable than GPFS replication. Depending on your requirements and 
assumptions for fault-tolerance one or the other might be the winner.

But .. if you're serious about performance - measure, measure, measure and 
pick the winner. (And we'd like to know what you found!)
And if you want the best performance, you'll probably need to (re)measure 
whenever you acquire new equipment.

GPFS can do and does striping for high performance when reading and 
writing metadata and data...


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160517/296937ab/attachment.htm>

From chair at spectrumscale.org  Wed May 18 19:35:44 2016
From: chair at spectrumscale.org (Spectrum scale UG Chair (Simon Thompson))
Date: Wed, 18 May 2016 19:35:44 +0100
Subject: [gpfsug-discuss] UK user group
Message-ID: <mailman.2.1653257204.13061.gpfsug-discuss_gpfsug.org@gpfsug.org>

Hi All,

Well, the UK user group is wrapped up. Most of the slides are now live on the user group site on the presentations pages. Hopefully we'll get the last few up in the next few days.

Thanks to all the user speakers, IBM speakers and those from our sponsors for supporting the event. And also to the members of the community who come along to support us, to ask questions and to challenge IBM for the future!

We had just under 30 speakers from across the globe, and a packed agenda over two days. You can get some insight on the days by checking the #SSUG16 hashtag on twitter.

Again thanks to the sponsors from Arcastream, DDN, IBM, Lenovo, Mellanox, NetApp, OCF, Seagate for supporting the event.

A final note of thanks to Oxford e-research centre for lending is their camera for the camera for two days, we have videos from many of the talks, we need to work out what to do with them, hopefully will get something online in the next few weeks! So keep an eye out!

If you came along and forget to give us your feedback forms, please drop myself and Claire an email and let us know!

And don't forget, if you missed us, there are user groups events over the next six weeks in New York, Chicago and Frankfurt!

Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160518/6959bfe4/attachment.htm>

From peter.botcherby at kcl.ac.uk  Wed May 18 19:41:57 2016
From: peter.botcherby at kcl.ac.uk (Botcherby, Peter)
Date: Wed, 18 May 2016 18:41:57 +0000
Subject: [gpfsug-discuss] UK user group
In-Reply-To: <20160518183607.9786A41C837@gpfsug.org>
References: <20160518183607.9786A41C837@gpfsug.org>
Message-ID: <VI1PR03MB1871E3D2415EC756778B0A35A4490@VI1PR03MB1871.eurprd03.prod.outlook.com>

Hi Simon,
Sorry I couldn't make it but look forward to seeing the presentations and the videos later.

All the best
Cheers
Peter
________________________________
From: Spectrum scale UG Chair (Simon Thompson)<mailto:chair at spectrumscale.org>
Sent: ?18/?05/?2016 19:36
To: gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] UK user group

Hi All,

Well, the UK user group is wrapped up. Most of the slides are now live on the user group site on the presentations pages. Hopefully we'll get the last few up in the next few days.

Thanks to all the user speakers, IBM speakers and those from our sponsors for supporting the event. And also to the members of the community who come along to support us, to ask questions and to challenge IBM for the future!

We had just under 30 speakers from across the globe, and a packed agenda over two days. You can get some insight on the days by checking the #SSUG16 hashtag on twitter.

Again thanks to the sponsors from Arcastream, DDN, IBM, Lenovo, Mellanox, NetApp, OCF, Seagate for supporting the event.

A final note of thanks to Oxford e-research centre for lending is their camera for the camera for two days, we have videos from many of the talks, we need to work out what to do with them, hopefully will get something online in the next few weeks! So keep an eye out!

If you came along and forget to give us your feedback forms, please drop myself and Claire an email and let us know!

And don't forget, if you missed us, there are user groups events over the next six weeks in New York, Chicago and Frankfurt!

Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160518/11688115/attachment.htm>

From S.J.Thompson at bham.ac.uk  Wed May 18 19:48:06 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Wed, 18 May 2016 18:48:06 +0000
Subject: [gpfsug-discuss] UK user group
In-Reply-To: <20160518183607.9786A41C837@gpfsug.org>
References: <20160518183607.9786A41C837@gpfsug.org>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F57805F4@EX13.adf.bham.ac.uk>


The presentations page is not obvious... Its under the menu that disappears....

www.spectrumscale.org/presentations/

Simon
________________________________________
From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Spectrum scale UG Chair (Simon Thompson) [chair at spectrumscale.org]
Sent: 18 May 2016 19:35
To: gpfsug-discuss at spectrumscale.org
Subject: [gpfsug-discuss] UK user group

Hi All,

Well, the UK user group is wrapped up. Most of the slides are now live on the user group site on the presentations pages. Hopefully we'll get the last few up in the next few days.

Thanks to all the user speakers, IBM speakers and those from our sponsors for supporting the event. And also to the members of the community who come along to support us, to ask questions and to challenge IBM for the future!

We had just under 30 speakers from across the globe, and a packed agenda over two days. You can get some insight on the days by checking the #SSUG16 hashtag on twitter.

Again thanks to the sponsors from Arcastream, DDN, IBM, Lenovo, Mellanox, NetApp, OCF, Seagate for supporting the event.

A final note of thanks to Oxford e-research centre for lending is their camera for the camera for two days, we have videos from many of the talks, we need to work out what to do with them, hopefully will get something online in the next few weeks! So keep an eye out!

If you came along and forget to give us your feedback forms, please drop myself and Claire an email and let us know!

And don't forget, if you missed us, there are user groups events over the next six weeks in New York, Chicago and Frankfurt!

Simon


From S.J.Thompson at bham.ac.uk  Thu May 19 16:08:12 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Thu, 19 May 2016 15:08:12 +0000
Subject: [gpfsug-discuss] UK user group
In-Reply-To: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F57805F4@EX13.adf.bham.ac.uk>
References: <20160518183607.9786A41C837@gpfsug.org>
	<CF45EE16DEF2FE4B9AA7FF2B6EE26545F57805F4@EX13.adf.bham.ac.uk>
Message-ID: <D3639532.2AAEB%s.j.thompson@bham.ac.uk>

And my take on the event ...

http://www.spectrumscale.org/uk-may-2016-group-report/

Don't forget, we're happy to take blog contributions from others as well,
events, musings on Spectrum Scale welcome ...

Simon



From S.J.Thompson at bham.ac.uk  Thu May 19 22:16:57 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Thu, 19 May 2016 21:16:57 +0000
Subject: [gpfsug-discuss] Flash for metadata
In-Reply-To: <201605172327.u4HNRYAv023425@d03av05.boulder.ibm.com>
References: <201605172327.u4HNRYAv023425@d03av05.boulder.ibm.com>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F578235F@EX13.adf.bham.ac.uk>

Thanks to all those who replied (including off list).

Someone reminded me that whilst the FlashSystem will support 512 and 4k blocks, it "prefers" 4k. Maybe there is a performance penalty for the smaller writes.

Second, a flash update is always going to be read erase write.

We don't get any choice with FlashSystem 900 on the RAID set - only RAID5 is available.

Given we have two FlashSystems in two data centres, looks like we are going with 4k sectors and GPFS replication.

I assume that creating several smaller LUNs on each FlashSystem in the same failure group is still preferable to one big LUN so we get more IO queues to play with?

Simon

-----Original Message-----
From: Marc A Kaplan [makaplan at us.ibm.com<mailto:makaplan at us.ibm.com>]
Sent: Wednesday, May 18, 2016 12:27 AM GMT Standard Time
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Flash for metadata

I think there are two points here:

A) RAID striping is probably a "loser" for GPFS metadata.
B) RAID mirroring for your metadata may or may not be faster and/or more reliable than GPFS replication. Depending on your requirements and assumptions for fault-tolerance one or the other might be the winner.

But .. if you're serious about performance - measure, measure, measure and pick the winner. (And we'd like to know what you found!)
And if you want the best performance, you'll probably need to (re)measure whenever you acquire new equipment.

GPFS can do and does striping for high performance when reading and writing metadata and data...

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160519/fdec78f4/attachment.htm>

From makaplan at us.ibm.com  Thu May 19 22:59:32 2016
From: makaplan at us.ibm.com (Marc A Kaplan)
Date: Thu, 19 May 2016 17:59:32 -0400
Subject: [gpfsug-discuss] Flash for metadata
In-Reply-To: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F578235F@EX13.adf.bham.ac.uk>
References: <201605172327.u4HNRYAv023425@d03av05.boulder.ibm.com>
	<CF45EE16DEF2FE4B9AA7FF2B6EE26545F578235F@EX13.adf.bham.ac.uk>
Message-ID: <201605192159.u4JLxZox006181@d03av05.boulder.ibm.com>

"I assume that creating several smaller LUNs on each FlashSystem in the 
same failure group is still preferable to one big LUN so we get more IO 
queues to play with?"

Traditionally, more spindles, more disk arms working in parallel => better 
overall performance.

HOWEVER Flash doesn't work that way... So it's going to depend... 
Perhaps some kind soul can point us to some information about this and how 
much it varies among today's flash based storage products.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160519/939c1ca6/attachment.htm>

From RWelp at uk.ibm.com  Fri May 20 20:47:15 2016
From: RWelp at uk.ibm.com (Richard Welp)
Date: Fri, 20 May 2016 20:47:15 +0100
Subject: [gpfsug-discuss] Flash for metadata
In-Reply-To: <201605192159.u4JLxZox006181@d03av05.boulder.ibm.com>
References: <201605172327.u4HNRYAv023425@d03av05.boulder.ibm.com>	<CF45EE16DEF2FE4B9AA7FF2B6EE26545F578235F@EX13.adf.bham.ac.uk>
	<201605192159.u4JLxZox006181@d03av05.boulder.ibm.com>
Message-ID: <201605201947.u4KJlKJT000383@d06av06.portsmouth.uk.ibm.com>

I checked with the FS900 performance expert in Tucson, and here is what I 
was told:

The 4KB and 512B blocks will both get equally great sub millisecond 
response times but 4KB can achieve a higher maximum IOPS rate.

As far as # of luns,  it doesn't really matter to the FS900, but the host 
operating system and other components in the data path can benefit from 
having more than 1 large lun.  If we are trying to get the maximum 
possible iops, we typically run with at least 16 luns.    I suspect with 4 
luns you would get within 10% of the maximum performance. 

Thanks,

Rick

===================
Rick Welp
Software Engineer
Master Inventor
Email: rwelp at uk.ibm.com
phone: +44 0161 214 0461 
IBM Systems - Manchester Lab
IBM UK Limited 
--------------------------




From:   "Marc A Kaplan" <makaplan at us.ibm.com>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   19/05/2016 11:00 pm
Subject:        Re: [gpfsug-discuss] Flash for metadata
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



"I assume that creating several smaller LUNs on each FlashSystem in the 
same failure group is still preferable to one big LUN so we get more IO 
queues to play with?"

Traditionally, more spindles, more disk arms working in parallel => better 
overall performance.

HOWEVER Flash doesn't work that way... So it's going to depend... 
Perhaps some kind soul can point us to some information about this and how 
much it varies among today's flash based storage products.
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160520/5f7d4126/attachment.htm>

From S.J.Thompson at bham.ac.uk  Fri May 20 22:15:57 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Fri, 20 May 2016 21:15:57 +0000
Subject: [gpfsug-discuss] Flash for metadata
In-Reply-To: <201605201947.u4KJlKJT000383@d06av06.portsmouth.uk.ibm.com>
References: <201605172327.u4HNRYAv023425@d03av05.boulder.ibm.com>
	<CF45EE16DEF2FE4B9AA7FF2B6EE26545F578235F@EX13.adf.bham.ac.uk>
	<201605192159.u4JLxZox006181@d03av05.boulder.ibm.com>,
	<201605201947.u4KJlKJT000383@d06av06.portsmouth.uk.ibm.com>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F5782A0D@EX13.adf.bham.ac.uk>


Thanks Rick for checking this.

On number of LUNs, I vaguely had this in the back of my head, when we deployed our first Storwise, that was one of the reasons we built lots of metadata mirrored pairs rather than bigger arrays.

Thinking back I remember reading it was something to do with multipath and how IO queues are processed back to the storage device. The storwise we have is dual active, so only one half "owns" the LUN, so I recall it was optimal to encourage gpfs to load over controllers by having more multipath accessible devices.

And then thinking how you cable up to you LSI fc adapters as well....

Simon

________________________________________
From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Richard Welp [RWelp at uk.ibm.com]
Sent: 20 May 2016 20:47
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Flash for metadata

I checked with the FS900 performance expert in Tucson, and here is what I was told:

The 4KB and 512B blocks will both get equally great sub millisecond response times but 4KB can achieve a higher maximum IOPS rate.

As far as # of luns,  it doesn't really matter to the FS900, but the host operating system and other components in the data path can benefit from having more than 1 large lun.  If we are trying to get the maximum possible iops, we typically run with at least 16 luns.    I suspect with 4 luns you would get within 10% of the maximum performance.

Thanks,

Rick

===================
Rick Welp
Software Engineer
Master Inventor
Email: rwelp at uk.ibm.com
phone: +44 0161 214 0461
IBM Systems - Manchester Lab
IBM UK Limited
--------------------------




From:        "Marc A Kaplan" <makaplan at us.ibm.com>
To:        gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:        19/05/2016 11:00 pm
Subject:        Re: [gpfsug-discuss] Flash for metadata
Sent by:        gpfsug-discuss-bounces at spectrumscale.org
________________________________



"I assume that creating several smaller LUNs on each FlashSystem in the same failure group is still preferable to one big LUN so we get more IO queues to play with?"

Traditionally, more spindles, more disk arms working in parallel => better overall performance.

HOWEVER Flash doesn't work that way... So it's going to depend...
Perhaps some kind soul can point us to some information about this and how much it varies among today's flash based storage products.
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss




From usa-principal at gpfsug.org  Mon May 23 12:31:39 2016
From: usa-principal at gpfsug.org (usa-principal-gpfsug.org)
Date: Mon, 23 May 2016 07:31:39 -0400
Subject: [gpfsug-discuss] Speakers Wanted - June 10, Argonne National Lab
Message-ID: <1fa21d083401a0a7acff84b7cd69ee34@webmail.gpfsug.org>

Hello all,

	We?d love to have 2 more site reports for the June 10th event at 
Argonne. As a reminder, the event details are here: 
http://www.spectrumscale.org/spectrum-scale-user-group-usa-meeting-june-10th/


	Please reply and let us know what you'd like to present. It's always 
good to hear real experiences.

	Thanks in advance for your participation.

Best,
Kristy



From p.childs at qmul.ac.uk  Tue May 24 15:00:46 2016
From: p.childs at qmul.ac.uk (Peter Childs)
Date: Tue, 24 May 2016 14:00:46 +0000
Subject: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
In-Reply-To: <5744538D.3030407@qmul.ac.uk>
References: <AM2PR07MB0948D12D638C1ABD1AD603E3A44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>,
	<5744538D.3030407@qmul.ac.uk>
Message-ID: <AM2PR07MB09480DBC9D614EE7B5826AAAA44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>

 Hi All,

We have an issue where the Linux kills off GPFS first when a computer runs out of memory, this happens when user processors have exhausted memory and swap and the out of memory killer in Linux kills the GPFS daemon as the largest user of memory, due to its large pinned memory foot print.

We have an issue where the Linux kills off GPFS first when a computer runs out of memory. We are running GPFS 3.5

We believe this happens when user processes have exhausted memory and swap and the out of memory killer in Linux chooses to  kill the GPFS daemon as the largest user of memory, due to its large pinned memory footprint.

This means that GPFS is killed and the whole cluster blocks for a minute before it resumes operation, this is not ideal, and kills and causes issues with most of the cluster.

What we see is users unable to login elsewhere on the cluster until we have powered off the node. We believe this is because while the node is still pingable, GPFS doesn't expel it from the cluster.

This issue mainly occurs on our frontend nodes of our HPC cluster but can effect the rest of the cluster when it occurs.

This issue mainly occurs on the login nodes of our HPC cluster but can affect the rest of the cluster when it occurs.

I've seen others on list with this issue.

We've come up with a solution where by the gpfs is adjusted so that is unlikely to be the first thing to be killed, and hopefully the user process is killed and not GPFS.

We've come up with a solution to adjust the OOM score of GPFS, so that it is unlikely to be the first thing to be killed, and hopefully the OOM killer picks a user process instead.

Out testing says this solution works, but I'm asking here firstly to share our knowledge and secondly to ask if there is anything we've missed with this solution and issues with this.

We've tested this and it seems to work. I'm asking here firstly to share our knowledge and secondly to ask if there is anything we've missed with this solution.

Its short which is part of its beauty.

/usr/local/sbin/gpfs-oom_score_adj

<pre>
#!/bin/bash

 for proc in $(pgrep mmfs); do
      echo -500 >/proc/$proc/oom_score_adj
 done
</pre>

This can then be called automatically on GPFS startup with the following:

<pre>
mmaddcallback startupoomkiller --command /usr/local/sbin/gpfs-oom_score_adj --event startup
</pre>

and either restart gpfs or just run the script on all nodes.

Peter Childs
ITS Research Infrastructure
Queen Mary, University of London

From Paul.Sanchez at deshaw.com  Tue May 24 15:32:55 2016
From: Paul.Sanchez at deshaw.com (Sanchez, Paul)
Date: Tue, 24 May 2016 14:32:55 +0000
Subject: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
In-Reply-To: <AM2PR07MB09480DBC9D614EE7B5826AAAA44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>
References: <AM2PR07MB0948D12D638C1ABD1AD603E3A44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>,
	<5744538D.3030407@qmul.ac.uk>
	<AM2PR07MB09480DBC9D614EE7B5826AAAA44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>
Message-ID: <d6028bcf19f8402b9315ab875d075ba4@mbxtoa1.winmail.deshaw.com>

Hi Peter,



This is mentioned explicitly in the Spectrum Scale docs (http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.pdg.doc/bl1pdg_kerncfg.htm?lang=en) as a problem for the admin to consider, and many of us have been bitten by this.  There are references going back at least to GPFS 3.1 in 2008 on developerworks complaining about this situation.



While the answer you described below is essentially what we do as well, I would argue that this is a problem which IBM should just own and fix for everyone. I cannot think of a situation in which you would want GPFS to be sacrificed on a node due to out-of-memory conditions, and I have seen several terrible consequences of this, including loss of cached, user-acknowledged writes.



I don't think there are any real gotchas.  But in addition, our own implementation also:



  * uses "--event preStartup" instead of "startup", since it runs earlier and reduces the risk of a race



  * reads the score back out and complains if it hasn't been set



  * includes "set -e" to ensure that errors will terminate the script and return a non-zero exit code to the callback parent



Thx
Paul



-----Original Message-----
From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Peter Childs
Sent: Tuesday, May 24, 2016 10:01 AM
To: gpfsug main discussion list
Subject: [gpfsug-discuss] OOM Killer killing off GPFS 3.5



Hi All,



We have an issue where the Linux kills off GPFS first when a computer runs out of memory, this happens when user processors have exhausted memory and swap and the out of memory killer in Linux kills the GPFS daemon as the largest user of memory, due to its large pinned memory foot print.



We have an issue where the Linux kills off GPFS first when a computer runs out of memory. We are running GPFS 3.5



We believe this happens when user processes have exhausted memory and swap and the out of memory killer in Linux chooses to  kill the GPFS daemon as the largest user of memory, due to its large pinned memory footprint.



This means that GPFS is killed and the whole cluster blocks for a minute before it resumes operation, this is not ideal, and kills and causes issues with most of the cluster.



What we see is users unable to login elsewhere on the cluster until we have powered off the node. We believe this is because while the node is still pingable, GPFS doesn't expel it from the cluster.



This issue mainly occurs on our frontend nodes of our HPC cluster but can effect the rest of the cluster when it occurs.



This issue mainly occurs on the login nodes of our HPC cluster but can affect the rest of the cluster when it occurs.



I've seen others on list with this issue.



We've come up with a solution where by the gpfs is adjusted so that is unlikely to be the first thing to be killed, and hopefully the user process is killed and not GPFS.



We've come up with a solution to adjust the OOM score of GPFS, so that it is unlikely to be the first thing to be killed, and hopefully the OOM killer picks a user process instead.



Out testing says this solution works, but I'm asking here firstly to share our knowledge and secondly to ask if there is anything we've missed with this solution and issues with this.



We've tested this and it seems to work. I'm asking here firstly to share our knowledge and secondly to ask if there is anything we've missed with this solution.



Its short which is part of its beauty.



/usr/local/sbin/gpfs-oom_score_adj



<pre>

#!/bin/bash



for proc in $(pgrep mmfs); do

      echo -500 >/proc/$proc/oom_score_adj  done </pre>



This can then be called automatically on GPFS startup with the following:



<pre>

mmaddcallback startupoomkiller --command /usr/local/sbin/gpfs-oom_score_adj --event startup </pre>



and either restart gpfs or just run the script on all nodes.



Peter Childs

ITS Research Infrastructure

Queen Mary, University of London

_______________________________________________

gpfsug-discuss mailing list

gpfsug-discuss at spectrumscale.org

http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160524/e0679f0a/attachment.htm>

From S.J.Thompson at bham.ac.uk  Tue May 24 15:33:07 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Tue, 24 May 2016 14:33:07 +0000
Subject: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
Message-ID: <D36A2390.2AD83%s.j.thompson@bham.ac.uk>

Hi Peter,

We used to see exactly this problem as well. We tweaked the init script on
our systems to adjust the OOM killer (we also tweaked it to wait for IB
links to come up as well). Its something we need to revisit now we are
running systemd based systems ...

Simon

On 24/05/2016, 15:00, "gpfsug-discuss-bounces at spectrumscale.org on behalf
of Peter Childs" <gpfsug-discuss-bounces at spectrumscale.org on behalf of
p.childs at qmul.ac.uk> wrote:

> Hi All,
>
>We have an issue where the Linux kills off GPFS first when a computer
>runs out of memory, this happens when user processors have exhausted
>memory and swap and the out of memory killer in Linux kills the GPFS
>daemon as the largest user of memory, due to its large pinned memory foot
>print.
>
>We have an issue where the Linux kills off GPFS first when a computer
>runs out of memory. We are running GPFS 3.5
>
>We believe this happens when user processes have exhausted memory and
>swap and the out of memory killer in Linux chooses to  kill the GPFS
>daemon as the largest user of memory, due to its large pinned memory
>footprint.
>
>This means that GPFS is killed and the whole cluster blocks for a minute
>before it resumes operation, this is not ideal, and kills and causes
>issues with most of the cluster.
>
>What we see is users unable to login elsewhere on the cluster until we
>have powered off the node. We believe this is because while the node is
>still pingable, GPFS doesn't expel it from the cluster.
>
>This issue mainly occurs on our frontend nodes of our HPC cluster but can
>effect the rest of the cluster when it occurs.
>
>This issue mainly occurs on the login nodes of our HPC cluster but can
>affect the rest of the cluster when it occurs.
>
>I've seen others on list with this issue.
>
>We've come up with a solution where by the gpfs is adjusted so that is
>unlikely to be the first thing to be killed, and hopefully the user
>process is killed and not GPFS.
>
>We've come up with a solution to adjust the OOM score of GPFS, so that it
>is unlikely to be the first thing to be killed, and hopefully the OOM
>killer picks a user process instead.
>
>Out testing says this solution works, but I'm asking here firstly to
>share our knowledge and secondly to ask if there is anything we've missed
>with this solution and issues with this.
>
>We've tested this and it seems to work. I'm asking here firstly to share
>our knowledge and secondly to ask if there is anything we've missed with
>this solution.
>
>Its short which is part of its beauty.
>
>/usr/local/sbin/gpfs-oom_score_adj
>
><pre>
>#!/bin/bash
>
> for proc in $(pgrep mmfs); do
>      echo -500 >/proc/$proc/oom_score_adj
> done
></pre>
>
>This can then be called automatically on GPFS startup with the following:
>
><pre>
>mmaddcallback startupoomkiller --command
>/usr/local/sbin/gpfs-oom_score_adj --event startup
></pre>
>
>and either restart gpfs or just run the script on all nodes.
>
>Peter Childs
>ITS Research Infrastructure
>Queen Mary, University of London
>_______________________________________________
>gpfsug-discuss mailing list
>gpfsug-discuss at spectrumscale.org
>http://gpfsug.org/mailman/listinfo/gpfsug-discuss



From volobuev at us.ibm.com  Tue May 24 17:17:12 2016
From: volobuev at us.ibm.com (Yuri L Volobuev)
Date: Tue, 24 May 2016 09:17:12 -0700
Subject: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
In-Reply-To: <d6028bcf19f8402b9315ab875d075ba4@mbxtoa1.winmail.deshaw.com>
References: <AM2PR07MB0948D12D638C1ABD1AD603E3A44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>,
	<5744538D.3030407@qmul.ac.uk><AM2PR07MB09480DBC9D614EE7B5826AAAA44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>
	<d6028bcf19f8402b9315ab875d075ba4@mbxtoa1.winmail.deshaw.com>
Message-ID: <20160524161719.AA629AC03F@b01ledav006.gho.pok.ibm.com>


This problem is more complex than it may seem.  The thing is, mmfsd runs as
root, as thus already possesses a certain amount of natural immunity to OOM
killer.  So adjusting mmfsd oom_score_adj doesn't radically change the
ranking of OOM killer victims, only tweaks it.  The way things are supposed
to work is: a user process eats up a lot of memory, and once a threshold is
hit, OOM killer picks off the memory hog, and the memory is released.
Unprivileged processes inherently have a higher OOM score, and should be
killed off first.  If that doesn't work, for some reason, the OOM killer
gets desperate and starts going after root processes.  Once things get to
this point, it's tough.  If you somehow manage to spare mmfsd per se,
what's going to happen next?  The OOM killer still needs a victim.  What
we've seen happen in such a situation is semi-random privileged process
killing.  mmfsd stays alive, but various other system processes are picked
off, and pretty quickly the node is a basket case.  A Linux node is not
very resilient to random process killing.  And it doesn't help that those
other privileged processes usually don't use much memory, so killing them
doesn't release much, and the carnage keeps on going.  The real problem is:
why wasn't the non-privileged memory hog process killed off first, before
root processes became fair game?  This is where things get pretty
complicated, and depend heavily on the Linux version.  There's one specific
issue that did get diagnosed.  If a process is using mmap and has page
faults going that result in GPFS IO, on older versions of GPFS the process
would fail to error out after a SIGKILL, due to locking complications
spanning Linux kernel VMM and GPFS mmap code.  This means the OOM killer
would attempt to kill a process, but that wouldn't produce the desired
result (the process is still around), and the OOM killer keeps moving down
the list.  This problem has been fixed in the current GPFS service levels.
It is possible that a similar problem may exist that prevents a memory hog
process from erroring out.  I strongly encourage opening a PMR to
investigate such a situation, instead of trying to work around it without
understanding why mmfsd was targeted in the first place.

This is the case of prevention being the best cure.  Where we've seen
success is customers using cgroups to prevent user processes from running a
node out of memory in the first place.  This has been shown to work well.
Dealing with the fallout from running out of memory is a much harder task.

The post-mmfsd-kill symptoms that are described in the original note are
not normal.  If an mmfsd process is killed, other nodes will become aware
of this fact faily quickly, and the node is going to be expelled from the
cluster (yes, expels *can* be a good thing).  In the normal case, TCP/IP
sockets are closed as soon as mmfsd is killed, and other nodes immediately
receive TCP RST packets, and close their connection endpoints.  If the
worst case, if a node just becomes catatonic, but RST is not sent out, the
troubled node is going to be expelled from the cluster after about 2
minutes of pinging (in a default configuration).  There should definitely
not be a permanent hang that necessitates a manual intervention.  Again,
older versions of GPFS had no protection against surprise OOM thread kills,
but in the current code some self-monitoring capabilities have been added,
and a single troubled node won't have a lasting impact on the cluster.  If
you aren't running with a reasonably current level of GPFS 3.5 service, I
strongly recommend upgrading.  If you see the symptoms originally described
with the current code, that's a bug that we need to fix, so please open a
PMR to address the issue.

yuri



From:	"Sanchez, Paul" <Paul.Sanchez at deshaw.com>
To:	gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>,
Date:	05/24/2016 07:33 AM
Subject:	Re: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



Hi Peter,

This is mentioned explicitly in the Spectrum Scale docs (
http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.pdg.doc/bl1pdg_kerncfg.htm?lang=en
) as a problem for the admin to consider, and many of us have been bitten
by this.  There are references going back at least to GPFS 3.1 in 2008 on
developerworks complaining about this situation.

While the answer you described below is essentially what we do as well, I
would argue that this is a problem which IBM should just own and fix for
everyone. I cannot think of a situation in which you would want GPFS to be
sacrificed on a node due to out-of-memory conditions, and I have seen
several terrible consequences of this, including loss of cached,
user-acknowledged writes.

I don't think there are any real gotchas.  But in addition, our own
implementation also:

  * uses "--event preStartup" instead of "startup", since it runs earlier
and reduces the risk of a race

  * reads the score back out and complains if it hasn't been set

  * includes "set -e" to ensure that errors will terminate the script and
return a non-zero exit code to the callback parent

Thx
Paul

-----Original Message-----
From: gpfsug-discuss-bounces at spectrumscale.org [
mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Peter Childs
Sent: Tuesday, May 24, 2016 10:01 AM
To: gpfsug main discussion list
Subject: [gpfsug-discuss] OOM Killer killing off GPFS 3.5

Hi All,

We have an issue where the Linux kills off GPFS first when a computer runs
out of memory, this happens when user processors have exhausted memory and
swap and the out of memory killer in Linux kills the GPFS daemon as the
largest user of memory, due to its large pinned memory foot print.

We have an issue where the Linux kills off GPFS first when a computer runs
out of memory. We are running GPFS 3.5

We believe this happens when user processes have exhausted memory and swap
and the out of memory killer in Linux chooses to  kill the GPFS daemon as
the largest user of memory, due to its large pinned memory footprint.

This means that GPFS is killed and the whole cluster blocks for a minute
before it resumes operation, this is not ideal, and kills and causes issues
with most of the cluster.

What we see is users unable to login elsewhere on the cluster until we have
powered off the node. We believe this is because while the node is still
pingable, GPFS doesn't expel it from the cluster.

This issue mainly occurs on our frontend nodes of our HPC cluster but can
effect the rest of the cluster when it occurs.

This issue mainly occurs on the login nodes of our HPC cluster but can
affect the rest of the cluster when it occurs.

I've seen others on list with this issue.

We've come up with a solution where by the gpfs is adjusted so that is
unlikely to be the first thing to be killed, and hopefully the user process
is killed and not GPFS.

We've come up with a solution to adjust the OOM score of GPFS, so that it
is unlikely to be the first thing to be killed, and hopefully the OOM
killer picks a user process instead.

Out testing says this solution works, but I'm asking here firstly to share
our knowledge and secondly to ask if there is anything we've missed with
this solution and issues with this.

We've tested this and it seems to work. I'm asking here firstly to share
our knowledge and secondly to ask if there is anything we've missed with
this solution.

Its short which is part of its beauty.

/usr/local/sbin/gpfs-oom_score_adj

<pre>
#!/bin/bash

for proc in $(pgrep mmfs); do
      echo -500 >/proc/$proc/oom_score_adj  done </pre>

This can then be called automatically on GPFS startup with the following:

<pre>
mmaddcallback startupoomkiller --command /usr/local/sbin/gpfs-oom_score_adj
--event startup </pre>

and either restart gpfs or just run the script on all nodes.

Peter Childs
ITS Research Infrastructure
Queen Mary, University of London
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160524/43490dfb/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160524/43490dfb/attachment.gif>

From volobuev at us.ibm.com  Tue May 24 20:25:06 2016
From: volobuev at us.ibm.com (Yuri L Volobuev)
Date: Tue, 24 May 2016 12:25:06 -0700
Subject: [gpfsug-discuss] Flash for metadata
In-Reply-To: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F577FC47@EX13.adf.bham.ac.uk>
References: <49a3ab3b493a49e59177c6c25dd34d23@mbxtoa1.winmail.deshaw.com>
	<CF45EE16DEF2FE4B9AA7FF2B6EE26545F577FC47@EX13.adf.bham.ac.uk>
Message-ID: <20160524192511.E6BFD6E041@b03ledav001.gho.boulder.ibm.com>

GPFS will issue 4K IOs if the file system is 4K-aligned, which is the
default for 4.1.1+, and may issue requests as small as 512 bytes otherwise.
If a given array presents 4K physical block size to the world, you won't be
able to format a non-4K-aligned file system which puts metadata on that
disk.  It is generally believed that 4K IO size produces optimal
performance with flash-based storage in general, because this is the native
physical block size, and 512-byte sector emulation entails doing a
read-modify-write.  We don't have the data on how much difference the
sector size actually makes.  If in doubt, I would recommend going with 4K
sectors and a 4K-aligned file system.

yuri



From:	"Simon Thompson (Research Computing - IT Services)"
            <S.J.Thompson at bham.ac.uk>
To:	"'gpfsug-discuss at spectrumscale.org'"
            <gpfsug-discuss at spectrumscale.org>,
Date:	05/16/2016 02:48 AM
Subject:	Re: [gpfsug-discuss] Flash for metadata
Sent by:	gpfsug-discuss-bounces at spectrumscale.org




Hi Paul,

Thanks, interestingly the FS-900 only does raid5. I get no choice about
that, I do get a sector size choice when I create a lun on it...

I was also wondering how gpfs would do a metadata update. Would it be smart
enough to know the sector size was 512b and only modify that, or would it
always do a read modify write on the whole 4k inode?

Simon

-----Original Message-----
From: Sanchez, Paul [Paul.Sanchez at deshaw.com]
Sent: Monday, May 16, 2016 10:28 AM GMT Standard Time
To: gpfsug-discuss at spectrumscale.org
Subject: Re: [gpfsug-discuss] Flash for metadata

Simon,

I can?t speak specifically to the FS-900, since I generally use Dell
Compellent and NetApp EF arrays for flash today.  But I also take care to
ensure that the system pool metadata NSDs use RAID10 (or equivalent) rather
than RAID5/6.  For small 4K metadata writes, on a 4K blocksize LUN this
could result in optimal writes.  For larger native blocksize LUNs, it
eliminates the parity calculation for read-modify-write operations.

Thanks,
Paul

From: gpfsug-discuss-bounces at spectrumscale.org [
mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon
Thompson (Research Computing - IT Services)
Sent: Monday, May 16, 2016 9:44 AM
To: gpfsug-discuss at spectrumscale.org
Subject: [gpfsug-discuss] Flash for metadata

Hi all

I'm currently about to deploy some flash systems (FS-900) to take our
metadata for one of our Spectrum Scale file-systems. We use 4k inodes for
this file-system.

On the FS-900, when creating a volume, we get a choice of 512b or 4k
sectors. Does anyone have a suggestion on this? On a spinning disk RAID
array, I get what a strip size is and we'd use multiples of the stripe
where possible for the block size.

Is the sector size on the FS-900 the smallest chunk that can be written
out? And so would it make sense to use 4k sectors for 4k GPFS inodes right?

Related. In the deployment guide for the FS-900 arrays, if one is using a
file-system (e.g. XFS) on the array the guide talks about aligning the
partitions with with array, when GPFS is used on a storage array, does it
automatically work out the sector alignment?


Thanks

Simon_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160524/3a332e95/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160524/3a332e95/attachment.gif>

From Paul.Sanchez at deshaw.com  Wed May 25 15:00:35 2016
From: Paul.Sanchez at deshaw.com (Sanchez, Paul)
Date: Wed, 25 May 2016 14:00:35 +0000
Subject: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
In-Reply-To: <20160524161719.AA629AC03F@b01ledav006.gho.pok.ibm.com>
References: <AM2PR07MB0948D12D638C1ABD1AD603E3A44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>,
	<5744538D.3030407@qmul.ac.uk><AM2PR07MB09480DBC9D614EE7B5826AAAA44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>
	<d6028bcf19f8402b9315ab875d075ba4@mbxtoa1.winmail.deshaw.com>,
	<20160524161719.AA629AC03F@b01ledav006.gho.pok.ibm.com>
Message-ID: <51fa215cfdc149c6b9c6cdeb50322d88@mbxtoa1.winmail.deshaw.com>

I'm sure that Yuri is right about the corner-case complexity across all linux and Spectrum/GPFS versions.

In situations where lots of outstanding tokens exist, and there are few token managers, we have seen the assassination of a large footprint mmfsd in GPFS 4.1 seem to impact entire clusters, potentially due to serialization in recovery of so many tokens, and overlapping access among nodes. We're looking forward to fixes in 4.2.1 to address some of this too.

But for what it's worth, on RH6/7 with 4.1, we have seen the end of OOM impacting GPFS since implementing the callback. One item I forgot is that we don't set it to -500, but to OOM_SCORE_ADJ_MIN, which on our systems is -1000. That causes the heuristic oom_badness to return the lowest possible score, more thoroughly immunizing it against selection.

Thx
Paul

Sent with Good Work (www.good.com)


From: Yuri L Volobuev <volobuev at us.ibm.com<mailto:volobuev at us.ibm.com>>
Date: Tuesday, May 24, 2016, 12:17 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] OOM Killer killing off GPFS 3.5


This problem is more complex than it may seem. The thing is, mmfsd runs as root, as thus already possesses a certain amount of natural immunity to OOM killer. So adjusting mmfsd oom_score_adj doesn't radically change the ranking of OOM killer victims, only tweaks it. The way things are supposed to work is: a user process eats up a lot of memory, and once a threshold is hit, OOM killer picks off the memory hog, and the memory is released. Unprivileged processes inherently have a higher OOM score, and should be killed off first. If that doesn't work, for some reason, the OOM killer gets desperate and starts going after root processes. Once things get to this point, it's tough. If you somehow manage to spare mmfsd per se, what's going to happen next? The OOM killer still needs a victim. What we've seen happen in such a situation is semi-random privileged process killing. mmfsd stays alive, but various other system processes are picked off, and pretty quickly the node is a basket case. A Linux node is not very resilient to random process killing. And it doesn't help that those other privileged processes usually don't use much memory, so killing them doesn't release much, and the carnage keeps on going. The real problem is: why wasn't the non-privileged memory hog process killed off first, before root processes became fair game? This is where things get pretty complicated, and depend heavily on the Linux version. There's one specific issue that did get diagnosed. If a process is using mmap and has page faults going that result in GPFS IO, on older versions of GPFS the process would fail to error out after a SIGKILL, due to locking complications spanning Linux kernel VMM and GPFS mmap code. This means the OOM killer would attempt to kill a process, but that wouldn't produce the desired result (the process is still around), and the OOM killer keeps moving down the list. This problem has been fixed in the current GPFS service levels. It is possible that a similar problem may exist that prevents a memory hog process from erroring out. I strongly encourage opening a PMR to investigate such a situation, instead of trying to work around it without understanding why mmfsd was targeted in the first place.

This is the case of prevention being the best cure. Where we've seen success is customers using cgroups to prevent user processes from running a node out of memory in the first place. This has been shown to work well. Dealing with the fallout from running out of memory is a much harder task.

The post-mmfsd-kill symptoms that are described in the original note are not normal. If an mmfsd process is killed, other nodes will become aware of this fact faily quickly, and the node is going to be expelled from the cluster (yes, expels *can* be a good thing). In the normal case, TCP/IP sockets are closed as soon as mmfsd is killed, and other nodes immediately receive TCP RST packets, and close their connection endpoints. If the worst case, if a node just becomes catatonic, but RST is not sent out, the troubled node is going to be expelled from the cluster after about 2 minutes of pinging (in a default configuration). There should definitely not be a permanent hang that necessitates a manual intervention. Again, older versions of GPFS had no protection against surprise OOM thread kills, but in the current code some self-monitoring capabilities have been added, and a single troubled node won't have a lasting impact on the cluster. If you aren't running with a reasonably current level of GPFS 3.5 service, I strongly recommend upgrading. If you see the symptoms originally described with the current code, that's a bug that we need to fix, so please open a PMR to address the issue.

yuri

[Inactive hide details for "Sanchez, Paul" ---05/24/2016 07:33:18 AM---Hi Peter, This is mentioned explicitly in the Spectrum Sc]"Sanchez, Paul" ---05/24/2016 07:33:18 AM---Hi Peter, This is mentioned explicitly in the Spectrum Scale docs (http://www.ibm.com/support/knowle

From: "Sanchez, Paul" <Paul.Sanchez at deshaw.com>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>,
Date: 05/24/2016 07:33 AM
Subject: Re: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
Sent by: gpfsug-discuss-bounces at spectrumscale.org

________________________________



Hi Peter,

This is mentioned explicitly in the Spectrum Scale docs (http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.pdg.doc/bl1pdg_kerncfg.htm?lang=en) as a problem for the admin to consider, and many of us have been bitten by this. There are references going back at least to GPFS 3.1 in 2008 on developerworks complaining about this situation.

While the answer you described below is essentially what we do as well, I would argue that this is a problem which IBM should just own and fix for everyone. I cannot think of a situation in which you would want GPFS to be sacrificed on a node due to out-of-memory conditions, and I have seen several terrible consequences of this, including loss of cached, user-acknowledged writes.

I don't think there are any real gotchas. But in addition, our own implementation also:

* uses "--event preStartup" instead of "startup", since it runs earlier and reduces the risk of a race

* reads the score back out and complains if it hasn't been set

* includes "set -e" to ensure that errors will terminate the script and return a non-zero exit code to the callback parent

Thx
Paul

-----Original Message-----
From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Peter Childs
Sent: Tuesday, May 24, 2016 10:01 AM
To: gpfsug main discussion list
Subject: [gpfsug-discuss] OOM Killer killing off GPFS 3.5

Hi All,

We have an issue where the Linux kills off GPFS first when a computer runs out of memory, this happens when user processors have exhausted memory and swap and the out of memory killer in Linux kills the GPFS daemon as the largest user of memory, due to its large pinned memory foot print.

We have an issue where the Linux kills off GPFS first when a computer runs out of memory. We are running GPFS 3.5

We believe this happens when user processes have exhausted memory and swap and the out of memory killer in Linux chooses to kill the GPFS daemon as the largest user of memory, due to its large pinned memory footprint.

This means that GPFS is killed and the whole cluster blocks for a minute before it resumes operation, this is not ideal, and kills and causes issues with most of the cluster.

What we see is users unable to login elsewhere on the cluster until we have powered off the node. We believe this is because while the node is still pingable, GPFS doesn't expel it from the cluster.

This issue mainly occurs on our frontend nodes of our HPC cluster but can effect the rest of the cluster when it occurs.

This issue mainly occurs on the login nodes of our HPC cluster but can affect the rest of the cluster when it occurs.

I've seen others on list with this issue.

We've come up with a solution where by the gpfs is adjusted so that is unlikely to be the first thing to be killed, and hopefully the user process is killed and not GPFS.

We've come up with a solution to adjust the OOM score of GPFS, so that it is unlikely to be the first thing to be killed, and hopefully the OOM killer picks a user process instead.

Out testing says this solution works, but I'm asking here firstly to share our knowledge and secondly to ask if there is anything we've missed with this solution and issues with this.

We've tested this and it seems to work. I'm asking here firstly to share our knowledge and secondly to ask if there is anything we've missed with this solution.

Its short which is part of its beauty.

/usr/local/sbin/gpfs-oom_score_adj

<pre>
#!/bin/bash

for proc in $(pgrep mmfs); do
echo -500 >/proc/$proc/oom_score_adj done </pre>

This can then be called automatically on GPFS startup with the following:

<pre>
mmaddcallback startupoomkiller --command /usr/local/sbin/gpfs-oom_score_adj --event startup </pre>

and either restart gpfs or just run the script on all nodes.

Peter Childs
ITS Research Infrastructure
Queen Mary, University of London
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160525/fed71308/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: graycol.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160525/fed71308/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: graycol.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160525/fed71308/attachment-0001.gif>

From bdeluca at gmail.com  Wed May 25 15:09:06 2016
From: bdeluca at gmail.com (Ben De Luca)
Date: Wed, 25 May 2016 17:09:06 +0300
Subject: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
In-Reply-To: <51fa215cfdc149c6b9c6cdeb50322d88@mbxtoa1.winmail.deshaw.com>
References: <AM2PR07MB0948D12D638C1ABD1AD603E3A44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>
	<5744538D.3030407@qmul.ac.uk>
	<AM2PR07MB09480DBC9D614EE7B5826AAAA44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>
	<d6028bcf19f8402b9315ab875d075ba4@mbxtoa1.winmail.deshaw.com>
	<20160524161719.AA629AC03F@b01ledav006.gho.pok.ibm.com>
	<51fa215cfdc149c6b9c6cdeb50322d88@mbxtoa1.winmail.deshaw.com>
Message-ID: <CAGC__DgTuomeon_tD2BSw0WdTFZQvEeKcQ-yX2EKbzykXHma0g@mail.gmail.com>

Not now, but in a previous role, we would specifically increase the oom
score on computer processes on our cluster that could consume a large
amount of ram, trying to protect system processes. Once did this we had 0
system processes die.



On 25 May 2016 at 17:00, Sanchez, Paul <Paul.Sanchez at deshaw.com> wrote:

> I'm sure that Yuri is right about the corner-case complexity across all
> linux and Spectrum/GPFS versions.
>
> In situations where lots of outstanding tokens exist, and there are few
> token managers, we have seen the assassination of a large footprint mmfsd
> in GPFS 4.1 seem to impact entire clusters, potentially due to
> serialization in recovery of so many tokens, and overlapping access among
> nodes. We're looking forward to fixes in 4.2.1 to address some of this too.
>
> But for what it's worth, on RH6/7 with 4.1, we have seen the end of OOM
> impacting GPFS since implementing the callback. One item I forgot is that
> we don't set it to -500, but to OOM_SCORE_ADJ_MIN, which on our systems is
> -1000. That causes the heuristic oom_badness to return the lowest possible
> score, more thoroughly immunizing it against selection.
>
> Thx
> Paul
>
> Sent with Good Work (www.good.com)
>
>
> *From: *Yuri L Volobuev <volobuev at us.ibm.com>
> *Date: *Tuesday, May 24, 2016, 12:17 PM
> *To: *gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> *Subject: *Re: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
>
> This problem is more complex than it may seem. The thing is, mmfsd runs as
> root, as thus already possesses a certain amount of natural immunity to OOM
> killer. So adjusting mmfsd oom_score_adj doesn't radically change the
> ranking of OOM killer victims, only tweaks it. The way things are supposed
> to work is: a user process eats up a lot of memory, and once a threshold is
> hit, OOM killer picks off the memory hog, and the memory is released.
> Unprivileged processes inherently have a higher OOM score, and should be
> killed off first. If that doesn't work, for some reason, the OOM killer
> gets desperate and starts going after root processes. Once things get to
> this point, it's tough. If you somehow manage to spare mmfsd per se, what's
> going to happen next? The OOM killer still needs a victim. What we've seen
> happen in such a situation is semi-random privileged process killing. mmfsd
> stays alive, but various other system processes are picked off, and pretty
> quickly the node is a basket case. A Linux node is not very resilient to
> random process killing. And it doesn't help that those other privileged
> processes usually don't use much memory, so killing them doesn't release
> much, and the carnage keeps on going. The real problem is: why wasn't the
> non-privileged memory hog process killed off first, before root processes
> became fair game? This is where things get pretty complicated, and depend
> heavily on the Linux version. There's one specific issue that did get
> diagnosed. If a process is using mmap and has page faults going that result
> in GPFS IO, on older versions of GPFS the process would fail to error out
> after a SIGKILL, due to locking complications spanning Linux kernel VMM and
> GPFS mmap code. This means the OOM killer would attempt to kill a process,
> but that wouldn't produce the desired result (the process is still around),
> and the OOM killer keeps moving down the list. This problem has been fixed
> in the current GPFS service levels. It is possible that a similar problem
> may exist that prevents a memory hog process from erroring out. I strongly
> encourage opening a PMR to investigate such a situation, instead of trying
> to work around it without understanding why mmfsd was targeted in the first
> place.
>
> This is the case of prevention being the best cure. Where we've seen
> success is customers using cgroups to prevent user processes from running a
> node out of memory in the first place. This has been shown to work well.
> Dealing with the fallout from running out of memory is a much harder task.
>
> The post-mmfsd-kill symptoms that are described in the original note are
> not normal. If an mmfsd process is killed, other nodes will become aware of
> this fact faily quickly, and the node is going to be expelled from the
> cluster (yes, expels *can* be a good thing). In the normal case, TCP/IP
> sockets are closed as soon as mmfsd is killed, and other nodes immediately
> receive TCP RST packets, and close their connection endpoints. If the worst
> case, if a node just becomes catatonic, but RST is not sent out, the
> troubled node is going to be expelled from the cluster after about 2
> minutes of pinging (in a default configuration). There should definitely
> not be a permanent hang that necessitates a manual intervention. Again,
> older versions of GPFS had no protection against surprise OOM thread kills,
> but in the current code some self-monitoring capabilities have been added,
> and a single troubled node won't have a lasting impact on the cluster. If
> you aren't running with a reasonably current level of GPFS 3.5 service, I
> strongly recommend upgrading. If you see the symptoms originally described
> with the current code, that's a bug that we need to fix, so please open a
> PMR to address the issue.
>
> yuri
>
> [image: Inactive hide details for "Sanchez, Paul" ---05/24/2016 07:33:18
> AM---Hi Peter, This is mentioned explicitly in the Spectrum Sc]"Sanchez,
> Paul" ---05/24/2016 07:33:18 AM---Hi Peter, This is mentioned explicitly in
> the Spectrum Scale docs (http://www.ibm.com/support/knowle
>
> From: "Sanchez, Paul" <Paul.Sanchez at deshaw.com>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>,
> Date: 05/24/2016 07:33 AM
> Subject: Re: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> ------------------------------
>
>
>
> Hi Peter,
>
> This is mentioned explicitly in the Spectrum Scale docs (
> http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.pdg.doc/bl1pdg_kerncfg.htm?lang=en)
> as a problem for the admin to consider, and many of us have been bitten by
> this. There are references going back at least to GPFS 3.1 in 2008 on
> developerworks complaining about this situation.
>
> While the answer you described below is essentially what we do as well, I
> would argue that this is a problem which IBM should just own and fix for
> everyone. I cannot think of a situation in which you would want GPFS to
> be sacrificed on a node due to out-of-memory conditions, and I have seen
> several terrible consequences of this, including loss of cached,
> user-acknowledged writes.
>
> I don't think there are any real gotchas. But in addition, our own
> implementation also:
>
> * uses "--event preStartup" instead of "startup", since it runs earlier
> and reduces the risk of a race
>
> * reads the score back out and complains if it hasn't been set
>
> * includes "set -e" to ensure that errors will terminate the script and
> return a non-zero exit code to the callback parent
>
> Thx
> Paul
>
> -----Original Message-----
> From: gpfsug-discuss-bounces at spectrumscale.org [
> mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>] On Behalf Of Peter Childs
> Sent: Tuesday, May 24, 2016 10:01 AM
> To: gpfsug main discussion list
> Subject: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
>
> Hi All,
>
> We have an issue where the Linux kills off GPFS first when a computer runs
> out of memory, this happens when user processors have exhausted memory and
> swap and the out of memory killer in Linux kills the GPFS daemon as the
> largest user of memory, due to its large pinned memory foot print.
>
> We have an issue where the Linux kills off GPFS first when a computer runs
> out of memory. We are running GPFS 3.5
>
> We believe this happens when user processes have exhausted memory and swap
> and the out of memory killer in Linux chooses to kill the GPFS daemon as
> the largest user of memory, due to its large pinned memory footprint.
>
> This means that GPFS is killed and the whole cluster blocks for a minute
> before it resumes operation, this is not ideal, and kills and causes issues
> with most of the cluster.
>
> What we see is users unable to login elsewhere on the cluster until we
> have powered off the node. We believe this is because while the node is
> still pingable, GPFS doesn't expel it from the cluster.
>
> This issue mainly occurs on our frontend nodes of our HPC cluster but can
> effect the rest of the cluster when it occurs.
>
> This issue mainly occurs on the login nodes of our HPC cluster but can
> affect the rest of the cluster when it occurs.
>
> I've seen others on list with this issue.
>
> We've come up with a solution where by the gpfs is adjusted so that is
> unlikely to be the first thing to be killed, and hopefully the user process
> is killed and not GPFS.
>
> We've come up with a solution to adjust the OOM score of GPFS, so that it
> is unlikely to be the first thing to be killed, and hopefully the OOM
> killer picks a user process instead.
>
> Out testing says this solution works, but I'm asking here firstly to share
> our knowledge and secondly to ask if there is anything we've missed with
> this solution and issues with this.
>
> We've tested this and it seems to work. I'm asking here firstly to share
> our knowledge and secondly to ask if there is anything we've missed with
> this solution.
>
> Its short which is part of its beauty.
>
> /usr/local/sbin/gpfs-oom_score_adj
>
> <pre>
> #!/bin/bash
>
> for proc in $(pgrep mmfs); do
> echo -500 >/proc/$proc/oom_score_adj done </pre>
>
> This can then be called automatically on GPFS startup with the following:
>
> <pre>
> mmaddcallback startupoomkiller --command
> /usr/local/sbin/gpfs-oom_score_adj --event startup </pre>
>
> and either restart gpfs or just run the script on all nodes.
>
> Peter Childs
> ITS Research Infrastructure
> Queen Mary, University of London
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160525/f137a550/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160525/f137a550/attachment.gif>

From Luke.Raimbach at crick.ac.uk  Thu May 26 09:20:02 2016
From: Luke.Raimbach at crick.ac.uk (Luke Raimbach)
Date: Thu, 26 May 2016 08:20:02 +0000
Subject: [gpfsug-discuss] CES and Centrify
Message-ID: <AMXPR03MB02105F338038F05AAD3C73BB0410@AMXPR03MB021.eurprd03.prod.outlook.com>

Hi,

Someone said "Centrify" at the recent UK user-group meeting.

Where can I find out more?

Cheers,
Luke.

Luke Raimbach?
Senior HPC Data and Storage Systems Engineer,
The Francis Crick Institute,
Gibbs Building,
215 Euston Road,
London NW1 2BE.

E: luke.raimbach at crick.ac.uk
W: www.crick.ac.uk

The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no. 06885462, with its registered office at 215 Euston Road, London NW1 2BE.

From scottcumbie at dynamixgroup.com  Thu May 26 16:57:14 2016
From: scottcumbie at dynamixgroup.com (Scott Cumbie)
Date: Thu, 26 May 2016 15:57:14 +0000
Subject: [gpfsug-discuss] mmbackup - Two at a time?
Message-ID: <35131C5F-56B9-4098-B711-4B59A025B5BB@dynamixgroup.com>

Just wondering if anyone has a quick answer to this question.  If mmbackup is running on a filesystem and a second mmbackup is executed on the same filesystem, what happens?  Does the second one end because the first one is running?

Thanks,
Scott




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160526/18d3fadb/attachment.htm>

From malone12 at illinois.edu  Thu May 26 19:33:33 2016
From: malone12 at illinois.edu (Maloney, John Daniel)
Date: Thu, 26 May 2016 18:33:33 +0000
Subject: [gpfsug-discuss] Speakers Wanted - June 10, Argonne National Lab
In-Reply-To: <1fa21d083401a0a7acff84b7cd69ee34@webmail.gpfsug.org>
References: <1fa21d083401a0a7acff84b7cd69ee34@webmail.gpfsug.org>
Message-ID: <2AEEB2A9-E7C2-4F55-AA42-7C34117A9F37@illinois.edu>

Hi Kristy,
I?d be willing to present at the User Group Meeting on the 10th at Argonne.  Our talk would be based on our roll out of a new GPFS environment that leverages remote cluster mounts, AFM, and the new cluster export services functionality in 4.2 to help us better serve our campus users who leverage U of I?s campus cluster.  Let me know if a slot is still available and if/when you need more information or slides.

Best,

J.D. Maloney
Storage Engineer | Storage Enabling Technologies Group
National Center for Supercomputing Applications (NCSA)
University of Illinois Urbana-Champaign
1205 W. Clark St. Urbana, IL 61801









On 5/23/16, 6:31 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of usa-principal-gpfsug.org" <gpfsug-discuss-bounces at spectrumscale.org on behalf of usa-principal at gpfsug.org> wrote:

>http://www.spectrumscale.org/spectrum-scale-user-group-usa-meeting-june-10th/

From Robert.Oesterlin at nuance.com  Thu May 26 20:03:44 2016
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Thu, 26 May 2016 19:03:44 +0000
Subject: [gpfsug-discuss] Speakers Wanted - June 10, Argonne National Lab
In-Reply-To: <2AEEB2A9-E7C2-4F55-AA42-7C34117A9F37@illinois.edu>
References: <1fa21d083401a0a7acff84b7cd69ee34@webmail.gpfsug.org>
	<2AEEB2A9-E7C2-4F55-AA42-7C34117A9F37@illinois.edu>
Message-ID: <BCB09BBF-2893-4CF3-83B9-78A2E217D314@nuance.com>

J.D.

We may still have one open slot - can you contact me directly and let's talk details?

robert.oesterlin  @ nuance.com


Bob Oesterlin
Sr Storage Engineer, Nuance HPC Grid
507-269-0413


From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of "Maloney, John Daniel" <malone12 at illinois.edu>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Thursday, May 26, 2016 at 1:33 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Speakers Wanted - June 10, Argonne National Lab

Hi Kristy,

I?d be willing to present at the User Group Meeting on the 10th at Argonne.  Our talk would be based on our roll out of a new GPFS environment that leverages remote cluster mounts, AFM, and the new cluster export services functionality in 4.2 to help us better serve our campus users who leverage U of I?s campus cluster.  Let me know if a slot is still available and if/when you need more information or slides.



Best,



J.D. Maloney

Storage Engineer | Storage Enabling Technologies Group

National Center for Supercomputing Applications (NCSA)

University of Illinois Urbana-Champaign

1205 W. Clark St. Urbana, IL 61801



















On 5/23/16, 6:31 AM, "gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> on behalf of usa-principal-gpfsug.org" <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> on behalf of usa-principal at gpfsug.org<mailto:usa-principal at gpfsug.org>> wrote:



https://urldefense.proofpoint.com/v2/url?u=http-3A__www.spectrumscale.org_spectrum-2Dscale-2Duser-2Dgroup-2Dusa-2Dmeeting-2Djune-2D10th_&d=CwIGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=MKKI9kld3C0B3tzQT64SPW1uCBHTb72hZlRP6x4V34E&s=sUYYuCYXPiSXA6Nhejs1H_sogi2WDfREIRc5yir2FJ4&e=

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwIGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=MKKI9kld3C0B3tzQT64SPW1uCBHTb72hZlRP6x4V34E&s=lORUKCzYSOh63hgiyP_35POyN_hMzfYGVVx8kQfFl-c&e=

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160526/cc0b7542/attachment.htm>

From craigawilson at gmail.com  Fri May 27 11:32:09 2016
From: craigawilson at gmail.com (Craig Wilson)
Date: Fri, 27 May 2016 11:32:09 +0100
Subject: [gpfsug-discuss] mmbackup - Two at a time?
In-Reply-To: <35131C5F-56B9-4098-B711-4B59A025B5BB@dynamixgroup.com>
References: <35131C5F-56B9-4098-B711-4B59A025B5BB@dynamixgroup.com>
Message-ID: <CALdwc7dd1b+oF3jXMnJFwKtf9QKJ7m-kRETViwA0cmp2oMYJZg@mail.gmail.com>

Hi Scott

I've not tried this one 4.# branch as I'm still running 3.5 but on 3.5 you
get a error message saying:

mmbackup: Failed to obtain the local environment update lock.
mmbackup: Command failed.  Examine previous error messages to determine
cause.

The old mmbackup keeps running through.

Kind Regards,

Craig Wilson.
HPC Systems Engineer
OCF Plc.

On 26 May 2016 at 16:57, Scott Cumbie <scottcumbie at dynamixgroup.com> wrote:

> Just wondering if anyone has a quick answer to this question.  If mmbackup
> is running on a filesystem and a second mmbackup is executed on the same
> filesystem, what happens?  Does the second one end because the first one is
> running?
>
> Thanks,
> Scott
>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160527/456339d2/attachment.htm>

From S.J.Thompson at bham.ac.uk  Fri May 27 11:49:27 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Fri, 27 May 2016 10:49:27 +0000
Subject: [gpfsug-discuss] mmbackup - Two at a time?
In-Reply-To: <CALdwc7dd1b+oF3jXMnJFwKtf9QKJ7m-kRETViwA0cmp2oMYJZg@mail.gmail.com>
References: <35131C5F-56B9-4098-B711-4B59A025B5BB@dynamixgroup.com>
	<CALdwc7dd1b+oF3jXMnJFwKtf9QKJ7m-kRETViwA0cmp2oMYJZg@mail.gmail.com>
Message-ID: <D36DE461.2AF90%s.j.thompson@bham.ac.uk>

Now we've seen different to this... But maybe it depends how you run mmbackup multiple times.

We have several nodes which are backing up via Spectrum Protect, and we thought it would be nice to have the client scheduler running on multiple nodes, we assumed it would only actually run one instance of the backup. Not so. We've seen two client schedulers kick off mmbackup (each then I think running across several nodes).

Simon

From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of Craig Wilson <craigawilson at gmail.com<mailto:craigawilson at gmail.com>>
Reply-To: "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Friday, 27 May 2016 at 11:32
To: "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] mmbackup - Two at a time?

Hi Scott

I've not tried this one 4.# branch as I'm still running 3.5 but on 3.5 you get a error message saying:

mmbackup: Failed to obtain the local environment update lock.
mmbackup: Command failed.  Examine previous error messages to determine cause.

The old mmbackup keeps running through.

Kind Regards,

Craig Wilson.
HPC Systems Engineer
OCF Plc.

On 26 May 2016 at 16:57, Scott Cumbie <scottcumbie at dynamixgroup.com<mailto:scottcumbie at dynamixgroup.com>> wrote:
Just wondering if anyone has a quick answer to this question.  If mmbackup is running on a filesystem and a second mmbackup is executed on the same filesystem, what happens?  Does the second one end because the first one is running?

Thanks,
Scott





_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160527/f431c56f/attachment.htm>

From craigawilson at gmail.com  Fri May 27 14:36:24 2016
From: craigawilson at gmail.com (Craig Wilson)
Date: Fri, 27 May 2016 14:36:24 +0100
Subject: [gpfsug-discuss] mmbackup - Two at a time?
In-Reply-To: <D36DE461.2AF90%s.j.thompson@bham.ac.uk>
References: <35131C5F-56B9-4098-B711-4B59A025B5BB@dynamixgroup.com>
	<CALdwc7dd1b+oF3jXMnJFwKtf9QKJ7m-kRETViwA0cmp2oMYJZg@mail.gmail.com>
	<D36DE461.2AF90%s.j.thompson@bham.ac.uk>
Message-ID: <CALdwc7cu5dmCaP__aAv5htnxV4vH8GO7pVMKmUg8WBuLxBo35w@mail.gmail.com>

That is interesting...

The only time I've seen two starting is when someone manually cleared the
environment lock. That said I've never tried running from a different node
at the same time. I presume they would be duplicating effort based on the
node name being used in the shadow database file name.

Kind regards,

Craig Wilson.

On 27 May 2016 at 11:49, Simon Thompson (Research Computing - IT Services) <
S.J.Thompson at bham.ac.uk> wrote:

> Now we've seen different to this... But maybe it depends how you run
> mmbackup multiple times.
>
> We have several nodes which are backing up via Spectrum Protect, and we
> thought it would be nice to have the client scheduler running on multiple
> nodes, we assumed it would only actually run one instance of the backup.
> Not so. We've seen two client schedulers kick off mmbackup (each then I
> think running across several nodes).
>
> Simon
>
> From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Craig
> Wilson <craigawilson at gmail.com>
> Reply-To: "gpfsug-discuss at spectrumscale.org" <
> gpfsug-discuss at spectrumscale.org>
> Date: Friday, 27 May 2016 at 11:32
> To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
> Subject: Re: [gpfsug-discuss] mmbackup - Two at a time?
>
> Hi Scott
>
> I've not tried this one 4.# branch as I'm still running 3.5 but on 3.5 you
> get a error message saying:
>
> mmbackup: Failed to obtain the local environment update lock.
> mmbackup: Command failed.  Examine previous error messages to determine
> cause.
>
> The old mmbackup keeps running through.
>
> Kind Regards,
>
> Craig Wilson.
> HPC Systems Engineer
> OCF Plc.
>
> On 26 May 2016 at 16:57, Scott Cumbie <scottcumbie at dynamixgroup.com>
> wrote:
>
>> Just wondering if anyone has a quick answer to this question.  If
>> mmbackup is running on a filesystem and a second mmbackup is executed on
>> the same filesystem, what happens?  Does the second one end because the
>> first one is running?
>>
>> Thanks,
>> Scott
>>
>>
>>
>>
>>
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>
>>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160527/c2fba71c/attachment.htm>

From Kevin.Buterbaugh at Vanderbilt.Edu  Fri May 27 21:23:01 2016
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Fri, 27 May 2016 20:23:01 +0000
Subject: [gpfsug-discuss] Building portability layer fails
Message-ID: <FEEC06BD-FE26-4F2B-84D6-AD604AB63208@vanderbilt.edu>

All,

We?re upgrading our cluster from GPFS 4.1.0.8 to GPFS 4.2.0.3 and all is going very well ? except for my tape backup server.

I?ve run the Spectrum Scale installer and installed the rpm?s in /usr/lpp/mmfs/4.2.0.3/gpfs_rpms, but when I run:

make LINUX_DISTRIBUTION=REDHAT_AS_LINUX Autoconfig

it errors with:

make[3]: *** [/usr/lpp/mmfs/src/gpl-linux/kdump-kern.o] Error 1
make[2]: *** [_module_/usr/lpp/mmfs/src/gpl-linux] Error 2
make[2]: Leaving directory `/usr/src/kernels/2.6.32-573.el6.x86_64'
make[1]: *** [modules] Error 1
make[1]: Leaving directory `/usr/lpp/mmfs/src/gpl-linux'
make: *** [Modules] Error 1

Any ideas?  Thanks in advance?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160527/9513f318/attachment.htm>

From sfadden at us.ibm.com  Fri May 27 21:35:58 2016
From: sfadden at us.ibm.com (Scott Fadden)
Date: Fri, 27 May 2016 13:35:58 -0700
Subject: [gpfsug-discuss] Building portability layer fails
In-Reply-To: <FEEC06BD-FE26-4F2B-84D6-AD604AB63208@vanderbilt.edu>
References: <FEEC06BD-FE26-4F2B-84D6-AD604AB63208@vanderbilt.edu>
Message-ID: <20160527203604.0F1F86A045@b03ledav003.gho.boulder.ibm.com>

On 4.2.0.3 run mmbuildgpl. See how that goes.


Scott Fadden
Spectrum Scale - Technical Marketing 
Phone: (503) 880-5833 
sfadden at us.ibm.com
http://www.ibm.com/systems/storage/spectrum/scale



From:   "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   05/27/2016 01:23 PM
Subject:        [gpfsug-discuss] Building portability layer fails
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



All, 

We?re upgrading our cluster from GPFS 4.1.0.8 to GPFS 4.2.0.3 and all is 
going very well ? except for my tape backup server.

I?ve run the Spectrum Scale installer and installed the rpm?s in 
/usr/lpp/mmfs/4.2.0.3/gpfs_rpms, but when I run:

make LINUX_DISTRIBUTION=REDHAT_AS_LINUX Autoconfig

it errors with:

make[3]: *** [/usr/lpp/mmfs/src/gpl-linux/kdump-kern.o] Error 1
make[2]: *** [_module_/usr/lpp/mmfs/src/gpl-linux] Error 2
make[2]: Leaving directory `/usr/src/kernels/2.6.32-573.el6.x86_64'
make[1]: *** [modules] Error 1
make[1]: Leaving directory `/usr/lpp/mmfs/src/gpl-linux'
make: *** [Modules] Error 1

Any ideas?  Thanks in advance?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and 
Education
Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160527/d0a3d54b/attachment.htm>

From Kevin.Buterbaugh at Vanderbilt.Edu  Fri May 27 21:51:18 2016
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Fri, 27 May 2016 20:51:18 +0000
Subject: [gpfsug-discuss] Building portability layer fails
In-Reply-To: <20160527203604.0F1F86A045@b03ledav003.gho.boulder.ibm.com>
References: <FEEC06BD-FE26-4F2B-84D6-AD604AB63208@vanderbilt.edu>
	<20160527203604.0F1F86A045@b03ledav003.gho.boulder.ibm.com>
Message-ID: <4E17167D-5A7A-4A04-8A56-03F4DDB092CB@vanderbilt.edu>

All,

After spending a couple of hours on this I finally took the key step towards fixing the problem myself ? I e-mailed this list!  ;-)

Five minutes later, I saw that the env.mcr file was munged, fixed that, and everything built just fine.  Sorry for the noise?

Kevin

On May 27, 2016, at 3:35 PM, Scott Fadden <sfadden at us.ibm.com<mailto:sfadden at us.ibm.com>> wrote:

On 4.2.0.3 run mmbuildgpl. See how that goes.


Scott Fadden
Spectrum Scale - Technical Marketing
Phone: (503) 880-5833
sfadden at us.ibm.com<mailto:sfadden at us.ibm.com>
http://www.ibm.com/systems/storage/spectrum/scale



From:        "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at vanderbilt.edu>>
To:        gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date:        05/27/2016 01:23 PM
Subject:        [gpfsug-discuss] Building portability layer fails
Sent by:        gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>
________________________________



All,

We?re upgrading our cluster from GPFS 4.1.0.8 to GPFS 4.2.0.3 and all is going very well ? except for my tape backup server.

I?ve run the Spectrum Scale installer and installed the rpm?s in /usr/lpp/mmfs/4.2.0.3/gpfs_rpms, but when I run:

make LINUX_DISTRIBUTION=REDHAT_AS_LINUX Autoconfig

it errors with:

make[3]: *** [/usr/lpp/mmfs/src/gpl-linux/kdump-kern.o] Error 1
make[2]: *** [_module_/usr/lpp/mmfs/src/gpl-linux] Error 2
make[2]: Leaving directory `/usr/src/kernels/2.6.32-573.el6.x86_64'
make[1]: *** [modules] Error 1
make[1]: Leaving directory `/usr/lpp/mmfs/src/gpl-linux'
make: *** [Modules] Error 1

Any ideas?  Thanks in advance?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu>- (615)875-9633


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160527/58a0de66/attachment.htm>

From S.J.Thompson at bham.ac.uk  Fri May 27 22:44:48 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Fri, 27 May 2016 21:44:48 +0000
Subject: [gpfsug-discuss] Building portability layer fails
In-Reply-To: <20160527203604.0F1F86A045@b03ledav003.gho.boulder.ibm.com>
References: <20160527203604.0F1F86A045@b03ledav003.gho.boulder.ibm.com>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F5786AF5@EX13.adf.bham.ac.uk>

Does mmbuildgpl work if its a RHEL clone like CentOS?

Simon

-----Original Message-----
From: Scott Fadden [sfadden at us.ibm.com<mailto:sfadden at us.ibm.com>]
Sent: Friday, May 27, 2016 09:36 PM GMT Standard Time
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Building portability layer fails

On 4.2.0.3 run mmbuildgpl. See how that goes.


Scott Fadden
Spectrum Scale - Technical Marketing
Phone: (503) 880-5833
sfadden at us.ibm.com
http://www.ibm.com/systems/storage/spectrum/scale



From:        "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu>
To:        gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:        05/27/2016 01:23 PM
Subject:        [gpfsug-discuss] Building portability layer fails
Sent by:        gpfsug-discuss-bounces at spectrumscale.org
________________________________



All,

We?re upgrading our cluster from GPFS 4.1.0.8 to GPFS 4.2.0.3 and all is going very well ? except for my tape backup server.

I?ve run the Spectrum Scale installer and installed the rpm?s in /usr/lpp/mmfs/4.2.0.3/gpfs_rpms, but when I run:

make LINUX_DISTRIBUTION=REDHAT_AS_LINUX Autoconfig

it errors with:

make[3]: *** [/usr/lpp/mmfs/src/gpl-linux/kdump-kern.o] Error 1
make[2]: *** [_module_/usr/lpp/mmfs/src/gpl-linux] Error 2
make[2]: Leaving directory `/usr/src/kernels/2.6.32-573.el6.x86_64'
make[1]: *** [modules] Error 1
make[1]: Leaving directory `/usr/lpp/mmfs/src/gpl-linux'
make: *** [Modules] Error 1

Any ideas?  Thanks in advance?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu>- (615)875-9633


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160527/c587e999/attachment.htm>

From sfadden at us.ibm.com  Sat May 28 01:47:05 2016
From: sfadden at us.ibm.com (Scott Fadden)
Date: Sat, 28 May 2016 00:47:05 +0000
Subject: [gpfsug-discuss] Building portability layer fails
In-Reply-To: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F5786AF5@EX13.adf.bham.ac.uk>
Message-ID: <20160528004711.1B4A46A03D@b03ledav003.gho.boulder.ibm.com>

It should

Sent from IBM Verse


   Simon Thompson (Research Computing - IT Services) --- RE: Re: [gpfsug-discuss] Building portability layer fails --- 
    From:"Simon Thompson (Research Computing - IT Services)" <S.J.Thompson at bham.ac.uk>To:"'gpfsug main discussion list'" <gpfsug-discuss at spectrumscale.org>, "Scott Fadden" <sfadden at us.ibm.com>Date:Fri, May 27, 2016 14:45Subject:RE: Re: [gpfsug-discuss] Building portability layer fails
  
         Does mmbuildgpl work if its a RHEL clone like CentOS?
  
  Simon
  
  -----Original Message-----
  From: Scott Fadden [sfadden at us.ibm.com]
  Sent: Friday, May 27, 2016 09:36 PM GMT Standard Time
  To: gpfsug main discussion list
  Subject: Re: [gpfsug-discuss] Building portability layer fails
  
  On 4.2.0.3 run mmbuildgpl. See how that goes.
  
  
  Scott Fadden
  Spectrum Scale - Technical Marketing
  Phone: (503) 880-5833  
  sfadden at us.ibm.com
  http://www.ibm.com/systems/storage/spectrum/scale
  
  
  
  From:        "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu>
  To:        gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
  Date:        05/27/2016 01:23 PM
  Subject:        [gpfsug-discuss] Building portability layer fails
  Sent by:        gpfsug-discuss-bounces at spectrumscale.org
   
  
  
  All,
  
  We?re upgrading our cluster from GPFS 4.1.0.8 to GPFS 4.2.0.3 and all is going very well ? except for my tape backup server.
  
  I?ve run the Spectrum Scale installer and installed the rpm?s in /usr/lpp/mmfs/4.2.0.3/gpfs_rpms, but when I run:
  
  make LINUX_DISTRIBUTION=REDHAT_AS_LINUX Autoconfig
  
  it errors with:
  
  make[3]: *** [/usr/lpp/mmfs/src/gpl-linux/kdump-kern.o] Error 1
  make[2]: *** [_module_/usr/lpp/mmfs/src/gpl-linux] Error 2
  make[2]: Leaving directory `/usr/src/kernels/2.6.32-573.el6.x86_64'
  make[1]: *** [modules] Error 1
  make[1]: Leaving directory `/usr/lpp/mmfs/src/gpl-linux'
  make: *** [Modules] Error 1
  
  Any ideas?  Thanks in advance?
  
  ?
  Kevin Buterbaugh - Senior System Administrator
  Vanderbilt University - Advanced Computing Center for Research and Education
  Kevin.Buterbaugh at vanderbilt.edu- (615)875-9633
  
  
  _______________________________________________
  gpfsug-discuss mailing list
  gpfsug-discuss at spectrumscale.org
  http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160528/e1576726/attachment.htm>

From laurence at qsplace.co.uk  Sat May 28 09:36:50 2016
From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow)
Date: Sat, 28 May 2016 09:36:50 +0100
Subject: [gpfsug-discuss] Building portability layer fails
In-Reply-To: <20160528004711.1B4A46A03D@b03ledav003.gho.boulder.ibm.com>
References: <20160528004711.1B4A46A03D@b03ledav003.gho.boulder.ibm.com>
Message-ID: <16F6CB71-3BC1-4F9F-BA2F-DC6911325C8A@qsplace.co.uk>

Yes it does.

But you still have to export LINUX_DISTRIBUTION=REDHAT_AS_LINUX? before the command.

i.e.
LINUX_DISTRIBUTION=REDHAT_AS_LINUX? mmbuildgpl

-- Lauz

On 28 May 2016 01:47:05 BST, Scott Fadden <sfadden at us.ibm.com> wrote:
>It should
>
>Sent from IBM Verse
>
>
>Simon Thompson (Research Computing - IT Services) --- RE: Re:
>[gpfsug-discuss] Building portability layer fails --- 
>From:"Simon Thompson (Research Computing - IT Services)"
><S.J.Thompson at bham.ac.uk>To:"'gpfsug main discussion list'"
><gpfsug-discuss at spectrumscale.org>, "Scott Fadden"
><sfadden at us.ibm.com>Date:Fri, May 27, 2016 14:45Subject:RE: Re:
>[gpfsug-discuss] Building portability layer fails
>  
>         Does mmbuildgpl work if its a RHEL clone like CentOS?
>  
>  Simon
>  
>  -----Original Message-----
>  From: Scott Fadden [sfadden at us.ibm.com]
>  Sent: Friday, May 27, 2016 09:36 PM GMT Standard Time
>  To: gpfsug main discussion list
>  Subject: Re: [gpfsug-discuss] Building portability layer fails
>  
>  On 4.2.0.3 run mmbuildgpl. See how that goes.
>  
>  
>  Scott Fadden
>  Spectrum Scale - Technical Marketing
>  Phone: (503) 880-5833  
>  sfadden at us.ibm.com
>  http://www.ibm.com/systems/storage/spectrum/scale
>  
>  
>  
>  From:        "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu>
>To:        gpfsug main discussion list
><gpfsug-discuss at spectrumscale.org>
>  Date:        05/27/2016 01:23 PM
>  Subject:        [gpfsug-discuss] Building portability layer fails
>  Sent by:        gpfsug-discuss-bounces at spectrumscale.org
>   
>  
>  
>  All,
>  
>We?re upgrading our cluster from GPFS 4.1.0.8 to GPFS 4.2.0.3 and all
>is going very well ? except for my tape backup server.
>  
>I?ve run the Spectrum Scale installer and installed the rpm?s in
>/usr/lpp/mmfs/4.2.0.3/gpfs_rpms, but when I run:
>  
>  make LINUX_DISTRIBUTION=REDHAT_AS_LINUX Autoconfig
>  
>  it errors with:
>  
>  make[3]: *** [/usr/lpp/mmfs/src/gpl-linux/kdump-kern.o] Error 1
>  make[2]: *** [_module_/usr/lpp/mmfs/src/gpl-linux] Error 2
>  make[2]: Leaving directory `/usr/src/kernels/2.6.32-573.el6.x86_64'
>  make[1]: *** [modules] Error 1
>  make[1]: Leaving directory `/usr/lpp/mmfs/src/gpl-linux'
>  make: *** [Modules] Error 1
>  
>  Any ideas?  Thanks in advance?
>  
>  ?
>  Kevin Buterbaugh - Senior System Administrator
>Vanderbilt University - Advanced Computing Center for Research and
>Education
>  Kevin.Buterbaugh at vanderbilt.edu- (615)875-9633
>  
>  
>  _______________________________________________
>  gpfsug-discuss mailing list
>  gpfsug-discuss at spectrumscale.org
>  http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>gpfsug-discuss mailing list
>gpfsug-discuss at spectrumscale.org
>http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160528/077f9264/attachment.htm>

From Robert.Oesterlin at nuance.com  Tue May 31 12:57:19 2016
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Tue, 31 May 2016 11:57:19 +0000
Subject: [gpfsug-discuss] GPFS/Spectrum Scale security vulernability - All
	versions
Message-ID: <E0694FAD-6D65-40AA-AB98-D6934E24D557@nuance.com>

IBM published a security vulnerability today that effects all current and prior levels of GPFS/Spectrum Scale. The short explanation is "IBM Spectrum Scale and IBM GPFS that could allow a local attacker to inject commands into setuid file parameters and execute commands as root." This is a "high" vulnerability (8.4).

Details here: http://www-01.ibm.com/support/docview.wss?uid=ssg1S1005781


Bob Oesterlin
Sr Storage Engineer, Nuance HPC Grid


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160531/184da280/attachment.htm>

From TROPPENS at de.ibm.com  Tue May  3 13:05:51 2016
From: TROPPENS at de.ibm.com (Ulf Troppens)
Date: Tue, 3 May 2016 14:05:51 +0200
Subject: [gpfsug-discuss] GPFS/Spectrum Scale Upcoming US Events - Save
 the Dates - Registration for New York May 26
In-Reply-To: <9489DBA2-1F12-4B05-A968-5D4855FBEA40@gpfsug.org>
References: <4192D644-E3AB-4B7B-AF7A-96D3D617FA7B@gpfsug.org>
	<9489DBA2-1F12-4B05-A968-5D4855FBEA40@gpfsug.org>
Message-ID: <201605031206.u43C5wss002725@d06av09.portsmouth.uk.ibm.com>

Registration for the first event in New York is up as well - the agenda
still needs some touch.

Please register here and select 'May 26 GPFS Day Registration', if you
don't attend SPXXL:
https://www.spxxl.org/?q=New-York-City-2016
https://www.eventbrite.com/e/spxxlscicomp-2016-summer-meeting-registration-24444020724


--
IBM Spectrum Scale Development - Client Engagements & Solutions Delivery
Consulting IT Specialist
Author "Storage Networks Explained"

IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Gesch?ftsf?hrung: Dirk Wittkopp
Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart,
HRB 243294




From:	GPFS UG USA Principal <usa-principal at gpfsug.org>
To:	gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Date:	28.04.2016 23:44
Subject:	Re: [gpfsug-discuss] GPFS/Spectrum Scale Upcoming US Events -
            Save	the Dates
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



All, the registration page for the second event listed below at Argonne
National Lab on June 10th is now up. An updated agenda is also at this
site.

Please register here: https://www.regonline.com/Spectrumscalemeeting

We look forward to seeing some of you at these upcoming events.

Feel free to send suggestions for future events in your area.

Cheers,
-Kristy


      On Apr 4, 2016, at 4:52 PM, GPFS UG USA Principal <
      usa-principal at gpfsug.org> wrote:

      Hello all,

      We?d like to announce two upcoming US GPFS/Spectrum Scale Events. One
      on the east coast, one in the midwest.

      1) May 26th (full day event): GPFS/Spectrum Scale Day at the SPXXL
      conference in NYC https://www.spxxl.org/?q=New-York-City-2016
      Developers and Engineers from IBM will be at the meeting to cover
      topics, and open dialogue between IBM and customers will be
      encouraged. More details about the agenda, hotel and other logistics
      will come later this month.

      Tentative Agenda:
      ? 10 reasons for upgrading from GPFS 3.5 to Spectrum Scale 4.2.1
      ? Enhancements for CORAL from IBM
      ? Panel discussion with customers, topic TBD
      ? AFM and integration with Spectrum Protect
      ? Best practices for GPFS or Spectrum Scale Tuning.
      ? At least one site update

      Location:
      New York Academy of Medicine
      1216 Fifth Avenue
      New York, NY 10029

      ??

      2) June 10th (full day event): GPFS/Spectrum Scale Users Group
      Meeting at Argonne National Lab (ANL). Thanks to Argonne for hosting
      this event. Developers and Engineers from IBM will be at the meeting
      to cover topics, and open dialogue between IBM and customers will be
      encouraged, as usual no marketing pitches!

      Location: Argonne National Lab more details and final agenda will
      come later.

      Tentative Agenda:

      9:00a-12:30p
      9-9:30a - Opening Remarks
      9:30-10a Deep Dive - Update on ESS
      10a-11a Deep Dive - Problem Determination (Presentation 30 min/Panel
      30 min?)
      11-11:30 Break
      11:30a-Noon - Deep Dive - Protect & Scale integration
      Noon-12:30p HDFS/Hadoop

      12:30 - 1:30p Lunch

      1:30p-5:00p
      1:30 - 2:00p IBM AFM Update
      2:00-2:30p ANL: AFM as a burst buffer
      2:30-3:00p ANL: GHI (GPFS HPSS Integration)
      3:00-3:30p Break
      3:30p - 4:00p LANL: ? or other site preso
      4:00-4:30p Nuance: GPFS Performance Sensors Deployment Experiences
      4:30p -5:00p Closing comments and Open Forum for Questions

      5:00 - ?
      Beer hunting?

      ??

      We hope you can attend one or both of these events.

      Best,
      Kristy Kallback-Rose & Bob Oesterlin
      GPFS Users Group - USA Chapter - Principal & Co-principal
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160503/c00a2018/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160503/c00a2018/attachment-0001.gif>

From A.K.Ghumra at bham.ac.uk  Tue May  3 14:45:23 2016
From: A.K.Ghumra at bham.ac.uk (Aslam Ghumra (IT Services, Facilities Management))
Date: Tue, 3 May 2016 13:45:23 +0000
Subject: [gpfsug-discuss] SMB access speed
Message-ID: <D50747684476FC4EAA59F1BE20FD0557D7AD34BC@EX12.adf.bham.ac.uk>

Hi Yaron, 

Local hdd to hdd copy speed is approx. 323.15MB/s, not sure if its's a different hdd or different partition, hopefully will get that information later.

Yup, gui is not an option and hopefully the researcher will use command line.

Regards, Aslam
Research Computing Team
DDI: +44 (121) 414 5877 | Skype: JanitorX | Twitter: @aslamghumra | a.k.ghumra at bham.ac.uk | intranet.birmingham.ac.uk/bear


-----Original Message-----
From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of gpfsug-discuss-request at spectrumscale.org
Sent: 30 April 2016 12:00

Today's Topics:

   1. Re: SMB access speed (Yaron Daniel)


----------------------------------------------------------------------

Message: 1
Date: Sat, 30 Apr 2016 08:17:28 +0300
From: "Yaron Daniel" <YARD at il.ibm.com>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] SMB access speed
Message-ID:
	<201604300517.u3U5HcbY022432 at d06av12.portsmouth.uk.ibm.com>
Content-Type: text/plain; charset="iso-8859-1"

Hi

It could be that GUI use in the "background" default command which use smb v1.

Regard copy files from GPFS to Local HDD, it might be related to the local HDD settings.
What is the speed transfer between the local HHD ? Cache Settings and so..

 
Regards
 


 
 
Yaron Daniel
 94 Em Ha'Moshavot Rd

Server, Storage and Data Services - Team Leader  Petach Tiqva, 49527 Global Technology Services  Israel
Phone:
+972-3-916-5672
 
 
Fax:
+972-3-916-5672
 
 
Mobile:
+972-52-8395593
 
 
e-mail:
yard at il.ibm.com
 
 
IBM Israel
 
 
 
 

 



From:   "Aslam Ghumra (IT Services, Facilities Management)" 
<A.K.Ghumra at bham.ac.uk>
To:     "gpfsug-discuss at spectrumscale.org" 
<gpfsug-discuss at spectrumscale.org>
Date:   04/29/2016 07:07 PM
Subject:        [gpfsug-discuss] SMB access speed
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Many thanks Yaron, after the change to disable encryption we were able to 
increase the speed via Ubuntu of copying files from the local desktop to 
our gpfs filestore with average speeds of 60Mbps.
 
We also tried changing the mount from vers=3.0 to vers=2.1, which gave 
similar figures
 
However, using the Ubuntu gui ( Unity ) the speed drops down to 7Mbps, 
however, we?re not concerned as the user will use rsync / cp. 
 
The other issue is copying data from gpfs filestore to the local HDD, 
which resulted in 4Mbps.
 
Aslam Ghumra
Research Data Management
____________________________
IT Services
Elms Road Data Centre
Building G5
Edgbaston
Birmingham B15 2TT
T: 0121 414 5877
F; 0121 414 3952
Skype : JanitorX
Twitter : @aslamghumra
http://intranet.bham.ac.uk/bear
 _______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160430/875e5aa0/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 1851 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160430/875e5aa0/attachment-0001.gif>

------------------------------

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


End of gpfsug-discuss Digest, Vol 51, Issue 57
**********************************************


From Kevin.Buterbaugh at Vanderbilt.Edu  Tue May  3 21:20:15 2016
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Tue, 3 May 2016 20:20:15 +0000
Subject: [gpfsug-discuss] CNFS and multiple IP addresses
Message-ID: <94CBCF33-0B97-440E-8E51-F4A51E9333FB@vanderbilt.edu>

Hi All,

I?ve looked in the documentation and on the GPFS Wiki for the answer to this question but haven?t found it.  Please feel free to point me to docs / web pages if I?ve missed something.

One of the GPFS clusters I administer uses CNFS, which I did not set up on this cluster ? and the person who did is now an employee of another organization.  The cluster in question runs GPFS 4.1.0.x, although I doubt that matters.

There are 4 CNFS servers, each of which has 2 IP addresses listed under the CNFS IP address list when running ?mmlscluster ?cnfs?.  I want to delete one of those two IP addresses for each server due to a planned network reorganization.  How do I do that?  Or do I need to completely delete the server from CNFS and add it back in with just the one IP address I want to keep?

Thanks in advance?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160503/76735cf0/attachment-0001.htm>

From Kevin.Buterbaugh at Vanderbilt.Edu  Tue May  3 21:27:12 2016
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Tue, 3 May 2016 20:27:12 +0000
Subject: [gpfsug-discuss] Data replication and fileset quotas
Message-ID: <AEE0A7CC-E792-462F-9BD0-C6DF9C551E41@vanderbilt.edu>

Hi again all,

I have another question on a completely different topic and therefore I decided it was better to send two separate e-mails.

For a number of years now we have had a GPFS filesystem where we use filesets and set fileset quotas.  Data replication is set to one.  I understand how that all works.

We are creating another GPFS filesystem where we intend to also use filesets and fileset quotas, but set data replication to two.  Based on my experience with data replication on a filesystem that doesn?t use filesets, I am expecting that setting data replication to two means that I will need to double the quota for each fileset (i.e. if a group has bought 5 TB of space I?ll need to set their fileset quota to 10 TB) but haven?t found where that is explicitly documented.  Is that correct?

Thanks again, all?

Kevin

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160503/c1274617/attachment-0001.htm>

From bbanister at jumptrading.com  Tue May  3 21:31:53 2016
From: bbanister at jumptrading.com (Bryan Banister)
Date: Tue, 3 May 2016 20:31:53 +0000
Subject: [gpfsug-discuss] CNFS and multiple IP addresses
In-Reply-To: <94CBCF33-0B97-440E-8E51-F4A51E9333FB@vanderbilt.edu>
References: <94CBCF33-0B97-440E-8E51-F4A51E9333FB@vanderbilt.edu>
Message-ID: <21BC488F0AEA2245B2C3E83FC0B33DBB060CA609@CHI-EXCHANGEW1.w2k.jumptrading.com>

Hi Kevin,

Look at the man page for mmchnode, you?ll need to use the ?--cnfs-interface=<ip_address_list>? option.

BTW, I get the cleanest looking man pages for GPFS commands by using the ?-E ascii? option.

You may also have a Round-Robin DNS record setup that includes the current IP addresses that you will likely want to update, not forgetting about DNS caching timeouts that you may need to let expire before no client attempts to mount from an old address.

Hope that helps!
-Bryan

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Buterbaugh, Kevin L
Sent: Tuesday, May 03, 2016 3:20 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] CNFS and multiple IP addresses

Hi All,

I?ve looked in the documentation and on the GPFS Wiki for the answer to this question but haven?t found it.  Please feel free to point me to docs / web pages if I?ve missed something.

One of the GPFS clusters I administer uses CNFS, which I did not set up on this cluster ? and the person who did is now an employee of another organization.  The cluster in question runs GPFS 4.1.0.x, although I doubt that matters.

There are 4 CNFS servers, each of which has 2 IP addresses listed under the CNFS IP address list when running ?mmlscluster ?cnfs?.  I want to delete one of those two IP addresses for each server due to a planned network reorganization.  How do I do that?  Or do I need to completely delete the server from CNFS and add it back in with just the one IP address I want to keep?

Thanks in advance?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633




________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160503/5ffdbf58/attachment-0001.htm>

From bbanister at jumptrading.com  Tue May  3 21:33:24 2016
From: bbanister at jumptrading.com (Bryan Banister)
Date: Tue, 3 May 2016 20:33:24 +0000
Subject: [gpfsug-discuss] Data replication and fileset quotas
In-Reply-To: <AEE0A7CC-E792-462F-9BD0-C6DF9C551E41@vanderbilt.edu>
References: <AEE0A7CC-E792-462F-9BD0-C6DF9C551E41@vanderbilt.edu>
Message-ID: <21BC488F0AEA2245B2C3E83FC0B33DBB060CA62F@CHI-EXCHANGEW1.w2k.jumptrading.com>

Yes, I believe that is the case due to the fact that every file is replicated therefore taking twice as much space within the file system.
-Bryan

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Buterbaugh, Kevin L
Sent: Tuesday, May 03, 2016 3:27 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] Data replication and fileset quotas

Hi again all,

I have another question on a completely different topic and therefore I decided it was better to send two separate e-mails.

For a number of years now we have had a GPFS filesystem where we use filesets and set fileset quotas.  Data replication is set to one.  I understand how that all works.

We are creating another GPFS filesystem where we intend to also use filesets and fileset quotas, but set data replication to two.  Based on my experience with data replication on a filesystem that doesn?t use filesets, I am expecting that setting data replication to two means that I will need to double the quota for each fileset (i.e. if a group has bought 5 TB of space I?ll need to set their fileset quota to 10 TB) but haven?t found where that is explicitly documented.  Is that correct?

Thanks again, all?

Kevin

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633




________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160503/67d1cbad/attachment-0001.htm>

From Kevin.Buterbaugh at Vanderbilt.Edu  Tue May  3 21:51:48 2016
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Tue, 3 May 2016 20:51:48 +0000
Subject: [gpfsug-discuss] CNFS and multiple IP addresses
In-Reply-To: <21BC488F0AEA2245B2C3E83FC0B33DBB060CA609@CHI-EXCHANGEW1.w2k.jumptrading.com>
References: <94CBCF33-0B97-440E-8E51-F4A51E9333FB@vanderbilt.edu>
	<21BC488F0AEA2245B2C3E83FC0B33DBB060CA609@CHI-EXCHANGEW1.w2k.jumptrading.com>
Message-ID: <FBB84DD9-9B12-4607-9D98-E7DD87C7C145@vanderbilt.edu>

Hi Bryan,

Thanks for the response.

I had looked at the man page for mmchnode previously, but was misinterpreting the wording.  I was under the impression that the ??cnfs-interface? option could only be used to add IP addresses or delete a node from CNFS with the ?DEFAULT? keyword.

Your explanation makes sense - thanks for clarifying?

Kevin

On May 3, 2016, at 3:31 PM, Bryan Banister <bbanister at jumptrading.com<mailto:bbanister at jumptrading.com>> wrote:

Hi Kevin,

Look at the man page for mmchnode, you?ll need to use the ?--cnfs-interface=<ip_address_list>? option.

BTW, I get the cleanest looking man pages for GPFS commands by using the ?-E ascii? option.

You may also have a Round-Robin DNS record setup that includes the current IP addresses that you will likely want to update, not forgetting about DNS caching timeouts that you may need to let expire before no client attempts to mount from an old address.

Hope that helps!
-Bryan

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Buterbaugh, Kevin L
Sent: Tuesday, May 03, 2016 3:20 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: [gpfsug-discuss] CNFS and multiple IP addresses

Hi All,

I?ve looked in the documentation and on the GPFS Wiki for the answer to this question but haven?t found it.  Please feel free to point me to docs / web pages if I?ve missed something.

One of the GPFS clusters I administer uses CNFS, which I did not set up on this cluster ? and the person who did is now an employee of another organization.  The cluster in question runs GPFS 4.1.0.x, although I doubt that matters.

There are 4 CNFS servers, each of which has 2 IP addresses listed under the CNFS IP address list when running ?mmlscluster ?cnfs?.  I want to delete one of those two IP addresses for each server due to a planned network reorganization.  How do I do that?  Or do I need to completely delete the server from CNFS and add it back in with just the one IP address I want to keep?

Thanks in advance?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633




________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160503/4af7fe04/attachment-0001.htm>

From S.J.Thompson at bham.ac.uk  Wed May  4 08:11:18 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Wed, 4 May 2016 07:11:18 +0000
Subject: [gpfsug-discuss] Data replication and fileset quotas
In-Reply-To: <21BC488F0AEA2245B2C3E83FC0B33DBB060CA62F@CHI-EXCHANGEW1.w2k.jumptrading.com>
References: <21BC488F0AEA2245B2C3E83FC0B33DBB060CA62F@CHI-EXCHANGEW1.w2k.jumptrading.com>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F576E8B6@EX13.adf.bham.ac.uk>




Yes, this correct (I think there was some discussion on the mailing list a little while back).

The quota is the space used on disk, so if you run compression its the compressed size, tape hsm don't count until you recall them.

I assume mcstore is the same, I.e. Only counts on recall, but I haven't tested this.

Simon

--

Simon Thompson

Research Computing Team, IT Services

+44 121 415 8675

-----Original Message-----
From: Bryan Banister [bbanister at jumptrading.com<mailto:bbanister at jumptrading.com>]
Sent: Tuesday, May 03, 2016 09:33 PM GMT Standard Time
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Data replication and fileset quotas

Yes, I believe that is the case due to the fact that every file is replicated therefore taking twice as much space within the file system.
-Bryan

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Buterbaugh, Kevin L
Sent: Tuesday, May 03, 2016 3:27 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] Data replication and fileset quotas

Hi again all,

I have another question on a completely different topic and therefore I decided it was better to send two separate e-mails.

For a number of years now we have had a GPFS filesystem where we use filesets and set fileset quotas.  Data replication is set to one.  I understand how that all works.

We are creating another GPFS filesystem where we intend to also use filesets and fileset quotas, but set data replication to two.  Based on my experience with data replication on a filesystem that doesn?t use filesets, I am expecting that setting data replication to two means that I will need to double the quota for each fileset (i.e. if a group has bought 5 TB of space I?ll need to set their fileset quota to 10 TB) but haven?t found where that is explicitly documented.  Is that correct?

Thanks again, all?

Kevin

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633




________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160504/34c11a8d/attachment-0001.htm>

From Robert.Oesterlin at nuance.com  Wed May  4 12:18:54 2016
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Wed, 4 May 2016 11:18:54 +0000
Subject: [gpfsug-discuss] Data replication and fileset quotas
Message-ID: <1C31AA0D-666C-4CB7-A54D-99CFC5AC8FAE@nuance.com>

From the documentation:

"When setting quota limits for a file system, replication within the file system should be considered. GPFS quota management takes replication into account when reporting on and determining if quota limits have been exceeded for both block and file usage. In a file system that has either type of replication set to a value of two, the values reported on by both the mmlsquota command and the mmrepquota command are double the value reported by the ls command."

Bob Oesterlin
Sr Storage Engineer, Nuance HPC Grid
507-269-0413


From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of "Simon Thompson (Research Computing - IT Services)" <S.J.Thompson at bham.ac.uk<mailto:S.J.Thompson at bham.ac.uk>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, May 4, 2016 at 2:11 AM
To: 'gpfsug main discussion list' <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Data replication and fileset quotas




Yes, this correct (I think there was some discussion on the mailing list a little while back).

The quota is the space used on disk, so if you run compression its the compressed size, tape hsm don't count until you recall them.

I assume mcstore is the same, I.e. Only counts on recall, but I haven't tested this.

Simon

--

Simon Thompson

Research Computing Team, IT Services

+44 121 415 8675

-----Original Message-----
From: Bryan Banister [bbanister at jumptrading.com<mailto:bbanister at jumptrading.com>]
Sent: Tuesday, May 03, 2016 09:33 PM GMT Standard Time
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Data replication and fileset quotas

Yes, I believe that is the case due to the fact that every file is replicated therefore taking twice as much space within the file system.
-Bryan

From:gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Buterbaugh, Kevin L
Sent: Tuesday, May 03, 2016 3:27 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: [gpfsug-discuss] Data replication and fileset quotas

Hi again all,

I have another question on a completely different topic and therefore I decided it was better to send two separate e-mails.

For a number of years now we have had a GPFS filesystem where we use filesets and set fileset quotas.  Data replication is set to one.  I understand how that all works.

We are creating another GPFS filesystem where we intend to also use filesets and fileset quotas, but set data replication to two.  Based on my experience with data replication on a filesystem that doesn?t use filesets, I am expecting that setting data replication to two means that I will need to double the quota for each fileset (i.e. if a group has bought 5 TB of space I?ll need to set their fileset quota to 10 TB) but haven?t found where that is explicitly documented.  Is that correct?

Thanks again, all?

Kevin

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633




________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160504/99bdbc87/attachment-0001.htm>

From daniel.kidger at uk.ibm.com  Wed May  4 12:36:06 2016
From: daniel.kidger at uk.ibm.com (Daniel Kidger)
Date: Wed, 4 May 2016 11:36:06 +0000
Subject: [gpfsug-discuss] Data replication and fileset quotas
In-Reply-To: <1C31AA0D-666C-4CB7-A54D-99CFC5AC8FAE@nuance.com>
References: <1C31AA0D-666C-4CB7-A54D-99CFC5AC8FAE@nuance.com>
Message-ID: <201605041136.u44BaBZT029213@d06av06.portsmouth.uk.ibm.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160504/597bfe75/attachment-0001.htm>

From D.R.Barker at exeter.ac.uk  Wed May  4 13:38:23 2016
From: D.R.Barker at exeter.ac.uk (Barker, David)
Date: Wed, 4 May 2016 12:38:23 +0000
Subject: [gpfsug-discuss] Data replication and fileset quotas
In-Reply-To: <201605041136.u44BaBZT029213@d06av06.portsmouth.uk.ibm.com>
References: <1C31AA0D-666C-4CB7-A54D-99CFC5AC8FAE@nuance.com>
	<201605041136.u44BaBZT029213@d06av06.portsmouth.uk.ibm.com>
Message-ID: <DB5PR03MB1366BB56598980F182E8B18CE67B0@DB5PR03MB1366.eurprd03.prod.outlook.com>

Hi Daniel,

It?s funny you mention compression & quotas ? it?s something I?ve hit on a.n.other system. The supplier ended up providing different ways of presenting quotas as a mount option. Options are:


-          Quotas are tracked using the post compression/dedupe actually used space. (i.e. users see the benefit/drawback of compression)

-          Quotas are tracked using the pre compression/dedupe filesize. (i.e. the sysadmin sees the benefit; users are unaware what?s happening underneath).

Would something similar be possible in GPFS land?

David

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Daniel Kidger
Sent: 04 May 2016 12:36
To: gpfsug-discuss at spectrumscale.org
Cc: gpfsug-discuss at spectrumscale.org
Subject: Re: [gpfsug-discuss] Data replication and fileset quotas

As Simon says, Quota measures used blocks in the filesystem.
Hence users can and should have behaviour that keeps within these limits.

GPFS Replication though is a system-administrator level concept - to protect data access in the case of power outages or though gross hardware failures. So as such should be transparent to the end users.

Unless users are enabled to choose 1 or 2 (or 3) way replication of their own files dependent on their importance (eg 1 copy for scratch files) then imho replication should not be measured in quota reporting.

On a related note, compression is great new feature, but it may confuse users if they delete some older but big 100GB files then try and recreate them only to find they can't because their quota is now exceeded (as compression is not at file creation but driven later by policies.

Thoughts?
Daniel






Dr Daniel Kidger
IBM Technical Sales Specialist
Software Defined Solution Sales

+44-07818 522 266
daniel.kidger at uk.ibm.com<mailto:daniel.kidger at uk.ibm.com>







----- Original message -----
From: "Oesterlin, Robert" <Robert.Oesterlin at nuance.com<mailto:Robert.Oesterlin at nuance.com>>
Sent by: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc:
Subject: Re: [gpfsug-discuss] Data replication and fileset quotas
Date: Wed, May 4, 2016 12:19 PM

From the documentation:

"When setting quota limits for a file system, replication within the file system should be considered. GPFS quota management takes replication into account when reporting on and determining if quota limits have been exceeded for both block and file usage. In a file system that has either type of replication set to a value of two, the values reported on by both the mmlsquota command and the mmrepquota command are double the value reported by the ls command."

Bob Oesterlin
Sr Storage Engineer, Nuance HPC Grid
507-269-0413


From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of "Simon Thompson (Research Computing - IT Services)" <S.J.Thompson at bham.ac.uk<mailto:S.J.Thompson at bham.ac.uk>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, May 4, 2016 at 2:11 AM
To: 'gpfsug main discussion list' <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Data replication and fileset quotas




Yes, this correct (I think there was some discussion on the mailing list a little while back).

The quota is the space used on disk, so if you run compression its the compressed size, tape hsm don't count until you recall them.

I assume mcstore is the same, I.e. Only counts on recall, but I haven't tested this.

Simon

--

Simon Thompson

Research Computing Team, IT Services

+44 121 415 8675

-----Original Message-----
From: Bryan Banister [bbanister at jumptrading.com<mailto:bbanister at jumptrading.com>]
Sent: Tuesday, May 03, 2016 09:33 PM GMT Standard Time
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Data replication and fileset quotas

Yes, I believe that is the case due to the fact that every file is replicated therefore taking twice as much space within the file system.

-Bryan



From:gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Buterbaugh, Kevin L
Sent: Tuesday, May 03, 2016 3:27 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: [gpfsug-discuss] Data replication and fileset quotas



Hi again all,



I have another question on a completely different topic and therefore I decided it was better to send two separate e-mails.



For a number of years now we have had a GPFS filesystem where we use filesets and set fileset quotas.  Data replication is set to one.  I understand how that all works.



We are creating another GPFS filesystem where we intend to also use filesets and fileset quotas, but set data replication to two.  Based on my experience with data replication on a filesystem that doesn?t use filesets, I am expecting that setting data replication to two means that I will need to double the quota for each fileset (i.e. if a group has bought 5 TB of space I?ll need to set their fileset quota to 10 TB) but haven?t found where that is explicitly documented.  Is that correct?



Thanks again, all?



Kevin



?

Kevin Buterbaugh - Senior System Administrator

Vanderbilt University - Advanced Computing Center for Research and Education

Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633







________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160504/46e3773a/attachment-0001.htm>

From r.sobey at imperial.ac.uk  Wed May  4 16:32:32 2016
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Wed, 4 May 2016 15:32:32 +0000
Subject: [gpfsug-discuss] TSM and re-linked filesets
Message-ID: <AMSPR06MB4053E885E19DC5F12F46F74DF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>

This is more a TSM question than GPFS but it involves GPFS to a point so I hope I'm ok asking this.

I've unlinked a fileset and relinked it at a different junction point. Will this cause TSM to see a "new" fileset and back up all its content? This is very undesireable so if I need to put it back where it belongs then I need to do this before tonight.

GPFS 3.5 and TSM 7.1.

Does anyone know?

Cheers

Richard

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160504/7399359a/attachment-0001.htm>

From r.sobey at imperial.ac.uk  Wed May  4 16:35:29 2016
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Wed, 4 May 2016 15:35:29 +0000
Subject: [gpfsug-discuss] TSM and re-linked filesets
In-Reply-To: <AMSPR06MB4053E885E19DC5F12F46F74DF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
References: <AMSPR06MB4053E885E19DC5F12F46F74DF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
Message-ID: <AMSPR06MB4053EB49583DA7B3623505EDF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>

A quick Google has given me the answer I dreaded:

"Attention: If you are using the TSM Backup-Archive client you must use caution when you unlink filesets that contain data backed up by TSM. TSM tracks files by pathname and does not track filesets. As a result, when you unlink a fileset, it appears to TSM that you deleted the contents of the fileset. Therefore, the TSM Backup-Archive client inactivates the data on the TSM server which may result in the loss of backup data during the expiration process."

So I need to relink it to the original place I think. Oh well. Not a disaster.

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A
Sent: 04 May 2016 16:33
To: 'gpfsug-discuss at spectrumscale.org' <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] TSM and re-linked filesets

This is more a TSM question than GPFS but it involves GPFS to a point so I hope I'm ok asking this.

I've unlinked a fileset and relinked it at a different junction point. Will this cause TSM to see a "new" fileset and back up all its content? This is very undesireable so if I need to put it back where it belongs then I need to do this before tonight.

GPFS 3.5 and TSM 7.1.

Does anyone know?

Cheers

Richard

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160504/f6c35dd3/attachment-0001.htm>

From S.J.Thompson at bham.ac.uk  Wed May  4 16:59:27 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Wed, 4 May 2016 15:59:27 +0000
Subject: [gpfsug-discuss] TSM and re-linked filesets
In-Reply-To: <AMSPR06MB4053EB49583DA7B3623505EDF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
References: <AMSPR06MB4053EB49583DA7B3623505EDF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F576F2A9@EX13.adf.bham.ac.uk>

Could you do something really nasty in the tsm db2 database to update the paths?

-----Original Message-----
From: Sobey, Richard A [r.sobey at imperial.ac.uk<mailto:r.sobey at imperial.ac.uk>]
Sent: Wednesday, May 04, 2016 04:35 PM GMT Standard Time
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] TSM and re-linked filesets

A quick Google has given me the answer I dreaded:

?Attention: If you are using the TSM Backup-Archive client you must use caution when you unlink filesets that contain data backed up by TSM. TSM tracks files by pathname and does not track filesets. As a result, when you unlink a fileset, it appears to TSM that you deleted the contents of the fileset. Therefore, the TSM Backup-Archive client inactivates the data on the TSM server which may result in the loss of backup data during the expiration process.?

So I need to relink it to the original place I think. Oh well. Not a disaster.

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A
Sent: 04 May 2016 16:33
To: 'gpfsug-discuss at spectrumscale.org' <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] TSM and re-linked filesets

This is more a TSM question than GPFS but it involves GPFS to a point so I hope I?m ok asking this.

I?ve unlinked a fileset and relinked it at a different junction point. Will this cause TSM to see a ?new? fileset and back up all its content? This is very undesireable so if I need to put it back where it belongs then I need to do this before tonight.

GPFS 3.5 and TSM 7.1.

Does anyone know?

Cheers

Richard

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160504/7874b433/attachment-0001.htm>

From makaplan at us.ibm.com  Wed May  4 17:14:44 2016
From: makaplan at us.ibm.com (Marc A Kaplan)
Date: Wed, 4 May 2016 12:14:44 -0400
Subject: [gpfsug-discuss] TSM and re-linked filesets
In-Reply-To: <AMSPR06MB4053EB49583DA7B3623505EDF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
References: <AMSPR06MB4053E885E19DC5F12F46F74DF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
	<AMSPR06MB4053EB49583DA7B3623505EDF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
Message-ID: <201605041614.u44GEnbI031747@d03av03.boulder.ibm.com>

I think you found your answer:  TSM tracks files by pathname. 

So... if a file had path /w/x/y/z on Monday.  But was moved to /w/x/q/p on 
Tuesday, how would TSM "know" it was the same file...?
It wouldn't! To TSM it seems you've deleted the first and created the 
second.

Technically there are some other possibilities, and some backup systems 
may use them, but NOT TSM:

1) Record the inode number and generation number and/or creation 
timestamp.   Within a given Posix-ish file system, that uniquely 
identifies the file.

2) Record a strong (cryptographic quality) checksum (hash) of the contents 
of the file.  If two files have the same checksum (hash) then the odds are
we can use the same backup data for both and don't have to make an extra 
copy in the backup system.    To make the odds really, really "long" you 
want to
take into account the "birthday paradox" and use lots and lots of bits. 
Long odds can also be compared to the probability of losing a file due to 
a bug
or an IO error or accident or disaster...

For example SHA-256, might be strong and long enough for you to believe 
in.
Backup is not generally a cryptographic game, so perhaps you should not 
worry much about some evil doer purposely trying to confound your backup 
system.
OTOH - if you have users who are adversaries, all backing up into the same 
system... In theory one might "destroy" another's backup.

This save transmission and storage of duplicates, but of course the backup 
system has to read the contents of each suspected new file and compute the 
hash...



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160504/b5aca332/attachment-0001.htm>

From laurence at qsplace.co.uk  Wed May  4 18:49:31 2016
From: laurence at qsplace.co.uk (Laurence Horrors-Barlow)
Date: Wed, 4 May 2016 18:49:31 +0100
Subject: [gpfsug-discuss] TSM and re-linked filesets
In-Reply-To: <AMSPR06MB4053EB49583DA7B3623505EDF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
References: <AMSPR06MB4053E885E19DC5F12F46F74DF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
	<AMSPR06MB4053EB49583DA7B3623505EDF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
Message-ID: <7DCCBF95-92CF-4473-8BAB-F845C5408A46@qsplace.co.uk>

Hi Richard,

That's correct, TSM will only look at the path and compare files it doesn't understand the file set. Once the path has changed it will then start expiring the missing files.

What are you trying to achieve here as you might be able to update the expiration or fudge the DB.

-- Lauz

Sent from my iPad

> On 4 May 2016, at 16:35, Sobey, Richard A <r.sobey at imperial.ac.uk> wrote:
> 
> A quick Google has given me the answer I dreaded:
>  
> ?Attention: If you are using the TSM Backup-Archive client you must use caution when you unlink filesets that contain data backed up by TSM. TSM tracks files by pathname and does not track filesets. As a result, when you unlink a fileset, it appears to TSM that you deleted the contents of the fileset. Therefore, the TSM Backup-Archive client inactivates the data on the TSM server which may result in the loss of backup data during the expiration process.?
>  
> So I need to relink it to the original place I think. Oh well. Not a disaster.
>  
> From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A
> Sent: 04 May 2016 16:33
> To: 'gpfsug-discuss at spectrumscale.org' <gpfsug-discuss at spectrumscale.org>
> Subject: [gpfsug-discuss] TSM and re-linked filesets
>  
> This is more a TSM question than GPFS but it involves GPFS to a point so I hope I?m ok asking this.
>  
> I?ve unlinked a fileset and relinked it at a different junction point. Will this cause TSM to see a ?new? fileset and back up all its content? This is very undesireable so if I need to put it back where it belongs then I need to do this before tonight.
>  
> GPFS 3.5 and TSM 7.1.
>  
> Does anyone know?
>  
> Cheers
>  
> Richard
>  
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160504/4e625d03/attachment-0001.htm>

From service at metamodul.com  Wed May  4 19:15:29 2016
From: service at metamodul.com (service at metamodul.com)
Date: Wed, 04 May 2016 20:15:29 +0200
Subject: [gpfsug-discuss] TSM and re-linked filesets
Message-ID: <tl3ri9d2ub9oacdlv1vloej2.1462385341788@email.android.com>

Have thought about the use of a submount ?
Meaning you link your fileset to the new directory and mount that dir on the old dir or you do not unlink at all but submount the old directory at the new directory.


Von Samsung Mobile gesendet

<div>-------- Urspr?ngliche Nachricht --------</div><div>Von: "Sobey, Richard A" <r.sobey at imperial.ac.uk> </div><div>Datum:2016.05.04  17:32  (GMT+01:00) </div><div>An: "'gpfsug-discuss at spectrumscale.org'" <gpfsug-discuss at spectrumscale.org> </div><div>Betreff: [gpfsug-discuss] TSM and re-linked filesets </div><div>
</div>This is more a TSM question than GPFS but it involves GPFS to a point so I hope I?m ok asking this.
 
I?ve unlinked a fileset and relinked it at a different junction point. Will this cause TSM to see a ?new? fileset and back up all its content? This is very undesireable so if I need to put it back where it belongs then I need to do this before tonight.
 
GPFS 3.5 and TSM 7.1.
 
Does anyone know?
 
Cheers
 
Richard
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160504/9a4c0a0d/attachment-0001.htm>

From r.sobey at imperial.ac.uk  Thu May  5 09:25:02 2016
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Thu, 5 May 2016 08:25:02 +0000
Subject: [gpfsug-discuss] TSM and re-linked filesets
In-Reply-To: <7DCCBF95-92CF-4473-8BAB-F845C5408A46@qsplace.co.uk>
References: <AMSPR06MB4053E885E19DC5F12F46F74DF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
	<AMSPR06MB4053EB49583DA7B3623505EDF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
	<7DCCBF95-92CF-4473-8BAB-F845C5408A46@qsplace.co.uk>
Message-ID: <AMSPR06MB405B89D8C47300501E4A447DF7C0@AMSPR06MB405.eurprd06.prod.outlook.com>

Hi Lauz

When we want to remove filesets we link them in an area that isn?t exported by CIFS or NFS. This allows us a grace period where the users most definitely cannot access the files before we delete them permanently.

Normally it?s not a problem as most of our filesets are only a few hundred GB, but this particular one contains 80TB of data (160TB replicated) and I don?t fancy it getting backed up again!

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Laurence Horrors-Barlow
Sent: 04 May 2016 18:50
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] TSM and re-linked filesets

Hi Richard,

That's correct, TSM will only look at the path and compare files it doesn't understand the file set. Once the path has changed it will then start expiring the missing files.

What are you trying to achieve here as you might be able to update the expiration or fudge the DB.

-- Lauz

Sent from my iPad

On 4 May 2016, at 16:35, Sobey, Richard A <r.sobey at imperial.ac.uk<mailto:r.sobey at imperial.ac.uk>> wrote:
A quick Google has given me the answer I dreaded:

?Attention: If you are using the TSM Backup-Archive client you must use caution when you unlink filesets that contain data backed up by TSM. TSM tracks files by pathname and does not track filesets. As a result, when you unlink a fileset, it appears to TSM that you deleted the contents of the fileset. Therefore, the TSM Backup-Archive client inactivates the data on the TSM server which may result in the loss of backup data during the expiration process.?

So I need to relink it to the original place I think. Oh well. Not a disaster.

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A
Sent: 04 May 2016 16:33
To: 'gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>' <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: [gpfsug-discuss] TSM and re-linked filesets

This is more a TSM question than GPFS but it involves GPFS to a point so I hope I?m ok asking this.

I?ve unlinked a fileset and relinked it at a different junction point. Will this cause TSM to see a ?new? fileset and back up all its content? This is very undesireable so if I need to put it back where it belongs then I need to do this before tonight.

GPFS 3.5 and TSM 7.1.

Does anyone know?

Cheers

Richard

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/f8406628/attachment-0001.htm>

From r.sobey at imperial.ac.uk  Thu May  5 09:25:49 2016
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Thu, 5 May 2016 08:25:49 +0000
Subject: [gpfsug-discuss] TSM and re-linked filesets
In-Reply-To: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F576F2A9@EX13.adf.bham.ac.uk>
References: <AMSPR06MB4053EB49583DA7B3623505EDF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
	<CF45EE16DEF2FE4B9AA7FF2B6EE26545F576F2A9@EX13.adf.bham.ac.uk>
Message-ID: <AMSPR06MB40548B61A9C4917DBA25F2BDF7C0@AMSPR06MB405.eurprd06.prod.outlook.com>

Err, I think you?ve already answered the question with the word ?nasty? heh.

Thanks for all the suggestions folks.

Richard

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (Research Computing - IT Services)
Sent: 04 May 2016 16:59
To: 'gpfsug main discussion list' <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] TSM and re-linked filesets

Could you do something really nasty in the tsm db2 database to update the paths?

-----Original Message-----
From: Sobey, Richard A [r.sobey at imperial.ac.uk<mailto:r.sobey at imperial.ac.uk>]
Sent: Wednesday, May 04, 2016 04:35 PM GMT Standard Time
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] TSM and re-linked filesets


A quick Google has given me the answer I dreaded:

?Attention: If you are using the TSM Backup-Archive client you must use caution when you unlink filesets that contain data backed up by TSM. TSM tracks files by pathname and does not track filesets. As a result, when you unlink a fileset, it appears to TSM that you deleted the contents of the fileset. Therefore, the TSM Backup-Archive client inactivates the data on the TSM server which may result in the loss of backup data during the expiration process.?

So I need to relink it to the original place I think. Oh well. Not a disaster.

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A
Sent: 04 May 2016 16:33
To: 'gpfsug-discuss at spectrumscale.org' <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: [gpfsug-discuss] TSM and re-linked filesets

This is more a TSM question than GPFS but it involves GPFS to a point so I hope I?m ok asking this.

I?ve unlinked a fileset and relinked it at a different junction point. Will this cause TSM to see a ?new? fileset and back up all its content? This is very undesireable so if I need to put it back where it belongs then I need to do this before tonight.

GPFS 3.5 and TSM 7.1.

Does anyone know?

Cheers

Richard

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/284f5a55/attachment-0001.htm>

From stijn.deweirdt at ugent.be  Thu May  5 09:33:45 2016
From: stijn.deweirdt at ugent.be (Stijn De Weirdt)
Date: Thu, 5 May 2016 10:33:45 +0200
Subject: [gpfsug-discuss] GPFS VM passthrough
Message-ID: <572B0569.1000201@ugent.be>

hi all,


we are examining the possibility to give VMs access to GPFS mounted on
the hypervisors. (use cases are readonly or read-mostly access to the
data on gpfs, the hypervisors have IB, the vms not yet; and we have no
idea how to handle possible explosion of gpfs client licenses if we
mount gpfs inside the VMs ;)

has anyone experience with using the 9p filesystem passthrough and gpfs
with qemu/kvm?

many thanks,

stijn


From laurence at qsplace.co.uk  Thu May  5 09:48:25 2016
From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow)
Date: Thu, 05 May 2016 09:48:25 +0100
Subject: [gpfsug-discuss] GPFS VM passthrough
In-Reply-To: <572B0569.1000201@ugent.be>
References: <572B0569.1000201@ugent.be>
Message-ID: <FF599E3E-FFEE-46B3-B2F1-BB1E0C882B6A@qsplace.co.uk>

I have a few KVM systems that use 9p to pass GPFS through and it does just work; although I've not really benchmarked or pushed it yet.

Regarding licencing, its my understanding that the guest inherits the hosts licence i.e. if you have a client license on the host all vms are covered by the same license (iirc there is a core/socket caveat) So that is worth exploring further.

-- Lauz

On 5 May 2016 09:33:45 BST, Stijn De Weirdt <stijn.deweirdt at ugent.be> wrote:
>hi all,
>
>
>we are examining the possibility to give VMs access to GPFS mounted on
>the hypervisors. (use cases are readonly or read-mostly access to the
>data on gpfs, the hypervisors have IB, the vms not yet; and we have no
>idea how to handle possible explosion of gpfs client licenses if we
>mount gpfs inside the VMs ;)
>
>has anyone experience with using the 9p filesystem passthrough and gpfs
>with qemu/kvm?
>
>many thanks,
>
>stijn
>_______________________________________________
>gpfsug-discuss mailing list
>gpfsug-discuss at spectrumscale.org
>http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/6f03a920/attachment-0002.htm>

From laurence at qsplace.co.uk  Thu May  5 09:48:25 2016
From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow)
Date: Thu, 05 May 2016 09:48:25 +0100
Subject: [gpfsug-discuss] GPFS VM passthrough
In-Reply-To: <572B0569.1000201@ugent.be>
References: <572B0569.1000201@ugent.be>
Message-ID: <FF599E3E-FFEE-46B3-B2F1-BB1E0C882B6A@qsplace.co.uk>

I have a few KVM systems that use 9p to pass GPFS through and it does just work; although I've not really benchmarked or pushed it yet.

Regarding licencing, its my understanding that the guest inherits the hosts licence i.e. if you have a client license on the host all vms are covered by the same license (iirc there is a core/socket caveat) So that is worth exploring further.

-- Lauz

On 5 May 2016 09:33:45 BST, Stijn De Weirdt <stijn.deweirdt at ugent.be> wrote:
>hi all,
>
>
>we are examining the possibility to give VMs access to GPFS mounted on
>the hypervisors. (use cases are readonly or read-mostly access to the
>data on gpfs, the hypervisors have IB, the vms not yet; and we have no
>idea how to handle possible explosion of gpfs client licenses if we
>mount gpfs inside the VMs ;)
>
>has anyone experience with using the 9p filesystem passthrough and gpfs
>with qemu/kvm?
>
>many thanks,
>
>stijn
>_______________________________________________
>gpfsug-discuss mailing list
>gpfsug-discuss at spectrumscale.org
>http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/6f03a920/attachment-0003.htm>

From daniel.kidger at uk.ibm.com  Thu May  5 09:52:22 2016
From: daniel.kidger at uk.ibm.com (Daniel Kidger)
Date: Thu, 5 May 2016 08:52:22 +0000
Subject: [gpfsug-discuss] GPFS VM passthrough
In-Reply-To: <572B0569.1000201@ugent.be>
Message-ID: <201605050852.u458qSQn011217@d06av04.portsmouth.uk.ibm.com>

Fortunately the licences issue is simple. 
The rules state that the sum of licences of all VMs on a node is capped at the number of sockets that server has.
So if you have 9 VMs using a mixture of different core counts each then you do not need more than 2 client licences if all hosted on a standard two socket server.

Daniel

Sent from IBM Verse


   Stijn De Weirdt --- [gpfsug-discuss] GPFS VM passthrough --- 
    From:"Stijn De Weirdt" <stijn.deweirdt at ugent.be>To:"gpfsug main discussion list" <gpfsug-discuss at gpfsug.org>Date:Thu, 5 May 2016 09:34Subject:[gpfsug-discuss] GPFS VM passthrough
  
    hi all,we are examining the possibility to give VMs access to GPFS mounted onthe hypervisors. (use cases are readonly or read-mostly access to thedata on gpfs, the hypervisors have IB, the vms not yet; and we have noidea how to handle possible explosion of gpfs client licenses if wemount gpfs inside the VMs ;)has anyone experience with using the 9p filesystem passthrough and gpfswith qemu/kvm?many thanks,stijn_______________________________________________gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discussUnless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/6fc5b68f/attachment-0001.htm>

From makaplan at us.ibm.com  Thu May  5 16:26:33 2016
From: makaplan at us.ibm.com (Marc A Kaplan)
Date: Thu, 5 May 2016 11:26:33 -0400
Subject: [gpfsug-discuss] TSM and re-linked filesets
In-Reply-To: <AMSPR06MB40548B61A9C4917DBA25F2BDF7C0@AMSPR06MB405.eurprd06.prod.outlook.com>
References: <AMSPR06MB4053EB49583DA7B3623505EDF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
	<CF45EE16DEF2FE4B9AA7FF2B6EE26545F576F2A9@EX13.adf.bham.ac.uk>
	<AMSPR06MB40548B61A9C4917DBA25F2BDF7C0@AMSPR06MB405.eurprd06.prod.outlook.com>
Message-ID: <201605051526.u45FQgxg012902@d03av02.boulder.ibm.com>

My little suggestion is to put it back where it was.  And then add a 
symlink from the desired (new) path to the TSM-needs-this actual path.



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/36637b8b/attachment-0001.htm>

From syi at ca.ibm.com  Thu May  5 22:09:49 2016
From: syi at ca.ibm.com (Yi Sun)
Date: Thu, 5 May 2016 17:09:49 -0400
Subject: [gpfsug-discuss] SPXXL meeting in NYC
Message-ID: <201605052109.u45L9vJ6015648@d03av01.boulder.ibm.com>





Hi Kristy and Bob,


For SPXXL meeting, if only interested in GPFS/Spectrum Scale day, is it
still nessary to make registration to SPXXL meeting?


Regards,


Yi Sun


syi at ca.ibm.com



>
>
> Message: 1
> Date: Mon, 4 Apr 2016 16:52:37 -0400
> From: GPFS UG USA Principal <usa-principal at gpfsug.org>
> To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
> Subject: [gpfsug-discuss] GPFS/Spectrum Scale Upcoming US Events -
> 		 		  Save the		 		  Dates
> Message-ID: <4192D644-E3AB-4B7B-AF7A-96D3D617FA7B at gpfsug.org>
> Content-Type: text/plain; charset="utf-8"
>
> Hello all,
>
> 		 		  We?d like to announce two upcoming US
GPFS/Spectrum Scale
> Events. One on the east coast, one in the midwest.
>
> 1) May 26th (full day event): GPFS/Spectrum Scale Day at the SPXXL
> conference in NYC https://www.spxxl.org/?q=New-York-City-2016 <
> https://www.spxxl.org/?q=New-York-City-2016> Developers and Engineers
from
> IBM will be at the meeting to cover topics, and open dialogue between IBM
> and customers will be encouraged. More details about the agenda, hotel
and
> other logistics will come later this month.
>
> Tentative Agenda:
> 		 		  ? 10 reasons for upgrading from GPFS 3.5 to
Spectrum Scale
> 4.2.1
> 		 		  ? Enhancements for CORAL from IBM
> 		 		  ? Panel discussion with customers, topic TBD
> 		 		  ? AFM and integration with Spectrum Protect
> 		 		  ? Best practices for GPFS or Spectrum Scale
Tuning.
> 		 		  ? At least one site update
>
> Location:
> New York Academy of Medicine
> 1216 Fifth Avenue
> New York, NY 10029
>
> ??
>
> 2) June 10th (full day event): GPFS/Spectrum Scale Users Group Meeting at
> Argonne National Lab (ANL). Thanks to Argonne for hosting this event.
> Developers and Engineers from IBM will be at the meeting to cover topics,
> and open dialogue between IBM and customers will be encouraged, as usual
no
> marketing pitches!
>
> Location: Argonne National Lab more details and final agenda will come
> later.
>
> Tentative Agenda:
>
> 9:00a-12:30p
> 9-9:30a - Opening Remarks
> 9:30-10a Deep Dive - Update on ESS
> 10a-11a Deep Dive - Problem Determination (Presentation 30 min/Panel 30
> min?)
> 11-11:30 Break
> 11:30a-Noon - Deep Dive - Protect & Scale integration
> Noon-12:30p HDFS/Hadoop
>
> 12:30 - 1:30p Lunch
>
> 1:30p-5:00p
> 1:30 - 2:00p IBM AFM Update
> 2:00-2:30p ANL: AFM as a burst buffer
> 2:30-3:00p ANL: GHI (GPFS HPSS Integration)
> 3:00-3:30p Break
> 3:30p - 4:00p LANL: ? or other site preso
> 4:00-4:30p Nuance: GPFS Performance Sensors Deployment Experiences
> 4:30p -5:00p Closing comments and Open Forum for Questions
>
> 5:00 - ?
> Beer hunting?
>
> ??
>
> We hope you can attend one or both of these events.
>
> Best,
> Kristy Kallback-Rose & Bob Oesterlin
> GPFS Users Group - USA Chapter - Principal & Co-principal
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
>
http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160404/2e191559/attachment-0001.html

> >
>
> ------------------------------
>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/d03f7737/attachment-0001.htm>

From kallbac at iu.edu  Thu May  5 22:31:43 2016
From: kallbac at iu.edu (Kallback-Rose, Kristy A)
Date: Thu, 5 May 2016 17:31:43 -0400
Subject: [gpfsug-discuss] SPXXL meeting in NYC
In-Reply-To: <201605052109.u45L9vJ6015648@d03av01.boulder.ibm.com>
References: <201605052109.u45L9vJ6015648@d03av01.boulder.ibm.com>
Message-ID: <106AD35B-EC40-45D6-A4C9-42F5C86600CB@iu.edu>

Yes, you should register so they know how many will attend and have seating/food. There is a $0 option for the GPFS day. Thanks for asking! If you have another question, let us know.

-Kristy

> On May 5, 2016, at 5:09 PM, Yi Sun <syi at ca.ibm.com> wrote:
> 
> Hi Kristy and Bob,
> 
> For SPXXL meeting, if only interested in GPFS/Spectrum Scale day, is it still nessary to make registration to SPXXL meeting?
> 
> Regards,
> 
> Yi Sun
> 
> syi at ca.ibm.com
> 
> 
> >
> >
> > Message: 1
> > Date: Mon, 4 Apr 2016 16:52:37 -0400
> > From: GPFS UG USA Principal <usa-principal at gpfsug.org>
> > To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
> > Subject: [gpfsug-discuss] GPFS/Spectrum Scale Upcoming US Events -
> >  Save the  Dates
> > Message-ID: <4192D644-E3AB-4B7B-AF7A-96D3D617FA7B at gpfsug.org>
> > Content-Type: text/plain; charset="utf-8"
> >
> > Hello all,
> >
> >  We?d like to announce two upcoming US GPFS/Spectrum Scale
> > Events. One on the east coast, one in the midwest.
> >
> > 1) May 26th (full day event): GPFS/Spectrum Scale Day at the SPXXL
> > conference in NYC https://www.spxxl.org/?q=New-York-City-2016 <https://www.spxxl.org/?q=New-York-City-2016> <
> > https://www.spxxl.org/?q=New-York-City-2016 <https://www.spxxl.org/?q=New-York-City-2016>> Developers and Engineers from
> > IBM will be at the meeting to cover topics, and open dialogue between IBM
> > and customers will be encouraged. More details about the agenda, hotel and
> > other logistics will come later this month.
> >
> > Tentative Agenda:
> >  ? 10 reasons for upgrading from GPFS 3.5 to Spectrum Scale
> > 4.2.1
> >  ? Enhancements for CORAL from IBM
> >  ? Panel discussion with customers, topic TBD
> >  ? AFM and integration with Spectrum Protect
> >  ? Best practices for GPFS or Spectrum Scale Tuning.
> >  ? At least one site update
> >
> > Location:
> > New York Academy of Medicine
> > 1216 Fifth Avenue
> > New York, NY 10029
> >
> > ??
> >
> > 2) June 10th (full day event): GPFS/Spectrum Scale Users Group Meeting at
> > Argonne National Lab (ANL). Thanks to Argonne for hosting this event.
> > Developers and Engineers from IBM will be at the meeting to cover topics,
> > and open dialogue between IBM and customers will be encouraged, as usual no
> > marketing pitches!
> >
> > Location: Argonne National Lab more details and final agenda will come
> > later.
> >
> > Tentative Agenda:
> >
> > 9:00a-12:30p
> > 9-9:30a - Opening Remarks
> > 9:30-10a Deep Dive - Update on ESS
> > 10a-11a Deep Dive - Problem Determination (Presentation 30 min/Panel 30
> > min?)
> > 11-11:30 Break
> > 11:30a-Noon - Deep Dive - Protect & Scale integration
> > Noon-12:30p HDFS/Hadoop
> >
> > 12:30 - 1:30p Lunch
> >
> > 1:30p-5:00p
> > 1:30 - 2:00p IBM AFM Update
> > 2:00-2:30p ANL: AFM as a burst buffer
> > 2:30-3:00p ANL: GHI (GPFS HPSS Integration)
> > 3:00-3:30p Break
> > 3:30p - 4:00p LANL: ? or other site preso
> > 4:00-4:30p Nuance: GPFS Performance Sensors Deployment Experiences
> > 4:30p -5:00p Closing comments and Open Forum for Questions
> >
> > 5:00 - ?
> > Beer hunting?
> >
> > ??
> >
> > We hope you can attend one or both of these events.
> >
> > Best,
> > Kristy Kallback-Rose & Bob Oesterlin
> > GPFS Users Group - USA Chapter - Principal & Co-principal
> > -------------- next part --------------
> > An HTML attachment was scrubbed...
> > URL: <
> > http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160404/2e191559/attachment-0001.html <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160404/2e191559/attachment-0001.html>
> > >
> >
> > ------------------------------
> >
> 
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/5ebe30d2/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/5ebe30d2/attachment-0001.sig>

From syi at ca.ibm.com  Thu May  5 22:36:05 2016
From: syi at ca.ibm.com (Yi Sun)
Date: Thu, 5 May 2016 17:36:05 -0400
Subject: [gpfsug-discuss] SPXXL meeting in NYC
Message-ID: <201605052136.u45LaEjv002729@d03av03.boulder.ibm.com>



Thank you Kristy.

=============================================================================================================
Date: Thu, 5 May 2016 17:31:43 -0400
From: "Kallback-Rose, Kristy A" <kallbac at iu.edu>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] SPXXL meeting in NYC
Message-ID: <106AD35B-EC40-45D6-A4C9-42F5C86600CB at iu.edu>
Content-Type: text/plain; charset="us-ascii"

Yes, you should register so they know how many will attend and have
seating/food. There is a $0 option for the GPFS day. Thanks for asking! If
you have another question, let us know.

-Kristy

> On May 5, 2016, at 5:09 PM, Yi Sun <syi at ca.ibm.com> wrote:
>
> Hi Kristy and Bob,
>
> For SPXXL meeting, if only interested in GPFS/Spectrum Scale day, is it
still nessary to make registration to SPXXL meeting?
>
> Regards,
>
> Yi Sun
>
> syi at ca.ibm.com




Regards,



Yi (Y.) Sun
Technical Account Manager
IBM SDI
IBM Systems


                                                                                      
                                                                                      
                                                                                      
 Phone: 1-905-316-2681                                                          IBM   
 E-mail: syi at ca.ibm.com                                                               
                                                           3600 Steeles Avenue East   
                                                                Markham, ON L3R 9Z7   
                                                                             Canada   
                                                                                      
                                                                                      
                                                                                      





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/2fb39b0e/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0C466617.gif
Type: image/gif
Size: 2022 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/2fb39b0e/attachment-0002.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ecblank.gif
Type: image/gif
Size: 45 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/2fb39b0e/attachment-0003.gif>

From TROPPENS at de.ibm.com  Tue May 10 13:00:16 2016
From: TROPPENS at de.ibm.com (Ulf Troppens)
Date: Tue, 10 May 2016 14:00:16 +0200
Subject: [gpfsug-discuss] GPFS/Spectrum Scale Upcoming US Events - Save
 the Dates - Agenda for New York May 26
In-Reply-To: <201605031206.u43C5wss002725@d06av09.portsmouth.uk.ibm.com>
References: <4192D644-E3AB-4B7B-AF7A-96D3D617FA7B@gpfsug.org><9489DBA2-1F12-4B05-A968-5D4855FBEA40@gpfsug.org>
	<201605031206.u43C5wss002725@d06av09.portsmouth.uk.ibm.com>
Message-ID: <201605101100.u4AB0Od3024998@d06av10.portsmouth.uk.ibm.com>

The agenda for the meeting in New York is now available here:
https://www.spxxl.org/?q=New-York-City-2016

For your convenience - here are the session titles:
- Top 10 reasons for upgrading from GPFS 3.5 to Spectrum Scale 4.2.1
- Panel Discussion: My favorite tool for monitoring/reporting
- Problem Determination
- User presentation : TBD
- Spectrum Scale in Life Sciences
- HPSS Update
- Best practices for GPFS or Spectrum Scale Tuning
- Enhancements for CORAL

Please use the link below for registration.

--
IBM Spectrum Scale Development - Client Engagements & Solutions Delivery
Consulting IT Specialist
Author "Storage Networks Explained"

IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Gesch?ftsf?hrung: Dirk Wittkopp
Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart,
HRB 243294




From:	Ulf Troppens/Germany/IBM at IBMDE
To:	gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:	03.05.2016 14:06
Subject:	Re: [gpfsug-discuss] GPFS/Spectrum Scale Upcoming US Events -
            Save the Dates - Registration for New York May 26
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



Registration for the first event in New York is up as well - the agenda
still needs some touch.

Please register here and select 'May 26 GPFS Day Registration', if you
don't attend SPXXL:
https://www.spxxl.org/?q=New-York-City-2016
https://www.eventbrite.com/e/spxxlscicomp-2016-summer-meeting-registration-24444020724



--
IBM Spectrum Scale Development - Client Engagements & Solutions Delivery
Consulting IT Specialist
Author "Storage Networks Explained"

IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Gesch?ftsf?hrung: Dirk Wittkopp
Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart,
HRB 243294


Inactive hide details for GPFS UG USA Principal ---28.04.2016
23:44:50---All, the registration page for the second event listedGPFS UG
USA Principal ---28.04.2016 23:44:50---All, the registration page for the
second event listed below at Argonne National Lab on June 10th is

From: GPFS UG USA Principal <usa-principal at gpfsug.org>
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Date: 28.04.2016 23:44
Subject: Re: [gpfsug-discuss] GPFS/Spectrum Scale Upcoming US Events - Save
the Dates
Sent by: gpfsug-discuss-bounces at spectrumscale.org



All, the registration page for the second event listed below at Argonne
National Lab on June 10th is now up. An updated agenda is also at this
site.

Please register here: https://www.regonline.com/Spectrumscalemeeting

We look forward to seeing some of you at these upcoming events.

Feel free to send suggestions for future events in your area.

Cheers,
-Kristy

            On Apr 4, 2016, at 4:52 PM, GPFS UG USA Principal <
            usa-principal at gpfsug.org> wrote:

            Hello all,

            We?d like to announce two upcoming US GPFS/Spectrum Scale
            Events. One on the east coast, one in the midwest.

            1) May 26th (full day event): GPFS/Spectrum Scale Day at the
            SPXXL conference in NYC
            https://www.spxxl.org/?q=New-York-City-2016 Developers and
            Engineers from IBM will be at the meeting to cover topics, and
            open dialogue between IBM and customers will be encouraged.
            More details about the agenda, hotel and other logistics will
            come later this month.

            Tentative Agenda:
            ? 10 reasons for upgrading from GPFS 3.5 to Spectrum Scale
            4.2.1
            ? Enhancements for CORAL from IBM
            ? Panel discussion with customers, topic TBD
            ? AFM and integration with Spectrum Protect
            ? Best practices for GPFS or Spectrum Scale Tuning.
            ? At least one site update

            Location:
            New York Academy of Medicine
            1216 Fifth Avenue
            New York, NY 10029

            ??

            2) June 10th (full day event): GPFS/Spectrum Scale Users Group
            Meeting at Argonne National Lab (ANL). Thanks to Argonne for
            hosting this event. Developers and Engineers from IBM will be
            at the meeting to cover topics, and open dialogue between IBM
            and customers will be encouraged, as usual no marketing
            pitches!

            Location: Argonne National Lab more details and final agenda
            will come later.

            Tentative Agenda:

            9:00a-12:30p
            9-9:30a - Opening Remarks
            9:30-10a Deep Dive - Update on ESS
            10a-11a Deep Dive - Problem Determination (Presentation 30
            min/Panel 30 min?)
            11-11:30 Break
            11:30a-Noon - Deep Dive - Protect & Scale integration
            Noon-12:30p HDFS/Hadoop

            12:30 - 1:30p Lunch

            1:30p-5:00p
            1:30 - 2:00p IBM AFM Update
            2:00-2:30p ANL: AFM as a burst buffer
            2:30-3:00p ANL: GHI (GPFS HPSS Integration)
            3:00-3:30p Break
            3:30p - 4:00p LANL: ? or other site preso
            4:00-4:30p Nuance: GPFS Performance Sensors Deployment
            Experiences
            4:30p -5:00p Closing comments and Open Forum for Questions

            5:00 - ?
            Beer hunting?

            ??

            We hope you can attend one or both of these events.

            Best,
            Kristy Kallback-Rose & Bob Oesterlin
            GPFS Users Group - USA Chapter - Principal & Co-principal


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160510/2d3525b5/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160510/2d3525b5/attachment-0001.gif>

From Mark.Bush at siriuscom.com  Wed May 11 03:02:30 2016
From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com)
Date: Wed, 11 May 2016 02:02:30 +0000
Subject: [gpfsug-discuss] Demo
Message-ID: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>

I?m tasked with coming up with a demo of Spectrum Scale for our customers.  Something that shows a little bit of most of the features of Scale.  Does anyone have some ideas of ?must see? features that might make for a good demo?

My thoughts are something like:

Overview of architecture
NSD Servers, Protocol Servers, Quorum, etc
Show adding a Node to the mix as a client and a server
Show creating a filesystem (and a fileset)
Show copying some files from a client (linux client with GPFS or through a filesystem thru protocol nodes)
Show GUI
Show Snapshot

Any other thoughts?


Mark R. Bush | Solutions Architect
Mobile: 210.237.8415 | mark.bush at siriuscom.com
Sirius Computer Solutions | www.siriuscom.com<http://www.siriuscom.com>
10100 Reunion Place, Suite 500, San Antonio, TX 78216


This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you.

Sirius Computer Solutions<http://www.siriuscom.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160511/b9bb4704/attachment-0001.htm>

From pinto at scinet.utoronto.ca  Wed May 11 03:35:19 2016
From: pinto at scinet.utoronto.ca (Jaime Pinto)
Date: Tue, 10 May 2016 22:35:19 -0400
Subject: [gpfsug-discuss] Demo
In-Reply-To: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
References: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
Message-ID: <20160510223519.11041xrqkfoeqbfr@support.scinet.utoronto.ca>

Make a clear distinction between the hardware and the software layers  
to start with.
Then between server infrastructure and client infrastructure (and  
licensing requirements)

NSD Servers, Protocol Servers, disks, LUNs, HW raid if any,  
configuration of the storage building blocks, NSD fail-over, etc

hardware raid vs. software raid vs, appliances

File system managers, cluster manager, clients addition/setup, FSmgr  
fail-over, Quorum, etc
File system creation, clustered vs. scattered, FS attributes,  
blocksize, capacity expansion/shrinkage, etc.

Fileset creation, junction linking, mounting, exportfs, protocol nodes.

storage cluster vs. compute cluster
snapshot, quota, (backup?), .....

For me there are areas/subjects that belong together, and a logic in  
the unveiling of features.

I would be interested in the end result

Thanks
Jaime


Quoting "Mark.Bush at siriuscom.com" <Mark.Bush at siriuscom.com>:

> I?m tasked with coming up with a demo of Spectrum Scale for our   
> customers.  Something that shows a little bit of most of the   
> features of Scale.  Does anyone have some ideas of ?must see?   
> features that might make for a good demo?
>
> My thoughts are something like:
>
> Overview of architecture
> NSD Servers, Protocol Servers, Quorum, etc
> Show adding a Node to the mix as a client and a server
> Show creating a filesystem (and a fileset)
> Show copying some files from a client (linux client with GPFS or   
> through a filesystem thru protocol nodes)
> Show GUI
> Show Snapshot
>
> Any other thoughts?
>
>
> Mark R. Bush | Solutions Architect
> Mobile: 210.237.8415 | mark.bush at siriuscom.com
> Sirius Computer Solutions | www.siriuscom.com<http://www.siriuscom.com>
> 10100 Reunion Place, Suite 500, San Antonio, TX 78216
>
>
> This message (including any attachments) is intended only for the   
> use of the individual or entity to which it is addressed and may   
> contain information that is non-public, proprietary, privileged,   
> confidential, and exempt from disclosure under applicable law. If   
> you are not the intended recipient, you are hereby notified that any  
>  use, dissemination, distribution, or copying of this communication   
> is strictly prohibited. This message may be viewed by parties at   
> Sirius Computer Solutions other than those named in the message   
> header. This message does not contain an official representation of   
> Sirius Computer Solutions. If you have received this communication   
> in error, notify Sirius Computer Solutions immediately and (i)   
> destroy this message if a facsimile or (ii) delete this message   
> immediately if this is an electronic communication. Thank you.
>
> Sirius Computer Solutions<http://www.siriuscom.com>
>




---
Jaime Pinto
SciNet HPC Consortium  - Compute/Calcul Canada
www.scinet.utoronto.ca - www.computecanada.org
University of Toronto
256 McCaul Street, Room 235
Toronto, ON, M5T1W5
P: 416-978-2755
C: 416-505-1477

----------------------------------------------------------------
This message was sent using IMP at SciNet Consortium, University of Toronto.



From sfadden at us.ibm.com  Wed May 11 06:51:38 2016
From: sfadden at us.ibm.com (Scott Fadden)
Date: Tue, 10 May 2016 22:51:38 -0700
Subject: [gpfsug-discuss] Demo
In-Reply-To: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
References: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
Message-ID: <201605110551.u4B5phC8017323@d03av02.boulder.ibm.com>

I developed a demo that shows how the file heat metric can be used to move 
"cool" data from a Flash tier to spinning disk. As visuals it displays a 
speedometer for transaction rate, a chart showing capacity used in each 
tier and total cost of storing the data ($/GiB). 

For a simpler demo maybe the GUI with live performance, maybe a policy on 
lots of files (use a list policy) or something like fileset level 
snapshots.


Scott Fadden
Spectrum Scale - Technical Marketing 
Phone: (503) 880-5833 
sfadden at us.ibm.com
http://www.ibm.com/systems/storage/spectrum/scale



From:   "Mark.Bush at siriuscom.com" <Mark.Bush at siriuscom.com>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   05/10/2016 07:02 PM
Subject:        [gpfsug-discuss] Demo
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



I?m tasked with coming up with a demo of Spectrum Scale for our customers. 
 Something that shows a little bit of most of the features of Scale.  Does 
anyone have some ideas of ?must see? features that might make for a good 
demo?

My thoughts are something like: 

Overview of architecture
NSD Servers, Protocol Servers, Quorum, etc
Show adding a Node to the mix as a client and a server
Show creating a filesystem (and a fileset)
Show copying some files from a client (linux client with GPFS or through a 
filesystem thru protocol nodes)
Show GUI
Show Snapshot

Any other thoughts?


Mark R. Bush | Solutions Architect
Mobile: 210.237.8415 | mark.bush at siriuscom.com
Sirius Computer Solutions | www.siriuscom.com
10100 Reunion Place, Suite 500, San Antonio, TX 78216 

This message (including any attachments) is intended only for the use of 
the individual or entity to which it is addressed and may contain 
information that is non-public, proprietary, privileged, confidential, and 
exempt from disclosure under applicable law. If you are not the intended 
recipient, you are hereby notified that any use, dissemination, 
distribution, or copying of this communication is strictly prohibited. 
This message may be viewed by parties at Sirius Computer Solutions other 
than those named in the message header. This message does not contain an 
official representation of Sirius Computer Solutions. If you have received 
this communication in error, notify Sirius Computer Solutions immediately 
and (i) destroy this message if a facsimile or (ii) delete this message 
immediately if this is an electronic communication. Thank you. 
Sirius Computer Solutions _______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160510/af957fe6/attachment-0001.htm>

From Mikal.Sande at altibox.no  Wed May 11 08:10:02 2016
From: Mikal.Sande at altibox.no (Sande, Mikal)
Date: Wed, 11 May 2016 07:10:02 +0000
Subject: [gpfsug-discuss] Demo
In-Reply-To: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
References: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
Message-ID: <84C0B9AB4719954985704BC46DF8602A9F7E480B@prdmbx01.lyse.no>

Apart from the usefulness of parallel access to the same LUNs, one feature I value for metadata heavy workloads is being able to split data and metadata. When we moved metadata from spindles to flash LUNs on one of our clusters it was a day vs. night kind of change in behavior in the cluster.

Just my two cents, I know exactly how to demo the split between data and metadata. Maybe when creating the filesystem? Or maybe afterwards and show how restripe is used to perform such a change?


Best regards,
Mikal Sande
________________________________
From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Mark.Bush at siriuscom.com [Mark.Bush at siriuscom.com]
Sent: Wednesday, May 11, 2016 4:02 AM
To: gpfsug main discussion list
Subject: [gpfsug-discuss] Demo

I?m tasked with coming up with a demo of Spectrum Scale for our customers.  Something that shows a little bit of most of the features of Scale.  Does anyone have some ideas of ?must see? features that might make for a good demo?

My thoughts are something like:

Overview of architecture
NSD Servers, Protocol Servers, Quorum, etc
Show adding a Node to the mix as a client and a server
Show creating a filesystem (and a fileset)
Show copying some files from a client (linux client with GPFS or through a filesystem thru protocol nodes)
Show GUI
Show Snapshot

Any other thoughts?


Mark R. Bush | Solutions Architect
Mobile: 210.237.8415 | mark.bush at siriuscom.com
Sirius Computer Solutions | www.siriuscom.com<http://www.siriuscom.com>
10100 Reunion Place, Suite 500, San Antonio, TX 78216


This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you.

Sirius Computer Solutions<http://www.siriuscom.com>


From D.R.Barker at exeter.ac.uk  Wed May 11 10:53:15 2016
From: D.R.Barker at exeter.ac.uk (Barker, David)
Date: Wed, 11 May 2016 09:53:15 +0000
Subject: [gpfsug-discuss] UK May Meeting
Message-ID: <A4DBAD9F-EE30-4A0E-AD59-51BFCB18F4DB@exeter.ac.uk>

Hi Simon - sorry to have to cancel at the last minute, but I won?t be able to attend now. Feel free to give my space to someone else. :-/

Best regards,
Dave

-- 
David Barker
01392 723900
www.exeter.ac.uk
Laver Building, North Park Road, EXETER, EX4 4QE

 
 









On 11/04/2016, 10:37, "gpfsug-discuss-bounces at spectrumscale.org on behalf of GPFS UG Chair (Simon Thompson)" <gpfsug-discuss-bounces at spectrumscale.org on behalf of chair at spectrumscale.org> wrote:

>
>Hi All,
>
>We are down to our last few places for the May user group meeting, if you
>are planning to come along, please do register:
>
>The draft agenda and registration for the day is at:
>http://www.eventbrite.com/e/spectrum-scale-gpfs-uk-user-group-spring-2016-t
>ickets-21724951916
>
>If you have registered and aren't able to attend now, please do let us
>know so that we can free the slot for other members of the group.
>
>We also have 1 slot left on the agenda for a user talk, so if you have an
>interesting deployment or plans and are able to speak, please let me know!
>
>Thanks
>
>Simon
>
>
>_______________________________________________
>gpfsug-discuss mailing list
>gpfsug-discuss at spectrumscale.org
>http://gpfsug.org/mailman/listinfo/gpfsug-discuss

From Robert.Oesterlin at nuance.com  Wed May 11 17:05:12 2016
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Wed, 11 May 2016 16:05:12 +0000
Subject: [gpfsug-discuss] Reminder: Spectrum Scale/GPFS UG meeting,
	June 10th at Argonne Lab
Message-ID: <AB6F0D83-4CBA-4651-BE78-384276359819@nuance.com>

A Reminder that there will be a user group meeting on June 10th at Argonne Labs, near Chicago, IL. A great agenda and a great opportunity to network with other users.

Details: http://spectrumscale.org/spectrum-scale-user-group-usa-meeting-june-10th/

Registration link:https://www.regonline.com/Spectrumscalemeeting

Registration deadline is May 27th!

Bob Oesterlin
GPFS UG Co-Principal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160511/e5e5c3f7/attachment-0001.htm>

From janfrode at tanso.net  Wed May 11 18:04:37 2016
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Wed, 11 May 2016 17:04:37 +0000
Subject: [gpfsug-discuss] Demo
In-Reply-To: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
References: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
Message-ID: <CAHwPathGPijoRstXc6GzBNJxfWiOF8kuNJFbJUui1xH-fJzFdA@mail.gmail.com>

Some suggestions for stuff I find demo-interesting..

Show snapshots accessible as previous version trough SMB/Windows Explorer.
Show offline files with X indicator in Explorer, if you can do HSM (maybe
trough via the cloud gateway beta).

Show object storage trough cyberduck, or similar clients.

Failover nfs/smb.

Mixed windows/linux cluster accessing same filesystem.

mmbackup doing backups of million of files in no time.



-jf

ons. 11. mai 2016 kl. 04.02 skrev Mark.Bush at siriuscom.com <
Mark.Bush at siriuscom.com>:

> I?m tasked with coming up with a demo of Spectrum Scale for our
> customers.  Something that shows a little bit of most of the features of
> Scale.  Does anyone have some ideas of ?must see? features that might make
> for a good demo?
>
> My thoughts are something like:
>
> Overview of architecture
> NSD Servers, Protocol Servers, Quorum, etc
> Show adding a Node to the mix as a client and a server
> Show creating a filesystem (and a fileset)
> Show copying some files from a client (linux client with GPFS or through a
> filesystem thru protocol nodes)
> Show GUI
> Show Snapshot
>
> Any other thoughts?
>
>
> Mark R. Bush | Solutions Architect
> Mobile: 210.237.8415 | mark.bush at siriuscom.com
> Sirius Computer Solutions | www.siriuscom.com
> 10100 Reunion Place, Suite 500, San Antonio, TX 78216
>
> This message (including any attachments) is intended only for the use of
> the individual or entity to which it is addressed and may contain
> information that is non-public, proprietary, privileged, confidential, and
> exempt from disclosure under applicable law. If you are not the intended
> recipient, you are hereby notified that any use, dissemination,
> distribution, or copying of this communication is strictly prohibited. This
> message may be viewed by parties at Sirius Computer Solutions other than
> those named in the message header. This message does not contain an
> official representation of Sirius Computer Solutions. If you have received
> this communication in error, notify Sirius Computer Solutions immediately
> and (i) destroy this message if a facsimile or (ii) delete this message
> immediately if this is an electronic communication. Thank you.
> Sirius Computer Solutions <http://www.siriuscom.com>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160511/5f4d6e64/attachment-0001.htm>

From YARD at il.ibm.com  Wed May 11 20:49:08 2016
From: YARD at il.ibm.com (Yaron Daniel)
Date: Wed, 11 May 2016 22:49:08 +0300
Subject: [gpfsug-discuss] Demo
In-Reply-To: <201605110551.u4B5phC8017323@d03av02.boulder.ibm.com>
References: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
	<201605110551.u4B5phC8017323@d03av02.boulder.ibm.com>
Message-ID: <201605111949.u4BJnF8L015520@d06av02.portsmouth.uk.ibm.com>

Hi

I think what is hot:

- Compression
- Mixed access to same files via CIFS/NFS/GPFS client
- AFM DR capabilities

 
Regards
 


 
 
Yaron Daniel
 94 Em Ha'Moshavot Rd

Server, Storage and Data Services - Team Leader  
 Petach Tiqva, 49527
Global Technology Services
 Israel
Phone:
+972-3-916-5672
 
 
Fax:
+972-3-916-5672
 
 
Mobile:
+972-52-8395593
 
 
e-mail:
yard at il.ibm.com
 
 
IBM Israel
 
 
 
 

 



From:   "Scott Fadden" <sfadden at us.ibm.com>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   05/11/2016 08:51 AM
Subject:        Re: [gpfsug-discuss] Demo
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



I developed a demo that shows how the file heat metric can be used to move 
"cool" data from a Flash tier to spinning disk. As visuals it displays a 
speedometer for transaction rate, a chart showing capacity used in each 
tier and total cost of storing the data ($/GiB). 

For a simpler demo maybe the GUI with live performance, maybe a policy on 
lots of files (use a list policy) or something like fileset level 
snapshots.


Scott Fadden
Spectrum Scale - Technical Marketing 
Phone: (503) 880-5833 
sfadden at us.ibm.com
http://www.ibm.com/systems/storage/spectrum/scale



From:        "Mark.Bush at siriuscom.com" <Mark.Bush at siriuscom.com>
To:        gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:        05/10/2016 07:02 PM
Subject:        [gpfsug-discuss] Demo
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



I?m tasked with coming up with a demo of Spectrum Scale for our customers. 
 Something that shows a little bit of most of the features of Scale.  Does 
anyone have some ideas of ?must see? features that might make for a good 
demo?

My thoughts are something like: 

Overview of architecture
NSD Servers, Protocol Servers, Quorum, etc
Show adding a Node to the mix as a client and a server
Show creating a filesystem (and a fileset)
Show copying some files from a client (linux client with GPFS or through a 
filesystem thru protocol nodes)
Show GUI
Show Snapshot

Any other thoughts?


Mark R. Bush| Solutions Architect
Mobile: 210.237.8415 | mark.bush at siriuscom.com
Sirius Computer Solutions | www.siriuscom.com
10100 Reunion Place, Suite 500, San Antonio, TX 78216 
This message (including any attachments) is intended only for the use of 
the individual or entity to which it is addressed and may contain 
information that is non-public, proprietary, privileged, confidential, and 
exempt from disclosure under applicable law. If you are not the intended 
recipient, you are hereby notified that any use, dissemination, 
distribution, or copying of this communication is strictly prohibited. 
This message may be viewed by parties at Sirius Computer Solutions other 
than those named in the message header. This message does not contain an 
official representation of Sirius Computer Solutions. If you have received 
this communication in error, notify Sirius Computer Solutions immediately 
and (i) destroy this message if a facsimile or (ii) delete this message 
immediately if this is an electronic communication. Thank you. 
Sirius Computer Solutions _______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160511/8d1a26c5/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 1851 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160511/8d1a26c5/attachment-0001.gif>

From pgalgali at us.ibm.com  Wed May 11 21:53:25 2016
From: pgalgali at us.ibm.com (Pallavi V Galgali)
Date: Wed, 11 May 2016 15:53:25 -0500
Subject: [gpfsug-discuss] Demo
In-Reply-To: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
References: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
Message-ID: <201605112053.u4BKrWeI005237@d01av04.pok.ibm.com>

Pls explore Spectrum Scale trial VM and its attached documentation -
http://www-03.ibm.com/systems/storage/spectrum/scale/trial.html
This comes with documentation ( Explore guide and Advanced user guide) that
has some interesting scenarios that can be used for demo purpose.

                                                                                     
   Thanks & Regards,                                                                 
                                                                                     
   Pallavi Galgali                                                                   
   Manager, Client Adoption and Solutions Delivery                                   
   IBM Spectrum Scale Development                                                    
   Cell: +1 914 433 9882                                                             
   11501 Burnet Road Austin TX 78758                                                 
                                                                                     








From:	"Mark.Bush at siriuscom.com" <Mark.Bush at siriuscom.com>
To:	gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:	05/10/2016 09:02 PM
Subject:	[gpfsug-discuss] Demo
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



I?m tasked with coming up with a demo of Spectrum Scale for our customers.
Something that shows a little bit of most of the features of Scale.  Does
anyone have some ideas of ?must see? features that might make for a good
demo?

My thoughts are something like:

Overview of architecture
NSD Servers, Protocol Servers, Quorum, etc
Show adding a Node to the mix as a client and a server
Show creating a filesystem (and a fileset)
Show copying some files from a client (linux client with GPFS or through a
filesystem thru protocol nodes)
Show GUI
Show Snapshot

Any other thoughts?


Mark R. Bush | Solutions Architect
Mobile: 210.237.8415 | mark.bush at siriuscom.com
Sirius Computer Solutions | www.siriuscom.com
10100 Reunion Place, Suite 500, San Antonio, TX 78216



This message (including any attachments) is intended only for the use of
the individual or entity to which it is addressed and may contain
information that is non-public, proprietary, privileged, confidential, and
exempt from disclosure under applicable law. If you are not the intended
recipient, you are hereby notified that any use, dissemination,
distribution, or copying of this communication is strictly prohibited. This
message may be viewed by parties at Sirius Computer Solutions other than
those named in the message header. This message does not contain an
official representation of Sirius Computer Solutions. If you have received
this communication in error, notify Sirius Computer Solutions immediately
and (i) destroy this message if a facsimile or (ii) delete this message
immediately if this is an electronic communication. Thank you.


Sirius Computer Solutions _______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160511/694b75a9/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160511/694b75a9/attachment-0001.gif>

From douglasof at us.ibm.com  Fri May 13 16:01:55 2016
From: douglasof at us.ibm.com (Douglas O'flaherty)
Date: Fri, 13 May 2016 11:01:55 -0400
Subject: [gpfsug-discuss] ISC U/G Agenda
Message-ID: <201605131502.u4DF23Bi020162@d03av05.boulder.ibm.com>

Greetings:

IBM is happy to announce the agenda for the joint IBM Spectrum Scale and 
Platform Computing User Group Agenda at ISC. We will finish on time to 
attend the opening reception. 

As with other user group meetings, the agenda includes user stories, 
updates on IBM Spectrum Scale & Platform LSF, and access to IBM experts 
and your peers. 

Please join us! 

To attend, please email Fabian.Beckmann at de.ibm.com so we can have an 
accurate count of attendees. 


Monday June 20, 2016  - 14:30-18:00 - Conference Room Konstant

14:30-14:40 [10 min] Welcome (Douglas o'Flaherty, IBM)
14:40-15:00 [20 min] Ten Reasons to Upgrade from GPFS 3.4 to Spectrum 
Scale 4.2 (Olaf Weiser, IBM)
15:00-15:30 [30 min] Shared Storage with in-memory latency: EMC DSSD D5 
and IBM Spectrum Scale (Stefan Radtke, EMC)
15:30-16:00 [30 min] Workload scheduling and data management in a private 
cloud (Uwe Sommer, Airbus)
16:00-16:30 [30 min] Spectrum Scale site report (To be confirmed by 
customer)
16:30-17:00 [30 min] What's new in Platform LSF 10.1 & storage integration 
(Bill McMillan, IBM)
17:00-17:30 [30 min] What's new in Spectrum Scale 4.2.1 (Mathias Dietz, 
IBM)
17:30-18:00 [30 min] CORAL enhancements for Spectrum Scale (Sven Oehme, 
IBM)

Looking forward to seeing you there!


doug

PS: IBMers can register their clients at this IBM link: 
https://w3-9033.ibm.com/events/ast/schedule/16isc.nsf 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160513/bba0eeae/attachment-0001.htm>

From billowen at us.ibm.com  Sat May 14 13:05:14 2016
From: billowen at us.ibm.com (Bill Owen)
Date: Sat, 14 May 2016 05:05:14 -0700
Subject: [gpfsug-discuss] OpenStack & Spectrum Scale Usage Survey
Message-ID: <201605141205.u4EC5KBm000338@d03av04.boulder.ibm.com>


We in Spectrum Scale development are very interested in learning about your
current & planned usage of OpenStack with Spectrum Scale, as well as plans
for deploying containers in your Spectrum Scale environment.

We would like to use this information to help shape our road map in this
area over the next 18-24 months.

Please take 10 minutes to answer the questions in this short survey:
https://www.surveygizmo.com/s3/2774614/IBMSpectrumScale-OpenStackUsageSurvey

Thank you,
Bill Owen
billowen at us.ibm.com
Spectrum Scale Object Storage
520-799-4829
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160514/be7206f9/attachment-0001.htm>

From S.J.Thompson at bham.ac.uk  Mon May 16 09:44:05 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Mon, 16 May 2016 08:44:05 +0000
Subject: [gpfsug-discuss] Flash for metadata
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F577FBC6@EX13.adf.bham.ac.uk>

Hi all

I'm currently about to deploy some flash systems (FS-900) to take our metadata for one of our Spectrum Scale file-systems. We use 4k inodes for this file-system.

On the FS-900, when creating a volume, we get a choice of 512b or 4k sectors. Does anyone have a suggestion on this? On a spinning disk RAID array, I get what a strip size is and we'd use multiples of the stripe where possible for the block size.

Is the sector size on the FS-900 the smallest chunk that can be written out? And so would it make sense to use 4k sectors for 4k GPFS inodes right?

Related. In the deployment guide for the FS-900 arrays, if one is using a file-system (e.g. XFS) on the array the guide talks about aligning the partitions with with array, when GPFS is used on a storage array, does it automatically work out the sector alignment?


Thanks

Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160516/2909e86d/attachment-0001.htm>

From Paul.Sanchez at deshaw.com  Mon May 16 10:28:35 2016
From: Paul.Sanchez at deshaw.com (Sanchez, Paul)
Date: Mon, 16 May 2016 09:28:35 +0000
Subject: [gpfsug-discuss] Flash for metadata
In-Reply-To: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F577FBC6@EX13.adf.bham.ac.uk>
References: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F577FBC6@EX13.adf.bham.ac.uk>
Message-ID: <49a3ab3b493a49e59177c6c25dd34d23@mbxtoa1.winmail.deshaw.com>

Simon,

I can't speak specifically to the FS-900, since I generally use Dell Compellent and NetApp EF arrays for flash today.  But I also take care to ensure that the system pool metadata NSDs use RAID10 (or equivalent) rather than RAID5/6.  For small 4K metadata writes, on a 4K blocksize LUN this could result in optimal writes.  For larger native blocksize LUNs, it eliminates the parity calculation for read-modify-write operations.

Thanks,
Paul

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (Research Computing - IT Services)
Sent: Monday, May 16, 2016 9:44 AM
To: gpfsug-discuss at spectrumscale.org
Subject: [gpfsug-discuss] Flash for metadata

Hi all

I'm currently about to deploy some flash systems (FS-900) to take our metadata for one of our Spectrum Scale file-systems. We use 4k inodes for this file-system.

On the FS-900, when creating a volume, we get a choice of 512b or 4k sectors. Does anyone have a suggestion on this? On a spinning disk RAID array, I get what a strip size is and we'd use multiples of the stripe where possible for the block size.

Is the sector size on the FS-900 the smallest chunk that can be written out? And so would it make sense to use 4k sectors for 4k GPFS inodes right?

Related. In the deployment guide for the FS-900 arrays, if one is using a file-system (e.g. XFS) on the array the guide talks about aligning the partitions with with array, when GPFS is used on a storage array, does it automatically work out the sector alignment?


Thanks

Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160516/f4964333/attachment-0001.htm>

From S.J.Thompson at bham.ac.uk  Mon May 16 10:48:12 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Mon, 16 May 2016 09:48:12 +0000
Subject: [gpfsug-discuss] Flash for metadata
In-Reply-To: <49a3ab3b493a49e59177c6c25dd34d23@mbxtoa1.winmail.deshaw.com>
References: <49a3ab3b493a49e59177c6c25dd34d23@mbxtoa1.winmail.deshaw.com>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F577FC47@EX13.adf.bham.ac.uk>


Hi Paul,

Thanks, interestingly the FS-900 only does raid5. I get no choice about that, I do get a sector size choice when I create a lun on it...

I was also wondering how gpfs would do a metadata update. Would it be smart enough to know the sector size was 512b and only modify that, or would it always do a read modify write on the whole 4k inode?

Simon

-----Original Message-----
From: Sanchez, Paul [Paul.Sanchez at deshaw.com<mailto:Paul.Sanchez at deshaw.com>]
Sent: Monday, May 16, 2016 10:28 AM GMT Standard Time
To: gpfsug-discuss at spectrumscale.org
Subject: Re: [gpfsug-discuss] Flash for metadata

Simon,

I can?t speak specifically to the FS-900, since I generally use Dell Compellent and NetApp EF arrays for flash today.  But I also take care to ensure that the system pool metadata NSDs use RAID10 (or equivalent) rather than RAID5/6.  For small 4K metadata writes, on a 4K blocksize LUN this could result in optimal writes.  For larger native blocksize LUNs, it eliminates the parity calculation for read-modify-write operations.

Thanks,
Paul

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (Research Computing - IT Services)
Sent: Monday, May 16, 2016 9:44 AM
To: gpfsug-discuss at spectrumscale.org
Subject: [gpfsug-discuss] Flash for metadata

Hi all

I'm currently about to deploy some flash systems (FS-900) to take our metadata for one of our Spectrum Scale file-systems. We use 4k inodes for this file-system.

On the FS-900, when creating a volume, we get a choice of 512b or 4k sectors. Does anyone have a suggestion on this? On a spinning disk RAID array, I get what a strip size is and we'd use multiples of the stripe where possible for the block size.

Is the sector size on the FS-900 the smallest chunk that can be written out? And so would it make sense to use 4k sectors for 4k GPFS inodes right?

Related. In the deployment guide for the FS-900 arrays, if one is using a file-system (e.g. XFS) on the array the guide talks about aligning the partitions with with array, when GPFS is used on a storage array, does it automatically work out the sector alignment?


Thanks

Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160516/9dbf385b/attachment-0001.htm>

From dacalder at co.ibm.com  Tue May 17 00:14:53 2016
From: dacalder at co.ibm.com (Danny Alexander Calderon Rodriguez)
Date: Mon, 16 May 2016 23:14:53 +0000
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 52,
	Issue 20 Flash for metadata
In-Reply-To: <mailman.3639.1463392096.1371.gpfsug-discuss@spectrumscale.org>
References: <mailman.3639.1463392096.1371.gpfsug-discuss@spectrumscale.org>
Message-ID: <201605162315.u4GNDqVo032100@mx0b-001b2d01.pphosted.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160516/14aea31a/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Image.14634403158690.gif
Type: image/gif
Size: 360 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160516/14aea31a/attachment-0001.gif>

From Luke.Raimbach at crick.ac.uk  Tue May 17 09:03:24 2016
From: Luke.Raimbach at crick.ac.uk (Luke Raimbach)
Date: Tue, 17 May 2016 08:03:24 +0000
Subject: [gpfsug-discuss] Feature Request: Clones Inheriting XATTRS
Message-ID: <AMSPR03MB020C6EAB8A9965F03517B16B0480@AMSPR03MB020.eurprd03.prod.outlook.com>

HI All,

Where do I send feature requests?

I'd like the mmclone command to have a switch which allows user defined attributes to be passed to file clones. I can understand why the "gpfs." defined extended attributes don't get passed along, because they might hold special information like how a DMAPI agent has fiddled with the data, which won't necessarily hold true for clones.

However, what would be really useful is something like "--xattr" switch for mmclone, which allows "user." defined attributes to be inherited by clones.

We are intending to use file cloning as an internal data sharing mechanism (to reduce physical disk space usage). Some scientific groups may wish to pass along all the extended attributes (e.g. instrument type, serial number, experiment ID, etc.) to those they share data with, and others may wish to strip out those attributes.

Where shall I send this one?

Cheers,
Luke.

Luke Raimbach?
Senior HPC Data and Storage Systems Engineer,
The Francis Crick Institute,
Gibbs Building,
215 Euston Road,
London NW1 2BE.

E: luke.raimbach at crick.ac.uk
W: www.crick.ac.uk


The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no. 06885462, with its registered office at 215 Euston Road, London NW1 2BE.

From samuel.guest at noaa.gov  Tue May 17 18:31:11 2016
From: samuel.guest at noaa.gov (Sam Guest)
Date: Tue, 17 May 2016 13:31:11 -0400
Subject: [gpfsug-discuss] New UG Subscriber
Message-ID: <0ca901d1b061$e7fd2ba0$b7f782e0$@noaa.gov>

Our organization is implementing a ESS/GPFS integrated solution via RedHat 7
on Power8 and IBM DCS3700 hardware to support the Comprehensive Large
Array-data Stewardship System (CLASS) for NOAA.gov. The ESS/GPFS solution
will be replacing a legacy shared file system solution from another storage
vendor. Our office is located in the Washington, DC metro area. The CLASS
application is expected to process over 500k files and approximately 30 TB
of data per day in the near future. I am new to the ESS/GPFS environment and
have an extensive background as a AIX and UNIX systems administrator. I am
excited to have the opportunity to be a part of such a dynamic technology
and look forward to learning, sharing, and networking with the group.

Sincerely,

Sam Guest, NOAA Affiliate
Systems Engineer
CSRA, DGP Team
NESDIS/OSD/GSD/CLASS
7855 Walker Drive, Suite 200, Greenbelt, MD 20770 
Desk: 240-542-1117 Cell : 301-706-1168 Fax : 301-474-4074





From jonathan at buzzard.me.uk  Tue May 17 21:22:39 2016
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Tue, 17 May 2016 21:22:39 +0100
Subject: [gpfsug-discuss] Flash for metadata
In-Reply-To: <49a3ab3b493a49e59177c6c25dd34d23@mbxtoa1.winmail.deshaw.com>
References: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F577FBC6@EX13.adf.bham.ac.uk>
	<49a3ab3b493a49e59177c6c25dd34d23@mbxtoa1.winmail.deshaw.com>
Message-ID: <573B7D8F.7010106@buzzard.me.uk>

On 16/05/16 10:28, Sanchez, Paul wrote:
> Simon,
>
> I can?t speak specifically to the FS-900, since I generally use Dell
> Compellent and NetApp EF arrays for flash today.  But I also take care
> to ensure that the system pool metadata NSDs use RAID10 (or equivalent)
> rather than RAID5/6.  For small 4K metadata writes, on a 4K blocksize
> LUN this could result in optimal writes.  For larger native blocksize
> LUNs, it eliminates the parity calculation for read-modify-write
> operations.
>

General point would it not make more sense to make multiple RAID1's and 
set replication of the metadata than make a RAID10? Preferably with the 
RAID1's on more than one storage array?

Always gives me a warm fuzzy feeling to have my metadata replicated even 
if the data is not.

JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.


From makaplan at us.ibm.com  Wed May 18 00:26:58 2016
From: makaplan at us.ibm.com (Marc A Kaplan)
Date: Tue, 17 May 2016 19:26:58 -0400
Subject: [gpfsug-discuss] Flash for metadata
In-Reply-To: <573B7D8F.7010106@buzzard.me.uk>
References: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F577FBC6@EX13.adf.bham.ac.uk><49a3ab3b493a49e59177c6c25dd34d23@mbxtoa1.winmail.deshaw.com>
	<573B7D8F.7010106@buzzard.me.uk>
Message-ID: <201605172327.u4HNRYAv023425@d03av05.boulder.ibm.com>

I think there are two points here:

A) RAID striping is probably a "loser" for GPFS metadata. 
B) RAID mirroring for your metadata may or may not be faster and/or more 
reliable than GPFS replication. Depending on your requirements and 
assumptions for fault-tolerance one or the other might be the winner.

But .. if you're serious about performance - measure, measure, measure and 
pick the winner. (And we'd like to know what you found!)
And if you want the best performance, you'll probably need to (re)measure 
whenever you acquire new equipment.

GPFS can do and does striping for high performance when reading and 
writing metadata and data...


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160517/296937ab/attachment-0001.htm>

From chair at spectrumscale.org  Wed May 18 19:35:44 2016
From: chair at spectrumscale.org (Spectrum scale UG Chair (Simon Thompson))
Date: Wed, 18 May 2016 19:35:44 +0100
Subject: [gpfsug-discuss] UK user group
Message-ID: <mailman.2.1653257833.13686.gpfsug-discuss_gpfsug.org@gpfsug.org>

Hi All,

Well, the UK user group is wrapped up. Most of the slides are now live on the user group site on the presentations pages. Hopefully we'll get the last few up in the next few days.

Thanks to all the user speakers, IBM speakers and those from our sponsors for supporting the event. And also to the members of the community who come along to support us, to ask questions and to challenge IBM for the future!

We had just under 30 speakers from across the globe, and a packed agenda over two days. You can get some insight on the days by checking the #SSUG16 hashtag on twitter.

Again thanks to the sponsors from Arcastream, DDN, IBM, Lenovo, Mellanox, NetApp, OCF, Seagate for supporting the event.

A final note of thanks to Oxford e-research centre for lending is their camera for the camera for two days, we have videos from many of the talks, we need to work out what to do with them, hopefully will get something online in the next few weeks! So keep an eye out!

If you came along and forget to give us your feedback forms, please drop myself and Claire an email and let us know!

And don't forget, if you missed us, there are user groups events over the next six weeks in New York, Chicago and Frankfurt!

Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160518/0b9b9d08/attachment.htm>

From peter.botcherby at kcl.ac.uk  Wed May 18 19:41:57 2016
From: peter.botcherby at kcl.ac.uk (Botcherby, Peter)
Date: Wed, 18 May 2016 18:41:57 +0000
Subject: [gpfsug-discuss] UK user group
In-Reply-To: <20160518183607.9786A41C837@gpfsug.org>
References: <20160518183607.9786A41C837@gpfsug.org>
Message-ID: <VI1PR03MB1871E3D2415EC756778B0A35A4490@VI1PR03MB1871.eurprd03.prod.outlook.com>

Hi Simon,
Sorry I couldn't make it but look forward to seeing the presentations and the videos later.

All the best
Cheers
Peter
________________________________
From: Spectrum scale UG Chair (Simon Thompson)<mailto:chair at spectrumscale.org>
Sent: ?18/?05/?2016 19:36
To: gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] UK user group

Hi All,

Well, the UK user group is wrapped up. Most of the slides are now live on the user group site on the presentations pages. Hopefully we'll get the last few up in the next few days.

Thanks to all the user speakers, IBM speakers and those from our sponsors for supporting the event. And also to the members of the community who come along to support us, to ask questions and to challenge IBM for the future!

We had just under 30 speakers from across the globe, and a packed agenda over two days. You can get some insight on the days by checking the #SSUG16 hashtag on twitter.

Again thanks to the sponsors from Arcastream, DDN, IBM, Lenovo, Mellanox, NetApp, OCF, Seagate for supporting the event.

A final note of thanks to Oxford e-research centre for lending is their camera for the camera for two days, we have videos from many of the talks, we need to work out what to do with them, hopefully will get something online in the next few weeks! So keep an eye out!

If you came along and forget to give us your feedback forms, please drop myself and Claire an email and let us know!

And don't forget, if you missed us, there are user groups events over the next six weeks in New York, Chicago and Frankfurt!

Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160518/11688115/attachment-0001.htm>

From S.J.Thompson at bham.ac.uk  Wed May 18 19:48:06 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Wed, 18 May 2016 18:48:06 +0000
Subject: [gpfsug-discuss] UK user group
In-Reply-To: <20160518183607.9786A41C837@gpfsug.org>
References: <20160518183607.9786A41C837@gpfsug.org>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F57805F4@EX13.adf.bham.ac.uk>


The presentations page is not obvious... Its under the menu that disappears....

www.spectrumscale.org/presentations/

Simon
________________________________________
From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Spectrum scale UG Chair (Simon Thompson) [chair at spectrumscale.org]
Sent: 18 May 2016 19:35
To: gpfsug-discuss at spectrumscale.org
Subject: [gpfsug-discuss] UK user group

Hi All,

Well, the UK user group is wrapped up. Most of the slides are now live on the user group site on the presentations pages. Hopefully we'll get the last few up in the next few days.

Thanks to all the user speakers, IBM speakers and those from our sponsors for supporting the event. And also to the members of the community who come along to support us, to ask questions and to challenge IBM for the future!

We had just under 30 speakers from across the globe, and a packed agenda over two days. You can get some insight on the days by checking the #SSUG16 hashtag on twitter.

Again thanks to the sponsors from Arcastream, DDN, IBM, Lenovo, Mellanox, NetApp, OCF, Seagate for supporting the event.

A final note of thanks to Oxford e-research centre for lending is their camera for the camera for two days, we have videos from many of the talks, we need to work out what to do with them, hopefully will get something online in the next few weeks! So keep an eye out!

If you came along and forget to give us your feedback forms, please drop myself and Claire an email and let us know!

And don't forget, if you missed us, there are user groups events over the next six weeks in New York, Chicago and Frankfurt!

Simon


From S.J.Thompson at bham.ac.uk  Thu May 19 16:08:12 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Thu, 19 May 2016 15:08:12 +0000
Subject: [gpfsug-discuss] UK user group
In-Reply-To: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F57805F4@EX13.adf.bham.ac.uk>
References: <20160518183607.9786A41C837@gpfsug.org>
	<CF45EE16DEF2FE4B9AA7FF2B6EE26545F57805F4@EX13.adf.bham.ac.uk>
Message-ID: <D3639532.2AAEB%s.j.thompson@bham.ac.uk>

And my take on the event ...

http://www.spectrumscale.org/uk-may-2016-group-report/

Don't forget, we're happy to take blog contributions from others as well,
events, musings on Spectrum Scale welcome ...

Simon



From S.J.Thompson at bham.ac.uk  Thu May 19 22:16:57 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Thu, 19 May 2016 21:16:57 +0000
Subject: [gpfsug-discuss] Flash for metadata
In-Reply-To: <201605172327.u4HNRYAv023425@d03av05.boulder.ibm.com>
References: <201605172327.u4HNRYAv023425@d03av05.boulder.ibm.com>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F578235F@EX13.adf.bham.ac.uk>

Thanks to all those who replied (including off list).

Someone reminded me that whilst the FlashSystem will support 512 and 4k blocks, it "prefers" 4k. Maybe there is a performance penalty for the smaller writes.

Second, a flash update is always going to be read erase write.

We don't get any choice with FlashSystem 900 on the RAID set - only RAID5 is available.

Given we have two FlashSystems in two data centres, looks like we are going with 4k sectors and GPFS replication.

I assume that creating several smaller LUNs on each FlashSystem in the same failure group is still preferable to one big LUN so we get more IO queues to play with?

Simon

-----Original Message-----
From: Marc A Kaplan [makaplan at us.ibm.com<mailto:makaplan at us.ibm.com>]
Sent: Wednesday, May 18, 2016 12:27 AM GMT Standard Time
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Flash for metadata

I think there are two points here:

A) RAID striping is probably a "loser" for GPFS metadata.
B) RAID mirroring for your metadata may or may not be faster and/or more reliable than GPFS replication. Depending on your requirements and assumptions for fault-tolerance one or the other might be the winner.

But .. if you're serious about performance - measure, measure, measure and pick the winner. (And we'd like to know what you found!)
And if you want the best performance, you'll probably need to (re)measure whenever you acquire new equipment.

GPFS can do and does striping for high performance when reading and writing metadata and data...

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160519/fdec78f4/attachment-0001.htm>

From makaplan at us.ibm.com  Thu May 19 22:59:32 2016
From: makaplan at us.ibm.com (Marc A Kaplan)
Date: Thu, 19 May 2016 17:59:32 -0400
Subject: [gpfsug-discuss] Flash for metadata
In-Reply-To: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F578235F@EX13.adf.bham.ac.uk>
References: <201605172327.u4HNRYAv023425@d03av05.boulder.ibm.com>
	<CF45EE16DEF2FE4B9AA7FF2B6EE26545F578235F@EX13.adf.bham.ac.uk>
Message-ID: <201605192159.u4JLxZox006181@d03av05.boulder.ibm.com>

"I assume that creating several smaller LUNs on each FlashSystem in the 
same failure group is still preferable to one big LUN so we get more IO 
queues to play with?"

Traditionally, more spindles, more disk arms working in parallel => better 
overall performance.

HOWEVER Flash doesn't work that way... So it's going to depend... 
Perhaps some kind soul can point us to some information about this and how 
much it varies among today's flash based storage products.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160519/939c1ca6/attachment-0001.htm>

From RWelp at uk.ibm.com  Fri May 20 20:47:15 2016
From: RWelp at uk.ibm.com (Richard Welp)
Date: Fri, 20 May 2016 20:47:15 +0100
Subject: [gpfsug-discuss] Flash for metadata
In-Reply-To: <201605192159.u4JLxZox006181@d03av05.boulder.ibm.com>
References: <201605172327.u4HNRYAv023425@d03av05.boulder.ibm.com>	<CF45EE16DEF2FE4B9AA7FF2B6EE26545F578235F@EX13.adf.bham.ac.uk>
	<201605192159.u4JLxZox006181@d03av05.boulder.ibm.com>
Message-ID: <201605201947.u4KJlKJT000383@d06av06.portsmouth.uk.ibm.com>

I checked with the FS900 performance expert in Tucson, and here is what I 
was told:

The 4KB and 512B blocks will both get equally great sub millisecond 
response times but 4KB can achieve a higher maximum IOPS rate.

As far as # of luns,  it doesn't really matter to the FS900, but the host 
operating system and other components in the data path can benefit from 
having more than 1 large lun.  If we are trying to get the maximum 
possible iops, we typically run with at least 16 luns.    I suspect with 4 
luns you would get within 10% of the maximum performance. 

Thanks,

Rick

===================
Rick Welp
Software Engineer
Master Inventor
Email: rwelp at uk.ibm.com
phone: +44 0161 214 0461 
IBM Systems - Manchester Lab
IBM UK Limited 
--------------------------




From:   "Marc A Kaplan" <makaplan at us.ibm.com>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   19/05/2016 11:00 pm
Subject:        Re: [gpfsug-discuss] Flash for metadata
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



"I assume that creating several smaller LUNs on each FlashSystem in the 
same failure group is still preferable to one big LUN so we get more IO 
queues to play with?"

Traditionally, more spindles, more disk arms working in parallel => better 
overall performance.

HOWEVER Flash doesn't work that way... So it's going to depend... 
Perhaps some kind soul can point us to some information about this and how 
much it varies among today's flash based storage products.
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160520/5f7d4126/attachment-0001.htm>

From S.J.Thompson at bham.ac.uk  Fri May 20 22:15:57 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Fri, 20 May 2016 21:15:57 +0000
Subject: [gpfsug-discuss] Flash for metadata
In-Reply-To: <201605201947.u4KJlKJT000383@d06av06.portsmouth.uk.ibm.com>
References: <201605172327.u4HNRYAv023425@d03av05.boulder.ibm.com>
	<CF45EE16DEF2FE4B9AA7FF2B6EE26545F578235F@EX13.adf.bham.ac.uk>
	<201605192159.u4JLxZox006181@d03av05.boulder.ibm.com>,
	<201605201947.u4KJlKJT000383@d06av06.portsmouth.uk.ibm.com>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F5782A0D@EX13.adf.bham.ac.uk>


Thanks Rick for checking this.

On number of LUNs, I vaguely had this in the back of my head, when we deployed our first Storwise, that was one of the reasons we built lots of metadata mirrored pairs rather than bigger arrays.

Thinking back I remember reading it was something to do with multipath and how IO queues are processed back to the storage device. The storwise we have is dual active, so only one half "owns" the LUN, so I recall it was optimal to encourage gpfs to load over controllers by having more multipath accessible devices.

And then thinking how you cable up to you LSI fc adapters as well....

Simon

________________________________________
From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Richard Welp [RWelp at uk.ibm.com]
Sent: 20 May 2016 20:47
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Flash for metadata

I checked with the FS900 performance expert in Tucson, and here is what I was told:

The 4KB and 512B blocks will both get equally great sub millisecond response times but 4KB can achieve a higher maximum IOPS rate.

As far as # of luns,  it doesn't really matter to the FS900, but the host operating system and other components in the data path can benefit from having more than 1 large lun.  If we are trying to get the maximum possible iops, we typically run with at least 16 luns.    I suspect with 4 luns you would get within 10% of the maximum performance.

Thanks,

Rick

===================
Rick Welp
Software Engineer
Master Inventor
Email: rwelp at uk.ibm.com
phone: +44 0161 214 0461
IBM Systems - Manchester Lab
IBM UK Limited
--------------------------




From:        "Marc A Kaplan" <makaplan at us.ibm.com>
To:        gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:        19/05/2016 11:00 pm
Subject:        Re: [gpfsug-discuss] Flash for metadata
Sent by:        gpfsug-discuss-bounces at spectrumscale.org
________________________________



"I assume that creating several smaller LUNs on each FlashSystem in the same failure group is still preferable to one big LUN so we get more IO queues to play with?"

Traditionally, more spindles, more disk arms working in parallel => better overall performance.

HOWEVER Flash doesn't work that way... So it's going to depend...
Perhaps some kind soul can point us to some information about this and how much it varies among today's flash based storage products.
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss




From usa-principal at gpfsug.org  Mon May 23 12:31:39 2016
From: usa-principal at gpfsug.org (usa-principal-gpfsug.org)
Date: Mon, 23 May 2016 07:31:39 -0400
Subject: [gpfsug-discuss] Speakers Wanted - June 10, Argonne National Lab
Message-ID: <1fa21d083401a0a7acff84b7cd69ee34@webmail.gpfsug.org>

Hello all,

	We?d love to have 2 more site reports for the June 10th event at 
Argonne. As a reminder, the event details are here: 
http://www.spectrumscale.org/spectrum-scale-user-group-usa-meeting-june-10th/


	Please reply and let us know what you'd like to present. It's always 
good to hear real experiences.

	Thanks in advance for your participation.

Best,
Kristy



From p.childs at qmul.ac.uk  Tue May 24 15:00:46 2016
From: p.childs at qmul.ac.uk (Peter Childs)
Date: Tue, 24 May 2016 14:00:46 +0000
Subject: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
In-Reply-To: <5744538D.3030407@qmul.ac.uk>
References: <AM2PR07MB0948D12D638C1ABD1AD603E3A44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>,
	<5744538D.3030407@qmul.ac.uk>
Message-ID: <AM2PR07MB09480DBC9D614EE7B5826AAAA44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>

 Hi All,

We have an issue where the Linux kills off GPFS first when a computer runs out of memory, this happens when user processors have exhausted memory and swap and the out of memory killer in Linux kills the GPFS daemon as the largest user of memory, due to its large pinned memory foot print.

We have an issue where the Linux kills off GPFS first when a computer runs out of memory. We are running GPFS 3.5

We believe this happens when user processes have exhausted memory and swap and the out of memory killer in Linux chooses to  kill the GPFS daemon as the largest user of memory, due to its large pinned memory footprint.

This means that GPFS is killed and the whole cluster blocks for a minute before it resumes operation, this is not ideal, and kills and causes issues with most of the cluster.

What we see is users unable to login elsewhere on the cluster until we have powered off the node. We believe this is because while the node is still pingable, GPFS doesn't expel it from the cluster.

This issue mainly occurs on our frontend nodes of our HPC cluster but can effect the rest of the cluster when it occurs.

This issue mainly occurs on the login nodes of our HPC cluster but can affect the rest of the cluster when it occurs.

I've seen others on list with this issue.

We've come up with a solution where by the gpfs is adjusted so that is unlikely to be the first thing to be killed, and hopefully the user process is killed and not GPFS.

We've come up with a solution to adjust the OOM score of GPFS, so that it is unlikely to be the first thing to be killed, and hopefully the OOM killer picks a user process instead.

Out testing says this solution works, but I'm asking here firstly to share our knowledge and secondly to ask if there is anything we've missed with this solution and issues with this.

We've tested this and it seems to work. I'm asking here firstly to share our knowledge and secondly to ask if there is anything we've missed with this solution.

Its short which is part of its beauty.

/usr/local/sbin/gpfs-oom_score_adj

<pre>
#!/bin/bash

 for proc in $(pgrep mmfs); do
      echo -500 >/proc/$proc/oom_score_adj
 done
</pre>

This can then be called automatically on GPFS startup with the following:

<pre>
mmaddcallback startupoomkiller --command /usr/local/sbin/gpfs-oom_score_adj --event startup
</pre>

and either restart gpfs or just run the script on all nodes.

Peter Childs
ITS Research Infrastructure
Queen Mary, University of London

From Paul.Sanchez at deshaw.com  Tue May 24 15:32:55 2016
From: Paul.Sanchez at deshaw.com (Sanchez, Paul)
Date: Tue, 24 May 2016 14:32:55 +0000
Subject: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
In-Reply-To: <AM2PR07MB09480DBC9D614EE7B5826AAAA44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>
References: <AM2PR07MB0948D12D638C1ABD1AD603E3A44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>,
	<5744538D.3030407@qmul.ac.uk>
	<AM2PR07MB09480DBC9D614EE7B5826AAAA44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>
Message-ID: <d6028bcf19f8402b9315ab875d075ba4@mbxtoa1.winmail.deshaw.com>

Hi Peter,



This is mentioned explicitly in the Spectrum Scale docs (http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.pdg.doc/bl1pdg_kerncfg.htm?lang=en) as a problem for the admin to consider, and many of us have been bitten by this.  There are references going back at least to GPFS 3.1 in 2008 on developerworks complaining about this situation.



While the answer you described below is essentially what we do as well, I would argue that this is a problem which IBM should just own and fix for everyone. I cannot think of a situation in which you would want GPFS to be sacrificed on a node due to out-of-memory conditions, and I have seen several terrible consequences of this, including loss of cached, user-acknowledged writes.



I don't think there are any real gotchas.  But in addition, our own implementation also:



  * uses "--event preStartup" instead of "startup", since it runs earlier and reduces the risk of a race



  * reads the score back out and complains if it hasn't been set



  * includes "set -e" to ensure that errors will terminate the script and return a non-zero exit code to the callback parent



Thx
Paul



-----Original Message-----
From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Peter Childs
Sent: Tuesday, May 24, 2016 10:01 AM
To: gpfsug main discussion list
Subject: [gpfsug-discuss] OOM Killer killing off GPFS 3.5



Hi All,



We have an issue where the Linux kills off GPFS first when a computer runs out of memory, this happens when user processors have exhausted memory and swap and the out of memory killer in Linux kills the GPFS daemon as the largest user of memory, due to its large pinned memory foot print.



We have an issue where the Linux kills off GPFS first when a computer runs out of memory. We are running GPFS 3.5



We believe this happens when user processes have exhausted memory and swap and the out of memory killer in Linux chooses to  kill the GPFS daemon as the largest user of memory, due to its large pinned memory footprint.



This means that GPFS is killed and the whole cluster blocks for a minute before it resumes operation, this is not ideal, and kills and causes issues with most of the cluster.



What we see is users unable to login elsewhere on the cluster until we have powered off the node. We believe this is because while the node is still pingable, GPFS doesn't expel it from the cluster.



This issue mainly occurs on our frontend nodes of our HPC cluster but can effect the rest of the cluster when it occurs.



This issue mainly occurs on the login nodes of our HPC cluster but can affect the rest of the cluster when it occurs.



I've seen others on list with this issue.



We've come up with a solution where by the gpfs is adjusted so that is unlikely to be the first thing to be killed, and hopefully the user process is killed and not GPFS.



We've come up with a solution to adjust the OOM score of GPFS, so that it is unlikely to be the first thing to be killed, and hopefully the OOM killer picks a user process instead.



Out testing says this solution works, but I'm asking here firstly to share our knowledge and secondly to ask if there is anything we've missed with this solution and issues with this.



We've tested this and it seems to work. I'm asking here firstly to share our knowledge and secondly to ask if there is anything we've missed with this solution.



Its short which is part of its beauty.



/usr/local/sbin/gpfs-oom_score_adj



<pre>

#!/bin/bash



for proc in $(pgrep mmfs); do

      echo -500 >/proc/$proc/oom_score_adj  done </pre>



This can then be called automatically on GPFS startup with the following:



<pre>

mmaddcallback startupoomkiller --command /usr/local/sbin/gpfs-oom_score_adj --event startup </pre>



and either restart gpfs or just run the script on all nodes.



Peter Childs

ITS Research Infrastructure

Queen Mary, University of London

_______________________________________________

gpfsug-discuss mailing list

gpfsug-discuss at spectrumscale.org

http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160524/e0679f0a/attachment-0001.htm>

From S.J.Thompson at bham.ac.uk  Tue May 24 15:33:07 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Tue, 24 May 2016 14:33:07 +0000
Subject: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
Message-ID: <D36A2390.2AD83%s.j.thompson@bham.ac.uk>

Hi Peter,

We used to see exactly this problem as well. We tweaked the init script on
our systems to adjust the OOM killer (we also tweaked it to wait for IB
links to come up as well). Its something we need to revisit now we are
running systemd based systems ...

Simon

On 24/05/2016, 15:00, "gpfsug-discuss-bounces at spectrumscale.org on behalf
of Peter Childs" <gpfsug-discuss-bounces at spectrumscale.org on behalf of
p.childs at qmul.ac.uk> wrote:

> Hi All,
>
>We have an issue where the Linux kills off GPFS first when a computer
>runs out of memory, this happens when user processors have exhausted
>memory and swap and the out of memory killer in Linux kills the GPFS
>daemon as the largest user of memory, due to its large pinned memory foot
>print.
>
>We have an issue where the Linux kills off GPFS first when a computer
>runs out of memory. We are running GPFS 3.5
>
>We believe this happens when user processes have exhausted memory and
>swap and the out of memory killer in Linux chooses to  kill the GPFS
>daemon as the largest user of memory, due to its large pinned memory
>footprint.
>
>This means that GPFS is killed and the whole cluster blocks for a minute
>before it resumes operation, this is not ideal, and kills and causes
>issues with most of the cluster.
>
>What we see is users unable to login elsewhere on the cluster until we
>have powered off the node. We believe this is because while the node is
>still pingable, GPFS doesn't expel it from the cluster.
>
>This issue mainly occurs on our frontend nodes of our HPC cluster but can
>effect the rest of the cluster when it occurs.
>
>This issue mainly occurs on the login nodes of our HPC cluster but can
>affect the rest of the cluster when it occurs.
>
>I've seen others on list with this issue.
>
>We've come up with a solution where by the gpfs is adjusted so that is
>unlikely to be the first thing to be killed, and hopefully the user
>process is killed and not GPFS.
>
>We've come up with a solution to adjust the OOM score of GPFS, so that it
>is unlikely to be the first thing to be killed, and hopefully the OOM
>killer picks a user process instead.
>
>Out testing says this solution works, but I'm asking here firstly to
>share our knowledge and secondly to ask if there is anything we've missed
>with this solution and issues with this.
>
>We've tested this and it seems to work. I'm asking here firstly to share
>our knowledge and secondly to ask if there is anything we've missed with
>this solution.
>
>Its short which is part of its beauty.
>
>/usr/local/sbin/gpfs-oom_score_adj
>
><pre>
>#!/bin/bash
>
> for proc in $(pgrep mmfs); do
>      echo -500 >/proc/$proc/oom_score_adj
> done
></pre>
>
>This can then be called automatically on GPFS startup with the following:
>
><pre>
>mmaddcallback startupoomkiller --command
>/usr/local/sbin/gpfs-oom_score_adj --event startup
></pre>
>
>and either restart gpfs or just run the script on all nodes.
>
>Peter Childs
>ITS Research Infrastructure
>Queen Mary, University of London
>_______________________________________________
>gpfsug-discuss mailing list
>gpfsug-discuss at spectrumscale.org
>http://gpfsug.org/mailman/listinfo/gpfsug-discuss



From volobuev at us.ibm.com  Tue May 24 17:17:12 2016
From: volobuev at us.ibm.com (Yuri L Volobuev)
Date: Tue, 24 May 2016 09:17:12 -0700
Subject: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
In-Reply-To: <d6028bcf19f8402b9315ab875d075ba4@mbxtoa1.winmail.deshaw.com>
References: <AM2PR07MB0948D12D638C1ABD1AD603E3A44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>,
	<5744538D.3030407@qmul.ac.uk><AM2PR07MB09480DBC9D614EE7B5826AAAA44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>
	<d6028bcf19f8402b9315ab875d075ba4@mbxtoa1.winmail.deshaw.com>
Message-ID: <20160524161719.AA629AC03F@b01ledav006.gho.pok.ibm.com>


This problem is more complex than it may seem.  The thing is, mmfsd runs as
root, as thus already possesses a certain amount of natural immunity to OOM
killer.  So adjusting mmfsd oom_score_adj doesn't radically change the
ranking of OOM killer victims, only tweaks it.  The way things are supposed
to work is: a user process eats up a lot of memory, and once a threshold is
hit, OOM killer picks off the memory hog, and the memory is released.
Unprivileged processes inherently have a higher OOM score, and should be
killed off first.  If that doesn't work, for some reason, the OOM killer
gets desperate and starts going after root processes.  Once things get to
this point, it's tough.  If you somehow manage to spare mmfsd per se,
what's going to happen next?  The OOM killer still needs a victim.  What
we've seen happen in such a situation is semi-random privileged process
killing.  mmfsd stays alive, but various other system processes are picked
off, and pretty quickly the node is a basket case.  A Linux node is not
very resilient to random process killing.  And it doesn't help that those
other privileged processes usually don't use much memory, so killing them
doesn't release much, and the carnage keeps on going.  The real problem is:
why wasn't the non-privileged memory hog process killed off first, before
root processes became fair game?  This is where things get pretty
complicated, and depend heavily on the Linux version.  There's one specific
issue that did get diagnosed.  If a process is using mmap and has page
faults going that result in GPFS IO, on older versions of GPFS the process
would fail to error out after a SIGKILL, due to locking complications
spanning Linux kernel VMM and GPFS mmap code.  This means the OOM killer
would attempt to kill a process, but that wouldn't produce the desired
result (the process is still around), and the OOM killer keeps moving down
the list.  This problem has been fixed in the current GPFS service levels.
It is possible that a similar problem may exist that prevents a memory hog
process from erroring out.  I strongly encourage opening a PMR to
investigate such a situation, instead of trying to work around it without
understanding why mmfsd was targeted in the first place.

This is the case of prevention being the best cure.  Where we've seen
success is customers using cgroups to prevent user processes from running a
node out of memory in the first place.  This has been shown to work well.
Dealing with the fallout from running out of memory is a much harder task.

The post-mmfsd-kill symptoms that are described in the original note are
not normal.  If an mmfsd process is killed, other nodes will become aware
of this fact faily quickly, and the node is going to be expelled from the
cluster (yes, expels *can* be a good thing).  In the normal case, TCP/IP
sockets are closed as soon as mmfsd is killed, and other nodes immediately
receive TCP RST packets, and close their connection endpoints.  If the
worst case, if a node just becomes catatonic, but RST is not sent out, the
troubled node is going to be expelled from the cluster after about 2
minutes of pinging (in a default configuration).  There should definitely
not be a permanent hang that necessitates a manual intervention.  Again,
older versions of GPFS had no protection against surprise OOM thread kills,
but in the current code some self-monitoring capabilities have been added,
and a single troubled node won't have a lasting impact on the cluster.  If
you aren't running with a reasonably current level of GPFS 3.5 service, I
strongly recommend upgrading.  If you see the symptoms originally described
with the current code, that's a bug that we need to fix, so please open a
PMR to address the issue.

yuri



From:	"Sanchez, Paul" <Paul.Sanchez at deshaw.com>
To:	gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>,
Date:	05/24/2016 07:33 AM
Subject:	Re: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



Hi Peter,

This is mentioned explicitly in the Spectrum Scale docs (
http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.pdg.doc/bl1pdg_kerncfg.htm?lang=en
) as a problem for the admin to consider, and many of us have been bitten
by this.  There are references going back at least to GPFS 3.1 in 2008 on
developerworks complaining about this situation.

While the answer you described below is essentially what we do as well, I
would argue that this is a problem which IBM should just own and fix for
everyone. I cannot think of a situation in which you would want GPFS to be
sacrificed on a node due to out-of-memory conditions, and I have seen
several terrible consequences of this, including loss of cached,
user-acknowledged writes.

I don't think there are any real gotchas.  But in addition, our own
implementation also:

  * uses "--event preStartup" instead of "startup", since it runs earlier
and reduces the risk of a race

  * reads the score back out and complains if it hasn't been set

  * includes "set -e" to ensure that errors will terminate the script and
return a non-zero exit code to the callback parent

Thx
Paul

-----Original Message-----
From: gpfsug-discuss-bounces at spectrumscale.org [
mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Peter Childs
Sent: Tuesday, May 24, 2016 10:01 AM
To: gpfsug main discussion list
Subject: [gpfsug-discuss] OOM Killer killing off GPFS 3.5

Hi All,

We have an issue where the Linux kills off GPFS first when a computer runs
out of memory, this happens when user processors have exhausted memory and
swap and the out of memory killer in Linux kills the GPFS daemon as the
largest user of memory, due to its large pinned memory foot print.

We have an issue where the Linux kills off GPFS first when a computer runs
out of memory. We are running GPFS 3.5

We believe this happens when user processes have exhausted memory and swap
and the out of memory killer in Linux chooses to  kill the GPFS daemon as
the largest user of memory, due to its large pinned memory footprint.

This means that GPFS is killed and the whole cluster blocks for a minute
before it resumes operation, this is not ideal, and kills and causes issues
with most of the cluster.

What we see is users unable to login elsewhere on the cluster until we have
powered off the node. We believe this is because while the node is still
pingable, GPFS doesn't expel it from the cluster.

This issue mainly occurs on our frontend nodes of our HPC cluster but can
effect the rest of the cluster when it occurs.

This issue mainly occurs on the login nodes of our HPC cluster but can
affect the rest of the cluster when it occurs.

I've seen others on list with this issue.

We've come up with a solution where by the gpfs is adjusted so that is
unlikely to be the first thing to be killed, and hopefully the user process
is killed and not GPFS.

We've come up with a solution to adjust the OOM score of GPFS, so that it
is unlikely to be the first thing to be killed, and hopefully the OOM
killer picks a user process instead.

Out testing says this solution works, but I'm asking here firstly to share
our knowledge and secondly to ask if there is anything we've missed with
this solution and issues with this.

We've tested this and it seems to work. I'm asking here firstly to share
our knowledge and secondly to ask if there is anything we've missed with
this solution.

Its short which is part of its beauty.

/usr/local/sbin/gpfs-oom_score_adj

<pre>
#!/bin/bash

for proc in $(pgrep mmfs); do
      echo -500 >/proc/$proc/oom_score_adj  done </pre>

This can then be called automatically on GPFS startup with the following:

<pre>
mmaddcallback startupoomkiller --command /usr/local/sbin/gpfs-oom_score_adj
--event startup </pre>

and either restart gpfs or just run the script on all nodes.

Peter Childs
ITS Research Infrastructure
Queen Mary, University of London
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160524/43490dfb/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160524/43490dfb/attachment-0001.gif>

From volobuev at us.ibm.com  Tue May 24 20:25:06 2016
From: volobuev at us.ibm.com (Yuri L Volobuev)
Date: Tue, 24 May 2016 12:25:06 -0700
Subject: [gpfsug-discuss] Flash for metadata
In-Reply-To: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F577FC47@EX13.adf.bham.ac.uk>
References: <49a3ab3b493a49e59177c6c25dd34d23@mbxtoa1.winmail.deshaw.com>
	<CF45EE16DEF2FE4B9AA7FF2B6EE26545F577FC47@EX13.adf.bham.ac.uk>
Message-ID: <20160524192511.E6BFD6E041@b03ledav001.gho.boulder.ibm.com>

GPFS will issue 4K IOs if the file system is 4K-aligned, which is the
default for 4.1.1+, and may issue requests as small as 512 bytes otherwise.
If a given array presents 4K physical block size to the world, you won't be
able to format a non-4K-aligned file system which puts metadata on that
disk.  It is generally believed that 4K IO size produces optimal
performance with flash-based storage in general, because this is the native
physical block size, and 512-byte sector emulation entails doing a
read-modify-write.  We don't have the data on how much difference the
sector size actually makes.  If in doubt, I would recommend going with 4K
sectors and a 4K-aligned file system.

yuri



From:	"Simon Thompson (Research Computing - IT Services)"
            <S.J.Thompson at bham.ac.uk>
To:	"'gpfsug-discuss at spectrumscale.org'"
            <gpfsug-discuss at spectrumscale.org>,
Date:	05/16/2016 02:48 AM
Subject:	Re: [gpfsug-discuss] Flash for metadata
Sent by:	gpfsug-discuss-bounces at spectrumscale.org




Hi Paul,

Thanks, interestingly the FS-900 only does raid5. I get no choice about
that, I do get a sector size choice when I create a lun on it...

I was also wondering how gpfs would do a metadata update. Would it be smart
enough to know the sector size was 512b and only modify that, or would it
always do a read modify write on the whole 4k inode?

Simon

-----Original Message-----
From: Sanchez, Paul [Paul.Sanchez at deshaw.com]
Sent: Monday, May 16, 2016 10:28 AM GMT Standard Time
To: gpfsug-discuss at spectrumscale.org
Subject: Re: [gpfsug-discuss] Flash for metadata

Simon,

I can?t speak specifically to the FS-900, since I generally use Dell
Compellent and NetApp EF arrays for flash today.  But I also take care to
ensure that the system pool metadata NSDs use RAID10 (or equivalent) rather
than RAID5/6.  For small 4K metadata writes, on a 4K blocksize LUN this
could result in optimal writes.  For larger native blocksize LUNs, it
eliminates the parity calculation for read-modify-write operations.

Thanks,
Paul

From: gpfsug-discuss-bounces at spectrumscale.org [
mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon
Thompson (Research Computing - IT Services)
Sent: Monday, May 16, 2016 9:44 AM
To: gpfsug-discuss at spectrumscale.org
Subject: [gpfsug-discuss] Flash for metadata

Hi all

I'm currently about to deploy some flash systems (FS-900) to take our
metadata for one of our Spectrum Scale file-systems. We use 4k inodes for
this file-system.

On the FS-900, when creating a volume, we get a choice of 512b or 4k
sectors. Does anyone have a suggestion on this? On a spinning disk RAID
array, I get what a strip size is and we'd use multiples of the stripe
where possible for the block size.

Is the sector size on the FS-900 the smallest chunk that can be written
out? And so would it make sense to use 4k sectors for 4k GPFS inodes right?

Related. In the deployment guide for the FS-900 arrays, if one is using a
file-system (e.g. XFS) on the array the guide talks about aligning the
partitions with with array, when GPFS is used on a storage array, does it
automatically work out the sector alignment?


Thanks

Simon_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160524/3a332e95/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160524/3a332e95/attachment-0001.gif>

From Paul.Sanchez at deshaw.com  Wed May 25 15:00:35 2016
From: Paul.Sanchez at deshaw.com (Sanchez, Paul)
Date: Wed, 25 May 2016 14:00:35 +0000
Subject: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
In-Reply-To: <20160524161719.AA629AC03F@b01ledav006.gho.pok.ibm.com>
References: <AM2PR07MB0948D12D638C1ABD1AD603E3A44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>,
	<5744538D.3030407@qmul.ac.uk><AM2PR07MB09480DBC9D614EE7B5826AAAA44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>
	<d6028bcf19f8402b9315ab875d075ba4@mbxtoa1.winmail.deshaw.com>,
	<20160524161719.AA629AC03F@b01ledav006.gho.pok.ibm.com>
Message-ID: <51fa215cfdc149c6b9c6cdeb50322d88@mbxtoa1.winmail.deshaw.com>

I'm sure that Yuri is right about the corner-case complexity across all linux and Spectrum/GPFS versions.

In situations where lots of outstanding tokens exist, and there are few token managers, we have seen the assassination of a large footprint mmfsd in GPFS 4.1 seem to impact entire clusters, potentially due to serialization in recovery of so many tokens, and overlapping access among nodes. We're looking forward to fixes in 4.2.1 to address some of this too.

But for what it's worth, on RH6/7 with 4.1, we have seen the end of OOM impacting GPFS since implementing the callback. One item I forgot is that we don't set it to -500, but to OOM_SCORE_ADJ_MIN, which on our systems is -1000. That causes the heuristic oom_badness to return the lowest possible score, more thoroughly immunizing it against selection.

Thx
Paul

Sent with Good Work (www.good.com)


From: Yuri L Volobuev <volobuev at us.ibm.com<mailto:volobuev at us.ibm.com>>
Date: Tuesday, May 24, 2016, 12:17 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] OOM Killer killing off GPFS 3.5


This problem is more complex than it may seem. The thing is, mmfsd runs as root, as thus already possesses a certain amount of natural immunity to OOM killer. So adjusting mmfsd oom_score_adj doesn't radically change the ranking of OOM killer victims, only tweaks it. The way things are supposed to work is: a user process eats up a lot of memory, and once a threshold is hit, OOM killer picks off the memory hog, and the memory is released. Unprivileged processes inherently have a higher OOM score, and should be killed off first. If that doesn't work, for some reason, the OOM killer gets desperate and starts going after root processes. Once things get to this point, it's tough. If you somehow manage to spare mmfsd per se, what's going to happen next? The OOM killer still needs a victim. What we've seen happen in such a situation is semi-random privileged process killing. mmfsd stays alive, but various other system processes are picked off, and pretty quickly the node is a basket case. A Linux node is not very resilient to random process killing. And it doesn't help that those other privileged processes usually don't use much memory, so killing them doesn't release much, and the carnage keeps on going. The real problem is: why wasn't the non-privileged memory hog process killed off first, before root processes became fair game? This is where things get pretty complicated, and depend heavily on the Linux version. There's one specific issue that did get diagnosed. If a process is using mmap and has page faults going that result in GPFS IO, on older versions of GPFS the process would fail to error out after a SIGKILL, due to locking complications spanning Linux kernel VMM and GPFS mmap code. This means the OOM killer would attempt to kill a process, but that wouldn't produce the desired result (the process is still around), and the OOM killer keeps moving down the list. This problem has been fixed in the current GPFS service levels. It is possible that a similar problem may exist that prevents a memory hog process from erroring out. I strongly encourage opening a PMR to investigate such a situation, instead of trying to work around it without understanding why mmfsd was targeted in the first place.

This is the case of prevention being the best cure. Where we've seen success is customers using cgroups to prevent user processes from running a node out of memory in the first place. This has been shown to work well. Dealing with the fallout from running out of memory is a much harder task.

The post-mmfsd-kill symptoms that are described in the original note are not normal. If an mmfsd process is killed, other nodes will become aware of this fact faily quickly, and the node is going to be expelled from the cluster (yes, expels *can* be a good thing). In the normal case, TCP/IP sockets are closed as soon as mmfsd is killed, and other nodes immediately receive TCP RST packets, and close their connection endpoints. If the worst case, if a node just becomes catatonic, but RST is not sent out, the troubled node is going to be expelled from the cluster after about 2 minutes of pinging (in a default configuration). There should definitely not be a permanent hang that necessitates a manual intervention. Again, older versions of GPFS had no protection against surprise OOM thread kills, but in the current code some self-monitoring capabilities have been added, and a single troubled node won't have a lasting impact on the cluster. If you aren't running with a reasonably current level of GPFS 3.5 service, I strongly recommend upgrading. If you see the symptoms originally described with the current code, that's a bug that we need to fix, so please open a PMR to address the issue.

yuri

[Inactive hide details for "Sanchez, Paul" ---05/24/2016 07:33:18 AM---Hi Peter, This is mentioned explicitly in the Spectrum Sc]"Sanchez, Paul" ---05/24/2016 07:33:18 AM---Hi Peter, This is mentioned explicitly in the Spectrum Scale docs (http://www.ibm.com/support/knowle

From: "Sanchez, Paul" <Paul.Sanchez at deshaw.com>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>,
Date: 05/24/2016 07:33 AM
Subject: Re: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
Sent by: gpfsug-discuss-bounces at spectrumscale.org

________________________________



Hi Peter,

This is mentioned explicitly in the Spectrum Scale docs (http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.pdg.doc/bl1pdg_kerncfg.htm?lang=en) as a problem for the admin to consider, and many of us have been bitten by this. There are references going back at least to GPFS 3.1 in 2008 on developerworks complaining about this situation.

While the answer you described below is essentially what we do as well, I would argue that this is a problem which IBM should just own and fix for everyone. I cannot think of a situation in which you would want GPFS to be sacrificed on a node due to out-of-memory conditions, and I have seen several terrible consequences of this, including loss of cached, user-acknowledged writes.

I don't think there are any real gotchas. But in addition, our own implementation also:

* uses "--event preStartup" instead of "startup", since it runs earlier and reduces the risk of a race

* reads the score back out and complains if it hasn't been set

* includes "set -e" to ensure that errors will terminate the script and return a non-zero exit code to the callback parent

Thx
Paul

-----Original Message-----
From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Peter Childs
Sent: Tuesday, May 24, 2016 10:01 AM
To: gpfsug main discussion list
Subject: [gpfsug-discuss] OOM Killer killing off GPFS 3.5

Hi All,

We have an issue where the Linux kills off GPFS first when a computer runs out of memory, this happens when user processors have exhausted memory and swap and the out of memory killer in Linux kills the GPFS daemon as the largest user of memory, due to its large pinned memory foot print.

We have an issue where the Linux kills off GPFS first when a computer runs out of memory. We are running GPFS 3.5

We believe this happens when user processes have exhausted memory and swap and the out of memory killer in Linux chooses to kill the GPFS daemon as the largest user of memory, due to its large pinned memory footprint.

This means that GPFS is killed and the whole cluster blocks for a minute before it resumes operation, this is not ideal, and kills and causes issues with most of the cluster.

What we see is users unable to login elsewhere on the cluster until we have powered off the node. We believe this is because while the node is still pingable, GPFS doesn't expel it from the cluster.

This issue mainly occurs on our frontend nodes of our HPC cluster but can effect the rest of the cluster when it occurs.

This issue mainly occurs on the login nodes of our HPC cluster but can affect the rest of the cluster when it occurs.

I've seen others on list with this issue.

We've come up with a solution where by the gpfs is adjusted so that is unlikely to be the first thing to be killed, and hopefully the user process is killed and not GPFS.

We've come up with a solution to adjust the OOM score of GPFS, so that it is unlikely to be the first thing to be killed, and hopefully the OOM killer picks a user process instead.

Out testing says this solution works, but I'm asking here firstly to share our knowledge and secondly to ask if there is anything we've missed with this solution and issues with this.

We've tested this and it seems to work. I'm asking here firstly to share our knowledge and secondly to ask if there is anything we've missed with this solution.

Its short which is part of its beauty.

/usr/local/sbin/gpfs-oom_score_adj

<pre>
#!/bin/bash

for proc in $(pgrep mmfs); do
echo -500 >/proc/$proc/oom_score_adj done </pre>

This can then be called automatically on GPFS startup with the following:

<pre>
mmaddcallback startupoomkiller --command /usr/local/sbin/gpfs-oom_score_adj --event startup </pre>

and either restart gpfs or just run the script on all nodes.

Peter Childs
ITS Research Infrastructure
Queen Mary, University of London
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160525/fed71308/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: graycol.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160525/fed71308/attachment-0002.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: graycol.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160525/fed71308/attachment-0003.gif>

From bdeluca at gmail.com  Wed May 25 15:09:06 2016
From: bdeluca at gmail.com (Ben De Luca)
Date: Wed, 25 May 2016 17:09:06 +0300
Subject: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
In-Reply-To: <51fa215cfdc149c6b9c6cdeb50322d88@mbxtoa1.winmail.deshaw.com>
References: <AM2PR07MB0948D12D638C1ABD1AD603E3A44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>
	<5744538D.3030407@qmul.ac.uk>
	<AM2PR07MB09480DBC9D614EE7B5826AAAA44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>
	<d6028bcf19f8402b9315ab875d075ba4@mbxtoa1.winmail.deshaw.com>
	<20160524161719.AA629AC03F@b01ledav006.gho.pok.ibm.com>
	<51fa215cfdc149c6b9c6cdeb50322d88@mbxtoa1.winmail.deshaw.com>
Message-ID: <CAGC__DgTuomeon_tD2BSw0WdTFZQvEeKcQ-yX2EKbzykXHma0g@mail.gmail.com>

Not now, but in a previous role, we would specifically increase the oom
score on computer processes on our cluster that could consume a large
amount of ram, trying to protect system processes. Once did this we had 0
system processes die.



On 25 May 2016 at 17:00, Sanchez, Paul <Paul.Sanchez at deshaw.com> wrote:

> I'm sure that Yuri is right about the corner-case complexity across all
> linux and Spectrum/GPFS versions.
>
> In situations where lots of outstanding tokens exist, and there are few
> token managers, we have seen the assassination of a large footprint mmfsd
> in GPFS 4.1 seem to impact entire clusters, potentially due to
> serialization in recovery of so many tokens, and overlapping access among
> nodes. We're looking forward to fixes in 4.2.1 to address some of this too.
>
> But for what it's worth, on RH6/7 with 4.1, we have seen the end of OOM
> impacting GPFS since implementing the callback. One item I forgot is that
> we don't set it to -500, but to OOM_SCORE_ADJ_MIN, which on our systems is
> -1000. That causes the heuristic oom_badness to return the lowest possible
> score, more thoroughly immunizing it against selection.
>
> Thx
> Paul
>
> Sent with Good Work (www.good.com)
>
>
> *From: *Yuri L Volobuev <volobuev at us.ibm.com>
> *Date: *Tuesday, May 24, 2016, 12:17 PM
> *To: *gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> *Subject: *Re: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
>
> This problem is more complex than it may seem. The thing is, mmfsd runs as
> root, as thus already possesses a certain amount of natural immunity to OOM
> killer. So adjusting mmfsd oom_score_adj doesn't radically change the
> ranking of OOM killer victims, only tweaks it. The way things are supposed
> to work is: a user process eats up a lot of memory, and once a threshold is
> hit, OOM killer picks off the memory hog, and the memory is released.
> Unprivileged processes inherently have a higher OOM score, and should be
> killed off first. If that doesn't work, for some reason, the OOM killer
> gets desperate and starts going after root processes. Once things get to
> this point, it's tough. If you somehow manage to spare mmfsd per se, what's
> going to happen next? The OOM killer still needs a victim. What we've seen
> happen in such a situation is semi-random privileged process killing. mmfsd
> stays alive, but various other system processes are picked off, and pretty
> quickly the node is a basket case. A Linux node is not very resilient to
> random process killing. And it doesn't help that those other privileged
> processes usually don't use much memory, so killing them doesn't release
> much, and the carnage keeps on going. The real problem is: why wasn't the
> non-privileged memory hog process killed off first, before root processes
> became fair game? This is where things get pretty complicated, and depend
> heavily on the Linux version. There's one specific issue that did get
> diagnosed. If a process is using mmap and has page faults going that result
> in GPFS IO, on older versions of GPFS the process would fail to error out
> after a SIGKILL, due to locking complications spanning Linux kernel VMM and
> GPFS mmap code. This means the OOM killer would attempt to kill a process,
> but that wouldn't produce the desired result (the process is still around),
> and the OOM killer keeps moving down the list. This problem has been fixed
> in the current GPFS service levels. It is possible that a similar problem
> may exist that prevents a memory hog process from erroring out. I strongly
> encourage opening a PMR to investigate such a situation, instead of trying
> to work around it without understanding why mmfsd was targeted in the first
> place.
>
> This is the case of prevention being the best cure. Where we've seen
> success is customers using cgroups to prevent user processes from running a
> node out of memory in the first place. This has been shown to work well.
> Dealing with the fallout from running out of memory is a much harder task.
>
> The post-mmfsd-kill symptoms that are described in the original note are
> not normal. If an mmfsd process is killed, other nodes will become aware of
> this fact faily quickly, and the node is going to be expelled from the
> cluster (yes, expels *can* be a good thing). In the normal case, TCP/IP
> sockets are closed as soon as mmfsd is killed, and other nodes immediately
> receive TCP RST packets, and close their connection endpoints. If the worst
> case, if a node just becomes catatonic, but RST is not sent out, the
> troubled node is going to be expelled from the cluster after about 2
> minutes of pinging (in a default configuration). There should definitely
> not be a permanent hang that necessitates a manual intervention. Again,
> older versions of GPFS had no protection against surprise OOM thread kills,
> but in the current code some self-monitoring capabilities have been added,
> and a single troubled node won't have a lasting impact on the cluster. If
> you aren't running with a reasonably current level of GPFS 3.5 service, I
> strongly recommend upgrading. If you see the symptoms originally described
> with the current code, that's a bug that we need to fix, so please open a
> PMR to address the issue.
>
> yuri
>
> [image: Inactive hide details for "Sanchez, Paul" ---05/24/2016 07:33:18
> AM---Hi Peter, This is mentioned explicitly in the Spectrum Sc]"Sanchez,
> Paul" ---05/24/2016 07:33:18 AM---Hi Peter, This is mentioned explicitly in
> the Spectrum Scale docs (http://www.ibm.com/support/knowle
>
> From: "Sanchez, Paul" <Paul.Sanchez at deshaw.com>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>,
> Date: 05/24/2016 07:33 AM
> Subject: Re: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> ------------------------------
>
>
>
> Hi Peter,
>
> This is mentioned explicitly in the Spectrum Scale docs (
> http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.pdg.doc/bl1pdg_kerncfg.htm?lang=en)
> as a problem for the admin to consider, and many of us have been bitten by
> this. There are references going back at least to GPFS 3.1 in 2008 on
> developerworks complaining about this situation.
>
> While the answer you described below is essentially what we do as well, I
> would argue that this is a problem which IBM should just own and fix for
> everyone. I cannot think of a situation in which you would want GPFS to
> be sacrificed on a node due to out-of-memory conditions, and I have seen
> several terrible consequences of this, including loss of cached,
> user-acknowledged writes.
>
> I don't think there are any real gotchas. But in addition, our own
> implementation also:
>
> * uses "--event preStartup" instead of "startup", since it runs earlier
> and reduces the risk of a race
>
> * reads the score back out and complains if it hasn't been set
>
> * includes "set -e" to ensure that errors will terminate the script and
> return a non-zero exit code to the callback parent
>
> Thx
> Paul
>
> -----Original Message-----
> From: gpfsug-discuss-bounces at spectrumscale.org [
> mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>] On Behalf Of Peter Childs
> Sent: Tuesday, May 24, 2016 10:01 AM
> To: gpfsug main discussion list
> Subject: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
>
> Hi All,
>
> We have an issue where the Linux kills off GPFS first when a computer runs
> out of memory, this happens when user processors have exhausted memory and
> swap and the out of memory killer in Linux kills the GPFS daemon as the
> largest user of memory, due to its large pinned memory foot print.
>
> We have an issue where the Linux kills off GPFS first when a computer runs
> out of memory. We are running GPFS 3.5
>
> We believe this happens when user processes have exhausted memory and swap
> and the out of memory killer in Linux chooses to kill the GPFS daemon as
> the largest user of memory, due to its large pinned memory footprint.
>
> This means that GPFS is killed and the whole cluster blocks for a minute
> before it resumes operation, this is not ideal, and kills and causes issues
> with most of the cluster.
>
> What we see is users unable to login elsewhere on the cluster until we
> have powered off the node. We believe this is because while the node is
> still pingable, GPFS doesn't expel it from the cluster.
>
> This issue mainly occurs on our frontend nodes of our HPC cluster but can
> effect the rest of the cluster when it occurs.
>
> This issue mainly occurs on the login nodes of our HPC cluster but can
> affect the rest of the cluster when it occurs.
>
> I've seen others on list with this issue.
>
> We've come up with a solution where by the gpfs is adjusted so that is
> unlikely to be the first thing to be killed, and hopefully the user process
> is killed and not GPFS.
>
> We've come up with a solution to adjust the OOM score of GPFS, so that it
> is unlikely to be the first thing to be killed, and hopefully the OOM
> killer picks a user process instead.
>
> Out testing says this solution works, but I'm asking here firstly to share
> our knowledge and secondly to ask if there is anything we've missed with
> this solution and issues with this.
>
> We've tested this and it seems to work. I'm asking here firstly to share
> our knowledge and secondly to ask if there is anything we've missed with
> this solution.
>
> Its short which is part of its beauty.
>
> /usr/local/sbin/gpfs-oom_score_adj
>
> <pre>
> #!/bin/bash
>
> for proc in $(pgrep mmfs); do
> echo -500 >/proc/$proc/oom_score_adj done </pre>
>
> This can then be called automatically on GPFS startup with the following:
>
> <pre>
> mmaddcallback startupoomkiller --command
> /usr/local/sbin/gpfs-oom_score_adj --event startup </pre>
>
> and either restart gpfs or just run the script on all nodes.
>
> Peter Childs
> ITS Research Infrastructure
> Queen Mary, University of London
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160525/f137a550/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160525/f137a550/attachment-0001.gif>

From Luke.Raimbach at crick.ac.uk  Thu May 26 09:20:02 2016
From: Luke.Raimbach at crick.ac.uk (Luke Raimbach)
Date: Thu, 26 May 2016 08:20:02 +0000
Subject: [gpfsug-discuss] CES and Centrify
Message-ID: <AMXPR03MB02105F338038F05AAD3C73BB0410@AMXPR03MB021.eurprd03.prod.outlook.com>

Hi,

Someone said "Centrify" at the recent UK user-group meeting.

Where can I find out more?

Cheers,
Luke.

Luke Raimbach?
Senior HPC Data and Storage Systems Engineer,
The Francis Crick Institute,
Gibbs Building,
215 Euston Road,
London NW1 2BE.

E: luke.raimbach at crick.ac.uk
W: www.crick.ac.uk

The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no. 06885462, with its registered office at 215 Euston Road, London NW1 2BE.

From scottcumbie at dynamixgroup.com  Thu May 26 16:57:14 2016
From: scottcumbie at dynamixgroup.com (Scott Cumbie)
Date: Thu, 26 May 2016 15:57:14 +0000
Subject: [gpfsug-discuss] mmbackup - Two at a time?
Message-ID: <35131C5F-56B9-4098-B711-4B59A025B5BB@dynamixgroup.com>

Just wondering if anyone has a quick answer to this question.  If mmbackup is running on a filesystem and a second mmbackup is executed on the same filesystem, what happens?  Does the second one end because the first one is running?

Thanks,
Scott




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160526/18d3fadb/attachment-0001.htm>

From malone12 at illinois.edu  Thu May 26 19:33:33 2016
From: malone12 at illinois.edu (Maloney, John Daniel)
Date: Thu, 26 May 2016 18:33:33 +0000
Subject: [gpfsug-discuss] Speakers Wanted - June 10, Argonne National Lab
In-Reply-To: <1fa21d083401a0a7acff84b7cd69ee34@webmail.gpfsug.org>
References: <1fa21d083401a0a7acff84b7cd69ee34@webmail.gpfsug.org>
Message-ID: <2AEEB2A9-E7C2-4F55-AA42-7C34117A9F37@illinois.edu>

Hi Kristy,
I?d be willing to present at the User Group Meeting on the 10th at Argonne.  Our talk would be based on our roll out of a new GPFS environment that leverages remote cluster mounts, AFM, and the new cluster export services functionality in 4.2 to help us better serve our campus users who leverage U of I?s campus cluster.  Let me know if a slot is still available and if/when you need more information or slides.

Best,

J.D. Maloney
Storage Engineer | Storage Enabling Technologies Group
National Center for Supercomputing Applications (NCSA)
University of Illinois Urbana-Champaign
1205 W. Clark St. Urbana, IL 61801









On 5/23/16, 6:31 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of usa-principal-gpfsug.org" <gpfsug-discuss-bounces at spectrumscale.org on behalf of usa-principal at gpfsug.org> wrote:

>http://www.spectrumscale.org/spectrum-scale-user-group-usa-meeting-june-10th/

From Robert.Oesterlin at nuance.com  Thu May 26 20:03:44 2016
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Thu, 26 May 2016 19:03:44 +0000
Subject: [gpfsug-discuss] Speakers Wanted - June 10, Argonne National Lab
In-Reply-To: <2AEEB2A9-E7C2-4F55-AA42-7C34117A9F37@illinois.edu>
References: <1fa21d083401a0a7acff84b7cd69ee34@webmail.gpfsug.org>
	<2AEEB2A9-E7C2-4F55-AA42-7C34117A9F37@illinois.edu>
Message-ID: <BCB09BBF-2893-4CF3-83B9-78A2E217D314@nuance.com>

J.D.

We may still have one open slot - can you contact me directly and let's talk details?

robert.oesterlin  @ nuance.com


Bob Oesterlin
Sr Storage Engineer, Nuance HPC Grid
507-269-0413


From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of "Maloney, John Daniel" <malone12 at illinois.edu>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Thursday, May 26, 2016 at 1:33 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Speakers Wanted - June 10, Argonne National Lab

Hi Kristy,

I?d be willing to present at the User Group Meeting on the 10th at Argonne.  Our talk would be based on our roll out of a new GPFS environment that leverages remote cluster mounts, AFM, and the new cluster export services functionality in 4.2 to help us better serve our campus users who leverage U of I?s campus cluster.  Let me know if a slot is still available and if/when you need more information or slides.



Best,



J.D. Maloney

Storage Engineer | Storage Enabling Technologies Group

National Center for Supercomputing Applications (NCSA)

University of Illinois Urbana-Champaign

1205 W. Clark St. Urbana, IL 61801



















On 5/23/16, 6:31 AM, "gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> on behalf of usa-principal-gpfsug.org" <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> on behalf of usa-principal at gpfsug.org<mailto:usa-principal at gpfsug.org>> wrote:



https://urldefense.proofpoint.com/v2/url?u=http-3A__www.spectrumscale.org_spectrum-2Dscale-2Duser-2Dgroup-2Dusa-2Dmeeting-2Djune-2D10th_&d=CwIGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=MKKI9kld3C0B3tzQT64SPW1uCBHTb72hZlRP6x4V34E&s=sUYYuCYXPiSXA6Nhejs1H_sogi2WDfREIRc5yir2FJ4&e=

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwIGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=MKKI9kld3C0B3tzQT64SPW1uCBHTb72hZlRP6x4V34E&s=lORUKCzYSOh63hgiyP_35POyN_hMzfYGVVx8kQfFl-c&e=

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160526/cc0b7542/attachment-0001.htm>

From craigawilson at gmail.com  Fri May 27 11:32:09 2016
From: craigawilson at gmail.com (Craig Wilson)
Date: Fri, 27 May 2016 11:32:09 +0100
Subject: [gpfsug-discuss] mmbackup - Two at a time?
In-Reply-To: <35131C5F-56B9-4098-B711-4B59A025B5BB@dynamixgroup.com>
References: <35131C5F-56B9-4098-B711-4B59A025B5BB@dynamixgroup.com>
Message-ID: <CALdwc7dd1b+oF3jXMnJFwKtf9QKJ7m-kRETViwA0cmp2oMYJZg@mail.gmail.com>

Hi Scott

I've not tried this one 4.# branch as I'm still running 3.5 but on 3.5 you
get a error message saying:

mmbackup: Failed to obtain the local environment update lock.
mmbackup: Command failed.  Examine previous error messages to determine
cause.

The old mmbackup keeps running through.

Kind Regards,

Craig Wilson.
HPC Systems Engineer
OCF Plc.

On 26 May 2016 at 16:57, Scott Cumbie <scottcumbie at dynamixgroup.com> wrote:

> Just wondering if anyone has a quick answer to this question.  If mmbackup
> is running on a filesystem and a second mmbackup is executed on the same
> filesystem, what happens?  Does the second one end because the first one is
> running?
>
> Thanks,
> Scott
>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160527/456339d2/attachment-0001.htm>

From S.J.Thompson at bham.ac.uk  Fri May 27 11:49:27 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Fri, 27 May 2016 10:49:27 +0000
Subject: [gpfsug-discuss] mmbackup - Two at a time?
In-Reply-To: <CALdwc7dd1b+oF3jXMnJFwKtf9QKJ7m-kRETViwA0cmp2oMYJZg@mail.gmail.com>
References: <35131C5F-56B9-4098-B711-4B59A025B5BB@dynamixgroup.com>
	<CALdwc7dd1b+oF3jXMnJFwKtf9QKJ7m-kRETViwA0cmp2oMYJZg@mail.gmail.com>
Message-ID: <D36DE461.2AF90%s.j.thompson@bham.ac.uk>

Now we've seen different to this... But maybe it depends how you run mmbackup multiple times.

We have several nodes which are backing up via Spectrum Protect, and we thought it would be nice to have the client scheduler running on multiple nodes, we assumed it would only actually run one instance of the backup. Not so. We've seen two client schedulers kick off mmbackup (each then I think running across several nodes).

Simon

From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of Craig Wilson <craigawilson at gmail.com<mailto:craigawilson at gmail.com>>
Reply-To: "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Friday, 27 May 2016 at 11:32
To: "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] mmbackup - Two at a time?

Hi Scott

I've not tried this one 4.# branch as I'm still running 3.5 but on 3.5 you get a error message saying:

mmbackup: Failed to obtain the local environment update lock.
mmbackup: Command failed.  Examine previous error messages to determine cause.

The old mmbackup keeps running through.

Kind Regards,

Craig Wilson.
HPC Systems Engineer
OCF Plc.

On 26 May 2016 at 16:57, Scott Cumbie <scottcumbie at dynamixgroup.com<mailto:scottcumbie at dynamixgroup.com>> wrote:
Just wondering if anyone has a quick answer to this question.  If mmbackup is running on a filesystem and a second mmbackup is executed on the same filesystem, what happens?  Does the second one end because the first one is running?

Thanks,
Scott





_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160527/f431c56f/attachment-0001.htm>

From craigawilson at gmail.com  Fri May 27 14:36:24 2016
From: craigawilson at gmail.com (Craig Wilson)
Date: Fri, 27 May 2016 14:36:24 +0100
Subject: [gpfsug-discuss] mmbackup - Two at a time?
In-Reply-To: <D36DE461.2AF90%s.j.thompson@bham.ac.uk>
References: <35131C5F-56B9-4098-B711-4B59A025B5BB@dynamixgroup.com>
	<CALdwc7dd1b+oF3jXMnJFwKtf9QKJ7m-kRETViwA0cmp2oMYJZg@mail.gmail.com>
	<D36DE461.2AF90%s.j.thompson@bham.ac.uk>
Message-ID: <CALdwc7cu5dmCaP__aAv5htnxV4vH8GO7pVMKmUg8WBuLxBo35w@mail.gmail.com>

That is interesting...

The only time I've seen two starting is when someone manually cleared the
environment lock. That said I've never tried running from a different node
at the same time. I presume they would be duplicating effort based on the
node name being used in the shadow database file name.

Kind regards,

Craig Wilson.

On 27 May 2016 at 11:49, Simon Thompson (Research Computing - IT Services) <
S.J.Thompson at bham.ac.uk> wrote:

> Now we've seen different to this... But maybe it depends how you run
> mmbackup multiple times.
>
> We have several nodes which are backing up via Spectrum Protect, and we
> thought it would be nice to have the client scheduler running on multiple
> nodes, we assumed it would only actually run one instance of the backup.
> Not so. We've seen two client schedulers kick off mmbackup (each then I
> think running across several nodes).
>
> Simon
>
> From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Craig
> Wilson <craigawilson at gmail.com>
> Reply-To: "gpfsug-discuss at spectrumscale.org" <
> gpfsug-discuss at spectrumscale.org>
> Date: Friday, 27 May 2016 at 11:32
> To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
> Subject: Re: [gpfsug-discuss] mmbackup - Two at a time?
>
> Hi Scott
>
> I've not tried this one 4.# branch as I'm still running 3.5 but on 3.5 you
> get a error message saying:
>
> mmbackup: Failed to obtain the local environment update lock.
> mmbackup: Command failed.  Examine previous error messages to determine
> cause.
>
> The old mmbackup keeps running through.
>
> Kind Regards,
>
> Craig Wilson.
> HPC Systems Engineer
> OCF Plc.
>
> On 26 May 2016 at 16:57, Scott Cumbie <scottcumbie at dynamixgroup.com>
> wrote:
>
>> Just wondering if anyone has a quick answer to this question.  If
>> mmbackup is running on a filesystem and a second mmbackup is executed on
>> the same filesystem, what happens?  Does the second one end because the
>> first one is running?
>>
>> Thanks,
>> Scott
>>
>>
>>
>>
>>
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>
>>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160527/c2fba71c/attachment-0001.htm>

From Kevin.Buterbaugh at Vanderbilt.Edu  Fri May 27 21:23:01 2016
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Fri, 27 May 2016 20:23:01 +0000
Subject: [gpfsug-discuss] Building portability layer fails
Message-ID: <FEEC06BD-FE26-4F2B-84D6-AD604AB63208@vanderbilt.edu>

All,

We?re upgrading our cluster from GPFS 4.1.0.8 to GPFS 4.2.0.3 and all is going very well ? except for my tape backup server.

I?ve run the Spectrum Scale installer and installed the rpm?s in /usr/lpp/mmfs/4.2.0.3/gpfs_rpms, but when I run:

make LINUX_DISTRIBUTION=REDHAT_AS_LINUX Autoconfig

it errors with:

make[3]: *** [/usr/lpp/mmfs/src/gpl-linux/kdump-kern.o] Error 1
make[2]: *** [_module_/usr/lpp/mmfs/src/gpl-linux] Error 2
make[2]: Leaving directory `/usr/src/kernels/2.6.32-573.el6.x86_64'
make[1]: *** [modules] Error 1
make[1]: Leaving directory `/usr/lpp/mmfs/src/gpl-linux'
make: *** [Modules] Error 1

Any ideas?  Thanks in advance?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160527/9513f318/attachment-0001.htm>

From sfadden at us.ibm.com  Fri May 27 21:35:58 2016
From: sfadden at us.ibm.com (Scott Fadden)
Date: Fri, 27 May 2016 13:35:58 -0700
Subject: [gpfsug-discuss] Building portability layer fails
In-Reply-To: <FEEC06BD-FE26-4F2B-84D6-AD604AB63208@vanderbilt.edu>
References: <FEEC06BD-FE26-4F2B-84D6-AD604AB63208@vanderbilt.edu>
Message-ID: <20160527203604.0F1F86A045@b03ledav003.gho.boulder.ibm.com>

On 4.2.0.3 run mmbuildgpl. See how that goes.


Scott Fadden
Spectrum Scale - Technical Marketing 
Phone: (503) 880-5833 
sfadden at us.ibm.com
http://www.ibm.com/systems/storage/spectrum/scale



From:   "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   05/27/2016 01:23 PM
Subject:        [gpfsug-discuss] Building portability layer fails
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



All, 

We?re upgrading our cluster from GPFS 4.1.0.8 to GPFS 4.2.0.3 and all is 
going very well ? except for my tape backup server.

I?ve run the Spectrum Scale installer and installed the rpm?s in 
/usr/lpp/mmfs/4.2.0.3/gpfs_rpms, but when I run:

make LINUX_DISTRIBUTION=REDHAT_AS_LINUX Autoconfig

it errors with:

make[3]: *** [/usr/lpp/mmfs/src/gpl-linux/kdump-kern.o] Error 1
make[2]: *** [_module_/usr/lpp/mmfs/src/gpl-linux] Error 2
make[2]: Leaving directory `/usr/src/kernels/2.6.32-573.el6.x86_64'
make[1]: *** [modules] Error 1
make[1]: Leaving directory `/usr/lpp/mmfs/src/gpl-linux'
make: *** [Modules] Error 1

Any ideas?  Thanks in advance?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and 
Education
Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160527/d0a3d54b/attachment-0001.htm>

From Kevin.Buterbaugh at Vanderbilt.Edu  Fri May 27 21:51:18 2016
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Fri, 27 May 2016 20:51:18 +0000
Subject: [gpfsug-discuss] Building portability layer fails
In-Reply-To: <20160527203604.0F1F86A045@b03ledav003.gho.boulder.ibm.com>
References: <FEEC06BD-FE26-4F2B-84D6-AD604AB63208@vanderbilt.edu>
	<20160527203604.0F1F86A045@b03ledav003.gho.boulder.ibm.com>
Message-ID: <4E17167D-5A7A-4A04-8A56-03F4DDB092CB@vanderbilt.edu>

All,

After spending a couple of hours on this I finally took the key step towards fixing the problem myself ? I e-mailed this list!  ;-)

Five minutes later, I saw that the env.mcr file was munged, fixed that, and everything built just fine.  Sorry for the noise?

Kevin

On May 27, 2016, at 3:35 PM, Scott Fadden <sfadden at us.ibm.com<mailto:sfadden at us.ibm.com>> wrote:

On 4.2.0.3 run mmbuildgpl. See how that goes.


Scott Fadden
Spectrum Scale - Technical Marketing
Phone: (503) 880-5833
sfadden at us.ibm.com<mailto:sfadden at us.ibm.com>
http://www.ibm.com/systems/storage/spectrum/scale



From:        "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at vanderbilt.edu>>
To:        gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date:        05/27/2016 01:23 PM
Subject:        [gpfsug-discuss] Building portability layer fails
Sent by:        gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>
________________________________



All,

We?re upgrading our cluster from GPFS 4.1.0.8 to GPFS 4.2.0.3 and all is going very well ? except for my tape backup server.

I?ve run the Spectrum Scale installer and installed the rpm?s in /usr/lpp/mmfs/4.2.0.3/gpfs_rpms, but when I run:

make LINUX_DISTRIBUTION=REDHAT_AS_LINUX Autoconfig

it errors with:

make[3]: *** [/usr/lpp/mmfs/src/gpl-linux/kdump-kern.o] Error 1
make[2]: *** [_module_/usr/lpp/mmfs/src/gpl-linux] Error 2
make[2]: Leaving directory `/usr/src/kernels/2.6.32-573.el6.x86_64'
make[1]: *** [modules] Error 1
make[1]: Leaving directory `/usr/lpp/mmfs/src/gpl-linux'
make: *** [Modules] Error 1

Any ideas?  Thanks in advance?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu>- (615)875-9633


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160527/58a0de66/attachment-0001.htm>

From S.J.Thompson at bham.ac.uk  Fri May 27 22:44:48 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Fri, 27 May 2016 21:44:48 +0000
Subject: [gpfsug-discuss] Building portability layer fails
In-Reply-To: <20160527203604.0F1F86A045@b03ledav003.gho.boulder.ibm.com>
References: <20160527203604.0F1F86A045@b03ledav003.gho.boulder.ibm.com>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F5786AF5@EX13.adf.bham.ac.uk>

Does mmbuildgpl work if its a RHEL clone like CentOS?

Simon

-----Original Message-----
From: Scott Fadden [sfadden at us.ibm.com<mailto:sfadden at us.ibm.com>]
Sent: Friday, May 27, 2016 09:36 PM GMT Standard Time
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Building portability layer fails

On 4.2.0.3 run mmbuildgpl. See how that goes.


Scott Fadden
Spectrum Scale - Technical Marketing
Phone: (503) 880-5833
sfadden at us.ibm.com
http://www.ibm.com/systems/storage/spectrum/scale



From:        "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu>
To:        gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:        05/27/2016 01:23 PM
Subject:        [gpfsug-discuss] Building portability layer fails
Sent by:        gpfsug-discuss-bounces at spectrumscale.org
________________________________



All,

We?re upgrading our cluster from GPFS 4.1.0.8 to GPFS 4.2.0.3 and all is going very well ? except for my tape backup server.

I?ve run the Spectrum Scale installer and installed the rpm?s in /usr/lpp/mmfs/4.2.0.3/gpfs_rpms, but when I run:

make LINUX_DISTRIBUTION=REDHAT_AS_LINUX Autoconfig

it errors with:

make[3]: *** [/usr/lpp/mmfs/src/gpl-linux/kdump-kern.o] Error 1
make[2]: *** [_module_/usr/lpp/mmfs/src/gpl-linux] Error 2
make[2]: Leaving directory `/usr/src/kernels/2.6.32-573.el6.x86_64'
make[1]: *** [modules] Error 1
make[1]: Leaving directory `/usr/lpp/mmfs/src/gpl-linux'
make: *** [Modules] Error 1

Any ideas?  Thanks in advance?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu>- (615)875-9633


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160527/c587e999/attachment-0001.htm>

From sfadden at us.ibm.com  Sat May 28 01:47:05 2016
From: sfadden at us.ibm.com (Scott Fadden)
Date: Sat, 28 May 2016 00:47:05 +0000
Subject: [gpfsug-discuss] Building portability layer fails
In-Reply-To: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F5786AF5@EX13.adf.bham.ac.uk>
Message-ID: <20160528004711.1B4A46A03D@b03ledav003.gho.boulder.ibm.com>

It should

Sent from IBM Verse


   Simon Thompson (Research Computing - IT Services) --- RE: Re: [gpfsug-discuss] Building portability layer fails --- 
    From:"Simon Thompson (Research Computing - IT Services)" <S.J.Thompson at bham.ac.uk>To:"'gpfsug main discussion list'" <gpfsug-discuss at spectrumscale.org>, "Scott Fadden" <sfadden at us.ibm.com>Date:Fri, May 27, 2016 14:45Subject:RE: Re: [gpfsug-discuss] Building portability layer fails
  
         Does mmbuildgpl work if its a RHEL clone like CentOS?
  
  Simon
  
  -----Original Message-----
  From: Scott Fadden [sfadden at us.ibm.com]
  Sent: Friday, May 27, 2016 09:36 PM GMT Standard Time
  To: gpfsug main discussion list
  Subject: Re: [gpfsug-discuss] Building portability layer fails
  
  On 4.2.0.3 run mmbuildgpl. See how that goes.
  
  
  Scott Fadden
  Spectrum Scale - Technical Marketing
  Phone: (503) 880-5833  
  sfadden at us.ibm.com
  http://www.ibm.com/systems/storage/spectrum/scale
  
  
  
  From:        "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu>
  To:        gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
  Date:        05/27/2016 01:23 PM
  Subject:        [gpfsug-discuss] Building portability layer fails
  Sent by:        gpfsug-discuss-bounces at spectrumscale.org
   
  
  
  All,
  
  We?re upgrading our cluster from GPFS 4.1.0.8 to GPFS 4.2.0.3 and all is going very well ? except for my tape backup server.
  
  I?ve run the Spectrum Scale installer and installed the rpm?s in /usr/lpp/mmfs/4.2.0.3/gpfs_rpms, but when I run:
  
  make LINUX_DISTRIBUTION=REDHAT_AS_LINUX Autoconfig
  
  it errors with:
  
  make[3]: *** [/usr/lpp/mmfs/src/gpl-linux/kdump-kern.o] Error 1
  make[2]: *** [_module_/usr/lpp/mmfs/src/gpl-linux] Error 2
  make[2]: Leaving directory `/usr/src/kernels/2.6.32-573.el6.x86_64'
  make[1]: *** [modules] Error 1
  make[1]: Leaving directory `/usr/lpp/mmfs/src/gpl-linux'
  make: *** [Modules] Error 1
  
  Any ideas?  Thanks in advance?
  
  ?
  Kevin Buterbaugh - Senior System Administrator
  Vanderbilt University - Advanced Computing Center for Research and Education
  Kevin.Buterbaugh at vanderbilt.edu- (615)875-9633
  
  
  _______________________________________________
  gpfsug-discuss mailing list
  gpfsug-discuss at spectrumscale.org
  http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160528/e1576726/attachment-0001.htm>

From laurence at qsplace.co.uk  Sat May 28 09:36:50 2016
From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow)
Date: Sat, 28 May 2016 09:36:50 +0100
Subject: [gpfsug-discuss] Building portability layer fails
In-Reply-To: <20160528004711.1B4A46A03D@b03ledav003.gho.boulder.ibm.com>
References: <20160528004711.1B4A46A03D@b03ledav003.gho.boulder.ibm.com>
Message-ID: <16F6CB71-3BC1-4F9F-BA2F-DC6911325C8A@qsplace.co.uk>

Yes it does.

But you still have to export LINUX_DISTRIBUTION=REDHAT_AS_LINUX? before the command.

i.e.
LINUX_DISTRIBUTION=REDHAT_AS_LINUX? mmbuildgpl

-- Lauz

On 28 May 2016 01:47:05 BST, Scott Fadden <sfadden at us.ibm.com> wrote:
>It should
>
>Sent from IBM Verse
>
>
>Simon Thompson (Research Computing - IT Services) --- RE: Re:
>[gpfsug-discuss] Building portability layer fails --- 
>From:"Simon Thompson (Research Computing - IT Services)"
><S.J.Thompson at bham.ac.uk>To:"'gpfsug main discussion list'"
><gpfsug-discuss at spectrumscale.org>, "Scott Fadden"
><sfadden at us.ibm.com>Date:Fri, May 27, 2016 14:45Subject:RE: Re:
>[gpfsug-discuss] Building portability layer fails
>  
>         Does mmbuildgpl work if its a RHEL clone like CentOS?
>  
>  Simon
>  
>  -----Original Message-----
>  From: Scott Fadden [sfadden at us.ibm.com]
>  Sent: Friday, May 27, 2016 09:36 PM GMT Standard Time
>  To: gpfsug main discussion list
>  Subject: Re: [gpfsug-discuss] Building portability layer fails
>  
>  On 4.2.0.3 run mmbuildgpl. See how that goes.
>  
>  
>  Scott Fadden
>  Spectrum Scale - Technical Marketing
>  Phone: (503) 880-5833  
>  sfadden at us.ibm.com
>  http://www.ibm.com/systems/storage/spectrum/scale
>  
>  
>  
>  From:        "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu>
>To:        gpfsug main discussion list
><gpfsug-discuss at spectrumscale.org>
>  Date:        05/27/2016 01:23 PM
>  Subject:        [gpfsug-discuss] Building portability layer fails
>  Sent by:        gpfsug-discuss-bounces at spectrumscale.org
>   
>  
>  
>  All,
>  
>We?re upgrading our cluster from GPFS 4.1.0.8 to GPFS 4.2.0.3 and all
>is going very well ? except for my tape backup server.
>  
>I?ve run the Spectrum Scale installer and installed the rpm?s in
>/usr/lpp/mmfs/4.2.0.3/gpfs_rpms, but when I run:
>  
>  make LINUX_DISTRIBUTION=REDHAT_AS_LINUX Autoconfig
>  
>  it errors with:
>  
>  make[3]: *** [/usr/lpp/mmfs/src/gpl-linux/kdump-kern.o] Error 1
>  make[2]: *** [_module_/usr/lpp/mmfs/src/gpl-linux] Error 2
>  make[2]: Leaving directory `/usr/src/kernels/2.6.32-573.el6.x86_64'
>  make[1]: *** [modules] Error 1
>  make[1]: Leaving directory `/usr/lpp/mmfs/src/gpl-linux'
>  make: *** [Modules] Error 1
>  
>  Any ideas?  Thanks in advance?
>  
>  ?
>  Kevin Buterbaugh - Senior System Administrator
>Vanderbilt University - Advanced Computing Center for Research and
>Education
>  Kevin.Buterbaugh at vanderbilt.edu- (615)875-9633
>  
>  
>  _______________________________________________
>  gpfsug-discuss mailing list
>  gpfsug-discuss at spectrumscale.org
>  http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>gpfsug-discuss mailing list
>gpfsug-discuss at spectrumscale.org
>http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160528/077f9264/attachment-0001.htm>

From Robert.Oesterlin at nuance.com  Tue May 31 12:57:19 2016
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Tue, 31 May 2016 11:57:19 +0000
Subject: [gpfsug-discuss] GPFS/Spectrum Scale security vulernability - All
	versions
Message-ID: <E0694FAD-6D65-40AA-AB98-D6934E24D557@nuance.com>

IBM published a security vulnerability today that effects all current and prior levels of GPFS/Spectrum Scale. The short explanation is "IBM Spectrum Scale and IBM GPFS that could allow a local attacker to inject commands into setuid file parameters and execute commands as root." This is a "high" vulnerability (8.4).

Details here: http://www-01.ibm.com/support/docview.wss?uid=ssg1S1005781


Bob Oesterlin
Sr Storage Engineer, Nuance HPC Grid


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160531/184da280/attachment-0001.htm>

From TROPPENS at de.ibm.com  Tue May  3 13:05:51 2016
From: TROPPENS at de.ibm.com (Ulf Troppens)
Date: Tue, 3 May 2016 14:05:51 +0200
Subject: [gpfsug-discuss] GPFS/Spectrum Scale Upcoming US Events - Save
 the Dates - Registration for New York May 26
In-Reply-To: <9489DBA2-1F12-4B05-A968-5D4855FBEA40@gpfsug.org>
References: <4192D644-E3AB-4B7B-AF7A-96D3D617FA7B@gpfsug.org>
	<9489DBA2-1F12-4B05-A968-5D4855FBEA40@gpfsug.org>
Message-ID: <201605031206.u43C5wss002725@d06av09.portsmouth.uk.ibm.com>

Registration for the first event in New York is up as well - the agenda
still needs some touch.

Please register here and select 'May 26 GPFS Day Registration', if you
don't attend SPXXL:
https://www.spxxl.org/?q=New-York-City-2016
https://www.eventbrite.com/e/spxxlscicomp-2016-summer-meeting-registration-24444020724


--
IBM Spectrum Scale Development - Client Engagements & Solutions Delivery
Consulting IT Specialist
Author "Storage Networks Explained"

IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Gesch?ftsf?hrung: Dirk Wittkopp
Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart,
HRB 243294




From:	GPFS UG USA Principal <usa-principal at gpfsug.org>
To:	gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Date:	28.04.2016 23:44
Subject:	Re: [gpfsug-discuss] GPFS/Spectrum Scale Upcoming US Events -
            Save	the Dates
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



All, the registration page for the second event listed below at Argonne
National Lab on June 10th is now up. An updated agenda is also at this
site.

Please register here: https://www.regonline.com/Spectrumscalemeeting

We look forward to seeing some of you at these upcoming events.

Feel free to send suggestions for future events in your area.

Cheers,
-Kristy


      On Apr 4, 2016, at 4:52 PM, GPFS UG USA Principal <
      usa-principal at gpfsug.org> wrote:

      Hello all,

      We?d like to announce two upcoming US GPFS/Spectrum Scale Events. One
      on the east coast, one in the midwest.

      1) May 26th (full day event): GPFS/Spectrum Scale Day at the SPXXL
      conference in NYC https://www.spxxl.org/?q=New-York-City-2016
      Developers and Engineers from IBM will be at the meeting to cover
      topics, and open dialogue between IBM and customers will be
      encouraged. More details about the agenda, hotel and other logistics
      will come later this month.

      Tentative Agenda:
      ? 10 reasons for upgrading from GPFS 3.5 to Spectrum Scale 4.2.1
      ? Enhancements for CORAL from IBM
      ? Panel discussion with customers, topic TBD
      ? AFM and integration with Spectrum Protect
      ? Best practices for GPFS or Spectrum Scale Tuning.
      ? At least one site update

      Location:
      New York Academy of Medicine
      1216 Fifth Avenue
      New York, NY 10029

      ??

      2) June 10th (full day event): GPFS/Spectrum Scale Users Group
      Meeting at Argonne National Lab (ANL). Thanks to Argonne for hosting
      this event. Developers and Engineers from IBM will be at the meeting
      to cover topics, and open dialogue between IBM and customers will be
      encouraged, as usual no marketing pitches!

      Location: Argonne National Lab more details and final agenda will
      come later.

      Tentative Agenda:

      9:00a-12:30p
      9-9:30a - Opening Remarks
      9:30-10a Deep Dive - Update on ESS
      10a-11a Deep Dive - Problem Determination (Presentation 30 min/Panel
      30 min?)
      11-11:30 Break
      11:30a-Noon - Deep Dive - Protect & Scale integration
      Noon-12:30p HDFS/Hadoop

      12:30 - 1:30p Lunch

      1:30p-5:00p
      1:30 - 2:00p IBM AFM Update
      2:00-2:30p ANL: AFM as a burst buffer
      2:30-3:00p ANL: GHI (GPFS HPSS Integration)
      3:00-3:30p Break
      3:30p - 4:00p LANL: ? or other site preso
      4:00-4:30p Nuance: GPFS Performance Sensors Deployment Experiences
      4:30p -5:00p Closing comments and Open Forum for Questions

      5:00 - ?
      Beer hunting?

      ??

      We hope you can attend one or both of these events.

      Best,
      Kristy Kallback-Rose & Bob Oesterlin
      GPFS Users Group - USA Chapter - Principal & Co-principal
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160503/c00a2018/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160503/c00a2018/attachment-0002.gif>

From A.K.Ghumra at bham.ac.uk  Tue May  3 14:45:23 2016
From: A.K.Ghumra at bham.ac.uk (Aslam Ghumra (IT Services, Facilities Management))
Date: Tue, 3 May 2016 13:45:23 +0000
Subject: [gpfsug-discuss] SMB access speed
Message-ID: <D50747684476FC4EAA59F1BE20FD0557D7AD34BC@EX12.adf.bham.ac.uk>

Hi Yaron, 

Local hdd to hdd copy speed is approx. 323.15MB/s, not sure if its's a different hdd or different partition, hopefully will get that information later.

Yup, gui is not an option and hopefully the researcher will use command line.

Regards, Aslam
Research Computing Team
DDI: +44 (121) 414 5877 | Skype: JanitorX | Twitter: @aslamghumra | a.k.ghumra at bham.ac.uk | intranet.birmingham.ac.uk/bear


-----Original Message-----
From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of gpfsug-discuss-request at spectrumscale.org
Sent: 30 April 2016 12:00

Today's Topics:

   1. Re: SMB access speed (Yaron Daniel)


----------------------------------------------------------------------

Message: 1
Date: Sat, 30 Apr 2016 08:17:28 +0300
From: "Yaron Daniel" <YARD at il.ibm.com>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] SMB access speed
Message-ID:
	<201604300517.u3U5HcbY022432 at d06av12.portsmouth.uk.ibm.com>
Content-Type: text/plain; charset="iso-8859-1"

Hi

It could be that GUI use in the "background" default command which use smb v1.

Regard copy files from GPFS to Local HDD, it might be related to the local HDD settings.
What is the speed transfer between the local HHD ? Cache Settings and so..

 
Regards
 


 
 
Yaron Daniel
 94 Em Ha'Moshavot Rd

Server, Storage and Data Services - Team Leader  Petach Tiqva, 49527 Global Technology Services  Israel
Phone:
+972-3-916-5672
 
 
Fax:
+972-3-916-5672
 
 
Mobile:
+972-52-8395593
 
 
e-mail:
yard at il.ibm.com
 
 
IBM Israel
 
 
 
 

 



From:   "Aslam Ghumra (IT Services, Facilities Management)" 
<A.K.Ghumra at bham.ac.uk>
To:     "gpfsug-discuss at spectrumscale.org" 
<gpfsug-discuss at spectrumscale.org>
Date:   04/29/2016 07:07 PM
Subject:        [gpfsug-discuss] SMB access speed
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Many thanks Yaron, after the change to disable encryption we were able to 
increase the speed via Ubuntu of copying files from the local desktop to 
our gpfs filestore with average speeds of 60Mbps.
 
We also tried changing the mount from vers=3.0 to vers=2.1, which gave 
similar figures
 
However, using the Ubuntu gui ( Unity ) the speed drops down to 7Mbps, 
however, we?re not concerned as the user will use rsync / cp. 
 
The other issue is copying data from gpfs filestore to the local HDD, 
which resulted in 4Mbps.
 
Aslam Ghumra
Research Data Management
____________________________
IT Services
Elms Road Data Centre
Building G5
Edgbaston
Birmingham B15 2TT
T: 0121 414 5877
F; 0121 414 3952
Skype : JanitorX
Twitter : @aslamghumra
http://intranet.bham.ac.uk/bear
 _______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160430/875e5aa0/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 1851 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160430/875e5aa0/attachment-0001.gif>

------------------------------

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


End of gpfsug-discuss Digest, Vol 51, Issue 57
**********************************************


From Kevin.Buterbaugh at Vanderbilt.Edu  Tue May  3 21:20:15 2016
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Tue, 3 May 2016 20:20:15 +0000
Subject: [gpfsug-discuss] CNFS and multiple IP addresses
Message-ID: <94CBCF33-0B97-440E-8E51-F4A51E9333FB@vanderbilt.edu>

Hi All,

I?ve looked in the documentation and on the GPFS Wiki for the answer to this question but haven?t found it.  Please feel free to point me to docs / web pages if I?ve missed something.

One of the GPFS clusters I administer uses CNFS, which I did not set up on this cluster ? and the person who did is now an employee of another organization.  The cluster in question runs GPFS 4.1.0.x, although I doubt that matters.

There are 4 CNFS servers, each of which has 2 IP addresses listed under the CNFS IP address list when running ?mmlscluster ?cnfs?.  I want to delete one of those two IP addresses for each server due to a planned network reorganization.  How do I do that?  Or do I need to completely delete the server from CNFS and add it back in with just the one IP address I want to keep?

Thanks in advance?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160503/76735cf0/attachment-0002.htm>

From Kevin.Buterbaugh at Vanderbilt.Edu  Tue May  3 21:27:12 2016
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Tue, 3 May 2016 20:27:12 +0000
Subject: [gpfsug-discuss] Data replication and fileset quotas
Message-ID: <AEE0A7CC-E792-462F-9BD0-C6DF9C551E41@vanderbilt.edu>

Hi again all,

I have another question on a completely different topic and therefore I decided it was better to send two separate e-mails.

For a number of years now we have had a GPFS filesystem where we use filesets and set fileset quotas.  Data replication is set to one.  I understand how that all works.

We are creating another GPFS filesystem where we intend to also use filesets and fileset quotas, but set data replication to two.  Based on my experience with data replication on a filesystem that doesn?t use filesets, I am expecting that setting data replication to two means that I will need to double the quota for each fileset (i.e. if a group has bought 5 TB of space I?ll need to set their fileset quota to 10 TB) but haven?t found where that is explicitly documented.  Is that correct?

Thanks again, all?

Kevin

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160503/c1274617/attachment-0002.htm>

From bbanister at jumptrading.com  Tue May  3 21:31:53 2016
From: bbanister at jumptrading.com (Bryan Banister)
Date: Tue, 3 May 2016 20:31:53 +0000
Subject: [gpfsug-discuss] CNFS and multiple IP addresses
In-Reply-To: <94CBCF33-0B97-440E-8E51-F4A51E9333FB@vanderbilt.edu>
References: <94CBCF33-0B97-440E-8E51-F4A51E9333FB@vanderbilt.edu>
Message-ID: <21BC488F0AEA2245B2C3E83FC0B33DBB060CA609@CHI-EXCHANGEW1.w2k.jumptrading.com>

Hi Kevin,

Look at the man page for mmchnode, you?ll need to use the ?--cnfs-interface=<ip_address_list>? option.

BTW, I get the cleanest looking man pages for GPFS commands by using the ?-E ascii? option.

You may also have a Round-Robin DNS record setup that includes the current IP addresses that you will likely want to update, not forgetting about DNS caching timeouts that you may need to let expire before no client attempts to mount from an old address.

Hope that helps!
-Bryan

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Buterbaugh, Kevin L
Sent: Tuesday, May 03, 2016 3:20 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] CNFS and multiple IP addresses

Hi All,

I?ve looked in the documentation and on the GPFS Wiki for the answer to this question but haven?t found it.  Please feel free to point me to docs / web pages if I?ve missed something.

One of the GPFS clusters I administer uses CNFS, which I did not set up on this cluster ? and the person who did is now an employee of another organization.  The cluster in question runs GPFS 4.1.0.x, although I doubt that matters.

There are 4 CNFS servers, each of which has 2 IP addresses listed under the CNFS IP address list when running ?mmlscluster ?cnfs?.  I want to delete one of those two IP addresses for each server due to a planned network reorganization.  How do I do that?  Or do I need to completely delete the server from CNFS and add it back in with just the one IP address I want to keep?

Thanks in advance?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633




________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160503/5ffdbf58/attachment-0002.htm>

From bbanister at jumptrading.com  Tue May  3 21:33:24 2016
From: bbanister at jumptrading.com (Bryan Banister)
Date: Tue, 3 May 2016 20:33:24 +0000
Subject: [gpfsug-discuss] Data replication and fileset quotas
In-Reply-To: <AEE0A7CC-E792-462F-9BD0-C6DF9C551E41@vanderbilt.edu>
References: <AEE0A7CC-E792-462F-9BD0-C6DF9C551E41@vanderbilt.edu>
Message-ID: <21BC488F0AEA2245B2C3E83FC0B33DBB060CA62F@CHI-EXCHANGEW1.w2k.jumptrading.com>

Yes, I believe that is the case due to the fact that every file is replicated therefore taking twice as much space within the file system.
-Bryan

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Buterbaugh, Kevin L
Sent: Tuesday, May 03, 2016 3:27 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] Data replication and fileset quotas

Hi again all,

I have another question on a completely different topic and therefore I decided it was better to send two separate e-mails.

For a number of years now we have had a GPFS filesystem where we use filesets and set fileset quotas.  Data replication is set to one.  I understand how that all works.

We are creating another GPFS filesystem where we intend to also use filesets and fileset quotas, but set data replication to two.  Based on my experience with data replication on a filesystem that doesn?t use filesets, I am expecting that setting data replication to two means that I will need to double the quota for each fileset (i.e. if a group has bought 5 TB of space I?ll need to set their fileset quota to 10 TB) but haven?t found where that is explicitly documented.  Is that correct?

Thanks again, all?

Kevin

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633




________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160503/67d1cbad/attachment-0002.htm>

From Kevin.Buterbaugh at Vanderbilt.Edu  Tue May  3 21:51:48 2016
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Tue, 3 May 2016 20:51:48 +0000
Subject: [gpfsug-discuss] CNFS and multiple IP addresses
In-Reply-To: <21BC488F0AEA2245B2C3E83FC0B33DBB060CA609@CHI-EXCHANGEW1.w2k.jumptrading.com>
References: <94CBCF33-0B97-440E-8E51-F4A51E9333FB@vanderbilt.edu>
	<21BC488F0AEA2245B2C3E83FC0B33DBB060CA609@CHI-EXCHANGEW1.w2k.jumptrading.com>
Message-ID: <FBB84DD9-9B12-4607-9D98-E7DD87C7C145@vanderbilt.edu>

Hi Bryan,

Thanks for the response.

I had looked at the man page for mmchnode previously, but was misinterpreting the wording.  I was under the impression that the ??cnfs-interface? option could only be used to add IP addresses or delete a node from CNFS with the ?DEFAULT? keyword.

Your explanation makes sense - thanks for clarifying?

Kevin

On May 3, 2016, at 3:31 PM, Bryan Banister <bbanister at jumptrading.com<mailto:bbanister at jumptrading.com>> wrote:

Hi Kevin,

Look at the man page for mmchnode, you?ll need to use the ?--cnfs-interface=<ip_address_list>? option.

BTW, I get the cleanest looking man pages for GPFS commands by using the ?-E ascii? option.

You may also have a Round-Robin DNS record setup that includes the current IP addresses that you will likely want to update, not forgetting about DNS caching timeouts that you may need to let expire before no client attempts to mount from an old address.

Hope that helps!
-Bryan

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Buterbaugh, Kevin L
Sent: Tuesday, May 03, 2016 3:20 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: [gpfsug-discuss] CNFS and multiple IP addresses

Hi All,

I?ve looked in the documentation and on the GPFS Wiki for the answer to this question but haven?t found it.  Please feel free to point me to docs / web pages if I?ve missed something.

One of the GPFS clusters I administer uses CNFS, which I did not set up on this cluster ? and the person who did is now an employee of another organization.  The cluster in question runs GPFS 4.1.0.x, although I doubt that matters.

There are 4 CNFS servers, each of which has 2 IP addresses listed under the CNFS IP address list when running ?mmlscluster ?cnfs?.  I want to delete one of those two IP addresses for each server due to a planned network reorganization.  How do I do that?  Or do I need to completely delete the server from CNFS and add it back in with just the one IP address I want to keep?

Thanks in advance?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633




________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160503/4af7fe04/attachment-0002.htm>

From S.J.Thompson at bham.ac.uk  Wed May  4 08:11:18 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Wed, 4 May 2016 07:11:18 +0000
Subject: [gpfsug-discuss] Data replication and fileset quotas
In-Reply-To: <21BC488F0AEA2245B2C3E83FC0B33DBB060CA62F@CHI-EXCHANGEW1.w2k.jumptrading.com>
References: <21BC488F0AEA2245B2C3E83FC0B33DBB060CA62F@CHI-EXCHANGEW1.w2k.jumptrading.com>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F576E8B6@EX13.adf.bham.ac.uk>




Yes, this correct (I think there was some discussion on the mailing list a little while back).

The quota is the space used on disk, so if you run compression its the compressed size, tape hsm don't count until you recall them.

I assume mcstore is the same, I.e. Only counts on recall, but I haven't tested this.

Simon

--

Simon Thompson

Research Computing Team, IT Services

+44 121 415 8675

-----Original Message-----
From: Bryan Banister [bbanister at jumptrading.com<mailto:bbanister at jumptrading.com>]
Sent: Tuesday, May 03, 2016 09:33 PM GMT Standard Time
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Data replication and fileset quotas

Yes, I believe that is the case due to the fact that every file is replicated therefore taking twice as much space within the file system.
-Bryan

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Buterbaugh, Kevin L
Sent: Tuesday, May 03, 2016 3:27 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] Data replication and fileset quotas

Hi again all,

I have another question on a completely different topic and therefore I decided it was better to send two separate e-mails.

For a number of years now we have had a GPFS filesystem where we use filesets and set fileset quotas.  Data replication is set to one.  I understand how that all works.

We are creating another GPFS filesystem where we intend to also use filesets and fileset quotas, but set data replication to two.  Based on my experience with data replication on a filesystem that doesn?t use filesets, I am expecting that setting data replication to two means that I will need to double the quota for each fileset (i.e. if a group has bought 5 TB of space I?ll need to set their fileset quota to 10 TB) but haven?t found where that is explicitly documented.  Is that correct?

Thanks again, all?

Kevin

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633




________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160504/34c11a8d/attachment-0002.htm>

From Robert.Oesterlin at nuance.com  Wed May  4 12:18:54 2016
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Wed, 4 May 2016 11:18:54 +0000
Subject: [gpfsug-discuss] Data replication and fileset quotas
Message-ID: <1C31AA0D-666C-4CB7-A54D-99CFC5AC8FAE@nuance.com>

From the documentation:

"When setting quota limits for a file system, replication within the file system should be considered. GPFS quota management takes replication into account when reporting on and determining if quota limits have been exceeded for both block and file usage. In a file system that has either type of replication set to a value of two, the values reported on by both the mmlsquota command and the mmrepquota command are double the value reported by the ls command."

Bob Oesterlin
Sr Storage Engineer, Nuance HPC Grid
507-269-0413


From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of "Simon Thompson (Research Computing - IT Services)" <S.J.Thompson at bham.ac.uk<mailto:S.J.Thompson at bham.ac.uk>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, May 4, 2016 at 2:11 AM
To: 'gpfsug main discussion list' <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Data replication and fileset quotas




Yes, this correct (I think there was some discussion on the mailing list a little while back).

The quota is the space used on disk, so if you run compression its the compressed size, tape hsm don't count until you recall them.

I assume mcstore is the same, I.e. Only counts on recall, but I haven't tested this.

Simon

--

Simon Thompson

Research Computing Team, IT Services

+44 121 415 8675

-----Original Message-----
From: Bryan Banister [bbanister at jumptrading.com<mailto:bbanister at jumptrading.com>]
Sent: Tuesday, May 03, 2016 09:33 PM GMT Standard Time
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Data replication and fileset quotas

Yes, I believe that is the case due to the fact that every file is replicated therefore taking twice as much space within the file system.
-Bryan

From:gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Buterbaugh, Kevin L
Sent: Tuesday, May 03, 2016 3:27 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: [gpfsug-discuss] Data replication and fileset quotas

Hi again all,

I have another question on a completely different topic and therefore I decided it was better to send two separate e-mails.

For a number of years now we have had a GPFS filesystem where we use filesets and set fileset quotas.  Data replication is set to one.  I understand how that all works.

We are creating another GPFS filesystem where we intend to also use filesets and fileset quotas, but set data replication to two.  Based on my experience with data replication on a filesystem that doesn?t use filesets, I am expecting that setting data replication to two means that I will need to double the quota for each fileset (i.e. if a group has bought 5 TB of space I?ll need to set their fileset quota to 10 TB) but haven?t found where that is explicitly documented.  Is that correct?

Thanks again, all?

Kevin

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633




________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160504/99bdbc87/attachment-0002.htm>

From daniel.kidger at uk.ibm.com  Wed May  4 12:36:06 2016
From: daniel.kidger at uk.ibm.com (Daniel Kidger)
Date: Wed, 4 May 2016 11:36:06 +0000
Subject: [gpfsug-discuss] Data replication and fileset quotas
In-Reply-To: <1C31AA0D-666C-4CB7-A54D-99CFC5AC8FAE@nuance.com>
References: <1C31AA0D-666C-4CB7-A54D-99CFC5AC8FAE@nuance.com>
Message-ID: <201605041136.u44BaBZT029213@d06av06.portsmouth.uk.ibm.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160504/597bfe75/attachment-0002.htm>

From D.R.Barker at exeter.ac.uk  Wed May  4 13:38:23 2016
From: D.R.Barker at exeter.ac.uk (Barker, David)
Date: Wed, 4 May 2016 12:38:23 +0000
Subject: [gpfsug-discuss] Data replication and fileset quotas
In-Reply-To: <201605041136.u44BaBZT029213@d06av06.portsmouth.uk.ibm.com>
References: <1C31AA0D-666C-4CB7-A54D-99CFC5AC8FAE@nuance.com>
	<201605041136.u44BaBZT029213@d06av06.portsmouth.uk.ibm.com>
Message-ID: <DB5PR03MB1366BB56598980F182E8B18CE67B0@DB5PR03MB1366.eurprd03.prod.outlook.com>

Hi Daniel,

It?s funny you mention compression & quotas ? it?s something I?ve hit on a.n.other system. The supplier ended up providing different ways of presenting quotas as a mount option. Options are:


-          Quotas are tracked using the post compression/dedupe actually used space. (i.e. users see the benefit/drawback of compression)

-          Quotas are tracked using the pre compression/dedupe filesize. (i.e. the sysadmin sees the benefit; users are unaware what?s happening underneath).

Would something similar be possible in GPFS land?

David

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Daniel Kidger
Sent: 04 May 2016 12:36
To: gpfsug-discuss at spectrumscale.org
Cc: gpfsug-discuss at spectrumscale.org
Subject: Re: [gpfsug-discuss] Data replication and fileset quotas

As Simon says, Quota measures used blocks in the filesystem.
Hence users can and should have behaviour that keeps within these limits.

GPFS Replication though is a system-administrator level concept - to protect data access in the case of power outages or though gross hardware failures. So as such should be transparent to the end users.

Unless users are enabled to choose 1 or 2 (or 3) way replication of their own files dependent on their importance (eg 1 copy for scratch files) then imho replication should not be measured in quota reporting.

On a related note, compression is great new feature, but it may confuse users if they delete some older but big 100GB files then try and recreate them only to find they can't because their quota is now exceeded (as compression is not at file creation but driven later by policies.

Thoughts?
Daniel






Dr Daniel Kidger
IBM Technical Sales Specialist
Software Defined Solution Sales

+44-07818 522 266
daniel.kidger at uk.ibm.com<mailto:daniel.kidger at uk.ibm.com>







----- Original message -----
From: "Oesterlin, Robert" <Robert.Oesterlin at nuance.com<mailto:Robert.Oesterlin at nuance.com>>
Sent by: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc:
Subject: Re: [gpfsug-discuss] Data replication and fileset quotas
Date: Wed, May 4, 2016 12:19 PM

From the documentation:

"When setting quota limits for a file system, replication within the file system should be considered. GPFS quota management takes replication into account when reporting on and determining if quota limits have been exceeded for both block and file usage. In a file system that has either type of replication set to a value of two, the values reported on by both the mmlsquota command and the mmrepquota command are double the value reported by the ls command."

Bob Oesterlin
Sr Storage Engineer, Nuance HPC Grid
507-269-0413


From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of "Simon Thompson (Research Computing - IT Services)" <S.J.Thompson at bham.ac.uk<mailto:S.J.Thompson at bham.ac.uk>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, May 4, 2016 at 2:11 AM
To: 'gpfsug main discussion list' <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Data replication and fileset quotas




Yes, this correct (I think there was some discussion on the mailing list a little while back).

The quota is the space used on disk, so if you run compression its the compressed size, tape hsm don't count until you recall them.

I assume mcstore is the same, I.e. Only counts on recall, but I haven't tested this.

Simon

--

Simon Thompson

Research Computing Team, IT Services

+44 121 415 8675

-----Original Message-----
From: Bryan Banister [bbanister at jumptrading.com<mailto:bbanister at jumptrading.com>]
Sent: Tuesday, May 03, 2016 09:33 PM GMT Standard Time
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Data replication and fileset quotas

Yes, I believe that is the case due to the fact that every file is replicated therefore taking twice as much space within the file system.

-Bryan



From:gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Buterbaugh, Kevin L
Sent: Tuesday, May 03, 2016 3:27 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: [gpfsug-discuss] Data replication and fileset quotas



Hi again all,



I have another question on a completely different topic and therefore I decided it was better to send two separate e-mails.



For a number of years now we have had a GPFS filesystem where we use filesets and set fileset quotas.  Data replication is set to one.  I understand how that all works.



We are creating another GPFS filesystem where we intend to also use filesets and fileset quotas, but set data replication to two.  Based on my experience with data replication on a filesystem that doesn?t use filesets, I am expecting that setting data replication to two means that I will need to double the quota for each fileset (i.e. if a group has bought 5 TB of space I?ll need to set their fileset quota to 10 TB) but haven?t found where that is explicitly documented.  Is that correct?



Thanks again, all?



Kevin



?

Kevin Buterbaugh - Senior System Administrator

Vanderbilt University - Advanced Computing Center for Research and Education

Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633







________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160504/46e3773a/attachment-0002.htm>

From r.sobey at imperial.ac.uk  Wed May  4 16:32:32 2016
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Wed, 4 May 2016 15:32:32 +0000
Subject: [gpfsug-discuss] TSM and re-linked filesets
Message-ID: <AMSPR06MB4053E885E19DC5F12F46F74DF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>

This is more a TSM question than GPFS but it involves GPFS to a point so I hope I'm ok asking this.

I've unlinked a fileset and relinked it at a different junction point. Will this cause TSM to see a "new" fileset and back up all its content? This is very undesireable so if I need to put it back where it belongs then I need to do this before tonight.

GPFS 3.5 and TSM 7.1.

Does anyone know?

Cheers

Richard

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160504/7399359a/attachment-0002.htm>

From r.sobey at imperial.ac.uk  Wed May  4 16:35:29 2016
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Wed, 4 May 2016 15:35:29 +0000
Subject: [gpfsug-discuss] TSM and re-linked filesets
In-Reply-To: <AMSPR06MB4053E885E19DC5F12F46F74DF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
References: <AMSPR06MB4053E885E19DC5F12F46F74DF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
Message-ID: <AMSPR06MB4053EB49583DA7B3623505EDF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>

A quick Google has given me the answer I dreaded:

"Attention: If you are using the TSM Backup-Archive client you must use caution when you unlink filesets that contain data backed up by TSM. TSM tracks files by pathname and does not track filesets. As a result, when you unlink a fileset, it appears to TSM that you deleted the contents of the fileset. Therefore, the TSM Backup-Archive client inactivates the data on the TSM server which may result in the loss of backup data during the expiration process."

So I need to relink it to the original place I think. Oh well. Not a disaster.

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A
Sent: 04 May 2016 16:33
To: 'gpfsug-discuss at spectrumscale.org' <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] TSM and re-linked filesets

This is more a TSM question than GPFS but it involves GPFS to a point so I hope I'm ok asking this.

I've unlinked a fileset and relinked it at a different junction point. Will this cause TSM to see a "new" fileset and back up all its content? This is very undesireable so if I need to put it back where it belongs then I need to do this before tonight.

GPFS 3.5 and TSM 7.1.

Does anyone know?

Cheers

Richard

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160504/f6c35dd3/attachment-0002.htm>

From S.J.Thompson at bham.ac.uk  Wed May  4 16:59:27 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Wed, 4 May 2016 15:59:27 +0000
Subject: [gpfsug-discuss] TSM and re-linked filesets
In-Reply-To: <AMSPR06MB4053EB49583DA7B3623505EDF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
References: <AMSPR06MB4053EB49583DA7B3623505EDF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F576F2A9@EX13.adf.bham.ac.uk>

Could you do something really nasty in the tsm db2 database to update the paths?

-----Original Message-----
From: Sobey, Richard A [r.sobey at imperial.ac.uk<mailto:r.sobey at imperial.ac.uk>]
Sent: Wednesday, May 04, 2016 04:35 PM GMT Standard Time
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] TSM and re-linked filesets

A quick Google has given me the answer I dreaded:

?Attention: If you are using the TSM Backup-Archive client you must use caution when you unlink filesets that contain data backed up by TSM. TSM tracks files by pathname and does not track filesets. As a result, when you unlink a fileset, it appears to TSM that you deleted the contents of the fileset. Therefore, the TSM Backup-Archive client inactivates the data on the TSM server which may result in the loss of backup data during the expiration process.?

So I need to relink it to the original place I think. Oh well. Not a disaster.

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A
Sent: 04 May 2016 16:33
To: 'gpfsug-discuss at spectrumscale.org' <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] TSM and re-linked filesets

This is more a TSM question than GPFS but it involves GPFS to a point so I hope I?m ok asking this.

I?ve unlinked a fileset and relinked it at a different junction point. Will this cause TSM to see a ?new? fileset and back up all its content? This is very undesireable so if I need to put it back where it belongs then I need to do this before tonight.

GPFS 3.5 and TSM 7.1.

Does anyone know?

Cheers

Richard

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160504/7874b433/attachment-0002.htm>

From makaplan at us.ibm.com  Wed May  4 17:14:44 2016
From: makaplan at us.ibm.com (Marc A Kaplan)
Date: Wed, 4 May 2016 12:14:44 -0400
Subject: [gpfsug-discuss] TSM and re-linked filesets
In-Reply-To: <AMSPR06MB4053EB49583DA7B3623505EDF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
References: <AMSPR06MB4053E885E19DC5F12F46F74DF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
	<AMSPR06MB4053EB49583DA7B3623505EDF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
Message-ID: <201605041614.u44GEnbI031747@d03av03.boulder.ibm.com>

I think you found your answer:  TSM tracks files by pathname. 

So... if a file had path /w/x/y/z on Monday.  But was moved to /w/x/q/p on 
Tuesday, how would TSM "know" it was the same file...?
It wouldn't! To TSM it seems you've deleted the first and created the 
second.

Technically there are some other possibilities, and some backup systems 
may use them, but NOT TSM:

1) Record the inode number and generation number and/or creation 
timestamp.   Within a given Posix-ish file system, that uniquely 
identifies the file.

2) Record a strong (cryptographic quality) checksum (hash) of the contents 
of the file.  If two files have the same checksum (hash) then the odds are
we can use the same backup data for both and don't have to make an extra 
copy in the backup system.    To make the odds really, really "long" you 
want to
take into account the "birthday paradox" and use lots and lots of bits. 
Long odds can also be compared to the probability of losing a file due to 
a bug
or an IO error or accident or disaster...

For example SHA-256, might be strong and long enough for you to believe 
in.
Backup is not generally a cryptographic game, so perhaps you should not 
worry much about some evil doer purposely trying to confound your backup 
system.
OTOH - if you have users who are adversaries, all backing up into the same 
system... In theory one might "destroy" another's backup.

This save transmission and storage of duplicates, but of course the backup 
system has to read the contents of each suspected new file and compute the 
hash...



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160504/b5aca332/attachment-0002.htm>

From laurence at qsplace.co.uk  Wed May  4 18:49:31 2016
From: laurence at qsplace.co.uk (Laurence Horrors-Barlow)
Date: Wed, 4 May 2016 18:49:31 +0100
Subject: [gpfsug-discuss] TSM and re-linked filesets
In-Reply-To: <AMSPR06MB4053EB49583DA7B3623505EDF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
References: <AMSPR06MB4053E885E19DC5F12F46F74DF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
	<AMSPR06MB4053EB49583DA7B3623505EDF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
Message-ID: <7DCCBF95-92CF-4473-8BAB-F845C5408A46@qsplace.co.uk>

Hi Richard,

That's correct, TSM will only look at the path and compare files it doesn't understand the file set. Once the path has changed it will then start expiring the missing files.

What are you trying to achieve here as you might be able to update the expiration or fudge the DB.

-- Lauz

Sent from my iPad

> On 4 May 2016, at 16:35, Sobey, Richard A <r.sobey at imperial.ac.uk> wrote:
> 
> A quick Google has given me the answer I dreaded:
>  
> ?Attention: If you are using the TSM Backup-Archive client you must use caution when you unlink filesets that contain data backed up by TSM. TSM tracks files by pathname and does not track filesets. As a result, when you unlink a fileset, it appears to TSM that you deleted the contents of the fileset. Therefore, the TSM Backup-Archive client inactivates the data on the TSM server which may result in the loss of backup data during the expiration process.?
>  
> So I need to relink it to the original place I think. Oh well. Not a disaster.
>  
> From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A
> Sent: 04 May 2016 16:33
> To: 'gpfsug-discuss at spectrumscale.org' <gpfsug-discuss at spectrumscale.org>
> Subject: [gpfsug-discuss] TSM and re-linked filesets
>  
> This is more a TSM question than GPFS but it involves GPFS to a point so I hope I?m ok asking this.
>  
> I?ve unlinked a fileset and relinked it at a different junction point. Will this cause TSM to see a ?new? fileset and back up all its content? This is very undesireable so if I need to put it back where it belongs then I need to do this before tonight.
>  
> GPFS 3.5 and TSM 7.1.
>  
> Does anyone know?
>  
> Cheers
>  
> Richard
>  
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160504/4e625d03/attachment-0002.htm>

From service at metamodul.com  Wed May  4 19:15:29 2016
From: service at metamodul.com (service at metamodul.com)
Date: Wed, 04 May 2016 20:15:29 +0200
Subject: [gpfsug-discuss] TSM and re-linked filesets
Message-ID: <tl3ri9d2ub9oacdlv1vloej2.1462385341788@email.android.com>

Have thought about the use of a submount ?
Meaning you link your fileset to the new directory and mount that dir on the old dir or you do not unlink at all but submount the old directory at the new directory.


Von Samsung Mobile gesendet

<div>-------- Urspr?ngliche Nachricht --------</div><div>Von: "Sobey, Richard A" <r.sobey at imperial.ac.uk> </div><div>Datum:2016.05.04  17:32  (GMT+01:00) </div><div>An: "'gpfsug-discuss at spectrumscale.org'" <gpfsug-discuss at spectrumscale.org> </div><div>Betreff: [gpfsug-discuss] TSM and re-linked filesets </div><div>
</div>This is more a TSM question than GPFS but it involves GPFS to a point so I hope I?m ok asking this.
 
I?ve unlinked a fileset and relinked it at a different junction point. Will this cause TSM to see a ?new? fileset and back up all its content? This is very undesireable so if I need to put it back where it belongs then I need to do this before tonight.
 
GPFS 3.5 and TSM 7.1.
 
Does anyone know?
 
Cheers
 
Richard
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160504/9a4c0a0d/attachment-0002.htm>

From r.sobey at imperial.ac.uk  Thu May  5 09:25:02 2016
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Thu, 5 May 2016 08:25:02 +0000
Subject: [gpfsug-discuss] TSM and re-linked filesets
In-Reply-To: <7DCCBF95-92CF-4473-8BAB-F845C5408A46@qsplace.co.uk>
References: <AMSPR06MB4053E885E19DC5F12F46F74DF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
	<AMSPR06MB4053EB49583DA7B3623505EDF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
	<7DCCBF95-92CF-4473-8BAB-F845C5408A46@qsplace.co.uk>
Message-ID: <AMSPR06MB405B89D8C47300501E4A447DF7C0@AMSPR06MB405.eurprd06.prod.outlook.com>

Hi Lauz

When we want to remove filesets we link them in an area that isn?t exported by CIFS or NFS. This allows us a grace period where the users most definitely cannot access the files before we delete them permanently.

Normally it?s not a problem as most of our filesets are only a few hundred GB, but this particular one contains 80TB of data (160TB replicated) and I don?t fancy it getting backed up again!

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Laurence Horrors-Barlow
Sent: 04 May 2016 18:50
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] TSM and re-linked filesets

Hi Richard,

That's correct, TSM will only look at the path and compare files it doesn't understand the file set. Once the path has changed it will then start expiring the missing files.

What are you trying to achieve here as you might be able to update the expiration or fudge the DB.

-- Lauz

Sent from my iPad

On 4 May 2016, at 16:35, Sobey, Richard A <r.sobey at imperial.ac.uk<mailto:r.sobey at imperial.ac.uk>> wrote:
A quick Google has given me the answer I dreaded:

?Attention: If you are using the TSM Backup-Archive client you must use caution when you unlink filesets that contain data backed up by TSM. TSM tracks files by pathname and does not track filesets. As a result, when you unlink a fileset, it appears to TSM that you deleted the contents of the fileset. Therefore, the TSM Backup-Archive client inactivates the data on the TSM server which may result in the loss of backup data during the expiration process.?

So I need to relink it to the original place I think. Oh well. Not a disaster.

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A
Sent: 04 May 2016 16:33
To: 'gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>' <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: [gpfsug-discuss] TSM and re-linked filesets

This is more a TSM question than GPFS but it involves GPFS to a point so I hope I?m ok asking this.

I?ve unlinked a fileset and relinked it at a different junction point. Will this cause TSM to see a ?new? fileset and back up all its content? This is very undesireable so if I need to put it back where it belongs then I need to do this before tonight.

GPFS 3.5 and TSM 7.1.

Does anyone know?

Cheers

Richard

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/f8406628/attachment-0002.htm>

From r.sobey at imperial.ac.uk  Thu May  5 09:25:49 2016
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Thu, 5 May 2016 08:25:49 +0000
Subject: [gpfsug-discuss] TSM and re-linked filesets
In-Reply-To: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F576F2A9@EX13.adf.bham.ac.uk>
References: <AMSPR06MB4053EB49583DA7B3623505EDF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
	<CF45EE16DEF2FE4B9AA7FF2B6EE26545F576F2A9@EX13.adf.bham.ac.uk>
Message-ID: <AMSPR06MB40548B61A9C4917DBA25F2BDF7C0@AMSPR06MB405.eurprd06.prod.outlook.com>

Err, I think you?ve already answered the question with the word ?nasty? heh.

Thanks for all the suggestions folks.

Richard

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (Research Computing - IT Services)
Sent: 04 May 2016 16:59
To: 'gpfsug main discussion list' <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] TSM and re-linked filesets

Could you do something really nasty in the tsm db2 database to update the paths?

-----Original Message-----
From: Sobey, Richard A [r.sobey at imperial.ac.uk<mailto:r.sobey at imperial.ac.uk>]
Sent: Wednesday, May 04, 2016 04:35 PM GMT Standard Time
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] TSM and re-linked filesets


A quick Google has given me the answer I dreaded:

?Attention: If you are using the TSM Backup-Archive client you must use caution when you unlink filesets that contain data backed up by TSM. TSM tracks files by pathname and does not track filesets. As a result, when you unlink a fileset, it appears to TSM that you deleted the contents of the fileset. Therefore, the TSM Backup-Archive client inactivates the data on the TSM server which may result in the loss of backup data during the expiration process.?

So I need to relink it to the original place I think. Oh well. Not a disaster.

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A
Sent: 04 May 2016 16:33
To: 'gpfsug-discuss at spectrumscale.org' <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: [gpfsug-discuss] TSM and re-linked filesets

This is more a TSM question than GPFS but it involves GPFS to a point so I hope I?m ok asking this.

I?ve unlinked a fileset and relinked it at a different junction point. Will this cause TSM to see a ?new? fileset and back up all its content? This is very undesireable so if I need to put it back where it belongs then I need to do this before tonight.

GPFS 3.5 and TSM 7.1.

Does anyone know?

Cheers

Richard

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/284f5a55/attachment-0002.htm>

From stijn.deweirdt at ugent.be  Thu May  5 09:33:45 2016
From: stijn.deweirdt at ugent.be (Stijn De Weirdt)
Date: Thu, 5 May 2016 10:33:45 +0200
Subject: [gpfsug-discuss] GPFS VM passthrough
Message-ID: <572B0569.1000201@ugent.be>

hi all,


we are examining the possibility to give VMs access to GPFS mounted on
the hypervisors. (use cases are readonly or read-mostly access to the
data on gpfs, the hypervisors have IB, the vms not yet; and we have no
idea how to handle possible explosion of gpfs client licenses if we
mount gpfs inside the VMs ;)

has anyone experience with using the 9p filesystem passthrough and gpfs
with qemu/kvm?

many thanks,

stijn


From laurence at qsplace.co.uk  Thu May  5 09:48:25 2016
From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow)
Date: Thu, 05 May 2016 09:48:25 +0100
Subject: [gpfsug-discuss] GPFS VM passthrough
In-Reply-To: <572B0569.1000201@ugent.be>
References: <572B0569.1000201@ugent.be>
Message-ID: <FF599E3E-FFEE-46B3-B2F1-BB1E0C882B6A@qsplace.co.uk>

I have a few KVM systems that use 9p to pass GPFS through and it does just work; although I've not really benchmarked or pushed it yet.

Regarding licencing, its my understanding that the guest inherits the hosts licence i.e. if you have a client license on the host all vms are covered by the same license (iirc there is a core/socket caveat) So that is worth exploring further.

-- Lauz

On 5 May 2016 09:33:45 BST, Stijn De Weirdt <stijn.deweirdt at ugent.be> wrote:
>hi all,
>
>
>we are examining the possibility to give VMs access to GPFS mounted on
>the hypervisors. (use cases are readonly or read-mostly access to the
>data on gpfs, the hypervisors have IB, the vms not yet; and we have no
>idea how to handle possible explosion of gpfs client licenses if we
>mount gpfs inside the VMs ;)
>
>has anyone experience with using the 9p filesystem passthrough and gpfs
>with qemu/kvm?
>
>many thanks,
>
>stijn
>_______________________________________________
>gpfsug-discuss mailing list
>gpfsug-discuss at spectrumscale.org
>http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/6f03a920/attachment-0004.htm>

From laurence at qsplace.co.uk  Thu May  5 09:48:25 2016
From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow)
Date: Thu, 05 May 2016 09:48:25 +0100
Subject: [gpfsug-discuss] GPFS VM passthrough
In-Reply-To: <572B0569.1000201@ugent.be>
References: <572B0569.1000201@ugent.be>
Message-ID: <FF599E3E-FFEE-46B3-B2F1-BB1E0C882B6A@qsplace.co.uk>

I have a few KVM systems that use 9p to pass GPFS through and it does just work; although I've not really benchmarked or pushed it yet.

Regarding licencing, its my understanding that the guest inherits the hosts licence i.e. if you have a client license on the host all vms are covered by the same license (iirc there is a core/socket caveat) So that is worth exploring further.

-- Lauz

On 5 May 2016 09:33:45 BST, Stijn De Weirdt <stijn.deweirdt at ugent.be> wrote:
>hi all,
>
>
>we are examining the possibility to give VMs access to GPFS mounted on
>the hypervisors. (use cases are readonly or read-mostly access to the
>data on gpfs, the hypervisors have IB, the vms not yet; and we have no
>idea how to handle possible explosion of gpfs client licenses if we
>mount gpfs inside the VMs ;)
>
>has anyone experience with using the 9p filesystem passthrough and gpfs
>with qemu/kvm?
>
>many thanks,
>
>stijn
>_______________________________________________
>gpfsug-discuss mailing list
>gpfsug-discuss at spectrumscale.org
>http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/6f03a920/attachment-0005.htm>

From daniel.kidger at uk.ibm.com  Thu May  5 09:52:22 2016
From: daniel.kidger at uk.ibm.com (Daniel Kidger)
Date: Thu, 5 May 2016 08:52:22 +0000
Subject: [gpfsug-discuss] GPFS VM passthrough
In-Reply-To: <572B0569.1000201@ugent.be>
Message-ID: <201605050852.u458qSQn011217@d06av04.portsmouth.uk.ibm.com>

Fortunately the licences issue is simple. 
The rules state that the sum of licences of all VMs on a node is capped at the number of sockets that server has.
So if you have 9 VMs using a mixture of different core counts each then you do not need more than 2 client licences if all hosted on a standard two socket server.

Daniel

Sent from IBM Verse


   Stijn De Weirdt --- [gpfsug-discuss] GPFS VM passthrough --- 
    From:"Stijn De Weirdt" <stijn.deweirdt at ugent.be>To:"gpfsug main discussion list" <gpfsug-discuss at gpfsug.org>Date:Thu, 5 May 2016 09:34Subject:[gpfsug-discuss] GPFS VM passthrough
  
    hi all,we are examining the possibility to give VMs access to GPFS mounted onthe hypervisors. (use cases are readonly or read-mostly access to thedata on gpfs, the hypervisors have IB, the vms not yet; and we have noidea how to handle possible explosion of gpfs client licenses if wemount gpfs inside the VMs ;)has anyone experience with using the 9p filesystem passthrough and gpfswith qemu/kvm?many thanks,stijn_______________________________________________gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discussUnless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/6fc5b68f/attachment-0002.htm>

From makaplan at us.ibm.com  Thu May  5 16:26:33 2016
From: makaplan at us.ibm.com (Marc A Kaplan)
Date: Thu, 5 May 2016 11:26:33 -0400
Subject: [gpfsug-discuss] TSM and re-linked filesets
In-Reply-To: <AMSPR06MB40548B61A9C4917DBA25F2BDF7C0@AMSPR06MB405.eurprd06.prod.outlook.com>
References: <AMSPR06MB4053EB49583DA7B3623505EDF7B0@AMSPR06MB405.eurprd06.prod.outlook.com>
	<CF45EE16DEF2FE4B9AA7FF2B6EE26545F576F2A9@EX13.adf.bham.ac.uk>
	<AMSPR06MB40548B61A9C4917DBA25F2BDF7C0@AMSPR06MB405.eurprd06.prod.outlook.com>
Message-ID: <201605051526.u45FQgxg012902@d03av02.boulder.ibm.com>

My little suggestion is to put it back where it was.  And then add a 
symlink from the desired (new) path to the TSM-needs-this actual path.



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/36637b8b/attachment-0002.htm>

From syi at ca.ibm.com  Thu May  5 22:09:49 2016
From: syi at ca.ibm.com (Yi Sun)
Date: Thu, 5 May 2016 17:09:49 -0400
Subject: [gpfsug-discuss] SPXXL meeting in NYC
Message-ID: <201605052109.u45L9vJ6015648@d03av01.boulder.ibm.com>





Hi Kristy and Bob,


For SPXXL meeting, if only interested in GPFS/Spectrum Scale day, is it
still nessary to make registration to SPXXL meeting?


Regards,


Yi Sun


syi at ca.ibm.com



>
>
> Message: 1
> Date: Mon, 4 Apr 2016 16:52:37 -0400
> From: GPFS UG USA Principal <usa-principal at gpfsug.org>
> To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
> Subject: [gpfsug-discuss] GPFS/Spectrum Scale Upcoming US Events -
> 		 		  Save the		 		  Dates
> Message-ID: <4192D644-E3AB-4B7B-AF7A-96D3D617FA7B at gpfsug.org>
> Content-Type: text/plain; charset="utf-8"
>
> Hello all,
>
> 		 		  We?d like to announce two upcoming US
GPFS/Spectrum Scale
> Events. One on the east coast, one in the midwest.
>
> 1) May 26th (full day event): GPFS/Spectrum Scale Day at the SPXXL
> conference in NYC https://www.spxxl.org/?q=New-York-City-2016 <
> https://www.spxxl.org/?q=New-York-City-2016> Developers and Engineers
from
> IBM will be at the meeting to cover topics, and open dialogue between IBM
> and customers will be encouraged. More details about the agenda, hotel
and
> other logistics will come later this month.
>
> Tentative Agenda:
> 		 		  ? 10 reasons for upgrading from GPFS 3.5 to
Spectrum Scale
> 4.2.1
> 		 		  ? Enhancements for CORAL from IBM
> 		 		  ? Panel discussion with customers, topic TBD
> 		 		  ? AFM and integration with Spectrum Protect
> 		 		  ? Best practices for GPFS or Spectrum Scale
Tuning.
> 		 		  ? At least one site update
>
> Location:
> New York Academy of Medicine
> 1216 Fifth Avenue
> New York, NY 10029
>
> ??
>
> 2) June 10th (full day event): GPFS/Spectrum Scale Users Group Meeting at
> Argonne National Lab (ANL). Thanks to Argonne for hosting this event.
> Developers and Engineers from IBM will be at the meeting to cover topics,
> and open dialogue between IBM and customers will be encouraged, as usual
no
> marketing pitches!
>
> Location: Argonne National Lab more details and final agenda will come
> later.
>
> Tentative Agenda:
>
> 9:00a-12:30p
> 9-9:30a - Opening Remarks
> 9:30-10a Deep Dive - Update on ESS
> 10a-11a Deep Dive - Problem Determination (Presentation 30 min/Panel 30
> min?)
> 11-11:30 Break
> 11:30a-Noon - Deep Dive - Protect & Scale integration
> Noon-12:30p HDFS/Hadoop
>
> 12:30 - 1:30p Lunch
>
> 1:30p-5:00p
> 1:30 - 2:00p IBM AFM Update
> 2:00-2:30p ANL: AFM as a burst buffer
> 2:30-3:00p ANL: GHI (GPFS HPSS Integration)
> 3:00-3:30p Break
> 3:30p - 4:00p LANL: ? or other site preso
> 4:00-4:30p Nuance: GPFS Performance Sensors Deployment Experiences
> 4:30p -5:00p Closing comments and Open Forum for Questions
>
> 5:00 - ?
> Beer hunting?
>
> ??
>
> We hope you can attend one or both of these events.
>
> Best,
> Kristy Kallback-Rose & Bob Oesterlin
> GPFS Users Group - USA Chapter - Principal & Co-principal
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
>
http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160404/2e191559/attachment-0001.html

> >
>
> ------------------------------
>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/d03f7737/attachment-0002.htm>

From kallbac at iu.edu  Thu May  5 22:31:43 2016
From: kallbac at iu.edu (Kallback-Rose, Kristy A)
Date: Thu, 5 May 2016 17:31:43 -0400
Subject: [gpfsug-discuss] SPXXL meeting in NYC
In-Reply-To: <201605052109.u45L9vJ6015648@d03av01.boulder.ibm.com>
References: <201605052109.u45L9vJ6015648@d03av01.boulder.ibm.com>
Message-ID: <106AD35B-EC40-45D6-A4C9-42F5C86600CB@iu.edu>

Yes, you should register so they know how many will attend and have seating/food. There is a $0 option for the GPFS day. Thanks for asking! If you have another question, let us know.

-Kristy

> On May 5, 2016, at 5:09 PM, Yi Sun <syi at ca.ibm.com> wrote:
> 
> Hi Kristy and Bob,
> 
> For SPXXL meeting, if only interested in GPFS/Spectrum Scale day, is it still nessary to make registration to SPXXL meeting?
> 
> Regards,
> 
> Yi Sun
> 
> syi at ca.ibm.com
> 
> 
> >
> >
> > Message: 1
> > Date: Mon, 4 Apr 2016 16:52:37 -0400
> > From: GPFS UG USA Principal <usa-principal at gpfsug.org>
> > To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
> > Subject: [gpfsug-discuss] GPFS/Spectrum Scale Upcoming US Events -
> >  Save the  Dates
> > Message-ID: <4192D644-E3AB-4B7B-AF7A-96D3D617FA7B at gpfsug.org>
> > Content-Type: text/plain; charset="utf-8"
> >
> > Hello all,
> >
> >  We?d like to announce two upcoming US GPFS/Spectrum Scale
> > Events. One on the east coast, one in the midwest.
> >
> > 1) May 26th (full day event): GPFS/Spectrum Scale Day at the SPXXL
> > conference in NYC https://www.spxxl.org/?q=New-York-City-2016 <https://www.spxxl.org/?q=New-York-City-2016> <
> > https://www.spxxl.org/?q=New-York-City-2016 <https://www.spxxl.org/?q=New-York-City-2016>> Developers and Engineers from
> > IBM will be at the meeting to cover topics, and open dialogue between IBM
> > and customers will be encouraged. More details about the agenda, hotel and
> > other logistics will come later this month.
> >
> > Tentative Agenda:
> >  ? 10 reasons for upgrading from GPFS 3.5 to Spectrum Scale
> > 4.2.1
> >  ? Enhancements for CORAL from IBM
> >  ? Panel discussion with customers, topic TBD
> >  ? AFM and integration with Spectrum Protect
> >  ? Best practices for GPFS or Spectrum Scale Tuning.
> >  ? At least one site update
> >
> > Location:
> > New York Academy of Medicine
> > 1216 Fifth Avenue
> > New York, NY 10029
> >
> > ??
> >
> > 2) June 10th (full day event): GPFS/Spectrum Scale Users Group Meeting at
> > Argonne National Lab (ANL). Thanks to Argonne for hosting this event.
> > Developers and Engineers from IBM will be at the meeting to cover topics,
> > and open dialogue between IBM and customers will be encouraged, as usual no
> > marketing pitches!
> >
> > Location: Argonne National Lab more details and final agenda will come
> > later.
> >
> > Tentative Agenda:
> >
> > 9:00a-12:30p
> > 9-9:30a - Opening Remarks
> > 9:30-10a Deep Dive - Update on ESS
> > 10a-11a Deep Dive - Problem Determination (Presentation 30 min/Panel 30
> > min?)
> > 11-11:30 Break
> > 11:30a-Noon - Deep Dive - Protect & Scale integration
> > Noon-12:30p HDFS/Hadoop
> >
> > 12:30 - 1:30p Lunch
> >
> > 1:30p-5:00p
> > 1:30 - 2:00p IBM AFM Update
> > 2:00-2:30p ANL: AFM as a burst buffer
> > 2:30-3:00p ANL: GHI (GPFS HPSS Integration)
> > 3:00-3:30p Break
> > 3:30p - 4:00p LANL: ? or other site preso
> > 4:00-4:30p Nuance: GPFS Performance Sensors Deployment Experiences
> > 4:30p -5:00p Closing comments and Open Forum for Questions
> >
> > 5:00 - ?
> > Beer hunting?
> >
> > ??
> >
> > We hope you can attend one or both of these events.
> >
> > Best,
> > Kristy Kallback-Rose & Bob Oesterlin
> > GPFS Users Group - USA Chapter - Principal & Co-principal
> > -------------- next part --------------
> > An HTML attachment was scrubbed...
> > URL: <
> > http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160404/2e191559/attachment-0001.html <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160404/2e191559/attachment-0001.html>
> > >
> >
> > ------------------------------
> >
> 
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/5ebe30d2/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/5ebe30d2/attachment-0002.sig>

From syi at ca.ibm.com  Thu May  5 22:36:05 2016
From: syi at ca.ibm.com (Yi Sun)
Date: Thu, 5 May 2016 17:36:05 -0400
Subject: [gpfsug-discuss] SPXXL meeting in NYC
Message-ID: <201605052136.u45LaEjv002729@d03av03.boulder.ibm.com>



Thank you Kristy.

=============================================================================================================
Date: Thu, 5 May 2016 17:31:43 -0400
From: "Kallback-Rose, Kristy A" <kallbac at iu.edu>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] SPXXL meeting in NYC
Message-ID: <106AD35B-EC40-45D6-A4C9-42F5C86600CB at iu.edu>
Content-Type: text/plain; charset="us-ascii"

Yes, you should register so they know how many will attend and have
seating/food. There is a $0 option for the GPFS day. Thanks for asking! If
you have another question, let us know.

-Kristy

> On May 5, 2016, at 5:09 PM, Yi Sun <syi at ca.ibm.com> wrote:
>
> Hi Kristy and Bob,
>
> For SPXXL meeting, if only interested in GPFS/Spectrum Scale day, is it
still nessary to make registration to SPXXL meeting?
>
> Regards,
>
> Yi Sun
>
> syi at ca.ibm.com




Regards,



Yi (Y.) Sun
Technical Account Manager
IBM SDI
IBM Systems


                                                                                      
                                                                                      
                                                                                      
 Phone: 1-905-316-2681                                                          IBM   
 E-mail: syi at ca.ibm.com                                                               
                                                           3600 Steeles Avenue East   
                                                                Markham, ON L3R 9Z7   
                                                                             Canada   
                                                                                      
                                                                                      
                                                                                      





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/2fb39b0e/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0C466617.gif
Type: image/gif
Size: 2022 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/2fb39b0e/attachment-0004.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ecblank.gif
Type: image/gif
Size: 45 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160505/2fb39b0e/attachment-0005.gif>

From TROPPENS at de.ibm.com  Tue May 10 13:00:16 2016
From: TROPPENS at de.ibm.com (Ulf Troppens)
Date: Tue, 10 May 2016 14:00:16 +0200
Subject: [gpfsug-discuss] GPFS/Spectrum Scale Upcoming US Events - Save
 the Dates - Agenda for New York May 26
In-Reply-To: <201605031206.u43C5wss002725@d06av09.portsmouth.uk.ibm.com>
References: <4192D644-E3AB-4B7B-AF7A-96D3D617FA7B@gpfsug.org><9489DBA2-1F12-4B05-A968-5D4855FBEA40@gpfsug.org>
	<201605031206.u43C5wss002725@d06av09.portsmouth.uk.ibm.com>
Message-ID: <201605101100.u4AB0Od3024998@d06av10.portsmouth.uk.ibm.com>

The agenda for the meeting in New York is now available here:
https://www.spxxl.org/?q=New-York-City-2016

For your convenience - here are the session titles:
- Top 10 reasons for upgrading from GPFS 3.5 to Spectrum Scale 4.2.1
- Panel Discussion: My favorite tool for monitoring/reporting
- Problem Determination
- User presentation : TBD
- Spectrum Scale in Life Sciences
- HPSS Update
- Best practices for GPFS or Spectrum Scale Tuning
- Enhancements for CORAL

Please use the link below for registration.

--
IBM Spectrum Scale Development - Client Engagements & Solutions Delivery
Consulting IT Specialist
Author "Storage Networks Explained"

IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Gesch?ftsf?hrung: Dirk Wittkopp
Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart,
HRB 243294




From:	Ulf Troppens/Germany/IBM at IBMDE
To:	gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:	03.05.2016 14:06
Subject:	Re: [gpfsug-discuss] GPFS/Spectrum Scale Upcoming US Events -
            Save the Dates - Registration for New York May 26
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



Registration for the first event in New York is up as well - the agenda
still needs some touch.

Please register here and select 'May 26 GPFS Day Registration', if you
don't attend SPXXL:
https://www.spxxl.org/?q=New-York-City-2016
https://www.eventbrite.com/e/spxxlscicomp-2016-summer-meeting-registration-24444020724



--
IBM Spectrum Scale Development - Client Engagements & Solutions Delivery
Consulting IT Specialist
Author "Storage Networks Explained"

IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Gesch?ftsf?hrung: Dirk Wittkopp
Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart,
HRB 243294


Inactive hide details for GPFS UG USA Principal ---28.04.2016
23:44:50---All, the registration page for the second event listedGPFS UG
USA Principal ---28.04.2016 23:44:50---All, the registration page for the
second event listed below at Argonne National Lab on June 10th is

From: GPFS UG USA Principal <usa-principal at gpfsug.org>
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Date: 28.04.2016 23:44
Subject: Re: [gpfsug-discuss] GPFS/Spectrum Scale Upcoming US Events - Save
the Dates
Sent by: gpfsug-discuss-bounces at spectrumscale.org



All, the registration page for the second event listed below at Argonne
National Lab on June 10th is now up. An updated agenda is also at this
site.

Please register here: https://www.regonline.com/Spectrumscalemeeting

We look forward to seeing some of you at these upcoming events.

Feel free to send suggestions for future events in your area.

Cheers,
-Kristy

            On Apr 4, 2016, at 4:52 PM, GPFS UG USA Principal <
            usa-principal at gpfsug.org> wrote:

            Hello all,

            We?d like to announce two upcoming US GPFS/Spectrum Scale
            Events. One on the east coast, one in the midwest.

            1) May 26th (full day event): GPFS/Spectrum Scale Day at the
            SPXXL conference in NYC
            https://www.spxxl.org/?q=New-York-City-2016 Developers and
            Engineers from IBM will be at the meeting to cover topics, and
            open dialogue between IBM and customers will be encouraged.
            More details about the agenda, hotel and other logistics will
            come later this month.

            Tentative Agenda:
            ? 10 reasons for upgrading from GPFS 3.5 to Spectrum Scale
            4.2.1
            ? Enhancements for CORAL from IBM
            ? Panel discussion with customers, topic TBD
            ? AFM and integration with Spectrum Protect
            ? Best practices for GPFS or Spectrum Scale Tuning.
            ? At least one site update

            Location:
            New York Academy of Medicine
            1216 Fifth Avenue
            New York, NY 10029

            ??

            2) June 10th (full day event): GPFS/Spectrum Scale Users Group
            Meeting at Argonne National Lab (ANL). Thanks to Argonne for
            hosting this event. Developers and Engineers from IBM will be
            at the meeting to cover topics, and open dialogue between IBM
            and customers will be encouraged, as usual no marketing
            pitches!

            Location: Argonne National Lab more details and final agenda
            will come later.

            Tentative Agenda:

            9:00a-12:30p
            9-9:30a - Opening Remarks
            9:30-10a Deep Dive - Update on ESS
            10a-11a Deep Dive - Problem Determination (Presentation 30
            min/Panel 30 min?)
            11-11:30 Break
            11:30a-Noon - Deep Dive - Protect & Scale integration
            Noon-12:30p HDFS/Hadoop

            12:30 - 1:30p Lunch

            1:30p-5:00p
            1:30 - 2:00p IBM AFM Update
            2:00-2:30p ANL: AFM as a burst buffer
            2:30-3:00p ANL: GHI (GPFS HPSS Integration)
            3:00-3:30p Break
            3:30p - 4:00p LANL: ? or other site preso
            4:00-4:30p Nuance: GPFS Performance Sensors Deployment
            Experiences
            4:30p -5:00p Closing comments and Open Forum for Questions

            5:00 - ?
            Beer hunting?

            ??

            We hope you can attend one or both of these events.

            Best,
            Kristy Kallback-Rose & Bob Oesterlin
            GPFS Users Group - USA Chapter - Principal & Co-principal


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160510/2d3525b5/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160510/2d3525b5/attachment-0002.gif>

From Mark.Bush at siriuscom.com  Wed May 11 03:02:30 2016
From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com)
Date: Wed, 11 May 2016 02:02:30 +0000
Subject: [gpfsug-discuss] Demo
Message-ID: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>

I?m tasked with coming up with a demo of Spectrum Scale for our customers.  Something that shows a little bit of most of the features of Scale.  Does anyone have some ideas of ?must see? features that might make for a good demo?

My thoughts are something like:

Overview of architecture
NSD Servers, Protocol Servers, Quorum, etc
Show adding a Node to the mix as a client and a server
Show creating a filesystem (and a fileset)
Show copying some files from a client (linux client with GPFS or through a filesystem thru protocol nodes)
Show GUI
Show Snapshot

Any other thoughts?


Mark R. Bush | Solutions Architect
Mobile: 210.237.8415 | mark.bush at siriuscom.com
Sirius Computer Solutions | www.siriuscom.com<http://www.siriuscom.com>
10100 Reunion Place, Suite 500, San Antonio, TX 78216


This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you.

Sirius Computer Solutions<http://www.siriuscom.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160511/b9bb4704/attachment-0002.htm>

From pinto at scinet.utoronto.ca  Wed May 11 03:35:19 2016
From: pinto at scinet.utoronto.ca (Jaime Pinto)
Date: Tue, 10 May 2016 22:35:19 -0400
Subject: [gpfsug-discuss] Demo
In-Reply-To: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
References: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
Message-ID: <20160510223519.11041xrqkfoeqbfr@support.scinet.utoronto.ca>

Make a clear distinction between the hardware and the software layers  
to start with.
Then between server infrastructure and client infrastructure (and  
licensing requirements)

NSD Servers, Protocol Servers, disks, LUNs, HW raid if any,  
configuration of the storage building blocks, NSD fail-over, etc

hardware raid vs. software raid vs, appliances

File system managers, cluster manager, clients addition/setup, FSmgr  
fail-over, Quorum, etc
File system creation, clustered vs. scattered, FS attributes,  
blocksize, capacity expansion/shrinkage, etc.

Fileset creation, junction linking, mounting, exportfs, protocol nodes.

storage cluster vs. compute cluster
snapshot, quota, (backup?), .....

For me there are areas/subjects that belong together, and a logic in  
the unveiling of features.

I would be interested in the end result

Thanks
Jaime


Quoting "Mark.Bush at siriuscom.com" <Mark.Bush at siriuscom.com>:

> I?m tasked with coming up with a demo of Spectrum Scale for our   
> customers.  Something that shows a little bit of most of the   
> features of Scale.  Does anyone have some ideas of ?must see?   
> features that might make for a good demo?
>
> My thoughts are something like:
>
> Overview of architecture
> NSD Servers, Protocol Servers, Quorum, etc
> Show adding a Node to the mix as a client and a server
> Show creating a filesystem (and a fileset)
> Show copying some files from a client (linux client with GPFS or   
> through a filesystem thru protocol nodes)
> Show GUI
> Show Snapshot
>
> Any other thoughts?
>
>
> Mark R. Bush | Solutions Architect
> Mobile: 210.237.8415 | mark.bush at siriuscom.com
> Sirius Computer Solutions | www.siriuscom.com<http://www.siriuscom.com>
> 10100 Reunion Place, Suite 500, San Antonio, TX 78216
>
>
> This message (including any attachments) is intended only for the   
> use of the individual or entity to which it is addressed and may   
> contain information that is non-public, proprietary, privileged,   
> confidential, and exempt from disclosure under applicable law. If   
> you are not the intended recipient, you are hereby notified that any  
>  use, dissemination, distribution, or copying of this communication   
> is strictly prohibited. This message may be viewed by parties at   
> Sirius Computer Solutions other than those named in the message   
> header. This message does not contain an official representation of   
> Sirius Computer Solutions. If you have received this communication   
> in error, notify Sirius Computer Solutions immediately and (i)   
> destroy this message if a facsimile or (ii) delete this message   
> immediately if this is an electronic communication. Thank you.
>
> Sirius Computer Solutions<http://www.siriuscom.com>
>




---
Jaime Pinto
SciNet HPC Consortium  - Compute/Calcul Canada
www.scinet.utoronto.ca - www.computecanada.org
University of Toronto
256 McCaul Street, Room 235
Toronto, ON, M5T1W5
P: 416-978-2755
C: 416-505-1477

----------------------------------------------------------------
This message was sent using IMP at SciNet Consortium, University of Toronto.



From sfadden at us.ibm.com  Wed May 11 06:51:38 2016
From: sfadden at us.ibm.com (Scott Fadden)
Date: Tue, 10 May 2016 22:51:38 -0700
Subject: [gpfsug-discuss] Demo
In-Reply-To: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
References: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
Message-ID: <201605110551.u4B5phC8017323@d03av02.boulder.ibm.com>

I developed a demo that shows how the file heat metric can be used to move 
"cool" data from a Flash tier to spinning disk. As visuals it displays a 
speedometer for transaction rate, a chart showing capacity used in each 
tier and total cost of storing the data ($/GiB). 

For a simpler demo maybe the GUI with live performance, maybe a policy on 
lots of files (use a list policy) or something like fileset level 
snapshots.


Scott Fadden
Spectrum Scale - Technical Marketing 
Phone: (503) 880-5833 
sfadden at us.ibm.com
http://www.ibm.com/systems/storage/spectrum/scale



From:   "Mark.Bush at siriuscom.com" <Mark.Bush at siriuscom.com>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   05/10/2016 07:02 PM
Subject:        [gpfsug-discuss] Demo
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



I?m tasked with coming up with a demo of Spectrum Scale for our customers. 
 Something that shows a little bit of most of the features of Scale.  Does 
anyone have some ideas of ?must see? features that might make for a good 
demo?

My thoughts are something like: 

Overview of architecture
NSD Servers, Protocol Servers, Quorum, etc
Show adding a Node to the mix as a client and a server
Show creating a filesystem (and a fileset)
Show copying some files from a client (linux client with GPFS or through a 
filesystem thru protocol nodes)
Show GUI
Show Snapshot

Any other thoughts?


Mark R. Bush | Solutions Architect
Mobile: 210.237.8415 | mark.bush at siriuscom.com
Sirius Computer Solutions | www.siriuscom.com
10100 Reunion Place, Suite 500, San Antonio, TX 78216 

This message (including any attachments) is intended only for the use of 
the individual or entity to which it is addressed and may contain 
information that is non-public, proprietary, privileged, confidential, and 
exempt from disclosure under applicable law. If you are not the intended 
recipient, you are hereby notified that any use, dissemination, 
distribution, or copying of this communication is strictly prohibited. 
This message may be viewed by parties at Sirius Computer Solutions other 
than those named in the message header. This message does not contain an 
official representation of Sirius Computer Solutions. If you have received 
this communication in error, notify Sirius Computer Solutions immediately 
and (i) destroy this message if a facsimile or (ii) delete this message 
immediately if this is an electronic communication. Thank you. 
Sirius Computer Solutions _______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160510/af957fe6/attachment-0002.htm>

From Mikal.Sande at altibox.no  Wed May 11 08:10:02 2016
From: Mikal.Sande at altibox.no (Sande, Mikal)
Date: Wed, 11 May 2016 07:10:02 +0000
Subject: [gpfsug-discuss] Demo
In-Reply-To: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
References: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
Message-ID: <84C0B9AB4719954985704BC46DF8602A9F7E480B@prdmbx01.lyse.no>

Apart from the usefulness of parallel access to the same LUNs, one feature I value for metadata heavy workloads is being able to split data and metadata. When we moved metadata from spindles to flash LUNs on one of our clusters it was a day vs. night kind of change in behavior in the cluster.

Just my two cents, I know exactly how to demo the split between data and metadata. Maybe when creating the filesystem? Or maybe afterwards and show how restripe is used to perform such a change?


Best regards,
Mikal Sande
________________________________
From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Mark.Bush at siriuscom.com [Mark.Bush at siriuscom.com]
Sent: Wednesday, May 11, 2016 4:02 AM
To: gpfsug main discussion list
Subject: [gpfsug-discuss] Demo

I?m tasked with coming up with a demo of Spectrum Scale for our customers.  Something that shows a little bit of most of the features of Scale.  Does anyone have some ideas of ?must see? features that might make for a good demo?

My thoughts are something like:

Overview of architecture
NSD Servers, Protocol Servers, Quorum, etc
Show adding a Node to the mix as a client and a server
Show creating a filesystem (and a fileset)
Show copying some files from a client (linux client with GPFS or through a filesystem thru protocol nodes)
Show GUI
Show Snapshot

Any other thoughts?


Mark R. Bush | Solutions Architect
Mobile: 210.237.8415 | mark.bush at siriuscom.com
Sirius Computer Solutions | www.siriuscom.com<http://www.siriuscom.com>
10100 Reunion Place, Suite 500, San Antonio, TX 78216


This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you.

Sirius Computer Solutions<http://www.siriuscom.com>


From D.R.Barker at exeter.ac.uk  Wed May 11 10:53:15 2016
From: D.R.Barker at exeter.ac.uk (Barker, David)
Date: Wed, 11 May 2016 09:53:15 +0000
Subject: [gpfsug-discuss] UK May Meeting
Message-ID: <A4DBAD9F-EE30-4A0E-AD59-51BFCB18F4DB@exeter.ac.uk>

Hi Simon - sorry to have to cancel at the last minute, but I won?t be able to attend now. Feel free to give my space to someone else. :-/

Best regards,
Dave

-- 
David Barker
01392 723900
www.exeter.ac.uk
Laver Building, North Park Road, EXETER, EX4 4QE

 
 









On 11/04/2016, 10:37, "gpfsug-discuss-bounces at spectrumscale.org on behalf of GPFS UG Chair (Simon Thompson)" <gpfsug-discuss-bounces at spectrumscale.org on behalf of chair at spectrumscale.org> wrote:

>
>Hi All,
>
>We are down to our last few places for the May user group meeting, if you
>are planning to come along, please do register:
>
>The draft agenda and registration for the day is at:
>http://www.eventbrite.com/e/spectrum-scale-gpfs-uk-user-group-spring-2016-t
>ickets-21724951916
>
>If you have registered and aren't able to attend now, please do let us
>know so that we can free the slot for other members of the group.
>
>We also have 1 slot left on the agenda for a user talk, so if you have an
>interesting deployment or plans and are able to speak, please let me know!
>
>Thanks
>
>Simon
>
>
>_______________________________________________
>gpfsug-discuss mailing list
>gpfsug-discuss at spectrumscale.org
>http://gpfsug.org/mailman/listinfo/gpfsug-discuss

From Robert.Oesterlin at nuance.com  Wed May 11 17:05:12 2016
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Wed, 11 May 2016 16:05:12 +0000
Subject: [gpfsug-discuss] Reminder: Spectrum Scale/GPFS UG meeting,
	June 10th at Argonne Lab
Message-ID: <AB6F0D83-4CBA-4651-BE78-384276359819@nuance.com>

A Reminder that there will be a user group meeting on June 10th at Argonne Labs, near Chicago, IL. A great agenda and a great opportunity to network with other users.

Details: http://spectrumscale.org/spectrum-scale-user-group-usa-meeting-june-10th/

Registration link:https://www.regonline.com/Spectrumscalemeeting

Registration deadline is May 27th!

Bob Oesterlin
GPFS UG Co-Principal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160511/e5e5c3f7/attachment-0002.htm>

From janfrode at tanso.net  Wed May 11 18:04:37 2016
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Wed, 11 May 2016 17:04:37 +0000
Subject: [gpfsug-discuss] Demo
In-Reply-To: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
References: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
Message-ID: <CAHwPathGPijoRstXc6GzBNJxfWiOF8kuNJFbJUui1xH-fJzFdA@mail.gmail.com>

Some suggestions for stuff I find demo-interesting..

Show snapshots accessible as previous version trough SMB/Windows Explorer.
Show offline files with X indicator in Explorer, if you can do HSM (maybe
trough via the cloud gateway beta).

Show object storage trough cyberduck, or similar clients.

Failover nfs/smb.

Mixed windows/linux cluster accessing same filesystem.

mmbackup doing backups of million of files in no time.



-jf

ons. 11. mai 2016 kl. 04.02 skrev Mark.Bush at siriuscom.com <
Mark.Bush at siriuscom.com>:

> I?m tasked with coming up with a demo of Spectrum Scale for our
> customers.  Something that shows a little bit of most of the features of
> Scale.  Does anyone have some ideas of ?must see? features that might make
> for a good demo?
>
> My thoughts are something like:
>
> Overview of architecture
> NSD Servers, Protocol Servers, Quorum, etc
> Show adding a Node to the mix as a client and a server
> Show creating a filesystem (and a fileset)
> Show copying some files from a client (linux client with GPFS or through a
> filesystem thru protocol nodes)
> Show GUI
> Show Snapshot
>
> Any other thoughts?
>
>
> Mark R. Bush | Solutions Architect
> Mobile: 210.237.8415 | mark.bush at siriuscom.com
> Sirius Computer Solutions | www.siriuscom.com
> 10100 Reunion Place, Suite 500, San Antonio, TX 78216
>
> This message (including any attachments) is intended only for the use of
> the individual or entity to which it is addressed and may contain
> information that is non-public, proprietary, privileged, confidential, and
> exempt from disclosure under applicable law. If you are not the intended
> recipient, you are hereby notified that any use, dissemination,
> distribution, or copying of this communication is strictly prohibited. This
> message may be viewed by parties at Sirius Computer Solutions other than
> those named in the message header. This message does not contain an
> official representation of Sirius Computer Solutions. If you have received
> this communication in error, notify Sirius Computer Solutions immediately
> and (i) destroy this message if a facsimile or (ii) delete this message
> immediately if this is an electronic communication. Thank you.
> Sirius Computer Solutions <http://www.siriuscom.com>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160511/5f4d6e64/attachment-0002.htm>

From YARD at il.ibm.com  Wed May 11 20:49:08 2016
From: YARD at il.ibm.com (Yaron Daniel)
Date: Wed, 11 May 2016 22:49:08 +0300
Subject: [gpfsug-discuss] Demo
In-Reply-To: <201605110551.u4B5phC8017323@d03av02.boulder.ibm.com>
References: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
	<201605110551.u4B5phC8017323@d03av02.boulder.ibm.com>
Message-ID: <201605111949.u4BJnF8L015520@d06av02.portsmouth.uk.ibm.com>

Hi

I think what is hot:

- Compression
- Mixed access to same files via CIFS/NFS/GPFS client
- AFM DR capabilities

 
Regards
 


 
 
Yaron Daniel
 94 Em Ha'Moshavot Rd

Server, Storage and Data Services - Team Leader  
 Petach Tiqva, 49527
Global Technology Services
 Israel
Phone:
+972-3-916-5672
 
 
Fax:
+972-3-916-5672
 
 
Mobile:
+972-52-8395593
 
 
e-mail:
yard at il.ibm.com
 
 
IBM Israel
 
 
 
 

 



From:   "Scott Fadden" <sfadden at us.ibm.com>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   05/11/2016 08:51 AM
Subject:        Re: [gpfsug-discuss] Demo
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



I developed a demo that shows how the file heat metric can be used to move 
"cool" data from a Flash tier to spinning disk. As visuals it displays a 
speedometer for transaction rate, a chart showing capacity used in each 
tier and total cost of storing the data ($/GiB). 

For a simpler demo maybe the GUI with live performance, maybe a policy on 
lots of files (use a list policy) or something like fileset level 
snapshots.


Scott Fadden
Spectrum Scale - Technical Marketing 
Phone: (503) 880-5833 
sfadden at us.ibm.com
http://www.ibm.com/systems/storage/spectrum/scale



From:        "Mark.Bush at siriuscom.com" <Mark.Bush at siriuscom.com>
To:        gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:        05/10/2016 07:02 PM
Subject:        [gpfsug-discuss] Demo
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



I?m tasked with coming up with a demo of Spectrum Scale for our customers. 
 Something that shows a little bit of most of the features of Scale.  Does 
anyone have some ideas of ?must see? features that might make for a good 
demo?

My thoughts are something like: 

Overview of architecture
NSD Servers, Protocol Servers, Quorum, etc
Show adding a Node to the mix as a client and a server
Show creating a filesystem (and a fileset)
Show copying some files from a client (linux client with GPFS or through a 
filesystem thru protocol nodes)
Show GUI
Show Snapshot

Any other thoughts?


Mark R. Bush| Solutions Architect
Mobile: 210.237.8415 | mark.bush at siriuscom.com
Sirius Computer Solutions | www.siriuscom.com
10100 Reunion Place, Suite 500, San Antonio, TX 78216 
This message (including any attachments) is intended only for the use of 
the individual or entity to which it is addressed and may contain 
information that is non-public, proprietary, privileged, confidential, and 
exempt from disclosure under applicable law. If you are not the intended 
recipient, you are hereby notified that any use, dissemination, 
distribution, or copying of this communication is strictly prohibited. 
This message may be viewed by parties at Sirius Computer Solutions other 
than those named in the message header. This message does not contain an 
official representation of Sirius Computer Solutions. If you have received 
this communication in error, notify Sirius Computer Solutions immediately 
and (i) destroy this message if a facsimile or (ii) delete this message 
immediately if this is an electronic communication. Thank you. 
Sirius Computer Solutions _______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160511/8d1a26c5/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 1851 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160511/8d1a26c5/attachment-0002.gif>

From pgalgali at us.ibm.com  Wed May 11 21:53:25 2016
From: pgalgali at us.ibm.com (Pallavi V Galgali)
Date: Wed, 11 May 2016 15:53:25 -0500
Subject: [gpfsug-discuss] Demo
In-Reply-To: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
References: <E4F5ECAD-2F3B-4F62-ACCF-DC020013A5E1@siriuscom.com>
Message-ID: <201605112053.u4BKrWeI005237@d01av04.pok.ibm.com>

Pls explore Spectrum Scale trial VM and its attached documentation -
http://www-03.ibm.com/systems/storage/spectrum/scale/trial.html
This comes with documentation ( Explore guide and Advanced user guide) that
has some interesting scenarios that can be used for demo purpose.

                                                                                     
   Thanks & Regards,                                                                 
                                                                                     
   Pallavi Galgali                                                                   
   Manager, Client Adoption and Solutions Delivery                                   
   IBM Spectrum Scale Development                                                    
   Cell: +1 914 433 9882                                                             
   11501 Burnet Road Austin TX 78758                                                 
                                                                                     








From:	"Mark.Bush at siriuscom.com" <Mark.Bush at siriuscom.com>
To:	gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:	05/10/2016 09:02 PM
Subject:	[gpfsug-discuss] Demo
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



I?m tasked with coming up with a demo of Spectrum Scale for our customers.
Something that shows a little bit of most of the features of Scale.  Does
anyone have some ideas of ?must see? features that might make for a good
demo?

My thoughts are something like:

Overview of architecture
NSD Servers, Protocol Servers, Quorum, etc
Show adding a Node to the mix as a client and a server
Show creating a filesystem (and a fileset)
Show copying some files from a client (linux client with GPFS or through a
filesystem thru protocol nodes)
Show GUI
Show Snapshot

Any other thoughts?


Mark R. Bush | Solutions Architect
Mobile: 210.237.8415 | mark.bush at siriuscom.com
Sirius Computer Solutions | www.siriuscom.com
10100 Reunion Place, Suite 500, San Antonio, TX 78216



This message (including any attachments) is intended only for the use of
the individual or entity to which it is addressed and may contain
information that is non-public, proprietary, privileged, confidential, and
exempt from disclosure under applicable law. If you are not the intended
recipient, you are hereby notified that any use, dissemination,
distribution, or copying of this communication is strictly prohibited. This
message may be viewed by parties at Sirius Computer Solutions other than
those named in the message header. This message does not contain an
official representation of Sirius Computer Solutions. If you have received
this communication in error, notify Sirius Computer Solutions immediately
and (i) destroy this message if a facsimile or (ii) delete this message
immediately if this is an electronic communication. Thank you.


Sirius Computer Solutions _______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160511/694b75a9/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160511/694b75a9/attachment-0002.gif>

From douglasof at us.ibm.com  Fri May 13 16:01:55 2016
From: douglasof at us.ibm.com (Douglas O'flaherty)
Date: Fri, 13 May 2016 11:01:55 -0400
Subject: [gpfsug-discuss] ISC U/G Agenda
Message-ID: <201605131502.u4DF23Bi020162@d03av05.boulder.ibm.com>

Greetings:

IBM is happy to announce the agenda for the joint IBM Spectrum Scale and 
Platform Computing User Group Agenda at ISC. We will finish on time to 
attend the opening reception. 

As with other user group meetings, the agenda includes user stories, 
updates on IBM Spectrum Scale & Platform LSF, and access to IBM experts 
and your peers. 

Please join us! 

To attend, please email Fabian.Beckmann at de.ibm.com so we can have an 
accurate count of attendees. 


Monday June 20, 2016  - 14:30-18:00 - Conference Room Konstant

14:30-14:40 [10 min] Welcome (Douglas o'Flaherty, IBM)
14:40-15:00 [20 min] Ten Reasons to Upgrade from GPFS 3.4 to Spectrum 
Scale 4.2 (Olaf Weiser, IBM)
15:00-15:30 [30 min] Shared Storage with in-memory latency: EMC DSSD D5 
and IBM Spectrum Scale (Stefan Radtke, EMC)
15:30-16:00 [30 min] Workload scheduling and data management in a private 
cloud (Uwe Sommer, Airbus)
16:00-16:30 [30 min] Spectrum Scale site report (To be confirmed by 
customer)
16:30-17:00 [30 min] What's new in Platform LSF 10.1 & storage integration 
(Bill McMillan, IBM)
17:00-17:30 [30 min] What's new in Spectrum Scale 4.2.1 (Mathias Dietz, 
IBM)
17:30-18:00 [30 min] CORAL enhancements for Spectrum Scale (Sven Oehme, 
IBM)

Looking forward to seeing you there!


doug

PS: IBMers can register their clients at this IBM link: 
https://w3-9033.ibm.com/events/ast/schedule/16isc.nsf 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160513/bba0eeae/attachment-0002.htm>

From billowen at us.ibm.com  Sat May 14 13:05:14 2016
From: billowen at us.ibm.com (Bill Owen)
Date: Sat, 14 May 2016 05:05:14 -0700
Subject: [gpfsug-discuss] OpenStack & Spectrum Scale Usage Survey
Message-ID: <201605141205.u4EC5KBm000338@d03av04.boulder.ibm.com>


We in Spectrum Scale development are very interested in learning about your
current & planned usage of OpenStack with Spectrum Scale, as well as plans
for deploying containers in your Spectrum Scale environment.

We would like to use this information to help shape our road map in this
area over the next 18-24 months.

Please take 10 minutes to answer the questions in this short survey:
https://www.surveygizmo.com/s3/2774614/IBMSpectrumScale-OpenStackUsageSurvey

Thank you,
Bill Owen
billowen at us.ibm.com
Spectrum Scale Object Storage
520-799-4829
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160514/be7206f9/attachment-0002.htm>

From S.J.Thompson at bham.ac.uk  Mon May 16 09:44:05 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Mon, 16 May 2016 08:44:05 +0000
Subject: [gpfsug-discuss] Flash for metadata
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F577FBC6@EX13.adf.bham.ac.uk>

Hi all

I'm currently about to deploy some flash systems (FS-900) to take our metadata for one of our Spectrum Scale file-systems. We use 4k inodes for this file-system.

On the FS-900, when creating a volume, we get a choice of 512b or 4k sectors. Does anyone have a suggestion on this? On a spinning disk RAID array, I get what a strip size is and we'd use multiples of the stripe where possible for the block size.

Is the sector size on the FS-900 the smallest chunk that can be written out? And so would it make sense to use 4k sectors for 4k GPFS inodes right?

Related. In the deployment guide for the FS-900 arrays, if one is using a file-system (e.g. XFS) on the array the guide talks about aligning the partitions with with array, when GPFS is used on a storage array, does it automatically work out the sector alignment?


Thanks

Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160516/2909e86d/attachment-0002.htm>

From Paul.Sanchez at deshaw.com  Mon May 16 10:28:35 2016
From: Paul.Sanchez at deshaw.com (Sanchez, Paul)
Date: Mon, 16 May 2016 09:28:35 +0000
Subject: [gpfsug-discuss] Flash for metadata
In-Reply-To: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F577FBC6@EX13.adf.bham.ac.uk>
References: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F577FBC6@EX13.adf.bham.ac.uk>
Message-ID: <49a3ab3b493a49e59177c6c25dd34d23@mbxtoa1.winmail.deshaw.com>

Simon,

I can't speak specifically to the FS-900, since I generally use Dell Compellent and NetApp EF arrays for flash today.  But I also take care to ensure that the system pool metadata NSDs use RAID10 (or equivalent) rather than RAID5/6.  For small 4K metadata writes, on a 4K blocksize LUN this could result in optimal writes.  For larger native blocksize LUNs, it eliminates the parity calculation for read-modify-write operations.

Thanks,
Paul

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (Research Computing - IT Services)
Sent: Monday, May 16, 2016 9:44 AM
To: gpfsug-discuss at spectrumscale.org
Subject: [gpfsug-discuss] Flash for metadata

Hi all

I'm currently about to deploy some flash systems (FS-900) to take our metadata for one of our Spectrum Scale file-systems. We use 4k inodes for this file-system.

On the FS-900, when creating a volume, we get a choice of 512b or 4k sectors. Does anyone have a suggestion on this? On a spinning disk RAID array, I get what a strip size is and we'd use multiples of the stripe where possible for the block size.

Is the sector size on the FS-900 the smallest chunk that can be written out? And so would it make sense to use 4k sectors for 4k GPFS inodes right?

Related. In the deployment guide for the FS-900 arrays, if one is using a file-system (e.g. XFS) on the array the guide talks about aligning the partitions with with array, when GPFS is used on a storage array, does it automatically work out the sector alignment?


Thanks

Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160516/f4964333/attachment-0002.htm>

From S.J.Thompson at bham.ac.uk  Mon May 16 10:48:12 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Mon, 16 May 2016 09:48:12 +0000
Subject: [gpfsug-discuss] Flash for metadata
In-Reply-To: <49a3ab3b493a49e59177c6c25dd34d23@mbxtoa1.winmail.deshaw.com>
References: <49a3ab3b493a49e59177c6c25dd34d23@mbxtoa1.winmail.deshaw.com>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F577FC47@EX13.adf.bham.ac.uk>


Hi Paul,

Thanks, interestingly the FS-900 only does raid5. I get no choice about that, I do get a sector size choice when I create a lun on it...

I was also wondering how gpfs would do a metadata update. Would it be smart enough to know the sector size was 512b and only modify that, or would it always do a read modify write on the whole 4k inode?

Simon

-----Original Message-----
From: Sanchez, Paul [Paul.Sanchez at deshaw.com<mailto:Paul.Sanchez at deshaw.com>]
Sent: Monday, May 16, 2016 10:28 AM GMT Standard Time
To: gpfsug-discuss at spectrumscale.org
Subject: Re: [gpfsug-discuss] Flash for metadata

Simon,

I can?t speak specifically to the FS-900, since I generally use Dell Compellent and NetApp EF arrays for flash today.  But I also take care to ensure that the system pool metadata NSDs use RAID10 (or equivalent) rather than RAID5/6.  For small 4K metadata writes, on a 4K blocksize LUN this could result in optimal writes.  For larger native blocksize LUNs, it eliminates the parity calculation for read-modify-write operations.

Thanks,
Paul

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (Research Computing - IT Services)
Sent: Monday, May 16, 2016 9:44 AM
To: gpfsug-discuss at spectrumscale.org
Subject: [gpfsug-discuss] Flash for metadata

Hi all

I'm currently about to deploy some flash systems (FS-900) to take our metadata for one of our Spectrum Scale file-systems. We use 4k inodes for this file-system.

On the FS-900, when creating a volume, we get a choice of 512b or 4k sectors. Does anyone have a suggestion on this? On a spinning disk RAID array, I get what a strip size is and we'd use multiples of the stripe where possible for the block size.

Is the sector size on the FS-900 the smallest chunk that can be written out? And so would it make sense to use 4k sectors for 4k GPFS inodes right?

Related. In the deployment guide for the FS-900 arrays, if one is using a file-system (e.g. XFS) on the array the guide talks about aligning the partitions with with array, when GPFS is used on a storage array, does it automatically work out the sector alignment?


Thanks

Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160516/9dbf385b/attachment-0002.htm>

From dacalder at co.ibm.com  Tue May 17 00:14:53 2016
From: dacalder at co.ibm.com (Danny Alexander Calderon Rodriguez)
Date: Mon, 16 May 2016 23:14:53 +0000
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 52,
	Issue 20 Flash for metadata
In-Reply-To: <mailman.3639.1463392096.1371.gpfsug-discuss@spectrumscale.org>
References: <mailman.3639.1463392096.1371.gpfsug-discuss@spectrumscale.org>
Message-ID: <201605162315.u4GNDqVo032100@mx0b-001b2d01.pphosted.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160516/14aea31a/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Image.14634403158690.gif
Type: image/gif
Size: 360 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160516/14aea31a/attachment-0002.gif>

From Luke.Raimbach at crick.ac.uk  Tue May 17 09:03:24 2016
From: Luke.Raimbach at crick.ac.uk (Luke Raimbach)
Date: Tue, 17 May 2016 08:03:24 +0000
Subject: [gpfsug-discuss] Feature Request: Clones Inheriting XATTRS
Message-ID: <AMSPR03MB020C6EAB8A9965F03517B16B0480@AMSPR03MB020.eurprd03.prod.outlook.com>

HI All,

Where do I send feature requests?

I'd like the mmclone command to have a switch which allows user defined attributes to be passed to file clones. I can understand why the "gpfs." defined extended attributes don't get passed along, because they might hold special information like how a DMAPI agent has fiddled with the data, which won't necessarily hold true for clones.

However, what would be really useful is something like "--xattr" switch for mmclone, which allows "user." defined attributes to be inherited by clones.

We are intending to use file cloning as an internal data sharing mechanism (to reduce physical disk space usage). Some scientific groups may wish to pass along all the extended attributes (e.g. instrument type, serial number, experiment ID, etc.) to those they share data with, and others may wish to strip out those attributes.

Where shall I send this one?

Cheers,
Luke.

Luke Raimbach?
Senior HPC Data and Storage Systems Engineer,
The Francis Crick Institute,
Gibbs Building,
215 Euston Road,
London NW1 2BE.

E: luke.raimbach at crick.ac.uk
W: www.crick.ac.uk


The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no. 06885462, with its registered office at 215 Euston Road, London NW1 2BE.

From samuel.guest at noaa.gov  Tue May 17 18:31:11 2016
From: samuel.guest at noaa.gov (Sam Guest)
Date: Tue, 17 May 2016 13:31:11 -0400
Subject: [gpfsug-discuss] New UG Subscriber
Message-ID: <0ca901d1b061$e7fd2ba0$b7f782e0$@noaa.gov>

Our organization is implementing a ESS/GPFS integrated solution via RedHat 7
on Power8 and IBM DCS3700 hardware to support the Comprehensive Large
Array-data Stewardship System (CLASS) for NOAA.gov. The ESS/GPFS solution
will be replacing a legacy shared file system solution from another storage
vendor. Our office is located in the Washington, DC metro area. The CLASS
application is expected to process over 500k files and approximately 30 TB
of data per day in the near future. I am new to the ESS/GPFS environment and
have an extensive background as a AIX and UNIX systems administrator. I am
excited to have the opportunity to be a part of such a dynamic technology
and look forward to learning, sharing, and networking with the group.

Sincerely,

Sam Guest, NOAA Affiliate
Systems Engineer
CSRA, DGP Team
NESDIS/OSD/GSD/CLASS
7855 Walker Drive, Suite 200, Greenbelt, MD 20770 
Desk: 240-542-1117 Cell : 301-706-1168 Fax : 301-474-4074





From jonathan at buzzard.me.uk  Tue May 17 21:22:39 2016
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Tue, 17 May 2016 21:22:39 +0100
Subject: [gpfsug-discuss] Flash for metadata
In-Reply-To: <49a3ab3b493a49e59177c6c25dd34d23@mbxtoa1.winmail.deshaw.com>
References: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F577FBC6@EX13.adf.bham.ac.uk>
	<49a3ab3b493a49e59177c6c25dd34d23@mbxtoa1.winmail.deshaw.com>
Message-ID: <573B7D8F.7010106@buzzard.me.uk>

On 16/05/16 10:28, Sanchez, Paul wrote:
> Simon,
>
> I can?t speak specifically to the FS-900, since I generally use Dell
> Compellent and NetApp EF arrays for flash today.  But I also take care
> to ensure that the system pool metadata NSDs use RAID10 (or equivalent)
> rather than RAID5/6.  For small 4K metadata writes, on a 4K blocksize
> LUN this could result in optimal writes.  For larger native blocksize
> LUNs, it eliminates the parity calculation for read-modify-write
> operations.
>

General point would it not make more sense to make multiple RAID1's and 
set replication of the metadata than make a RAID10? Preferably with the 
RAID1's on more than one storage array?

Always gives me a warm fuzzy feeling to have my metadata replicated even 
if the data is not.

JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.


From makaplan at us.ibm.com  Wed May 18 00:26:58 2016
From: makaplan at us.ibm.com (Marc A Kaplan)
Date: Tue, 17 May 2016 19:26:58 -0400
Subject: [gpfsug-discuss] Flash for metadata
In-Reply-To: <573B7D8F.7010106@buzzard.me.uk>
References: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F577FBC6@EX13.adf.bham.ac.uk><49a3ab3b493a49e59177c6c25dd34d23@mbxtoa1.winmail.deshaw.com>
	<573B7D8F.7010106@buzzard.me.uk>
Message-ID: <201605172327.u4HNRYAv023425@d03av05.boulder.ibm.com>

I think there are two points here:

A) RAID striping is probably a "loser" for GPFS metadata. 
B) RAID mirroring for your metadata may or may not be faster and/or more 
reliable than GPFS replication. Depending on your requirements and 
assumptions for fault-tolerance one or the other might be the winner.

But .. if you're serious about performance - measure, measure, measure and 
pick the winner. (And we'd like to know what you found!)
And if you want the best performance, you'll probably need to (re)measure 
whenever you acquire new equipment.

GPFS can do and does striping for high performance when reading and 
writing metadata and data...


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160517/296937ab/attachment-0002.htm>

From chair at spectrumscale.org  Wed May 18 19:35:44 2016
From: chair at spectrumscale.org (Spectrum scale UG Chair (Simon Thompson))
Date: Wed, 18 May 2016 19:35:44 +0100
Subject: [gpfsug-discuss] UK user group
Message-ID: <mailman.2.1679913370.3671611.gpfsug-discuss_gpfsug.org@gpfsug.org>

Hi All,

Well, the UK user group is wrapped up. Most of the slides are now live on the user group site on the presentations pages. Hopefully we'll get the last few up in the next few days.

Thanks to all the user speakers, IBM speakers and those from our sponsors for supporting the event. And also to the members of the community who come along to support us, to ask questions and to challenge IBM for the future!

We had just under 30 speakers from across the globe, and a packed agenda over two days. You can get some insight on the days by checking the #SSUG16 hashtag on twitter.

Again thanks to the sponsors from Arcastream, DDN, IBM, Lenovo, Mellanox, NetApp, OCF, Seagate for supporting the event.

A final note of thanks to Oxford e-research centre for lending is their camera for the camera for two days, we have videos from many of the talks, we need to work out what to do with them, hopefully will get something online in the next few weeks! So keep an eye out!

If you came along and forget to give us your feedback forms, please drop myself and Claire an email and let us know!

And don't forget, if you missed us, there are user groups events over the next six weeks in New York, Chicago and Frankfurt!

Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160518/4920a224/attachment.htm>

From peter.botcherby at kcl.ac.uk  Wed May 18 19:41:57 2016
From: peter.botcherby at kcl.ac.uk (Botcherby, Peter)
Date: Wed, 18 May 2016 18:41:57 +0000
Subject: [gpfsug-discuss] UK user group
In-Reply-To: <20160518183607.9786A41C837@gpfsug.org>
References: <20160518183607.9786A41C837@gpfsug.org>
Message-ID: <VI1PR03MB1871E3D2415EC756778B0A35A4490@VI1PR03MB1871.eurprd03.prod.outlook.com>

Hi Simon,
Sorry I couldn't make it but look forward to seeing the presentations and the videos later.

All the best
Cheers
Peter
________________________________
From: Spectrum scale UG Chair (Simon Thompson)<mailto:chair at spectrumscale.org>
Sent: ?18/?05/?2016 19:36
To: gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] UK user group

Hi All,

Well, the UK user group is wrapped up. Most of the slides are now live on the user group site on the presentations pages. Hopefully we'll get the last few up in the next few days.

Thanks to all the user speakers, IBM speakers and those from our sponsors for supporting the event. And also to the members of the community who come along to support us, to ask questions and to challenge IBM for the future!

We had just under 30 speakers from across the globe, and a packed agenda over two days. You can get some insight on the days by checking the #SSUG16 hashtag on twitter.

Again thanks to the sponsors from Arcastream, DDN, IBM, Lenovo, Mellanox, NetApp, OCF, Seagate for supporting the event.

A final note of thanks to Oxford e-research centre for lending is their camera for the camera for two days, we have videos from many of the talks, we need to work out what to do with them, hopefully will get something online in the next few weeks! So keep an eye out!

If you came along and forget to give us your feedback forms, please drop myself and Claire an email and let us know!

And don't forget, if you missed us, there are user groups events over the next six weeks in New York, Chicago and Frankfurt!

Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160518/11688115/attachment-0002.htm>

From S.J.Thompson at bham.ac.uk  Wed May 18 19:48:06 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Wed, 18 May 2016 18:48:06 +0000
Subject: [gpfsug-discuss] UK user group
In-Reply-To: <20160518183607.9786A41C837@gpfsug.org>
References: <20160518183607.9786A41C837@gpfsug.org>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F57805F4@EX13.adf.bham.ac.uk>


The presentations page is not obvious... Its under the menu that disappears....

www.spectrumscale.org/presentations/

Simon
________________________________________
From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Spectrum scale UG Chair (Simon Thompson) [chair at spectrumscale.org]
Sent: 18 May 2016 19:35
To: gpfsug-discuss at spectrumscale.org
Subject: [gpfsug-discuss] UK user group

Hi All,

Well, the UK user group is wrapped up. Most of the slides are now live on the user group site on the presentations pages. Hopefully we'll get the last few up in the next few days.

Thanks to all the user speakers, IBM speakers and those from our sponsors for supporting the event. And also to the members of the community who come along to support us, to ask questions and to challenge IBM for the future!

We had just under 30 speakers from across the globe, and a packed agenda over two days. You can get some insight on the days by checking the #SSUG16 hashtag on twitter.

Again thanks to the sponsors from Arcastream, DDN, IBM, Lenovo, Mellanox, NetApp, OCF, Seagate for supporting the event.

A final note of thanks to Oxford e-research centre for lending is their camera for the camera for two days, we have videos from many of the talks, we need to work out what to do with them, hopefully will get something online in the next few weeks! So keep an eye out!

If you came along and forget to give us your feedback forms, please drop myself and Claire an email and let us know!

And don't forget, if you missed us, there are user groups events over the next six weeks in New York, Chicago and Frankfurt!

Simon


From S.J.Thompson at bham.ac.uk  Thu May 19 16:08:12 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Thu, 19 May 2016 15:08:12 +0000
Subject: [gpfsug-discuss] UK user group
In-Reply-To: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F57805F4@EX13.adf.bham.ac.uk>
References: <20160518183607.9786A41C837@gpfsug.org>
	<CF45EE16DEF2FE4B9AA7FF2B6EE26545F57805F4@EX13.adf.bham.ac.uk>
Message-ID: <D3639532.2AAEB%s.j.thompson@bham.ac.uk>

And my take on the event ...

http://www.spectrumscale.org/uk-may-2016-group-report/

Don't forget, we're happy to take blog contributions from others as well,
events, musings on Spectrum Scale welcome ...

Simon



From S.J.Thompson at bham.ac.uk  Thu May 19 22:16:57 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Thu, 19 May 2016 21:16:57 +0000
Subject: [gpfsug-discuss] Flash for metadata
In-Reply-To: <201605172327.u4HNRYAv023425@d03av05.boulder.ibm.com>
References: <201605172327.u4HNRYAv023425@d03av05.boulder.ibm.com>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F578235F@EX13.adf.bham.ac.uk>

Thanks to all those who replied (including off list).

Someone reminded me that whilst the FlashSystem will support 512 and 4k blocks, it "prefers" 4k. Maybe there is a performance penalty for the smaller writes.

Second, a flash update is always going to be read erase write.

We don't get any choice with FlashSystem 900 on the RAID set - only RAID5 is available.

Given we have two FlashSystems in two data centres, looks like we are going with 4k sectors and GPFS replication.

I assume that creating several smaller LUNs on each FlashSystem in the same failure group is still preferable to one big LUN so we get more IO queues to play with?

Simon

-----Original Message-----
From: Marc A Kaplan [makaplan at us.ibm.com<mailto:makaplan at us.ibm.com>]
Sent: Wednesday, May 18, 2016 12:27 AM GMT Standard Time
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Flash for metadata

I think there are two points here:

A) RAID striping is probably a "loser" for GPFS metadata.
B) RAID mirroring for your metadata may or may not be faster and/or more reliable than GPFS replication. Depending on your requirements and assumptions for fault-tolerance one or the other might be the winner.

But .. if you're serious about performance - measure, measure, measure and pick the winner. (And we'd like to know what you found!)
And if you want the best performance, you'll probably need to (re)measure whenever you acquire new equipment.

GPFS can do and does striping for high performance when reading and writing metadata and data...

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160519/fdec78f4/attachment-0002.htm>

From makaplan at us.ibm.com  Thu May 19 22:59:32 2016
From: makaplan at us.ibm.com (Marc A Kaplan)
Date: Thu, 19 May 2016 17:59:32 -0400
Subject: [gpfsug-discuss] Flash for metadata
In-Reply-To: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F578235F@EX13.adf.bham.ac.uk>
References: <201605172327.u4HNRYAv023425@d03av05.boulder.ibm.com>
	<CF45EE16DEF2FE4B9AA7FF2B6EE26545F578235F@EX13.adf.bham.ac.uk>
Message-ID: <201605192159.u4JLxZox006181@d03av05.boulder.ibm.com>

"I assume that creating several smaller LUNs on each FlashSystem in the 
same failure group is still preferable to one big LUN so we get more IO 
queues to play with?"

Traditionally, more spindles, more disk arms working in parallel => better 
overall performance.

HOWEVER Flash doesn't work that way... So it's going to depend... 
Perhaps some kind soul can point us to some information about this and how 
much it varies among today's flash based storage products.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160519/939c1ca6/attachment-0002.htm>

From RWelp at uk.ibm.com  Fri May 20 20:47:15 2016
From: RWelp at uk.ibm.com (Richard Welp)
Date: Fri, 20 May 2016 20:47:15 +0100
Subject: [gpfsug-discuss] Flash for metadata
In-Reply-To: <201605192159.u4JLxZox006181@d03av05.boulder.ibm.com>
References: <201605172327.u4HNRYAv023425@d03av05.boulder.ibm.com>	<CF45EE16DEF2FE4B9AA7FF2B6EE26545F578235F@EX13.adf.bham.ac.uk>
	<201605192159.u4JLxZox006181@d03av05.boulder.ibm.com>
Message-ID: <201605201947.u4KJlKJT000383@d06av06.portsmouth.uk.ibm.com>

I checked with the FS900 performance expert in Tucson, and here is what I 
was told:

The 4KB and 512B blocks will both get equally great sub millisecond 
response times but 4KB can achieve a higher maximum IOPS rate.

As far as # of luns,  it doesn't really matter to the FS900, but the host 
operating system and other components in the data path can benefit from 
having more than 1 large lun.  If we are trying to get the maximum 
possible iops, we typically run with at least 16 luns.    I suspect with 4 
luns you would get within 10% of the maximum performance. 

Thanks,

Rick

===================
Rick Welp
Software Engineer
Master Inventor
Email: rwelp at uk.ibm.com
phone: +44 0161 214 0461 
IBM Systems - Manchester Lab
IBM UK Limited 
--------------------------




From:   "Marc A Kaplan" <makaplan at us.ibm.com>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   19/05/2016 11:00 pm
Subject:        Re: [gpfsug-discuss] Flash for metadata
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



"I assume that creating several smaller LUNs on each FlashSystem in the 
same failure group is still preferable to one big LUN so we get more IO 
queues to play with?"

Traditionally, more spindles, more disk arms working in parallel => better 
overall performance.

HOWEVER Flash doesn't work that way... So it's going to depend... 
Perhaps some kind soul can point us to some information about this and how 
much it varies among today's flash based storage products.
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160520/5f7d4126/attachment-0002.htm>

From S.J.Thompson at bham.ac.uk  Fri May 20 22:15:57 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Fri, 20 May 2016 21:15:57 +0000
Subject: [gpfsug-discuss] Flash for metadata
In-Reply-To: <201605201947.u4KJlKJT000383@d06av06.portsmouth.uk.ibm.com>
References: <201605172327.u4HNRYAv023425@d03av05.boulder.ibm.com>
	<CF45EE16DEF2FE4B9AA7FF2B6EE26545F578235F@EX13.adf.bham.ac.uk>
	<201605192159.u4JLxZox006181@d03av05.boulder.ibm.com>,
	<201605201947.u4KJlKJT000383@d06av06.portsmouth.uk.ibm.com>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F5782A0D@EX13.adf.bham.ac.uk>


Thanks Rick for checking this.

On number of LUNs, I vaguely had this in the back of my head, when we deployed our first Storwise, that was one of the reasons we built lots of metadata mirrored pairs rather than bigger arrays.

Thinking back I remember reading it was something to do with multipath and how IO queues are processed back to the storage device. The storwise we have is dual active, so only one half "owns" the LUN, so I recall it was optimal to encourage gpfs to load over controllers by having more multipath accessible devices.

And then thinking how you cable up to you LSI fc adapters as well....

Simon

________________________________________
From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Richard Welp [RWelp at uk.ibm.com]
Sent: 20 May 2016 20:47
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Flash for metadata

I checked with the FS900 performance expert in Tucson, and here is what I was told:

The 4KB and 512B blocks will both get equally great sub millisecond response times but 4KB can achieve a higher maximum IOPS rate.

As far as # of luns,  it doesn't really matter to the FS900, but the host operating system and other components in the data path can benefit from having more than 1 large lun.  If we are trying to get the maximum possible iops, we typically run with at least 16 luns.    I suspect with 4 luns you would get within 10% of the maximum performance.

Thanks,

Rick

===================
Rick Welp
Software Engineer
Master Inventor
Email: rwelp at uk.ibm.com
phone: +44 0161 214 0461
IBM Systems - Manchester Lab
IBM UK Limited
--------------------------




From:        "Marc A Kaplan" <makaplan at us.ibm.com>
To:        gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:        19/05/2016 11:00 pm
Subject:        Re: [gpfsug-discuss] Flash for metadata
Sent by:        gpfsug-discuss-bounces at spectrumscale.org
________________________________



"I assume that creating several smaller LUNs on each FlashSystem in the same failure group is still preferable to one big LUN so we get more IO queues to play with?"

Traditionally, more spindles, more disk arms working in parallel => better overall performance.

HOWEVER Flash doesn't work that way... So it's going to depend...
Perhaps some kind soul can point us to some information about this and how much it varies among today's flash based storage products.
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss




From usa-principal at gpfsug.org  Mon May 23 12:31:39 2016
From: usa-principal at gpfsug.org (usa-principal-gpfsug.org)
Date: Mon, 23 May 2016 07:31:39 -0400
Subject: [gpfsug-discuss] Speakers Wanted - June 10, Argonne National Lab
Message-ID: <1fa21d083401a0a7acff84b7cd69ee34@webmail.gpfsug.org>

Hello all,

	We?d love to have 2 more site reports for the June 10th event at 
Argonne. As a reminder, the event details are here: 
http://www.spectrumscale.org/spectrum-scale-user-group-usa-meeting-june-10th/


	Please reply and let us know what you'd like to present. It's always 
good to hear real experiences.

	Thanks in advance for your participation.

Best,
Kristy



From p.childs at qmul.ac.uk  Tue May 24 15:00:46 2016
From: p.childs at qmul.ac.uk (Peter Childs)
Date: Tue, 24 May 2016 14:00:46 +0000
Subject: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
In-Reply-To: <5744538D.3030407@qmul.ac.uk>
References: <AM2PR07MB0948D12D638C1ABD1AD603E3A44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>,
	<5744538D.3030407@qmul.ac.uk>
Message-ID: <AM2PR07MB09480DBC9D614EE7B5826AAAA44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>

 Hi All,

We have an issue where the Linux kills off GPFS first when a computer runs out of memory, this happens when user processors have exhausted memory and swap and the out of memory killer in Linux kills the GPFS daemon as the largest user of memory, due to its large pinned memory foot print.

We have an issue where the Linux kills off GPFS first when a computer runs out of memory. We are running GPFS 3.5

We believe this happens when user processes have exhausted memory and swap and the out of memory killer in Linux chooses to  kill the GPFS daemon as the largest user of memory, due to its large pinned memory footprint.

This means that GPFS is killed and the whole cluster blocks for a minute before it resumes operation, this is not ideal, and kills and causes issues with most of the cluster.

What we see is users unable to login elsewhere on the cluster until we have powered off the node. We believe this is because while the node is still pingable, GPFS doesn't expel it from the cluster.

This issue mainly occurs on our frontend nodes of our HPC cluster but can effect the rest of the cluster when it occurs.

This issue mainly occurs on the login nodes of our HPC cluster but can affect the rest of the cluster when it occurs.

I've seen others on list with this issue.

We've come up with a solution where by the gpfs is adjusted so that is unlikely to be the first thing to be killed, and hopefully the user process is killed and not GPFS.

We've come up with a solution to adjust the OOM score of GPFS, so that it is unlikely to be the first thing to be killed, and hopefully the OOM killer picks a user process instead.

Out testing says this solution works, but I'm asking here firstly to share our knowledge and secondly to ask if there is anything we've missed with this solution and issues with this.

We've tested this and it seems to work. I'm asking here firstly to share our knowledge and secondly to ask if there is anything we've missed with this solution.

Its short which is part of its beauty.

/usr/local/sbin/gpfs-oom_score_adj

<pre>
#!/bin/bash

 for proc in $(pgrep mmfs); do
      echo -500 >/proc/$proc/oom_score_adj
 done
</pre>

This can then be called automatically on GPFS startup with the following:

<pre>
mmaddcallback startupoomkiller --command /usr/local/sbin/gpfs-oom_score_adj --event startup
</pre>

and either restart gpfs or just run the script on all nodes.

Peter Childs
ITS Research Infrastructure
Queen Mary, University of London

From Paul.Sanchez at deshaw.com  Tue May 24 15:32:55 2016
From: Paul.Sanchez at deshaw.com (Sanchez, Paul)
Date: Tue, 24 May 2016 14:32:55 +0000
Subject: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
In-Reply-To: <AM2PR07MB09480DBC9D614EE7B5826AAAA44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>
References: <AM2PR07MB0948D12D638C1ABD1AD603E3A44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>,
	<5744538D.3030407@qmul.ac.uk>
	<AM2PR07MB09480DBC9D614EE7B5826AAAA44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>
Message-ID: <d6028bcf19f8402b9315ab875d075ba4@mbxtoa1.winmail.deshaw.com>

Hi Peter,



This is mentioned explicitly in the Spectrum Scale docs (http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.pdg.doc/bl1pdg_kerncfg.htm?lang=en) as a problem for the admin to consider, and many of us have been bitten by this.  There are references going back at least to GPFS 3.1 in 2008 on developerworks complaining about this situation.



While the answer you described below is essentially what we do as well, I would argue that this is a problem which IBM should just own and fix for everyone. I cannot think of a situation in which you would want GPFS to be sacrificed on a node due to out-of-memory conditions, and I have seen several terrible consequences of this, including loss of cached, user-acknowledged writes.



I don't think there are any real gotchas.  But in addition, our own implementation also:



  * uses "--event preStartup" instead of "startup", since it runs earlier and reduces the risk of a race



  * reads the score back out and complains if it hasn't been set



  * includes "set -e" to ensure that errors will terminate the script and return a non-zero exit code to the callback parent



Thx
Paul



-----Original Message-----
From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Peter Childs
Sent: Tuesday, May 24, 2016 10:01 AM
To: gpfsug main discussion list
Subject: [gpfsug-discuss] OOM Killer killing off GPFS 3.5



Hi All,



We have an issue where the Linux kills off GPFS first when a computer runs out of memory, this happens when user processors have exhausted memory and swap and the out of memory killer in Linux kills the GPFS daemon as the largest user of memory, due to its large pinned memory foot print.



We have an issue where the Linux kills off GPFS first when a computer runs out of memory. We are running GPFS 3.5



We believe this happens when user processes have exhausted memory and swap and the out of memory killer in Linux chooses to  kill the GPFS daemon as the largest user of memory, due to its large pinned memory footprint.



This means that GPFS is killed and the whole cluster blocks for a minute before it resumes operation, this is not ideal, and kills and causes issues with most of the cluster.



What we see is users unable to login elsewhere on the cluster until we have powered off the node. We believe this is because while the node is still pingable, GPFS doesn't expel it from the cluster.



This issue mainly occurs on our frontend nodes of our HPC cluster but can effect the rest of the cluster when it occurs.



This issue mainly occurs on the login nodes of our HPC cluster but can affect the rest of the cluster when it occurs.



I've seen others on list with this issue.



We've come up with a solution where by the gpfs is adjusted so that is unlikely to be the first thing to be killed, and hopefully the user process is killed and not GPFS.



We've come up with a solution to adjust the OOM score of GPFS, so that it is unlikely to be the first thing to be killed, and hopefully the OOM killer picks a user process instead.



Out testing says this solution works, but I'm asking here firstly to share our knowledge and secondly to ask if there is anything we've missed with this solution and issues with this.



We've tested this and it seems to work. I'm asking here firstly to share our knowledge and secondly to ask if there is anything we've missed with this solution.



Its short which is part of its beauty.



/usr/local/sbin/gpfs-oom_score_adj



<pre>

#!/bin/bash



for proc in $(pgrep mmfs); do

      echo -500 >/proc/$proc/oom_score_adj  done </pre>



This can then be called automatically on GPFS startup with the following:



<pre>

mmaddcallback startupoomkiller --command /usr/local/sbin/gpfs-oom_score_adj --event startup </pre>



and either restart gpfs or just run the script on all nodes.



Peter Childs

ITS Research Infrastructure

Queen Mary, University of London

_______________________________________________

gpfsug-discuss mailing list

gpfsug-discuss at spectrumscale.org

http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160524/e0679f0a/attachment-0002.htm>

From S.J.Thompson at bham.ac.uk  Tue May 24 15:33:07 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Tue, 24 May 2016 14:33:07 +0000
Subject: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
Message-ID: <D36A2390.2AD83%s.j.thompson@bham.ac.uk>

Hi Peter,

We used to see exactly this problem as well. We tweaked the init script on
our systems to adjust the OOM killer (we also tweaked it to wait for IB
links to come up as well). Its something we need to revisit now we are
running systemd based systems ...

Simon

On 24/05/2016, 15:00, "gpfsug-discuss-bounces at spectrumscale.org on behalf
of Peter Childs" <gpfsug-discuss-bounces at spectrumscale.org on behalf of
p.childs at qmul.ac.uk> wrote:

> Hi All,
>
>We have an issue where the Linux kills off GPFS first when a computer
>runs out of memory, this happens when user processors have exhausted
>memory and swap and the out of memory killer in Linux kills the GPFS
>daemon as the largest user of memory, due to its large pinned memory foot
>print.
>
>We have an issue where the Linux kills off GPFS first when a computer
>runs out of memory. We are running GPFS 3.5
>
>We believe this happens when user processes have exhausted memory and
>swap and the out of memory killer in Linux chooses to  kill the GPFS
>daemon as the largest user of memory, due to its large pinned memory
>footprint.
>
>This means that GPFS is killed and the whole cluster blocks for a minute
>before it resumes operation, this is not ideal, and kills and causes
>issues with most of the cluster.
>
>What we see is users unable to login elsewhere on the cluster until we
>have powered off the node. We believe this is because while the node is
>still pingable, GPFS doesn't expel it from the cluster.
>
>This issue mainly occurs on our frontend nodes of our HPC cluster but can
>effect the rest of the cluster when it occurs.
>
>This issue mainly occurs on the login nodes of our HPC cluster but can
>affect the rest of the cluster when it occurs.
>
>I've seen others on list with this issue.
>
>We've come up with a solution where by the gpfs is adjusted so that is
>unlikely to be the first thing to be killed, and hopefully the user
>process is killed and not GPFS.
>
>We've come up with a solution to adjust the OOM score of GPFS, so that it
>is unlikely to be the first thing to be killed, and hopefully the OOM
>killer picks a user process instead.
>
>Out testing says this solution works, but I'm asking here firstly to
>share our knowledge and secondly to ask if there is anything we've missed
>with this solution and issues with this.
>
>We've tested this and it seems to work. I'm asking here firstly to share
>our knowledge and secondly to ask if there is anything we've missed with
>this solution.
>
>Its short which is part of its beauty.
>
>/usr/local/sbin/gpfs-oom_score_adj
>
><pre>
>#!/bin/bash
>
> for proc in $(pgrep mmfs); do
>      echo -500 >/proc/$proc/oom_score_adj
> done
></pre>
>
>This can then be called automatically on GPFS startup with the following:
>
><pre>
>mmaddcallback startupoomkiller --command
>/usr/local/sbin/gpfs-oom_score_adj --event startup
></pre>
>
>and either restart gpfs or just run the script on all nodes.
>
>Peter Childs
>ITS Research Infrastructure
>Queen Mary, University of London
>_______________________________________________
>gpfsug-discuss mailing list
>gpfsug-discuss at spectrumscale.org
>http://gpfsug.org/mailman/listinfo/gpfsug-discuss



From volobuev at us.ibm.com  Tue May 24 17:17:12 2016
From: volobuev at us.ibm.com (Yuri L Volobuev)
Date: Tue, 24 May 2016 09:17:12 -0700
Subject: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
In-Reply-To: <d6028bcf19f8402b9315ab875d075ba4@mbxtoa1.winmail.deshaw.com>
References: <AM2PR07MB0948D12D638C1ABD1AD603E3A44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>,
	<5744538D.3030407@qmul.ac.uk><AM2PR07MB09480DBC9D614EE7B5826AAAA44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>
	<d6028bcf19f8402b9315ab875d075ba4@mbxtoa1.winmail.deshaw.com>
Message-ID: <20160524161719.AA629AC03F@b01ledav006.gho.pok.ibm.com>


This problem is more complex than it may seem.  The thing is, mmfsd runs as
root, as thus already possesses a certain amount of natural immunity to OOM
killer.  So adjusting mmfsd oom_score_adj doesn't radically change the
ranking of OOM killer victims, only tweaks it.  The way things are supposed
to work is: a user process eats up a lot of memory, and once a threshold is
hit, OOM killer picks off the memory hog, and the memory is released.
Unprivileged processes inherently have a higher OOM score, and should be
killed off first.  If that doesn't work, for some reason, the OOM killer
gets desperate and starts going after root processes.  Once things get to
this point, it's tough.  If you somehow manage to spare mmfsd per se,
what's going to happen next?  The OOM killer still needs a victim.  What
we've seen happen in such a situation is semi-random privileged process
killing.  mmfsd stays alive, but various other system processes are picked
off, and pretty quickly the node is a basket case.  A Linux node is not
very resilient to random process killing.  And it doesn't help that those
other privileged processes usually don't use much memory, so killing them
doesn't release much, and the carnage keeps on going.  The real problem is:
why wasn't the non-privileged memory hog process killed off first, before
root processes became fair game?  This is where things get pretty
complicated, and depend heavily on the Linux version.  There's one specific
issue that did get diagnosed.  If a process is using mmap and has page
faults going that result in GPFS IO, on older versions of GPFS the process
would fail to error out after a SIGKILL, due to locking complications
spanning Linux kernel VMM and GPFS mmap code.  This means the OOM killer
would attempt to kill a process, but that wouldn't produce the desired
result (the process is still around), and the OOM killer keeps moving down
the list.  This problem has been fixed in the current GPFS service levels.
It is possible that a similar problem may exist that prevents a memory hog
process from erroring out.  I strongly encourage opening a PMR to
investigate such a situation, instead of trying to work around it without
understanding why mmfsd was targeted in the first place.

This is the case of prevention being the best cure.  Where we've seen
success is customers using cgroups to prevent user processes from running a
node out of memory in the first place.  This has been shown to work well.
Dealing with the fallout from running out of memory is a much harder task.

The post-mmfsd-kill symptoms that are described in the original note are
not normal.  If an mmfsd process is killed, other nodes will become aware
of this fact faily quickly, and the node is going to be expelled from the
cluster (yes, expels *can* be a good thing).  In the normal case, TCP/IP
sockets are closed as soon as mmfsd is killed, and other nodes immediately
receive TCP RST packets, and close their connection endpoints.  If the
worst case, if a node just becomes catatonic, but RST is not sent out, the
troubled node is going to be expelled from the cluster after about 2
minutes of pinging (in a default configuration).  There should definitely
not be a permanent hang that necessitates a manual intervention.  Again,
older versions of GPFS had no protection against surprise OOM thread kills,
but in the current code some self-monitoring capabilities have been added,
and a single troubled node won't have a lasting impact on the cluster.  If
you aren't running with a reasonably current level of GPFS 3.5 service, I
strongly recommend upgrading.  If you see the symptoms originally described
with the current code, that's a bug that we need to fix, so please open a
PMR to address the issue.

yuri



From:	"Sanchez, Paul" <Paul.Sanchez at deshaw.com>
To:	gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>,
Date:	05/24/2016 07:33 AM
Subject:	Re: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



Hi Peter,

This is mentioned explicitly in the Spectrum Scale docs (
http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.pdg.doc/bl1pdg_kerncfg.htm?lang=en
) as a problem for the admin to consider, and many of us have been bitten
by this.  There are references going back at least to GPFS 3.1 in 2008 on
developerworks complaining about this situation.

While the answer you described below is essentially what we do as well, I
would argue that this is a problem which IBM should just own and fix for
everyone. I cannot think of a situation in which you would want GPFS to be
sacrificed on a node due to out-of-memory conditions, and I have seen
several terrible consequences of this, including loss of cached,
user-acknowledged writes.

I don't think there are any real gotchas.  But in addition, our own
implementation also:

  * uses "--event preStartup" instead of "startup", since it runs earlier
and reduces the risk of a race

  * reads the score back out and complains if it hasn't been set

  * includes "set -e" to ensure that errors will terminate the script and
return a non-zero exit code to the callback parent

Thx
Paul

-----Original Message-----
From: gpfsug-discuss-bounces at spectrumscale.org [
mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Peter Childs
Sent: Tuesday, May 24, 2016 10:01 AM
To: gpfsug main discussion list
Subject: [gpfsug-discuss] OOM Killer killing off GPFS 3.5

Hi All,

We have an issue where the Linux kills off GPFS first when a computer runs
out of memory, this happens when user processors have exhausted memory and
swap and the out of memory killer in Linux kills the GPFS daemon as the
largest user of memory, due to its large pinned memory foot print.

We have an issue where the Linux kills off GPFS first when a computer runs
out of memory. We are running GPFS 3.5

We believe this happens when user processes have exhausted memory and swap
and the out of memory killer in Linux chooses to  kill the GPFS daemon as
the largest user of memory, due to its large pinned memory footprint.

This means that GPFS is killed and the whole cluster blocks for a minute
before it resumes operation, this is not ideal, and kills and causes issues
with most of the cluster.

What we see is users unable to login elsewhere on the cluster until we have
powered off the node. We believe this is because while the node is still
pingable, GPFS doesn't expel it from the cluster.

This issue mainly occurs on our frontend nodes of our HPC cluster but can
effect the rest of the cluster when it occurs.

This issue mainly occurs on the login nodes of our HPC cluster but can
affect the rest of the cluster when it occurs.

I've seen others on list with this issue.

We've come up with a solution where by the gpfs is adjusted so that is
unlikely to be the first thing to be killed, and hopefully the user process
is killed and not GPFS.

We've come up with a solution to adjust the OOM score of GPFS, so that it
is unlikely to be the first thing to be killed, and hopefully the OOM
killer picks a user process instead.

Out testing says this solution works, but I'm asking here firstly to share
our knowledge and secondly to ask if there is anything we've missed with
this solution and issues with this.

We've tested this and it seems to work. I'm asking here firstly to share
our knowledge and secondly to ask if there is anything we've missed with
this solution.

Its short which is part of its beauty.

/usr/local/sbin/gpfs-oom_score_adj

<pre>
#!/bin/bash

for proc in $(pgrep mmfs); do
      echo -500 >/proc/$proc/oom_score_adj  done </pre>

This can then be called automatically on GPFS startup with the following:

<pre>
mmaddcallback startupoomkiller --command /usr/local/sbin/gpfs-oom_score_adj
--event startup </pre>

and either restart gpfs or just run the script on all nodes.

Peter Childs
ITS Research Infrastructure
Queen Mary, University of London
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160524/43490dfb/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160524/43490dfb/attachment-0002.gif>

From volobuev at us.ibm.com  Tue May 24 20:25:06 2016
From: volobuev at us.ibm.com (Yuri L Volobuev)
Date: Tue, 24 May 2016 12:25:06 -0700
Subject: [gpfsug-discuss] Flash for metadata
In-Reply-To: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F577FC47@EX13.adf.bham.ac.uk>
References: <49a3ab3b493a49e59177c6c25dd34d23@mbxtoa1.winmail.deshaw.com>
	<CF45EE16DEF2FE4B9AA7FF2B6EE26545F577FC47@EX13.adf.bham.ac.uk>
Message-ID: <20160524192511.E6BFD6E041@b03ledav001.gho.boulder.ibm.com>

GPFS will issue 4K IOs if the file system is 4K-aligned, which is the
default for 4.1.1+, and may issue requests as small as 512 bytes otherwise.
If a given array presents 4K physical block size to the world, you won't be
able to format a non-4K-aligned file system which puts metadata on that
disk.  It is generally believed that 4K IO size produces optimal
performance with flash-based storage in general, because this is the native
physical block size, and 512-byte sector emulation entails doing a
read-modify-write.  We don't have the data on how much difference the
sector size actually makes.  If in doubt, I would recommend going with 4K
sectors and a 4K-aligned file system.

yuri



From:	"Simon Thompson (Research Computing - IT Services)"
            <S.J.Thompson at bham.ac.uk>
To:	"'gpfsug-discuss at spectrumscale.org'"
            <gpfsug-discuss at spectrumscale.org>,
Date:	05/16/2016 02:48 AM
Subject:	Re: [gpfsug-discuss] Flash for metadata
Sent by:	gpfsug-discuss-bounces at spectrumscale.org




Hi Paul,

Thanks, interestingly the FS-900 only does raid5. I get no choice about
that, I do get a sector size choice when I create a lun on it...

I was also wondering how gpfs would do a metadata update. Would it be smart
enough to know the sector size was 512b and only modify that, or would it
always do a read modify write on the whole 4k inode?

Simon

-----Original Message-----
From: Sanchez, Paul [Paul.Sanchez at deshaw.com]
Sent: Monday, May 16, 2016 10:28 AM GMT Standard Time
To: gpfsug-discuss at spectrumscale.org
Subject: Re: [gpfsug-discuss] Flash for metadata

Simon,

I can?t speak specifically to the FS-900, since I generally use Dell
Compellent and NetApp EF arrays for flash today.  But I also take care to
ensure that the system pool metadata NSDs use RAID10 (or equivalent) rather
than RAID5/6.  For small 4K metadata writes, on a 4K blocksize LUN this
could result in optimal writes.  For larger native blocksize LUNs, it
eliminates the parity calculation for read-modify-write operations.

Thanks,
Paul

From: gpfsug-discuss-bounces at spectrumscale.org [
mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon
Thompson (Research Computing - IT Services)
Sent: Monday, May 16, 2016 9:44 AM
To: gpfsug-discuss at spectrumscale.org
Subject: [gpfsug-discuss] Flash for metadata

Hi all

I'm currently about to deploy some flash systems (FS-900) to take our
metadata for one of our Spectrum Scale file-systems. We use 4k inodes for
this file-system.

On the FS-900, when creating a volume, we get a choice of 512b or 4k
sectors. Does anyone have a suggestion on this? On a spinning disk RAID
array, I get what a strip size is and we'd use multiples of the stripe
where possible for the block size.

Is the sector size on the FS-900 the smallest chunk that can be written
out? And so would it make sense to use 4k sectors for 4k GPFS inodes right?

Related. In the deployment guide for the FS-900 arrays, if one is using a
file-system (e.g. XFS) on the array the guide talks about aligning the
partitions with with array, when GPFS is used on a storage array, does it
automatically work out the sector alignment?


Thanks

Simon_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160524/3a332e95/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160524/3a332e95/attachment-0002.gif>

From Paul.Sanchez at deshaw.com  Wed May 25 15:00:35 2016
From: Paul.Sanchez at deshaw.com (Sanchez, Paul)
Date: Wed, 25 May 2016 14:00:35 +0000
Subject: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
In-Reply-To: <20160524161719.AA629AC03F@b01ledav006.gho.pok.ibm.com>
References: <AM2PR07MB0948D12D638C1ABD1AD603E3A44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>,
	<5744538D.3030407@qmul.ac.uk><AM2PR07MB09480DBC9D614EE7B5826AAAA44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>
	<d6028bcf19f8402b9315ab875d075ba4@mbxtoa1.winmail.deshaw.com>,
	<20160524161719.AA629AC03F@b01ledav006.gho.pok.ibm.com>
Message-ID: <51fa215cfdc149c6b9c6cdeb50322d88@mbxtoa1.winmail.deshaw.com>

I'm sure that Yuri is right about the corner-case complexity across all linux and Spectrum/GPFS versions.

In situations where lots of outstanding tokens exist, and there are few token managers, we have seen the assassination of a large footprint mmfsd in GPFS 4.1 seem to impact entire clusters, potentially due to serialization in recovery of so many tokens, and overlapping access among nodes. We're looking forward to fixes in 4.2.1 to address some of this too.

But for what it's worth, on RH6/7 with 4.1, we have seen the end of OOM impacting GPFS since implementing the callback. One item I forgot is that we don't set it to -500, but to OOM_SCORE_ADJ_MIN, which on our systems is -1000. That causes the heuristic oom_badness to return the lowest possible score, more thoroughly immunizing it against selection.

Thx
Paul

Sent with Good Work (www.good.com)


From: Yuri L Volobuev <volobuev at us.ibm.com<mailto:volobuev at us.ibm.com>>
Date: Tuesday, May 24, 2016, 12:17 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] OOM Killer killing off GPFS 3.5


This problem is more complex than it may seem. The thing is, mmfsd runs as root, as thus already possesses a certain amount of natural immunity to OOM killer. So adjusting mmfsd oom_score_adj doesn't radically change the ranking of OOM killer victims, only tweaks it. The way things are supposed to work is: a user process eats up a lot of memory, and once a threshold is hit, OOM killer picks off the memory hog, and the memory is released. Unprivileged processes inherently have a higher OOM score, and should be killed off first. If that doesn't work, for some reason, the OOM killer gets desperate and starts going after root processes. Once things get to this point, it's tough. If you somehow manage to spare mmfsd per se, what's going to happen next? The OOM killer still needs a victim. What we've seen happen in such a situation is semi-random privileged process killing. mmfsd stays alive, but various other system processes are picked off, and pretty quickly the node is a basket case. A Linux node is not very resilient to random process killing. And it doesn't help that those other privileged processes usually don't use much memory, so killing them doesn't release much, and the carnage keeps on going. The real problem is: why wasn't the non-privileged memory hog process killed off first, before root processes became fair game? This is where things get pretty complicated, and depend heavily on the Linux version. There's one specific issue that did get diagnosed. If a process is using mmap and has page faults going that result in GPFS IO, on older versions of GPFS the process would fail to error out after a SIGKILL, due to locking complications spanning Linux kernel VMM and GPFS mmap code. This means the OOM killer would attempt to kill a process, but that wouldn't produce the desired result (the process is still around), and the OOM killer keeps moving down the list. This problem has been fixed in the current GPFS service levels. It is possible that a similar problem may exist that prevents a memory hog process from erroring out. I strongly encourage opening a PMR to investigate such a situation, instead of trying to work around it without understanding why mmfsd was targeted in the first place.

This is the case of prevention being the best cure. Where we've seen success is customers using cgroups to prevent user processes from running a node out of memory in the first place. This has been shown to work well. Dealing with the fallout from running out of memory is a much harder task.

The post-mmfsd-kill symptoms that are described in the original note are not normal. If an mmfsd process is killed, other nodes will become aware of this fact faily quickly, and the node is going to be expelled from the cluster (yes, expels *can* be a good thing). In the normal case, TCP/IP sockets are closed as soon as mmfsd is killed, and other nodes immediately receive TCP RST packets, and close their connection endpoints. If the worst case, if a node just becomes catatonic, but RST is not sent out, the troubled node is going to be expelled from the cluster after about 2 minutes of pinging (in a default configuration). There should definitely not be a permanent hang that necessitates a manual intervention. Again, older versions of GPFS had no protection against surprise OOM thread kills, but in the current code some self-monitoring capabilities have been added, and a single troubled node won't have a lasting impact on the cluster. If you aren't running with a reasonably current level of GPFS 3.5 service, I strongly recommend upgrading. If you see the symptoms originally described with the current code, that's a bug that we need to fix, so please open a PMR to address the issue.

yuri

[Inactive hide details for "Sanchez, Paul" ---05/24/2016 07:33:18 AM---Hi Peter, This is mentioned explicitly in the Spectrum Sc]"Sanchez, Paul" ---05/24/2016 07:33:18 AM---Hi Peter, This is mentioned explicitly in the Spectrum Scale docs (http://www.ibm.com/support/knowle

From: "Sanchez, Paul" <Paul.Sanchez at deshaw.com>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>,
Date: 05/24/2016 07:33 AM
Subject: Re: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
Sent by: gpfsug-discuss-bounces at spectrumscale.org

________________________________



Hi Peter,

This is mentioned explicitly in the Spectrum Scale docs (http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.pdg.doc/bl1pdg_kerncfg.htm?lang=en) as a problem for the admin to consider, and many of us have been bitten by this. There are references going back at least to GPFS 3.1 in 2008 on developerworks complaining about this situation.

While the answer you described below is essentially what we do as well, I would argue that this is a problem which IBM should just own and fix for everyone. I cannot think of a situation in which you would want GPFS to be sacrificed on a node due to out-of-memory conditions, and I have seen several terrible consequences of this, including loss of cached, user-acknowledged writes.

I don't think there are any real gotchas. But in addition, our own implementation also:

* uses "--event preStartup" instead of "startup", since it runs earlier and reduces the risk of a race

* reads the score back out and complains if it hasn't been set

* includes "set -e" to ensure that errors will terminate the script and return a non-zero exit code to the callback parent

Thx
Paul

-----Original Message-----
From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Peter Childs
Sent: Tuesday, May 24, 2016 10:01 AM
To: gpfsug main discussion list
Subject: [gpfsug-discuss] OOM Killer killing off GPFS 3.5

Hi All,

We have an issue where the Linux kills off GPFS first when a computer runs out of memory, this happens when user processors have exhausted memory and swap and the out of memory killer in Linux kills the GPFS daemon as the largest user of memory, due to its large pinned memory foot print.

We have an issue where the Linux kills off GPFS first when a computer runs out of memory. We are running GPFS 3.5

We believe this happens when user processes have exhausted memory and swap and the out of memory killer in Linux chooses to kill the GPFS daemon as the largest user of memory, due to its large pinned memory footprint.

This means that GPFS is killed and the whole cluster blocks for a minute before it resumes operation, this is not ideal, and kills and causes issues with most of the cluster.

What we see is users unable to login elsewhere on the cluster until we have powered off the node. We believe this is because while the node is still pingable, GPFS doesn't expel it from the cluster.

This issue mainly occurs on our frontend nodes of our HPC cluster but can effect the rest of the cluster when it occurs.

This issue mainly occurs on the login nodes of our HPC cluster but can affect the rest of the cluster when it occurs.

I've seen others on list with this issue.

We've come up with a solution where by the gpfs is adjusted so that is unlikely to be the first thing to be killed, and hopefully the user process is killed and not GPFS.

We've come up with a solution to adjust the OOM score of GPFS, so that it is unlikely to be the first thing to be killed, and hopefully the OOM killer picks a user process instead.

Out testing says this solution works, but I'm asking here firstly to share our knowledge and secondly to ask if there is anything we've missed with this solution and issues with this.

We've tested this and it seems to work. I'm asking here firstly to share our knowledge and secondly to ask if there is anything we've missed with this solution.

Its short which is part of its beauty.

/usr/local/sbin/gpfs-oom_score_adj

<pre>
#!/bin/bash

for proc in $(pgrep mmfs); do
echo -500 >/proc/$proc/oom_score_adj done </pre>

This can then be called automatically on GPFS startup with the following:

<pre>
mmaddcallback startupoomkiller --command /usr/local/sbin/gpfs-oom_score_adj --event startup </pre>

and either restart gpfs or just run the script on all nodes.

Peter Childs
ITS Research Infrastructure
Queen Mary, University of London
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160525/fed71308/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: graycol.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160525/fed71308/attachment-0004.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: graycol.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160525/fed71308/attachment-0005.gif>

From bdeluca at gmail.com  Wed May 25 15:09:06 2016
From: bdeluca at gmail.com (Ben De Luca)
Date: Wed, 25 May 2016 17:09:06 +0300
Subject: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
In-Reply-To: <51fa215cfdc149c6b9c6cdeb50322d88@mbxtoa1.winmail.deshaw.com>
References: <AM2PR07MB0948D12D638C1ABD1AD603E3A44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>
	<5744538D.3030407@qmul.ac.uk>
	<AM2PR07MB09480DBC9D614EE7B5826AAAA44F0@AM2PR07MB0948.eurprd07.prod.outlook.com>
	<d6028bcf19f8402b9315ab875d075ba4@mbxtoa1.winmail.deshaw.com>
	<20160524161719.AA629AC03F@b01ledav006.gho.pok.ibm.com>
	<51fa215cfdc149c6b9c6cdeb50322d88@mbxtoa1.winmail.deshaw.com>
Message-ID: <CAGC__DgTuomeon_tD2BSw0WdTFZQvEeKcQ-yX2EKbzykXHma0g@mail.gmail.com>

Not now, but in a previous role, we would specifically increase the oom
score on computer processes on our cluster that could consume a large
amount of ram, trying to protect system processes. Once did this we had 0
system processes die.



On 25 May 2016 at 17:00, Sanchez, Paul <Paul.Sanchez at deshaw.com> wrote:

> I'm sure that Yuri is right about the corner-case complexity across all
> linux and Spectrum/GPFS versions.
>
> In situations where lots of outstanding tokens exist, and there are few
> token managers, we have seen the assassination of a large footprint mmfsd
> in GPFS 4.1 seem to impact entire clusters, potentially due to
> serialization in recovery of so many tokens, and overlapping access among
> nodes. We're looking forward to fixes in 4.2.1 to address some of this too.
>
> But for what it's worth, on RH6/7 with 4.1, we have seen the end of OOM
> impacting GPFS since implementing the callback. One item I forgot is that
> we don't set it to -500, but to OOM_SCORE_ADJ_MIN, which on our systems is
> -1000. That causes the heuristic oom_badness to return the lowest possible
> score, more thoroughly immunizing it against selection.
>
> Thx
> Paul
>
> Sent with Good Work (www.good.com)
>
>
> *From: *Yuri L Volobuev <volobuev at us.ibm.com>
> *Date: *Tuesday, May 24, 2016, 12:17 PM
> *To: *gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> *Subject: *Re: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
>
> This problem is more complex than it may seem. The thing is, mmfsd runs as
> root, as thus already possesses a certain amount of natural immunity to OOM
> killer. So adjusting mmfsd oom_score_adj doesn't radically change the
> ranking of OOM killer victims, only tweaks it. The way things are supposed
> to work is: a user process eats up a lot of memory, and once a threshold is
> hit, OOM killer picks off the memory hog, and the memory is released.
> Unprivileged processes inherently have a higher OOM score, and should be
> killed off first. If that doesn't work, for some reason, the OOM killer
> gets desperate and starts going after root processes. Once things get to
> this point, it's tough. If you somehow manage to spare mmfsd per se, what's
> going to happen next? The OOM killer still needs a victim. What we've seen
> happen in such a situation is semi-random privileged process killing. mmfsd
> stays alive, but various other system processes are picked off, and pretty
> quickly the node is a basket case. A Linux node is not very resilient to
> random process killing. And it doesn't help that those other privileged
> processes usually don't use much memory, so killing them doesn't release
> much, and the carnage keeps on going. The real problem is: why wasn't the
> non-privileged memory hog process killed off first, before root processes
> became fair game? This is where things get pretty complicated, and depend
> heavily on the Linux version. There's one specific issue that did get
> diagnosed. If a process is using mmap and has page faults going that result
> in GPFS IO, on older versions of GPFS the process would fail to error out
> after a SIGKILL, due to locking complications spanning Linux kernel VMM and
> GPFS mmap code. This means the OOM killer would attempt to kill a process,
> but that wouldn't produce the desired result (the process is still around),
> and the OOM killer keeps moving down the list. This problem has been fixed
> in the current GPFS service levels. It is possible that a similar problem
> may exist that prevents a memory hog process from erroring out. I strongly
> encourage opening a PMR to investigate such a situation, instead of trying
> to work around it without understanding why mmfsd was targeted in the first
> place.
>
> This is the case of prevention being the best cure. Where we've seen
> success is customers using cgroups to prevent user processes from running a
> node out of memory in the first place. This has been shown to work well.
> Dealing with the fallout from running out of memory is a much harder task.
>
> The post-mmfsd-kill symptoms that are described in the original note are
> not normal. If an mmfsd process is killed, other nodes will become aware of
> this fact faily quickly, and the node is going to be expelled from the
> cluster (yes, expels *can* be a good thing). In the normal case, TCP/IP
> sockets are closed as soon as mmfsd is killed, and other nodes immediately
> receive TCP RST packets, and close their connection endpoints. If the worst
> case, if a node just becomes catatonic, but RST is not sent out, the
> troubled node is going to be expelled from the cluster after about 2
> minutes of pinging (in a default configuration). There should definitely
> not be a permanent hang that necessitates a manual intervention. Again,
> older versions of GPFS had no protection against surprise OOM thread kills,
> but in the current code some self-monitoring capabilities have been added,
> and a single troubled node won't have a lasting impact on the cluster. If
> you aren't running with a reasonably current level of GPFS 3.5 service, I
> strongly recommend upgrading. If you see the symptoms originally described
> with the current code, that's a bug that we need to fix, so please open a
> PMR to address the issue.
>
> yuri
>
> [image: Inactive hide details for "Sanchez, Paul" ---05/24/2016 07:33:18
> AM---Hi Peter, This is mentioned explicitly in the Spectrum Sc]"Sanchez,
> Paul" ---05/24/2016 07:33:18 AM---Hi Peter, This is mentioned explicitly in
> the Spectrum Scale docs (http://www.ibm.com/support/knowle
>
> From: "Sanchez, Paul" <Paul.Sanchez at deshaw.com>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>,
> Date: 05/24/2016 07:33 AM
> Subject: Re: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> ------------------------------
>
>
>
> Hi Peter,
>
> This is mentioned explicitly in the Spectrum Scale docs (
> http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.pdg.doc/bl1pdg_kerncfg.htm?lang=en)
> as a problem for the admin to consider, and many of us have been bitten by
> this. There are references going back at least to GPFS 3.1 in 2008 on
> developerworks complaining about this situation.
>
> While the answer you described below is essentially what we do as well, I
> would argue that this is a problem which IBM should just own and fix for
> everyone. I cannot think of a situation in which you would want GPFS to
> be sacrificed on a node due to out-of-memory conditions, and I have seen
> several terrible consequences of this, including loss of cached,
> user-acknowledged writes.
>
> I don't think there are any real gotchas. But in addition, our own
> implementation also:
>
> * uses "--event preStartup" instead of "startup", since it runs earlier
> and reduces the risk of a race
>
> * reads the score back out and complains if it hasn't been set
>
> * includes "set -e" to ensure that errors will terminate the script and
> return a non-zero exit code to the callback parent
>
> Thx
> Paul
>
> -----Original Message-----
> From: gpfsug-discuss-bounces at spectrumscale.org [
> mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>] On Behalf Of Peter Childs
> Sent: Tuesday, May 24, 2016 10:01 AM
> To: gpfsug main discussion list
> Subject: [gpfsug-discuss] OOM Killer killing off GPFS 3.5
>
> Hi All,
>
> We have an issue where the Linux kills off GPFS first when a computer runs
> out of memory, this happens when user processors have exhausted memory and
> swap and the out of memory killer in Linux kills the GPFS daemon as the
> largest user of memory, due to its large pinned memory foot print.
>
> We have an issue where the Linux kills off GPFS first when a computer runs
> out of memory. We are running GPFS 3.5
>
> We believe this happens when user processes have exhausted memory and swap
> and the out of memory killer in Linux chooses to kill the GPFS daemon as
> the largest user of memory, due to its large pinned memory footprint.
>
> This means that GPFS is killed and the whole cluster blocks for a minute
> before it resumes operation, this is not ideal, and kills and causes issues
> with most of the cluster.
>
> What we see is users unable to login elsewhere on the cluster until we
> have powered off the node. We believe this is because while the node is
> still pingable, GPFS doesn't expel it from the cluster.
>
> This issue mainly occurs on our frontend nodes of our HPC cluster but can
> effect the rest of the cluster when it occurs.
>
> This issue mainly occurs on the login nodes of our HPC cluster but can
> affect the rest of the cluster when it occurs.
>
> I've seen others on list with this issue.
>
> We've come up with a solution where by the gpfs is adjusted so that is
> unlikely to be the first thing to be killed, and hopefully the user process
> is killed and not GPFS.
>
> We've come up with a solution to adjust the OOM score of GPFS, so that it
> is unlikely to be the first thing to be killed, and hopefully the OOM
> killer picks a user process instead.
>
> Out testing says this solution works, but I'm asking here firstly to share
> our knowledge and secondly to ask if there is anything we've missed with
> this solution and issues with this.
>
> We've tested this and it seems to work. I'm asking here firstly to share
> our knowledge and secondly to ask if there is anything we've missed with
> this solution.
>
> Its short which is part of its beauty.
>
> /usr/local/sbin/gpfs-oom_score_adj
>
> <pre>
> #!/bin/bash
>
> for proc in $(pgrep mmfs); do
> echo -500 >/proc/$proc/oom_score_adj done </pre>
>
> This can then be called automatically on GPFS startup with the following:
>
> <pre>
> mmaddcallback startupoomkiller --command
> /usr/local/sbin/gpfs-oom_score_adj --event startup </pre>
>
> and either restart gpfs or just run the script on all nodes.
>
> Peter Childs
> ITS Research Infrastructure
> Queen Mary, University of London
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160525/f137a550/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160525/f137a550/attachment-0002.gif>

From Luke.Raimbach at crick.ac.uk  Thu May 26 09:20:02 2016
From: Luke.Raimbach at crick.ac.uk (Luke Raimbach)
Date: Thu, 26 May 2016 08:20:02 +0000
Subject: [gpfsug-discuss] CES and Centrify
Message-ID: <AMXPR03MB02105F338038F05AAD3C73BB0410@AMXPR03MB021.eurprd03.prod.outlook.com>

Hi,

Someone said "Centrify" at the recent UK user-group meeting.

Where can I find out more?

Cheers,
Luke.

Luke Raimbach?
Senior HPC Data and Storage Systems Engineer,
The Francis Crick Institute,
Gibbs Building,
215 Euston Road,
London NW1 2BE.

E: luke.raimbach at crick.ac.uk
W: www.crick.ac.uk

The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no. 06885462, with its registered office at 215 Euston Road, London NW1 2BE.

From scottcumbie at dynamixgroup.com  Thu May 26 16:57:14 2016
From: scottcumbie at dynamixgroup.com (Scott Cumbie)
Date: Thu, 26 May 2016 15:57:14 +0000
Subject: [gpfsug-discuss] mmbackup - Two at a time?
Message-ID: <35131C5F-56B9-4098-B711-4B59A025B5BB@dynamixgroup.com>

Just wondering if anyone has a quick answer to this question.  If mmbackup is running on a filesystem and a second mmbackup is executed on the same filesystem, what happens?  Does the second one end because the first one is running?

Thanks,
Scott




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160526/18d3fadb/attachment-0002.htm>

From malone12 at illinois.edu  Thu May 26 19:33:33 2016
From: malone12 at illinois.edu (Maloney, John Daniel)
Date: Thu, 26 May 2016 18:33:33 +0000
Subject: [gpfsug-discuss] Speakers Wanted - June 10, Argonne National Lab
In-Reply-To: <1fa21d083401a0a7acff84b7cd69ee34@webmail.gpfsug.org>
References: <1fa21d083401a0a7acff84b7cd69ee34@webmail.gpfsug.org>
Message-ID: <2AEEB2A9-E7C2-4F55-AA42-7C34117A9F37@illinois.edu>

Hi Kristy,
I?d be willing to present at the User Group Meeting on the 10th at Argonne.  Our talk would be based on our roll out of a new GPFS environment that leverages remote cluster mounts, AFM, and the new cluster export services functionality in 4.2 to help us better serve our campus users who leverage U of I?s campus cluster.  Let me know if a slot is still available and if/when you need more information or slides.

Best,

J.D. Maloney
Storage Engineer | Storage Enabling Technologies Group
National Center for Supercomputing Applications (NCSA)
University of Illinois Urbana-Champaign
1205 W. Clark St. Urbana, IL 61801









On 5/23/16, 6:31 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of usa-principal-gpfsug.org" <gpfsug-discuss-bounces at spectrumscale.org on behalf of usa-principal at gpfsug.org> wrote:

>http://www.spectrumscale.org/spectrum-scale-user-group-usa-meeting-june-10th/

From Robert.Oesterlin at nuance.com  Thu May 26 20:03:44 2016
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Thu, 26 May 2016 19:03:44 +0000
Subject: [gpfsug-discuss] Speakers Wanted - June 10, Argonne National Lab
In-Reply-To: <2AEEB2A9-E7C2-4F55-AA42-7C34117A9F37@illinois.edu>
References: <1fa21d083401a0a7acff84b7cd69ee34@webmail.gpfsug.org>
	<2AEEB2A9-E7C2-4F55-AA42-7C34117A9F37@illinois.edu>
Message-ID: <BCB09BBF-2893-4CF3-83B9-78A2E217D314@nuance.com>

J.D.

We may still have one open slot - can you contact me directly and let's talk details?

robert.oesterlin  @ nuance.com


Bob Oesterlin
Sr Storage Engineer, Nuance HPC Grid
507-269-0413


From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of "Maloney, John Daniel" <malone12 at illinois.edu>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Thursday, May 26, 2016 at 1:33 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Speakers Wanted - June 10, Argonne National Lab

Hi Kristy,

I?d be willing to present at the User Group Meeting on the 10th at Argonne.  Our talk would be based on our roll out of a new GPFS environment that leverages remote cluster mounts, AFM, and the new cluster export services functionality in 4.2 to help us better serve our campus users who leverage U of I?s campus cluster.  Let me know if a slot is still available and if/when you need more information or slides.



Best,



J.D. Maloney

Storage Engineer | Storage Enabling Technologies Group

National Center for Supercomputing Applications (NCSA)

University of Illinois Urbana-Champaign

1205 W. Clark St. Urbana, IL 61801



















On 5/23/16, 6:31 AM, "gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> on behalf of usa-principal-gpfsug.org" <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> on behalf of usa-principal at gpfsug.org<mailto:usa-principal at gpfsug.org>> wrote:



https://urldefense.proofpoint.com/v2/url?u=http-3A__www.spectrumscale.org_spectrum-2Dscale-2Duser-2Dgroup-2Dusa-2Dmeeting-2Djune-2D10th_&d=CwIGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=MKKI9kld3C0B3tzQT64SPW1uCBHTb72hZlRP6x4V34E&s=sUYYuCYXPiSXA6Nhejs1H_sogi2WDfREIRc5yir2FJ4&e=

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwIGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=MKKI9kld3C0B3tzQT64SPW1uCBHTb72hZlRP6x4V34E&s=lORUKCzYSOh63hgiyP_35POyN_hMzfYGVVx8kQfFl-c&e=

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160526/cc0b7542/attachment-0002.htm>

From craigawilson at gmail.com  Fri May 27 11:32:09 2016
From: craigawilson at gmail.com (Craig Wilson)
Date: Fri, 27 May 2016 11:32:09 +0100
Subject: [gpfsug-discuss] mmbackup - Two at a time?
In-Reply-To: <35131C5F-56B9-4098-B711-4B59A025B5BB@dynamixgroup.com>
References: <35131C5F-56B9-4098-B711-4B59A025B5BB@dynamixgroup.com>
Message-ID: <CALdwc7dd1b+oF3jXMnJFwKtf9QKJ7m-kRETViwA0cmp2oMYJZg@mail.gmail.com>

Hi Scott

I've not tried this one 4.# branch as I'm still running 3.5 but on 3.5 you
get a error message saying:

mmbackup: Failed to obtain the local environment update lock.
mmbackup: Command failed.  Examine previous error messages to determine
cause.

The old mmbackup keeps running through.

Kind Regards,

Craig Wilson.
HPC Systems Engineer
OCF Plc.

On 26 May 2016 at 16:57, Scott Cumbie <scottcumbie at dynamixgroup.com> wrote:

> Just wondering if anyone has a quick answer to this question.  If mmbackup
> is running on a filesystem and a second mmbackup is executed on the same
> filesystem, what happens?  Does the second one end because the first one is
> running?
>
> Thanks,
> Scott
>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160527/456339d2/attachment-0002.htm>

From S.J.Thompson at bham.ac.uk  Fri May 27 11:49:27 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Fri, 27 May 2016 10:49:27 +0000
Subject: [gpfsug-discuss] mmbackup - Two at a time?
In-Reply-To: <CALdwc7dd1b+oF3jXMnJFwKtf9QKJ7m-kRETViwA0cmp2oMYJZg@mail.gmail.com>
References: <35131C5F-56B9-4098-B711-4B59A025B5BB@dynamixgroup.com>
	<CALdwc7dd1b+oF3jXMnJFwKtf9QKJ7m-kRETViwA0cmp2oMYJZg@mail.gmail.com>
Message-ID: <D36DE461.2AF90%s.j.thompson@bham.ac.uk>

Now we've seen different to this... But maybe it depends how you run mmbackup multiple times.

We have several nodes which are backing up via Spectrum Protect, and we thought it would be nice to have the client scheduler running on multiple nodes, we assumed it would only actually run one instance of the backup. Not so. We've seen two client schedulers kick off mmbackup (each then I think running across several nodes).

Simon

From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of Craig Wilson <craigawilson at gmail.com<mailto:craigawilson at gmail.com>>
Reply-To: "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Friday, 27 May 2016 at 11:32
To: "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] mmbackup - Two at a time?

Hi Scott

I've not tried this one 4.# branch as I'm still running 3.5 but on 3.5 you get a error message saying:

mmbackup: Failed to obtain the local environment update lock.
mmbackup: Command failed.  Examine previous error messages to determine cause.

The old mmbackup keeps running through.

Kind Regards,

Craig Wilson.
HPC Systems Engineer
OCF Plc.

On 26 May 2016 at 16:57, Scott Cumbie <scottcumbie at dynamixgroup.com<mailto:scottcumbie at dynamixgroup.com>> wrote:
Just wondering if anyone has a quick answer to this question.  If mmbackup is running on a filesystem and a second mmbackup is executed on the same filesystem, what happens?  Does the second one end because the first one is running?

Thanks,
Scott





_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160527/f431c56f/attachment-0002.htm>

From craigawilson at gmail.com  Fri May 27 14:36:24 2016
From: craigawilson at gmail.com (Craig Wilson)
Date: Fri, 27 May 2016 14:36:24 +0100
Subject: [gpfsug-discuss] mmbackup - Two at a time?
In-Reply-To: <D36DE461.2AF90%s.j.thompson@bham.ac.uk>
References: <35131C5F-56B9-4098-B711-4B59A025B5BB@dynamixgroup.com>
	<CALdwc7dd1b+oF3jXMnJFwKtf9QKJ7m-kRETViwA0cmp2oMYJZg@mail.gmail.com>
	<D36DE461.2AF90%s.j.thompson@bham.ac.uk>
Message-ID: <CALdwc7cu5dmCaP__aAv5htnxV4vH8GO7pVMKmUg8WBuLxBo35w@mail.gmail.com>

That is interesting...

The only time I've seen two starting is when someone manually cleared the
environment lock. That said I've never tried running from a different node
at the same time. I presume they would be duplicating effort based on the
node name being used in the shadow database file name.

Kind regards,

Craig Wilson.

On 27 May 2016 at 11:49, Simon Thompson (Research Computing - IT Services) <
S.J.Thompson at bham.ac.uk> wrote:

> Now we've seen different to this... But maybe it depends how you run
> mmbackup multiple times.
>
> We have several nodes which are backing up via Spectrum Protect, and we
> thought it would be nice to have the client scheduler running on multiple
> nodes, we assumed it would only actually run one instance of the backup.
> Not so. We've seen two client schedulers kick off mmbackup (each then I
> think running across several nodes).
>
> Simon
>
> From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Craig
> Wilson <craigawilson at gmail.com>
> Reply-To: "gpfsug-discuss at spectrumscale.org" <
> gpfsug-discuss at spectrumscale.org>
> Date: Friday, 27 May 2016 at 11:32
> To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
> Subject: Re: [gpfsug-discuss] mmbackup - Two at a time?
>
> Hi Scott
>
> I've not tried this one 4.# branch as I'm still running 3.5 but on 3.5 you
> get a error message saying:
>
> mmbackup: Failed to obtain the local environment update lock.
> mmbackup: Command failed.  Examine previous error messages to determine
> cause.
>
> The old mmbackup keeps running through.
>
> Kind Regards,
>
> Craig Wilson.
> HPC Systems Engineer
> OCF Plc.
>
> On 26 May 2016 at 16:57, Scott Cumbie <scottcumbie at dynamixgroup.com>
> wrote:
>
>> Just wondering if anyone has a quick answer to this question.  If
>> mmbackup is running on a filesystem and a second mmbackup is executed on
>> the same filesystem, what happens?  Does the second one end because the
>> first one is running?
>>
>> Thanks,
>> Scott
>>
>>
>>
>>
>>
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>
>>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160527/c2fba71c/attachment-0002.htm>

From Kevin.Buterbaugh at Vanderbilt.Edu  Fri May 27 21:23:01 2016
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Fri, 27 May 2016 20:23:01 +0000
Subject: [gpfsug-discuss] Building portability layer fails
Message-ID: <FEEC06BD-FE26-4F2B-84D6-AD604AB63208@vanderbilt.edu>

All,

We?re upgrading our cluster from GPFS 4.1.0.8 to GPFS 4.2.0.3 and all is going very well ? except for my tape backup server.

I?ve run the Spectrum Scale installer and installed the rpm?s in /usr/lpp/mmfs/4.2.0.3/gpfs_rpms, but when I run:

make LINUX_DISTRIBUTION=REDHAT_AS_LINUX Autoconfig

it errors with:

make[3]: *** [/usr/lpp/mmfs/src/gpl-linux/kdump-kern.o] Error 1
make[2]: *** [_module_/usr/lpp/mmfs/src/gpl-linux] Error 2
make[2]: Leaving directory `/usr/src/kernels/2.6.32-573.el6.x86_64'
make[1]: *** [modules] Error 1
make[1]: Leaving directory `/usr/lpp/mmfs/src/gpl-linux'
make: *** [Modules] Error 1

Any ideas?  Thanks in advance?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160527/9513f318/attachment-0002.htm>

From sfadden at us.ibm.com  Fri May 27 21:35:58 2016
From: sfadden at us.ibm.com (Scott Fadden)
Date: Fri, 27 May 2016 13:35:58 -0700
Subject: [gpfsug-discuss] Building portability layer fails
In-Reply-To: <FEEC06BD-FE26-4F2B-84D6-AD604AB63208@vanderbilt.edu>
References: <FEEC06BD-FE26-4F2B-84D6-AD604AB63208@vanderbilt.edu>
Message-ID: <20160527203604.0F1F86A045@b03ledav003.gho.boulder.ibm.com>

On 4.2.0.3 run mmbuildgpl. See how that goes.


Scott Fadden
Spectrum Scale - Technical Marketing 
Phone: (503) 880-5833 
sfadden at us.ibm.com
http://www.ibm.com/systems/storage/spectrum/scale



From:   "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   05/27/2016 01:23 PM
Subject:        [gpfsug-discuss] Building portability layer fails
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



All, 

We?re upgrading our cluster from GPFS 4.1.0.8 to GPFS 4.2.0.3 and all is 
going very well ? except for my tape backup server.

I?ve run the Spectrum Scale installer and installed the rpm?s in 
/usr/lpp/mmfs/4.2.0.3/gpfs_rpms, but when I run:

make LINUX_DISTRIBUTION=REDHAT_AS_LINUX Autoconfig

it errors with:

make[3]: *** [/usr/lpp/mmfs/src/gpl-linux/kdump-kern.o] Error 1
make[2]: *** [_module_/usr/lpp/mmfs/src/gpl-linux] Error 2
make[2]: Leaving directory `/usr/src/kernels/2.6.32-573.el6.x86_64'
make[1]: *** [modules] Error 1
make[1]: Leaving directory `/usr/lpp/mmfs/src/gpl-linux'
make: *** [Modules] Error 1

Any ideas?  Thanks in advance?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and 
Education
Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160527/d0a3d54b/attachment-0002.htm>

From Kevin.Buterbaugh at Vanderbilt.Edu  Fri May 27 21:51:18 2016
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Fri, 27 May 2016 20:51:18 +0000
Subject: [gpfsug-discuss] Building portability layer fails
In-Reply-To: <20160527203604.0F1F86A045@b03ledav003.gho.boulder.ibm.com>
References: <FEEC06BD-FE26-4F2B-84D6-AD604AB63208@vanderbilt.edu>
	<20160527203604.0F1F86A045@b03ledav003.gho.boulder.ibm.com>
Message-ID: <4E17167D-5A7A-4A04-8A56-03F4DDB092CB@vanderbilt.edu>

All,

After spending a couple of hours on this I finally took the key step towards fixing the problem myself ? I e-mailed this list!  ;-)

Five minutes later, I saw that the env.mcr file was munged, fixed that, and everything built just fine.  Sorry for the noise?

Kevin

On May 27, 2016, at 3:35 PM, Scott Fadden <sfadden at us.ibm.com<mailto:sfadden at us.ibm.com>> wrote:

On 4.2.0.3 run mmbuildgpl. See how that goes.


Scott Fadden
Spectrum Scale - Technical Marketing
Phone: (503) 880-5833
sfadden at us.ibm.com<mailto:sfadden at us.ibm.com>
http://www.ibm.com/systems/storage/spectrum/scale



From:        "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at vanderbilt.edu>>
To:        gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date:        05/27/2016 01:23 PM
Subject:        [gpfsug-discuss] Building portability layer fails
Sent by:        gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>
________________________________



All,

We?re upgrading our cluster from GPFS 4.1.0.8 to GPFS 4.2.0.3 and all is going very well ? except for my tape backup server.

I?ve run the Spectrum Scale installer and installed the rpm?s in /usr/lpp/mmfs/4.2.0.3/gpfs_rpms, but when I run:

make LINUX_DISTRIBUTION=REDHAT_AS_LINUX Autoconfig

it errors with:

make[3]: *** [/usr/lpp/mmfs/src/gpl-linux/kdump-kern.o] Error 1
make[2]: *** [_module_/usr/lpp/mmfs/src/gpl-linux] Error 2
make[2]: Leaving directory `/usr/src/kernels/2.6.32-573.el6.x86_64'
make[1]: *** [modules] Error 1
make[1]: Leaving directory `/usr/lpp/mmfs/src/gpl-linux'
make: *** [Modules] Error 1

Any ideas?  Thanks in advance?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu>- (615)875-9633


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160527/58a0de66/attachment-0002.htm>

From S.J.Thompson at bham.ac.uk  Fri May 27 22:44:48 2016
From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services))
Date: Fri, 27 May 2016 21:44:48 +0000
Subject: [gpfsug-discuss] Building portability layer fails
In-Reply-To: <20160527203604.0F1F86A045@b03ledav003.gho.boulder.ibm.com>
References: <20160527203604.0F1F86A045@b03ledav003.gho.boulder.ibm.com>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F5786AF5@EX13.adf.bham.ac.uk>

Does mmbuildgpl work if its a RHEL clone like CentOS?

Simon

-----Original Message-----
From: Scott Fadden [sfadden at us.ibm.com<mailto:sfadden at us.ibm.com>]
Sent: Friday, May 27, 2016 09:36 PM GMT Standard Time
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Building portability layer fails

On 4.2.0.3 run mmbuildgpl. See how that goes.


Scott Fadden
Spectrum Scale - Technical Marketing
Phone: (503) 880-5833
sfadden at us.ibm.com
http://www.ibm.com/systems/storage/spectrum/scale



From:        "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu>
To:        gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:        05/27/2016 01:23 PM
Subject:        [gpfsug-discuss] Building portability layer fails
Sent by:        gpfsug-discuss-bounces at spectrumscale.org
________________________________



All,

We?re upgrading our cluster from GPFS 4.1.0.8 to GPFS 4.2.0.3 and all is going very well ? except for my tape backup server.

I?ve run the Spectrum Scale installer and installed the rpm?s in /usr/lpp/mmfs/4.2.0.3/gpfs_rpms, but when I run:

make LINUX_DISTRIBUTION=REDHAT_AS_LINUX Autoconfig

it errors with:

make[3]: *** [/usr/lpp/mmfs/src/gpl-linux/kdump-kern.o] Error 1
make[2]: *** [_module_/usr/lpp/mmfs/src/gpl-linux] Error 2
make[2]: Leaving directory `/usr/src/kernels/2.6.32-573.el6.x86_64'
make[1]: *** [modules] Error 1
make[1]: Leaving directory `/usr/lpp/mmfs/src/gpl-linux'
make: *** [Modules] Error 1

Any ideas?  Thanks in advance?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu>- (615)875-9633


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160527/c587e999/attachment-0002.htm>

From sfadden at us.ibm.com  Sat May 28 01:47:05 2016
From: sfadden at us.ibm.com (Scott Fadden)
Date: Sat, 28 May 2016 00:47:05 +0000
Subject: [gpfsug-discuss] Building portability layer fails
In-Reply-To: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F5786AF5@EX13.adf.bham.ac.uk>
Message-ID: <20160528004711.1B4A46A03D@b03ledav003.gho.boulder.ibm.com>

It should

Sent from IBM Verse


   Simon Thompson (Research Computing - IT Services) --- RE: Re: [gpfsug-discuss] Building portability layer fails --- 
    From:"Simon Thompson (Research Computing - IT Services)" <S.J.Thompson at bham.ac.uk>To:"'gpfsug main discussion list'" <gpfsug-discuss at spectrumscale.org>, "Scott Fadden" <sfadden at us.ibm.com>Date:Fri, May 27, 2016 14:45Subject:RE: Re: [gpfsug-discuss] Building portability layer fails
  
         Does mmbuildgpl work if its a RHEL clone like CentOS?
  
  Simon
  
  -----Original Message-----
  From: Scott Fadden [sfadden at us.ibm.com]
  Sent: Friday, May 27, 2016 09:36 PM GMT Standard Time
  To: gpfsug main discussion list
  Subject: Re: [gpfsug-discuss] Building portability layer fails
  
  On 4.2.0.3 run mmbuildgpl. See how that goes.
  
  
  Scott Fadden
  Spectrum Scale - Technical Marketing
  Phone: (503) 880-5833  
  sfadden at us.ibm.com
  http://www.ibm.com/systems/storage/spectrum/scale
  
  
  
  From:        "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu>
  To:        gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
  Date:        05/27/2016 01:23 PM
  Subject:        [gpfsug-discuss] Building portability layer fails
  Sent by:        gpfsug-discuss-bounces at spectrumscale.org
   
  
  
  All,
  
  We?re upgrading our cluster from GPFS 4.1.0.8 to GPFS 4.2.0.3 and all is going very well ? except for my tape backup server.
  
  I?ve run the Spectrum Scale installer and installed the rpm?s in /usr/lpp/mmfs/4.2.0.3/gpfs_rpms, but when I run:
  
  make LINUX_DISTRIBUTION=REDHAT_AS_LINUX Autoconfig
  
  it errors with:
  
  make[3]: *** [/usr/lpp/mmfs/src/gpl-linux/kdump-kern.o] Error 1
  make[2]: *** [_module_/usr/lpp/mmfs/src/gpl-linux] Error 2
  make[2]: Leaving directory `/usr/src/kernels/2.6.32-573.el6.x86_64'
  make[1]: *** [modules] Error 1
  make[1]: Leaving directory `/usr/lpp/mmfs/src/gpl-linux'
  make: *** [Modules] Error 1
  
  Any ideas?  Thanks in advance?
  
  ?
  Kevin Buterbaugh - Senior System Administrator
  Vanderbilt University - Advanced Computing Center for Research and Education
  Kevin.Buterbaugh at vanderbilt.edu- (615)875-9633
  
  
  _______________________________________________
  gpfsug-discuss mailing list
  gpfsug-discuss at spectrumscale.org
  http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160528/e1576726/attachment-0002.htm>

From laurence at qsplace.co.uk  Sat May 28 09:36:50 2016
From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow)
Date: Sat, 28 May 2016 09:36:50 +0100
Subject: [gpfsug-discuss] Building portability layer fails
In-Reply-To: <20160528004711.1B4A46A03D@b03ledav003.gho.boulder.ibm.com>
References: <20160528004711.1B4A46A03D@b03ledav003.gho.boulder.ibm.com>
Message-ID: <16F6CB71-3BC1-4F9F-BA2F-DC6911325C8A@qsplace.co.uk>

Yes it does.

But you still have to export LINUX_DISTRIBUTION=REDHAT_AS_LINUX? before the command.

i.e.
LINUX_DISTRIBUTION=REDHAT_AS_LINUX? mmbuildgpl

-- Lauz

On 28 May 2016 01:47:05 BST, Scott Fadden <sfadden at us.ibm.com> wrote:
>It should
>
>Sent from IBM Verse
>
>
>Simon Thompson (Research Computing - IT Services) --- RE: Re:
>[gpfsug-discuss] Building portability layer fails --- 
>From:"Simon Thompson (Research Computing - IT Services)"
><S.J.Thompson at bham.ac.uk>To:"'gpfsug main discussion list'"
><gpfsug-discuss at spectrumscale.org>, "Scott Fadden"
><sfadden at us.ibm.com>Date:Fri, May 27, 2016 14:45Subject:RE: Re:
>[gpfsug-discuss] Building portability layer fails
>  
>         Does mmbuildgpl work if its a RHEL clone like CentOS?
>  
>  Simon
>  
>  -----Original Message-----
>  From: Scott Fadden [sfadden at us.ibm.com]
>  Sent: Friday, May 27, 2016 09:36 PM GMT Standard Time
>  To: gpfsug main discussion list
>  Subject: Re: [gpfsug-discuss] Building portability layer fails
>  
>  On 4.2.0.3 run mmbuildgpl. See how that goes.
>  
>  
>  Scott Fadden
>  Spectrum Scale - Technical Marketing
>  Phone: (503) 880-5833  
>  sfadden at us.ibm.com
>  http://www.ibm.com/systems/storage/spectrum/scale
>  
>  
>  
>  From:        "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu>
>To:        gpfsug main discussion list
><gpfsug-discuss at spectrumscale.org>
>  Date:        05/27/2016 01:23 PM
>  Subject:        [gpfsug-discuss] Building portability layer fails
>  Sent by:        gpfsug-discuss-bounces at spectrumscale.org
>   
>  
>  
>  All,
>  
>We?re upgrading our cluster from GPFS 4.1.0.8 to GPFS 4.2.0.3 and all
>is going very well ? except for my tape backup server.
>  
>I?ve run the Spectrum Scale installer and installed the rpm?s in
>/usr/lpp/mmfs/4.2.0.3/gpfs_rpms, but when I run:
>  
>  make LINUX_DISTRIBUTION=REDHAT_AS_LINUX Autoconfig
>  
>  it errors with:
>  
>  make[3]: *** [/usr/lpp/mmfs/src/gpl-linux/kdump-kern.o] Error 1
>  make[2]: *** [_module_/usr/lpp/mmfs/src/gpl-linux] Error 2
>  make[2]: Leaving directory `/usr/src/kernels/2.6.32-573.el6.x86_64'
>  make[1]: *** [modules] Error 1
>  make[1]: Leaving directory `/usr/lpp/mmfs/src/gpl-linux'
>  make: *** [Modules] Error 1
>  
>  Any ideas?  Thanks in advance?
>  
>  ?
>  Kevin Buterbaugh - Senior System Administrator
>Vanderbilt University - Advanced Computing Center for Research and
>Education
>  Kevin.Buterbaugh at vanderbilt.edu- (615)875-9633
>  
>  
>  _______________________________________________
>  gpfsug-discuss mailing list
>  gpfsug-discuss at spectrumscale.org
>  http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>gpfsug-discuss mailing list
>gpfsug-discuss at spectrumscale.org
>http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160528/077f9264/attachment-0002.htm>

From Robert.Oesterlin at nuance.com  Tue May 31 12:57:19 2016
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Tue, 31 May 2016 11:57:19 +0000
Subject: [gpfsug-discuss] GPFS/Spectrum Scale security vulernability - All
	versions
Message-ID: <E0694FAD-6D65-40AA-AB98-D6934E24D557@nuance.com>

IBM published a security vulnerability today that effects all current and prior levels of GPFS/Spectrum Scale. The short explanation is "IBM Spectrum Scale and IBM GPFS that could allow a local attacker to inject commands into setuid file parameters and execute commands as root." This is a "high" vulnerability (8.4).

Details here: http://www-01.ibm.com/support/docview.wss?uid=ssg1S1005781


Bob Oesterlin
Sr Storage Engineer, Nuance HPC Grid


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160531/184da280/attachment-0002.htm>