From novosirj at rutgers.edu  Mon Jul  1 06:42:51 2019
From: novosirj at rutgers.edu (Ryan Novosielski)
Date: Mon, 1 Jul 2019 05:42:51 +0000
Subject: [gpfsug-discuss] Any guidelines for choosing vdisk size?
Message-ID: <E55F8C7F-48A4-4A66-ABF8-880A5232F39D@oarc.rutgers.edu>

Good morning,

Was wondering if anyone could point me to any tips or provide some regarding choosing vdisk size? My understanding is that too small is a waste of resources in the form of overhead, and that there is an upper limit. but that generally within a pool, you want them to be the same size, so that if you aren?t allocating the entire storage space on the system straight off, you?ll need to choose a reasonable size or be left with unusable space (eg. you will waste space if you went with, say, 200 GB vdisk sizes on a 500 GB array).

Anyone have any tips? Do people just generally allocate the whole thing and have one 2 vdisks (redundancy)? Seems you have some more flexibility if you don?t do that and have to, say, create a storage pool or filesystem from scratch to take advantage of features in a newer FS version or what have you.

Thanks for the help ? spent a lot of time looking for this previously, but never asked on the list.
 
--
____
|| \\UTGERS,  	 |---------------------------*O*---------------------------
||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
     `'


From abeattie at au1.ibm.com  Mon Jul  1 06:54:37 2019
From: abeattie at au1.ibm.com (Andrew Beattie)
Date: Mon, 1 Jul 2019 05:54:37 +0000
Subject: [gpfsug-discuss] Any guidelines for choosing vdisk size?
In-Reply-To: <E55F8C7F-48A4-4A66-ABF8-880A5232F39D@oarc.rutgers.edu>
References: <E55F8C7F-48A4-4A66-ABF8-880A5232F39D@oarc.rutgers.edu>
Message-ID: <OF54C435D4.B3E41DA2-ON0025842A.001FD56B-0025842A.002077AE@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190701/6acfe5f8/attachment.htm>

From novosirj at rutgers.edu  Mon Jul  1 07:01:46 2019
From: novosirj at rutgers.edu (Ryan Novosielski)
Date: Mon, 1 Jul 2019 06:01:46 +0000
Subject: [gpfsug-discuss] Any guidelines for choosing vdisk size?
In-Reply-To: <OF54C435D4.B3E41DA2-ON0025842A.001FD56B-0025842A.002077AE@notes.na.collabserv.com>
References: <E55F8C7F-48A4-4A66-ABF8-880A5232F39D@oarc.rutgers.edu>
	<OF54C435D4.B3E41DA2-ON0025842A.001FD56B-0025842A.002077AE@notes.na.collabserv.com>
Message-ID: <D599D062-6286-477C-B494-4AD0984691AD@rutgers.edu>

Sure; thanks, I meant to add those details:

In our case, all GSS/DSS-G with GNR.

The context here is general information, but specifically designing new filesystems/setting up one of these systems from scratch in what I?d figure to be the traditional way: don?t allocate the entire storage if you?re not totally sure the way the growth will go, but don?t leave behind unusable space or space that forces you to later change vdisk sizes. Essentially allocate something today that makes sense, but leaves room open for sensible growth with both the existing hardware and new, if expanded. If I?m thinking about this generally the wrong way, let me know.

The systems we currently own are GSS26 (6 TB drives, fully populated ? I forget how many drives, 350 or so?) , and a few DSS-G 220 w/10 TB drives (also fully populated with 168 drives).

> On Jul 1, 2019, at 1:54 AM, Andrew Beattie <abeattie at au1.ibm.com> wrote:
> 
> Hi Ryan,
>  
> Can I ask a couple of clarifying questions?
>  
> Is this in an ESS / GSS environment with Scale native raid ?
> Is this in classic San attached storage environment / or Direct Attached Disk without Scale Native Raid?
>  
> What are you trying to achieve?
> Are you presenting the Vdisk into an existing filesystem ?
> Are you creating an new filesystem?
>  
>  
> Regards,
>  
> Andrew Beattie
> File and Object Storage Technical Specialist - A/NZ
> IBM Systems - Storage
> Phone: 614-2133-7927
> E-mail: abeattie at au1.ibm.com
>  
>  
> ----- Original message -----
> From: Ryan Novosielski <novosirj at rutgers.edu>
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Cc:
> Subject: [EXTERNAL] [gpfsug-discuss] Any guidelines for choosing vdisk size?
> Date: Mon, Jul 1, 2019 3:43 PM
>  
> Good morning,
> 
> Was wondering if anyone could point me to any tips or provide some regarding choosing vdisk size? My understanding is that too small is a waste of resources in the form of overhead, and that there is an upper limit. but that generally within a pool, you want them to be the same size, so that if you aren?t allocating the entire storage space on the system straight off, you?ll need to choose a reasonable size or be left with unusable space (eg. you will waste space if you went with, say, 200 GB vdisk sizes on a 500 GB array).
> 
> Anyone have any tips? Do people just generally allocate the whole thing and have one 2 vdisks (redundancy)? Seems you have some more flexibility if you don?t do that and have to, say, create a storage pool or filesystem from scratch to take advantage of features in a newer FS version or what have you.
> 
> Thanks for the help ? spent a lot of time looking for this previously, but never asked on the list.
>  
> --
> ____
> || \\UTGERS,   |---------------------------*O*---------------------------
> ||_// the State |         Ryan Novosielski - novosirj at rutgers.edu
> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
> ||  \\    of NJ | Office of Advanced Research Computing - MSB C630, Newark
>      `'
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss 
>  
>  
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss


From mhennecke at lenovo.com  Mon Jul  1 08:25:23 2019
From: mhennecke at lenovo.com (Michael Hennecke)
Date: Mon, 1 Jul 2019 07:25:23 +0000
Subject: [gpfsug-discuss] Any guidelines for choosing vdisk size?
Message-ID: <TY2PR03MB446249F071E9252F75D87E5ACCF90@TY2PR03MB4462.apcprd03.prod.outlook.com>

Ryan,

That is a question with many variables. Some of the factors influencing the decision are:
* Are you sharing data and metadata (the GNR default these days), or are you building separate data and metadata vdisks (the "older" default, and still useful for some use cases)? If you have separate metadata, you need to ensure you leave enough free space in the RGs to account for metadata growth beyond your initial sizing.
* Are you creating vdisks for multiple filesystems on the same building block, or just for one filesystem? 
* How likely is it that space needs to be (re-)allocated between different filesystems?
* How likely is it that new storage will be added to existing filesystems?
* Is there a requirement to include building blocks of different size (different #disks, or different capacity of disks) in the same filesystem?

All of these influence the decision on the "best" vdisk quantity and capacities. If you have a specific configuration question, I'm happy to discuss that offline.
 
In general I advise customers to use at least two equally sized vdisks per RG (4 per building block), so there is more flexibility when capacity needs to be moved around. But many sites that have homogeneous building blocks and don't expect changes in their storage configuration just go with a single vdisk per RG (2 per building block).

Either way, it is a good idea (in environments with more than a single building block) to put aside a small "test" vdisk per RG just for performance testing. So you can do performance validations on each building block individually even if your main filesystem spans all building block. Size of that "test" vdisk should be big enough so you can run a few minutes of sequential IOR against it without filling it up 100%.


Mit freundlichen Gr?ssen / Best regards,

Michael Hennecke
HPC Chief Technologist - HPC and AI Business Unit  
mailto:mhennecke at lenovo.com
--
Lenovo Global Technology (Germany) GmbH * Am Zehnthof 77 * D-45307 Essen * Germany 
Gesch?ftsf?hrung: Colm Gleeson, Christophe Laurent * Sitz der Gesellschaft: Stuttgart * HRB-Nr.: 758298, AG Stuttgart

-----Original Message-----
From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of Ryan Novosielski
Sent: Monday, 1 July, 2019 07:43
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [External] [gpfsug-discuss] Any guidelines for choosing vdisk size?

Good morning,

Was wondering if anyone could point me to any tips or provide some regarding choosing vdisk size? My understanding is that too small is a waste of resources in the form of overhead, and that there is an upper limit. but that generally within a pool, you want them to be the same size, so that if you aren?t allocating the entire storage space on the system straight off, you?ll need to choose a reasonable size or be left with unusable space (eg. you will waste space if you went with, say, 200 GB vdisk sizes on a 500 GB array).

Anyone have any tips? Do people just generally allocate the whole thing and have one 2 vdisks (redundancy)? Seems you have some more flexibility if you don?t do that and have to, say, create a storage pool or filesystem from scratch to take advantage of features in a newer FS version or what have you.

Thanks for the help ? spent a lot of time looking for this previously, but never asked on the list.
 
--
____
|| \\UTGERS,  	 |---------------------------*O*---------------------------
||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
     `'

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

From janfrode at tanso.net  Mon Jul  1 08:56:22 2019
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Mon, 1 Jul 2019 09:56:22 +0200
Subject: [gpfsug-discuss] Any guidelines for choosing vdisk size?
In-Reply-To: <E55F8C7F-48A4-4A66-ABF8-880A5232F39D@oarc.rutgers.edu>
References: <E55F8C7F-48A4-4A66-ABF8-880A5232F39D@oarc.rutgers.edu>
Message-ID: <CAHwPatihe-0esH-JbOzfENS65RvjY9ViQKvEyPnXr1+qhv_yqQ@mail.gmail.com>

I would mainly consider future upgrades. F.ex. do one vdisk per disk shelf
per rg. F.ex. for a GL6S you would have 12 vdisks, and if you add a GL4S
you would add 8 more vdisks, then each spindle of both systems should get
approximately the same number of IOs.

Another thing to consider is re-allocating capacity. If your vdisks are
very large, it might be difficult to free up a full vdisk if you need to
reorganize and create new filesystems, or add more 3way replicated
metadata..

But mainly I create large vdisks.. haven?t been able to show any
performance difference between one vdisk per RG vs. 10 vdisks per RG.


  -jf

man. 1. jul. 2019 kl. 07:42 skrev Ryan Novosielski <novosirj at rutgers.edu>:

> Good morning,
>
> Was wondering if anyone could point me to any tips or provide some
> regarding choosing vdisk size? My understanding is that too small is a
> waste of resources in the form of overhead, and that there is an upper
> limit. but that generally within a pool, you want them to be the same size,
> so that if you aren?t allocating the entire storage space on the system
> straight off, you?ll need to choose a reasonable size or be left with
> unusable space (eg. you will waste space if you went with, say, 200 GB
> vdisk sizes on a 500 GB array).
>
> Anyone have any tips? Do people just generally allocate the whole thing
> and have one 2 vdisks (redundancy)? Seems you have some more flexibility if
> you don?t do that and have to, say, create a storage pool or filesystem
> from scratch to take advantage of features in a newer FS version or what
> have you.
>
> Thanks for the help ? spent a lot of time looking for this previously, but
> never asked on the list.
>
> --
> ____
> || \\UTGERS,     |---------------------------*O*---------------------------
> ||_// the State  |         Ryan Novosielski - novosirj at rutgers.edu
> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
> ||  \\    of NJ  | Office of Advanced Research Computing - MSB C630, Newark
>      `'
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190701/9a8c578e/attachment.htm>

From abeattie at au1.ibm.com  Mon Jul  1 09:12:41 2019
From: abeattie at au1.ibm.com (Andrew Beattie)
Date: Mon, 1 Jul 2019 08:12:41 +0000
Subject: [gpfsug-discuss] Any guidelines for choosing vdisk size?
In-Reply-To: <D599D062-6286-477C-B494-4AD0984691AD@rutgers.edu>
References: <D599D062-6286-477C-B494-4AD0984691AD@rutgers.edu>,
	<E55F8C7F-48A4-4A66-ABF8-880A5232F39D@oarc.rutgers.edu><OF54C435D4.B3E41DA2-ON0025842A.001FD56B-0025842A.002077AE@notes.na.collabserv.com>
Message-ID: <OF10ACC789.BB227E77-ON0025842A.002C56F8-0025842A.002D1B7B@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190701/2e915488/attachment.htm>

From UWEFALKE at de.ibm.com  Mon Jul  1 09:29:49 2019
From: UWEFALKE at de.ibm.com (Uwe Falke)
Date: Mon, 1 Jul 2019 10:29:49 +0200
Subject: [gpfsug-discuss] Any guidelines for choosing vdisk size?
In-Reply-To: <TY2PR03MB446249F071E9252F75D87E5ACCF90@TY2PR03MB4462.apcprd03.prod.outlook.com>
References: <TY2PR03MB446249F071E9252F75D87E5ACCF90@TY2PR03MB4462.apcprd03.prod.outlook.com>
Message-ID: <OF01F0AAB3.D10E313F-ONC125842A.002D8C69-C125842A.002EAD36@notes.na.collabserv.com>

gpfsug-discuss-bounces at spectrumscale.org wrote on 01/07/2019 09:25:23:

> From: Michael Hennecke <mhennecke at lenovo.com>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Date: 01/07/2019 09:38
> Subject: [EXTERNAL] Re: [gpfsug-discuss] Any guidelines for choosing
> vdisk size?
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
...
> Either way, it is a good idea (in environments with more than a 
> single building block) to put aside a small "test" vdisk per RG just
> for performance testing. So you can do performance validations on 
> each building block individually even if your main filesystem spans 
> all building block. Size of that "test" vdisk should be big enough 
> so you can run a few minutes of sequential IOR against it without 
> filling it up 100%.

However, be aware that GNR limits the number of ptracks per vdisk 
according to the vdisk size. That could limit the seen performance 
especially for writes. 
I've experienced that with vdisk sizes of 509GiB in DAs of 288TiB of ESS 
GL4 (can't recall exactly if ptracks are assigned by absolute or by 
relative vdisk size). 


 
Mit freundlichen Gr??en / Kind regards

 
Dr. Uwe Falke
 
IT Specialist
High Performance Computing Services / Integrated Technology Services / 
Data Center Services
-------------------------------------------------------------------------------------------------------------------------------------------
IBM Deutschland
Rathausstr. 7
09111 Chemnitz
Phone: +49 371 6978 2165
Mobile: +49 175 575 2877
E-Mail: uwefalke at de.ibm.com
-------------------------------------------------------------------------------------------------------------------------------------------
IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: 
Thomas Wolter, Sven Schoo?
Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, 
HRB 17122 


gpfsug-discuss-bounces at spectrumscale.org wrote on 01/07/2019 09:25:23:

> From: Michael Hennecke <mhennecke at lenovo.com>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Date: 01/07/2019 09:38
> Subject: [EXTERNAL] Re: [gpfsug-discuss] Any guidelines for choosing
> vdisk size?
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> 
> Ryan,
> 
> That is a question with many variables. Some of the factors 
> influencing the decision are:
> * Are you sharing data and metadata (the GNR default these days), or
> are you building separate data and metadata vdisks (the "older" 
> default, and still useful for some use cases)? If you have separate 
> metadata, you need to ensure you leave enough free space in the RGs 
> to account for metadata growth beyond your initial sizing.
> * Are you creating vdisks for multiple filesystems on the same 
> building block, or just for one filesystem? 
> * How likely is it that space needs to be (re-)allocated between 
> different filesystems?
> * How likely is it that new storage will be added to existing 
filesystems?
> * Is there a requirement to include building blocks of different 
> size (different #disks, or different capacity of disks) in the same 
> filesystem?
> 
> All of these influence the decision on the "best" vdisk quantity and
> capacities. If you have a specific configuration question, I'm happy
> to discuss that offline.
> 
> In general I advise customers to use at least two equally sized 
> vdisks per RG (4 per building block), so there is more flexibility 
> when capacity needs to be moved around. But many sites that have 
> homogeneous building blocks and don't expect changes in their 
> storage configuration just go with a single vdisk per RG (2 per 
> building block).
> 
> Either way, it is a good idea (in environments with more than a 
> single building block) to put aside a small "test" vdisk per RG just
> for performance testing. So you can do performance validations on 
> each building block individually even if your main filesystem spans 
> all building block. Size of that "test" vdisk should be big enough 
> so you can run a few minutes of sequential IOR against it without 
> filling it up 100%.
> 
> 
> Mit freundlichen Gr?ssen / Best regards,
> 
> Michael Hennecke
> HPC Chief Technologist - HPC and AI Business Unit 
> mailto:mhennecke at lenovo.com
> --
> Lenovo Global Technology (Germany) GmbH * Am Zehnthof 77 * D-45307 
> Essen * Germany 
> Gesch?ftsf?hrung: Colm Gleeson, Christophe Laurent * Sitz der 
> Gesellschaft: Stuttgart * HRB-Nr.: 758298, AG Stuttgart
> 
> -----Original Message-----
> From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-
> bounces at spectrumscale.org> On Behalf Of Ryan Novosielski
> Sent: Monday, 1 July, 2019 07:43
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: [External] [gpfsug-discuss] Any guidelines for choosing vdisk 
size?
> 
> Good morning,
> 
> Was wondering if anyone could point me to any tips or provide some 
> regarding choosing vdisk size? My understanding is that too small is
> a waste of resources in the form of overhead, and that there is an 
> upper limit. but that generally within a pool, you want them to be 
> the same size, so that if you aren?t allocating the entire storage 
> space on the system straight off, you?ll need to choose a reasonable
> size or be left with unusable space (eg. you will waste space if you
> went with, say, 200 GB vdisk sizes on a 500 GB array).
> 
> Anyone have any tips? Do people just generally allocate the whole 
> thing and have one 2 vdisks (redundancy)? Seems you have some more 
> flexibility if you don?t do that and have to, say, create a storage 
> pool or filesystem from scratch to take advantage of features in a 
> newer FS version or what have you.
> 
> Thanks for the help ? spent a lot of time looking for this 
> previously, but never asked on the list.
> 
> --
> ____
> || \\UTGERS, |---------------------------*O*---------------------------
> ||_// the State    |         Ryan Novosielski - novosirj at rutgers.edu
> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS 
Campus
> ||  \\    of NJ    | Office of Advanced Research Computing - MSB C630, 
Newark
>      `'
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?
> 
u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-
> 
siA1ZOg&r=fTuVGtgq6A14KiNeaGfNZzOOgtHW5Lm4crZU6lJxtB8&m=XHrjWG2mnps3wxUISQHIZDR08HKkIKdcekRQibivZB4&s=wO1NamNaVBSnk_Kbe7Xl45aGtu7W6CBj0tHOlysTui4&e=
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?
> 
u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-
> 
siA1ZOg&r=fTuVGtgq6A14KiNeaGfNZzOOgtHW5Lm4crZU6lJxtB8&m=XHrjWG2mnps3wxUISQHIZDR08HKkIKdcekRQibivZB4&s=wO1NamNaVBSnk_Kbe7Xl45aGtu7W6CBj0tHOlysTui4&e=
> 




From kkr at lbl.gov  Tue Jul  2 18:45:30 2019
From: kkr at lbl.gov (Kristy Kallback-Rose)
Date: Tue, 2 Jul 2019 10:45:30 -0700
Subject: [gpfsug-discuss] Hold the Date - September 23 and 24
Message-ID: <3F2B08E9-C6E3-412B-9308-D79E3480C5DA@lbl.gov>

Hello,

HPCXXL will be hosted by NERSC (Berkeley, CA) this September. As part of this event, there will be approximately a day and a half on GPFS content. We have done this type of event in the past, and as before, the GPFS days will be free to attend, but you do need to register. 

We?ll have more details soon, mark your calendars. 

Initial details: https://www.spectrumscaleug.org/event/spectrum-scale-gpfs-days-part-of-hpcxxl/ <https://www.spectrumscaleug.org/event/spectrum-scale-gpfs-days-part-of-hpcxxl/>

Best,
Kristy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190702/9747f62d/attachment.htm>

From novosirj at rutgers.edu  Thu Jul  4 04:55:56 2019
From: novosirj at rutgers.edu (Ryan Novosielski)
Date: Thu, 4 Jul 2019 03:55:56 +0000
Subject: [gpfsug-discuss] Combined system (metadata) and data pool on GPFS
	5.x?
Message-ID: <DCA32300-AA3F-4E63-8248-5A241F6D4E4C@rutgers.edu>

Hi there,

Was at the Lenovo GNR/GPFS 5.x seminar the other week and we briefly discussed whether or not the recommendation was to place data in the system pool (eg. have one combined system/data pool), or to have them separate. We?ve got a GSS26 that we are creating new filesystems on for high speed scratch (16 MB blocks).

I unfortunately did not note all of the considerations/caveats related to this choice. Does anyone who knows/was there and can remember remind me?

The recollection I have is that you may have some contention for I/O if your metadata is on your data disks, and you won?t be able to have different redundancy settings. Once upon a time, block size also mattered, but not so much on 5.x.

Thanks!

--
____
|| \\UTGERS,  	 |---------------------------*O*---------------------------
||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
     `'


From S.J.Thompson at bham.ac.uk  Thu Jul  4 09:51:18 2019
From: S.J.Thompson at bham.ac.uk (Simon Thompson)
Date: Thu, 4 Jul 2019 08:51:18 +0000
Subject: [gpfsug-discuss] Combined system (metadata) and data pool on
 GPFS 5.x?
In-Reply-To: <DCA32300-AA3F-4E63-8248-5A241F6D4E4C@rutgers.edu>
References: <DCA32300-AA3F-4E63-8248-5A241F6D4E4C@rutgers.edu>
Message-ID: <205352CF-6035-4DDA-812F-9BD9FA0152B7@bham.ac.uk>

I wasn't but ...

If your metadata is in the same RG/DA as your data, even if it?s a separate pool, the contention is the same. But if you have 2 DAs, then you have more spinning disks underneath to provide iops anyway.

We only separate our metadata and data into pools because we want to replicate the metadata between DSS-G systems, but not data. And also have some data which is replicated but not all, and the only way we could figure to do that is by having separate pools and failure groups. i.e. we didn't want data in the system pool that wasn't replicated being placed on the second site.

And I agree, block sizes don't matter so much anymore, except remember the "variable" sub-blocks are based on the smallest block size used, so we ended up with 16MB blocks for both data and metadata.

Simon

?On 04/07/2019, 04:56, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Ryan Novosielski" <gpfsug-discuss-bounces at spectrumscale.org on behalf of novosirj at rutgers.edu> wrote:

    Hi there,
    
    Was at the Lenovo GNR/GPFS 5.x seminar the other week and we briefly discussed whether or not the recommendation was to place data in the system pool (eg. have one combined system/data pool), or to have them separate. We?ve got a GSS26 that we are creating new filesystems on for high speed scratch (16 MB blocks).
    
    I unfortunately did not note all of the considerations/caveats related to this choice. Does anyone who knows/was there and can remember remind me?
    
    The recollection I have is that you may have some contention for I/O if your metadata is on your data disks, and you won?t be able to have different redundancy settings. Once upon a time, block size also mattered, but not so much on 5.x.
    
    Thanks!
    
    --
    ____
    || \\UTGERS,  	 |---------------------------*O*---------------------------
    ||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
    || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
    ||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
         `'
    
    _______________________________________________
    gpfsug-discuss mailing list
    gpfsug-discuss at spectrumscale.org
    http://gpfsug.org/mailman/listinfo/gpfsug-discuss
    


From damir.krstic at gmail.com  Wed Jul 10 13:16:31 2019
From: damir.krstic at gmail.com (Damir Krstic)
Date: Wed, 10 Jul 2019 07:16:31 -0500
Subject: [gpfsug-discuss] slow filesystem
Message-ID: <CAKV+WqctLVWdr0E5sA+G9pZpc9F16Pc7Q-VT1C81zf6R8gkGmA@mail.gmail.com>

Over last couple of days our reads and writes on our compute cluster are
experiencing real slow reads and writes on one of the filesystems in the
cluster. We are talking with IBM and have Sev. 1 ticket open, but I figured
to ask here about the warning message we are seeing in GPFS logs.

The cluster is configured as following:

4 IO servers in the main gpfs cluster
700+ compute nodes in the gpfs cluster

home filesystem is slow but projects filesystem seems to be fast. Not many
waiters on the IO servers (almost none) but a lot of waiters on the remote
cluster.

The message that is giving us a pause is the following:

Jul 10 07:05:31 gssio4 mmfs: [N] Writing into file
/var/mmfs/gen/LastLeaseRequestSent took 10.5 seconds

Why is taking so long to write to to the local file?

Thanks,
Damir
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190710/5aa0023a/attachment.htm>

From Kevin.Buterbaugh at Vanderbilt.Edu  Wed Jul 10 13:21:52 2019
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Wed, 10 Jul 2019 12:21:52 +0000
Subject: [gpfsug-discuss] slow filesystem
In-Reply-To: <CAKV+WqctLVWdr0E5sA+G9pZpc9F16Pc7Q-VT1C81zf6R8gkGmA@mail.gmail.com>
References: <CAKV+WqctLVWdr0E5sA+G9pZpc9F16Pc7Q-VT1C81zf6R8gkGmA@mail.gmail.com>
Message-ID: <BEF9A8D8-DFFE-4102-8ADF-6A919F9A5094@vanderbilt.edu>

Hi Damir,

Have you checked to see whether gssio4 might have a failing internal HD / SSD?  Thanks?

Kevin

On Jul 10, 2019, at 7:16 AM, Damir Krstic <damir.krstic at gmail.com<mailto:damir.krstic at gmail.com>> wrote:

Over last couple of days our reads and writes on our compute cluster are experiencing real slow reads and writes on one of the filesystems in the cluster. We are talking with IBM and have Sev. 1 ticket open, but I figured to ask here about the warning message we are seeing in GPFS logs.

The cluster is configured as following:

4 IO servers in the main gpfs cluster
700+ compute nodes in the gpfs cluster

home filesystem is slow but projects filesystem seems to be fast. Not many waiters on the IO servers (almost none) but a lot of waiters on the remote cluster.

The message that is giving us a pause is the following:
Jul 10 07:05:31 gssio4 mmfs: [N] Writing into file /var/mmfs/gen/LastLeaseRequestSent took 10.5 seconds

Why is taking so long to write to to the local file?

Thanks,
Damir

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&amp;data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cede76a6c2bd743c836b708d705307a86%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636983578133164831&amp;sdata=ATtqXDDChaZTouZZRkjf%2F79pIK%2Fc1DAwq6KUU%2FKYca4%3D&amp;reserved=0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190710/014c4ff6/attachment.htm>

From damir.krstic at gmail.com  Wed Jul 10 13:22:37 2019
From: damir.krstic at gmail.com (Damir Krstic)
Date: Wed, 10 Jul 2019 07:22:37 -0500
Subject: [gpfsug-discuss] slow filesystem
In-Reply-To: <BEF9A8D8-DFFE-4102-8ADF-6A919F9A5094@vanderbilt.edu>
References: <CAKV+WqctLVWdr0E5sA+G9pZpc9F16Pc7Q-VT1C81zf6R8gkGmA@mail.gmail.com>
	<BEF9A8D8-DFFE-4102-8ADF-6A919F9A5094@vanderbilt.edu>
Message-ID: <CAKV+WqfUQ3wWaKqOB-aVKDRNsvw7SutWT64owkCJvpkndWtWUA@mail.gmail.com>

mmlspdisk all --not-ok does not indicate any failed hard drives.

On Wed, Jul 10, 2019 at 7:22 AM Buterbaugh, Kevin L <
Kevin.Buterbaugh at vanderbilt.edu> wrote:

> Hi Damir,
>
> Have you checked to see whether gssio4 might have a failing internal HD /
> SSD?  Thanks?
>
> Kevin
>
> On Jul 10, 2019, at 7:16 AM, Damir Krstic <damir.krstic at gmail.com> wrote:
>
> Over last couple of days our reads and writes on our compute cluster are
> experiencing real slow reads and writes on one of the filesystems in the
> cluster. We are talking with IBM and have Sev. 1 ticket open, but I figured
> to ask here about the warning message we are seeing in GPFS logs.
>
> The cluster is configured as following:
>
> 4 IO servers in the main gpfs cluster
> 700+ compute nodes in the gpfs cluster
>
> home filesystem is slow but projects filesystem seems to be fast. Not many
> waiters on the IO servers (almost none) but a lot of waiters on the remote
> cluster.
>
> The message that is giving us a pause is the following:
> Jul 10 07:05:31 gssio4 mmfs: [N] Writing into file
> /var/mmfs/gen/LastLeaseRequestSent took 10.5 seconds
>
> Why is taking so long to write to to the local file?
>
> Thanks,
> Damir
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
>
> https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&amp;data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cede76a6c2bd743c836b708d705307a86%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636983578133164831&amp;sdata=ATtqXDDChaZTouZZRkjf%2F79pIK%2Fc1DAwq6KUU%2FKYca4%3D&amp;reserved=0
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190710/dac7390e/attachment.htm>

From valdis.kletnieks at vt.edu  Wed Jul 10 14:14:31 2019
From: valdis.kletnieks at vt.edu (Valdis Kl=?utf-8?Q?=c4=93?=tnieks)
Date: Wed, 10 Jul 2019 09:14:31 -0400
Subject: [gpfsug-discuss] slow filesystem
In-Reply-To: <CAKV+WqfUQ3wWaKqOB-aVKDRNsvw7SutWT64owkCJvpkndWtWUA@mail.gmail.com>
References: <CAKV+WqctLVWdr0E5sA+G9pZpc9F16Pc7Q-VT1C81zf6R8gkGmA@mail.gmail.com>
	<BEF9A8D8-DFFE-4102-8ADF-6A919F9A5094@vanderbilt.edu>
	<CAKV+WqfUQ3wWaKqOB-aVKDRNsvw7SutWT64owkCJvpkndWtWUA@mail.gmail.com>
Message-ID: <5311.1562764471@turing-police>

On Wed, 10 Jul 2019 07:22:37 -0500, Damir Krstic said:

> mmlspdisk all --not-ok does not indicate any failed hard drives.

Not a GPFS drive - if a write to /var is taking 10 seconds, that indicates
that there is likely a problem with the disk(s) that the system lives on,
and you're hitting recoverable write errors....

Jul 10 07:05:31 gssio4 mmfs: [N] Writing into file /var/mmfs/gen/LastLeaseRequestSent took 10.5 seconds

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190710/f532fd53/attachment.sig>

From heinrich.billich at id.ethz.ch  Wed Jul 17 12:41:54 2019
From: heinrich.billich at id.ethz.ch (Billich  Heinrich Rainer (ID SD))
Date: Wed, 17 Jul 2019 11:41:54 +0000
Subject: [gpfsug-discuss] How to prove that data is in inode
Message-ID: <71647F14-8BEC-4E89-A30B-F0D880A81A84@id.ethz.ch>

Hello,

How can I prove that data of a small file is stored in the inode (and not on a data nsd)?

We have a filesystem with 4k inodes on Scale 5.0.2 , but it seems there is no file data in the inodes?

I would expect that 'stat' reports 'Blocks: 0'  for a small file, but I see 'Blocks:1'.

Cheers,

Heiner

I tried

[]# rm -f test; echo hello > test
[]# ls -ls test
1 -rw-r--r-- 1 root root 6 Jul 17 13:11 test
[root at testnas13ems01 test]# stat test
  File: ?test?
  Size: 6         	Blocks: 1          IO Block: 1048576 regular file
Device: 2dh/45d	Inode: 353314      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-07-17 13:11:03.037049000 +0200
Modify: 2019-07-17 13:11:03.037331000 +0200
Change: 2019-07-17 13:11:03.037259319 +0200
 Birth: -
[root at testnas13ems01 test]# du test
1	test
[root at testnas13ems01 test]# du -b test
6	test
[root at testnas13ems01 test]#

Filesystem

# mmlsfs f****
flag                value                    description
------------------- ------------------------ -----------------------------------
 -f                 32768                    Minimum fragment (subblock) size in bytes
 -i                 4096                     Inode size in bytes
 -I                 32768                    Indirect block size in bytes
 -m                 1                        Default number of metadata replicas
 -M                 2                        Maximum number of metadata replicas
 -r                 1                        Default number of data replicas
 -R                 2                        Maximum number of data replicas
 -j                 cluster                  Block allocation type
 -D                 nfs4                     File locking semantics in effect
 -k                 nfs4                     ACL semantics in effect
 -n                 32                       Estimated number of nodes that will mount file system
 -B                 1048576                  Block size
 -Q                 user;group;fileset       Quotas accounting enabled
                    user;group;fileset       Quotas enforced
                    user;group;fileset       Default quotas enabled
 --perfileset-quota Yes                      Per-fileset quota enforcement
 --filesetdf        Yes                      Fileset df enabled?
 -V                 20.01 (5.0.2.0)          Current file system version
                    15.01 (4.2.0.0)          Original file system version
 --create-time      ***** 2017 File system creation time
 -z                 No                       Is DMAPI enabled?
 -L                 33554432                 Logfile size
 -E                 Yes                      Exact mtime mount option
 -S                 relatime                 Suppress atime mount option
 -K                 whenpossible             Strict replica allocation option
 --fastea           Yes                      Fast external attributes enabled?
 --encryption       No                       Encryption enabled?
 --inode-limit      1294592                  Maximum number of inodes in all inode spaces
 --log-replicas     0                        Number of log replicas
 --is4KAligned      Yes                      is4KAligned?
 --rapid-repair     Yes                      rapidRepair enabled?
 --write-cache-threshold 0                   HAWC Threshold (max 65536)
 --subblocks-per-full-block 32               Number of subblocks per full block
 -P                 system;data              Disk storage pools in file system
 --file-audit-log   No                       File Audit Logging enabled?
 --maintenance-mode No                       Maintenance Mode enabled?
 -d                 ******
-A                 yes                      Automatic mount option
 -o                 nfssync,nodev            Additional mount options
 -T                 /****                  Default mount point
 --mount-priority   0                        Mount priority

-- 
=======================
Heinrich Billich
ETH Z?rich
Informatikdienste
Tel.: +41 44 632 72 56
heinrich.billich at id.ethz.ch
========================
 
 


From kums at us.ibm.com  Wed Jul 17 13:37:35 2019
From: kums at us.ibm.com (Kumaran Rajaram)
Date: Wed, 17 Jul 2019 08:37:35 -0400
Subject: [gpfsug-discuss] How to prove that data is in inode
In-Reply-To: <71647F14-8BEC-4E89-A30B-F0D880A81A84@id.ethz.ch>
References: <71647F14-8BEC-4E89-A30B-F0D880A81A84@id.ethz.ch>
Message-ID: <OF776130C0.2EA026CD-ON0025843A.0044FED5-8525843A.00455C19@notes.na.collabserv.com>

Hi,

>> How can I prove that data of a small file is stored in the inode (and
not on a data nsd)?

You may use echo "inode file_inode_number" | tsdbfs fs_device | grep
indirectionLevel and if it points to INODE, then the file is stored in the
inodes

# 4K Inode Size
# mmlsfs gpfs3a | grep 'Inode size'
 -i                 4096                     Inode size in bytes

# Small file
# ls -l /mnt/gpfs3a/hello.txt
-rw-r--r-- 1 root root 6 Jul 17 08:32 /mnt/gpfs3a/hello.txt

# ls -i /mnt/gpfs3a/hello.txt
91649 /mnt/gpfs3a/hello.txt

#File is inlined within Inode
# echo "inode 91649" | tsdbfs gpfs3a | grep indirectionLevel
  indirectionLevel=INODE status=USERFILE

Regards,
-Kums





From:	"Billich  Heinrich Rainer (ID SD)"
            <heinrich.billich at id.ethz.ch>
To:	gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:	07/17/2019 07:49 AM
Subject:	[EXTERNAL] [gpfsug-discuss] How to prove that data is in inode
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



Hello,

How can I prove that data of a small file is stored in the inode (and not
on a data nsd)?

We have a filesystem with 4k inodes on Scale 5.0.2 , but it seems there is
no file data in the inodes?

I would expect that 'stat' reports 'Blocks: 0'  for a small file, but I see
'Blocks:1'.

Cheers,

Heiner

I tried

[]# rm -f test; echo hello > test
[]# ls -ls test
1 -rw-r--r-- 1 root root 6 Jul 17 13:11 test
[root at testnas13ems01 test]# stat test
  File: ?test?
  Size: 6         		 Blocks: 1          IO Block: 1048576 regular
file
Device: 2dh/45d		 Inode: 353314      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-07-17 13:11:03.037049000 +0200
Modify: 2019-07-17 13:11:03.037331000 +0200
Change: 2019-07-17 13:11:03.037259319 +0200
 Birth: -
[root at testnas13ems01 test]# du test
1		 test
[root at testnas13ems01 test]# du -b test
6		 test
[root at testnas13ems01 test]#

Filesystem

# mmlsfs f****
flag                value                    description
------------------- ------------------------
-----------------------------------
 -f                 32768                    Minimum fragment (subblock)
size in bytes
 -i                 4096                     Inode size in bytes
 -I                 32768                    Indirect block size in bytes
 -m                 1                        Default number of metadata
replicas
 -M                 2                        Maximum number of metadata
replicas
 -r                 1                        Default number of data
replicas
 -R                 2                        Maximum number of data
replicas
 -j                 cluster                  Block allocation type
 -D                 nfs4                     File locking semantics in
effect
 -k                 nfs4                     ACL semantics in effect
 -n                 32                       Estimated number of nodes that
will mount file system
 -B                 1048576                  Block size
 -Q                 user;group;fileset       Quotas accounting enabled
                    user;group;fileset       Quotas enforced
                    user;group;fileset       Default quotas enabled
 --perfileset-quota Yes                      Per-fileset quota enforcement
 --filesetdf        Yes                      Fileset df enabled?
 -V                 20.01 (5.0.2.0)          Current file system version
                    15.01 (4.2.0.0)          Original file system version
 --create-time      ***** 2017 File system creation time
 -z                 No                       Is DMAPI enabled?
 -L                 33554432                 Logfile size
 -E                 Yes                      Exact mtime mount option
 -S                 relatime                 Suppress atime mount option
 -K                 whenpossible             Strict replica allocation
option
 --fastea           Yes                      Fast external attributes
enabled?
 --encryption       No                       Encryption enabled?
 --inode-limit      1294592                  Maximum number of inodes in
all inode spaces
 --log-replicas     0                        Number of log replicas
 --is4KAligned      Yes                      is4KAligned?
 --rapid-repair     Yes                      rapidRepair enabled?
 --write-cache-threshold 0                   HAWC Threshold (max 65536)
 --subblocks-per-full-block 32               Number of subblocks per full
block
 -P                 system;data              Disk storage pools in file
system
 --file-audit-log   No                       File Audit Logging enabled?
 --maintenance-mode No                       Maintenance Mode enabled?
 -d                 ******
-A                 yes                      Automatic mount option
 -o                 nfssync,nodev            Additional mount options
 -T                 /****                  Default mount point
 --mount-priority   0                        Mount priority

--
=======================
Heinrich Billich
ETH Z?rich
Informatikdienste
Tel.: +41 44 632 72 56
heinrich.billich at id.ethz.ch
========================



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=McIf98wfiVqHU8ZygezLrQ&m=VUhXK3bFtfQfiPegK_nm2SStc9jabkxIAUQruQxyXdI&s=JobCqlXTY87VyE6RxErKI46SdzyDhW5kvZKc9xw6DHM&e=




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190717/da506a92/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190717/da506a92/attachment.gif>

From p.ward at nhm.ac.uk  Wed Jul 17 14:00:16 2019
From: p.ward at nhm.ac.uk (Paul Ward)
Date: Wed, 17 Jul 2019 13:00:16 +0000
Subject: [gpfsug-discuss] Break-out session for new user and
	prospects	[London Usergroup]
In-Reply-To: <776770B2-5F84-4462-B900-58EBB982DC1C@spectrumscale.org>
References: <776770B2-5F84-4462-B900-58EBB982DC1C@spectrumscale.org>
Message-ID: <DB7PR10MB2201EAAA629BB9C10BBE24E0D3C90@DB7PR10MB2201.EURPRD10.PROD.OUTLOOK.COM>

Hi Simon,

Was this a slide presentation? Is it available?

Kindest regards,
Paul

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (Spectrum Scale User Group Chair)
Sent: 30 April 2019 10:25
To: gpfsug-discuss at spectrumscale.org
Subject: [gpfsug-discuss] Break-out session for new user and prospects [London Usergroup]

Hi all,

We know that a lot of the talks at the user groups are for experienced users, following feedback from the USA user group, we thought we?d advertise that this year we?re planning to run a break-out for new users on day 1.

Break-out session for new user and prospects (Wed May 8th, 13:00 - 16:45)

This year we will offer a break-out session for new Spectrum Scale user and prospects to get started with Spectrum Scale. In this session we will cover Spectrum Scale Use Cases, the architecture of a Spectrum Scale environment, and discuss how the manifold Spectrum Scale features support the different use case. Please inform customers and colleagues who are interested to learn about Spectrum Scale to grab one of the last seats.

Registration link:
https://www.spectrumscaleug.org/event/uk-user-group-meeting/

There?s just a couple of places left for the usergroup, so please do share and register if you plan to attend.

Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190717/b7294b4c/attachment.htm>

From S.J.Thompson at bham.ac.uk  Wed Jul 17 14:10:21 2019
From: S.J.Thompson at bham.ac.uk (Simon Thompson)
Date: Wed, 17 Jul 2019 13:10:21 +0000
Subject: [gpfsug-discuss] Break-out session for new user and prospects
 [London Usergroup]
In-Reply-To: <DB7PR10MB2201EAAA629BB9C10BBE24E0D3C90@DB7PR10MB2201.EURPRD10.PROD.OUTLOOK.COM>
References: <776770B2-5F84-4462-B900-58EBB982DC1C@spectrumscale.org>
	<DB7PR10MB2201EAAA629BB9C10BBE24E0D3C90@DB7PR10MB2201.EURPRD10.PROD.OUTLOOK.COM>
Message-ID: <6916EC29-2057-4396-A285-7A905DB215EE@bham.ac.uk>

Hi Paul,

There was a slide deck, I?ve asked the IBMers if it?s something we can share.

Simon

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Paul Ward <p.ward at nhm.ac.uk>
Reply-To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Date: Wednesday, 17 July 2019 at 14:01
To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Break-out session for new user and prospects [London Usergroup]

Hi Simon,

Was this a slide presentation? Is it available?

Kindest regards,
Paul

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (Spectrum Scale User Group Chair)
Sent: 30 April 2019 10:25
To: gpfsug-discuss at spectrumscale.org
Subject: [gpfsug-discuss] Break-out session for new user and prospects [London Usergroup]

Hi all,

We know that a lot of the talks at the user groups are for experienced users, following feedback from the USA user group, we thought we?d advertise that this year we?re planning to run a break-out for new users on day 1.

Break-out session for new user and prospects (Wed May 8th, 13:00 - 16:45)

This year we will offer a break-out session for new Spectrum Scale user and prospects to get started with Spectrum Scale. In this session we will cover Spectrum Scale Use Cases, the architecture of a Spectrum Scale environment, and discuss how the manifold Spectrum Scale features support the different use case. Please inform customers and colleagues who are interested to learn about Spectrum Scale to grab one of the last seats.

Registration link:
https://www.spectrumscaleug.org/event/uk-user-group-meeting/

There?s just a couple of places left for the usergroup, so please do share and register if you plan to attend.

Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190717/386a9e5b/attachment.htm>

From leonardo.sala at psi.ch  Wed Jul 17 15:01:06 2019
From: leonardo.sala at psi.ch (Leonardo Sala)
Date: Wed, 17 Jul 2019 16:01:06 +0200
Subject: [gpfsug-discuss] Mistakenly AFM resync triggered: should I stop it?
Message-ID: <e69518bc-d525-fa95-d271-e3e8e81ebcf5@psi.ch>

Dear all,

I do have a possibly stupid question, based on something stupid I did. 
During a procedure to recover AFM transfers (which stopped for still to 
be investigated reasons), I triggered by mistake a resync operation on 
an healthy SW fileset (state: Inactive). The AFM cache in this case is 
big, so this has triggered a big backlog and quite some network 
activity. afmPrefetchThreshold is set to 0, so I am not too scared about 
partially zero-filled data files. But what would it be the best course 
of action now? Should I just let it run (ETA: ~20 hours, give or take), 
or try to stop it? Also, given that this is some experimental data, what 
would the safest course of action?

Thanks a lot!

cheers

leo


-- 
Paul Scherrer Institut
Dr. Leonardo Sala
Group Leader High Performance Computing
Deputy Section Head Science IT
Science IT
WHGA/106
5232 Villigen PSI
Switzerland

Phone: +41 56 310 3369
leonardo.sala at psi.ch
www.psi.ch

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190717/fbaaae4f/attachment.htm>

From bbanister at jumptrading.com  Wed Jul 17 18:24:12 2019
From: bbanister at jumptrading.com (Bryan Banister)
Date: Wed, 17 Jul 2019 17:24:12 +0000
Subject: [gpfsug-discuss] GPFS quota inode/files maximum setting of
	2147483647
Message-ID: <DM6PR14MB2890B7F5B0283B4444C68BAEA2C90@DM6PR14MB2890.namprd14.prod.outlook.com>

Thought this might be of interest to others on the list.

IBM, are there any plans to increase the maximum files setting for quotas over 2147483647?

Thanks,
-Bryan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190717/e373b695/attachment.htm>

From vpuvvada at in.ibm.com  Thu Jul 18 07:41:08 2019
From: vpuvvada at in.ibm.com (Venkateswara R Puvvada)
Date: Thu, 18 Jul 2019 12:11:08 +0530
Subject: [gpfsug-discuss] Mistakenly AFM resync triggered: should I stop
	it?
In-Reply-To: <e69518bc-d525-fa95-d271-e3e8e81ebcf5@psi.ch>
References: <e69518bc-d525-fa95-d271-e3e8e81ebcf5@psi.ch>
Message-ID: <OF11AFAFD0.1AB097A2-ON6525843B.0022AC38-6525843B.0024B9A2@notes.na.collabserv.com>

Hi,

You can stop the AFM resync if it is not required.

mmafmctl device dropPending -j fileset
mmafmctl device resetResync -j fileset

Next access to the fileset would run the recovery and impact should be 
minimal as it recovers only the data which was not replicated.

~Venkat (vpuvvada at in.ibm.com)



From:   Leonardo Sala <leonardo.sala at psi.ch>
To:     "gpfsug-discuss at spectrumscale.org" 
<gpfsug-discuss at spectrumscale.org>
Date:   07/17/2019 07:32 PM
Subject:        [EXTERNAL] [gpfsug-discuss] Mistakenly AFM resync 
triggered: should I stop it?
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Dear all,
I do have a possibly stupid question, based on something stupid I did. 
During a procedure to recover AFM transfers (which stopped for still to be 
investigated reasons), I triggered by mistake a resync operation on an 
healthy SW fileset (state: Inactive). The AFM cache in this case is big, 
so this has triggered a big backlog and quite some network activity. 
afmPrefetchThreshold is set to 0, so I am not too scared about partially 
zero-filled data files. But what would it be the best course of action 
now? Should I just let it run (ETA: ~20 hours, give or take), or try to 
stop it? Also, given that this is some experimental data, what would the 
safest course of action?
Thanks a lot!
cheers
leo

-- 
Paul Scherrer Institut
Dr. Leonardo Sala
Group Leader High Performance Computing
Deputy Section Head Science IT
Science IT
WHGA/106
5232 Villigen PSI
Switzerland

Phone: +41 56 310 3369
leonardo.sala at psi.ch
www.psi.ch_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=MapU9QQbtQx-G9hGAGBPzuB-Qxkl0HONn6ByIz4nXoI&s=F_iFVjqia2agf0fTNncWoaZqmKbhSCO-opqFb6cjG6A&e= 





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190718/748db732/attachment.htm>

From heinrich.billich at id.ethz.ch  Thu Jul 18 15:15:12 2019
From: heinrich.billich at id.ethz.ch (Billich  Heinrich Rainer (ID SD))
Date: Thu, 18 Jul 2019 14:15:12 +0000
Subject: [gpfsug-discuss] How to prove that data is in inode
In-Reply-To: <OF776130C0.2EA026CD-ON0025843A.0044FED5-8525843A.00455C19@notes.na.collabserv.com>
References: <71647F14-8BEC-4E89-A30B-F0D880A81A84@id.ethz.ch>
	<OF776130C0.2EA026CD-ON0025843A.0044FED5-8525843A.00455C19@notes.na.collabserv.com>
Message-ID: <A5096632-88B9-4A6B-9C22-E8DC0884987F@id.ethz.ch>

Hello Kums,

Thank you; I could verify that the data of small files is in the inode. In the table below you see the filesize and the result of the tsdbfs query. Up to 3k all data is in the inode.

The result of stat calls on small files is very puzzling. When data is in the inode stat reports one block of 512 bytes used, even for 3k of data. I don?t see how this would affect any of our applications, so it?s just something to note.

I append some output and the script to generate it, just for completeness.

Cheers,

Heiner

# stat -c "inode %i" $f | tsdbfs fs1301 | grep indirectionLevel
0  indirectionLevel=DIRECT status=USERFILE. <empty file is special>
1  indirectionLevel=INODE status=USERFILE
16  indirectionLevel=INODE status=USERFILE
512  indirectionLevel=INODE status=USERFILE
1k  indirectionLevel=INODE status=USERFILE
2k  indirectionLevel=INODE status=USERFILE
3k  indirectionLevel=INODE status=USERFILE.  < up to 3k data is in inode >

4k  indirectionLevel=DIRECT status=USERFILE
16k  indirectionLevel=DIRECT status=USERFILE
64k  indirectionLevel=DIRECT status=USERFILE
1M  indirectionLevel=DIRECT status=USERFILE
2M  indirectionLevel=DIRECT status=USERFILE


Stat output

# stat -c ?%n size: %s  allocated: %b*%B?

# stat -c %n size: %s  allocated: %b*%B
0 size: 0  allocated: 0*512
1 size: 1  allocated: 1*512
16 size: 16  allocated: 1*512
512 size: 512  allocated: 1*512
1k size: 1024  allocated: 1*512
2k size: 2048  allocated: 1*512
3k size: 3072  allocated: 1*512  < 3k file and  all data in inode:  stat reports 1*512 allocated/used)
4k size: 4096  allocated: 64*512
16k size: 16384  allocated: 64*512.   < as expected, 32k subblock size >
64k size: 65536  allocated: 128*512
1M size: 1048576  allocated: 2048*512
2M size: 2097152  allocated: 4096*512

The script:

# test-data-in-inode.sh
sizes="0 1 16 512 1k 2k 3k 4k 16k 64k 1M 2M"

echo create files
for s in $sizes
do
  head -c $s /dev/zero > $s
done

echo sleep 20. # give gpfs some time to update metadata
sleep 20

echo
echo "# ls -ls $sizes"
ls -ls $sizes

echo
echo "# stat -c %n size: %s  allocated: %b*%B"
stat -c "%n size: %s  allocated: %b*%B" $sizes

echo
echo '# stat -c "inode %i" $f | tsdbfs fs1301 | grep indirectionLevel'
for f in $sizes
do
echo -n $f
stat -c "inode %i" $f | tsdbfs fs1301 | grep indirectionLevel
done

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Kumaran Rajaram <kums at us.ibm.com>
Reply to: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Wednesday, 17 July 2019 at 14:38
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] How to prove that data is in inode


Hi,

>> How can I prove that data of a small file is stored in the inode (and not on a data nsd)?

You may use echo "inode file_inode_number" | tsdbfs fs_device | grep indirectionLevel and if it points to INODE, then the file is stored in the inodes

# 4K Inode Size
# mmlsfs gpfs3a | grep 'Inode size'
-i 4096 Inode size in bytes

# Small file
# ls -l /mnt/gpfs3a/hello.txt
-rw-r--r-- 1 root root 6 Jul 17 08:32 /mnt/gpfs3a/hello.txt

# ls -i /mnt/gpfs3a/hello.txt
91649 /mnt/gpfs3a/hello.txt

#File is inlined within Inode
# echo "inode 91649" | tsdbfs gpfs3a | grep indirectionLevel
indirectionLevel=INODE status=USERFILE

Regards,
-Kums



[Inactive hide details for "Billich  Heinrich Rainer (ID SD)" ---07/17/2019 07:49:56 AM---Hello, How can I prove that data of a]"Billich Heinrich Rainer (ID SD)" ---07/17/2019 07:49:56 AM---Hello, How can I prove that data of a small file is stored in the inode (and not on a data nsd)?

From: "Billich Heinrich Rainer (ID SD)" <heinrich.billich at id.ethz.ch>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: 07/17/2019 07:49 AM
Subject: [EXTERNAL] [gpfsug-discuss] How to prove that data is in inode
Sent by: gpfsug-discuss-bounces at spectrumscale.org

________________________________



Hello,

How can I prove that data of a small file is stored in the inode (and not on a data nsd)?

We have a filesystem with 4k inodes on Scale 5.0.2 , but it seems there is no file data in the inodes?

I would expect that 'stat' reports 'Blocks: 0'  for a small file, but I see 'Blocks:1'.

Cheers,

Heiner

I tried

[]# rm -f test; echo hello > test
[]# ls -ls test
1 -rw-r--r-- 1 root root 6 Jul 17 13:11 test
[root at testnas13ems01 test]# stat test
 File: ?test?
 Size: 6         Blocks: 1          IO Block: 1048576 regular file
Device: 2dh/45d Inode: 353314      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-07-17 13:11:03.037049000 +0200
Modify: 2019-07-17 13:11:03.037331000 +0200
Change: 2019-07-17 13:11:03.037259319 +0200
Birth: -
[root at testnas13ems01 test]# du test
1 test
[root at testnas13ems01 test]# du -b test
6 test
[root at testnas13ems01 test]#

Filesystem

# mmlsfs f****
flag                value                    description
------------------- ------------------------ -----------------------------------
-f                 32768                    Minimum fragment (subblock) size in bytes
-i                 4096                     Inode size in bytes
-I                 32768                    Indirect block size in bytes
-m                 1                        Default number of metadata replicas
-M                 2                        Maximum number of metadata replicas
-r                 1                        Default number of data replicas
-R                 2                        Maximum number of data replicas
-j                 cluster                  Block allocation type
-D                 nfs4                     File locking semantics in effect
-k                 nfs4                     ACL semantics in effect
-n                 32                       Estimated number of nodes that will mount file system
-B                 1048576                  Block size
-Q                 user;group;fileset       Quotas accounting enabled
                   user;group;fileset       Quotas enforced
                   user;group;fileset       Default quotas enabled
--perfileset-quota Yes                      Per-fileset quota enforcement
--filesetdf        Yes                      Fileset df enabled?
-V                 20.01 (5.0.2.0)          Current file system version
                   15.01 (4.2.0.0)          Original file system version
--create-time      ***** 2017 File system creation time
-z                 No                       Is DMAPI enabled?
-L                 33554432                 Logfile size
-E                 Yes                      Exact mtime mount option
-S                 relatime                 Suppress atime mount option
-K                 whenpossible             Strict replica allocation option
--fastea           Yes                      Fast external attributes enabled?
--encryption       No                       Encryption enabled?
--inode-limit      1294592                  Maximum number of inodes in all inode spaces
--log-replicas     0                        Number of log replicas
--is4KAligned      Yes                      is4KAligned?
--rapid-repair     Yes                      rapidRepair enabled?
--write-cache-threshold 0                   HAWC Threshold (max 65536)
--subblocks-per-full-block 32               Number of subblocks per full block
-P                 system;data              Disk storage pools in file system
--file-audit-log   No                       File Audit Logging enabled?
--maintenance-mode No                       Maintenance Mode enabled?
-d                 ******
-A                 yes                      Automatic mount option
-o                 nfssync,nodev            Additional mount options
-T                 /****                  Default mount point
--mount-priority   0                        Mount priority

--
=======================
Heinrich Billich
ETH Z?rich
Informatikdienste
Tel.: +41 44 632 72 56
heinrich.billich at id.ethz.ch
========================



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190718/d044120a/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.gif
Type: image/gif
Size: 106 bytes
Desc: image001.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190718/d044120a/attachment.gif>

From sxiao at us.ibm.com  Thu Jul 18 16:03:38 2019
From: sxiao at us.ibm.com (Steve Xiao)
Date: Thu, 18 Jul 2019 11:03:38 -0400
Subject: [gpfsug-discuss] How to prove that data is in inode
In-Reply-To: <mailman.267.1563459398.1289.gpfsug-discuss@spectrumscale.org>
References: <mailman.267.1563459398.1289.gpfsug-discuss@spectrumscale.org>
Message-ID: <OF10D4C475.21C37EF2-ON8525843B.00521027-8525843B.0052BAEF@notes.na.collabserv.com>

I believe GPFS report  1 block used in stat so the file will not be 
treated as empty by applications.    If i remember correctly, some backup 
programs was skipping small files with data in inode because GPFS was 
returning 0 block used for these small files. 

Steve Y. Xiao



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190718/6df9881d/attachment.htm>

From novosirj at rutgers.edu  Thu Jul 18 22:15:52 2019
From: novosirj at rutgers.edu (Ryan Novosielski)
Date: Thu, 18 Jul 2019 21:15:52 +0000
Subject: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
In-Reply-To: <OFE9B5AE08.76EA4744-ON002582F2.003C63C0-C12582F2.00410D62@notes.na.collabserv.com>
References: <OF67F2CFA5.B278755D-ON002582F2.002CFDE9-C12582F2.0030AC8F@notes.na.collabserv.com>
	<CAAxuGpE4fM+X1Gwc-OySE+ZWc_jygx5TACusAP4rO0BWNgtaKA@mail.gmail.com>
	<OF237C2868.64DA65AF-ON002582F2.003784F8-002582F2.0038210B@notes.na.collabserv.com>
	<OFE9B5AE08.76EA4744-ON002582F2.003C63C0-C12582F2.00410D62@notes.na.collabserv.com>
Message-ID: <225AEBED-69EF-4E3C-9A67-5A9CB04B4309@rutgers.edu>

I happened across this message because I?ve already done this in the past and was trying to figure out how I did it (apparently didn?t write it down).

Most of it appeared to be adding to /etc/sysconfig/gpfsgui the following:

HTTP_PORT=8080
HTTPS_PORT=8443

?but that hasn?t completely done it yet. Going to have a look and see what else I might need to do.

--
____
|| \\UTGERS,  	 |---------------------------*O*---------------------------
||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
     `'

> On Aug 23, 2018, at 7:50 AM, Markus Rohwedder <rohwedder at de.ibm.com> wrote:
> 
> Hello Juri, Keith,
> 
> thank you for your responses.
> 
> The internal services communicate on the privileged ports, for backwards compatibility and firewall simplicity reasons. We can not just assume all nodes in the cluster are at the latest level.
> 
> Running two services at the same port on different IP addresses could be an option to consider for co-existance of the GUI and another service on the same node.
> However we have not set up, tested nor documented such a configuration as of today. 
> 
> Currently the GUI service manages the iptables redirect bring up and tear down.
> If this would be managed externally it would be possible to bind services to specific ports based on specific IPs.
> 
> In order to create custom redirect rules based on IP address it is necessary to instruct the GUI to 
> - not check for already used ports when the GUI service tries to start up
> - don't create/destroy port forwarding rules during GUI service start and stop.
> This GUI behavior can be configured using the internal flag UPDATE_IPTABLES in the service configuration with the 5.0.1.2 GUI code level.
> 
> The service configuration is not stored in the cluster configuration and may be overwritten during code upgrades, so these settings may have to be added again after an upgrade.
> 
> See this KC link:
> https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.1/com.ibm.spectrum.scale.v5r01.doc/bl1adv_firewallforgui.htm
> 
> Mit freundlichen Gr??en / Kind regards
> 
> Dr. Markus Rohwedder
> 
> Spectrum Scale GUI Development
> <ecblank.gif>
> Phone:	+49 7034 6430190	IBM Deutschland Research & Development	
> <17153317.gif>
> E-Mail:	rohwedder at de.ibm.com	Am Weiher 24
> <ecblank.gif>	<ecblank.gif>	65451 Kelsterbach
> <ecblank.gif>	<ecblank.gif>	Germany
> <ecblank.gif>
> 
> <graycol.gif>"Daniel Kidger" ---23.08.2018 12:13:36---Keith, I have another IBM customer who also wished to move Scale GUI's https ports. In their case
> 
> From:  "Daniel Kidger" <daniel.kidger at uk.ibm.com>
> To:  gpfsug-discuss at spectrumscale.org
> Cc:  gpfsug-discuss at spectrumscale.org
> Date:  23.08.2018 12:13
> Subject:  Re: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
> Sent by:  gpfsug-discuss-bounces at spectrumscale.org
> 
> 
> 
> 
> Keith,
> 
> I have another IBM customer who also wished to move Scale GUI's https ports.
> In their case because they had their own web based management interface on the same https port.
> Is this the same reason that you have?
> If so I wonder how many other sites have the same issue?
> 
> One workaround that was suggested at the time, was to add a second IP address to the node (piggy-backing on 'eth0').
> Then run the two different GUIs, one per IP address.
> Is this an option, albeit a little ugly?
> Daniel
> 
> <17310450.gif>				Dr Daniel Kidger
> IBM Technical Sales Specialist
> Software Defined Solution Sales
> 
> +44-(0)7818 522 266 
> daniel.kidger at uk.ibm.com
> 
> 
> 
> ----- Original message -----
> From: "Markus Rohwedder" <rohwedder at de.ibm.com>
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Cc:
> Subject: Re: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
> Date: Thu, Aug 23, 2018 9:51 AM
> Hello Keith,
> 
> it is not so easy.
> 
> The GUI receives events from other scale components using the currently defined ports.
> Changing the GUI ports will cause breakage in the GUI stack at several places (internal watchdog functions, interlock with health events, interlock with CES).
> Therefore at this point there is no procedure to change this behaviour across all components.
> 
> Because the GUI service does not run as root. the GUI server does not serve the privileged ports 80 and 443 directly but rather 47443 and 47080.
> Tweaking the ports in the server.xml file will only change the native ports that the GUI uses.
> The GUI manages IPTABLES rules to forward ports 443 and 80 to 47443 and 47080. 
> If these ports are already used by another service, the GUI will not start up.
> 
> Making the GUI ports freely configurable is therefore not a strightforward change, and currently no on our roadmap.
> If you want to emphasize your case as future development item, please let me know.
> 
> I would also be interested in:
> > Scale version you are running
> > Do you need port 80 or 443 as well?
> > Would it work for you if the xCAT service was bound to a single IP address?
> 
> Mit freundlichen Gr??en / Kind regards
> 
> Dr. Markus Rohwedder
> 
> Spectrum Scale GUI Development
> 
> <ecblank.gif>
> Phone:	+49 7034 6430190	IBM Deutschland Research & Development	
> <17153317.gif>
> E-Mail:	rohwedder at de.ibm.com	Am Weiher 24
> <ecblank.gif>	<ecblank.gif>	65451 Kelsterbach
> <ecblank.gif>	<ecblank.gif>	Germany
> <ecblank.gif>
> 
> <graycol.gif>Keith Ball ---22.08.2018 21:33:25---Hello All, Does anyone know how to change the HTTP ports for the Spectrum Scale GUI?
> 
> From: Keith Ball <bipcuds at gmail.com>
> To: gpfsug-discuss at spectrumscale.org
> Date: 22.08.2018 21:33
> Subject: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> 
> 
> 
> 
> Hello All,
> 
> Does anyone know how to change the HTTP ports for the Spectrum Scale GUI? Any documentation or RedPaper I have found deftly avoids discussing this. The most promising thing I see is in /opt/ibm/wlp/usr/servers/gpfsgui/server.xml:
> 
> <httpEndpoint id="defaultHttpEndpoint" host="*" httpPort="47080" httpsPort="47443">
> <tcpOptions soReuseAddr="true"/>
> </httpEndpoint>
> 
> but it appears that port 80 specifically is used also by the GUI's Web service. I already have an HTTP server using port 80 for provisioning (xCAT), so would rather change the Specturm Scale GUI configuration if I can.
> 
> Many Thanks,
> Keith
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number 741598. 
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss


From knop at us.ibm.com  Thu Jul 18 22:24:50 2019
From: knop at us.ibm.com (Felipe Knop)
Date: Thu, 18 Jul 2019 17:24:50 -0400
Subject: [gpfsug-discuss]
 =?utf-8?q?GPFS_quota_inode/files_maximum_setting?=
 =?utf-8?q?_of=092147483647?=
In-Reply-To: <DM6PR14MB2890B7F5B0283B4444C68BAEA2C90@DM6PR14MB2890.namprd14.prod.outlook.com>
References: <DM6PR14MB2890B7F5B0283B4444C68BAEA2C90@DM6PR14MB2890.namprd14.prod.outlook.com>
Message-ID: <OFD656CADB.20E2D1BC-ON0025843B.006EB6EC-8525843B.0075A180@notes.na.collabserv.com>


Bryan,

Could you open an RFE for this item?  We took an initial look at what it
will take to raise this limit and found that it will require an RPC format
change. So it's something that would need to go through a normal release
cycle.

Thanks and regards,

  Felipe

----
Felipe Knop                                     knop at us.ibm.com
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601
(845) 433-9314  T/L 293-9314





From:	Bryan Banister <bbanister at jumptrading.com>
To:	gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:	07/17/2019 01:24 PM
Subject:	[EXTERNAL] [gpfsug-discuss] GPFS quota inode/files maximum
            setting of	2147483647
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



Thought this might be of interest to others on the list.

IBM, are there any plans to increase the maximum files setting for quotas
over 2147483647?

Thanks,
-Bryan_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=oNT2koCZX0xmWlSlLblR9Q&m=gA7MMIPzQmpVqvoaVl9D2jeytjZOXtkGikPGPpNg_HU&s=pdkFa7ir9MdaAdAuMic15m4-cbL4FqlsESaTYOnxAq4&e=



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190718/374e3cc2/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190718/374e3cc2/attachment.gif>

From novosirj at rutgers.edu  Thu Jul 18 22:26:50 2019
From: novosirj at rutgers.edu (Ryan Novosielski)
Date: Thu, 18 Jul 2019 21:26:50 +0000
Subject: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
In-Reply-To: <225AEBED-69EF-4E3C-9A67-5A9CB04B4309@rutgers.edu>
References: <OF67F2CFA5.B278755D-ON002582F2.002CFDE9-C12582F2.0030AC8F@notes.na.collabserv.com>
	<CAAxuGpE4fM+X1Gwc-OySE+ZWc_jygx5TACusAP4rO0BWNgtaKA@mail.gmail.com>
	<OF237C2868.64DA65AF-ON002582F2.003784F8-002582F2.0038210B@notes.na.collabserv.com>
	<OFE9B5AE08.76EA4744-ON002582F2.003C63C0-C12582F2.00410D62@notes.na.collabserv.com>
	<225AEBED-69EF-4E3C-9A67-5A9CB04B4309@rutgers.edu>
Message-ID: <B5538DB3-4625-43DA-B8F9-E600F48054B4@rutgers.edu>

Nope, that appears to be all of it. I also had a problem with the postgresql service, which was why the gpfsgui wouldn?t start. But once I fixed that, I can log in on https://<GUI_HOST>:8443.

HTH.

> On Jul 18, 2019, at 5:15 PM, Ryan Novosielski <novosirj at rutgers.edu> wrote:
> 
> I happened across this message because I?ve already done this in the past and was trying to figure out how I did it (apparently didn?t write it down).
> 
> Most of it appeared to be adding to /etc/sysconfig/gpfsgui the following:
> 
> HTTP_PORT=8080
> HTTPS_PORT=8443
> 
> ?but that hasn?t completely done it yet. Going to have a look and see what else I might need to do.
> 
> --
> ____
> || \\UTGERS,  	 |---------------------------*O*---------------------------
> ||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
> ||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
>     `'
> 
>> On Aug 23, 2018, at 7:50 AM, Markus Rohwedder <rohwedder at de.ibm.com> wrote:
>> 
>> Hello Juri, Keith,
>> 
>> thank you for your responses.
>> 
>> The internal services communicate on the privileged ports, for backwards compatibility and firewall simplicity reasons. We can not just assume all nodes in the cluster are at the latest level.
>> 
>> Running two services at the same port on different IP addresses could be an option to consider for co-existance of the GUI and another service on the same node.
>> However we have not set up, tested nor documented such a configuration as of today. 
>> 
>> Currently the GUI service manages the iptables redirect bring up and tear down.
>> If this would be managed externally it would be possible to bind services to specific ports based on specific IPs.
>> 
>> In order to create custom redirect rules based on IP address it is necessary to instruct the GUI to 
>> - not check for already used ports when the GUI service tries to start up
>> - don't create/destroy port forwarding rules during GUI service start and stop.
>> This GUI behavior can be configured using the internal flag UPDATE_IPTABLES in the service configuration with the 5.0.1.2 GUI code level.
>> 
>> The service configuration is not stored in the cluster configuration and may be overwritten during code upgrades, so these settings may have to be added again after an upgrade.
>> 
>> See this KC link:
>> https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.1/com.ibm.spectrum.scale.v5r01.doc/bl1adv_firewallforgui.htm
>> 
>> Mit freundlichen Gr??en / Kind regards
>> 
>> Dr. Markus Rohwedder
>> 
>> Spectrum Scale GUI Development
>> <ecblank.gif>
>> Phone:	+49 7034 6430190	IBM Deutschland Research & Development	
>> <17153317.gif>
>> E-Mail:	rohwedder at de.ibm.com	Am Weiher 24
>> <ecblank.gif>	<ecblank.gif>	65451 Kelsterbach
>> <ecblank.gif>	<ecblank.gif>	Germany
>> <ecblank.gif>
>> 
>> <graycol.gif>"Daniel Kidger" ---23.08.2018 12:13:36---Keith, I have another IBM customer who also wished to move Scale GUI's https ports. In their case
>> 
>> From:  "Daniel Kidger" <daniel.kidger at uk.ibm.com>
>> To:  gpfsug-discuss at spectrumscale.org
>> Cc:  gpfsug-discuss at spectrumscale.org
>> Date:  23.08.2018 12:13
>> Subject:  Re: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
>> Sent by:  gpfsug-discuss-bounces at spectrumscale.org
>> 
>> 
>> 
>> 
>> Keith,
>> 
>> I have another IBM customer who also wished to move Scale GUI's https ports.
>> In their case because they had their own web based management interface on the same https port.
>> Is this the same reason that you have?
>> If so I wonder how many other sites have the same issue?
>> 
>> One workaround that was suggested at the time, was to add a second IP address to the node (piggy-backing on 'eth0').
>> Then run the two different GUIs, one per IP address.
>> Is this an option, albeit a little ugly?
>> Daniel
>> 
>> <17310450.gif>				Dr Daniel Kidger
>> IBM Technical Sales Specialist
>> Software Defined Solution Sales
>> 
>> +44-(0)7818 522 266 
>> daniel.kidger at uk.ibm.com
>> 
>> 
>> 
>> ----- Original message -----
>> From: "Markus Rohwedder" <rohwedder at de.ibm.com>
>> Sent by: gpfsug-discuss-bounces at spectrumscale.org
>> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
>> Cc:
>> Subject: Re: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
>> Date: Thu, Aug 23, 2018 9:51 AM
>> Hello Keith,
>> 
>> it is not so easy.
>> 
>> The GUI receives events from other scale components using the currently defined ports.
>> Changing the GUI ports will cause breakage in the GUI stack at several places (internal watchdog functions, interlock with health events, interlock with CES).
>> Therefore at this point there is no procedure to change this behaviour across all components.
>> 
>> Because the GUI service does not run as root. the GUI server does not serve the privileged ports 80 and 443 directly but rather 47443 and 47080.
>> Tweaking the ports in the server.xml file will only change the native ports that the GUI uses.
>> The GUI manages IPTABLES rules to forward ports 443 and 80 to 47443 and 47080. 
>> If these ports are already used by another service, the GUI will not start up.
>> 
>> Making the GUI ports freely configurable is therefore not a strightforward change, and currently no on our roadmap.
>> If you want to emphasize your case as future development item, please let me know.
>> 
>> I would also be interested in:
>>> Scale version you are running
>>> Do you need port 80 or 443 as well?
>>> Would it work for you if the xCAT service was bound to a single IP address?
>> 
>> Mit freundlichen Gr??en / Kind regards
>> 
>> Dr. Markus Rohwedder
>> 
>> Spectrum Scale GUI Development
>> 
>> <ecblank.gif>
>> Phone:	+49 7034 6430190	IBM Deutschland Research & Development	
>> <17153317.gif>
>> E-Mail:	rohwedder at de.ibm.com	Am Weiher 24
>> <ecblank.gif>	<ecblank.gif>	65451 Kelsterbach
>> <ecblank.gif>	<ecblank.gif>	Germany
>> <ecblank.gif>
>> 
>> <graycol.gif>Keith Ball ---22.08.2018 21:33:25---Hello All, Does anyone know how to change the HTTP ports for the Spectrum Scale GUI?
>> 
>> From: Keith Ball <bipcuds at gmail.com>
>> To: gpfsug-discuss at spectrumscale.org
>> Date: 22.08.2018 21:33
>> Subject: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
>> Sent by: gpfsug-discuss-bounces at spectrumscale.org
>> 
>> 
>> 
>> 
>> Hello All,
>> 
>> Does anyone know how to change the HTTP ports for the Spectrum Scale GUI? Any documentation or RedPaper I have found deftly avoids discussing this. The most promising thing I see is in /opt/ibm/wlp/usr/servers/gpfsgui/server.xml:
>> 
>> <httpEndpoint id="defaultHttpEndpoint" host="*" httpPort="47080" httpsPort="47443">
>> <tcpOptions soReuseAddr="true"/>
>> </httpEndpoint>
>> 
>> but it appears that port 80 specifically is used also by the GUI's Web service. I already have an HTTP server using port 80 for provisioning (xCAT), so would rather change the Specturm Scale GUI configuration if I can.
>> 
>> Many Thanks,
>> Keith
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>> 
>> 
>> 
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>> 
>> Unless stated otherwise above:
>> IBM United Kingdom Limited - Registered in England and Wales with number 741598. 
>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>> 
>> 
>> 
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss


From novosirj at rutgers.edu  Fri Jul 19 17:43:03 2019
From: novosirj at rutgers.edu (Ryan Novosielski)
Date: Fri, 19 Jul 2019 16:43:03 +0000
Subject: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
In-Reply-To: <B5538DB3-4625-43DA-B8F9-E600F48054B4@rutgers.edu>
References: <OF67F2CFA5.B278755D-ON002582F2.002CFDE9-C12582F2.0030AC8F@notes.na.collabserv.com>
	<CAAxuGpE4fM+X1Gwc-OySE+ZWc_jygx5TACusAP4rO0BWNgtaKA@mail.gmail.com>
	<OF237C2868.64DA65AF-ON002582F2.003784F8-002582F2.0038210B@notes.na.collabserv.com>
	<OFE9B5AE08.76EA4744-ON002582F2.003C63C0-C12582F2.00410D62@notes.na.collabserv.com>
	<225AEBED-69EF-4E3C-9A67-5A9CB04B4309@rutgers.edu>
	<B5538DB3-4625-43DA-B8F9-E600F48054B4@rutgers.edu>
Message-ID: <C7EE62BE-118A-4680-B64E-4A2050F1BFBA@rutgers.edu>

Support has since pointed out to me, coincidentally, that all of the GUI callback scripts in /usr/lpp/mmfs/gui/callbacks contain a hard-coded port. I changed them this way, with PWD=/usr/lpp/mmfs/gui/callbacks:

find . -name *.sh -exec sed -i.bak 's/PORT=443/PORT=8443/g' {} \;

?as you can see, the only change was, for example:

[root at quorum02 callbacks]# diff gnr/PhysicalDiskCallback.sh.bak gnr/PhysicalDiskCallback.sh 
21c21
< PORT=443
---
> PORT=8443

That caused me to look to see if port 443 is hard-coded anyplace else, and I found this other one, /usr/lpp/mmfs/gui/bin-sudo/functions_iptables.sh:

23: . /etc/sysconfig/gpfsgui
24: 
25: HTTP_PORT=80
26: HTTPS_PORT=443

?this is peculiar to me because a) it works ? it would seem like these two override my /etc/sysconfig/gpfsgui settings, but the web server is reachable at 8443. Also, these lines would seem to make way more sense in the reverse (eg. let the sysconfig file redefine the ports if they contain values).

Ideally, IBM would let you change those two environment variables in the sysconfig file, or somewhere else, and the callback scripts would use that value from the environment. I?ve not tried setting PORT=$HTTPS_PORT to see if those callback scripts have access to that variable. 

--
____
|| \\UTGERS,  	 |---------------------------*O*---------------------------
||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
     `'

> On Jul 18, 2019, at 5:26 PM, Ryan Novosielski <novosirj at rutgers.edu> wrote:
> 
> Nope, that appears to be all of it. I also had a problem with the postgresql service, which was why the gpfsgui wouldn?t start. But once I fixed that, I can log in on https://<GUI_HOST>:8443.
> 
> HTH.
> 
>> On Jul 18, 2019, at 5:15 PM, Ryan Novosielski <novosirj at rutgers.edu> wrote:
>> 
>> I happened across this message because I?ve already done this in the past and was trying to figure out how I did it (apparently didn?t write it down).
>> 
>> Most of it appeared to be adding to /etc/sysconfig/gpfsgui the following:
>> 
>> HTTP_PORT=8080
>> HTTPS_PORT=8443
>> 
>> ?but that hasn?t completely done it yet. Going to have a look and see what else I might need to do.
>> 
>> --
>> ____
>> || \\UTGERS,  	 |---------------------------*O*---------------------------
>> ||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
>> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
>> ||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
>>    `'
>> 
>>> On Aug 23, 2018, at 7:50 AM, Markus Rohwedder <rohwedder at de.ibm.com> wrote:
>>> 
>>> Hello Juri, Keith,
>>> 
>>> thank you for your responses.
>>> 
>>> The internal services communicate on the privileged ports, for backwards compatibility and firewall simplicity reasons. We can not just assume all nodes in the cluster are at the latest level.
>>> 
>>> Running two services at the same port on different IP addresses could be an option to consider for co-existance of the GUI and another service on the same node.
>>> However we have not set up, tested nor documented such a configuration as of today. 
>>> 
>>> Currently the GUI service manages the iptables redirect bring up and tear down.
>>> If this would be managed externally it would be possible to bind services to specific ports based on specific IPs.
>>> 
>>> In order to create custom redirect rules based on IP address it is necessary to instruct the GUI to 
>>> - not check for already used ports when the GUI service tries to start up
>>> - don't create/destroy port forwarding rules during GUI service start and stop.
>>> This GUI behavior can be configured using the internal flag UPDATE_IPTABLES in the service configuration with the 5.0.1.2 GUI code level.
>>> 
>>> The service configuration is not stored in the cluster configuration and may be overwritten during code upgrades, so these settings may have to be added again after an upgrade.
>>> 
>>> See this KC link:
>>> https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.1/com.ibm.spectrum.scale.v5r01.doc/bl1adv_firewallforgui.htm
>>> 
>>> Mit freundlichen Gr??en / Kind regards
>>> 
>>> Dr. Markus Rohwedder
>>> 
>>> Spectrum Scale GUI Development
>>> <ecblank.gif>
>>> Phone:	+49 7034 6430190	IBM Deutschland Research & Development	
>>> <17153317.gif>
>>> E-Mail:	rohwedder at de.ibm.com	Am Weiher 24
>>> <ecblank.gif>	<ecblank.gif>	65451 Kelsterbach
>>> <ecblank.gif>	<ecblank.gif>	Germany
>>> <ecblank.gif>
>>> 
>>> <graycol.gif>"Daniel Kidger" ---23.08.2018 12:13:36---Keith, I have another IBM customer who also wished to move Scale GUI's https ports. In their case
>>> 
>>> From:  "Daniel Kidger" <daniel.kidger at uk.ibm.com>
>>> To:  gpfsug-discuss at spectrumscale.org
>>> Cc:  gpfsug-discuss at spectrumscale.org
>>> Date:  23.08.2018 12:13
>>> Subject:  Re: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
>>> Sent by:  gpfsug-discuss-bounces at spectrumscale.org
>>> 
>>> 
>>> 
>>> 
>>> Keith,
>>> 
>>> I have another IBM customer who also wished to move Scale GUI's https ports.
>>> In their case because they had their own web based management interface on the same https port.
>>> Is this the same reason that you have?
>>> If so I wonder how many other sites have the same issue?
>>> 
>>> One workaround that was suggested at the time, was to add a second IP address to the node (piggy-backing on 'eth0').
>>> Then run the two different GUIs, one per IP address.
>>> Is this an option, albeit a little ugly?
>>> Daniel
>>> 
>>> <17310450.gif>				Dr Daniel Kidger
>>> IBM Technical Sales Specialist
>>> Software Defined Solution Sales
>>> 
>>> +44-(0)7818 522 266 
>>> daniel.kidger at uk.ibm.com
>>> 
>>> 
>>> 
>>> ----- Original message -----
>>> From: "Markus Rohwedder" <rohwedder at de.ibm.com>
>>> Sent by: gpfsug-discuss-bounces at spectrumscale.org
>>> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
>>> Cc:
>>> Subject: Re: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
>>> Date: Thu, Aug 23, 2018 9:51 AM
>>> Hello Keith,
>>> 
>>> it is not so easy.
>>> 
>>> The GUI receives events from other scale components using the currently defined ports.
>>> Changing the GUI ports will cause breakage in the GUI stack at several places (internal watchdog functions, interlock with health events, interlock with CES).
>>> Therefore at this point there is no procedure to change this behaviour across all components.
>>> 
>>> Because the GUI service does not run as root. the GUI server does not serve the privileged ports 80 and 443 directly but rather 47443 and 47080.
>>> Tweaking the ports in the server.xml file will only change the native ports that the GUI uses.
>>> The GUI manages IPTABLES rules to forward ports 443 and 80 to 47443 and 47080. 
>>> If these ports are already used by another service, the GUI will not start up.
>>> 
>>> Making the GUI ports freely configurable is therefore not a strightforward change, and currently no on our roadmap.
>>> If you want to emphasize your case as future development item, please let me know.
>>> 
>>> I would also be interested in:
>>>> Scale version you are running
>>>> Do you need port 80 or 443 as well?
>>>> Would it work for you if the xCAT service was bound to a single IP address?
>>> 
>>> Mit freundlichen Gr??en / Kind regards
>>> 
>>> Dr. Markus Rohwedder
>>> 
>>> Spectrum Scale GUI Development
>>> 
>>> <ecblank.gif>
>>> Phone:	+49 7034 6430190	IBM Deutschland Research & Development	
>>> <17153317.gif>
>>> E-Mail:	rohwedder at de.ibm.com	Am Weiher 24
>>> <ecblank.gif>	<ecblank.gif>	65451 Kelsterbach
>>> <ecblank.gif>	<ecblank.gif>	Germany
>>> <ecblank.gif>
>>> 
>>> <graycol.gif>Keith Ball ---22.08.2018 21:33:25---Hello All, Does anyone know how to change the HTTP ports for the Spectrum Scale GUI?
>>> 
>>> From: Keith Ball <bipcuds at gmail.com>
>>> To: gpfsug-discuss at spectrumscale.org
>>> Date: 22.08.2018 21:33
>>> Subject: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
>>> Sent by: gpfsug-discuss-bounces at spectrumscale.org
>>> 
>>> 
>>> 
>>> 
>>> Hello All,
>>> 
>>> Does anyone know how to change the HTTP ports for the Spectrum Scale GUI? Any documentation or RedPaper I have found deftly avoids discussing this. The most promising thing I see is in /opt/ibm/wlp/usr/servers/gpfsgui/server.xml:
>>> 
>>> <httpEndpoint id="defaultHttpEndpoint" host="*" httpPort="47080" httpsPort="47443">
>>> <tcpOptions soReuseAddr="true"/>
>>> </httpEndpoint>
>>> 
>>> but it appears that port 80 specifically is used also by the GUI's Web service. I already have an HTTP server using port 80 for provisioning (xCAT), so would rather change the Specturm Scale GUI configuration if I can.
>>> 
>>> Many Thanks,
>>> Keith
>>> _______________________________________________
>>> gpfsug-discuss mailing list
>>> gpfsug-discuss at spectrumscale.org
>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> gpfsug-discuss mailing list
>>> gpfsug-discuss at spectrumscale.org
>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>> 
>>> Unless stated otherwise above:
>>> IBM United Kingdom Limited - Registered in England and Wales with number 741598. 
>>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>>> _______________________________________________
>>> gpfsug-discuss mailing list
>>> gpfsug-discuss at spectrumscale.org
>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> gpfsug-discuss mailing list
>>> gpfsug-discuss at spectrumscale.org
>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>> 
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss


From lore at cscs.ch  Fri Jul 19 17:59:05 2019
From: lore at cscs.ch (Lo Re  Giuseppe)
Date: Fri, 19 Jul 2019 16:59:05 +0000
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 90, Issue 11
In-Reply-To: <mailman.1.1563534001.39239.gpfsug-discuss@spectrumscale.org>
References: <mailman.1.1563534001.39239.gpfsug-discuss@spectrumscale.org>
Message-ID: <1F7E4727-C978-45D5-A348-22DC7B62AF88@cscs.ch>

Hi,

We have also hit this limitation at CSCS.
We have a CES Object Service cluster where one node cannot run the Openstack Swift proxy-server service on port 443 as it is already taken by the GUI.
I have opened a RFE, didn?t help much though __ , so far.

https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=107778

Giuseppe

?On 19.07.19, 04:01, "gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org" <gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org> wrote:

    /etc/sysconfig/gpfsgui


From novosirj at rutgers.edu  Wed Jul 24 17:19:47 2019
From: novosirj at rutgers.edu (Ryan Novosielski)
Date: Wed, 24 Jul 2019 16:19:47 +0000
Subject: [gpfsug-discuss] Spectrum Scale with RHEL7.6 kernel
 3.10.0-957.21.2
In-Reply-To: <OF248C1D60.E94029E1-ON00258412.00769D60-85258412.0077861A@notes.na.collabserv.com>
References: <OF248C1D60.E94029E1-ON00258412.00769D60-85258412.0077861A@notes.na.collabserv.com>
Message-ID: <3DFDAC1A-ECD5-4752-8342-55389804FBDD@rutgers.edu>

Hi Felipe and others:

We didn?t hear about this apart from via this list and from a colleague. What is the way to get on an official notifications list for this sort of thing? I don?t see anything anywhere in the IBM support pages.

Thanks!

--
____
|| \\UTGERS,  	 |---------------------------*O*---------------------------
||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
     `'

> On Jun 7, 2019, at 5:45 PM, Felipe Knop <knop at us.ibm.com> wrote:
> 
> All,
> 
> There have been reported issues (including kernel crashes) on Spectrum Scale with the latest RHEL7.6 kernel 3.10.0-957.21.2. Please consider delaying upgrades to this kernel until further information is provided. 
> 
> Thanks,
> 
> Felipe
> 
> ----
> Felipe Knop knop at us.ibm.com
> GPFS Development and Security
> IBM Systems
> IBM Building 008
> 2455 South Rd, Poughkeepsie, NY 12601
> (845) 433-9314 T/L 293-9314
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss


From S.J.Thompson at bham.ac.uk  Wed Jul 24 18:30:31 2019
From: S.J.Thompson at bham.ac.uk (Simon Thompson)
Date: Wed, 24 Jul 2019 17:30:31 +0000
Subject: [gpfsug-discuss] Spectrum Scale with RHEL7.6 kernel
 3.10.0-957.21.2
In-Reply-To: <3DFDAC1A-ECD5-4752-8342-55389804FBDD@rutgers.edu>
References: <OF248C1D60.E94029E1-ON00258412.00769D60-85258412.0077861A@notes.na.collabserv.com>,
	<3DFDAC1A-ECD5-4752-8342-55389804FBDD@rutgers.edu>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545012993054A@EX13.adf.bham.ac.uk>

There is a notifications service. I think you have to subscribe to them via fix central or passport advantage (I don't remember which).

Though this notification was on the list a while before the official Comms.

Simon
________________________________________
From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Ryan Novosielski [novosirj at rutgers.edu]
Sent: 24 July 2019 17:19
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Spectrum Scale with RHEL7.6 kernel 3.10.0-957.21.2

Hi Felipe and others:

We didn?t hear about this apart from via this list and from a colleague. What is the way to get on an official notifications list for this sort of thing? I don?t see anything anywhere in the IBM support pages.

Thanks!

--
____
|| \\UTGERS,     |---------------------------*O*---------------------------
||_// the State  |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ  | Office of Advanced Research Computing - MSB C630, Newark
     `'

> On Jun 7, 2019, at 5:45 PM, Felipe Knop <knop at us.ibm.com> wrote:
>
> All,
>
> There have been reported issues (including kernel crashes) on Spectrum Scale with the latest RHEL7.6 kernel 3.10.0-957.21.2. Please consider delaying upgrades to this kernel until further information is provided.
>
> Thanks,
>
> Felipe
>
> ----
> Felipe Knop knop at us.ibm.com
> GPFS Development and Security
> IBM Systems
> IBM Building 008
> 2455 South Rd, Poughkeepsie, NY 12601
> (845) 433-9314 T/L 293-9314
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


From mark.bergman at uphs.upenn.edu  Fri Jul 26 00:31:21 2019
From: mark.bergman at uphs.upenn.edu (mark.bergman at uphs.upenn.edu)
Date: Thu, 25 Jul 2019 19:31:21 -0400
Subject: [gpfsug-discuss] Question concerning integration of CES with AD
	authentication system
In-Reply-To: Your message of "Thu, 24 May 2018 17:07:02 -0000."
	<OF0CCFED2D.FE89B800-ON00258297.005DFD38-00258297.005E075F@notes.na.collabserv.com>
References: <OF0CCFED2D.FE89B800-ON00258297.005DFD38-00258297.005E075F@notes.na.collabserv.com>
	<1527173192.28106.18.camel@strath.ac.uk>,
	<83A6EEB0EC738F459A39439733AE804522F1B13B@MBX214.d.ethz.ch><20180524141632.xuah3dxu4bxx372z@utumno.gs.washington.edu>
Message-ID: <10843-1564097481.984955@sYvE.K-ht.6Q-W>

In the message dated: Thu, 24 May 2018 17:07:02 -0000,
The pithy ruminations from Christof Schmitt on 
[Re: [gpfsug-discuss] Question concerning integration of CES with AD authentication system] were:
=> 

Following up on an old, old post...

=> > Basically Samba ignores the separate GID field in RFC2307bis, so one
=> > imagines the options for changing the LDAP attributes are none
=> > existent.
=>  
=> mmuserauth now has an option to use either the gid from the actual primary
=> group or the gid defined for the user. See:
=>  
=> https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.0/
=> com.ibm.spectrum.scale.v5r00.doc/bl1adm_mmuserauth.htm
=>  
=> --unixmap-domains unixDomainMap
=> [...]
=>     win: Specifies the system to read the primary group set as Windows
=> primary group of a user on the Active Directory.
=>      unix: Specifies the system to read the primary group as set in "UNIX
=> attributes" of a user on the Active Directory.  
=>     For example,
=>      --unixmap-domains "MYDOMAIN1(20000-50000:unix);MYDOMAIN2
=> (100000-200000:win)"

I see this is refering to UNIX attributes within AD, but I'm curious about mapping to attributes in LDAP.

=> This gets mapped to 'idmap config ... : unix_primary_group' in the
=> internal config.

Does that correspond to setting the smb.conf parameter
	
	unix_primary_group = yes




Specifically, under Spectrum Scale 5.0.2, if I run:

	mmuserauth service create --data-access-method file --ldapmap-domains "DOMAIN(type=stand-alone:ldap_srv=ldapserver:range=1001-65535:usr_dn=ou=People,dc=DC,dc=TLD:grp_dn=ou=Group,dc=DC,dc=TLD)" --type ad

(some args removed in this example), will that map the user's primary group to

	the primaryGroupID supplied by AD
  or
	the primaryGroupID LDAP field
  or
	the gidNumber LDAP field

or something else?

Thanks,

Mark


=> 
=> Christof Schmitt || IBM || Spectrum Scale Development || Tucson, AZ
=> christof.schmitt at us.ibm.com  ||  +1-520-799-2469    (T/L: 321-2469)
=>  


From christof.schmitt at us.ibm.com  Fri Jul 26 17:38:17 2019
From: christof.schmitt at us.ibm.com (Christof Schmitt)
Date: Fri, 26 Jul 2019 16:38:17 +0000
Subject: [gpfsug-discuss] Question concerning integration of CES with AD
 authentication system
In-Reply-To: <10843-1564097481.984955@sYvE.K-ht.6Q-W>
References: <10843-1564097481.984955@sYvE.K-ht.6Q-W>,
	<OF0CCFED2D.FE89B800-ON00258297.005DFD38-00258297.005E075F@notes.na.collabserv.com>
	<1527173192.28106.18.camel@strath.ac.uk>,
	<83A6EEB0EC738F459A39439733AE804522F1B13B@MBX214.d.ethz.ch><20180524141632.xuah3dxu4bxx372z@utumno.gs.washington.edu>
Message-ID: <OF5FF29174.4A08CF99-ON00258443.005B64E3-00258443.005B658E@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190726/68876eda/attachment.htm>

From david_johnson at brown.edu  Mon Jul 29 17:04:38 2019
From: david_johnson at brown.edu (David Johnson)
Date: Mon, 29 Jul 2019 12:04:38 -0400
Subject: [gpfsug-discuss] Building GPFS filesystem system data pool on
 shared nothing NVMe drives
Message-ID: <B5D9698F-23E7-432E-A35E-747569E7422F@brown.edu>

We are planning a 5.0.x upgrade onto new hardware to make use of the new 5.x GPFS features.
The goal is to use up to four NSD nodes for metadata, each one with 6 NVMe drives (to be determined
whether we use Intel VROC for raid 5 or raid 1, or just straight disks).  

So questions ? 
Has anyone done system pool on shared nothing cluster?  How did you set it up?
With default metadata replication set at 3, can you make use of four NSD nodes effectively?
How would one design the location vectors and failure groups so that the system metadata is
spread evenly across the four servers?

Thanks,
 ? ddj
Dave Johnson

From xhejtman at ics.muni.cz  Mon Jul 29 17:11:38 2019
From: xhejtman at ics.muni.cz (Lukas Hejtmanek)
Date: Mon, 29 Jul 2019 18:11:38 +0200
Subject: [gpfsug-discuss] Asymetric SAN with GPFS
Message-ID: <20190729161138.znuss5dt2rhig6cv@ics.muni.cz>

Hello,

is there any settings for GPFS 5.x so that you could mitigate slow down of
asymmetric SAN? The asymmetric SAN means, that not every LUN has the same
speed, or not every disk array has the same number of LUNs. It seems that
overal speed is degraded to the slowest LUN. Is there any workaround for this
except avoiding that LUN at all?

-- 
Luk?? Hejtm?nek

Linux Administrator only because
  Full Time Multitasking Ninja 
  is not an official job title


From S.J.Thompson at bham.ac.uk  Mon Jul 29 17:19:45 2019
From: S.J.Thompson at bham.ac.uk (Simon Thompson)
Date: Mon, 29 Jul 2019 16:19:45 +0000
Subject: [gpfsug-discuss] Asymetric SAN with GPFS
In-Reply-To: <20190729161138.znuss5dt2rhig6cv@ics.muni.cz>
References: <20190729161138.znuss5dt2rhig6cv@ics.muni.cz>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545012994D59A@EX13.adf.bham.ac.uk>

Surely you would need to use different data pools for this. Given in a single pool data will be distributed over all available nsd devices in that pool, you are highly likely to only ever go as fast as the slowest LUN.

Simon
________________________________________
From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of xhejtman at ics.muni.cz [xhejtman at ics.muni.cz]
Sent: 29 July 2019 17:11
To: gpfsug-discuss at spectrumscale.org
Subject: [gpfsug-discuss] Asymetric SAN with GPFS

Hello,

is there any settings for GPFS 5.x so that you could mitigate slow down of
asymmetric SAN? The asymmetric SAN means, that not every LUN has the same
speed, or not every disk array has the same number of LUNs. It seems that
overal speed is degraded to the slowest LUN. Is there any workaround for this
except avoiding that LUN at all?

--
Luk?? Hejtm?nek

Linux Administrator only because
  Full Time Multitasking Ninja
  is not an official job title
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


From luis.bolinches at fi.ibm.com  Mon Jul 29 18:34:58 2019
From: luis.bolinches at fi.ibm.com (Luis Bolinches)
Date: Mon, 29 Jul 2019 17:34:58 +0000
Subject: [gpfsug-discuss] Building GPFS filesystem system data pool on
 shared nothing NVMe drives
In-Reply-To: <B5D9698F-23E7-432E-A35E-747569E7422F@brown.edu>
Message-ID: <OFC1ED3470.4BE6E16D-ON00258446.006095F7-1564421698664@notes.na.collabserv.com>

Hi, from phone so sorry for typos. 

I really think you should look into Spectrum Scale Erasure Code Edition (ECE) for this. 

Sure you could do a RAID on each node as you mention here but that sounds like a lot of waste to me on storage capacity. Not to forget you get other goodies like end to end checksum and rapid rebuilds with ECE, among others. 

Four servers is the minimum requirement for ECE (4+3p) and from top of my head 12 disk per RG, you are fine with both requirements. 

There is a presentation on ECE on the user group web page from London May 2019 were we talk about ECE. 

And the ibm page of the product https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_intro.htm
--
Cheers

> El 29 jul 2019, a las 19:06, David Johnson <david_johnson at brown.edu> escribi?:
> 
> We are planning a 5.0.x upgrade onto new hardware to make use of the new 5.x GPFS features.
> The goal is to use up to four NSD nodes for metadata, each one with 6 NVMe drives (to be determined
> whether we use Intel VROC for raid 5 or raid 1, or just straight disks).  
> 
> So questions ? 
> Has anyone done system pool on shared nothing cluster?  How did you set it up?
> With default metadata replication set at 3, can you make use of four NSD nodes effectively?
> How would one design the location vectors and failure groups so that the system metadata is
> spread evenly across the four servers?
> 
> Thanks,
> ? ddj
> Dave Johnson
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=1mZ896psa5caYzBeaugTlc7TtRejJp3uvKYxas3S7Xc&m=Pu3G5GzwRsM4iwztIiWzzngNSgsPzrRe8cd3pPAFVM0&s=q3w_lvsuB88gMM_J5psgKuWDex_wSW1v5h16WNMCtzU&e= 
> 

Ellei edell? ole toisin mainittu: / Unless stated otherwise above:
Oy IBM Finland Ab
PL 265, 00101 Helsinki, Finland
Business ID, Y-tunnus: 0195876-3 
Registered in Finland

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190729/0d0c5e04/attachment.htm>

From david_johnson at brown.edu  Tue Jul 30 12:46:14 2019
From: david_johnson at brown.edu (David Johnson)
Date: Tue, 30 Jul 2019 07:46:14 -0400
Subject: [gpfsug-discuss] Building GPFS filesystem system data pool on
 shared nothing NVMe drives
In-Reply-To: <OFC1ED3470.4BE6E16D-ON00258446.006095F7-1564421698664@notes.na.collabserv.com>
References: <OFC1ED3470.4BE6E16D-ON00258446.006095F7-1564421698664@notes.na.collabserv.com>
Message-ID: <D53515DA-94D8-400B-80E0-8033633A5F21@brown.edu>

Can we confirm the requirement for disks per RG?  I have 4 RG, but only 6 x 3TB NVMe drives per box.

> On Jul 29, 2019, at 1:34 PM, Luis Bolinches <luis.bolinches at fi.ibm.com> wrote:
> 
> Hi, from phone so sorry for typos. 
> 
> I really think you should look into Spectrum Scale Erasure Code Edition (ECE) for this. 
> 
> Sure you could do a RAID on each node as you mention here but that sounds like a lot of waste to me on storage capacity. Not to forget you get other goodies like end to end checksum and rapid rebuilds with ECE, among others. 
> 
> Four servers is the minimum requirement for ECE (4+3p) and from top of my head 12 disk per RG, you are fine with both requirements. 
> 
> There is a presentation on ECE on the user group web page from London May 2019 were we talk about ECE. 
> 
> And the ibm page of the product https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_intro.htm <https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_intro.htm>
> --
> Cheers
> 
> El 29 jul 2019, a las 19:06, David Johnson <david_johnson at brown.edu <mailto:david_johnson at brown.edu>> escribi?:
> 
>> We are planning a 5.0.x upgrade onto new hardware to make use of the new 5.x GPFS features.
>> The goal is to use up to four NSD nodes for metadata, each one with 6 NVMe drives (to be determined
>> whether we use Intel VROC for raid 5 or raid 1, or just straight disks).  
>> 
>> So questions ? 
>> Has anyone done system pool on shared nothing cluster?  How did you set it up?
>> With default metadata replication set at 3, can you make use of four NSD nodes effectively?
>> How would one design the location vectors and failure groups so that the system metadata is
>> spread evenly across the four servers?
>> 
>> Thanks,
>> ? ddj
>> Dave Johnson
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org <http://spectrumscale.org/>
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss <http://gpfsug.org/mailman/listinfo/gpfsug-discuss> 
>> 
> 
> Ellei edell? ole toisin mainittu: / Unless stated otherwise above:
> Oy IBM Finland Ab
> PL 265, 00101 Helsinki, Finland
> Business ID, Y-tunnus: 0195876-3 
> Registered in Finland
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190730/b61f3695/attachment.htm>

From p.ward at nhm.ac.uk  Tue Jul 30 13:16:31 2019
From: p.ward at nhm.ac.uk (Paul Ward)
Date: Tue, 30 Jul 2019 12:16:31 +0000
Subject: [gpfsug-discuss] Break-out session for new user and prospects
 [London Usergroup]
In-Reply-To: <6916EC29-2057-4396-A285-7A905DB215EE@bham.ac.uk>
References: <776770B2-5F84-4462-B900-58EBB982DC1C@spectrumscale.org>
	<DB7PR10MB2201EAAA629BB9C10BBE24E0D3C90@DB7PR10MB2201.EURPRD10.PROD.OUTLOOK.COM>
	<6916EC29-2057-4396-A285-7A905DB215EE@bham.ac.uk>
Message-ID: <DB7PR10MB22017537C2C2102FD904B11CD3DC0@DB7PR10MB2201.EURPRD10.PROD.OUTLOOK.COM>

Hi Simon,

Any update on this please.


Paul Ward
Technical Solutions Infrastructure Architect
Natural History Museum
T: 02079426450
E: p.ward at nhm.ac.uk<mailto:p.ward at nhm.ac.uk>

From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of Simon Thompson
Sent: 17 July 2019 14:10
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Break-out session for new user and prospects [London Usergroup]

Hi Paul,

There was a slide deck, I?ve asked the IBMers if it?s something we can share.

Simon

From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of Paul Ward <p.ward at nhm.ac.uk<mailto:p.ward at nhm.ac.uk>>
Reply-To: "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, 17 July 2019 at 14:01
To: "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Break-out session for new user and prospects [London Usergroup]

Hi Simon,

Was this a slide presentation? Is it available?

Kindest regards,
Paul

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (Spectrum Scale User Group Chair)
Sent: 30 April 2019 10:25
To: gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] Break-out session for new user and prospects [London Usergroup]

Hi all,

We know that a lot of the talks at the user groups are for experienced users, following feedback from the USA user group, we thought we?d advertise that this year we?re planning to run a break-out for new users on day 1.

Break-out session for new user and prospects (Wed May 8th, 13:00 - 16:45)

This year we will offer a break-out session for new Spectrum Scale user and prospects to get started with Spectrum Scale. In this session we will cover Spectrum Scale Use Cases, the architecture of a Spectrum Scale environment, and discuss how the manifold Spectrum Scale features support the different use case. Please inform customers and colleagues who are interested to learn about Spectrum Scale to grab one of the last seats.

Registration link:
https://www.spectrumscaleug.org/event/uk-user-group-meeting/<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.spectrumscaleug.org%2Fevent%2Fuk-user-group-meeting%2F&data=02%7C01%7Cp.ward%40nhm.ac.uk%7C7d6995c8217f453c8dd408d70ab82dad%7C73a29c014e78437fa0d4c8553e1960c1%7C1%7C0%7C636989658468906223&sdata=%2BmSDjCczU4GYNWbtOfeWhxovF0YtYzkMX%2BJNRmcIpNo%3D&reserved=0>

There?s just a couple of places left for the usergroup, so please do share and register if you plan to attend.

Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190730/eeac3325/attachment.htm>

From Paul.Sanchez at deshaw.com  Tue Jul 30 13:19:56 2019
From: Paul.Sanchez at deshaw.com (Sanchez, Paul)
Date: Tue, 30 Jul 2019 12:19:56 +0000
Subject: [gpfsug-discuss] Building GPFS filesystem system data pool on
 shared nothing NVMe drives
In-Reply-To: <D53515DA-94D8-400B-80E0-8033633A5F21@brown.edu>
References: <OFC1ED3470.4BE6E16D-ON00258446.006095F7-1564421698664@notes.na.collabserv.com>
	<D53515DA-94D8-400B-80E0-8033633A5F21@brown.edu>
Message-ID: <5664c4aaea5e4fbfac1f8a72acb080dc@deshaw.com>

Hi David,

In an ECE configuration, it would be typical to put all of the NVMe disks in all 4 of your servers into a single recovery group.   So in your case, all 24 NVMe drives would be in one recovery group and the 4 servers would be ?log group? servers in the recovery group, distributing the I/O load for the NSD/vdisks that are hosted on the RG.  (The minimum disks for a single RG config is 12, and you meet that easily.)

https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_plan_recommendations.htm
outlines the recommendations for raidCode protection.  Your configuration (4 nodes) would use vdisks with 4+3P, which gives you a slightly better capacity yield than RAID10 would, but with much better recovery characteristics:


?         No single failed node will result in a down system NSD.

?         No single drive failure will require a critical priority rebuild, and can be handled in the background without killing performance.

So from that perspective, ECE is a win here and avoids a problem with the non-ECE, shared-nothing designs: the manual ?mmchdisk <fsname> start -a? operation that is needed after any traditional shared-nothing metadata NSD goes offline to bring it back and protect against further failures.

Despite the operational challenges of the non-ECE design, it can sometimes survive two server failures (if replication factor is 3 and the filesystem descriptor quorum wasn?t lost by the two failures) which a 4 node ECE cluster cannot.  Given that the world is complex and unexpected things can happen, I?d personally recommend redistributing the 24 disks across 6 servers if you can, so that the design could always survive 2 node failures.  I?ve run this design and it?s fairly robust.

In any event, you should of course test the failure scenarios yourself before going into production to validate them and familiarize yourself with the process.  And a special note on ECE: due to the cooperative nature at the pdisk level, the network between the servers in the RG should be as reliable as possible and any network redundancy should also be tested ahead of time.

-Paul

From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of David Johnson
Sent: Tuesday, July 30, 2019 7:46 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Building GPFS filesystem system data pool on shared nothing NVMe drives


This message was sent by an external party.

Can we confirm the requirement for disks per RG?  I have 4 RG, but only 6 x 3TB NVMe drives per box.


On Jul 29, 2019, at 1:34 PM, Luis Bolinches <luis.bolinches at fi.ibm.com<mailto:luis.bolinches at fi.ibm.com>> wrote:

Hi, from phone so sorry for typos.

I really think you should look into Spectrum Scale Erasure Code Edition (ECE) for this.

Sure you could do a RAID on each node as you mention here but that sounds like a lot of waste to me on storage capacity. Not to forget you get other goodies like end to end checksum and rapid rebuilds with ECE, among others.

Four servers is the minimum requirement for ECE (4+3p) and from top of my head 12 disk per RG, you are fine with both requirements.

There is a presentation on ECE on the user group web page from London May 2019 were we talk about ECE.

And the ibm page of the product https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_intro.htm
--
Cheers

El 29 jul 2019, a las 19:06, David Johnson <david_johnson at brown.edu<mailto:david_johnson at brown.edu>> escribi?:
We are planning a 5.0.x upgrade onto new hardware to make use of the new 5.x GPFS features.
The goal is to use up to four NSD nodes for metadata, each one with 6 NVMe drives (to be determined
whether we use Intel VROC for raid 5 or raid 1, or just straight disks).

So questions ?
Has anyone done system pool on shared nothing cluster?  How did you set it up?
With default metadata replication set at 3, can you make use of four NSD nodes effectively?
How would one design the location vectors and failure groups so that the system metadata is
spread evenly across the four servers?

Thanks,
? ddj
Dave Johnson
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Ellei edell? ole toisin mainittu: / Unless stated otherwise above:
Oy IBM Finland Ab
PL 265, 00101 Helsinki, Finland
Business ID, Y-tunnus: 0195876-3
Registered in Finland
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190730/539e5159/attachment.htm>

From david_johnson at brown.edu  Tue Jul 30 14:03:54 2019
From: david_johnson at brown.edu (David Johnson)
Date: Tue, 30 Jul 2019 09:03:54 -0400
Subject: [gpfsug-discuss] Building GPFS filesystem system data pool on
 shared nothing NVMe drives
In-Reply-To: <5664c4aaea5e4fbfac1f8a72acb080dc@deshaw.com>
References: <OFC1ED3470.4BE6E16D-ON00258446.006095F7-1564421698664@notes.na.collabserv.com>
	<D53515DA-94D8-400B-80E0-8033633A5F21@brown.edu>
	<5664c4aaea5e4fbfac1f8a72acb080dc@deshaw.com>
Message-ID: <261C855E-FC23-43C6-8A76-FEAE5B48221C@brown.edu>

OK, so the ECE recovery group is the four NSD servers with the System storage pool disks, and somehow I have to read the docs
and find out how to define pdisks that spread the replication across the four servers, but three disks at a time.  
Three pdisks of 7 drives, three I can't do anything with, or are those for rebuilding space?

Can you provide me details of your six-node non-ECE configuration?  Basically how the NSDs are defined...

The remainder of our new filesystem will have a fast pool of 12 nodes of excelero, and 2Pb of spinning disks, so another possibility
would be to license four more nodes and put the system pool under excelero.  

 -- ddj

> On Jul 30, 2019, at 8:19 AM, Sanchez, Paul <Paul.Sanchez at deshaw.com> wrote:
> 
> Hi David,
>  
> In an ECE configuration, it would be typical to put all of the NVMe disks in all 4 of your servers into a single recovery group.   So in your case, all 24 NVMe drives would be in one recovery group and the 4 servers would be ?log group? servers in the recovery group, distributing the I/O load for the NSD/vdisks that are hosted on the RG.  (The minimum disks for a single RG config is 12, and you meet that easily.)
>  
> https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_plan_recommendations.htm <https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_plan_recommendations.htm>
> outlines the recommendations for raidCode protection.  Your configuration (4 nodes) would use vdisks with 4+3P, which gives you a slightly better capacity yield than RAID10 would, but with much better recovery characteristics:
>  
> ?         No single failed node will result in a down system NSD.
> ?         No single drive failure will require a critical priority rebuild, and can be handled in the background without killing performance.
>  
> So from that perspective, ECE is a win here and avoids a problem with the non-ECE, shared-nothing designs: the manual ?mmchdisk <fsname> start -a? operation that is needed after any traditional shared-nothing metadata NSD goes offline to bring it back and protect against further failures.
>  
> Despite the operational challenges of the non-ECE design, it can sometimes survive two server failures (if replication factor is 3 and the filesystem descriptor quorum wasn?t lost by the two failures) which a 4 node ECE cluster cannot.  Given that the world is complex and unexpected things can happen, I?d personally recommend redistributing the 24 disks across 6 servers if you can, so that the design could always survive 2 node failures.  I?ve run this design and it?s fairly robust.
>  
> In any event, you should of course test the failure scenarios yourself before going into production to validate them and familiarize yourself with the process.  And a special note on ECE: due to the cooperative nature at the pdisk level, the network between the servers in the RG should be as reliable as possible and any network redundancy should also be tested ahead of time.
>  
> -Paul
>  
> From: gpfsug-discuss-bounces at spectrumscale.org <mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org <mailto:gpfsug-discuss-bounces at spectrumscale.org>> On Behalf Of David Johnson
> Sent: Tuesday, July 30, 2019 7:46 AM
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org <mailto:gpfsug-discuss at spectrumscale.org>>
> Subject: Re: [gpfsug-discuss] Building GPFS filesystem system data pool on shared nothing NVMe drives
>  
> This message was sent by an external party.
> 
>  
> Can we confirm the requirement for disks per RG?  I have 4 RG, but only 6 x 3TB NVMe drives per box.
> 
> 
> On Jul 29, 2019, at 1:34 PM, Luis Bolinches <luis.bolinches at fi.ibm.com <mailto:luis.bolinches at fi.ibm.com>> wrote:
>  
> Hi, from phone so sorry for typos.  
>  
> I really think you should look into Spectrum Scale Erasure Code Edition (ECE) for this. 
>  
> Sure you could do a RAID on each node as you mention here but that sounds like a lot of waste to me on storage capacity. Not to forget you get other goodies like end to end checksum and rapid rebuilds with ECE, among others. 
>  
> Four servers is the minimum requirement for ECE (4+3p) and from top of my head 12 disk per RG, you are fine with both requirements. 
> 
> There is a presentation on ECE on the user group web page from London May 2019 were we talk about ECE. 
>  
> And the ibm page of the product https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_intro.htm <https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_intro.htm>
> -- 
> Cheers
> 
> El 29 jul 2019, a las 19:06, David Johnson <david_johnson at brown.edu <mailto:david_johnson at brown.edu>> escribi?:
> 
> We are planning a 5.0.x upgrade onto new hardware to make use of the new 5.x GPFS features.
> The goal is to use up to four NSD nodes for metadata, each one with 6 NVMe drives (to be determined
> whether we use Intel VROC for raid 5 or raid 1, or just straight disks).  
> 
> So questions ? 
> Has anyone done system pool on shared nothing cluster?  How did you set it up?
> With default metadata replication set at 3, can you make use of four NSD nodes effectively?
> How would one design the location vectors and failure groups so that the system metadata is
> spread evenly across the four servers?
> 
> Thanks,
> ? ddj
> Dave Johnson
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org <http://spectrumscale.org/>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
> 
> Ellei edell? ole toisin mainittu: / Unless stated otherwise above:
> Oy IBM Finland Ab
> PL 265, 00101 Helsinki, Finland
> Business ID, Y-tunnus: 0195876-3 
> Registered in Finland
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org <http://spectrumscale.org/>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
>  
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org <http://spectrumscale.org/>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190730/e8dc0611/attachment.htm>

From S.J.Thompson at bham.ac.uk  Tue Jul 30 14:49:04 2019
From: S.J.Thompson at bham.ac.uk (Simon Thompson)
Date: Tue, 30 Jul 2019 13:49:04 +0000
Subject: [gpfsug-discuss] Break-out session for new user and prospects
 [London Usergroup]
In-Reply-To: <DB7PR10MB22017537C2C2102FD904B11CD3DC0@DB7PR10MB2201.EURPRD10.PROD.OUTLOOK.COM>
References: <776770B2-5F84-4462-B900-58EBB982DC1C@spectrumscale.org>
	<DB7PR10MB2201EAAA629BB9C10BBE24E0D3C90@DB7PR10MB2201.EURPRD10.PROD.OUTLOOK.COM>
	<6916EC29-2057-4396-A285-7A905DB215EE@bham.ac.uk>
	<DB7PR10MB22017537C2C2102FD904B11CD3DC0@DB7PR10MB2201.EURPRD10.PROD.OUTLOOK.COM>
Message-ID: <BB09590F-57DE-4BCF-BD14-5DFED3BC2AD8@bham.ac.uk>

Hi Sorry, yes. I got the slides and forgot to upload them!

https://www.spectrumscaleug.org/wp-content/uploads/2019/07/IBM-Spectrum-Scale-Use-Cases.pdf

Simon

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Paul Ward <p.ward at nhm.ac.uk>
Reply-To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Date: Tuesday, 30 July 2019 at 13:16
To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Break-out session for new user and prospects [London Usergroup]

Hi Simon,

Any update on this please.


Paul Ward
Technical Solutions Infrastructure Architect
Natural History Museum
T: 02079426450
E: p.ward at nhm.ac.uk<mailto:p.ward at nhm.ac.uk>

From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of Simon Thompson
Sent: 17 July 2019 14:10
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Break-out session for new user and prospects [London Usergroup]

Hi Paul,

There was a slide deck, I?ve asked the IBMers if it?s something we can share.

Simon

From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of Paul Ward <p.ward at nhm.ac.uk<mailto:p.ward at nhm.ac.uk>>
Reply-To: "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, 17 July 2019 at 14:01
To: "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Break-out session for new user and prospects [London Usergroup]

Hi Simon,

Was this a slide presentation? Is it available?

Kindest regards,
Paul

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (Spectrum Scale User Group Chair)
Sent: 30 April 2019 10:25
To: gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] Break-out session for new user and prospects [London Usergroup]

Hi all,

We know that a lot of the talks at the user groups are for experienced users, following feedback from the USA user group, we thought we?d advertise that this year we?re planning to run a break-out for new users on day 1.

Break-out session for new user and prospects (Wed May 8th, 13:00 - 16:45)

This year we will offer a break-out session for new Spectrum Scale user and prospects to get started with Spectrum Scale. In this session we will cover Spectrum Scale Use Cases, the architecture of a Spectrum Scale environment, and discuss how the manifold Spectrum Scale features support the different use case. Please inform customers and colleagues who are interested to learn about Spectrum Scale to grab one of the last seats.

Registration link:
https://www.spectrumscaleug.org/event/uk-user-group-meeting/<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.spectrumscaleug.org%2Fevent%2Fuk-user-group-meeting%2F&data=02%7C01%7Cp.ward%40nhm.ac.uk%7C7d6995c8217f453c8dd408d70ab82dad%7C73a29c014e78437fa0d4c8553e1960c1%7C1%7C0%7C636989658468906223&sdata=%2BmSDjCczU4GYNWbtOfeWhxovF0YtYzkMX%2BJNRmcIpNo%3D&reserved=0>

There?s just a couple of places left for the usergroup, so please do share and register if you plan to attend.

Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190730/731e27a0/attachment.htm>

From Paul.Sanchez at deshaw.com  Tue Jul 30 15:36:35 2019
From: Paul.Sanchez at deshaw.com (Sanchez, Paul)
Date: Tue, 30 Jul 2019 14:36:35 +0000
Subject: [gpfsug-discuss] Building GPFS filesystem system data pool on
 shared nothing NVMe drives
In-Reply-To: <261C855E-FC23-43C6-8A76-FEAE5B48221C@brown.edu>
References: <OFC1ED3470.4BE6E16D-ON00258446.006095F7-1564421698664@notes.na.collabserv.com>
	<D53515DA-94D8-400B-80E0-8033633A5F21@brown.edu>
	<5664c4aaea5e4fbfac1f8a72acb080dc@deshaw.com>
	<261C855E-FC23-43C6-8A76-FEAE5B48221C@brown.edu>
Message-ID: <b227c5eadc5c46cd9e63b19e51416502@deshaw.com>

Yes, read the documentation for the mmvdisk command to get a better idea of how this actually works.  There?s definitely a small paradigmatic shift in thinking that you?ll encounter between traditional NSDs and ECE.  The mmvdisk command?s defaults handle just about everything for you and the defaults are sane.

The short answer is that in a 6-node ECE recoverygroup, each of the nodes will normally provide access to 2 NSDs and the system pool would therefore have 12 NSDs total.  If a node fails, each of its two NSDs will continue to be served each by a different one of the remaining servers.  If you?ve used ESS before, then you can think about the RAID/spare/rebuild space similarly to that: all drives have some spare space on them and erasure-code stripes are evenly distributed among all of the disks so that they are all used and fill at approximately the same rate.

From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of David Johnson
Sent: Tuesday, July 30, 2019 9:04 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Building GPFS filesystem system data pool on shared nothing NVMe drives


This message was sent by an external party.

OK, so the ECE recovery group is the four NSD servers with the System storage pool disks, and somehow I have to read the docs
and find out how to define pdisks that spread the replication across the four servers, but three disks at a time.
Three pdisks of 7 drives, three I can't do anything with, or are those for rebuilding space?

Can you provide me details of your six-node non-ECE configuration?  Basically how the NSDs are defined...

The remainder of our new filesystem will have a fast pool of 12 nodes of excelero, and 2Pb of spinning disks, so another possibility
would be to license four more nodes and put the system pool under excelero.

 -- ddj


On Jul 30, 2019, at 8:19 AM, Sanchez, Paul <Paul.Sanchez at deshaw.com<mailto:Paul.Sanchez at deshaw.com>> wrote:

Hi David,

In an ECE configuration, it would be typical to put all of the NVMe disks in all 4 of your servers into a single recovery group.   So in your case, all 24 NVMe drives would be in one recovery group and the 4 servers would be ?log group? servers in the recovery group, distributing the I/O load for the NSD/vdisks that are hosted on the RG.  (The minimum disks for a single RG config is 12, and you meet that easily.)

https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_plan_recommendations.htm
outlines the recommendations for raidCode protection.  Your configuration (4 nodes) would use vdisks with 4+3P, which gives you a slightly better capacity yield than RAID10 would, but with much better recovery characteristics:

?         No single failed node will result in a down system NSD.
?         No single drive failure will require a critical priority rebuild, and can be handled in the background without killing performance.

So from that perspective, ECE is a win here and avoids a problem with the non-ECE, shared-nothing designs: the manual ?mmchdisk <fsname> start -a? operation that is needed after any traditional shared-nothing metadata NSD goes offline to bring it back and protect against further failures.

Despite the operational challenges of the non-ECE design, it can sometimes survive two server failures (if replication factor is 3 and the filesystem descriptor quorum wasn?t lost by the two failures) which a 4 node ECE cluster cannot.  Given that the world is complex and unexpected things can happen, I?d personally recommend redistributing the 24 disks across 6 servers if you can, so that the design could always survive 2 node failures.  I?ve run this design and it?s fairly robust.

In any event, you should of course test the failure scenarios yourself before going into production to validate them and familiarize yourself with the process.  And a special note on ECE: due to the cooperative nature at the pdisk level, the network between the servers in the RG should be as reliable as possible and any network redundancy should also be tested ahead of time.

-Paul

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> On Behalf Of David Johnson
Sent: Tuesday, July 30, 2019 7:46 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Building GPFS filesystem system data pool on shared nothing NVMe drives

This message was sent by an external party.

Can we confirm the requirement for disks per RG?  I have 4 RG, but only 6 x 3TB NVMe drives per box.



On Jul 29, 2019, at 1:34 PM, Luis Bolinches <luis.bolinches at fi.ibm.com<mailto:luis.bolinches at fi.ibm.com>> wrote:

Hi, from phone so sorry for typos.

I really think you should look into Spectrum Scale Erasure Code Edition (ECE) for this.

Sure you could do a RAID on each node as you mention here but that sounds like a lot of waste to me on storage capacity. Not to forget you get other goodies like end to end checksum and rapid rebuilds with ECE, among others.

Four servers is the minimum requirement for ECE (4+3p) and from top of my head 12 disk per RG, you are fine with both requirements.

There is a presentation on ECE on the user group web page from London May 2019 were we talk about ECE.

And the ibm page of the product https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_intro.htm
--
Cheers

El 29 jul 2019, a las 19:06, David Johnson <david_johnson at brown.edu<mailto:david_johnson at brown.edu>> escribi?:
We are planning a 5.0.x upgrade onto new hardware to make use of the new 5.x GPFS features.
The goal is to use up to four NSD nodes for metadata, each one with 6 NVMe drives (to be determined
whether we use Intel VROC for raid 5 or raid 1, or just straight disks).

So questions ?
Has anyone done system pool on shared nothing cluster?  How did you set it up?
With default metadata replication set at 3, can you make use of four NSD nodes effectively?
How would one design the location vectors and failure groups so that the system metadata is
spread evenly across the four servers?

Thanks,
? ddj
Dave Johnson
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Ellei edell? ole toisin mainittu: / Unless stated otherwise above:
Oy IBM Finland Ab
PL 265, 00101 Helsinki, Finland
Business ID, Y-tunnus: 0195876-3
Registered in Finland
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190730/d782a1c1/attachment.htm>

From luis.bolinches at fi.ibm.com  Tue Jul 30 15:54:42 2019
From: luis.bolinches at fi.ibm.com (Luis Bolinches)
Date: Tue, 30 Jul 2019 14:54:42 +0000
Subject: [gpfsug-discuss] Building GPFS filesystem system data pool on
 shared nothing NVMe drives
In-Reply-To: <b227c5eadc5c46cd9e63b19e51416502@deshaw.com>
References: <b227c5eadc5c46cd9e63b19e51416502@deshaw.com>,
	<OFC1ED3470.4BE6E16D-ON00258446.006095F7-1564421698664@notes.na.collabserv.com><D53515DA-94D8-400B-80E0-8033633A5F21@brown.edu><5664c4aaea5e4fbfac1f8a72acb080dc@deshaw.com><261C855E-FC23-43C6-8A76-FEAE5B48221C@brown.edu>
Message-ID: <OF31F22F72.D63B7538-ON00258447.00510DA2-00258447.0051E9DD@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190730/7157fa12/attachment.htm>

From novosirj at rutgers.edu  Mon Jul  1 06:42:51 2019
From: novosirj at rutgers.edu (Ryan Novosielski)
Date: Mon, 1 Jul 2019 05:42:51 +0000
Subject: [gpfsug-discuss] Any guidelines for choosing vdisk size?
Message-ID: <E55F8C7F-48A4-4A66-ABF8-880A5232F39D@oarc.rutgers.edu>

Good morning,

Was wondering if anyone could point me to any tips or provide some regarding choosing vdisk size? My understanding is that too small is a waste of resources in the form of overhead, and that there is an upper limit. but that generally within a pool, you want them to be the same size, so that if you aren?t allocating the entire storage space on the system straight off, you?ll need to choose a reasonable size or be left with unusable space (eg. you will waste space if you went with, say, 200 GB vdisk sizes on a 500 GB array).

Anyone have any tips? Do people just generally allocate the whole thing and have one 2 vdisks (redundancy)? Seems you have some more flexibility if you don?t do that and have to, say, create a storage pool or filesystem from scratch to take advantage of features in a newer FS version or what have you.

Thanks for the help ? spent a lot of time looking for this previously, but never asked on the list.
 
--
____
|| \\UTGERS,  	 |---------------------------*O*---------------------------
||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
     `'


From abeattie at au1.ibm.com  Mon Jul  1 06:54:37 2019
From: abeattie at au1.ibm.com (Andrew Beattie)
Date: Mon, 1 Jul 2019 05:54:37 +0000
Subject: [gpfsug-discuss] Any guidelines for choosing vdisk size?
In-Reply-To: <E55F8C7F-48A4-4A66-ABF8-880A5232F39D@oarc.rutgers.edu>
References: <E55F8C7F-48A4-4A66-ABF8-880A5232F39D@oarc.rutgers.edu>
Message-ID: <OF54C435D4.B3E41DA2-ON0025842A.001FD56B-0025842A.002077AE@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190701/6acfe5f8/attachment-0001.htm>

From novosirj at rutgers.edu  Mon Jul  1 07:01:46 2019
From: novosirj at rutgers.edu (Ryan Novosielski)
Date: Mon, 1 Jul 2019 06:01:46 +0000
Subject: [gpfsug-discuss] Any guidelines for choosing vdisk size?
In-Reply-To: <OF54C435D4.B3E41DA2-ON0025842A.001FD56B-0025842A.002077AE@notes.na.collabserv.com>
References: <E55F8C7F-48A4-4A66-ABF8-880A5232F39D@oarc.rutgers.edu>
	<OF54C435D4.B3E41DA2-ON0025842A.001FD56B-0025842A.002077AE@notes.na.collabserv.com>
Message-ID: <D599D062-6286-477C-B494-4AD0984691AD@rutgers.edu>

Sure; thanks, I meant to add those details:

In our case, all GSS/DSS-G with GNR.

The context here is general information, but specifically designing new filesystems/setting up one of these systems from scratch in what I?d figure to be the traditional way: don?t allocate the entire storage if you?re not totally sure the way the growth will go, but don?t leave behind unusable space or space that forces you to later change vdisk sizes. Essentially allocate something today that makes sense, but leaves room open for sensible growth with both the existing hardware and new, if expanded. If I?m thinking about this generally the wrong way, let me know.

The systems we currently own are GSS26 (6 TB drives, fully populated ? I forget how many drives, 350 or so?) , and a few DSS-G 220 w/10 TB drives (also fully populated with 168 drives).

> On Jul 1, 2019, at 1:54 AM, Andrew Beattie <abeattie at au1.ibm.com> wrote:
> 
> Hi Ryan,
>  
> Can I ask a couple of clarifying questions?
>  
> Is this in an ESS / GSS environment with Scale native raid ?
> Is this in classic San attached storage environment / or Direct Attached Disk without Scale Native Raid?
>  
> What are you trying to achieve?
> Are you presenting the Vdisk into an existing filesystem ?
> Are you creating an new filesystem?
>  
>  
> Regards,
>  
> Andrew Beattie
> File and Object Storage Technical Specialist - A/NZ
> IBM Systems - Storage
> Phone: 614-2133-7927
> E-mail: abeattie at au1.ibm.com
>  
>  
> ----- Original message -----
> From: Ryan Novosielski <novosirj at rutgers.edu>
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Cc:
> Subject: [EXTERNAL] [gpfsug-discuss] Any guidelines for choosing vdisk size?
> Date: Mon, Jul 1, 2019 3:43 PM
>  
> Good morning,
> 
> Was wondering if anyone could point me to any tips or provide some regarding choosing vdisk size? My understanding is that too small is a waste of resources in the form of overhead, and that there is an upper limit. but that generally within a pool, you want them to be the same size, so that if you aren?t allocating the entire storage space on the system straight off, you?ll need to choose a reasonable size or be left with unusable space (eg. you will waste space if you went with, say, 200 GB vdisk sizes on a 500 GB array).
> 
> Anyone have any tips? Do people just generally allocate the whole thing and have one 2 vdisks (redundancy)? Seems you have some more flexibility if you don?t do that and have to, say, create a storage pool or filesystem from scratch to take advantage of features in a newer FS version or what have you.
> 
> Thanks for the help ? spent a lot of time looking for this previously, but never asked on the list.
>  
> --
> ____
> || \\UTGERS,   |---------------------------*O*---------------------------
> ||_// the State |         Ryan Novosielski - novosirj at rutgers.edu
> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
> ||  \\    of NJ | Office of Advanced Research Computing - MSB C630, Newark
>      `'
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss 
>  
>  
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss


From mhennecke at lenovo.com  Mon Jul  1 08:25:23 2019
From: mhennecke at lenovo.com (Michael Hennecke)
Date: Mon, 1 Jul 2019 07:25:23 +0000
Subject: [gpfsug-discuss] Any guidelines for choosing vdisk size?
Message-ID: <TY2PR03MB446249F071E9252F75D87E5ACCF90@TY2PR03MB4462.apcprd03.prod.outlook.com>

Ryan,

That is a question with many variables. Some of the factors influencing the decision are:
* Are you sharing data and metadata (the GNR default these days), or are you building separate data and metadata vdisks (the "older" default, and still useful for some use cases)? If you have separate metadata, you need to ensure you leave enough free space in the RGs to account for metadata growth beyond your initial sizing.
* Are you creating vdisks for multiple filesystems on the same building block, or just for one filesystem? 
* How likely is it that space needs to be (re-)allocated between different filesystems?
* How likely is it that new storage will be added to existing filesystems?
* Is there a requirement to include building blocks of different size (different #disks, or different capacity of disks) in the same filesystem?

All of these influence the decision on the "best" vdisk quantity and capacities. If you have a specific configuration question, I'm happy to discuss that offline.
 
In general I advise customers to use at least two equally sized vdisks per RG (4 per building block), so there is more flexibility when capacity needs to be moved around. But many sites that have homogeneous building blocks and don't expect changes in their storage configuration just go with a single vdisk per RG (2 per building block).

Either way, it is a good idea (in environments with more than a single building block) to put aside a small "test" vdisk per RG just for performance testing. So you can do performance validations on each building block individually even if your main filesystem spans all building block. Size of that "test" vdisk should be big enough so you can run a few minutes of sequential IOR against it without filling it up 100%.


Mit freundlichen Gr?ssen / Best regards,

Michael Hennecke
HPC Chief Technologist - HPC and AI Business Unit  
mailto:mhennecke at lenovo.com
--
Lenovo Global Technology (Germany) GmbH * Am Zehnthof 77 * D-45307 Essen * Germany 
Gesch?ftsf?hrung: Colm Gleeson, Christophe Laurent * Sitz der Gesellschaft: Stuttgart * HRB-Nr.: 758298, AG Stuttgart

-----Original Message-----
From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of Ryan Novosielski
Sent: Monday, 1 July, 2019 07:43
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [External] [gpfsug-discuss] Any guidelines for choosing vdisk size?

Good morning,

Was wondering if anyone could point me to any tips or provide some regarding choosing vdisk size? My understanding is that too small is a waste of resources in the form of overhead, and that there is an upper limit. but that generally within a pool, you want them to be the same size, so that if you aren?t allocating the entire storage space on the system straight off, you?ll need to choose a reasonable size or be left with unusable space (eg. you will waste space if you went with, say, 200 GB vdisk sizes on a 500 GB array).

Anyone have any tips? Do people just generally allocate the whole thing and have one 2 vdisks (redundancy)? Seems you have some more flexibility if you don?t do that and have to, say, create a storage pool or filesystem from scratch to take advantage of features in a newer FS version or what have you.

Thanks for the help ? spent a lot of time looking for this previously, but never asked on the list.
 
--
____
|| \\UTGERS,  	 |---------------------------*O*---------------------------
||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
     `'

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

From janfrode at tanso.net  Mon Jul  1 08:56:22 2019
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Mon, 1 Jul 2019 09:56:22 +0200
Subject: [gpfsug-discuss] Any guidelines for choosing vdisk size?
In-Reply-To: <E55F8C7F-48A4-4A66-ABF8-880A5232F39D@oarc.rutgers.edu>
References: <E55F8C7F-48A4-4A66-ABF8-880A5232F39D@oarc.rutgers.edu>
Message-ID: <CAHwPatihe-0esH-JbOzfENS65RvjY9ViQKvEyPnXr1+qhv_yqQ@mail.gmail.com>

I would mainly consider future upgrades. F.ex. do one vdisk per disk shelf
per rg. F.ex. for a GL6S you would have 12 vdisks, and if you add a GL4S
you would add 8 more vdisks, then each spindle of both systems should get
approximately the same number of IOs.

Another thing to consider is re-allocating capacity. If your vdisks are
very large, it might be difficult to free up a full vdisk if you need to
reorganize and create new filesystems, or add more 3way replicated
metadata..

But mainly I create large vdisks.. haven?t been able to show any
performance difference between one vdisk per RG vs. 10 vdisks per RG.


  -jf

man. 1. jul. 2019 kl. 07:42 skrev Ryan Novosielski <novosirj at rutgers.edu>:

> Good morning,
>
> Was wondering if anyone could point me to any tips or provide some
> regarding choosing vdisk size? My understanding is that too small is a
> waste of resources in the form of overhead, and that there is an upper
> limit. but that generally within a pool, you want them to be the same size,
> so that if you aren?t allocating the entire storage space on the system
> straight off, you?ll need to choose a reasonable size or be left with
> unusable space (eg. you will waste space if you went with, say, 200 GB
> vdisk sizes on a 500 GB array).
>
> Anyone have any tips? Do people just generally allocate the whole thing
> and have one 2 vdisks (redundancy)? Seems you have some more flexibility if
> you don?t do that and have to, say, create a storage pool or filesystem
> from scratch to take advantage of features in a newer FS version or what
> have you.
>
> Thanks for the help ? spent a lot of time looking for this previously, but
> never asked on the list.
>
> --
> ____
> || \\UTGERS,     |---------------------------*O*---------------------------
> ||_// the State  |         Ryan Novosielski - novosirj at rutgers.edu
> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
> ||  \\    of NJ  | Office of Advanced Research Computing - MSB C630, Newark
>      `'
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190701/9a8c578e/attachment-0001.htm>

From abeattie at au1.ibm.com  Mon Jul  1 09:12:41 2019
From: abeattie at au1.ibm.com (Andrew Beattie)
Date: Mon, 1 Jul 2019 08:12:41 +0000
Subject: [gpfsug-discuss] Any guidelines for choosing vdisk size?
In-Reply-To: <D599D062-6286-477C-B494-4AD0984691AD@rutgers.edu>
References: <D599D062-6286-477C-B494-4AD0984691AD@rutgers.edu>,
	<E55F8C7F-48A4-4A66-ABF8-880A5232F39D@oarc.rutgers.edu><OF54C435D4.B3E41DA2-ON0025842A.001FD56B-0025842A.002077AE@notes.na.collabserv.com>
Message-ID: <OF10ACC789.BB227E77-ON0025842A.002C56F8-0025842A.002D1B7B@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190701/2e915488/attachment-0001.htm>

From UWEFALKE at de.ibm.com  Mon Jul  1 09:29:49 2019
From: UWEFALKE at de.ibm.com (Uwe Falke)
Date: Mon, 1 Jul 2019 10:29:49 +0200
Subject: [gpfsug-discuss] Any guidelines for choosing vdisk size?
In-Reply-To: <TY2PR03MB446249F071E9252F75D87E5ACCF90@TY2PR03MB4462.apcprd03.prod.outlook.com>
References: <TY2PR03MB446249F071E9252F75D87E5ACCF90@TY2PR03MB4462.apcprd03.prod.outlook.com>
Message-ID: <OF01F0AAB3.D10E313F-ONC125842A.002D8C69-C125842A.002EAD36@notes.na.collabserv.com>

gpfsug-discuss-bounces at spectrumscale.org wrote on 01/07/2019 09:25:23:

> From: Michael Hennecke <mhennecke at lenovo.com>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Date: 01/07/2019 09:38
> Subject: [EXTERNAL] Re: [gpfsug-discuss] Any guidelines for choosing
> vdisk size?
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
...
> Either way, it is a good idea (in environments with more than a 
> single building block) to put aside a small "test" vdisk per RG just
> for performance testing. So you can do performance validations on 
> each building block individually even if your main filesystem spans 
> all building block. Size of that "test" vdisk should be big enough 
> so you can run a few minutes of sequential IOR against it without 
> filling it up 100%.

However, be aware that GNR limits the number of ptracks per vdisk 
according to the vdisk size. That could limit the seen performance 
especially for writes. 
I've experienced that with vdisk sizes of 509GiB in DAs of 288TiB of ESS 
GL4 (can't recall exactly if ptracks are assigned by absolute or by 
relative vdisk size). 


 
Mit freundlichen Gr??en / Kind regards

 
Dr. Uwe Falke
 
IT Specialist
High Performance Computing Services / Integrated Technology Services / 
Data Center Services
-------------------------------------------------------------------------------------------------------------------------------------------
IBM Deutschland
Rathausstr. 7
09111 Chemnitz
Phone: +49 371 6978 2165
Mobile: +49 175 575 2877
E-Mail: uwefalke at de.ibm.com
-------------------------------------------------------------------------------------------------------------------------------------------
IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: 
Thomas Wolter, Sven Schoo?
Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, 
HRB 17122 


gpfsug-discuss-bounces at spectrumscale.org wrote on 01/07/2019 09:25:23:

> From: Michael Hennecke <mhennecke at lenovo.com>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Date: 01/07/2019 09:38
> Subject: [EXTERNAL] Re: [gpfsug-discuss] Any guidelines for choosing
> vdisk size?
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> 
> Ryan,
> 
> That is a question with many variables. Some of the factors 
> influencing the decision are:
> * Are you sharing data and metadata (the GNR default these days), or
> are you building separate data and metadata vdisks (the "older" 
> default, and still useful for some use cases)? If you have separate 
> metadata, you need to ensure you leave enough free space in the RGs 
> to account for metadata growth beyond your initial sizing.
> * Are you creating vdisks for multiple filesystems on the same 
> building block, or just for one filesystem? 
> * How likely is it that space needs to be (re-)allocated between 
> different filesystems?
> * How likely is it that new storage will be added to existing 
filesystems?
> * Is there a requirement to include building blocks of different 
> size (different #disks, or different capacity of disks) in the same 
> filesystem?
> 
> All of these influence the decision on the "best" vdisk quantity and
> capacities. If you have a specific configuration question, I'm happy
> to discuss that offline.
> 
> In general I advise customers to use at least two equally sized 
> vdisks per RG (4 per building block), so there is more flexibility 
> when capacity needs to be moved around. But many sites that have 
> homogeneous building blocks and don't expect changes in their 
> storage configuration just go with a single vdisk per RG (2 per 
> building block).
> 
> Either way, it is a good idea (in environments with more than a 
> single building block) to put aside a small "test" vdisk per RG just
> for performance testing. So you can do performance validations on 
> each building block individually even if your main filesystem spans 
> all building block. Size of that "test" vdisk should be big enough 
> so you can run a few minutes of sequential IOR against it without 
> filling it up 100%.
> 
> 
> Mit freundlichen Gr?ssen / Best regards,
> 
> Michael Hennecke
> HPC Chief Technologist - HPC and AI Business Unit 
> mailto:mhennecke at lenovo.com
> --
> Lenovo Global Technology (Germany) GmbH * Am Zehnthof 77 * D-45307 
> Essen * Germany 
> Gesch?ftsf?hrung: Colm Gleeson, Christophe Laurent * Sitz der 
> Gesellschaft: Stuttgart * HRB-Nr.: 758298, AG Stuttgart
> 
> -----Original Message-----
> From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-
> bounces at spectrumscale.org> On Behalf Of Ryan Novosielski
> Sent: Monday, 1 July, 2019 07:43
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: [External] [gpfsug-discuss] Any guidelines for choosing vdisk 
size?
> 
> Good morning,
> 
> Was wondering if anyone could point me to any tips or provide some 
> regarding choosing vdisk size? My understanding is that too small is
> a waste of resources in the form of overhead, and that there is an 
> upper limit. but that generally within a pool, you want them to be 
> the same size, so that if you aren?t allocating the entire storage 
> space on the system straight off, you?ll need to choose a reasonable
> size or be left with unusable space (eg. you will waste space if you
> went with, say, 200 GB vdisk sizes on a 500 GB array).
> 
> Anyone have any tips? Do people just generally allocate the whole 
> thing and have one 2 vdisks (redundancy)? Seems you have some more 
> flexibility if you don?t do that and have to, say, create a storage 
> pool or filesystem from scratch to take advantage of features in a 
> newer FS version or what have you.
> 
> Thanks for the help ? spent a lot of time looking for this 
> previously, but never asked on the list.
> 
> --
> ____
> || \\UTGERS, |---------------------------*O*---------------------------
> ||_// the State    |         Ryan Novosielski - novosirj at rutgers.edu
> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS 
Campus
> ||  \\    of NJ    | Office of Advanced Research Computing - MSB C630, 
Newark
>      `'
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?
> 
u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-
> 
siA1ZOg&r=fTuVGtgq6A14KiNeaGfNZzOOgtHW5Lm4crZU6lJxtB8&m=XHrjWG2mnps3wxUISQHIZDR08HKkIKdcekRQibivZB4&s=wO1NamNaVBSnk_Kbe7Xl45aGtu7W6CBj0tHOlysTui4&e=
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?
> 
u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-
> 
siA1ZOg&r=fTuVGtgq6A14KiNeaGfNZzOOgtHW5Lm4crZU6lJxtB8&m=XHrjWG2mnps3wxUISQHIZDR08HKkIKdcekRQibivZB4&s=wO1NamNaVBSnk_Kbe7Xl45aGtu7W6CBj0tHOlysTui4&e=
> 




From kkr at lbl.gov  Tue Jul  2 18:45:30 2019
From: kkr at lbl.gov (Kristy Kallback-Rose)
Date: Tue, 2 Jul 2019 10:45:30 -0700
Subject: [gpfsug-discuss] Hold the Date - September 23 and 24
Message-ID: <3F2B08E9-C6E3-412B-9308-D79E3480C5DA@lbl.gov>

Hello,

HPCXXL will be hosted by NERSC (Berkeley, CA) this September. As part of this event, there will be approximately a day and a half on GPFS content. We have done this type of event in the past, and as before, the GPFS days will be free to attend, but you do need to register. 

We?ll have more details soon, mark your calendars. 

Initial details: https://www.spectrumscaleug.org/event/spectrum-scale-gpfs-days-part-of-hpcxxl/ <https://www.spectrumscaleug.org/event/spectrum-scale-gpfs-days-part-of-hpcxxl/>

Best,
Kristy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190702/9747f62d/attachment-0001.htm>

From novosirj at rutgers.edu  Thu Jul  4 04:55:56 2019
From: novosirj at rutgers.edu (Ryan Novosielski)
Date: Thu, 4 Jul 2019 03:55:56 +0000
Subject: [gpfsug-discuss] Combined system (metadata) and data pool on GPFS
	5.x?
Message-ID: <DCA32300-AA3F-4E63-8248-5A241F6D4E4C@rutgers.edu>

Hi there,

Was at the Lenovo GNR/GPFS 5.x seminar the other week and we briefly discussed whether or not the recommendation was to place data in the system pool (eg. have one combined system/data pool), or to have them separate. We?ve got a GSS26 that we are creating new filesystems on for high speed scratch (16 MB blocks).

I unfortunately did not note all of the considerations/caveats related to this choice. Does anyone who knows/was there and can remember remind me?

The recollection I have is that you may have some contention for I/O if your metadata is on your data disks, and you won?t be able to have different redundancy settings. Once upon a time, block size also mattered, but not so much on 5.x.

Thanks!

--
____
|| \\UTGERS,  	 |---------------------------*O*---------------------------
||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
     `'


From S.J.Thompson at bham.ac.uk  Thu Jul  4 09:51:18 2019
From: S.J.Thompson at bham.ac.uk (Simon Thompson)
Date: Thu, 4 Jul 2019 08:51:18 +0000
Subject: [gpfsug-discuss] Combined system (metadata) and data pool on
 GPFS 5.x?
In-Reply-To: <DCA32300-AA3F-4E63-8248-5A241F6D4E4C@rutgers.edu>
References: <DCA32300-AA3F-4E63-8248-5A241F6D4E4C@rutgers.edu>
Message-ID: <205352CF-6035-4DDA-812F-9BD9FA0152B7@bham.ac.uk>

I wasn't but ...

If your metadata is in the same RG/DA as your data, even if it?s a separate pool, the contention is the same. But if you have 2 DAs, then you have more spinning disks underneath to provide iops anyway.

We only separate our metadata and data into pools because we want to replicate the metadata between DSS-G systems, but not data. And also have some data which is replicated but not all, and the only way we could figure to do that is by having separate pools and failure groups. i.e. we didn't want data in the system pool that wasn't replicated being placed on the second site.

And I agree, block sizes don't matter so much anymore, except remember the "variable" sub-blocks are based on the smallest block size used, so we ended up with 16MB blocks for both data and metadata.

Simon

?On 04/07/2019, 04:56, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Ryan Novosielski" <gpfsug-discuss-bounces at spectrumscale.org on behalf of novosirj at rutgers.edu> wrote:

    Hi there,
    
    Was at the Lenovo GNR/GPFS 5.x seminar the other week and we briefly discussed whether or not the recommendation was to place data in the system pool (eg. have one combined system/data pool), or to have them separate. We?ve got a GSS26 that we are creating new filesystems on for high speed scratch (16 MB blocks).
    
    I unfortunately did not note all of the considerations/caveats related to this choice. Does anyone who knows/was there and can remember remind me?
    
    The recollection I have is that you may have some contention for I/O if your metadata is on your data disks, and you won?t be able to have different redundancy settings. Once upon a time, block size also mattered, but not so much on 5.x.
    
    Thanks!
    
    --
    ____
    || \\UTGERS,  	 |---------------------------*O*---------------------------
    ||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
    || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
    ||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
         `'
    
    _______________________________________________
    gpfsug-discuss mailing list
    gpfsug-discuss at spectrumscale.org
    http://gpfsug.org/mailman/listinfo/gpfsug-discuss
    


From damir.krstic at gmail.com  Wed Jul 10 13:16:31 2019
From: damir.krstic at gmail.com (Damir Krstic)
Date: Wed, 10 Jul 2019 07:16:31 -0500
Subject: [gpfsug-discuss] slow filesystem
Message-ID: <CAKV+WqctLVWdr0E5sA+G9pZpc9F16Pc7Q-VT1C81zf6R8gkGmA@mail.gmail.com>

Over last couple of days our reads and writes on our compute cluster are
experiencing real slow reads and writes on one of the filesystems in the
cluster. We are talking with IBM and have Sev. 1 ticket open, but I figured
to ask here about the warning message we are seeing in GPFS logs.

The cluster is configured as following:

4 IO servers in the main gpfs cluster
700+ compute nodes in the gpfs cluster

home filesystem is slow but projects filesystem seems to be fast. Not many
waiters on the IO servers (almost none) but a lot of waiters on the remote
cluster.

The message that is giving us a pause is the following:

Jul 10 07:05:31 gssio4 mmfs: [N] Writing into file
/var/mmfs/gen/LastLeaseRequestSent took 10.5 seconds

Why is taking so long to write to to the local file?

Thanks,
Damir
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190710/5aa0023a/attachment-0001.htm>

From Kevin.Buterbaugh at Vanderbilt.Edu  Wed Jul 10 13:21:52 2019
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Wed, 10 Jul 2019 12:21:52 +0000
Subject: [gpfsug-discuss] slow filesystem
In-Reply-To: <CAKV+WqctLVWdr0E5sA+G9pZpc9F16Pc7Q-VT1C81zf6R8gkGmA@mail.gmail.com>
References: <CAKV+WqctLVWdr0E5sA+G9pZpc9F16Pc7Q-VT1C81zf6R8gkGmA@mail.gmail.com>
Message-ID: <BEF9A8D8-DFFE-4102-8ADF-6A919F9A5094@vanderbilt.edu>

Hi Damir,

Have you checked to see whether gssio4 might have a failing internal HD / SSD?  Thanks?

Kevin

On Jul 10, 2019, at 7:16 AM, Damir Krstic <damir.krstic at gmail.com<mailto:damir.krstic at gmail.com>> wrote:

Over last couple of days our reads and writes on our compute cluster are experiencing real slow reads and writes on one of the filesystems in the cluster. We are talking with IBM and have Sev. 1 ticket open, but I figured to ask here about the warning message we are seeing in GPFS logs.

The cluster is configured as following:

4 IO servers in the main gpfs cluster
700+ compute nodes in the gpfs cluster

home filesystem is slow but projects filesystem seems to be fast. Not many waiters on the IO servers (almost none) but a lot of waiters on the remote cluster.

The message that is giving us a pause is the following:
Jul 10 07:05:31 gssio4 mmfs: [N] Writing into file /var/mmfs/gen/LastLeaseRequestSent took 10.5 seconds

Why is taking so long to write to to the local file?

Thanks,
Damir

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&amp;data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cede76a6c2bd743c836b708d705307a86%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636983578133164831&amp;sdata=ATtqXDDChaZTouZZRkjf%2F79pIK%2Fc1DAwq6KUU%2FKYca4%3D&amp;reserved=0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190710/014c4ff6/attachment-0001.htm>

From damir.krstic at gmail.com  Wed Jul 10 13:22:37 2019
From: damir.krstic at gmail.com (Damir Krstic)
Date: Wed, 10 Jul 2019 07:22:37 -0500
Subject: [gpfsug-discuss] slow filesystem
In-Reply-To: <BEF9A8D8-DFFE-4102-8ADF-6A919F9A5094@vanderbilt.edu>
References: <CAKV+WqctLVWdr0E5sA+G9pZpc9F16Pc7Q-VT1C81zf6R8gkGmA@mail.gmail.com>
	<BEF9A8D8-DFFE-4102-8ADF-6A919F9A5094@vanderbilt.edu>
Message-ID: <CAKV+WqfUQ3wWaKqOB-aVKDRNsvw7SutWT64owkCJvpkndWtWUA@mail.gmail.com>

mmlspdisk all --not-ok does not indicate any failed hard drives.

On Wed, Jul 10, 2019 at 7:22 AM Buterbaugh, Kevin L <
Kevin.Buterbaugh at vanderbilt.edu> wrote:

> Hi Damir,
>
> Have you checked to see whether gssio4 might have a failing internal HD /
> SSD?  Thanks?
>
> Kevin
>
> On Jul 10, 2019, at 7:16 AM, Damir Krstic <damir.krstic at gmail.com> wrote:
>
> Over last couple of days our reads and writes on our compute cluster are
> experiencing real slow reads and writes on one of the filesystems in the
> cluster. We are talking with IBM and have Sev. 1 ticket open, but I figured
> to ask here about the warning message we are seeing in GPFS logs.
>
> The cluster is configured as following:
>
> 4 IO servers in the main gpfs cluster
> 700+ compute nodes in the gpfs cluster
>
> home filesystem is slow but projects filesystem seems to be fast. Not many
> waiters on the IO servers (almost none) but a lot of waiters on the remote
> cluster.
>
> The message that is giving us a pause is the following:
> Jul 10 07:05:31 gssio4 mmfs: [N] Writing into file
> /var/mmfs/gen/LastLeaseRequestSent took 10.5 seconds
>
> Why is taking so long to write to to the local file?
>
> Thanks,
> Damir
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
>
> https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&amp;data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cede76a6c2bd743c836b708d705307a86%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636983578133164831&amp;sdata=ATtqXDDChaZTouZZRkjf%2F79pIK%2Fc1DAwq6KUU%2FKYca4%3D&amp;reserved=0
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190710/dac7390e/attachment-0001.htm>

From valdis.kletnieks at vt.edu  Wed Jul 10 14:14:31 2019
From: valdis.kletnieks at vt.edu (Valdis Kl=?utf-8?Q?=c4=93?=tnieks)
Date: Wed, 10 Jul 2019 09:14:31 -0400
Subject: [gpfsug-discuss] slow filesystem
In-Reply-To: <CAKV+WqfUQ3wWaKqOB-aVKDRNsvw7SutWT64owkCJvpkndWtWUA@mail.gmail.com>
References: <CAKV+WqctLVWdr0E5sA+G9pZpc9F16Pc7Q-VT1C81zf6R8gkGmA@mail.gmail.com>
	<BEF9A8D8-DFFE-4102-8ADF-6A919F9A5094@vanderbilt.edu>
	<CAKV+WqfUQ3wWaKqOB-aVKDRNsvw7SutWT64owkCJvpkndWtWUA@mail.gmail.com>
Message-ID: <5311.1562764471@turing-police>

On Wed, 10 Jul 2019 07:22:37 -0500, Damir Krstic said:

> mmlspdisk all --not-ok does not indicate any failed hard drives.

Not a GPFS drive - if a write to /var is taking 10 seconds, that indicates
that there is likely a problem with the disk(s) that the system lives on,
and you're hitting recoverable write errors....

Jul 10 07:05:31 gssio4 mmfs: [N] Writing into file /var/mmfs/gen/LastLeaseRequestSent took 10.5 seconds

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190710/f532fd53/attachment-0001.sig>

From heinrich.billich at id.ethz.ch  Wed Jul 17 12:41:54 2019
From: heinrich.billich at id.ethz.ch (Billich  Heinrich Rainer (ID SD))
Date: Wed, 17 Jul 2019 11:41:54 +0000
Subject: [gpfsug-discuss] How to prove that data is in inode
Message-ID: <71647F14-8BEC-4E89-A30B-F0D880A81A84@id.ethz.ch>

Hello,

How can I prove that data of a small file is stored in the inode (and not on a data nsd)?

We have a filesystem with 4k inodes on Scale 5.0.2 , but it seems there is no file data in the inodes?

I would expect that 'stat' reports 'Blocks: 0'  for a small file, but I see 'Blocks:1'.

Cheers,

Heiner

I tried

[]# rm -f test; echo hello > test
[]# ls -ls test
1 -rw-r--r-- 1 root root 6 Jul 17 13:11 test
[root at testnas13ems01 test]# stat test
  File: ?test?
  Size: 6         	Blocks: 1          IO Block: 1048576 regular file
Device: 2dh/45d	Inode: 353314      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-07-17 13:11:03.037049000 +0200
Modify: 2019-07-17 13:11:03.037331000 +0200
Change: 2019-07-17 13:11:03.037259319 +0200
 Birth: -
[root at testnas13ems01 test]# du test
1	test
[root at testnas13ems01 test]# du -b test
6	test
[root at testnas13ems01 test]#

Filesystem

# mmlsfs f****
flag                value                    description
------------------- ------------------------ -----------------------------------
 -f                 32768                    Minimum fragment (subblock) size in bytes
 -i                 4096                     Inode size in bytes
 -I                 32768                    Indirect block size in bytes
 -m                 1                        Default number of metadata replicas
 -M                 2                        Maximum number of metadata replicas
 -r                 1                        Default number of data replicas
 -R                 2                        Maximum number of data replicas
 -j                 cluster                  Block allocation type
 -D                 nfs4                     File locking semantics in effect
 -k                 nfs4                     ACL semantics in effect
 -n                 32                       Estimated number of nodes that will mount file system
 -B                 1048576                  Block size
 -Q                 user;group;fileset       Quotas accounting enabled
                    user;group;fileset       Quotas enforced
                    user;group;fileset       Default quotas enabled
 --perfileset-quota Yes                      Per-fileset quota enforcement
 --filesetdf        Yes                      Fileset df enabled?
 -V                 20.01 (5.0.2.0)          Current file system version
                    15.01 (4.2.0.0)          Original file system version
 --create-time      ***** 2017 File system creation time
 -z                 No                       Is DMAPI enabled?
 -L                 33554432                 Logfile size
 -E                 Yes                      Exact mtime mount option
 -S                 relatime                 Suppress atime mount option
 -K                 whenpossible             Strict replica allocation option
 --fastea           Yes                      Fast external attributes enabled?
 --encryption       No                       Encryption enabled?
 --inode-limit      1294592                  Maximum number of inodes in all inode spaces
 --log-replicas     0                        Number of log replicas
 --is4KAligned      Yes                      is4KAligned?
 --rapid-repair     Yes                      rapidRepair enabled?
 --write-cache-threshold 0                   HAWC Threshold (max 65536)
 --subblocks-per-full-block 32               Number of subblocks per full block
 -P                 system;data              Disk storage pools in file system
 --file-audit-log   No                       File Audit Logging enabled?
 --maintenance-mode No                       Maintenance Mode enabled?
 -d                 ******
-A                 yes                      Automatic mount option
 -o                 nfssync,nodev            Additional mount options
 -T                 /****                  Default mount point
 --mount-priority   0                        Mount priority

-- 
=======================
Heinrich Billich
ETH Z?rich
Informatikdienste
Tel.: +41 44 632 72 56
heinrich.billich at id.ethz.ch
========================
 
 


From kums at us.ibm.com  Wed Jul 17 13:37:35 2019
From: kums at us.ibm.com (Kumaran Rajaram)
Date: Wed, 17 Jul 2019 08:37:35 -0400
Subject: [gpfsug-discuss] How to prove that data is in inode
In-Reply-To: <71647F14-8BEC-4E89-A30B-F0D880A81A84@id.ethz.ch>
References: <71647F14-8BEC-4E89-A30B-F0D880A81A84@id.ethz.ch>
Message-ID: <OF776130C0.2EA026CD-ON0025843A.0044FED5-8525843A.00455C19@notes.na.collabserv.com>

Hi,

>> How can I prove that data of a small file is stored in the inode (and
not on a data nsd)?

You may use echo "inode file_inode_number" | tsdbfs fs_device | grep
indirectionLevel and if it points to INODE, then the file is stored in the
inodes

# 4K Inode Size
# mmlsfs gpfs3a | grep 'Inode size'
 -i                 4096                     Inode size in bytes

# Small file
# ls -l /mnt/gpfs3a/hello.txt
-rw-r--r-- 1 root root 6 Jul 17 08:32 /mnt/gpfs3a/hello.txt

# ls -i /mnt/gpfs3a/hello.txt
91649 /mnt/gpfs3a/hello.txt

#File is inlined within Inode
# echo "inode 91649" | tsdbfs gpfs3a | grep indirectionLevel
  indirectionLevel=INODE status=USERFILE

Regards,
-Kums





From:	"Billich  Heinrich Rainer (ID SD)"
            <heinrich.billich at id.ethz.ch>
To:	gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:	07/17/2019 07:49 AM
Subject:	[EXTERNAL] [gpfsug-discuss] How to prove that data is in inode
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



Hello,

How can I prove that data of a small file is stored in the inode (and not
on a data nsd)?

We have a filesystem with 4k inodes on Scale 5.0.2 , but it seems there is
no file data in the inodes?

I would expect that 'stat' reports 'Blocks: 0'  for a small file, but I see
'Blocks:1'.

Cheers,

Heiner

I tried

[]# rm -f test; echo hello > test
[]# ls -ls test
1 -rw-r--r-- 1 root root 6 Jul 17 13:11 test
[root at testnas13ems01 test]# stat test
  File: ?test?
  Size: 6         		 Blocks: 1          IO Block: 1048576 regular
file
Device: 2dh/45d		 Inode: 353314      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-07-17 13:11:03.037049000 +0200
Modify: 2019-07-17 13:11:03.037331000 +0200
Change: 2019-07-17 13:11:03.037259319 +0200
 Birth: -
[root at testnas13ems01 test]# du test
1		 test
[root at testnas13ems01 test]# du -b test
6		 test
[root at testnas13ems01 test]#

Filesystem

# mmlsfs f****
flag                value                    description
------------------- ------------------------
-----------------------------------
 -f                 32768                    Minimum fragment (subblock)
size in bytes
 -i                 4096                     Inode size in bytes
 -I                 32768                    Indirect block size in bytes
 -m                 1                        Default number of metadata
replicas
 -M                 2                        Maximum number of metadata
replicas
 -r                 1                        Default number of data
replicas
 -R                 2                        Maximum number of data
replicas
 -j                 cluster                  Block allocation type
 -D                 nfs4                     File locking semantics in
effect
 -k                 nfs4                     ACL semantics in effect
 -n                 32                       Estimated number of nodes that
will mount file system
 -B                 1048576                  Block size
 -Q                 user;group;fileset       Quotas accounting enabled
                    user;group;fileset       Quotas enforced
                    user;group;fileset       Default quotas enabled
 --perfileset-quota Yes                      Per-fileset quota enforcement
 --filesetdf        Yes                      Fileset df enabled?
 -V                 20.01 (5.0.2.0)          Current file system version
                    15.01 (4.2.0.0)          Original file system version
 --create-time      ***** 2017 File system creation time
 -z                 No                       Is DMAPI enabled?
 -L                 33554432                 Logfile size
 -E                 Yes                      Exact mtime mount option
 -S                 relatime                 Suppress atime mount option
 -K                 whenpossible             Strict replica allocation
option
 --fastea           Yes                      Fast external attributes
enabled?
 --encryption       No                       Encryption enabled?
 --inode-limit      1294592                  Maximum number of inodes in
all inode spaces
 --log-replicas     0                        Number of log replicas
 --is4KAligned      Yes                      is4KAligned?
 --rapid-repair     Yes                      rapidRepair enabled?
 --write-cache-threshold 0                   HAWC Threshold (max 65536)
 --subblocks-per-full-block 32               Number of subblocks per full
block
 -P                 system;data              Disk storage pools in file
system
 --file-audit-log   No                       File Audit Logging enabled?
 --maintenance-mode No                       Maintenance Mode enabled?
 -d                 ******
-A                 yes                      Automatic mount option
 -o                 nfssync,nodev            Additional mount options
 -T                 /****                  Default mount point
 --mount-priority   0                        Mount priority

--
=======================
Heinrich Billich
ETH Z?rich
Informatikdienste
Tel.: +41 44 632 72 56
heinrich.billich at id.ethz.ch
========================



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=McIf98wfiVqHU8ZygezLrQ&m=VUhXK3bFtfQfiPegK_nm2SStc9jabkxIAUQruQxyXdI&s=JobCqlXTY87VyE6RxErKI46SdzyDhW5kvZKc9xw6DHM&e=




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190717/da506a92/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190717/da506a92/attachment-0001.gif>

From p.ward at nhm.ac.uk  Wed Jul 17 14:00:16 2019
From: p.ward at nhm.ac.uk (Paul Ward)
Date: Wed, 17 Jul 2019 13:00:16 +0000
Subject: [gpfsug-discuss] Break-out session for new user and
	prospects	[London Usergroup]
In-Reply-To: <776770B2-5F84-4462-B900-58EBB982DC1C@spectrumscale.org>
References: <776770B2-5F84-4462-B900-58EBB982DC1C@spectrumscale.org>
Message-ID: <DB7PR10MB2201EAAA629BB9C10BBE24E0D3C90@DB7PR10MB2201.EURPRD10.PROD.OUTLOOK.COM>

Hi Simon,

Was this a slide presentation? Is it available?

Kindest regards,
Paul

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (Spectrum Scale User Group Chair)
Sent: 30 April 2019 10:25
To: gpfsug-discuss at spectrumscale.org
Subject: [gpfsug-discuss] Break-out session for new user and prospects [London Usergroup]

Hi all,

We know that a lot of the talks at the user groups are for experienced users, following feedback from the USA user group, we thought we?d advertise that this year we?re planning to run a break-out for new users on day 1.

Break-out session for new user and prospects (Wed May 8th, 13:00 - 16:45)

This year we will offer a break-out session for new Spectrum Scale user and prospects to get started with Spectrum Scale. In this session we will cover Spectrum Scale Use Cases, the architecture of a Spectrum Scale environment, and discuss how the manifold Spectrum Scale features support the different use case. Please inform customers and colleagues who are interested to learn about Spectrum Scale to grab one of the last seats.

Registration link:
https://www.spectrumscaleug.org/event/uk-user-group-meeting/

There?s just a couple of places left for the usergroup, so please do share and register if you plan to attend.

Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190717/b7294b4c/attachment-0001.htm>

From S.J.Thompson at bham.ac.uk  Wed Jul 17 14:10:21 2019
From: S.J.Thompson at bham.ac.uk (Simon Thompson)
Date: Wed, 17 Jul 2019 13:10:21 +0000
Subject: [gpfsug-discuss] Break-out session for new user and prospects
 [London Usergroup]
In-Reply-To: <DB7PR10MB2201EAAA629BB9C10BBE24E0D3C90@DB7PR10MB2201.EURPRD10.PROD.OUTLOOK.COM>
References: <776770B2-5F84-4462-B900-58EBB982DC1C@spectrumscale.org>
	<DB7PR10MB2201EAAA629BB9C10BBE24E0D3C90@DB7PR10MB2201.EURPRD10.PROD.OUTLOOK.COM>
Message-ID: <6916EC29-2057-4396-A285-7A905DB215EE@bham.ac.uk>

Hi Paul,

There was a slide deck, I?ve asked the IBMers if it?s something we can share.

Simon

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Paul Ward <p.ward at nhm.ac.uk>
Reply-To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Date: Wednesday, 17 July 2019 at 14:01
To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Break-out session for new user and prospects [London Usergroup]

Hi Simon,

Was this a slide presentation? Is it available?

Kindest regards,
Paul

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (Spectrum Scale User Group Chair)
Sent: 30 April 2019 10:25
To: gpfsug-discuss at spectrumscale.org
Subject: [gpfsug-discuss] Break-out session for new user and prospects [London Usergroup]

Hi all,

We know that a lot of the talks at the user groups are for experienced users, following feedback from the USA user group, we thought we?d advertise that this year we?re planning to run a break-out for new users on day 1.

Break-out session for new user and prospects (Wed May 8th, 13:00 - 16:45)

This year we will offer a break-out session for new Spectrum Scale user and prospects to get started with Spectrum Scale. In this session we will cover Spectrum Scale Use Cases, the architecture of a Spectrum Scale environment, and discuss how the manifold Spectrum Scale features support the different use case. Please inform customers and colleagues who are interested to learn about Spectrum Scale to grab one of the last seats.

Registration link:
https://www.spectrumscaleug.org/event/uk-user-group-meeting/

There?s just a couple of places left for the usergroup, so please do share and register if you plan to attend.

Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190717/386a9e5b/attachment-0001.htm>

From leonardo.sala at psi.ch  Wed Jul 17 15:01:06 2019
From: leonardo.sala at psi.ch (Leonardo Sala)
Date: Wed, 17 Jul 2019 16:01:06 +0200
Subject: [gpfsug-discuss] Mistakenly AFM resync triggered: should I stop it?
Message-ID: <e69518bc-d525-fa95-d271-e3e8e81ebcf5@psi.ch>

Dear all,

I do have a possibly stupid question, based on something stupid I did. 
During a procedure to recover AFM transfers (which stopped for still to 
be investigated reasons), I triggered by mistake a resync operation on 
an healthy SW fileset (state: Inactive). The AFM cache in this case is 
big, so this has triggered a big backlog and quite some network 
activity. afmPrefetchThreshold is set to 0, so I am not too scared about 
partially zero-filled data files. But what would it be the best course 
of action now? Should I just let it run (ETA: ~20 hours, give or take), 
or try to stop it? Also, given that this is some experimental data, what 
would the safest course of action?

Thanks a lot!

cheers

leo


-- 
Paul Scherrer Institut
Dr. Leonardo Sala
Group Leader High Performance Computing
Deputy Section Head Science IT
Science IT
WHGA/106
5232 Villigen PSI
Switzerland

Phone: +41 56 310 3369
leonardo.sala at psi.ch
www.psi.ch

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190717/fbaaae4f/attachment-0001.htm>

From bbanister at jumptrading.com  Wed Jul 17 18:24:12 2019
From: bbanister at jumptrading.com (Bryan Banister)
Date: Wed, 17 Jul 2019 17:24:12 +0000
Subject: [gpfsug-discuss] GPFS quota inode/files maximum setting of
	2147483647
Message-ID: <DM6PR14MB2890B7F5B0283B4444C68BAEA2C90@DM6PR14MB2890.namprd14.prod.outlook.com>

Thought this might be of interest to others on the list.

IBM, are there any plans to increase the maximum files setting for quotas over 2147483647?

Thanks,
-Bryan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190717/e373b695/attachment-0001.htm>

From vpuvvada at in.ibm.com  Thu Jul 18 07:41:08 2019
From: vpuvvada at in.ibm.com (Venkateswara R Puvvada)
Date: Thu, 18 Jul 2019 12:11:08 +0530
Subject: [gpfsug-discuss] Mistakenly AFM resync triggered: should I stop
	it?
In-Reply-To: <e69518bc-d525-fa95-d271-e3e8e81ebcf5@psi.ch>
References: <e69518bc-d525-fa95-d271-e3e8e81ebcf5@psi.ch>
Message-ID: <OF11AFAFD0.1AB097A2-ON6525843B.0022AC38-6525843B.0024B9A2@notes.na.collabserv.com>

Hi,

You can stop the AFM resync if it is not required.

mmafmctl device dropPending -j fileset
mmafmctl device resetResync -j fileset

Next access to the fileset would run the recovery and impact should be 
minimal as it recovers only the data which was not replicated.

~Venkat (vpuvvada at in.ibm.com)



From:   Leonardo Sala <leonardo.sala at psi.ch>
To:     "gpfsug-discuss at spectrumscale.org" 
<gpfsug-discuss at spectrumscale.org>
Date:   07/17/2019 07:32 PM
Subject:        [EXTERNAL] [gpfsug-discuss] Mistakenly AFM resync 
triggered: should I stop it?
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Dear all,
I do have a possibly stupid question, based on something stupid I did. 
During a procedure to recover AFM transfers (which stopped for still to be 
investigated reasons), I triggered by mistake a resync operation on an 
healthy SW fileset (state: Inactive). The AFM cache in this case is big, 
so this has triggered a big backlog and quite some network activity. 
afmPrefetchThreshold is set to 0, so I am not too scared about partially 
zero-filled data files. But what would it be the best course of action 
now? Should I just let it run (ETA: ~20 hours, give or take), or try to 
stop it? Also, given that this is some experimental data, what would the 
safest course of action?
Thanks a lot!
cheers
leo

-- 
Paul Scherrer Institut
Dr. Leonardo Sala
Group Leader High Performance Computing
Deputy Section Head Science IT
Science IT
WHGA/106
5232 Villigen PSI
Switzerland

Phone: +41 56 310 3369
leonardo.sala at psi.ch
www.psi.ch_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=MapU9QQbtQx-G9hGAGBPzuB-Qxkl0HONn6ByIz4nXoI&s=F_iFVjqia2agf0fTNncWoaZqmKbhSCO-opqFb6cjG6A&e= 





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190718/748db732/attachment-0001.htm>

From heinrich.billich at id.ethz.ch  Thu Jul 18 15:15:12 2019
From: heinrich.billich at id.ethz.ch (Billich  Heinrich Rainer (ID SD))
Date: Thu, 18 Jul 2019 14:15:12 +0000
Subject: [gpfsug-discuss] How to prove that data is in inode
In-Reply-To: <OF776130C0.2EA026CD-ON0025843A.0044FED5-8525843A.00455C19@notes.na.collabserv.com>
References: <71647F14-8BEC-4E89-A30B-F0D880A81A84@id.ethz.ch>
	<OF776130C0.2EA026CD-ON0025843A.0044FED5-8525843A.00455C19@notes.na.collabserv.com>
Message-ID: <A5096632-88B9-4A6B-9C22-E8DC0884987F@id.ethz.ch>

Hello Kums,

Thank you; I could verify that the data of small files is in the inode. In the table below you see the filesize and the result of the tsdbfs query. Up to 3k all data is in the inode.

The result of stat calls on small files is very puzzling. When data is in the inode stat reports one block of 512 bytes used, even for 3k of data. I don?t see how this would affect any of our applications, so it?s just something to note.

I append some output and the script to generate it, just for completeness.

Cheers,

Heiner

# stat -c "inode %i" $f | tsdbfs fs1301 | grep indirectionLevel
0  indirectionLevel=DIRECT status=USERFILE. <empty file is special>
1  indirectionLevel=INODE status=USERFILE
16  indirectionLevel=INODE status=USERFILE
512  indirectionLevel=INODE status=USERFILE
1k  indirectionLevel=INODE status=USERFILE
2k  indirectionLevel=INODE status=USERFILE
3k  indirectionLevel=INODE status=USERFILE.  < up to 3k data is in inode >

4k  indirectionLevel=DIRECT status=USERFILE
16k  indirectionLevel=DIRECT status=USERFILE
64k  indirectionLevel=DIRECT status=USERFILE
1M  indirectionLevel=DIRECT status=USERFILE
2M  indirectionLevel=DIRECT status=USERFILE


Stat output

# stat -c ?%n size: %s  allocated: %b*%B?

# stat -c %n size: %s  allocated: %b*%B
0 size: 0  allocated: 0*512
1 size: 1  allocated: 1*512
16 size: 16  allocated: 1*512
512 size: 512  allocated: 1*512
1k size: 1024  allocated: 1*512
2k size: 2048  allocated: 1*512
3k size: 3072  allocated: 1*512  < 3k file and  all data in inode:  stat reports 1*512 allocated/used)
4k size: 4096  allocated: 64*512
16k size: 16384  allocated: 64*512.   < as expected, 32k subblock size >
64k size: 65536  allocated: 128*512
1M size: 1048576  allocated: 2048*512
2M size: 2097152  allocated: 4096*512

The script:

# test-data-in-inode.sh
sizes="0 1 16 512 1k 2k 3k 4k 16k 64k 1M 2M"

echo create files
for s in $sizes
do
  head -c $s /dev/zero > $s
done

echo sleep 20. # give gpfs some time to update metadata
sleep 20

echo
echo "# ls -ls $sizes"
ls -ls $sizes

echo
echo "# stat -c %n size: %s  allocated: %b*%B"
stat -c "%n size: %s  allocated: %b*%B" $sizes

echo
echo '# stat -c "inode %i" $f | tsdbfs fs1301 | grep indirectionLevel'
for f in $sizes
do
echo -n $f
stat -c "inode %i" $f | tsdbfs fs1301 | grep indirectionLevel
done

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Kumaran Rajaram <kums at us.ibm.com>
Reply to: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Wednesday, 17 July 2019 at 14:38
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] How to prove that data is in inode


Hi,

>> How can I prove that data of a small file is stored in the inode (and not on a data nsd)?

You may use echo "inode file_inode_number" | tsdbfs fs_device | grep indirectionLevel and if it points to INODE, then the file is stored in the inodes

# 4K Inode Size
# mmlsfs gpfs3a | grep 'Inode size'
-i 4096 Inode size in bytes

# Small file
# ls -l /mnt/gpfs3a/hello.txt
-rw-r--r-- 1 root root 6 Jul 17 08:32 /mnt/gpfs3a/hello.txt

# ls -i /mnt/gpfs3a/hello.txt
91649 /mnt/gpfs3a/hello.txt

#File is inlined within Inode
# echo "inode 91649" | tsdbfs gpfs3a | grep indirectionLevel
indirectionLevel=INODE status=USERFILE

Regards,
-Kums



[Inactive hide details for "Billich  Heinrich Rainer (ID SD)" ---07/17/2019 07:49:56 AM---Hello, How can I prove that data of a]"Billich Heinrich Rainer (ID SD)" ---07/17/2019 07:49:56 AM---Hello, How can I prove that data of a small file is stored in the inode (and not on a data nsd)?

From: "Billich Heinrich Rainer (ID SD)" <heinrich.billich at id.ethz.ch>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: 07/17/2019 07:49 AM
Subject: [EXTERNAL] [gpfsug-discuss] How to prove that data is in inode
Sent by: gpfsug-discuss-bounces at spectrumscale.org

________________________________



Hello,

How can I prove that data of a small file is stored in the inode (and not on a data nsd)?

We have a filesystem with 4k inodes on Scale 5.0.2 , but it seems there is no file data in the inodes?

I would expect that 'stat' reports 'Blocks: 0'  for a small file, but I see 'Blocks:1'.

Cheers,

Heiner

I tried

[]# rm -f test; echo hello > test
[]# ls -ls test
1 -rw-r--r-- 1 root root 6 Jul 17 13:11 test
[root at testnas13ems01 test]# stat test
 File: ?test?
 Size: 6         Blocks: 1          IO Block: 1048576 regular file
Device: 2dh/45d Inode: 353314      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-07-17 13:11:03.037049000 +0200
Modify: 2019-07-17 13:11:03.037331000 +0200
Change: 2019-07-17 13:11:03.037259319 +0200
Birth: -
[root at testnas13ems01 test]# du test
1 test
[root at testnas13ems01 test]# du -b test
6 test
[root at testnas13ems01 test]#

Filesystem

# mmlsfs f****
flag                value                    description
------------------- ------------------------ -----------------------------------
-f                 32768                    Minimum fragment (subblock) size in bytes
-i                 4096                     Inode size in bytes
-I                 32768                    Indirect block size in bytes
-m                 1                        Default number of metadata replicas
-M                 2                        Maximum number of metadata replicas
-r                 1                        Default number of data replicas
-R                 2                        Maximum number of data replicas
-j                 cluster                  Block allocation type
-D                 nfs4                     File locking semantics in effect
-k                 nfs4                     ACL semantics in effect
-n                 32                       Estimated number of nodes that will mount file system
-B                 1048576                  Block size
-Q                 user;group;fileset       Quotas accounting enabled
                   user;group;fileset       Quotas enforced
                   user;group;fileset       Default quotas enabled
--perfileset-quota Yes                      Per-fileset quota enforcement
--filesetdf        Yes                      Fileset df enabled?
-V                 20.01 (5.0.2.0)          Current file system version
                   15.01 (4.2.0.0)          Original file system version
--create-time      ***** 2017 File system creation time
-z                 No                       Is DMAPI enabled?
-L                 33554432                 Logfile size
-E                 Yes                      Exact mtime mount option
-S                 relatime                 Suppress atime mount option
-K                 whenpossible             Strict replica allocation option
--fastea           Yes                      Fast external attributes enabled?
--encryption       No                       Encryption enabled?
--inode-limit      1294592                  Maximum number of inodes in all inode spaces
--log-replicas     0                        Number of log replicas
--is4KAligned      Yes                      is4KAligned?
--rapid-repair     Yes                      rapidRepair enabled?
--write-cache-threshold 0                   HAWC Threshold (max 65536)
--subblocks-per-full-block 32               Number of subblocks per full block
-P                 system;data              Disk storage pools in file system
--file-audit-log   No                       File Audit Logging enabled?
--maintenance-mode No                       Maintenance Mode enabled?
-d                 ******
-A                 yes                      Automatic mount option
-o                 nfssync,nodev            Additional mount options
-T                 /****                  Default mount point
--mount-priority   0                        Mount priority

--
=======================
Heinrich Billich
ETH Z?rich
Informatikdienste
Tel.: +41 44 632 72 56
heinrich.billich at id.ethz.ch
========================



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190718/d044120a/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.gif
Type: image/gif
Size: 106 bytes
Desc: image001.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190718/d044120a/attachment-0001.gif>

From sxiao at us.ibm.com  Thu Jul 18 16:03:38 2019
From: sxiao at us.ibm.com (Steve Xiao)
Date: Thu, 18 Jul 2019 11:03:38 -0400
Subject: [gpfsug-discuss] How to prove that data is in inode
In-Reply-To: <mailman.267.1563459398.1289.gpfsug-discuss@spectrumscale.org>
References: <mailman.267.1563459398.1289.gpfsug-discuss@spectrumscale.org>
Message-ID: <OF10D4C475.21C37EF2-ON8525843B.00521027-8525843B.0052BAEF@notes.na.collabserv.com>

I believe GPFS report  1 block used in stat so the file will not be 
treated as empty by applications.    If i remember correctly, some backup 
programs was skipping small files with data in inode because GPFS was 
returning 0 block used for these small files. 

Steve Y. Xiao



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190718/6df9881d/attachment-0001.htm>

From novosirj at rutgers.edu  Thu Jul 18 22:15:52 2019
From: novosirj at rutgers.edu (Ryan Novosielski)
Date: Thu, 18 Jul 2019 21:15:52 +0000
Subject: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
In-Reply-To: <OFE9B5AE08.76EA4744-ON002582F2.003C63C0-C12582F2.00410D62@notes.na.collabserv.com>
References: <OF67F2CFA5.B278755D-ON002582F2.002CFDE9-C12582F2.0030AC8F@notes.na.collabserv.com>
	<CAAxuGpE4fM+X1Gwc-OySE+ZWc_jygx5TACusAP4rO0BWNgtaKA@mail.gmail.com>
	<OF237C2868.64DA65AF-ON002582F2.003784F8-002582F2.0038210B@notes.na.collabserv.com>
	<OFE9B5AE08.76EA4744-ON002582F2.003C63C0-C12582F2.00410D62@notes.na.collabserv.com>
Message-ID: <225AEBED-69EF-4E3C-9A67-5A9CB04B4309@rutgers.edu>

I happened across this message because I?ve already done this in the past and was trying to figure out how I did it (apparently didn?t write it down).

Most of it appeared to be adding to /etc/sysconfig/gpfsgui the following:

HTTP_PORT=8080
HTTPS_PORT=8443

?but that hasn?t completely done it yet. Going to have a look and see what else I might need to do.

--
____
|| \\UTGERS,  	 |---------------------------*O*---------------------------
||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
     `'

> On Aug 23, 2018, at 7:50 AM, Markus Rohwedder <rohwedder at de.ibm.com> wrote:
> 
> Hello Juri, Keith,
> 
> thank you for your responses.
> 
> The internal services communicate on the privileged ports, for backwards compatibility and firewall simplicity reasons. We can not just assume all nodes in the cluster are at the latest level.
> 
> Running two services at the same port on different IP addresses could be an option to consider for co-existance of the GUI and another service on the same node.
> However we have not set up, tested nor documented such a configuration as of today. 
> 
> Currently the GUI service manages the iptables redirect bring up and tear down.
> If this would be managed externally it would be possible to bind services to specific ports based on specific IPs.
> 
> In order to create custom redirect rules based on IP address it is necessary to instruct the GUI to 
> - not check for already used ports when the GUI service tries to start up
> - don't create/destroy port forwarding rules during GUI service start and stop.
> This GUI behavior can be configured using the internal flag UPDATE_IPTABLES in the service configuration with the 5.0.1.2 GUI code level.
> 
> The service configuration is not stored in the cluster configuration and may be overwritten during code upgrades, so these settings may have to be added again after an upgrade.
> 
> See this KC link:
> https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.1/com.ibm.spectrum.scale.v5r01.doc/bl1adv_firewallforgui.htm
> 
> Mit freundlichen Gr??en / Kind regards
> 
> Dr. Markus Rohwedder
> 
> Spectrum Scale GUI Development
> <ecblank.gif>
> Phone:	+49 7034 6430190	IBM Deutschland Research & Development	
> <17153317.gif>
> E-Mail:	rohwedder at de.ibm.com	Am Weiher 24
> <ecblank.gif>	<ecblank.gif>	65451 Kelsterbach
> <ecblank.gif>	<ecblank.gif>	Germany
> <ecblank.gif>
> 
> <graycol.gif>"Daniel Kidger" ---23.08.2018 12:13:36---Keith, I have another IBM customer who also wished to move Scale GUI's https ports. In their case
> 
> From:  "Daniel Kidger" <daniel.kidger at uk.ibm.com>
> To:  gpfsug-discuss at spectrumscale.org
> Cc:  gpfsug-discuss at spectrumscale.org
> Date:  23.08.2018 12:13
> Subject:  Re: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
> Sent by:  gpfsug-discuss-bounces at spectrumscale.org
> 
> 
> 
> 
> Keith,
> 
> I have another IBM customer who also wished to move Scale GUI's https ports.
> In their case because they had their own web based management interface on the same https port.
> Is this the same reason that you have?
> If so I wonder how many other sites have the same issue?
> 
> One workaround that was suggested at the time, was to add a second IP address to the node (piggy-backing on 'eth0').
> Then run the two different GUIs, one per IP address.
> Is this an option, albeit a little ugly?
> Daniel
> 
> <17310450.gif>				Dr Daniel Kidger
> IBM Technical Sales Specialist
> Software Defined Solution Sales
> 
> +44-(0)7818 522 266 
> daniel.kidger at uk.ibm.com
> 
> 
> 
> ----- Original message -----
> From: "Markus Rohwedder" <rohwedder at de.ibm.com>
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Cc:
> Subject: Re: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
> Date: Thu, Aug 23, 2018 9:51 AM
> Hello Keith,
> 
> it is not so easy.
> 
> The GUI receives events from other scale components using the currently defined ports.
> Changing the GUI ports will cause breakage in the GUI stack at several places (internal watchdog functions, interlock with health events, interlock with CES).
> Therefore at this point there is no procedure to change this behaviour across all components.
> 
> Because the GUI service does not run as root. the GUI server does not serve the privileged ports 80 and 443 directly but rather 47443 and 47080.
> Tweaking the ports in the server.xml file will only change the native ports that the GUI uses.
> The GUI manages IPTABLES rules to forward ports 443 and 80 to 47443 and 47080. 
> If these ports are already used by another service, the GUI will not start up.
> 
> Making the GUI ports freely configurable is therefore not a strightforward change, and currently no on our roadmap.
> If you want to emphasize your case as future development item, please let me know.
> 
> I would also be interested in:
> > Scale version you are running
> > Do you need port 80 or 443 as well?
> > Would it work for you if the xCAT service was bound to a single IP address?
> 
> Mit freundlichen Gr??en / Kind regards
> 
> Dr. Markus Rohwedder
> 
> Spectrum Scale GUI Development
> 
> <ecblank.gif>
> Phone:	+49 7034 6430190	IBM Deutschland Research & Development	
> <17153317.gif>
> E-Mail:	rohwedder at de.ibm.com	Am Weiher 24
> <ecblank.gif>	<ecblank.gif>	65451 Kelsterbach
> <ecblank.gif>	<ecblank.gif>	Germany
> <ecblank.gif>
> 
> <graycol.gif>Keith Ball ---22.08.2018 21:33:25---Hello All, Does anyone know how to change the HTTP ports for the Spectrum Scale GUI?
> 
> From: Keith Ball <bipcuds at gmail.com>
> To: gpfsug-discuss at spectrumscale.org
> Date: 22.08.2018 21:33
> Subject: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> 
> 
> 
> 
> Hello All,
> 
> Does anyone know how to change the HTTP ports for the Spectrum Scale GUI? Any documentation or RedPaper I have found deftly avoids discussing this. The most promising thing I see is in /opt/ibm/wlp/usr/servers/gpfsgui/server.xml:
> 
> <httpEndpoint id="defaultHttpEndpoint" host="*" httpPort="47080" httpsPort="47443">
> <tcpOptions soReuseAddr="true"/>
> </httpEndpoint>
> 
> but it appears that port 80 specifically is used also by the GUI's Web service. I already have an HTTP server using port 80 for provisioning (xCAT), so would rather change the Specturm Scale GUI configuration if I can.
> 
> Many Thanks,
> Keith
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number 741598. 
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss


From knop at us.ibm.com  Thu Jul 18 22:24:50 2019
From: knop at us.ibm.com (Felipe Knop)
Date: Thu, 18 Jul 2019 17:24:50 -0400
Subject: [gpfsug-discuss]
 =?utf-8?q?GPFS_quota_inode/files_maximum_setting?=
 =?utf-8?q?_of=092147483647?=
In-Reply-To: <DM6PR14MB2890B7F5B0283B4444C68BAEA2C90@DM6PR14MB2890.namprd14.prod.outlook.com>
References: <DM6PR14MB2890B7F5B0283B4444C68BAEA2C90@DM6PR14MB2890.namprd14.prod.outlook.com>
Message-ID: <OFD656CADB.20E2D1BC-ON0025843B.006EB6EC-8525843B.0075A180@notes.na.collabserv.com>


Bryan,

Could you open an RFE for this item?  We took an initial look at what it
will take to raise this limit and found that it will require an RPC format
change. So it's something that would need to go through a normal release
cycle.

Thanks and regards,

  Felipe

----
Felipe Knop                                     knop at us.ibm.com
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601
(845) 433-9314  T/L 293-9314





From:	Bryan Banister <bbanister at jumptrading.com>
To:	gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:	07/17/2019 01:24 PM
Subject:	[EXTERNAL] [gpfsug-discuss] GPFS quota inode/files maximum
            setting of	2147483647
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



Thought this might be of interest to others on the list.

IBM, are there any plans to increase the maximum files setting for quotas
over 2147483647?

Thanks,
-Bryan_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=oNT2koCZX0xmWlSlLblR9Q&m=gA7MMIPzQmpVqvoaVl9D2jeytjZOXtkGikPGPpNg_HU&s=pdkFa7ir9MdaAdAuMic15m4-cbL4FqlsESaTYOnxAq4&e=



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190718/374e3cc2/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190718/374e3cc2/attachment-0001.gif>

From novosirj at rutgers.edu  Thu Jul 18 22:26:50 2019
From: novosirj at rutgers.edu (Ryan Novosielski)
Date: Thu, 18 Jul 2019 21:26:50 +0000
Subject: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
In-Reply-To: <225AEBED-69EF-4E3C-9A67-5A9CB04B4309@rutgers.edu>
References: <OF67F2CFA5.B278755D-ON002582F2.002CFDE9-C12582F2.0030AC8F@notes.na.collabserv.com>
	<CAAxuGpE4fM+X1Gwc-OySE+ZWc_jygx5TACusAP4rO0BWNgtaKA@mail.gmail.com>
	<OF237C2868.64DA65AF-ON002582F2.003784F8-002582F2.0038210B@notes.na.collabserv.com>
	<OFE9B5AE08.76EA4744-ON002582F2.003C63C0-C12582F2.00410D62@notes.na.collabserv.com>
	<225AEBED-69EF-4E3C-9A67-5A9CB04B4309@rutgers.edu>
Message-ID: <B5538DB3-4625-43DA-B8F9-E600F48054B4@rutgers.edu>

Nope, that appears to be all of it. I also had a problem with the postgresql service, which was why the gpfsgui wouldn?t start. But once I fixed that, I can log in on https://<GUI_HOST>:8443.

HTH.

> On Jul 18, 2019, at 5:15 PM, Ryan Novosielski <novosirj at rutgers.edu> wrote:
> 
> I happened across this message because I?ve already done this in the past and was trying to figure out how I did it (apparently didn?t write it down).
> 
> Most of it appeared to be adding to /etc/sysconfig/gpfsgui the following:
> 
> HTTP_PORT=8080
> HTTPS_PORT=8443
> 
> ?but that hasn?t completely done it yet. Going to have a look and see what else I might need to do.
> 
> --
> ____
> || \\UTGERS,  	 |---------------------------*O*---------------------------
> ||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
> ||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
>     `'
> 
>> On Aug 23, 2018, at 7:50 AM, Markus Rohwedder <rohwedder at de.ibm.com> wrote:
>> 
>> Hello Juri, Keith,
>> 
>> thank you for your responses.
>> 
>> The internal services communicate on the privileged ports, for backwards compatibility and firewall simplicity reasons. We can not just assume all nodes in the cluster are at the latest level.
>> 
>> Running two services at the same port on different IP addresses could be an option to consider for co-existance of the GUI and another service on the same node.
>> However we have not set up, tested nor documented such a configuration as of today. 
>> 
>> Currently the GUI service manages the iptables redirect bring up and tear down.
>> If this would be managed externally it would be possible to bind services to specific ports based on specific IPs.
>> 
>> In order to create custom redirect rules based on IP address it is necessary to instruct the GUI to 
>> - not check for already used ports when the GUI service tries to start up
>> - don't create/destroy port forwarding rules during GUI service start and stop.
>> This GUI behavior can be configured using the internal flag UPDATE_IPTABLES in the service configuration with the 5.0.1.2 GUI code level.
>> 
>> The service configuration is not stored in the cluster configuration and may be overwritten during code upgrades, so these settings may have to be added again after an upgrade.
>> 
>> See this KC link:
>> https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.1/com.ibm.spectrum.scale.v5r01.doc/bl1adv_firewallforgui.htm
>> 
>> Mit freundlichen Gr??en / Kind regards
>> 
>> Dr. Markus Rohwedder
>> 
>> Spectrum Scale GUI Development
>> <ecblank.gif>
>> Phone:	+49 7034 6430190	IBM Deutschland Research & Development	
>> <17153317.gif>
>> E-Mail:	rohwedder at de.ibm.com	Am Weiher 24
>> <ecblank.gif>	<ecblank.gif>	65451 Kelsterbach
>> <ecblank.gif>	<ecblank.gif>	Germany
>> <ecblank.gif>
>> 
>> <graycol.gif>"Daniel Kidger" ---23.08.2018 12:13:36---Keith, I have another IBM customer who also wished to move Scale GUI's https ports. In their case
>> 
>> From:  "Daniel Kidger" <daniel.kidger at uk.ibm.com>
>> To:  gpfsug-discuss at spectrumscale.org
>> Cc:  gpfsug-discuss at spectrumscale.org
>> Date:  23.08.2018 12:13
>> Subject:  Re: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
>> Sent by:  gpfsug-discuss-bounces at spectrumscale.org
>> 
>> 
>> 
>> 
>> Keith,
>> 
>> I have another IBM customer who also wished to move Scale GUI's https ports.
>> In their case because they had their own web based management interface on the same https port.
>> Is this the same reason that you have?
>> If so I wonder how many other sites have the same issue?
>> 
>> One workaround that was suggested at the time, was to add a second IP address to the node (piggy-backing on 'eth0').
>> Then run the two different GUIs, one per IP address.
>> Is this an option, albeit a little ugly?
>> Daniel
>> 
>> <17310450.gif>				Dr Daniel Kidger
>> IBM Technical Sales Specialist
>> Software Defined Solution Sales
>> 
>> +44-(0)7818 522 266 
>> daniel.kidger at uk.ibm.com
>> 
>> 
>> 
>> ----- Original message -----
>> From: "Markus Rohwedder" <rohwedder at de.ibm.com>
>> Sent by: gpfsug-discuss-bounces at spectrumscale.org
>> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
>> Cc:
>> Subject: Re: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
>> Date: Thu, Aug 23, 2018 9:51 AM
>> Hello Keith,
>> 
>> it is not so easy.
>> 
>> The GUI receives events from other scale components using the currently defined ports.
>> Changing the GUI ports will cause breakage in the GUI stack at several places (internal watchdog functions, interlock with health events, interlock with CES).
>> Therefore at this point there is no procedure to change this behaviour across all components.
>> 
>> Because the GUI service does not run as root. the GUI server does not serve the privileged ports 80 and 443 directly but rather 47443 and 47080.
>> Tweaking the ports in the server.xml file will only change the native ports that the GUI uses.
>> The GUI manages IPTABLES rules to forward ports 443 and 80 to 47443 and 47080. 
>> If these ports are already used by another service, the GUI will not start up.
>> 
>> Making the GUI ports freely configurable is therefore not a strightforward change, and currently no on our roadmap.
>> If you want to emphasize your case as future development item, please let me know.
>> 
>> I would also be interested in:
>>> Scale version you are running
>>> Do you need port 80 or 443 as well?
>>> Would it work for you if the xCAT service was bound to a single IP address?
>> 
>> Mit freundlichen Gr??en / Kind regards
>> 
>> Dr. Markus Rohwedder
>> 
>> Spectrum Scale GUI Development
>> 
>> <ecblank.gif>
>> Phone:	+49 7034 6430190	IBM Deutschland Research & Development	
>> <17153317.gif>
>> E-Mail:	rohwedder at de.ibm.com	Am Weiher 24
>> <ecblank.gif>	<ecblank.gif>	65451 Kelsterbach
>> <ecblank.gif>	<ecblank.gif>	Germany
>> <ecblank.gif>
>> 
>> <graycol.gif>Keith Ball ---22.08.2018 21:33:25---Hello All, Does anyone know how to change the HTTP ports for the Spectrum Scale GUI?
>> 
>> From: Keith Ball <bipcuds at gmail.com>
>> To: gpfsug-discuss at spectrumscale.org
>> Date: 22.08.2018 21:33
>> Subject: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
>> Sent by: gpfsug-discuss-bounces at spectrumscale.org
>> 
>> 
>> 
>> 
>> Hello All,
>> 
>> Does anyone know how to change the HTTP ports for the Spectrum Scale GUI? Any documentation or RedPaper I have found deftly avoids discussing this. The most promising thing I see is in /opt/ibm/wlp/usr/servers/gpfsgui/server.xml:
>> 
>> <httpEndpoint id="defaultHttpEndpoint" host="*" httpPort="47080" httpsPort="47443">
>> <tcpOptions soReuseAddr="true"/>
>> </httpEndpoint>
>> 
>> but it appears that port 80 specifically is used also by the GUI's Web service. I already have an HTTP server using port 80 for provisioning (xCAT), so would rather change the Specturm Scale GUI configuration if I can.
>> 
>> Many Thanks,
>> Keith
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>> 
>> 
>> 
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>> 
>> Unless stated otherwise above:
>> IBM United Kingdom Limited - Registered in England and Wales with number 741598. 
>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>> 
>> 
>> 
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss


From novosirj at rutgers.edu  Fri Jul 19 17:43:03 2019
From: novosirj at rutgers.edu (Ryan Novosielski)
Date: Fri, 19 Jul 2019 16:43:03 +0000
Subject: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
In-Reply-To: <B5538DB3-4625-43DA-B8F9-E600F48054B4@rutgers.edu>
References: <OF67F2CFA5.B278755D-ON002582F2.002CFDE9-C12582F2.0030AC8F@notes.na.collabserv.com>
	<CAAxuGpE4fM+X1Gwc-OySE+ZWc_jygx5TACusAP4rO0BWNgtaKA@mail.gmail.com>
	<OF237C2868.64DA65AF-ON002582F2.003784F8-002582F2.0038210B@notes.na.collabserv.com>
	<OFE9B5AE08.76EA4744-ON002582F2.003C63C0-C12582F2.00410D62@notes.na.collabserv.com>
	<225AEBED-69EF-4E3C-9A67-5A9CB04B4309@rutgers.edu>
	<B5538DB3-4625-43DA-B8F9-E600F48054B4@rutgers.edu>
Message-ID: <C7EE62BE-118A-4680-B64E-4A2050F1BFBA@rutgers.edu>

Support has since pointed out to me, coincidentally, that all of the GUI callback scripts in /usr/lpp/mmfs/gui/callbacks contain a hard-coded port. I changed them this way, with PWD=/usr/lpp/mmfs/gui/callbacks:

find . -name *.sh -exec sed -i.bak 's/PORT=443/PORT=8443/g' {} \;

?as you can see, the only change was, for example:

[root at quorum02 callbacks]# diff gnr/PhysicalDiskCallback.sh.bak gnr/PhysicalDiskCallback.sh 
21c21
< PORT=443
---
> PORT=8443

That caused me to look to see if port 443 is hard-coded anyplace else, and I found this other one, /usr/lpp/mmfs/gui/bin-sudo/functions_iptables.sh:

23: . /etc/sysconfig/gpfsgui
24: 
25: HTTP_PORT=80
26: HTTPS_PORT=443

?this is peculiar to me because a) it works ? it would seem like these two override my /etc/sysconfig/gpfsgui settings, but the web server is reachable at 8443. Also, these lines would seem to make way more sense in the reverse (eg. let the sysconfig file redefine the ports if they contain values).

Ideally, IBM would let you change those two environment variables in the sysconfig file, or somewhere else, and the callback scripts would use that value from the environment. I?ve not tried setting PORT=$HTTPS_PORT to see if those callback scripts have access to that variable. 

--
____
|| \\UTGERS,  	 |---------------------------*O*---------------------------
||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
     `'

> On Jul 18, 2019, at 5:26 PM, Ryan Novosielski <novosirj at rutgers.edu> wrote:
> 
> Nope, that appears to be all of it. I also had a problem with the postgresql service, which was why the gpfsgui wouldn?t start. But once I fixed that, I can log in on https://<GUI_HOST>:8443.
> 
> HTH.
> 
>> On Jul 18, 2019, at 5:15 PM, Ryan Novosielski <novosirj at rutgers.edu> wrote:
>> 
>> I happened across this message because I?ve already done this in the past and was trying to figure out how I did it (apparently didn?t write it down).
>> 
>> Most of it appeared to be adding to /etc/sysconfig/gpfsgui the following:
>> 
>> HTTP_PORT=8080
>> HTTPS_PORT=8443
>> 
>> ?but that hasn?t completely done it yet. Going to have a look and see what else I might need to do.
>> 
>> --
>> ____
>> || \\UTGERS,  	 |---------------------------*O*---------------------------
>> ||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
>> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
>> ||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
>>    `'
>> 
>>> On Aug 23, 2018, at 7:50 AM, Markus Rohwedder <rohwedder at de.ibm.com> wrote:
>>> 
>>> Hello Juri, Keith,
>>> 
>>> thank you for your responses.
>>> 
>>> The internal services communicate on the privileged ports, for backwards compatibility and firewall simplicity reasons. We can not just assume all nodes in the cluster are at the latest level.
>>> 
>>> Running two services at the same port on different IP addresses could be an option to consider for co-existance of the GUI and another service on the same node.
>>> However we have not set up, tested nor documented such a configuration as of today. 
>>> 
>>> Currently the GUI service manages the iptables redirect bring up and tear down.
>>> If this would be managed externally it would be possible to bind services to specific ports based on specific IPs.
>>> 
>>> In order to create custom redirect rules based on IP address it is necessary to instruct the GUI to 
>>> - not check for already used ports when the GUI service tries to start up
>>> - don't create/destroy port forwarding rules during GUI service start and stop.
>>> This GUI behavior can be configured using the internal flag UPDATE_IPTABLES in the service configuration with the 5.0.1.2 GUI code level.
>>> 
>>> The service configuration is not stored in the cluster configuration and may be overwritten during code upgrades, so these settings may have to be added again after an upgrade.
>>> 
>>> See this KC link:
>>> https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.1/com.ibm.spectrum.scale.v5r01.doc/bl1adv_firewallforgui.htm
>>> 
>>> Mit freundlichen Gr??en / Kind regards
>>> 
>>> Dr. Markus Rohwedder
>>> 
>>> Spectrum Scale GUI Development
>>> <ecblank.gif>
>>> Phone:	+49 7034 6430190	IBM Deutschland Research & Development	
>>> <17153317.gif>
>>> E-Mail:	rohwedder at de.ibm.com	Am Weiher 24
>>> <ecblank.gif>	<ecblank.gif>	65451 Kelsterbach
>>> <ecblank.gif>	<ecblank.gif>	Germany
>>> <ecblank.gif>
>>> 
>>> <graycol.gif>"Daniel Kidger" ---23.08.2018 12:13:36---Keith, I have another IBM customer who also wished to move Scale GUI's https ports. In their case
>>> 
>>> From:  "Daniel Kidger" <daniel.kidger at uk.ibm.com>
>>> To:  gpfsug-discuss at spectrumscale.org
>>> Cc:  gpfsug-discuss at spectrumscale.org
>>> Date:  23.08.2018 12:13
>>> Subject:  Re: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
>>> Sent by:  gpfsug-discuss-bounces at spectrumscale.org
>>> 
>>> 
>>> 
>>> 
>>> Keith,
>>> 
>>> I have another IBM customer who also wished to move Scale GUI's https ports.
>>> In their case because they had their own web based management interface on the same https port.
>>> Is this the same reason that you have?
>>> If so I wonder how many other sites have the same issue?
>>> 
>>> One workaround that was suggested at the time, was to add a second IP address to the node (piggy-backing on 'eth0').
>>> Then run the two different GUIs, one per IP address.
>>> Is this an option, albeit a little ugly?
>>> Daniel
>>> 
>>> <17310450.gif>				Dr Daniel Kidger
>>> IBM Technical Sales Specialist
>>> Software Defined Solution Sales
>>> 
>>> +44-(0)7818 522 266 
>>> daniel.kidger at uk.ibm.com
>>> 
>>> 
>>> 
>>> ----- Original message -----
>>> From: "Markus Rohwedder" <rohwedder at de.ibm.com>
>>> Sent by: gpfsug-discuss-bounces at spectrumscale.org
>>> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
>>> Cc:
>>> Subject: Re: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
>>> Date: Thu, Aug 23, 2018 9:51 AM
>>> Hello Keith,
>>> 
>>> it is not so easy.
>>> 
>>> The GUI receives events from other scale components using the currently defined ports.
>>> Changing the GUI ports will cause breakage in the GUI stack at several places (internal watchdog functions, interlock with health events, interlock with CES).
>>> Therefore at this point there is no procedure to change this behaviour across all components.
>>> 
>>> Because the GUI service does not run as root. the GUI server does not serve the privileged ports 80 and 443 directly but rather 47443 and 47080.
>>> Tweaking the ports in the server.xml file will only change the native ports that the GUI uses.
>>> The GUI manages IPTABLES rules to forward ports 443 and 80 to 47443 and 47080. 
>>> If these ports are already used by another service, the GUI will not start up.
>>> 
>>> Making the GUI ports freely configurable is therefore not a strightforward change, and currently no on our roadmap.
>>> If you want to emphasize your case as future development item, please let me know.
>>> 
>>> I would also be interested in:
>>>> Scale version you are running
>>>> Do you need port 80 or 443 as well?
>>>> Would it work for you if the xCAT service was bound to a single IP address?
>>> 
>>> Mit freundlichen Gr??en / Kind regards
>>> 
>>> Dr. Markus Rohwedder
>>> 
>>> Spectrum Scale GUI Development
>>> 
>>> <ecblank.gif>
>>> Phone:	+49 7034 6430190	IBM Deutschland Research & Development	
>>> <17153317.gif>
>>> E-Mail:	rohwedder at de.ibm.com	Am Weiher 24
>>> <ecblank.gif>	<ecblank.gif>	65451 Kelsterbach
>>> <ecblank.gif>	<ecblank.gif>	Germany
>>> <ecblank.gif>
>>> 
>>> <graycol.gif>Keith Ball ---22.08.2018 21:33:25---Hello All, Does anyone know how to change the HTTP ports for the Spectrum Scale GUI?
>>> 
>>> From: Keith Ball <bipcuds at gmail.com>
>>> To: gpfsug-discuss at spectrumscale.org
>>> Date: 22.08.2018 21:33
>>> Subject: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
>>> Sent by: gpfsug-discuss-bounces at spectrumscale.org
>>> 
>>> 
>>> 
>>> 
>>> Hello All,
>>> 
>>> Does anyone know how to change the HTTP ports for the Spectrum Scale GUI? Any documentation or RedPaper I have found deftly avoids discussing this. The most promising thing I see is in /opt/ibm/wlp/usr/servers/gpfsgui/server.xml:
>>> 
>>> <httpEndpoint id="defaultHttpEndpoint" host="*" httpPort="47080" httpsPort="47443">
>>> <tcpOptions soReuseAddr="true"/>
>>> </httpEndpoint>
>>> 
>>> but it appears that port 80 specifically is used also by the GUI's Web service. I already have an HTTP server using port 80 for provisioning (xCAT), so would rather change the Specturm Scale GUI configuration if I can.
>>> 
>>> Many Thanks,
>>> Keith
>>> _______________________________________________
>>> gpfsug-discuss mailing list
>>> gpfsug-discuss at spectrumscale.org
>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> gpfsug-discuss mailing list
>>> gpfsug-discuss at spectrumscale.org
>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>> 
>>> Unless stated otherwise above:
>>> IBM United Kingdom Limited - Registered in England and Wales with number 741598. 
>>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>>> _______________________________________________
>>> gpfsug-discuss mailing list
>>> gpfsug-discuss at spectrumscale.org
>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> gpfsug-discuss mailing list
>>> gpfsug-discuss at spectrumscale.org
>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>> 
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss


From lore at cscs.ch  Fri Jul 19 17:59:05 2019
From: lore at cscs.ch (Lo Re  Giuseppe)
Date: Fri, 19 Jul 2019 16:59:05 +0000
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 90, Issue 11
In-Reply-To: <mailman.1.1563534001.39239.gpfsug-discuss@spectrumscale.org>
References: <mailman.1.1563534001.39239.gpfsug-discuss@spectrumscale.org>
Message-ID: <1F7E4727-C978-45D5-A348-22DC7B62AF88@cscs.ch>

Hi,

We have also hit this limitation at CSCS.
We have a CES Object Service cluster where one node cannot run the Openstack Swift proxy-server service on port 443 as it is already taken by the GUI.
I have opened a RFE, didn?t help much though __ , so far.

https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=107778

Giuseppe

?On 19.07.19, 04:01, "gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org" <gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org> wrote:

    /etc/sysconfig/gpfsgui


From novosirj at rutgers.edu  Wed Jul 24 17:19:47 2019
From: novosirj at rutgers.edu (Ryan Novosielski)
Date: Wed, 24 Jul 2019 16:19:47 +0000
Subject: [gpfsug-discuss] Spectrum Scale with RHEL7.6 kernel
 3.10.0-957.21.2
In-Reply-To: <OF248C1D60.E94029E1-ON00258412.00769D60-85258412.0077861A@notes.na.collabserv.com>
References: <OF248C1D60.E94029E1-ON00258412.00769D60-85258412.0077861A@notes.na.collabserv.com>
Message-ID: <3DFDAC1A-ECD5-4752-8342-55389804FBDD@rutgers.edu>

Hi Felipe and others:

We didn?t hear about this apart from via this list and from a colleague. What is the way to get on an official notifications list for this sort of thing? I don?t see anything anywhere in the IBM support pages.

Thanks!

--
____
|| \\UTGERS,  	 |---------------------------*O*---------------------------
||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
     `'

> On Jun 7, 2019, at 5:45 PM, Felipe Knop <knop at us.ibm.com> wrote:
> 
> All,
> 
> There have been reported issues (including kernel crashes) on Spectrum Scale with the latest RHEL7.6 kernel 3.10.0-957.21.2. Please consider delaying upgrades to this kernel until further information is provided. 
> 
> Thanks,
> 
> Felipe
> 
> ----
> Felipe Knop knop at us.ibm.com
> GPFS Development and Security
> IBM Systems
> IBM Building 008
> 2455 South Rd, Poughkeepsie, NY 12601
> (845) 433-9314 T/L 293-9314
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss


From S.J.Thompson at bham.ac.uk  Wed Jul 24 18:30:31 2019
From: S.J.Thompson at bham.ac.uk (Simon Thompson)
Date: Wed, 24 Jul 2019 17:30:31 +0000
Subject: [gpfsug-discuss] Spectrum Scale with RHEL7.6 kernel
 3.10.0-957.21.2
In-Reply-To: <3DFDAC1A-ECD5-4752-8342-55389804FBDD@rutgers.edu>
References: <OF248C1D60.E94029E1-ON00258412.00769D60-85258412.0077861A@notes.na.collabserv.com>,
	<3DFDAC1A-ECD5-4752-8342-55389804FBDD@rutgers.edu>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545012993054A@EX13.adf.bham.ac.uk>

There is a notifications service. I think you have to subscribe to them via fix central or passport advantage (I don't remember which).

Though this notification was on the list a while before the official Comms.

Simon
________________________________________
From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Ryan Novosielski [novosirj at rutgers.edu]
Sent: 24 July 2019 17:19
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Spectrum Scale with RHEL7.6 kernel 3.10.0-957.21.2

Hi Felipe and others:

We didn?t hear about this apart from via this list and from a colleague. What is the way to get on an official notifications list for this sort of thing? I don?t see anything anywhere in the IBM support pages.

Thanks!

--
____
|| \\UTGERS,     |---------------------------*O*---------------------------
||_// the State  |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ  | Office of Advanced Research Computing - MSB C630, Newark
     `'

> On Jun 7, 2019, at 5:45 PM, Felipe Knop <knop at us.ibm.com> wrote:
>
> All,
>
> There have been reported issues (including kernel crashes) on Spectrum Scale with the latest RHEL7.6 kernel 3.10.0-957.21.2. Please consider delaying upgrades to this kernel until further information is provided.
>
> Thanks,
>
> Felipe
>
> ----
> Felipe Knop knop at us.ibm.com
> GPFS Development and Security
> IBM Systems
> IBM Building 008
> 2455 South Rd, Poughkeepsie, NY 12601
> (845) 433-9314 T/L 293-9314
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


From mark.bergman at uphs.upenn.edu  Fri Jul 26 00:31:21 2019
From: mark.bergman at uphs.upenn.edu (mark.bergman at uphs.upenn.edu)
Date: Thu, 25 Jul 2019 19:31:21 -0400
Subject: [gpfsug-discuss] Question concerning integration of CES with AD
	authentication system
In-Reply-To: Your message of "Thu, 24 May 2018 17:07:02 -0000."
	<OF0CCFED2D.FE89B800-ON00258297.005DFD38-00258297.005E075F@notes.na.collabserv.com>
References: <OF0CCFED2D.FE89B800-ON00258297.005DFD38-00258297.005E075F@notes.na.collabserv.com>
	<1527173192.28106.18.camel@strath.ac.uk>,
	<83A6EEB0EC738F459A39439733AE804522F1B13B@MBX214.d.ethz.ch><20180524141632.xuah3dxu4bxx372z@utumno.gs.washington.edu>
Message-ID: <10843-1564097481.984955@sYvE.K-ht.6Q-W>

In the message dated: Thu, 24 May 2018 17:07:02 -0000,
The pithy ruminations from Christof Schmitt on 
[Re: [gpfsug-discuss] Question concerning integration of CES with AD authentication system] were:
=> 

Following up on an old, old post...

=> > Basically Samba ignores the separate GID field in RFC2307bis, so one
=> > imagines the options for changing the LDAP attributes are none
=> > existent.
=>  
=> mmuserauth now has an option to use either the gid from the actual primary
=> group or the gid defined for the user. See:
=>  
=> https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.0/
=> com.ibm.spectrum.scale.v5r00.doc/bl1adm_mmuserauth.htm
=>  
=> --unixmap-domains unixDomainMap
=> [...]
=>     win: Specifies the system to read the primary group set as Windows
=> primary group of a user on the Active Directory.
=>      unix: Specifies the system to read the primary group as set in "UNIX
=> attributes" of a user on the Active Directory.  
=>     For example,
=>      --unixmap-domains "MYDOMAIN1(20000-50000:unix);MYDOMAIN2
=> (100000-200000:win)"

I see this is refering to UNIX attributes within AD, but I'm curious about mapping to attributes in LDAP.

=> This gets mapped to 'idmap config ... : unix_primary_group' in the
=> internal config.

Does that correspond to setting the smb.conf parameter
	
	unix_primary_group = yes




Specifically, under Spectrum Scale 5.0.2, if I run:

	mmuserauth service create --data-access-method file --ldapmap-domains "DOMAIN(type=stand-alone:ldap_srv=ldapserver:range=1001-65535:usr_dn=ou=People,dc=DC,dc=TLD:grp_dn=ou=Group,dc=DC,dc=TLD)" --type ad

(some args removed in this example), will that map the user's primary group to

	the primaryGroupID supplied by AD
  or
	the primaryGroupID LDAP field
  or
	the gidNumber LDAP field

or something else?

Thanks,

Mark


=> 
=> Christof Schmitt || IBM || Spectrum Scale Development || Tucson, AZ
=> christof.schmitt at us.ibm.com  ||  +1-520-799-2469    (T/L: 321-2469)
=>  


From christof.schmitt at us.ibm.com  Fri Jul 26 17:38:17 2019
From: christof.schmitt at us.ibm.com (Christof Schmitt)
Date: Fri, 26 Jul 2019 16:38:17 +0000
Subject: [gpfsug-discuss] Question concerning integration of CES with AD
 authentication system
In-Reply-To: <10843-1564097481.984955@sYvE.K-ht.6Q-W>
References: <10843-1564097481.984955@sYvE.K-ht.6Q-W>,
	<OF0CCFED2D.FE89B800-ON00258297.005DFD38-00258297.005E075F@notes.na.collabserv.com>
	<1527173192.28106.18.camel@strath.ac.uk>,
	<83A6EEB0EC738F459A39439733AE804522F1B13B@MBX214.d.ethz.ch><20180524141632.xuah3dxu4bxx372z@utumno.gs.washington.edu>
Message-ID: <OF5FF29174.4A08CF99-ON00258443.005B64E3-00258443.005B658E@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190726/68876eda/attachment-0001.htm>

From david_johnson at brown.edu  Mon Jul 29 17:04:38 2019
From: david_johnson at brown.edu (David Johnson)
Date: Mon, 29 Jul 2019 12:04:38 -0400
Subject: [gpfsug-discuss] Building GPFS filesystem system data pool on
 shared nothing NVMe drives
Message-ID: <B5D9698F-23E7-432E-A35E-747569E7422F@brown.edu>

We are planning a 5.0.x upgrade onto new hardware to make use of the new 5.x GPFS features.
The goal is to use up to four NSD nodes for metadata, each one with 6 NVMe drives (to be determined
whether we use Intel VROC for raid 5 or raid 1, or just straight disks).  

So questions ? 
Has anyone done system pool on shared nothing cluster?  How did you set it up?
With default metadata replication set at 3, can you make use of four NSD nodes effectively?
How would one design the location vectors and failure groups so that the system metadata is
spread evenly across the four servers?

Thanks,
 ? ddj
Dave Johnson

From xhejtman at ics.muni.cz  Mon Jul 29 17:11:38 2019
From: xhejtman at ics.muni.cz (Lukas Hejtmanek)
Date: Mon, 29 Jul 2019 18:11:38 +0200
Subject: [gpfsug-discuss] Asymetric SAN with GPFS
Message-ID: <20190729161138.znuss5dt2rhig6cv@ics.muni.cz>

Hello,

is there any settings for GPFS 5.x so that you could mitigate slow down of
asymmetric SAN? The asymmetric SAN means, that not every LUN has the same
speed, or not every disk array has the same number of LUNs. It seems that
overal speed is degraded to the slowest LUN. Is there any workaround for this
except avoiding that LUN at all?

-- 
Luk?? Hejtm?nek

Linux Administrator only because
  Full Time Multitasking Ninja 
  is not an official job title


From S.J.Thompson at bham.ac.uk  Mon Jul 29 17:19:45 2019
From: S.J.Thompson at bham.ac.uk (Simon Thompson)
Date: Mon, 29 Jul 2019 16:19:45 +0000
Subject: [gpfsug-discuss] Asymetric SAN with GPFS
In-Reply-To: <20190729161138.znuss5dt2rhig6cv@ics.muni.cz>
References: <20190729161138.znuss5dt2rhig6cv@ics.muni.cz>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545012994D59A@EX13.adf.bham.ac.uk>

Surely you would need to use different data pools for this. Given in a single pool data will be distributed over all available nsd devices in that pool, you are highly likely to only ever go as fast as the slowest LUN.

Simon
________________________________________
From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of xhejtman at ics.muni.cz [xhejtman at ics.muni.cz]
Sent: 29 July 2019 17:11
To: gpfsug-discuss at spectrumscale.org
Subject: [gpfsug-discuss] Asymetric SAN with GPFS

Hello,

is there any settings for GPFS 5.x so that you could mitigate slow down of
asymmetric SAN? The asymmetric SAN means, that not every LUN has the same
speed, or not every disk array has the same number of LUNs. It seems that
overal speed is degraded to the slowest LUN. Is there any workaround for this
except avoiding that LUN at all?

--
Luk?? Hejtm?nek

Linux Administrator only because
  Full Time Multitasking Ninja
  is not an official job title
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


From luis.bolinches at fi.ibm.com  Mon Jul 29 18:34:58 2019
From: luis.bolinches at fi.ibm.com (Luis Bolinches)
Date: Mon, 29 Jul 2019 17:34:58 +0000
Subject: [gpfsug-discuss] Building GPFS filesystem system data pool on
 shared nothing NVMe drives
In-Reply-To: <B5D9698F-23E7-432E-A35E-747569E7422F@brown.edu>
Message-ID: <OFC1ED3470.4BE6E16D-ON00258446.006095F7-1564421698664@notes.na.collabserv.com>

Hi, from phone so sorry for typos. 

I really think you should look into Spectrum Scale Erasure Code Edition (ECE) for this. 

Sure you could do a RAID on each node as you mention here but that sounds like a lot of waste to me on storage capacity. Not to forget you get other goodies like end to end checksum and rapid rebuilds with ECE, among others. 

Four servers is the minimum requirement for ECE (4+3p) and from top of my head 12 disk per RG, you are fine with both requirements. 

There is a presentation on ECE on the user group web page from London May 2019 were we talk about ECE. 

And the ibm page of the product https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_intro.htm
--
Cheers

> El 29 jul 2019, a las 19:06, David Johnson <david_johnson at brown.edu> escribi?:
> 
> We are planning a 5.0.x upgrade onto new hardware to make use of the new 5.x GPFS features.
> The goal is to use up to four NSD nodes for metadata, each one with 6 NVMe drives (to be determined
> whether we use Intel VROC for raid 5 or raid 1, or just straight disks).  
> 
> So questions ? 
> Has anyone done system pool on shared nothing cluster?  How did you set it up?
> With default metadata replication set at 3, can you make use of four NSD nodes effectively?
> How would one design the location vectors and failure groups so that the system metadata is
> spread evenly across the four servers?
> 
> Thanks,
> ? ddj
> Dave Johnson
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=1mZ896psa5caYzBeaugTlc7TtRejJp3uvKYxas3S7Xc&m=Pu3G5GzwRsM4iwztIiWzzngNSgsPzrRe8cd3pPAFVM0&s=q3w_lvsuB88gMM_J5psgKuWDex_wSW1v5h16WNMCtzU&e= 
> 

Ellei edell? ole toisin mainittu: / Unless stated otherwise above:
Oy IBM Finland Ab
PL 265, 00101 Helsinki, Finland
Business ID, Y-tunnus: 0195876-3 
Registered in Finland

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190729/0d0c5e04/attachment-0001.htm>

From david_johnson at brown.edu  Tue Jul 30 12:46:14 2019
From: david_johnson at brown.edu (David Johnson)
Date: Tue, 30 Jul 2019 07:46:14 -0400
Subject: [gpfsug-discuss] Building GPFS filesystem system data pool on
 shared nothing NVMe drives
In-Reply-To: <OFC1ED3470.4BE6E16D-ON00258446.006095F7-1564421698664@notes.na.collabserv.com>
References: <OFC1ED3470.4BE6E16D-ON00258446.006095F7-1564421698664@notes.na.collabserv.com>
Message-ID: <D53515DA-94D8-400B-80E0-8033633A5F21@brown.edu>

Can we confirm the requirement for disks per RG?  I have 4 RG, but only 6 x 3TB NVMe drives per box.

> On Jul 29, 2019, at 1:34 PM, Luis Bolinches <luis.bolinches at fi.ibm.com> wrote:
> 
> Hi, from phone so sorry for typos. 
> 
> I really think you should look into Spectrum Scale Erasure Code Edition (ECE) for this. 
> 
> Sure you could do a RAID on each node as you mention here but that sounds like a lot of waste to me on storage capacity. Not to forget you get other goodies like end to end checksum and rapid rebuilds with ECE, among others. 
> 
> Four servers is the minimum requirement for ECE (4+3p) and from top of my head 12 disk per RG, you are fine with both requirements. 
> 
> There is a presentation on ECE on the user group web page from London May 2019 were we talk about ECE. 
> 
> And the ibm page of the product https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_intro.htm <https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_intro.htm>
> --
> Cheers
> 
> El 29 jul 2019, a las 19:06, David Johnson <david_johnson at brown.edu <mailto:david_johnson at brown.edu>> escribi?:
> 
>> We are planning a 5.0.x upgrade onto new hardware to make use of the new 5.x GPFS features.
>> The goal is to use up to four NSD nodes for metadata, each one with 6 NVMe drives (to be determined
>> whether we use Intel VROC for raid 5 or raid 1, or just straight disks).  
>> 
>> So questions ? 
>> Has anyone done system pool on shared nothing cluster?  How did you set it up?
>> With default metadata replication set at 3, can you make use of four NSD nodes effectively?
>> How would one design the location vectors and failure groups so that the system metadata is
>> spread evenly across the four servers?
>> 
>> Thanks,
>> ? ddj
>> Dave Johnson
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org <http://spectrumscale.org/>
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss <http://gpfsug.org/mailman/listinfo/gpfsug-discuss> 
>> 
> 
> Ellei edell? ole toisin mainittu: / Unless stated otherwise above:
> Oy IBM Finland Ab
> PL 265, 00101 Helsinki, Finland
> Business ID, Y-tunnus: 0195876-3 
> Registered in Finland
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190730/b61f3695/attachment-0001.htm>

From p.ward at nhm.ac.uk  Tue Jul 30 13:16:31 2019
From: p.ward at nhm.ac.uk (Paul Ward)
Date: Tue, 30 Jul 2019 12:16:31 +0000
Subject: [gpfsug-discuss] Break-out session for new user and prospects
 [London Usergroup]
In-Reply-To: <6916EC29-2057-4396-A285-7A905DB215EE@bham.ac.uk>
References: <776770B2-5F84-4462-B900-58EBB982DC1C@spectrumscale.org>
	<DB7PR10MB2201EAAA629BB9C10BBE24E0D3C90@DB7PR10MB2201.EURPRD10.PROD.OUTLOOK.COM>
	<6916EC29-2057-4396-A285-7A905DB215EE@bham.ac.uk>
Message-ID: <DB7PR10MB22017537C2C2102FD904B11CD3DC0@DB7PR10MB2201.EURPRD10.PROD.OUTLOOK.COM>

Hi Simon,

Any update on this please.


Paul Ward
Technical Solutions Infrastructure Architect
Natural History Museum
T: 02079426450
E: p.ward at nhm.ac.uk<mailto:p.ward at nhm.ac.uk>

From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of Simon Thompson
Sent: 17 July 2019 14:10
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Break-out session for new user and prospects [London Usergroup]

Hi Paul,

There was a slide deck, I?ve asked the IBMers if it?s something we can share.

Simon

From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of Paul Ward <p.ward at nhm.ac.uk<mailto:p.ward at nhm.ac.uk>>
Reply-To: "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, 17 July 2019 at 14:01
To: "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Break-out session for new user and prospects [London Usergroup]

Hi Simon,

Was this a slide presentation? Is it available?

Kindest regards,
Paul

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (Spectrum Scale User Group Chair)
Sent: 30 April 2019 10:25
To: gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] Break-out session for new user and prospects [London Usergroup]

Hi all,

We know that a lot of the talks at the user groups are for experienced users, following feedback from the USA user group, we thought we?d advertise that this year we?re planning to run a break-out for new users on day 1.

Break-out session for new user and prospects (Wed May 8th, 13:00 - 16:45)

This year we will offer a break-out session for new Spectrum Scale user and prospects to get started with Spectrum Scale. In this session we will cover Spectrum Scale Use Cases, the architecture of a Spectrum Scale environment, and discuss how the manifold Spectrum Scale features support the different use case. Please inform customers and colleagues who are interested to learn about Spectrum Scale to grab one of the last seats.

Registration link:
https://www.spectrumscaleug.org/event/uk-user-group-meeting/<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.spectrumscaleug.org%2Fevent%2Fuk-user-group-meeting%2F&data=02%7C01%7Cp.ward%40nhm.ac.uk%7C7d6995c8217f453c8dd408d70ab82dad%7C73a29c014e78437fa0d4c8553e1960c1%7C1%7C0%7C636989658468906223&sdata=%2BmSDjCczU4GYNWbtOfeWhxovF0YtYzkMX%2BJNRmcIpNo%3D&reserved=0>

There?s just a couple of places left for the usergroup, so please do share and register if you plan to attend.

Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190730/eeac3325/attachment-0001.htm>

From Paul.Sanchez at deshaw.com  Tue Jul 30 13:19:56 2019
From: Paul.Sanchez at deshaw.com (Sanchez, Paul)
Date: Tue, 30 Jul 2019 12:19:56 +0000
Subject: [gpfsug-discuss] Building GPFS filesystem system data pool on
 shared nothing NVMe drives
In-Reply-To: <D53515DA-94D8-400B-80E0-8033633A5F21@brown.edu>
References: <OFC1ED3470.4BE6E16D-ON00258446.006095F7-1564421698664@notes.na.collabserv.com>
	<D53515DA-94D8-400B-80E0-8033633A5F21@brown.edu>
Message-ID: <5664c4aaea5e4fbfac1f8a72acb080dc@deshaw.com>

Hi David,

In an ECE configuration, it would be typical to put all of the NVMe disks in all 4 of your servers into a single recovery group.   So in your case, all 24 NVMe drives would be in one recovery group and the 4 servers would be ?log group? servers in the recovery group, distributing the I/O load for the NSD/vdisks that are hosted on the RG.  (The minimum disks for a single RG config is 12, and you meet that easily.)

https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_plan_recommendations.htm
outlines the recommendations for raidCode protection.  Your configuration (4 nodes) would use vdisks with 4+3P, which gives you a slightly better capacity yield than RAID10 would, but with much better recovery characteristics:


?         No single failed node will result in a down system NSD.

?         No single drive failure will require a critical priority rebuild, and can be handled in the background without killing performance.

So from that perspective, ECE is a win here and avoids a problem with the non-ECE, shared-nothing designs: the manual ?mmchdisk <fsname> start -a? operation that is needed after any traditional shared-nothing metadata NSD goes offline to bring it back and protect against further failures.

Despite the operational challenges of the non-ECE design, it can sometimes survive two server failures (if replication factor is 3 and the filesystem descriptor quorum wasn?t lost by the two failures) which a 4 node ECE cluster cannot.  Given that the world is complex and unexpected things can happen, I?d personally recommend redistributing the 24 disks across 6 servers if you can, so that the design could always survive 2 node failures.  I?ve run this design and it?s fairly robust.

In any event, you should of course test the failure scenarios yourself before going into production to validate them and familiarize yourself with the process.  And a special note on ECE: due to the cooperative nature at the pdisk level, the network between the servers in the RG should be as reliable as possible and any network redundancy should also be tested ahead of time.

-Paul

From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of David Johnson
Sent: Tuesday, July 30, 2019 7:46 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Building GPFS filesystem system data pool on shared nothing NVMe drives


This message was sent by an external party.

Can we confirm the requirement for disks per RG?  I have 4 RG, but only 6 x 3TB NVMe drives per box.


On Jul 29, 2019, at 1:34 PM, Luis Bolinches <luis.bolinches at fi.ibm.com<mailto:luis.bolinches at fi.ibm.com>> wrote:

Hi, from phone so sorry for typos.

I really think you should look into Spectrum Scale Erasure Code Edition (ECE) for this.

Sure you could do a RAID on each node as you mention here but that sounds like a lot of waste to me on storage capacity. Not to forget you get other goodies like end to end checksum and rapid rebuilds with ECE, among others.

Four servers is the minimum requirement for ECE (4+3p) and from top of my head 12 disk per RG, you are fine with both requirements.

There is a presentation on ECE on the user group web page from London May 2019 were we talk about ECE.

And the ibm page of the product https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_intro.htm
--
Cheers

El 29 jul 2019, a las 19:06, David Johnson <david_johnson at brown.edu<mailto:david_johnson at brown.edu>> escribi?:
We are planning a 5.0.x upgrade onto new hardware to make use of the new 5.x GPFS features.
The goal is to use up to four NSD nodes for metadata, each one with 6 NVMe drives (to be determined
whether we use Intel VROC for raid 5 or raid 1, or just straight disks).

So questions ?
Has anyone done system pool on shared nothing cluster?  How did you set it up?
With default metadata replication set at 3, can you make use of four NSD nodes effectively?
How would one design the location vectors and failure groups so that the system metadata is
spread evenly across the four servers?

Thanks,
? ddj
Dave Johnson
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Ellei edell? ole toisin mainittu: / Unless stated otherwise above:
Oy IBM Finland Ab
PL 265, 00101 Helsinki, Finland
Business ID, Y-tunnus: 0195876-3
Registered in Finland
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190730/539e5159/attachment-0001.htm>

From david_johnson at brown.edu  Tue Jul 30 14:03:54 2019
From: david_johnson at brown.edu (David Johnson)
Date: Tue, 30 Jul 2019 09:03:54 -0400
Subject: [gpfsug-discuss] Building GPFS filesystem system data pool on
 shared nothing NVMe drives
In-Reply-To: <5664c4aaea5e4fbfac1f8a72acb080dc@deshaw.com>
References: <OFC1ED3470.4BE6E16D-ON00258446.006095F7-1564421698664@notes.na.collabserv.com>
	<D53515DA-94D8-400B-80E0-8033633A5F21@brown.edu>
	<5664c4aaea5e4fbfac1f8a72acb080dc@deshaw.com>
Message-ID: <261C855E-FC23-43C6-8A76-FEAE5B48221C@brown.edu>

OK, so the ECE recovery group is the four NSD servers with the System storage pool disks, and somehow I have to read the docs
and find out how to define pdisks that spread the replication across the four servers, but three disks at a time.  
Three pdisks of 7 drives, three I can't do anything with, or are those for rebuilding space?

Can you provide me details of your six-node non-ECE configuration?  Basically how the NSDs are defined...

The remainder of our new filesystem will have a fast pool of 12 nodes of excelero, and 2Pb of spinning disks, so another possibility
would be to license four more nodes and put the system pool under excelero.  

 -- ddj

> On Jul 30, 2019, at 8:19 AM, Sanchez, Paul <Paul.Sanchez at deshaw.com> wrote:
> 
> Hi David,
>  
> In an ECE configuration, it would be typical to put all of the NVMe disks in all 4 of your servers into a single recovery group.   So in your case, all 24 NVMe drives would be in one recovery group and the 4 servers would be ?log group? servers in the recovery group, distributing the I/O load for the NSD/vdisks that are hosted on the RG.  (The minimum disks for a single RG config is 12, and you meet that easily.)
>  
> https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_plan_recommendations.htm <https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_plan_recommendations.htm>
> outlines the recommendations for raidCode protection.  Your configuration (4 nodes) would use vdisks with 4+3P, which gives you a slightly better capacity yield than RAID10 would, but with much better recovery characteristics:
>  
> ?         No single failed node will result in a down system NSD.
> ?         No single drive failure will require a critical priority rebuild, and can be handled in the background without killing performance.
>  
> So from that perspective, ECE is a win here and avoids a problem with the non-ECE, shared-nothing designs: the manual ?mmchdisk <fsname> start -a? operation that is needed after any traditional shared-nothing metadata NSD goes offline to bring it back and protect against further failures.
>  
> Despite the operational challenges of the non-ECE design, it can sometimes survive two server failures (if replication factor is 3 and the filesystem descriptor quorum wasn?t lost by the two failures) which a 4 node ECE cluster cannot.  Given that the world is complex and unexpected things can happen, I?d personally recommend redistributing the 24 disks across 6 servers if you can, so that the design could always survive 2 node failures.  I?ve run this design and it?s fairly robust.
>  
> In any event, you should of course test the failure scenarios yourself before going into production to validate them and familiarize yourself with the process.  And a special note on ECE: due to the cooperative nature at the pdisk level, the network between the servers in the RG should be as reliable as possible and any network redundancy should also be tested ahead of time.
>  
> -Paul
>  
> From: gpfsug-discuss-bounces at spectrumscale.org <mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org <mailto:gpfsug-discuss-bounces at spectrumscale.org>> On Behalf Of David Johnson
> Sent: Tuesday, July 30, 2019 7:46 AM
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org <mailto:gpfsug-discuss at spectrumscale.org>>
> Subject: Re: [gpfsug-discuss] Building GPFS filesystem system data pool on shared nothing NVMe drives
>  
> This message was sent by an external party.
> 
>  
> Can we confirm the requirement for disks per RG?  I have 4 RG, but only 6 x 3TB NVMe drives per box.
> 
> 
> On Jul 29, 2019, at 1:34 PM, Luis Bolinches <luis.bolinches at fi.ibm.com <mailto:luis.bolinches at fi.ibm.com>> wrote:
>  
> Hi, from phone so sorry for typos.  
>  
> I really think you should look into Spectrum Scale Erasure Code Edition (ECE) for this. 
>  
> Sure you could do a RAID on each node as you mention here but that sounds like a lot of waste to me on storage capacity. Not to forget you get other goodies like end to end checksum and rapid rebuilds with ECE, among others. 
>  
> Four servers is the minimum requirement for ECE (4+3p) and from top of my head 12 disk per RG, you are fine with both requirements. 
> 
> There is a presentation on ECE on the user group web page from London May 2019 were we talk about ECE. 
>  
> And the ibm page of the product https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_intro.htm <https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_intro.htm>
> -- 
> Cheers
> 
> El 29 jul 2019, a las 19:06, David Johnson <david_johnson at brown.edu <mailto:david_johnson at brown.edu>> escribi?:
> 
> We are planning a 5.0.x upgrade onto new hardware to make use of the new 5.x GPFS features.
> The goal is to use up to four NSD nodes for metadata, each one with 6 NVMe drives (to be determined
> whether we use Intel VROC for raid 5 or raid 1, or just straight disks).  
> 
> So questions ? 
> Has anyone done system pool on shared nothing cluster?  How did you set it up?
> With default metadata replication set at 3, can you make use of four NSD nodes effectively?
> How would one design the location vectors and failure groups so that the system metadata is
> spread evenly across the four servers?
> 
> Thanks,
> ? ddj
> Dave Johnson
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org <http://spectrumscale.org/>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
> 
> Ellei edell? ole toisin mainittu: / Unless stated otherwise above:
> Oy IBM Finland Ab
> PL 265, 00101 Helsinki, Finland
> Business ID, Y-tunnus: 0195876-3 
> Registered in Finland
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org <http://spectrumscale.org/>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
>  
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org <http://spectrumscale.org/>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190730/e8dc0611/attachment-0001.htm>

From S.J.Thompson at bham.ac.uk  Tue Jul 30 14:49:04 2019
From: S.J.Thompson at bham.ac.uk (Simon Thompson)
Date: Tue, 30 Jul 2019 13:49:04 +0000
Subject: [gpfsug-discuss] Break-out session for new user and prospects
 [London Usergroup]
In-Reply-To: <DB7PR10MB22017537C2C2102FD904B11CD3DC0@DB7PR10MB2201.EURPRD10.PROD.OUTLOOK.COM>
References: <776770B2-5F84-4462-B900-58EBB982DC1C@spectrumscale.org>
	<DB7PR10MB2201EAAA629BB9C10BBE24E0D3C90@DB7PR10MB2201.EURPRD10.PROD.OUTLOOK.COM>
	<6916EC29-2057-4396-A285-7A905DB215EE@bham.ac.uk>
	<DB7PR10MB22017537C2C2102FD904B11CD3DC0@DB7PR10MB2201.EURPRD10.PROD.OUTLOOK.COM>
Message-ID: <BB09590F-57DE-4BCF-BD14-5DFED3BC2AD8@bham.ac.uk>

Hi Sorry, yes. I got the slides and forgot to upload them!

https://www.spectrumscaleug.org/wp-content/uploads/2019/07/IBM-Spectrum-Scale-Use-Cases.pdf

Simon

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Paul Ward <p.ward at nhm.ac.uk>
Reply-To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Date: Tuesday, 30 July 2019 at 13:16
To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Break-out session for new user and prospects [London Usergroup]

Hi Simon,

Any update on this please.


Paul Ward
Technical Solutions Infrastructure Architect
Natural History Museum
T: 02079426450
E: p.ward at nhm.ac.uk<mailto:p.ward at nhm.ac.uk>

From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of Simon Thompson
Sent: 17 July 2019 14:10
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Break-out session for new user and prospects [London Usergroup]

Hi Paul,

There was a slide deck, I?ve asked the IBMers if it?s something we can share.

Simon

From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of Paul Ward <p.ward at nhm.ac.uk<mailto:p.ward at nhm.ac.uk>>
Reply-To: "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, 17 July 2019 at 14:01
To: "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Break-out session for new user and prospects [London Usergroup]

Hi Simon,

Was this a slide presentation? Is it available?

Kindest regards,
Paul

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (Spectrum Scale User Group Chair)
Sent: 30 April 2019 10:25
To: gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] Break-out session for new user and prospects [London Usergroup]

Hi all,

We know that a lot of the talks at the user groups are for experienced users, following feedback from the USA user group, we thought we?d advertise that this year we?re planning to run a break-out for new users on day 1.

Break-out session for new user and prospects (Wed May 8th, 13:00 - 16:45)

This year we will offer a break-out session for new Spectrum Scale user and prospects to get started with Spectrum Scale. In this session we will cover Spectrum Scale Use Cases, the architecture of a Spectrum Scale environment, and discuss how the manifold Spectrum Scale features support the different use case. Please inform customers and colleagues who are interested to learn about Spectrum Scale to grab one of the last seats.

Registration link:
https://www.spectrumscaleug.org/event/uk-user-group-meeting/<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.spectrumscaleug.org%2Fevent%2Fuk-user-group-meeting%2F&data=02%7C01%7Cp.ward%40nhm.ac.uk%7C7d6995c8217f453c8dd408d70ab82dad%7C73a29c014e78437fa0d4c8553e1960c1%7C1%7C0%7C636989658468906223&sdata=%2BmSDjCczU4GYNWbtOfeWhxovF0YtYzkMX%2BJNRmcIpNo%3D&reserved=0>

There?s just a couple of places left for the usergroup, so please do share and register if you plan to attend.

Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190730/731e27a0/attachment-0001.htm>

From Paul.Sanchez at deshaw.com  Tue Jul 30 15:36:35 2019
From: Paul.Sanchez at deshaw.com (Sanchez, Paul)
Date: Tue, 30 Jul 2019 14:36:35 +0000
Subject: [gpfsug-discuss] Building GPFS filesystem system data pool on
 shared nothing NVMe drives
In-Reply-To: <261C855E-FC23-43C6-8A76-FEAE5B48221C@brown.edu>
References: <OFC1ED3470.4BE6E16D-ON00258446.006095F7-1564421698664@notes.na.collabserv.com>
	<D53515DA-94D8-400B-80E0-8033633A5F21@brown.edu>
	<5664c4aaea5e4fbfac1f8a72acb080dc@deshaw.com>
	<261C855E-FC23-43C6-8A76-FEAE5B48221C@brown.edu>
Message-ID: <b227c5eadc5c46cd9e63b19e51416502@deshaw.com>

Yes, read the documentation for the mmvdisk command to get a better idea of how this actually works.  There?s definitely a small paradigmatic shift in thinking that you?ll encounter between traditional NSDs and ECE.  The mmvdisk command?s defaults handle just about everything for you and the defaults are sane.

The short answer is that in a 6-node ECE recoverygroup, each of the nodes will normally provide access to 2 NSDs and the system pool would therefore have 12 NSDs total.  If a node fails, each of its two NSDs will continue to be served each by a different one of the remaining servers.  If you?ve used ESS before, then you can think about the RAID/spare/rebuild space similarly to that: all drives have some spare space on them and erasure-code stripes are evenly distributed among all of the disks so that they are all used and fill at approximately the same rate.

From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of David Johnson
Sent: Tuesday, July 30, 2019 9:04 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Building GPFS filesystem system data pool on shared nothing NVMe drives


This message was sent by an external party.

OK, so the ECE recovery group is the four NSD servers with the System storage pool disks, and somehow I have to read the docs
and find out how to define pdisks that spread the replication across the four servers, but three disks at a time.
Three pdisks of 7 drives, three I can't do anything with, or are those for rebuilding space?

Can you provide me details of your six-node non-ECE configuration?  Basically how the NSDs are defined...

The remainder of our new filesystem will have a fast pool of 12 nodes of excelero, and 2Pb of spinning disks, so another possibility
would be to license four more nodes and put the system pool under excelero.

 -- ddj


On Jul 30, 2019, at 8:19 AM, Sanchez, Paul <Paul.Sanchez at deshaw.com<mailto:Paul.Sanchez at deshaw.com>> wrote:

Hi David,

In an ECE configuration, it would be typical to put all of the NVMe disks in all 4 of your servers into a single recovery group.   So in your case, all 24 NVMe drives would be in one recovery group and the 4 servers would be ?log group? servers in the recovery group, distributing the I/O load for the NSD/vdisks that are hosted on the RG.  (The minimum disks for a single RG config is 12, and you meet that easily.)

https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_plan_recommendations.htm
outlines the recommendations for raidCode protection.  Your configuration (4 nodes) would use vdisks with 4+3P, which gives you a slightly better capacity yield than RAID10 would, but with much better recovery characteristics:

?         No single failed node will result in a down system NSD.
?         No single drive failure will require a critical priority rebuild, and can be handled in the background without killing performance.

So from that perspective, ECE is a win here and avoids a problem with the non-ECE, shared-nothing designs: the manual ?mmchdisk <fsname> start -a? operation that is needed after any traditional shared-nothing metadata NSD goes offline to bring it back and protect against further failures.

Despite the operational challenges of the non-ECE design, it can sometimes survive two server failures (if replication factor is 3 and the filesystem descriptor quorum wasn?t lost by the two failures) which a 4 node ECE cluster cannot.  Given that the world is complex and unexpected things can happen, I?d personally recommend redistributing the 24 disks across 6 servers if you can, so that the design could always survive 2 node failures.  I?ve run this design and it?s fairly robust.

In any event, you should of course test the failure scenarios yourself before going into production to validate them and familiarize yourself with the process.  And a special note on ECE: due to the cooperative nature at the pdisk level, the network between the servers in the RG should be as reliable as possible and any network redundancy should also be tested ahead of time.

-Paul

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> On Behalf Of David Johnson
Sent: Tuesday, July 30, 2019 7:46 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Building GPFS filesystem system data pool on shared nothing NVMe drives

This message was sent by an external party.

Can we confirm the requirement for disks per RG?  I have 4 RG, but only 6 x 3TB NVMe drives per box.



On Jul 29, 2019, at 1:34 PM, Luis Bolinches <luis.bolinches at fi.ibm.com<mailto:luis.bolinches at fi.ibm.com>> wrote:

Hi, from phone so sorry for typos.

I really think you should look into Spectrum Scale Erasure Code Edition (ECE) for this.

Sure you could do a RAID on each node as you mention here but that sounds like a lot of waste to me on storage capacity. Not to forget you get other goodies like end to end checksum and rapid rebuilds with ECE, among others.

Four servers is the minimum requirement for ECE (4+3p) and from top of my head 12 disk per RG, you are fine with both requirements.

There is a presentation on ECE on the user group web page from London May 2019 were we talk about ECE.

And the ibm page of the product https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_intro.htm
--
Cheers

El 29 jul 2019, a las 19:06, David Johnson <david_johnson at brown.edu<mailto:david_johnson at brown.edu>> escribi?:
We are planning a 5.0.x upgrade onto new hardware to make use of the new 5.x GPFS features.
The goal is to use up to four NSD nodes for metadata, each one with 6 NVMe drives (to be determined
whether we use Intel VROC for raid 5 or raid 1, or just straight disks).

So questions ?
Has anyone done system pool on shared nothing cluster?  How did you set it up?
With default metadata replication set at 3, can you make use of four NSD nodes effectively?
How would one design the location vectors and failure groups so that the system metadata is
spread evenly across the four servers?

Thanks,
? ddj
Dave Johnson
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Ellei edell? ole toisin mainittu: / Unless stated otherwise above:
Oy IBM Finland Ab
PL 265, 00101 Helsinki, Finland
Business ID, Y-tunnus: 0195876-3
Registered in Finland
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190730/d782a1c1/attachment-0001.htm>

From luis.bolinches at fi.ibm.com  Tue Jul 30 15:54:42 2019
From: luis.bolinches at fi.ibm.com (Luis Bolinches)
Date: Tue, 30 Jul 2019 14:54:42 +0000
Subject: [gpfsug-discuss] Building GPFS filesystem system data pool on
 shared nothing NVMe drives
In-Reply-To: <b227c5eadc5c46cd9e63b19e51416502@deshaw.com>
References: <b227c5eadc5c46cd9e63b19e51416502@deshaw.com>,
	<OFC1ED3470.4BE6E16D-ON00258446.006095F7-1564421698664@notes.na.collabserv.com><D53515DA-94D8-400B-80E0-8033633A5F21@brown.edu><5664c4aaea5e4fbfac1f8a72acb080dc@deshaw.com><261C855E-FC23-43C6-8A76-FEAE5B48221C@brown.edu>
Message-ID: <OF31F22F72.D63B7538-ON00258447.00510DA2-00258447.0051E9DD@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190730/7157fa12/attachment-0001.htm>

From novosirj at rutgers.edu  Mon Jul  1 06:42:51 2019
From: novosirj at rutgers.edu (Ryan Novosielski)
Date: Mon, 1 Jul 2019 05:42:51 +0000
Subject: [gpfsug-discuss] Any guidelines for choosing vdisk size?
Message-ID: <E55F8C7F-48A4-4A66-ABF8-880A5232F39D@oarc.rutgers.edu>

Good morning,

Was wondering if anyone could point me to any tips or provide some regarding choosing vdisk size? My understanding is that too small is a waste of resources in the form of overhead, and that there is an upper limit. but that generally within a pool, you want them to be the same size, so that if you aren?t allocating the entire storage space on the system straight off, you?ll need to choose a reasonable size or be left with unusable space (eg. you will waste space if you went with, say, 200 GB vdisk sizes on a 500 GB array).

Anyone have any tips? Do people just generally allocate the whole thing and have one 2 vdisks (redundancy)? Seems you have some more flexibility if you don?t do that and have to, say, create a storage pool or filesystem from scratch to take advantage of features in a newer FS version or what have you.

Thanks for the help ? spent a lot of time looking for this previously, but never asked on the list.
 
--
____
|| \\UTGERS,  	 |---------------------------*O*---------------------------
||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
     `'


From abeattie at au1.ibm.com  Mon Jul  1 06:54:37 2019
From: abeattie at au1.ibm.com (Andrew Beattie)
Date: Mon, 1 Jul 2019 05:54:37 +0000
Subject: [gpfsug-discuss] Any guidelines for choosing vdisk size?
In-Reply-To: <E55F8C7F-48A4-4A66-ABF8-880A5232F39D@oarc.rutgers.edu>
References: <E55F8C7F-48A4-4A66-ABF8-880A5232F39D@oarc.rutgers.edu>
Message-ID: <OF54C435D4.B3E41DA2-ON0025842A.001FD56B-0025842A.002077AE@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190701/6acfe5f8/attachment-0002.htm>

From novosirj at rutgers.edu  Mon Jul  1 07:01:46 2019
From: novosirj at rutgers.edu (Ryan Novosielski)
Date: Mon, 1 Jul 2019 06:01:46 +0000
Subject: [gpfsug-discuss] Any guidelines for choosing vdisk size?
In-Reply-To: <OF54C435D4.B3E41DA2-ON0025842A.001FD56B-0025842A.002077AE@notes.na.collabserv.com>
References: <E55F8C7F-48A4-4A66-ABF8-880A5232F39D@oarc.rutgers.edu>
	<OF54C435D4.B3E41DA2-ON0025842A.001FD56B-0025842A.002077AE@notes.na.collabserv.com>
Message-ID: <D599D062-6286-477C-B494-4AD0984691AD@rutgers.edu>

Sure; thanks, I meant to add those details:

In our case, all GSS/DSS-G with GNR.

The context here is general information, but specifically designing new filesystems/setting up one of these systems from scratch in what I?d figure to be the traditional way: don?t allocate the entire storage if you?re not totally sure the way the growth will go, but don?t leave behind unusable space or space that forces you to later change vdisk sizes. Essentially allocate something today that makes sense, but leaves room open for sensible growth with both the existing hardware and new, if expanded. If I?m thinking about this generally the wrong way, let me know.

The systems we currently own are GSS26 (6 TB drives, fully populated ? I forget how many drives, 350 or so?) , and a few DSS-G 220 w/10 TB drives (also fully populated with 168 drives).

> On Jul 1, 2019, at 1:54 AM, Andrew Beattie <abeattie at au1.ibm.com> wrote:
> 
> Hi Ryan,
>  
> Can I ask a couple of clarifying questions?
>  
> Is this in an ESS / GSS environment with Scale native raid ?
> Is this in classic San attached storage environment / or Direct Attached Disk without Scale Native Raid?
>  
> What are you trying to achieve?
> Are you presenting the Vdisk into an existing filesystem ?
> Are you creating an new filesystem?
>  
>  
> Regards,
>  
> Andrew Beattie
> File and Object Storage Technical Specialist - A/NZ
> IBM Systems - Storage
> Phone: 614-2133-7927
> E-mail: abeattie at au1.ibm.com
>  
>  
> ----- Original message -----
> From: Ryan Novosielski <novosirj at rutgers.edu>
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Cc:
> Subject: [EXTERNAL] [gpfsug-discuss] Any guidelines for choosing vdisk size?
> Date: Mon, Jul 1, 2019 3:43 PM
>  
> Good morning,
> 
> Was wondering if anyone could point me to any tips or provide some regarding choosing vdisk size? My understanding is that too small is a waste of resources in the form of overhead, and that there is an upper limit. but that generally within a pool, you want them to be the same size, so that if you aren?t allocating the entire storage space on the system straight off, you?ll need to choose a reasonable size or be left with unusable space (eg. you will waste space if you went with, say, 200 GB vdisk sizes on a 500 GB array).
> 
> Anyone have any tips? Do people just generally allocate the whole thing and have one 2 vdisks (redundancy)? Seems you have some more flexibility if you don?t do that and have to, say, create a storage pool or filesystem from scratch to take advantage of features in a newer FS version or what have you.
> 
> Thanks for the help ? spent a lot of time looking for this previously, but never asked on the list.
>  
> --
> ____
> || \\UTGERS,   |---------------------------*O*---------------------------
> ||_// the State |         Ryan Novosielski - novosirj at rutgers.edu
> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
> ||  \\    of NJ | Office of Advanced Research Computing - MSB C630, Newark
>      `'
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss 
>  
>  
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss


From mhennecke at lenovo.com  Mon Jul  1 08:25:23 2019
From: mhennecke at lenovo.com (Michael Hennecke)
Date: Mon, 1 Jul 2019 07:25:23 +0000
Subject: [gpfsug-discuss] Any guidelines for choosing vdisk size?
Message-ID: <TY2PR03MB446249F071E9252F75D87E5ACCF90@TY2PR03MB4462.apcprd03.prod.outlook.com>

Ryan,

That is a question with many variables. Some of the factors influencing the decision are:
* Are you sharing data and metadata (the GNR default these days), or are you building separate data and metadata vdisks (the "older" default, and still useful for some use cases)? If you have separate metadata, you need to ensure you leave enough free space in the RGs to account for metadata growth beyond your initial sizing.
* Are you creating vdisks for multiple filesystems on the same building block, or just for one filesystem? 
* How likely is it that space needs to be (re-)allocated between different filesystems?
* How likely is it that new storage will be added to existing filesystems?
* Is there a requirement to include building blocks of different size (different #disks, or different capacity of disks) in the same filesystem?

All of these influence the decision on the "best" vdisk quantity and capacities. If you have a specific configuration question, I'm happy to discuss that offline.
 
In general I advise customers to use at least two equally sized vdisks per RG (4 per building block), so there is more flexibility when capacity needs to be moved around. But many sites that have homogeneous building blocks and don't expect changes in their storage configuration just go with a single vdisk per RG (2 per building block).

Either way, it is a good idea (in environments with more than a single building block) to put aside a small "test" vdisk per RG just for performance testing. So you can do performance validations on each building block individually even if your main filesystem spans all building block. Size of that "test" vdisk should be big enough so you can run a few minutes of sequential IOR against it without filling it up 100%.


Mit freundlichen Gr?ssen / Best regards,

Michael Hennecke
HPC Chief Technologist - HPC and AI Business Unit  
mailto:mhennecke at lenovo.com
--
Lenovo Global Technology (Germany) GmbH * Am Zehnthof 77 * D-45307 Essen * Germany 
Gesch?ftsf?hrung: Colm Gleeson, Christophe Laurent * Sitz der Gesellschaft: Stuttgart * HRB-Nr.: 758298, AG Stuttgart

-----Original Message-----
From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of Ryan Novosielski
Sent: Monday, 1 July, 2019 07:43
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [External] [gpfsug-discuss] Any guidelines for choosing vdisk size?

Good morning,

Was wondering if anyone could point me to any tips or provide some regarding choosing vdisk size? My understanding is that too small is a waste of resources in the form of overhead, and that there is an upper limit. but that generally within a pool, you want them to be the same size, so that if you aren?t allocating the entire storage space on the system straight off, you?ll need to choose a reasonable size or be left with unusable space (eg. you will waste space if you went with, say, 200 GB vdisk sizes on a 500 GB array).

Anyone have any tips? Do people just generally allocate the whole thing and have one 2 vdisks (redundancy)? Seems you have some more flexibility if you don?t do that and have to, say, create a storage pool or filesystem from scratch to take advantage of features in a newer FS version or what have you.

Thanks for the help ? spent a lot of time looking for this previously, but never asked on the list.
 
--
____
|| \\UTGERS,  	 |---------------------------*O*---------------------------
||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
     `'

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

From janfrode at tanso.net  Mon Jul  1 08:56:22 2019
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Mon, 1 Jul 2019 09:56:22 +0200
Subject: [gpfsug-discuss] Any guidelines for choosing vdisk size?
In-Reply-To: <E55F8C7F-48A4-4A66-ABF8-880A5232F39D@oarc.rutgers.edu>
References: <E55F8C7F-48A4-4A66-ABF8-880A5232F39D@oarc.rutgers.edu>
Message-ID: <CAHwPatihe-0esH-JbOzfENS65RvjY9ViQKvEyPnXr1+qhv_yqQ@mail.gmail.com>

I would mainly consider future upgrades. F.ex. do one vdisk per disk shelf
per rg. F.ex. for a GL6S you would have 12 vdisks, and if you add a GL4S
you would add 8 more vdisks, then each spindle of both systems should get
approximately the same number of IOs.

Another thing to consider is re-allocating capacity. If your vdisks are
very large, it might be difficult to free up a full vdisk if you need to
reorganize and create new filesystems, or add more 3way replicated
metadata..

But mainly I create large vdisks.. haven?t been able to show any
performance difference between one vdisk per RG vs. 10 vdisks per RG.


  -jf

man. 1. jul. 2019 kl. 07:42 skrev Ryan Novosielski <novosirj at rutgers.edu>:

> Good morning,
>
> Was wondering if anyone could point me to any tips or provide some
> regarding choosing vdisk size? My understanding is that too small is a
> waste of resources in the form of overhead, and that there is an upper
> limit. but that generally within a pool, you want them to be the same size,
> so that if you aren?t allocating the entire storage space on the system
> straight off, you?ll need to choose a reasonable size or be left with
> unusable space (eg. you will waste space if you went with, say, 200 GB
> vdisk sizes on a 500 GB array).
>
> Anyone have any tips? Do people just generally allocate the whole thing
> and have one 2 vdisks (redundancy)? Seems you have some more flexibility if
> you don?t do that and have to, say, create a storage pool or filesystem
> from scratch to take advantage of features in a newer FS version or what
> have you.
>
> Thanks for the help ? spent a lot of time looking for this previously, but
> never asked on the list.
>
> --
> ____
> || \\UTGERS,     |---------------------------*O*---------------------------
> ||_// the State  |         Ryan Novosielski - novosirj at rutgers.edu
> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
> ||  \\    of NJ  | Office of Advanced Research Computing - MSB C630, Newark
>      `'
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190701/9a8c578e/attachment-0002.htm>

From abeattie at au1.ibm.com  Mon Jul  1 09:12:41 2019
From: abeattie at au1.ibm.com (Andrew Beattie)
Date: Mon, 1 Jul 2019 08:12:41 +0000
Subject: [gpfsug-discuss] Any guidelines for choosing vdisk size?
In-Reply-To: <D599D062-6286-477C-B494-4AD0984691AD@rutgers.edu>
References: <D599D062-6286-477C-B494-4AD0984691AD@rutgers.edu>,
	<E55F8C7F-48A4-4A66-ABF8-880A5232F39D@oarc.rutgers.edu><OF54C435D4.B3E41DA2-ON0025842A.001FD56B-0025842A.002077AE@notes.na.collabserv.com>
Message-ID: <OF10ACC789.BB227E77-ON0025842A.002C56F8-0025842A.002D1B7B@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190701/2e915488/attachment-0002.htm>

From UWEFALKE at de.ibm.com  Mon Jul  1 09:29:49 2019
From: UWEFALKE at de.ibm.com (Uwe Falke)
Date: Mon, 1 Jul 2019 10:29:49 +0200
Subject: [gpfsug-discuss] Any guidelines for choosing vdisk size?
In-Reply-To: <TY2PR03MB446249F071E9252F75D87E5ACCF90@TY2PR03MB4462.apcprd03.prod.outlook.com>
References: <TY2PR03MB446249F071E9252F75D87E5ACCF90@TY2PR03MB4462.apcprd03.prod.outlook.com>
Message-ID: <OF01F0AAB3.D10E313F-ONC125842A.002D8C69-C125842A.002EAD36@notes.na.collabserv.com>

gpfsug-discuss-bounces at spectrumscale.org wrote on 01/07/2019 09:25:23:

> From: Michael Hennecke <mhennecke at lenovo.com>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Date: 01/07/2019 09:38
> Subject: [EXTERNAL] Re: [gpfsug-discuss] Any guidelines for choosing
> vdisk size?
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
...
> Either way, it is a good idea (in environments with more than a 
> single building block) to put aside a small "test" vdisk per RG just
> for performance testing. So you can do performance validations on 
> each building block individually even if your main filesystem spans 
> all building block. Size of that "test" vdisk should be big enough 
> so you can run a few minutes of sequential IOR against it without 
> filling it up 100%.

However, be aware that GNR limits the number of ptracks per vdisk 
according to the vdisk size. That could limit the seen performance 
especially for writes. 
I've experienced that with vdisk sizes of 509GiB in DAs of 288TiB of ESS 
GL4 (can't recall exactly if ptracks are assigned by absolute or by 
relative vdisk size). 


 
Mit freundlichen Gr??en / Kind regards

 
Dr. Uwe Falke
 
IT Specialist
High Performance Computing Services / Integrated Technology Services / 
Data Center Services
-------------------------------------------------------------------------------------------------------------------------------------------
IBM Deutschland
Rathausstr. 7
09111 Chemnitz
Phone: +49 371 6978 2165
Mobile: +49 175 575 2877
E-Mail: uwefalke at de.ibm.com
-------------------------------------------------------------------------------------------------------------------------------------------
IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: 
Thomas Wolter, Sven Schoo?
Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, 
HRB 17122 


gpfsug-discuss-bounces at spectrumscale.org wrote on 01/07/2019 09:25:23:

> From: Michael Hennecke <mhennecke at lenovo.com>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Date: 01/07/2019 09:38
> Subject: [EXTERNAL] Re: [gpfsug-discuss] Any guidelines for choosing
> vdisk size?
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> 
> Ryan,
> 
> That is a question with many variables. Some of the factors 
> influencing the decision are:
> * Are you sharing data and metadata (the GNR default these days), or
> are you building separate data and metadata vdisks (the "older" 
> default, and still useful for some use cases)? If you have separate 
> metadata, you need to ensure you leave enough free space in the RGs 
> to account for metadata growth beyond your initial sizing.
> * Are you creating vdisks for multiple filesystems on the same 
> building block, or just for one filesystem? 
> * How likely is it that space needs to be (re-)allocated between 
> different filesystems?
> * How likely is it that new storage will be added to existing 
filesystems?
> * Is there a requirement to include building blocks of different 
> size (different #disks, or different capacity of disks) in the same 
> filesystem?
> 
> All of these influence the decision on the "best" vdisk quantity and
> capacities. If you have a specific configuration question, I'm happy
> to discuss that offline.
> 
> In general I advise customers to use at least two equally sized 
> vdisks per RG (4 per building block), so there is more flexibility 
> when capacity needs to be moved around. But many sites that have 
> homogeneous building blocks and don't expect changes in their 
> storage configuration just go with a single vdisk per RG (2 per 
> building block).
> 
> Either way, it is a good idea (in environments with more than a 
> single building block) to put aside a small "test" vdisk per RG just
> for performance testing. So you can do performance validations on 
> each building block individually even if your main filesystem spans 
> all building block. Size of that "test" vdisk should be big enough 
> so you can run a few minutes of sequential IOR against it without 
> filling it up 100%.
> 
> 
> Mit freundlichen Gr?ssen / Best regards,
> 
> Michael Hennecke
> HPC Chief Technologist - HPC and AI Business Unit 
> mailto:mhennecke at lenovo.com
> --
> Lenovo Global Technology (Germany) GmbH * Am Zehnthof 77 * D-45307 
> Essen * Germany 
> Gesch?ftsf?hrung: Colm Gleeson, Christophe Laurent * Sitz der 
> Gesellschaft: Stuttgart * HRB-Nr.: 758298, AG Stuttgart
> 
> -----Original Message-----
> From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-
> bounces at spectrumscale.org> On Behalf Of Ryan Novosielski
> Sent: Monday, 1 July, 2019 07:43
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: [External] [gpfsug-discuss] Any guidelines for choosing vdisk 
size?
> 
> Good morning,
> 
> Was wondering if anyone could point me to any tips or provide some 
> regarding choosing vdisk size? My understanding is that too small is
> a waste of resources in the form of overhead, and that there is an 
> upper limit. but that generally within a pool, you want them to be 
> the same size, so that if you aren?t allocating the entire storage 
> space on the system straight off, you?ll need to choose a reasonable
> size or be left with unusable space (eg. you will waste space if you
> went with, say, 200 GB vdisk sizes on a 500 GB array).
> 
> Anyone have any tips? Do people just generally allocate the whole 
> thing and have one 2 vdisks (redundancy)? Seems you have some more 
> flexibility if you don?t do that and have to, say, create a storage 
> pool or filesystem from scratch to take advantage of features in a 
> newer FS version or what have you.
> 
> Thanks for the help ? spent a lot of time looking for this 
> previously, but never asked on the list.
> 
> --
> ____
> || \\UTGERS, |---------------------------*O*---------------------------
> ||_// the State    |         Ryan Novosielski - novosirj at rutgers.edu
> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS 
Campus
> ||  \\    of NJ    | Office of Advanced Research Computing - MSB C630, 
Newark
>      `'
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?
> 
u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-
> 
siA1ZOg&r=fTuVGtgq6A14KiNeaGfNZzOOgtHW5Lm4crZU6lJxtB8&m=XHrjWG2mnps3wxUISQHIZDR08HKkIKdcekRQibivZB4&s=wO1NamNaVBSnk_Kbe7Xl45aGtu7W6CBj0tHOlysTui4&e=
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?
> 
u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-
> 
siA1ZOg&r=fTuVGtgq6A14KiNeaGfNZzOOgtHW5Lm4crZU6lJxtB8&m=XHrjWG2mnps3wxUISQHIZDR08HKkIKdcekRQibivZB4&s=wO1NamNaVBSnk_Kbe7Xl45aGtu7W6CBj0tHOlysTui4&e=
> 




From kkr at lbl.gov  Tue Jul  2 18:45:30 2019
From: kkr at lbl.gov (Kristy Kallback-Rose)
Date: Tue, 2 Jul 2019 10:45:30 -0700
Subject: [gpfsug-discuss] Hold the Date - September 23 and 24
Message-ID: <3F2B08E9-C6E3-412B-9308-D79E3480C5DA@lbl.gov>

Hello,

HPCXXL will be hosted by NERSC (Berkeley, CA) this September. As part of this event, there will be approximately a day and a half on GPFS content. We have done this type of event in the past, and as before, the GPFS days will be free to attend, but you do need to register. 

We?ll have more details soon, mark your calendars. 

Initial details: https://www.spectrumscaleug.org/event/spectrum-scale-gpfs-days-part-of-hpcxxl/ <https://www.spectrumscaleug.org/event/spectrum-scale-gpfs-days-part-of-hpcxxl/>

Best,
Kristy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190702/9747f62d/attachment-0002.htm>

From novosirj at rutgers.edu  Thu Jul  4 04:55:56 2019
From: novosirj at rutgers.edu (Ryan Novosielski)
Date: Thu, 4 Jul 2019 03:55:56 +0000
Subject: [gpfsug-discuss] Combined system (metadata) and data pool on GPFS
	5.x?
Message-ID: <DCA32300-AA3F-4E63-8248-5A241F6D4E4C@rutgers.edu>

Hi there,

Was at the Lenovo GNR/GPFS 5.x seminar the other week and we briefly discussed whether or not the recommendation was to place data in the system pool (eg. have one combined system/data pool), or to have them separate. We?ve got a GSS26 that we are creating new filesystems on for high speed scratch (16 MB blocks).

I unfortunately did not note all of the considerations/caveats related to this choice. Does anyone who knows/was there and can remember remind me?

The recollection I have is that you may have some contention for I/O if your metadata is on your data disks, and you won?t be able to have different redundancy settings. Once upon a time, block size also mattered, but not so much on 5.x.

Thanks!

--
____
|| \\UTGERS,  	 |---------------------------*O*---------------------------
||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
     `'


From S.J.Thompson at bham.ac.uk  Thu Jul  4 09:51:18 2019
From: S.J.Thompson at bham.ac.uk (Simon Thompson)
Date: Thu, 4 Jul 2019 08:51:18 +0000
Subject: [gpfsug-discuss] Combined system (metadata) and data pool on
 GPFS 5.x?
In-Reply-To: <DCA32300-AA3F-4E63-8248-5A241F6D4E4C@rutgers.edu>
References: <DCA32300-AA3F-4E63-8248-5A241F6D4E4C@rutgers.edu>
Message-ID: <205352CF-6035-4DDA-812F-9BD9FA0152B7@bham.ac.uk>

I wasn't but ...

If your metadata is in the same RG/DA as your data, even if it?s a separate pool, the contention is the same. But if you have 2 DAs, then you have more spinning disks underneath to provide iops anyway.

We only separate our metadata and data into pools because we want to replicate the metadata between DSS-G systems, but not data. And also have some data which is replicated but not all, and the only way we could figure to do that is by having separate pools and failure groups. i.e. we didn't want data in the system pool that wasn't replicated being placed on the second site.

And I agree, block sizes don't matter so much anymore, except remember the "variable" sub-blocks are based on the smallest block size used, so we ended up with 16MB blocks for both data and metadata.

Simon

?On 04/07/2019, 04:56, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Ryan Novosielski" <gpfsug-discuss-bounces at spectrumscale.org on behalf of novosirj at rutgers.edu> wrote:

    Hi there,
    
    Was at the Lenovo GNR/GPFS 5.x seminar the other week and we briefly discussed whether or not the recommendation was to place data in the system pool (eg. have one combined system/data pool), or to have them separate. We?ve got a GSS26 that we are creating new filesystems on for high speed scratch (16 MB blocks).
    
    I unfortunately did not note all of the considerations/caveats related to this choice. Does anyone who knows/was there and can remember remind me?
    
    The recollection I have is that you may have some contention for I/O if your metadata is on your data disks, and you won?t be able to have different redundancy settings. Once upon a time, block size also mattered, but not so much on 5.x.
    
    Thanks!
    
    --
    ____
    || \\UTGERS,  	 |---------------------------*O*---------------------------
    ||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
    || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
    ||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
         `'
    
    _______________________________________________
    gpfsug-discuss mailing list
    gpfsug-discuss at spectrumscale.org
    http://gpfsug.org/mailman/listinfo/gpfsug-discuss
    


From damir.krstic at gmail.com  Wed Jul 10 13:16:31 2019
From: damir.krstic at gmail.com (Damir Krstic)
Date: Wed, 10 Jul 2019 07:16:31 -0500
Subject: [gpfsug-discuss] slow filesystem
Message-ID: <CAKV+WqctLVWdr0E5sA+G9pZpc9F16Pc7Q-VT1C81zf6R8gkGmA@mail.gmail.com>

Over last couple of days our reads and writes on our compute cluster are
experiencing real slow reads and writes on one of the filesystems in the
cluster. We are talking with IBM and have Sev. 1 ticket open, but I figured
to ask here about the warning message we are seeing in GPFS logs.

The cluster is configured as following:

4 IO servers in the main gpfs cluster
700+ compute nodes in the gpfs cluster

home filesystem is slow but projects filesystem seems to be fast. Not many
waiters on the IO servers (almost none) but a lot of waiters on the remote
cluster.

The message that is giving us a pause is the following:

Jul 10 07:05:31 gssio4 mmfs: [N] Writing into file
/var/mmfs/gen/LastLeaseRequestSent took 10.5 seconds

Why is taking so long to write to to the local file?

Thanks,
Damir
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190710/5aa0023a/attachment-0002.htm>

From Kevin.Buterbaugh at Vanderbilt.Edu  Wed Jul 10 13:21:52 2019
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Wed, 10 Jul 2019 12:21:52 +0000
Subject: [gpfsug-discuss] slow filesystem
In-Reply-To: <CAKV+WqctLVWdr0E5sA+G9pZpc9F16Pc7Q-VT1C81zf6R8gkGmA@mail.gmail.com>
References: <CAKV+WqctLVWdr0E5sA+G9pZpc9F16Pc7Q-VT1C81zf6R8gkGmA@mail.gmail.com>
Message-ID: <BEF9A8D8-DFFE-4102-8ADF-6A919F9A5094@vanderbilt.edu>

Hi Damir,

Have you checked to see whether gssio4 might have a failing internal HD / SSD?  Thanks?

Kevin

On Jul 10, 2019, at 7:16 AM, Damir Krstic <damir.krstic at gmail.com<mailto:damir.krstic at gmail.com>> wrote:

Over last couple of days our reads and writes on our compute cluster are experiencing real slow reads and writes on one of the filesystems in the cluster. We are talking with IBM and have Sev. 1 ticket open, but I figured to ask here about the warning message we are seeing in GPFS logs.

The cluster is configured as following:

4 IO servers in the main gpfs cluster
700+ compute nodes in the gpfs cluster

home filesystem is slow but projects filesystem seems to be fast. Not many waiters on the IO servers (almost none) but a lot of waiters on the remote cluster.

The message that is giving us a pause is the following:
Jul 10 07:05:31 gssio4 mmfs: [N] Writing into file /var/mmfs/gen/LastLeaseRequestSent took 10.5 seconds

Why is taking so long to write to to the local file?

Thanks,
Damir

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&amp;data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cede76a6c2bd743c836b708d705307a86%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636983578133164831&amp;sdata=ATtqXDDChaZTouZZRkjf%2F79pIK%2Fc1DAwq6KUU%2FKYca4%3D&amp;reserved=0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190710/014c4ff6/attachment-0002.htm>

From damir.krstic at gmail.com  Wed Jul 10 13:22:37 2019
From: damir.krstic at gmail.com (Damir Krstic)
Date: Wed, 10 Jul 2019 07:22:37 -0500
Subject: [gpfsug-discuss] slow filesystem
In-Reply-To: <BEF9A8D8-DFFE-4102-8ADF-6A919F9A5094@vanderbilt.edu>
References: <CAKV+WqctLVWdr0E5sA+G9pZpc9F16Pc7Q-VT1C81zf6R8gkGmA@mail.gmail.com>
	<BEF9A8D8-DFFE-4102-8ADF-6A919F9A5094@vanderbilt.edu>
Message-ID: <CAKV+WqfUQ3wWaKqOB-aVKDRNsvw7SutWT64owkCJvpkndWtWUA@mail.gmail.com>

mmlspdisk all --not-ok does not indicate any failed hard drives.

On Wed, Jul 10, 2019 at 7:22 AM Buterbaugh, Kevin L <
Kevin.Buterbaugh at vanderbilt.edu> wrote:

> Hi Damir,
>
> Have you checked to see whether gssio4 might have a failing internal HD /
> SSD?  Thanks?
>
> Kevin
>
> On Jul 10, 2019, at 7:16 AM, Damir Krstic <damir.krstic at gmail.com> wrote:
>
> Over last couple of days our reads and writes on our compute cluster are
> experiencing real slow reads and writes on one of the filesystems in the
> cluster. We are talking with IBM and have Sev. 1 ticket open, but I figured
> to ask here about the warning message we are seeing in GPFS logs.
>
> The cluster is configured as following:
>
> 4 IO servers in the main gpfs cluster
> 700+ compute nodes in the gpfs cluster
>
> home filesystem is slow but projects filesystem seems to be fast. Not many
> waiters on the IO servers (almost none) but a lot of waiters on the remote
> cluster.
>
> The message that is giving us a pause is the following:
> Jul 10 07:05:31 gssio4 mmfs: [N] Writing into file
> /var/mmfs/gen/LastLeaseRequestSent took 10.5 seconds
>
> Why is taking so long to write to to the local file?
>
> Thanks,
> Damir
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
>
> https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&amp;data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cede76a6c2bd743c836b708d705307a86%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636983578133164831&amp;sdata=ATtqXDDChaZTouZZRkjf%2F79pIK%2Fc1DAwq6KUU%2FKYca4%3D&amp;reserved=0
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190710/dac7390e/attachment-0002.htm>

From valdis.kletnieks at vt.edu  Wed Jul 10 14:14:31 2019
From: valdis.kletnieks at vt.edu (Valdis Kl=?utf-8?Q?=c4=93?=tnieks)
Date: Wed, 10 Jul 2019 09:14:31 -0400
Subject: [gpfsug-discuss] slow filesystem
In-Reply-To: <CAKV+WqfUQ3wWaKqOB-aVKDRNsvw7SutWT64owkCJvpkndWtWUA@mail.gmail.com>
References: <CAKV+WqctLVWdr0E5sA+G9pZpc9F16Pc7Q-VT1C81zf6R8gkGmA@mail.gmail.com>
	<BEF9A8D8-DFFE-4102-8ADF-6A919F9A5094@vanderbilt.edu>
	<CAKV+WqfUQ3wWaKqOB-aVKDRNsvw7SutWT64owkCJvpkndWtWUA@mail.gmail.com>
Message-ID: <5311.1562764471@turing-police>

On Wed, 10 Jul 2019 07:22:37 -0500, Damir Krstic said:

> mmlspdisk all --not-ok does not indicate any failed hard drives.

Not a GPFS drive - if a write to /var is taking 10 seconds, that indicates
that there is likely a problem with the disk(s) that the system lives on,
and you're hitting recoverable write errors....

Jul 10 07:05:31 gssio4 mmfs: [N] Writing into file /var/mmfs/gen/LastLeaseRequestSent took 10.5 seconds

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190710/f532fd53/attachment-0002.sig>

From heinrich.billich at id.ethz.ch  Wed Jul 17 12:41:54 2019
From: heinrich.billich at id.ethz.ch (Billich  Heinrich Rainer (ID SD))
Date: Wed, 17 Jul 2019 11:41:54 +0000
Subject: [gpfsug-discuss] How to prove that data is in inode
Message-ID: <71647F14-8BEC-4E89-A30B-F0D880A81A84@id.ethz.ch>

Hello,

How can I prove that data of a small file is stored in the inode (and not on a data nsd)?

We have a filesystem with 4k inodes on Scale 5.0.2 , but it seems there is no file data in the inodes?

I would expect that 'stat' reports 'Blocks: 0'  for a small file, but I see 'Blocks:1'.

Cheers,

Heiner

I tried

[]# rm -f test; echo hello > test
[]# ls -ls test
1 -rw-r--r-- 1 root root 6 Jul 17 13:11 test
[root at testnas13ems01 test]# stat test
  File: ?test?
  Size: 6         	Blocks: 1          IO Block: 1048576 regular file
Device: 2dh/45d	Inode: 353314      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-07-17 13:11:03.037049000 +0200
Modify: 2019-07-17 13:11:03.037331000 +0200
Change: 2019-07-17 13:11:03.037259319 +0200
 Birth: -
[root at testnas13ems01 test]# du test
1	test
[root at testnas13ems01 test]# du -b test
6	test
[root at testnas13ems01 test]#

Filesystem

# mmlsfs f****
flag                value                    description
------------------- ------------------------ -----------------------------------
 -f                 32768                    Minimum fragment (subblock) size in bytes
 -i                 4096                     Inode size in bytes
 -I                 32768                    Indirect block size in bytes
 -m                 1                        Default number of metadata replicas
 -M                 2                        Maximum number of metadata replicas
 -r                 1                        Default number of data replicas
 -R                 2                        Maximum number of data replicas
 -j                 cluster                  Block allocation type
 -D                 nfs4                     File locking semantics in effect
 -k                 nfs4                     ACL semantics in effect
 -n                 32                       Estimated number of nodes that will mount file system
 -B                 1048576                  Block size
 -Q                 user;group;fileset       Quotas accounting enabled
                    user;group;fileset       Quotas enforced
                    user;group;fileset       Default quotas enabled
 --perfileset-quota Yes                      Per-fileset quota enforcement
 --filesetdf        Yes                      Fileset df enabled?
 -V                 20.01 (5.0.2.0)          Current file system version
                    15.01 (4.2.0.0)          Original file system version
 --create-time      ***** 2017 File system creation time
 -z                 No                       Is DMAPI enabled?
 -L                 33554432                 Logfile size
 -E                 Yes                      Exact mtime mount option
 -S                 relatime                 Suppress atime mount option
 -K                 whenpossible             Strict replica allocation option
 --fastea           Yes                      Fast external attributes enabled?
 --encryption       No                       Encryption enabled?
 --inode-limit      1294592                  Maximum number of inodes in all inode spaces
 --log-replicas     0                        Number of log replicas
 --is4KAligned      Yes                      is4KAligned?
 --rapid-repair     Yes                      rapidRepair enabled?
 --write-cache-threshold 0                   HAWC Threshold (max 65536)
 --subblocks-per-full-block 32               Number of subblocks per full block
 -P                 system;data              Disk storage pools in file system
 --file-audit-log   No                       File Audit Logging enabled?
 --maintenance-mode No                       Maintenance Mode enabled?
 -d                 ******
-A                 yes                      Automatic mount option
 -o                 nfssync,nodev            Additional mount options
 -T                 /****                  Default mount point
 --mount-priority   0                        Mount priority

-- 
=======================
Heinrich Billich
ETH Z?rich
Informatikdienste
Tel.: +41 44 632 72 56
heinrich.billich at id.ethz.ch
========================
 
 


From kums at us.ibm.com  Wed Jul 17 13:37:35 2019
From: kums at us.ibm.com (Kumaran Rajaram)
Date: Wed, 17 Jul 2019 08:37:35 -0400
Subject: [gpfsug-discuss] How to prove that data is in inode
In-Reply-To: <71647F14-8BEC-4E89-A30B-F0D880A81A84@id.ethz.ch>
References: <71647F14-8BEC-4E89-A30B-F0D880A81A84@id.ethz.ch>
Message-ID: <OF776130C0.2EA026CD-ON0025843A.0044FED5-8525843A.00455C19@notes.na.collabserv.com>

Hi,

>> How can I prove that data of a small file is stored in the inode (and
not on a data nsd)?

You may use echo "inode file_inode_number" | tsdbfs fs_device | grep
indirectionLevel and if it points to INODE, then the file is stored in the
inodes

# 4K Inode Size
# mmlsfs gpfs3a | grep 'Inode size'
 -i                 4096                     Inode size in bytes

# Small file
# ls -l /mnt/gpfs3a/hello.txt
-rw-r--r-- 1 root root 6 Jul 17 08:32 /mnt/gpfs3a/hello.txt

# ls -i /mnt/gpfs3a/hello.txt
91649 /mnt/gpfs3a/hello.txt

#File is inlined within Inode
# echo "inode 91649" | tsdbfs gpfs3a | grep indirectionLevel
  indirectionLevel=INODE status=USERFILE

Regards,
-Kums





From:	"Billich  Heinrich Rainer (ID SD)"
            <heinrich.billich at id.ethz.ch>
To:	gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:	07/17/2019 07:49 AM
Subject:	[EXTERNAL] [gpfsug-discuss] How to prove that data is in inode
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



Hello,

How can I prove that data of a small file is stored in the inode (and not
on a data nsd)?

We have a filesystem with 4k inodes on Scale 5.0.2 , but it seems there is
no file data in the inodes?

I would expect that 'stat' reports 'Blocks: 0'  for a small file, but I see
'Blocks:1'.

Cheers,

Heiner

I tried

[]# rm -f test; echo hello > test
[]# ls -ls test
1 -rw-r--r-- 1 root root 6 Jul 17 13:11 test
[root at testnas13ems01 test]# stat test
  File: ?test?
  Size: 6         		 Blocks: 1          IO Block: 1048576 regular
file
Device: 2dh/45d		 Inode: 353314      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-07-17 13:11:03.037049000 +0200
Modify: 2019-07-17 13:11:03.037331000 +0200
Change: 2019-07-17 13:11:03.037259319 +0200
 Birth: -
[root at testnas13ems01 test]# du test
1		 test
[root at testnas13ems01 test]# du -b test
6		 test
[root at testnas13ems01 test]#

Filesystem

# mmlsfs f****
flag                value                    description
------------------- ------------------------
-----------------------------------
 -f                 32768                    Minimum fragment (subblock)
size in bytes
 -i                 4096                     Inode size in bytes
 -I                 32768                    Indirect block size in bytes
 -m                 1                        Default number of metadata
replicas
 -M                 2                        Maximum number of metadata
replicas
 -r                 1                        Default number of data
replicas
 -R                 2                        Maximum number of data
replicas
 -j                 cluster                  Block allocation type
 -D                 nfs4                     File locking semantics in
effect
 -k                 nfs4                     ACL semantics in effect
 -n                 32                       Estimated number of nodes that
will mount file system
 -B                 1048576                  Block size
 -Q                 user;group;fileset       Quotas accounting enabled
                    user;group;fileset       Quotas enforced
                    user;group;fileset       Default quotas enabled
 --perfileset-quota Yes                      Per-fileset quota enforcement
 --filesetdf        Yes                      Fileset df enabled?
 -V                 20.01 (5.0.2.0)          Current file system version
                    15.01 (4.2.0.0)          Original file system version
 --create-time      ***** 2017 File system creation time
 -z                 No                       Is DMAPI enabled?
 -L                 33554432                 Logfile size
 -E                 Yes                      Exact mtime mount option
 -S                 relatime                 Suppress atime mount option
 -K                 whenpossible             Strict replica allocation
option
 --fastea           Yes                      Fast external attributes
enabled?
 --encryption       No                       Encryption enabled?
 --inode-limit      1294592                  Maximum number of inodes in
all inode spaces
 --log-replicas     0                        Number of log replicas
 --is4KAligned      Yes                      is4KAligned?
 --rapid-repair     Yes                      rapidRepair enabled?
 --write-cache-threshold 0                   HAWC Threshold (max 65536)
 --subblocks-per-full-block 32               Number of subblocks per full
block
 -P                 system;data              Disk storage pools in file
system
 --file-audit-log   No                       File Audit Logging enabled?
 --maintenance-mode No                       Maintenance Mode enabled?
 -d                 ******
-A                 yes                      Automatic mount option
 -o                 nfssync,nodev            Additional mount options
 -T                 /****                  Default mount point
 --mount-priority   0                        Mount priority

--
=======================
Heinrich Billich
ETH Z?rich
Informatikdienste
Tel.: +41 44 632 72 56
heinrich.billich at id.ethz.ch
========================



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=McIf98wfiVqHU8ZygezLrQ&m=VUhXK3bFtfQfiPegK_nm2SStc9jabkxIAUQruQxyXdI&s=JobCqlXTY87VyE6RxErKI46SdzyDhW5kvZKc9xw6DHM&e=




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190717/da506a92/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190717/da506a92/attachment-0002.gif>

From p.ward at nhm.ac.uk  Wed Jul 17 14:00:16 2019
From: p.ward at nhm.ac.uk (Paul Ward)
Date: Wed, 17 Jul 2019 13:00:16 +0000
Subject: [gpfsug-discuss] Break-out session for new user and
	prospects	[London Usergroup]
In-Reply-To: <776770B2-5F84-4462-B900-58EBB982DC1C@spectrumscale.org>
References: <776770B2-5F84-4462-B900-58EBB982DC1C@spectrumscale.org>
Message-ID: <DB7PR10MB2201EAAA629BB9C10BBE24E0D3C90@DB7PR10MB2201.EURPRD10.PROD.OUTLOOK.COM>

Hi Simon,

Was this a slide presentation? Is it available?

Kindest regards,
Paul

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (Spectrum Scale User Group Chair)
Sent: 30 April 2019 10:25
To: gpfsug-discuss at spectrumscale.org
Subject: [gpfsug-discuss] Break-out session for new user and prospects [London Usergroup]

Hi all,

We know that a lot of the talks at the user groups are for experienced users, following feedback from the USA user group, we thought we?d advertise that this year we?re planning to run a break-out for new users on day 1.

Break-out session for new user and prospects (Wed May 8th, 13:00 - 16:45)

This year we will offer a break-out session for new Spectrum Scale user and prospects to get started with Spectrum Scale. In this session we will cover Spectrum Scale Use Cases, the architecture of a Spectrum Scale environment, and discuss how the manifold Spectrum Scale features support the different use case. Please inform customers and colleagues who are interested to learn about Spectrum Scale to grab one of the last seats.

Registration link:
https://www.spectrumscaleug.org/event/uk-user-group-meeting/

There?s just a couple of places left for the usergroup, so please do share and register if you plan to attend.

Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190717/b7294b4c/attachment-0002.htm>

From S.J.Thompson at bham.ac.uk  Wed Jul 17 14:10:21 2019
From: S.J.Thompson at bham.ac.uk (Simon Thompson)
Date: Wed, 17 Jul 2019 13:10:21 +0000
Subject: [gpfsug-discuss] Break-out session for new user and prospects
 [London Usergroup]
In-Reply-To: <DB7PR10MB2201EAAA629BB9C10BBE24E0D3C90@DB7PR10MB2201.EURPRD10.PROD.OUTLOOK.COM>
References: <776770B2-5F84-4462-B900-58EBB982DC1C@spectrumscale.org>
	<DB7PR10MB2201EAAA629BB9C10BBE24E0D3C90@DB7PR10MB2201.EURPRD10.PROD.OUTLOOK.COM>
Message-ID: <6916EC29-2057-4396-A285-7A905DB215EE@bham.ac.uk>

Hi Paul,

There was a slide deck, I?ve asked the IBMers if it?s something we can share.

Simon

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Paul Ward <p.ward at nhm.ac.uk>
Reply-To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Date: Wednesday, 17 July 2019 at 14:01
To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Break-out session for new user and prospects [London Usergroup]

Hi Simon,

Was this a slide presentation? Is it available?

Kindest regards,
Paul

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (Spectrum Scale User Group Chair)
Sent: 30 April 2019 10:25
To: gpfsug-discuss at spectrumscale.org
Subject: [gpfsug-discuss] Break-out session for new user and prospects [London Usergroup]

Hi all,

We know that a lot of the talks at the user groups are for experienced users, following feedback from the USA user group, we thought we?d advertise that this year we?re planning to run a break-out for new users on day 1.

Break-out session for new user and prospects (Wed May 8th, 13:00 - 16:45)

This year we will offer a break-out session for new Spectrum Scale user and prospects to get started with Spectrum Scale. In this session we will cover Spectrum Scale Use Cases, the architecture of a Spectrum Scale environment, and discuss how the manifold Spectrum Scale features support the different use case. Please inform customers and colleagues who are interested to learn about Spectrum Scale to grab one of the last seats.

Registration link:
https://www.spectrumscaleug.org/event/uk-user-group-meeting/

There?s just a couple of places left for the usergroup, so please do share and register if you plan to attend.

Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190717/386a9e5b/attachment-0002.htm>

From leonardo.sala at psi.ch  Wed Jul 17 15:01:06 2019
From: leonardo.sala at psi.ch (Leonardo Sala)
Date: Wed, 17 Jul 2019 16:01:06 +0200
Subject: [gpfsug-discuss] Mistakenly AFM resync triggered: should I stop it?
Message-ID: <e69518bc-d525-fa95-d271-e3e8e81ebcf5@psi.ch>

Dear all,

I do have a possibly stupid question, based on something stupid I did. 
During a procedure to recover AFM transfers (which stopped for still to 
be investigated reasons), I triggered by mistake a resync operation on 
an healthy SW fileset (state: Inactive). The AFM cache in this case is 
big, so this has triggered a big backlog and quite some network 
activity. afmPrefetchThreshold is set to 0, so I am not too scared about 
partially zero-filled data files. But what would it be the best course 
of action now? Should I just let it run (ETA: ~20 hours, give or take), 
or try to stop it? Also, given that this is some experimental data, what 
would the safest course of action?

Thanks a lot!

cheers

leo


-- 
Paul Scherrer Institut
Dr. Leonardo Sala
Group Leader High Performance Computing
Deputy Section Head Science IT
Science IT
WHGA/106
5232 Villigen PSI
Switzerland

Phone: +41 56 310 3369
leonardo.sala at psi.ch
www.psi.ch

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190717/fbaaae4f/attachment-0002.htm>

From bbanister at jumptrading.com  Wed Jul 17 18:24:12 2019
From: bbanister at jumptrading.com (Bryan Banister)
Date: Wed, 17 Jul 2019 17:24:12 +0000
Subject: [gpfsug-discuss] GPFS quota inode/files maximum setting of
	2147483647
Message-ID: <DM6PR14MB2890B7F5B0283B4444C68BAEA2C90@DM6PR14MB2890.namprd14.prod.outlook.com>

Thought this might be of interest to others on the list.

IBM, are there any plans to increase the maximum files setting for quotas over 2147483647?

Thanks,
-Bryan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190717/e373b695/attachment-0002.htm>

From vpuvvada at in.ibm.com  Thu Jul 18 07:41:08 2019
From: vpuvvada at in.ibm.com (Venkateswara R Puvvada)
Date: Thu, 18 Jul 2019 12:11:08 +0530
Subject: [gpfsug-discuss] Mistakenly AFM resync triggered: should I stop
	it?
In-Reply-To: <e69518bc-d525-fa95-d271-e3e8e81ebcf5@psi.ch>
References: <e69518bc-d525-fa95-d271-e3e8e81ebcf5@psi.ch>
Message-ID: <OF11AFAFD0.1AB097A2-ON6525843B.0022AC38-6525843B.0024B9A2@notes.na.collabserv.com>

Hi,

You can stop the AFM resync if it is not required.

mmafmctl device dropPending -j fileset
mmafmctl device resetResync -j fileset

Next access to the fileset would run the recovery and impact should be 
minimal as it recovers only the data which was not replicated.

~Venkat (vpuvvada at in.ibm.com)



From:   Leonardo Sala <leonardo.sala at psi.ch>
To:     "gpfsug-discuss at spectrumscale.org" 
<gpfsug-discuss at spectrumscale.org>
Date:   07/17/2019 07:32 PM
Subject:        [EXTERNAL] [gpfsug-discuss] Mistakenly AFM resync 
triggered: should I stop it?
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Dear all,
I do have a possibly stupid question, based on something stupid I did. 
During a procedure to recover AFM transfers (which stopped for still to be 
investigated reasons), I triggered by mistake a resync operation on an 
healthy SW fileset (state: Inactive). The AFM cache in this case is big, 
so this has triggered a big backlog and quite some network activity. 
afmPrefetchThreshold is set to 0, so I am not too scared about partially 
zero-filled data files. But what would it be the best course of action 
now? Should I just let it run (ETA: ~20 hours, give or take), or try to 
stop it? Also, given that this is some experimental data, what would the 
safest course of action?
Thanks a lot!
cheers
leo

-- 
Paul Scherrer Institut
Dr. Leonardo Sala
Group Leader High Performance Computing
Deputy Section Head Science IT
Science IT
WHGA/106
5232 Villigen PSI
Switzerland

Phone: +41 56 310 3369
leonardo.sala at psi.ch
www.psi.ch_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=MapU9QQbtQx-G9hGAGBPzuB-Qxkl0HONn6ByIz4nXoI&s=F_iFVjqia2agf0fTNncWoaZqmKbhSCO-opqFb6cjG6A&e= 





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190718/748db732/attachment-0002.htm>

From heinrich.billich at id.ethz.ch  Thu Jul 18 15:15:12 2019
From: heinrich.billich at id.ethz.ch (Billich  Heinrich Rainer (ID SD))
Date: Thu, 18 Jul 2019 14:15:12 +0000
Subject: [gpfsug-discuss] How to prove that data is in inode
In-Reply-To: <OF776130C0.2EA026CD-ON0025843A.0044FED5-8525843A.00455C19@notes.na.collabserv.com>
References: <71647F14-8BEC-4E89-A30B-F0D880A81A84@id.ethz.ch>
	<OF776130C0.2EA026CD-ON0025843A.0044FED5-8525843A.00455C19@notes.na.collabserv.com>
Message-ID: <A5096632-88B9-4A6B-9C22-E8DC0884987F@id.ethz.ch>

Hello Kums,

Thank you; I could verify that the data of small files is in the inode. In the table below you see the filesize and the result of the tsdbfs query. Up to 3k all data is in the inode.

The result of stat calls on small files is very puzzling. When data is in the inode stat reports one block of 512 bytes used, even for 3k of data. I don?t see how this would affect any of our applications, so it?s just something to note.

I append some output and the script to generate it, just for completeness.

Cheers,

Heiner

# stat -c "inode %i" $f | tsdbfs fs1301 | grep indirectionLevel
0  indirectionLevel=DIRECT status=USERFILE. <empty file is special>
1  indirectionLevel=INODE status=USERFILE
16  indirectionLevel=INODE status=USERFILE
512  indirectionLevel=INODE status=USERFILE
1k  indirectionLevel=INODE status=USERFILE
2k  indirectionLevel=INODE status=USERFILE
3k  indirectionLevel=INODE status=USERFILE.  < up to 3k data is in inode >

4k  indirectionLevel=DIRECT status=USERFILE
16k  indirectionLevel=DIRECT status=USERFILE
64k  indirectionLevel=DIRECT status=USERFILE
1M  indirectionLevel=DIRECT status=USERFILE
2M  indirectionLevel=DIRECT status=USERFILE


Stat output

# stat -c ?%n size: %s  allocated: %b*%B?

# stat -c %n size: %s  allocated: %b*%B
0 size: 0  allocated: 0*512
1 size: 1  allocated: 1*512
16 size: 16  allocated: 1*512
512 size: 512  allocated: 1*512
1k size: 1024  allocated: 1*512
2k size: 2048  allocated: 1*512
3k size: 3072  allocated: 1*512  < 3k file and  all data in inode:  stat reports 1*512 allocated/used)
4k size: 4096  allocated: 64*512
16k size: 16384  allocated: 64*512.   < as expected, 32k subblock size >
64k size: 65536  allocated: 128*512
1M size: 1048576  allocated: 2048*512
2M size: 2097152  allocated: 4096*512

The script:

# test-data-in-inode.sh
sizes="0 1 16 512 1k 2k 3k 4k 16k 64k 1M 2M"

echo create files
for s in $sizes
do
  head -c $s /dev/zero > $s
done

echo sleep 20. # give gpfs some time to update metadata
sleep 20

echo
echo "# ls -ls $sizes"
ls -ls $sizes

echo
echo "# stat -c %n size: %s  allocated: %b*%B"
stat -c "%n size: %s  allocated: %b*%B" $sizes

echo
echo '# stat -c "inode %i" $f | tsdbfs fs1301 | grep indirectionLevel'
for f in $sizes
do
echo -n $f
stat -c "inode %i" $f | tsdbfs fs1301 | grep indirectionLevel
done

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Kumaran Rajaram <kums at us.ibm.com>
Reply to: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Wednesday, 17 July 2019 at 14:38
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] How to prove that data is in inode


Hi,

>> How can I prove that data of a small file is stored in the inode (and not on a data nsd)?

You may use echo "inode file_inode_number" | tsdbfs fs_device | grep indirectionLevel and if it points to INODE, then the file is stored in the inodes

# 4K Inode Size
# mmlsfs gpfs3a | grep 'Inode size'
-i 4096 Inode size in bytes

# Small file
# ls -l /mnt/gpfs3a/hello.txt
-rw-r--r-- 1 root root 6 Jul 17 08:32 /mnt/gpfs3a/hello.txt

# ls -i /mnt/gpfs3a/hello.txt
91649 /mnt/gpfs3a/hello.txt

#File is inlined within Inode
# echo "inode 91649" | tsdbfs gpfs3a | grep indirectionLevel
indirectionLevel=INODE status=USERFILE

Regards,
-Kums



[Inactive hide details for "Billich  Heinrich Rainer (ID SD)" ---07/17/2019 07:49:56 AM---Hello, How can I prove that data of a]"Billich Heinrich Rainer (ID SD)" ---07/17/2019 07:49:56 AM---Hello, How can I prove that data of a small file is stored in the inode (and not on a data nsd)?

From: "Billich Heinrich Rainer (ID SD)" <heinrich.billich at id.ethz.ch>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: 07/17/2019 07:49 AM
Subject: [EXTERNAL] [gpfsug-discuss] How to prove that data is in inode
Sent by: gpfsug-discuss-bounces at spectrumscale.org

________________________________



Hello,

How can I prove that data of a small file is stored in the inode (and not on a data nsd)?

We have a filesystem with 4k inodes on Scale 5.0.2 , but it seems there is no file data in the inodes?

I would expect that 'stat' reports 'Blocks: 0'  for a small file, but I see 'Blocks:1'.

Cheers,

Heiner

I tried

[]# rm -f test; echo hello > test
[]# ls -ls test
1 -rw-r--r-- 1 root root 6 Jul 17 13:11 test
[root at testnas13ems01 test]# stat test
 File: ?test?
 Size: 6         Blocks: 1          IO Block: 1048576 regular file
Device: 2dh/45d Inode: 353314      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-07-17 13:11:03.037049000 +0200
Modify: 2019-07-17 13:11:03.037331000 +0200
Change: 2019-07-17 13:11:03.037259319 +0200
Birth: -
[root at testnas13ems01 test]# du test
1 test
[root at testnas13ems01 test]# du -b test
6 test
[root at testnas13ems01 test]#

Filesystem

# mmlsfs f****
flag                value                    description
------------------- ------------------------ -----------------------------------
-f                 32768                    Minimum fragment (subblock) size in bytes
-i                 4096                     Inode size in bytes
-I                 32768                    Indirect block size in bytes
-m                 1                        Default number of metadata replicas
-M                 2                        Maximum number of metadata replicas
-r                 1                        Default number of data replicas
-R                 2                        Maximum number of data replicas
-j                 cluster                  Block allocation type
-D                 nfs4                     File locking semantics in effect
-k                 nfs4                     ACL semantics in effect
-n                 32                       Estimated number of nodes that will mount file system
-B                 1048576                  Block size
-Q                 user;group;fileset       Quotas accounting enabled
                   user;group;fileset       Quotas enforced
                   user;group;fileset       Default quotas enabled
--perfileset-quota Yes                      Per-fileset quota enforcement
--filesetdf        Yes                      Fileset df enabled?
-V                 20.01 (5.0.2.0)          Current file system version
                   15.01 (4.2.0.0)          Original file system version
--create-time      ***** 2017 File system creation time
-z                 No                       Is DMAPI enabled?
-L                 33554432                 Logfile size
-E                 Yes                      Exact mtime mount option
-S                 relatime                 Suppress atime mount option
-K                 whenpossible             Strict replica allocation option
--fastea           Yes                      Fast external attributes enabled?
--encryption       No                       Encryption enabled?
--inode-limit      1294592                  Maximum number of inodes in all inode spaces
--log-replicas     0                        Number of log replicas
--is4KAligned      Yes                      is4KAligned?
--rapid-repair     Yes                      rapidRepair enabled?
--write-cache-threshold 0                   HAWC Threshold (max 65536)
--subblocks-per-full-block 32               Number of subblocks per full block
-P                 system;data              Disk storage pools in file system
--file-audit-log   No                       File Audit Logging enabled?
--maintenance-mode No                       Maintenance Mode enabled?
-d                 ******
-A                 yes                      Automatic mount option
-o                 nfssync,nodev            Additional mount options
-T                 /****                  Default mount point
--mount-priority   0                        Mount priority

--
=======================
Heinrich Billich
ETH Z?rich
Informatikdienste
Tel.: +41 44 632 72 56
heinrich.billich at id.ethz.ch
========================



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190718/d044120a/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.gif
Type: image/gif
Size: 106 bytes
Desc: image001.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190718/d044120a/attachment-0002.gif>

From sxiao at us.ibm.com  Thu Jul 18 16:03:38 2019
From: sxiao at us.ibm.com (Steve Xiao)
Date: Thu, 18 Jul 2019 11:03:38 -0400
Subject: [gpfsug-discuss] How to prove that data is in inode
In-Reply-To: <mailman.267.1563459398.1289.gpfsug-discuss@spectrumscale.org>
References: <mailman.267.1563459398.1289.gpfsug-discuss@spectrumscale.org>
Message-ID: <OF10D4C475.21C37EF2-ON8525843B.00521027-8525843B.0052BAEF@notes.na.collabserv.com>

I believe GPFS report  1 block used in stat so the file will not be 
treated as empty by applications.    If i remember correctly, some backup 
programs was skipping small files with data in inode because GPFS was 
returning 0 block used for these small files. 

Steve Y. Xiao



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190718/6df9881d/attachment-0002.htm>

From novosirj at rutgers.edu  Thu Jul 18 22:15:52 2019
From: novosirj at rutgers.edu (Ryan Novosielski)
Date: Thu, 18 Jul 2019 21:15:52 +0000
Subject: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
In-Reply-To: <OFE9B5AE08.76EA4744-ON002582F2.003C63C0-C12582F2.00410D62@notes.na.collabserv.com>
References: <OF67F2CFA5.B278755D-ON002582F2.002CFDE9-C12582F2.0030AC8F@notes.na.collabserv.com>
	<CAAxuGpE4fM+X1Gwc-OySE+ZWc_jygx5TACusAP4rO0BWNgtaKA@mail.gmail.com>
	<OF237C2868.64DA65AF-ON002582F2.003784F8-002582F2.0038210B@notes.na.collabserv.com>
	<OFE9B5AE08.76EA4744-ON002582F2.003C63C0-C12582F2.00410D62@notes.na.collabserv.com>
Message-ID: <225AEBED-69EF-4E3C-9A67-5A9CB04B4309@rutgers.edu>

I happened across this message because I?ve already done this in the past and was trying to figure out how I did it (apparently didn?t write it down).

Most of it appeared to be adding to /etc/sysconfig/gpfsgui the following:

HTTP_PORT=8080
HTTPS_PORT=8443

?but that hasn?t completely done it yet. Going to have a look and see what else I might need to do.

--
____
|| \\UTGERS,  	 |---------------------------*O*---------------------------
||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
     `'

> On Aug 23, 2018, at 7:50 AM, Markus Rohwedder <rohwedder at de.ibm.com> wrote:
> 
> Hello Juri, Keith,
> 
> thank you for your responses.
> 
> The internal services communicate on the privileged ports, for backwards compatibility and firewall simplicity reasons. We can not just assume all nodes in the cluster are at the latest level.
> 
> Running two services at the same port on different IP addresses could be an option to consider for co-existance of the GUI and another service on the same node.
> However we have not set up, tested nor documented such a configuration as of today. 
> 
> Currently the GUI service manages the iptables redirect bring up and tear down.
> If this would be managed externally it would be possible to bind services to specific ports based on specific IPs.
> 
> In order to create custom redirect rules based on IP address it is necessary to instruct the GUI to 
> - not check for already used ports when the GUI service tries to start up
> - don't create/destroy port forwarding rules during GUI service start and stop.
> This GUI behavior can be configured using the internal flag UPDATE_IPTABLES in the service configuration with the 5.0.1.2 GUI code level.
> 
> The service configuration is not stored in the cluster configuration and may be overwritten during code upgrades, so these settings may have to be added again after an upgrade.
> 
> See this KC link:
> https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.1/com.ibm.spectrum.scale.v5r01.doc/bl1adv_firewallforgui.htm
> 
> Mit freundlichen Gr??en / Kind regards
> 
> Dr. Markus Rohwedder
> 
> Spectrum Scale GUI Development
> <ecblank.gif>
> Phone:	+49 7034 6430190	IBM Deutschland Research & Development	
> <17153317.gif>
> E-Mail:	rohwedder at de.ibm.com	Am Weiher 24
> <ecblank.gif>	<ecblank.gif>	65451 Kelsterbach
> <ecblank.gif>	<ecblank.gif>	Germany
> <ecblank.gif>
> 
> <graycol.gif>"Daniel Kidger" ---23.08.2018 12:13:36---Keith, I have another IBM customer who also wished to move Scale GUI's https ports. In their case
> 
> From:  "Daniel Kidger" <daniel.kidger at uk.ibm.com>
> To:  gpfsug-discuss at spectrumscale.org
> Cc:  gpfsug-discuss at spectrumscale.org
> Date:  23.08.2018 12:13
> Subject:  Re: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
> Sent by:  gpfsug-discuss-bounces at spectrumscale.org
> 
> 
> 
> 
> Keith,
> 
> I have another IBM customer who also wished to move Scale GUI's https ports.
> In their case because they had their own web based management interface on the same https port.
> Is this the same reason that you have?
> If so I wonder how many other sites have the same issue?
> 
> One workaround that was suggested at the time, was to add a second IP address to the node (piggy-backing on 'eth0').
> Then run the two different GUIs, one per IP address.
> Is this an option, albeit a little ugly?
> Daniel
> 
> <17310450.gif>				Dr Daniel Kidger
> IBM Technical Sales Specialist
> Software Defined Solution Sales
> 
> +44-(0)7818 522 266 
> daniel.kidger at uk.ibm.com
> 
> 
> 
> ----- Original message -----
> From: "Markus Rohwedder" <rohwedder at de.ibm.com>
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Cc:
> Subject: Re: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
> Date: Thu, Aug 23, 2018 9:51 AM
> Hello Keith,
> 
> it is not so easy.
> 
> The GUI receives events from other scale components using the currently defined ports.
> Changing the GUI ports will cause breakage in the GUI stack at several places (internal watchdog functions, interlock with health events, interlock with CES).
> Therefore at this point there is no procedure to change this behaviour across all components.
> 
> Because the GUI service does not run as root. the GUI server does not serve the privileged ports 80 and 443 directly but rather 47443 and 47080.
> Tweaking the ports in the server.xml file will only change the native ports that the GUI uses.
> The GUI manages IPTABLES rules to forward ports 443 and 80 to 47443 and 47080. 
> If these ports are already used by another service, the GUI will not start up.
> 
> Making the GUI ports freely configurable is therefore not a strightforward change, and currently no on our roadmap.
> If you want to emphasize your case as future development item, please let me know.
> 
> I would also be interested in:
> > Scale version you are running
> > Do you need port 80 or 443 as well?
> > Would it work for you if the xCAT service was bound to a single IP address?
> 
> Mit freundlichen Gr??en / Kind regards
> 
> Dr. Markus Rohwedder
> 
> Spectrum Scale GUI Development
> 
> <ecblank.gif>
> Phone:	+49 7034 6430190	IBM Deutschland Research & Development	
> <17153317.gif>
> E-Mail:	rohwedder at de.ibm.com	Am Weiher 24
> <ecblank.gif>	<ecblank.gif>	65451 Kelsterbach
> <ecblank.gif>	<ecblank.gif>	Germany
> <ecblank.gif>
> 
> <graycol.gif>Keith Ball ---22.08.2018 21:33:25---Hello All, Does anyone know how to change the HTTP ports for the Spectrum Scale GUI?
> 
> From: Keith Ball <bipcuds at gmail.com>
> To: gpfsug-discuss at spectrumscale.org
> Date: 22.08.2018 21:33
> Subject: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> 
> 
> 
> 
> Hello All,
> 
> Does anyone know how to change the HTTP ports for the Spectrum Scale GUI? Any documentation or RedPaper I have found deftly avoids discussing this. The most promising thing I see is in /opt/ibm/wlp/usr/servers/gpfsgui/server.xml:
> 
> <httpEndpoint id="defaultHttpEndpoint" host="*" httpPort="47080" httpsPort="47443">
> <tcpOptions soReuseAddr="true"/>
> </httpEndpoint>
> 
> but it appears that port 80 specifically is used also by the GUI's Web service. I already have an HTTP server using port 80 for provisioning (xCAT), so would rather change the Specturm Scale GUI configuration if I can.
> 
> Many Thanks,
> Keith
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number 741598. 
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss


From knop at us.ibm.com  Thu Jul 18 22:24:50 2019
From: knop at us.ibm.com (Felipe Knop)
Date: Thu, 18 Jul 2019 17:24:50 -0400
Subject: [gpfsug-discuss]
 =?utf-8?q?GPFS_quota_inode/files_maximum_setting?=
 =?utf-8?q?_of=092147483647?=
In-Reply-To: <DM6PR14MB2890B7F5B0283B4444C68BAEA2C90@DM6PR14MB2890.namprd14.prod.outlook.com>
References: <DM6PR14MB2890B7F5B0283B4444C68BAEA2C90@DM6PR14MB2890.namprd14.prod.outlook.com>
Message-ID: <OFD656CADB.20E2D1BC-ON0025843B.006EB6EC-8525843B.0075A180@notes.na.collabserv.com>


Bryan,

Could you open an RFE for this item?  We took an initial look at what it
will take to raise this limit and found that it will require an RPC format
change. So it's something that would need to go through a normal release
cycle.

Thanks and regards,

  Felipe

----
Felipe Knop                                     knop at us.ibm.com
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601
(845) 433-9314  T/L 293-9314





From:	Bryan Banister <bbanister at jumptrading.com>
To:	gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:	07/17/2019 01:24 PM
Subject:	[EXTERNAL] [gpfsug-discuss] GPFS quota inode/files maximum
            setting of	2147483647
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



Thought this might be of interest to others on the list.

IBM, are there any plans to increase the maximum files setting for quotas
over 2147483647?

Thanks,
-Bryan_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=oNT2koCZX0xmWlSlLblR9Q&m=gA7MMIPzQmpVqvoaVl9D2jeytjZOXtkGikPGPpNg_HU&s=pdkFa7ir9MdaAdAuMic15m4-cbL4FqlsESaTYOnxAq4&e=



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190718/374e3cc2/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190718/374e3cc2/attachment-0002.gif>

From novosirj at rutgers.edu  Thu Jul 18 22:26:50 2019
From: novosirj at rutgers.edu (Ryan Novosielski)
Date: Thu, 18 Jul 2019 21:26:50 +0000
Subject: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
In-Reply-To: <225AEBED-69EF-4E3C-9A67-5A9CB04B4309@rutgers.edu>
References: <OF67F2CFA5.B278755D-ON002582F2.002CFDE9-C12582F2.0030AC8F@notes.na.collabserv.com>
	<CAAxuGpE4fM+X1Gwc-OySE+ZWc_jygx5TACusAP4rO0BWNgtaKA@mail.gmail.com>
	<OF237C2868.64DA65AF-ON002582F2.003784F8-002582F2.0038210B@notes.na.collabserv.com>
	<OFE9B5AE08.76EA4744-ON002582F2.003C63C0-C12582F2.00410D62@notes.na.collabserv.com>
	<225AEBED-69EF-4E3C-9A67-5A9CB04B4309@rutgers.edu>
Message-ID: <B5538DB3-4625-43DA-B8F9-E600F48054B4@rutgers.edu>

Nope, that appears to be all of it. I also had a problem with the postgresql service, which was why the gpfsgui wouldn?t start. But once I fixed that, I can log in on https://<GUI_HOST>:8443.

HTH.

> On Jul 18, 2019, at 5:15 PM, Ryan Novosielski <novosirj at rutgers.edu> wrote:
> 
> I happened across this message because I?ve already done this in the past and was trying to figure out how I did it (apparently didn?t write it down).
> 
> Most of it appeared to be adding to /etc/sysconfig/gpfsgui the following:
> 
> HTTP_PORT=8080
> HTTPS_PORT=8443
> 
> ?but that hasn?t completely done it yet. Going to have a look and see what else I might need to do.
> 
> --
> ____
> || \\UTGERS,  	 |---------------------------*O*---------------------------
> ||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
> ||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
>     `'
> 
>> On Aug 23, 2018, at 7:50 AM, Markus Rohwedder <rohwedder at de.ibm.com> wrote:
>> 
>> Hello Juri, Keith,
>> 
>> thank you for your responses.
>> 
>> The internal services communicate on the privileged ports, for backwards compatibility and firewall simplicity reasons. We can not just assume all nodes in the cluster are at the latest level.
>> 
>> Running two services at the same port on different IP addresses could be an option to consider for co-existance of the GUI and another service on the same node.
>> However we have not set up, tested nor documented such a configuration as of today. 
>> 
>> Currently the GUI service manages the iptables redirect bring up and tear down.
>> If this would be managed externally it would be possible to bind services to specific ports based on specific IPs.
>> 
>> In order to create custom redirect rules based on IP address it is necessary to instruct the GUI to 
>> - not check for already used ports when the GUI service tries to start up
>> - don't create/destroy port forwarding rules during GUI service start and stop.
>> This GUI behavior can be configured using the internal flag UPDATE_IPTABLES in the service configuration with the 5.0.1.2 GUI code level.
>> 
>> The service configuration is not stored in the cluster configuration and may be overwritten during code upgrades, so these settings may have to be added again after an upgrade.
>> 
>> See this KC link:
>> https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.1/com.ibm.spectrum.scale.v5r01.doc/bl1adv_firewallforgui.htm
>> 
>> Mit freundlichen Gr??en / Kind regards
>> 
>> Dr. Markus Rohwedder
>> 
>> Spectrum Scale GUI Development
>> <ecblank.gif>
>> Phone:	+49 7034 6430190	IBM Deutschland Research & Development	
>> <17153317.gif>
>> E-Mail:	rohwedder at de.ibm.com	Am Weiher 24
>> <ecblank.gif>	<ecblank.gif>	65451 Kelsterbach
>> <ecblank.gif>	<ecblank.gif>	Germany
>> <ecblank.gif>
>> 
>> <graycol.gif>"Daniel Kidger" ---23.08.2018 12:13:36---Keith, I have another IBM customer who also wished to move Scale GUI's https ports. In their case
>> 
>> From:  "Daniel Kidger" <daniel.kidger at uk.ibm.com>
>> To:  gpfsug-discuss at spectrumscale.org
>> Cc:  gpfsug-discuss at spectrumscale.org
>> Date:  23.08.2018 12:13
>> Subject:  Re: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
>> Sent by:  gpfsug-discuss-bounces at spectrumscale.org
>> 
>> 
>> 
>> 
>> Keith,
>> 
>> I have another IBM customer who also wished to move Scale GUI's https ports.
>> In their case because they had their own web based management interface on the same https port.
>> Is this the same reason that you have?
>> If so I wonder how many other sites have the same issue?
>> 
>> One workaround that was suggested at the time, was to add a second IP address to the node (piggy-backing on 'eth0').
>> Then run the two different GUIs, one per IP address.
>> Is this an option, albeit a little ugly?
>> Daniel
>> 
>> <17310450.gif>				Dr Daniel Kidger
>> IBM Technical Sales Specialist
>> Software Defined Solution Sales
>> 
>> +44-(0)7818 522 266 
>> daniel.kidger at uk.ibm.com
>> 
>> 
>> 
>> ----- Original message -----
>> From: "Markus Rohwedder" <rohwedder at de.ibm.com>
>> Sent by: gpfsug-discuss-bounces at spectrumscale.org
>> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
>> Cc:
>> Subject: Re: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
>> Date: Thu, Aug 23, 2018 9:51 AM
>> Hello Keith,
>> 
>> it is not so easy.
>> 
>> The GUI receives events from other scale components using the currently defined ports.
>> Changing the GUI ports will cause breakage in the GUI stack at several places (internal watchdog functions, interlock with health events, interlock with CES).
>> Therefore at this point there is no procedure to change this behaviour across all components.
>> 
>> Because the GUI service does not run as root. the GUI server does not serve the privileged ports 80 and 443 directly but rather 47443 and 47080.
>> Tweaking the ports in the server.xml file will only change the native ports that the GUI uses.
>> The GUI manages IPTABLES rules to forward ports 443 and 80 to 47443 and 47080. 
>> If these ports are already used by another service, the GUI will not start up.
>> 
>> Making the GUI ports freely configurable is therefore not a strightforward change, and currently no on our roadmap.
>> If you want to emphasize your case as future development item, please let me know.
>> 
>> I would also be interested in:
>>> Scale version you are running
>>> Do you need port 80 or 443 as well?
>>> Would it work for you if the xCAT service was bound to a single IP address?
>> 
>> Mit freundlichen Gr??en / Kind regards
>> 
>> Dr. Markus Rohwedder
>> 
>> Spectrum Scale GUI Development
>> 
>> <ecblank.gif>
>> Phone:	+49 7034 6430190	IBM Deutschland Research & Development	
>> <17153317.gif>
>> E-Mail:	rohwedder at de.ibm.com	Am Weiher 24
>> <ecblank.gif>	<ecblank.gif>	65451 Kelsterbach
>> <ecblank.gif>	<ecblank.gif>	Germany
>> <ecblank.gif>
>> 
>> <graycol.gif>Keith Ball ---22.08.2018 21:33:25---Hello All, Does anyone know how to change the HTTP ports for the Spectrum Scale GUI?
>> 
>> From: Keith Ball <bipcuds at gmail.com>
>> To: gpfsug-discuss at spectrumscale.org
>> Date: 22.08.2018 21:33
>> Subject: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
>> Sent by: gpfsug-discuss-bounces at spectrumscale.org
>> 
>> 
>> 
>> 
>> Hello All,
>> 
>> Does anyone know how to change the HTTP ports for the Spectrum Scale GUI? Any documentation or RedPaper I have found deftly avoids discussing this. The most promising thing I see is in /opt/ibm/wlp/usr/servers/gpfsgui/server.xml:
>> 
>> <httpEndpoint id="defaultHttpEndpoint" host="*" httpPort="47080" httpsPort="47443">
>> <tcpOptions soReuseAddr="true"/>
>> </httpEndpoint>
>> 
>> but it appears that port 80 specifically is used also by the GUI's Web service. I already have an HTTP server using port 80 for provisioning (xCAT), so would rather change the Specturm Scale GUI configuration if I can.
>> 
>> Many Thanks,
>> Keith
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>> 
>> 
>> 
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>> 
>> Unless stated otherwise above:
>> IBM United Kingdom Limited - Registered in England and Wales with number 741598. 
>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>> 
>> 
>> 
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss


From novosirj at rutgers.edu  Fri Jul 19 17:43:03 2019
From: novosirj at rutgers.edu (Ryan Novosielski)
Date: Fri, 19 Jul 2019 16:43:03 +0000
Subject: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
In-Reply-To: <B5538DB3-4625-43DA-B8F9-E600F48054B4@rutgers.edu>
References: <OF67F2CFA5.B278755D-ON002582F2.002CFDE9-C12582F2.0030AC8F@notes.na.collabserv.com>
	<CAAxuGpE4fM+X1Gwc-OySE+ZWc_jygx5TACusAP4rO0BWNgtaKA@mail.gmail.com>
	<OF237C2868.64DA65AF-ON002582F2.003784F8-002582F2.0038210B@notes.na.collabserv.com>
	<OFE9B5AE08.76EA4744-ON002582F2.003C63C0-C12582F2.00410D62@notes.na.collabserv.com>
	<225AEBED-69EF-4E3C-9A67-5A9CB04B4309@rutgers.edu>
	<B5538DB3-4625-43DA-B8F9-E600F48054B4@rutgers.edu>
Message-ID: <C7EE62BE-118A-4680-B64E-4A2050F1BFBA@rutgers.edu>

Support has since pointed out to me, coincidentally, that all of the GUI callback scripts in /usr/lpp/mmfs/gui/callbacks contain a hard-coded port. I changed them this way, with PWD=/usr/lpp/mmfs/gui/callbacks:

find . -name *.sh -exec sed -i.bak 's/PORT=443/PORT=8443/g' {} \;

?as you can see, the only change was, for example:

[root at quorum02 callbacks]# diff gnr/PhysicalDiskCallback.sh.bak gnr/PhysicalDiskCallback.sh 
21c21
< PORT=443
---
> PORT=8443

That caused me to look to see if port 443 is hard-coded anyplace else, and I found this other one, /usr/lpp/mmfs/gui/bin-sudo/functions_iptables.sh:

23: . /etc/sysconfig/gpfsgui
24: 
25: HTTP_PORT=80
26: HTTPS_PORT=443

?this is peculiar to me because a) it works ? it would seem like these two override my /etc/sysconfig/gpfsgui settings, but the web server is reachable at 8443. Also, these lines would seem to make way more sense in the reverse (eg. let the sysconfig file redefine the ports if they contain values).

Ideally, IBM would let you change those two environment variables in the sysconfig file, or somewhere else, and the callback scripts would use that value from the environment. I?ve not tried setting PORT=$HTTPS_PORT to see if those callback scripts have access to that variable. 

--
____
|| \\UTGERS,  	 |---------------------------*O*---------------------------
||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
     `'

> On Jul 18, 2019, at 5:26 PM, Ryan Novosielski <novosirj at rutgers.edu> wrote:
> 
> Nope, that appears to be all of it. I also had a problem with the postgresql service, which was why the gpfsgui wouldn?t start. But once I fixed that, I can log in on https://<GUI_HOST>:8443.
> 
> HTH.
> 
>> On Jul 18, 2019, at 5:15 PM, Ryan Novosielski <novosirj at rutgers.edu> wrote:
>> 
>> I happened across this message because I?ve already done this in the past and was trying to figure out how I did it (apparently didn?t write it down).
>> 
>> Most of it appeared to be adding to /etc/sysconfig/gpfsgui the following:
>> 
>> HTTP_PORT=8080
>> HTTPS_PORT=8443
>> 
>> ?but that hasn?t completely done it yet. Going to have a look and see what else I might need to do.
>> 
>> --
>> ____
>> || \\UTGERS,  	 |---------------------------*O*---------------------------
>> ||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
>> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
>> ||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
>>    `'
>> 
>>> On Aug 23, 2018, at 7:50 AM, Markus Rohwedder <rohwedder at de.ibm.com> wrote:
>>> 
>>> Hello Juri, Keith,
>>> 
>>> thank you for your responses.
>>> 
>>> The internal services communicate on the privileged ports, for backwards compatibility and firewall simplicity reasons. We can not just assume all nodes in the cluster are at the latest level.
>>> 
>>> Running two services at the same port on different IP addresses could be an option to consider for co-existance of the GUI and another service on the same node.
>>> However we have not set up, tested nor documented such a configuration as of today. 
>>> 
>>> Currently the GUI service manages the iptables redirect bring up and tear down.
>>> If this would be managed externally it would be possible to bind services to specific ports based on specific IPs.
>>> 
>>> In order to create custom redirect rules based on IP address it is necessary to instruct the GUI to 
>>> - not check for already used ports when the GUI service tries to start up
>>> - don't create/destroy port forwarding rules during GUI service start and stop.
>>> This GUI behavior can be configured using the internal flag UPDATE_IPTABLES in the service configuration with the 5.0.1.2 GUI code level.
>>> 
>>> The service configuration is not stored in the cluster configuration and may be overwritten during code upgrades, so these settings may have to be added again after an upgrade.
>>> 
>>> See this KC link:
>>> https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.1/com.ibm.spectrum.scale.v5r01.doc/bl1adv_firewallforgui.htm
>>> 
>>> Mit freundlichen Gr??en / Kind regards
>>> 
>>> Dr. Markus Rohwedder
>>> 
>>> Spectrum Scale GUI Development
>>> <ecblank.gif>
>>> Phone:	+49 7034 6430190	IBM Deutschland Research & Development	
>>> <17153317.gif>
>>> E-Mail:	rohwedder at de.ibm.com	Am Weiher 24
>>> <ecblank.gif>	<ecblank.gif>	65451 Kelsterbach
>>> <ecblank.gif>	<ecblank.gif>	Germany
>>> <ecblank.gif>
>>> 
>>> <graycol.gif>"Daniel Kidger" ---23.08.2018 12:13:36---Keith, I have another IBM customer who also wished to move Scale GUI's https ports. In their case
>>> 
>>> From:  "Daniel Kidger" <daniel.kidger at uk.ibm.com>
>>> To:  gpfsug-discuss at spectrumscale.org
>>> Cc:  gpfsug-discuss at spectrumscale.org
>>> Date:  23.08.2018 12:13
>>> Subject:  Re: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
>>> Sent by:  gpfsug-discuss-bounces at spectrumscale.org
>>> 
>>> 
>>> 
>>> 
>>> Keith,
>>> 
>>> I have another IBM customer who also wished to move Scale GUI's https ports.
>>> In their case because they had their own web based management interface on the same https port.
>>> Is this the same reason that you have?
>>> If so I wonder how many other sites have the same issue?
>>> 
>>> One workaround that was suggested at the time, was to add a second IP address to the node (piggy-backing on 'eth0').
>>> Then run the two different GUIs, one per IP address.
>>> Is this an option, albeit a little ugly?
>>> Daniel
>>> 
>>> <17310450.gif>				Dr Daniel Kidger
>>> IBM Technical Sales Specialist
>>> Software Defined Solution Sales
>>> 
>>> +44-(0)7818 522 266 
>>> daniel.kidger at uk.ibm.com
>>> 
>>> 
>>> 
>>> ----- Original message -----
>>> From: "Markus Rohwedder" <rohwedder at de.ibm.com>
>>> Sent by: gpfsug-discuss-bounces at spectrumscale.org
>>> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
>>> Cc:
>>> Subject: Re: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
>>> Date: Thu, Aug 23, 2018 9:51 AM
>>> Hello Keith,
>>> 
>>> it is not so easy.
>>> 
>>> The GUI receives events from other scale components using the currently defined ports.
>>> Changing the GUI ports will cause breakage in the GUI stack at several places (internal watchdog functions, interlock with health events, interlock with CES).
>>> Therefore at this point there is no procedure to change this behaviour across all components.
>>> 
>>> Because the GUI service does not run as root. the GUI server does not serve the privileged ports 80 and 443 directly but rather 47443 and 47080.
>>> Tweaking the ports in the server.xml file will only change the native ports that the GUI uses.
>>> The GUI manages IPTABLES rules to forward ports 443 and 80 to 47443 and 47080. 
>>> If these ports are already used by another service, the GUI will not start up.
>>> 
>>> Making the GUI ports freely configurable is therefore not a strightforward change, and currently no on our roadmap.
>>> If you want to emphasize your case as future development item, please let me know.
>>> 
>>> I would also be interested in:
>>>> Scale version you are running
>>>> Do you need port 80 or 443 as well?
>>>> Would it work for you if the xCAT service was bound to a single IP address?
>>> 
>>> Mit freundlichen Gr??en / Kind regards
>>> 
>>> Dr. Markus Rohwedder
>>> 
>>> Spectrum Scale GUI Development
>>> 
>>> <ecblank.gif>
>>> Phone:	+49 7034 6430190	IBM Deutschland Research & Development	
>>> <17153317.gif>
>>> E-Mail:	rohwedder at de.ibm.com	Am Weiher 24
>>> <ecblank.gif>	<ecblank.gif>	65451 Kelsterbach
>>> <ecblank.gif>	<ecblank.gif>	Germany
>>> <ecblank.gif>
>>> 
>>> <graycol.gif>Keith Ball ---22.08.2018 21:33:25---Hello All, Does anyone know how to change the HTTP ports for the Spectrum Scale GUI?
>>> 
>>> From: Keith Ball <bipcuds at gmail.com>
>>> To: gpfsug-discuss at spectrumscale.org
>>> Date: 22.08.2018 21:33
>>> Subject: [gpfsug-discuss] Changing Web ports for the Spectrum Scale GUI
>>> Sent by: gpfsug-discuss-bounces at spectrumscale.org
>>> 
>>> 
>>> 
>>> 
>>> Hello All,
>>> 
>>> Does anyone know how to change the HTTP ports for the Spectrum Scale GUI? Any documentation or RedPaper I have found deftly avoids discussing this. The most promising thing I see is in /opt/ibm/wlp/usr/servers/gpfsgui/server.xml:
>>> 
>>> <httpEndpoint id="defaultHttpEndpoint" host="*" httpPort="47080" httpsPort="47443">
>>> <tcpOptions soReuseAddr="true"/>
>>> </httpEndpoint>
>>> 
>>> but it appears that port 80 specifically is used also by the GUI's Web service. I already have an HTTP server using port 80 for provisioning (xCAT), so would rather change the Specturm Scale GUI configuration if I can.
>>> 
>>> Many Thanks,
>>> Keith
>>> _______________________________________________
>>> gpfsug-discuss mailing list
>>> gpfsug-discuss at spectrumscale.org
>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> gpfsug-discuss mailing list
>>> gpfsug-discuss at spectrumscale.org
>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>> 
>>> Unless stated otherwise above:
>>> IBM United Kingdom Limited - Registered in England and Wales with number 741598. 
>>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>>> _______________________________________________
>>> gpfsug-discuss mailing list
>>> gpfsug-discuss at spectrumscale.org
>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> gpfsug-discuss mailing list
>>> gpfsug-discuss at spectrumscale.org
>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>> 
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss


From lore at cscs.ch  Fri Jul 19 17:59:05 2019
From: lore at cscs.ch (Lo Re  Giuseppe)
Date: Fri, 19 Jul 2019 16:59:05 +0000
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 90, Issue 11
In-Reply-To: <mailman.1.1563534001.39239.gpfsug-discuss@spectrumscale.org>
References: <mailman.1.1563534001.39239.gpfsug-discuss@spectrumscale.org>
Message-ID: <1F7E4727-C978-45D5-A348-22DC7B62AF88@cscs.ch>

Hi,

We have also hit this limitation at CSCS.
We have a CES Object Service cluster where one node cannot run the Openstack Swift proxy-server service on port 443 as it is already taken by the GUI.
I have opened a RFE, didn?t help much though __ , so far.

https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=107778

Giuseppe

?On 19.07.19, 04:01, "gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org" <gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org> wrote:

    /etc/sysconfig/gpfsgui


From novosirj at rutgers.edu  Wed Jul 24 17:19:47 2019
From: novosirj at rutgers.edu (Ryan Novosielski)
Date: Wed, 24 Jul 2019 16:19:47 +0000
Subject: [gpfsug-discuss] Spectrum Scale with RHEL7.6 kernel
 3.10.0-957.21.2
In-Reply-To: <OF248C1D60.E94029E1-ON00258412.00769D60-85258412.0077861A@notes.na.collabserv.com>
References: <OF248C1D60.E94029E1-ON00258412.00769D60-85258412.0077861A@notes.na.collabserv.com>
Message-ID: <3DFDAC1A-ECD5-4752-8342-55389804FBDD@rutgers.edu>

Hi Felipe and others:

We didn?t hear about this apart from via this list and from a colleague. What is the way to get on an official notifications list for this sort of thing? I don?t see anything anywhere in the IBM support pages.

Thanks!

--
____
|| \\UTGERS,  	 |---------------------------*O*---------------------------
||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
     `'

> On Jun 7, 2019, at 5:45 PM, Felipe Knop <knop at us.ibm.com> wrote:
> 
> All,
> 
> There have been reported issues (including kernel crashes) on Spectrum Scale with the latest RHEL7.6 kernel 3.10.0-957.21.2. Please consider delaying upgrades to this kernel until further information is provided. 
> 
> Thanks,
> 
> Felipe
> 
> ----
> Felipe Knop knop at us.ibm.com
> GPFS Development and Security
> IBM Systems
> IBM Building 008
> 2455 South Rd, Poughkeepsie, NY 12601
> (845) 433-9314 T/L 293-9314
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss


From S.J.Thompson at bham.ac.uk  Wed Jul 24 18:30:31 2019
From: S.J.Thompson at bham.ac.uk (Simon Thompson)
Date: Wed, 24 Jul 2019 17:30:31 +0000
Subject: [gpfsug-discuss] Spectrum Scale with RHEL7.6 kernel
 3.10.0-957.21.2
In-Reply-To: <3DFDAC1A-ECD5-4752-8342-55389804FBDD@rutgers.edu>
References: <OF248C1D60.E94029E1-ON00258412.00769D60-85258412.0077861A@notes.na.collabserv.com>,
	<3DFDAC1A-ECD5-4752-8342-55389804FBDD@rutgers.edu>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545012993054A@EX13.adf.bham.ac.uk>

There is a notifications service. I think you have to subscribe to them via fix central or passport advantage (I don't remember which).

Though this notification was on the list a while before the official Comms.

Simon
________________________________________
From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Ryan Novosielski [novosirj at rutgers.edu]
Sent: 24 July 2019 17:19
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Spectrum Scale with RHEL7.6 kernel 3.10.0-957.21.2

Hi Felipe and others:

We didn?t hear about this apart from via this list and from a colleague. What is the way to get on an official notifications list for this sort of thing? I don?t see anything anywhere in the IBM support pages.

Thanks!

--
____
|| \\UTGERS,     |---------------------------*O*---------------------------
||_// the State  |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ  | Office of Advanced Research Computing - MSB C630, Newark
     `'

> On Jun 7, 2019, at 5:45 PM, Felipe Knop <knop at us.ibm.com> wrote:
>
> All,
>
> There have been reported issues (including kernel crashes) on Spectrum Scale with the latest RHEL7.6 kernel 3.10.0-957.21.2. Please consider delaying upgrades to this kernel until further information is provided.
>
> Thanks,
>
> Felipe
>
> ----
> Felipe Knop knop at us.ibm.com
> GPFS Development and Security
> IBM Systems
> IBM Building 008
> 2455 South Rd, Poughkeepsie, NY 12601
> (845) 433-9314 T/L 293-9314
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


From mark.bergman at uphs.upenn.edu  Fri Jul 26 00:31:21 2019
From: mark.bergman at uphs.upenn.edu (mark.bergman at uphs.upenn.edu)
Date: Thu, 25 Jul 2019 19:31:21 -0400
Subject: [gpfsug-discuss] Question concerning integration of CES with AD
	authentication system
In-Reply-To: Your message of "Thu, 24 May 2018 17:07:02 -0000."
	<OF0CCFED2D.FE89B800-ON00258297.005DFD38-00258297.005E075F@notes.na.collabserv.com>
References: <OF0CCFED2D.FE89B800-ON00258297.005DFD38-00258297.005E075F@notes.na.collabserv.com>
	<1527173192.28106.18.camel@strath.ac.uk>,
	<83A6EEB0EC738F459A39439733AE804522F1B13B@MBX214.d.ethz.ch><20180524141632.xuah3dxu4bxx372z@utumno.gs.washington.edu>
Message-ID: <10843-1564097481.984955@sYvE.K-ht.6Q-W>

In the message dated: Thu, 24 May 2018 17:07:02 -0000,
The pithy ruminations from Christof Schmitt on 
[Re: [gpfsug-discuss] Question concerning integration of CES with AD authentication system] were:
=> 

Following up on an old, old post...

=> > Basically Samba ignores the separate GID field in RFC2307bis, so one
=> > imagines the options for changing the LDAP attributes are none
=> > existent.
=>  
=> mmuserauth now has an option to use either the gid from the actual primary
=> group or the gid defined for the user. See:
=>  
=> https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.0/
=> com.ibm.spectrum.scale.v5r00.doc/bl1adm_mmuserauth.htm
=>  
=> --unixmap-domains unixDomainMap
=> [...]
=>     win: Specifies the system to read the primary group set as Windows
=> primary group of a user on the Active Directory.
=>      unix: Specifies the system to read the primary group as set in "UNIX
=> attributes" of a user on the Active Directory.  
=>     For example,
=>      --unixmap-domains "MYDOMAIN1(20000-50000:unix);MYDOMAIN2
=> (100000-200000:win)"

I see this is refering to UNIX attributes within AD, but I'm curious about mapping to attributes in LDAP.

=> This gets mapped to 'idmap config ... : unix_primary_group' in the
=> internal config.

Does that correspond to setting the smb.conf parameter
	
	unix_primary_group = yes




Specifically, under Spectrum Scale 5.0.2, if I run:

	mmuserauth service create --data-access-method file --ldapmap-domains "DOMAIN(type=stand-alone:ldap_srv=ldapserver:range=1001-65535:usr_dn=ou=People,dc=DC,dc=TLD:grp_dn=ou=Group,dc=DC,dc=TLD)" --type ad

(some args removed in this example), will that map the user's primary group to

	the primaryGroupID supplied by AD
  or
	the primaryGroupID LDAP field
  or
	the gidNumber LDAP field

or something else?

Thanks,

Mark


=> 
=> Christof Schmitt || IBM || Spectrum Scale Development || Tucson, AZ
=> christof.schmitt at us.ibm.com  ||  +1-520-799-2469    (T/L: 321-2469)
=>  


From christof.schmitt at us.ibm.com  Fri Jul 26 17:38:17 2019
From: christof.schmitt at us.ibm.com (Christof Schmitt)
Date: Fri, 26 Jul 2019 16:38:17 +0000
Subject: [gpfsug-discuss] Question concerning integration of CES with AD
 authentication system
In-Reply-To: <10843-1564097481.984955@sYvE.K-ht.6Q-W>
References: <10843-1564097481.984955@sYvE.K-ht.6Q-W>,
	<OF0CCFED2D.FE89B800-ON00258297.005DFD38-00258297.005E075F@notes.na.collabserv.com>
	<1527173192.28106.18.camel@strath.ac.uk>,
	<83A6EEB0EC738F459A39439733AE804522F1B13B@MBX214.d.ethz.ch><20180524141632.xuah3dxu4bxx372z@utumno.gs.washington.edu>
Message-ID: <OF5FF29174.4A08CF99-ON00258443.005B64E3-00258443.005B658E@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190726/68876eda/attachment-0002.htm>

From david_johnson at brown.edu  Mon Jul 29 17:04:38 2019
From: david_johnson at brown.edu (David Johnson)
Date: Mon, 29 Jul 2019 12:04:38 -0400
Subject: [gpfsug-discuss] Building GPFS filesystem system data pool on
 shared nothing NVMe drives
Message-ID: <B5D9698F-23E7-432E-A35E-747569E7422F@brown.edu>

We are planning a 5.0.x upgrade onto new hardware to make use of the new 5.x GPFS features.
The goal is to use up to four NSD nodes for metadata, each one with 6 NVMe drives (to be determined
whether we use Intel VROC for raid 5 or raid 1, or just straight disks).  

So questions ? 
Has anyone done system pool on shared nothing cluster?  How did you set it up?
With default metadata replication set at 3, can you make use of four NSD nodes effectively?
How would one design the location vectors and failure groups so that the system metadata is
spread evenly across the four servers?

Thanks,
 ? ddj
Dave Johnson

From xhejtman at ics.muni.cz  Mon Jul 29 17:11:38 2019
From: xhejtman at ics.muni.cz (Lukas Hejtmanek)
Date: Mon, 29 Jul 2019 18:11:38 +0200
Subject: [gpfsug-discuss] Asymetric SAN with GPFS
Message-ID: <20190729161138.znuss5dt2rhig6cv@ics.muni.cz>

Hello,

is there any settings for GPFS 5.x so that you could mitigate slow down of
asymmetric SAN? The asymmetric SAN means, that not every LUN has the same
speed, or not every disk array has the same number of LUNs. It seems that
overal speed is degraded to the slowest LUN. Is there any workaround for this
except avoiding that LUN at all?

-- 
Luk?? Hejtm?nek

Linux Administrator only because
  Full Time Multitasking Ninja 
  is not an official job title


From S.J.Thompson at bham.ac.uk  Mon Jul 29 17:19:45 2019
From: S.J.Thompson at bham.ac.uk (Simon Thompson)
Date: Mon, 29 Jul 2019 16:19:45 +0000
Subject: [gpfsug-discuss] Asymetric SAN with GPFS
In-Reply-To: <20190729161138.znuss5dt2rhig6cv@ics.muni.cz>
References: <20190729161138.znuss5dt2rhig6cv@ics.muni.cz>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545012994D59A@EX13.adf.bham.ac.uk>

Surely you would need to use different data pools for this. Given in a single pool data will be distributed over all available nsd devices in that pool, you are highly likely to only ever go as fast as the slowest LUN.

Simon
________________________________________
From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of xhejtman at ics.muni.cz [xhejtman at ics.muni.cz]
Sent: 29 July 2019 17:11
To: gpfsug-discuss at spectrumscale.org
Subject: [gpfsug-discuss] Asymetric SAN with GPFS

Hello,

is there any settings for GPFS 5.x so that you could mitigate slow down of
asymmetric SAN? The asymmetric SAN means, that not every LUN has the same
speed, or not every disk array has the same number of LUNs. It seems that
overal speed is degraded to the slowest LUN. Is there any workaround for this
except avoiding that LUN at all?

--
Luk?? Hejtm?nek

Linux Administrator only because
  Full Time Multitasking Ninja
  is not an official job title
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


From luis.bolinches at fi.ibm.com  Mon Jul 29 18:34:58 2019
From: luis.bolinches at fi.ibm.com (Luis Bolinches)
Date: Mon, 29 Jul 2019 17:34:58 +0000
Subject: [gpfsug-discuss] Building GPFS filesystem system data pool on
 shared nothing NVMe drives
In-Reply-To: <B5D9698F-23E7-432E-A35E-747569E7422F@brown.edu>
Message-ID: <OFC1ED3470.4BE6E16D-ON00258446.006095F7-1564421698664@notes.na.collabserv.com>

Hi, from phone so sorry for typos. 

I really think you should look into Spectrum Scale Erasure Code Edition (ECE) for this. 

Sure you could do a RAID on each node as you mention here but that sounds like a lot of waste to me on storage capacity. Not to forget you get other goodies like end to end checksum and rapid rebuilds with ECE, among others. 

Four servers is the minimum requirement for ECE (4+3p) and from top of my head 12 disk per RG, you are fine with both requirements. 

There is a presentation on ECE on the user group web page from London May 2019 were we talk about ECE. 

And the ibm page of the product https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_intro.htm
--
Cheers

> El 29 jul 2019, a las 19:06, David Johnson <david_johnson at brown.edu> escribi?:
> 
> We are planning a 5.0.x upgrade onto new hardware to make use of the new 5.x GPFS features.
> The goal is to use up to four NSD nodes for metadata, each one with 6 NVMe drives (to be determined
> whether we use Intel VROC for raid 5 or raid 1, or just straight disks).  
> 
> So questions ? 
> Has anyone done system pool on shared nothing cluster?  How did you set it up?
> With default metadata replication set at 3, can you make use of four NSD nodes effectively?
> How would one design the location vectors and failure groups so that the system metadata is
> spread evenly across the four servers?
> 
> Thanks,
> ? ddj
> Dave Johnson
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=1mZ896psa5caYzBeaugTlc7TtRejJp3uvKYxas3S7Xc&m=Pu3G5GzwRsM4iwztIiWzzngNSgsPzrRe8cd3pPAFVM0&s=q3w_lvsuB88gMM_J5psgKuWDex_wSW1v5h16WNMCtzU&e= 
> 

Ellei edell? ole toisin mainittu: / Unless stated otherwise above:
Oy IBM Finland Ab
PL 265, 00101 Helsinki, Finland
Business ID, Y-tunnus: 0195876-3 
Registered in Finland

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190729/0d0c5e04/attachment-0002.htm>

From david_johnson at brown.edu  Tue Jul 30 12:46:14 2019
From: david_johnson at brown.edu (David Johnson)
Date: Tue, 30 Jul 2019 07:46:14 -0400
Subject: [gpfsug-discuss] Building GPFS filesystem system data pool on
 shared nothing NVMe drives
In-Reply-To: <OFC1ED3470.4BE6E16D-ON00258446.006095F7-1564421698664@notes.na.collabserv.com>
References: <OFC1ED3470.4BE6E16D-ON00258446.006095F7-1564421698664@notes.na.collabserv.com>
Message-ID: <D53515DA-94D8-400B-80E0-8033633A5F21@brown.edu>

Can we confirm the requirement for disks per RG?  I have 4 RG, but only 6 x 3TB NVMe drives per box.

> On Jul 29, 2019, at 1:34 PM, Luis Bolinches <luis.bolinches at fi.ibm.com> wrote:
> 
> Hi, from phone so sorry for typos. 
> 
> I really think you should look into Spectrum Scale Erasure Code Edition (ECE) for this. 
> 
> Sure you could do a RAID on each node as you mention here but that sounds like a lot of waste to me on storage capacity. Not to forget you get other goodies like end to end checksum and rapid rebuilds with ECE, among others. 
> 
> Four servers is the minimum requirement for ECE (4+3p) and from top of my head 12 disk per RG, you are fine with both requirements. 
> 
> There is a presentation on ECE on the user group web page from London May 2019 were we talk about ECE. 
> 
> And the ibm page of the product https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_intro.htm <https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_intro.htm>
> --
> Cheers
> 
> El 29 jul 2019, a las 19:06, David Johnson <david_johnson at brown.edu <mailto:david_johnson at brown.edu>> escribi?:
> 
>> We are planning a 5.0.x upgrade onto new hardware to make use of the new 5.x GPFS features.
>> The goal is to use up to four NSD nodes for metadata, each one with 6 NVMe drives (to be determined
>> whether we use Intel VROC for raid 5 or raid 1, or just straight disks).  
>> 
>> So questions ? 
>> Has anyone done system pool on shared nothing cluster?  How did you set it up?
>> With default metadata replication set at 3, can you make use of four NSD nodes effectively?
>> How would one design the location vectors and failure groups so that the system metadata is
>> spread evenly across the four servers?
>> 
>> Thanks,
>> ? ddj
>> Dave Johnson
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org <http://spectrumscale.org/>
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss <http://gpfsug.org/mailman/listinfo/gpfsug-discuss> 
>> 
> 
> Ellei edell? ole toisin mainittu: / Unless stated otherwise above:
> Oy IBM Finland Ab
> PL 265, 00101 Helsinki, Finland
> Business ID, Y-tunnus: 0195876-3 
> Registered in Finland
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190730/b61f3695/attachment-0002.htm>

From p.ward at nhm.ac.uk  Tue Jul 30 13:16:31 2019
From: p.ward at nhm.ac.uk (Paul Ward)
Date: Tue, 30 Jul 2019 12:16:31 +0000
Subject: [gpfsug-discuss] Break-out session for new user and prospects
 [London Usergroup]
In-Reply-To: <6916EC29-2057-4396-A285-7A905DB215EE@bham.ac.uk>
References: <776770B2-5F84-4462-B900-58EBB982DC1C@spectrumscale.org>
	<DB7PR10MB2201EAAA629BB9C10BBE24E0D3C90@DB7PR10MB2201.EURPRD10.PROD.OUTLOOK.COM>
	<6916EC29-2057-4396-A285-7A905DB215EE@bham.ac.uk>
Message-ID: <DB7PR10MB22017537C2C2102FD904B11CD3DC0@DB7PR10MB2201.EURPRD10.PROD.OUTLOOK.COM>

Hi Simon,

Any update on this please.


Paul Ward
Technical Solutions Infrastructure Architect
Natural History Museum
T: 02079426450
E: p.ward at nhm.ac.uk<mailto:p.ward at nhm.ac.uk>

From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of Simon Thompson
Sent: 17 July 2019 14:10
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Break-out session for new user and prospects [London Usergroup]

Hi Paul,

There was a slide deck, I?ve asked the IBMers if it?s something we can share.

Simon

From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of Paul Ward <p.ward at nhm.ac.uk<mailto:p.ward at nhm.ac.uk>>
Reply-To: "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, 17 July 2019 at 14:01
To: "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Break-out session for new user and prospects [London Usergroup]

Hi Simon,

Was this a slide presentation? Is it available?

Kindest regards,
Paul

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (Spectrum Scale User Group Chair)
Sent: 30 April 2019 10:25
To: gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] Break-out session for new user and prospects [London Usergroup]

Hi all,

We know that a lot of the talks at the user groups are for experienced users, following feedback from the USA user group, we thought we?d advertise that this year we?re planning to run a break-out for new users on day 1.

Break-out session for new user and prospects (Wed May 8th, 13:00 - 16:45)

This year we will offer a break-out session for new Spectrum Scale user and prospects to get started with Spectrum Scale. In this session we will cover Spectrum Scale Use Cases, the architecture of a Spectrum Scale environment, and discuss how the manifold Spectrum Scale features support the different use case. Please inform customers and colleagues who are interested to learn about Spectrum Scale to grab one of the last seats.

Registration link:
https://www.spectrumscaleug.org/event/uk-user-group-meeting/<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.spectrumscaleug.org%2Fevent%2Fuk-user-group-meeting%2F&data=02%7C01%7Cp.ward%40nhm.ac.uk%7C7d6995c8217f453c8dd408d70ab82dad%7C73a29c014e78437fa0d4c8553e1960c1%7C1%7C0%7C636989658468906223&sdata=%2BmSDjCczU4GYNWbtOfeWhxovF0YtYzkMX%2BJNRmcIpNo%3D&reserved=0>

There?s just a couple of places left for the usergroup, so please do share and register if you plan to attend.

Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190730/eeac3325/attachment-0002.htm>

From Paul.Sanchez at deshaw.com  Tue Jul 30 13:19:56 2019
From: Paul.Sanchez at deshaw.com (Sanchez, Paul)
Date: Tue, 30 Jul 2019 12:19:56 +0000
Subject: [gpfsug-discuss] Building GPFS filesystem system data pool on
 shared nothing NVMe drives
In-Reply-To: <D53515DA-94D8-400B-80E0-8033633A5F21@brown.edu>
References: <OFC1ED3470.4BE6E16D-ON00258446.006095F7-1564421698664@notes.na.collabserv.com>
	<D53515DA-94D8-400B-80E0-8033633A5F21@brown.edu>
Message-ID: <5664c4aaea5e4fbfac1f8a72acb080dc@deshaw.com>

Hi David,

In an ECE configuration, it would be typical to put all of the NVMe disks in all 4 of your servers into a single recovery group.   So in your case, all 24 NVMe drives would be in one recovery group and the 4 servers would be ?log group? servers in the recovery group, distributing the I/O load for the NSD/vdisks that are hosted on the RG.  (The minimum disks for a single RG config is 12, and you meet that easily.)

https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_plan_recommendations.htm
outlines the recommendations for raidCode protection.  Your configuration (4 nodes) would use vdisks with 4+3P, which gives you a slightly better capacity yield than RAID10 would, but with much better recovery characteristics:


?         No single failed node will result in a down system NSD.

?         No single drive failure will require a critical priority rebuild, and can be handled in the background without killing performance.

So from that perspective, ECE is a win here and avoids a problem with the non-ECE, shared-nothing designs: the manual ?mmchdisk <fsname> start -a? operation that is needed after any traditional shared-nothing metadata NSD goes offline to bring it back and protect against further failures.

Despite the operational challenges of the non-ECE design, it can sometimes survive two server failures (if replication factor is 3 and the filesystem descriptor quorum wasn?t lost by the two failures) which a 4 node ECE cluster cannot.  Given that the world is complex and unexpected things can happen, I?d personally recommend redistributing the 24 disks across 6 servers if you can, so that the design could always survive 2 node failures.  I?ve run this design and it?s fairly robust.

In any event, you should of course test the failure scenarios yourself before going into production to validate them and familiarize yourself with the process.  And a special note on ECE: due to the cooperative nature at the pdisk level, the network between the servers in the RG should be as reliable as possible and any network redundancy should also be tested ahead of time.

-Paul

From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of David Johnson
Sent: Tuesday, July 30, 2019 7:46 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Building GPFS filesystem system data pool on shared nothing NVMe drives


This message was sent by an external party.

Can we confirm the requirement for disks per RG?  I have 4 RG, but only 6 x 3TB NVMe drives per box.


On Jul 29, 2019, at 1:34 PM, Luis Bolinches <luis.bolinches at fi.ibm.com<mailto:luis.bolinches at fi.ibm.com>> wrote:

Hi, from phone so sorry for typos.

I really think you should look into Spectrum Scale Erasure Code Edition (ECE) for this.

Sure you could do a RAID on each node as you mention here but that sounds like a lot of waste to me on storage capacity. Not to forget you get other goodies like end to end checksum and rapid rebuilds with ECE, among others.

Four servers is the minimum requirement for ECE (4+3p) and from top of my head 12 disk per RG, you are fine with both requirements.

There is a presentation on ECE on the user group web page from London May 2019 were we talk about ECE.

And the ibm page of the product https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_intro.htm
--
Cheers

El 29 jul 2019, a las 19:06, David Johnson <david_johnson at brown.edu<mailto:david_johnson at brown.edu>> escribi?:
We are planning a 5.0.x upgrade onto new hardware to make use of the new 5.x GPFS features.
The goal is to use up to four NSD nodes for metadata, each one with 6 NVMe drives (to be determined
whether we use Intel VROC for raid 5 or raid 1, or just straight disks).

So questions ?
Has anyone done system pool on shared nothing cluster?  How did you set it up?
With default metadata replication set at 3, can you make use of four NSD nodes effectively?
How would one design the location vectors and failure groups so that the system metadata is
spread evenly across the four servers?

Thanks,
? ddj
Dave Johnson
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Ellei edell? ole toisin mainittu: / Unless stated otherwise above:
Oy IBM Finland Ab
PL 265, 00101 Helsinki, Finland
Business ID, Y-tunnus: 0195876-3
Registered in Finland
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190730/539e5159/attachment-0002.htm>

From david_johnson at brown.edu  Tue Jul 30 14:03:54 2019
From: david_johnson at brown.edu (David Johnson)
Date: Tue, 30 Jul 2019 09:03:54 -0400
Subject: [gpfsug-discuss] Building GPFS filesystem system data pool on
 shared nothing NVMe drives
In-Reply-To: <5664c4aaea5e4fbfac1f8a72acb080dc@deshaw.com>
References: <OFC1ED3470.4BE6E16D-ON00258446.006095F7-1564421698664@notes.na.collabserv.com>
	<D53515DA-94D8-400B-80E0-8033633A5F21@brown.edu>
	<5664c4aaea5e4fbfac1f8a72acb080dc@deshaw.com>
Message-ID: <261C855E-FC23-43C6-8A76-FEAE5B48221C@brown.edu>

OK, so the ECE recovery group is the four NSD servers with the System storage pool disks, and somehow I have to read the docs
and find out how to define pdisks that spread the replication across the four servers, but three disks at a time.  
Three pdisks of 7 drives, three I can't do anything with, or are those for rebuilding space?

Can you provide me details of your six-node non-ECE configuration?  Basically how the NSDs are defined...

The remainder of our new filesystem will have a fast pool of 12 nodes of excelero, and 2Pb of spinning disks, so another possibility
would be to license four more nodes and put the system pool under excelero.  

 -- ddj

> On Jul 30, 2019, at 8:19 AM, Sanchez, Paul <Paul.Sanchez at deshaw.com> wrote:
> 
> Hi David,
>  
> In an ECE configuration, it would be typical to put all of the NVMe disks in all 4 of your servers into a single recovery group.   So in your case, all 24 NVMe drives would be in one recovery group and the 4 servers would be ?log group? servers in the recovery group, distributing the I/O load for the NSD/vdisks that are hosted on the RG.  (The minimum disks for a single RG config is 12, and you meet that easily.)
>  
> https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_plan_recommendations.htm <https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_plan_recommendations.htm>
> outlines the recommendations for raidCode protection.  Your configuration (4 nodes) would use vdisks with 4+3P, which gives you a slightly better capacity yield than RAID10 would, but with much better recovery characteristics:
>  
> ?         No single failed node will result in a down system NSD.
> ?         No single drive failure will require a critical priority rebuild, and can be handled in the background without killing performance.
>  
> So from that perspective, ECE is a win here and avoids a problem with the non-ECE, shared-nothing designs: the manual ?mmchdisk <fsname> start -a? operation that is needed after any traditional shared-nothing metadata NSD goes offline to bring it back and protect against further failures.
>  
> Despite the operational challenges of the non-ECE design, it can sometimes survive two server failures (if replication factor is 3 and the filesystem descriptor quorum wasn?t lost by the two failures) which a 4 node ECE cluster cannot.  Given that the world is complex and unexpected things can happen, I?d personally recommend redistributing the 24 disks across 6 servers if you can, so that the design could always survive 2 node failures.  I?ve run this design and it?s fairly robust.
>  
> In any event, you should of course test the failure scenarios yourself before going into production to validate them and familiarize yourself with the process.  And a special note on ECE: due to the cooperative nature at the pdisk level, the network between the servers in the RG should be as reliable as possible and any network redundancy should also be tested ahead of time.
>  
> -Paul
>  
> From: gpfsug-discuss-bounces at spectrumscale.org <mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org <mailto:gpfsug-discuss-bounces at spectrumscale.org>> On Behalf Of David Johnson
> Sent: Tuesday, July 30, 2019 7:46 AM
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org <mailto:gpfsug-discuss at spectrumscale.org>>
> Subject: Re: [gpfsug-discuss] Building GPFS filesystem system data pool on shared nothing NVMe drives
>  
> This message was sent by an external party.
> 
>  
> Can we confirm the requirement for disks per RG?  I have 4 RG, but only 6 x 3TB NVMe drives per box.
> 
> 
> On Jul 29, 2019, at 1:34 PM, Luis Bolinches <luis.bolinches at fi.ibm.com <mailto:luis.bolinches at fi.ibm.com>> wrote:
>  
> Hi, from phone so sorry for typos.  
>  
> I really think you should look into Spectrum Scale Erasure Code Edition (ECE) for this. 
>  
> Sure you could do a RAID on each node as you mention here but that sounds like a lot of waste to me on storage capacity. Not to forget you get other goodies like end to end checksum and rapid rebuilds with ECE, among others. 
>  
> Four servers is the minimum requirement for ECE (4+3p) and from top of my head 12 disk per RG, you are fine with both requirements. 
> 
> There is a presentation on ECE on the user group web page from London May 2019 were we talk about ECE. 
>  
> And the ibm page of the product https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_intro.htm <https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_intro.htm>
> -- 
> Cheers
> 
> El 29 jul 2019, a las 19:06, David Johnson <david_johnson at brown.edu <mailto:david_johnson at brown.edu>> escribi?:
> 
> We are planning a 5.0.x upgrade onto new hardware to make use of the new 5.x GPFS features.
> The goal is to use up to four NSD nodes for metadata, each one with 6 NVMe drives (to be determined
> whether we use Intel VROC for raid 5 or raid 1, or just straight disks).  
> 
> So questions ? 
> Has anyone done system pool on shared nothing cluster?  How did you set it up?
> With default metadata replication set at 3, can you make use of four NSD nodes effectively?
> How would one design the location vectors and failure groups so that the system metadata is
> spread evenly across the four servers?
> 
> Thanks,
> ? ddj
> Dave Johnson
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org <http://spectrumscale.org/>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
> 
> Ellei edell? ole toisin mainittu: / Unless stated otherwise above:
> Oy IBM Finland Ab
> PL 265, 00101 Helsinki, Finland
> Business ID, Y-tunnus: 0195876-3 
> Registered in Finland
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org <http://spectrumscale.org/>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
>  
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org <http://spectrumscale.org/>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190730/e8dc0611/attachment-0002.htm>

From S.J.Thompson at bham.ac.uk  Tue Jul 30 14:49:04 2019
From: S.J.Thompson at bham.ac.uk (Simon Thompson)
Date: Tue, 30 Jul 2019 13:49:04 +0000
Subject: [gpfsug-discuss] Break-out session for new user and prospects
 [London Usergroup]
In-Reply-To: <DB7PR10MB22017537C2C2102FD904B11CD3DC0@DB7PR10MB2201.EURPRD10.PROD.OUTLOOK.COM>
References: <776770B2-5F84-4462-B900-58EBB982DC1C@spectrumscale.org>
	<DB7PR10MB2201EAAA629BB9C10BBE24E0D3C90@DB7PR10MB2201.EURPRD10.PROD.OUTLOOK.COM>
	<6916EC29-2057-4396-A285-7A905DB215EE@bham.ac.uk>
	<DB7PR10MB22017537C2C2102FD904B11CD3DC0@DB7PR10MB2201.EURPRD10.PROD.OUTLOOK.COM>
Message-ID: <BB09590F-57DE-4BCF-BD14-5DFED3BC2AD8@bham.ac.uk>

Hi Sorry, yes. I got the slides and forgot to upload them!

https://www.spectrumscaleug.org/wp-content/uploads/2019/07/IBM-Spectrum-Scale-Use-Cases.pdf

Simon

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Paul Ward <p.ward at nhm.ac.uk>
Reply-To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Date: Tuesday, 30 July 2019 at 13:16
To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Break-out session for new user and prospects [London Usergroup]

Hi Simon,

Any update on this please.


Paul Ward
Technical Solutions Infrastructure Architect
Natural History Museum
T: 02079426450
E: p.ward at nhm.ac.uk<mailto:p.ward at nhm.ac.uk>

From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of Simon Thompson
Sent: 17 July 2019 14:10
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Break-out session for new user and prospects [London Usergroup]

Hi Paul,

There was a slide deck, I?ve asked the IBMers if it?s something we can share.

Simon

From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of Paul Ward <p.ward at nhm.ac.uk<mailto:p.ward at nhm.ac.uk>>
Reply-To: "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, 17 July 2019 at 14:01
To: "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Break-out session for new user and prospects [London Usergroup]

Hi Simon,

Was this a slide presentation? Is it available?

Kindest regards,
Paul

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (Spectrum Scale User Group Chair)
Sent: 30 April 2019 10:25
To: gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] Break-out session for new user and prospects [London Usergroup]

Hi all,

We know that a lot of the talks at the user groups are for experienced users, following feedback from the USA user group, we thought we?d advertise that this year we?re planning to run a break-out for new users on day 1.

Break-out session for new user and prospects (Wed May 8th, 13:00 - 16:45)

This year we will offer a break-out session for new Spectrum Scale user and prospects to get started with Spectrum Scale. In this session we will cover Spectrum Scale Use Cases, the architecture of a Spectrum Scale environment, and discuss how the manifold Spectrum Scale features support the different use case. Please inform customers and colleagues who are interested to learn about Spectrum Scale to grab one of the last seats.

Registration link:
https://www.spectrumscaleug.org/event/uk-user-group-meeting/<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.spectrumscaleug.org%2Fevent%2Fuk-user-group-meeting%2F&data=02%7C01%7Cp.ward%40nhm.ac.uk%7C7d6995c8217f453c8dd408d70ab82dad%7C73a29c014e78437fa0d4c8553e1960c1%7C1%7C0%7C636989658468906223&sdata=%2BmSDjCczU4GYNWbtOfeWhxovF0YtYzkMX%2BJNRmcIpNo%3D&reserved=0>

There?s just a couple of places left for the usergroup, so please do share and register if you plan to attend.

Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190730/731e27a0/attachment-0002.htm>

From Paul.Sanchez at deshaw.com  Tue Jul 30 15:36:35 2019
From: Paul.Sanchez at deshaw.com (Sanchez, Paul)
Date: Tue, 30 Jul 2019 14:36:35 +0000
Subject: [gpfsug-discuss] Building GPFS filesystem system data pool on
 shared nothing NVMe drives
In-Reply-To: <261C855E-FC23-43C6-8A76-FEAE5B48221C@brown.edu>
References: <OFC1ED3470.4BE6E16D-ON00258446.006095F7-1564421698664@notes.na.collabserv.com>
	<D53515DA-94D8-400B-80E0-8033633A5F21@brown.edu>
	<5664c4aaea5e4fbfac1f8a72acb080dc@deshaw.com>
	<261C855E-FC23-43C6-8A76-FEAE5B48221C@brown.edu>
Message-ID: <b227c5eadc5c46cd9e63b19e51416502@deshaw.com>

Yes, read the documentation for the mmvdisk command to get a better idea of how this actually works.  There?s definitely a small paradigmatic shift in thinking that you?ll encounter between traditional NSDs and ECE.  The mmvdisk command?s defaults handle just about everything for you and the defaults are sane.

The short answer is that in a 6-node ECE recoverygroup, each of the nodes will normally provide access to 2 NSDs and the system pool would therefore have 12 NSDs total.  If a node fails, each of its two NSDs will continue to be served each by a different one of the remaining servers.  If you?ve used ESS before, then you can think about the RAID/spare/rebuild space similarly to that: all drives have some spare space on them and erasure-code stripes are evenly distributed among all of the disks so that they are all used and fill at approximately the same rate.

From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of David Johnson
Sent: Tuesday, July 30, 2019 9:04 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Building GPFS filesystem system data pool on shared nothing NVMe drives


This message was sent by an external party.

OK, so the ECE recovery group is the four NSD servers with the System storage pool disks, and somehow I have to read the docs
and find out how to define pdisks that spread the replication across the four servers, but three disks at a time.
Three pdisks of 7 drives, three I can't do anything with, or are those for rebuilding space?

Can you provide me details of your six-node non-ECE configuration?  Basically how the NSDs are defined...

The remainder of our new filesystem will have a fast pool of 12 nodes of excelero, and 2Pb of spinning disks, so another possibility
would be to license four more nodes and put the system pool under excelero.

 -- ddj


On Jul 30, 2019, at 8:19 AM, Sanchez, Paul <Paul.Sanchez at deshaw.com<mailto:Paul.Sanchez at deshaw.com>> wrote:

Hi David,

In an ECE configuration, it would be typical to put all of the NVMe disks in all 4 of your servers into a single recovery group.   So in your case, all 24 NVMe drives would be in one recovery group and the 4 servers would be ?log group? servers in the recovery group, distributing the I/O load for the NSD/vdisks that are hosted on the RG.  (The minimum disks for a single RG config is 12, and you meet that easily.)

https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_plan_recommendations.htm
outlines the recommendations for raidCode protection.  Your configuration (4 nodes) would use vdisks with 4+3P, which gives you a slightly better capacity yield than RAID10 would, but with much better recovery characteristics:

?         No single failed node will result in a down system NSD.
?         No single drive failure will require a critical priority rebuild, and can be handled in the background without killing performance.

So from that perspective, ECE is a win here and avoids a problem with the non-ECE, shared-nothing designs: the manual ?mmchdisk <fsname> start -a? operation that is needed after any traditional shared-nothing metadata NSD goes offline to bring it back and protect against further failures.

Despite the operational challenges of the non-ECE design, it can sometimes survive two server failures (if replication factor is 3 and the filesystem descriptor quorum wasn?t lost by the two failures) which a 4 node ECE cluster cannot.  Given that the world is complex and unexpected things can happen, I?d personally recommend redistributing the 24 disks across 6 servers if you can, so that the design could always survive 2 node failures.  I?ve run this design and it?s fairly robust.

In any event, you should of course test the failure scenarios yourself before going into production to validate them and familiarize yourself with the process.  And a special note on ECE: due to the cooperative nature at the pdisk level, the network between the servers in the RG should be as reliable as possible and any network redundancy should also be tested ahead of time.

-Paul

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> On Behalf Of David Johnson
Sent: Tuesday, July 30, 2019 7:46 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Building GPFS filesystem system data pool on shared nothing NVMe drives

This message was sent by an external party.

Can we confirm the requirement for disks per RG?  I have 4 RG, but only 6 x 3TB NVMe drives per box.



On Jul 29, 2019, at 1:34 PM, Luis Bolinches <luis.bolinches at fi.ibm.com<mailto:luis.bolinches at fi.ibm.com>> wrote:

Hi, from phone so sorry for typos.

I really think you should look into Spectrum Scale Erasure Code Edition (ECE) for this.

Sure you could do a RAID on each node as you mention here but that sounds like a lot of waste to me on storage capacity. Not to forget you get other goodies like end to end checksum and rapid rebuilds with ECE, among others.

Four servers is the minimum requirement for ECE (4+3p) and from top of my head 12 disk per RG, you are fine with both requirements.

There is a presentation on ECE on the user group web page from London May 2019 were we talk about ECE.

And the ibm page of the product https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_intro.htm
--
Cheers

El 29 jul 2019, a las 19:06, David Johnson <david_johnson at brown.edu<mailto:david_johnson at brown.edu>> escribi?:
We are planning a 5.0.x upgrade onto new hardware to make use of the new 5.x GPFS features.
The goal is to use up to four NSD nodes for metadata, each one with 6 NVMe drives (to be determined
whether we use Intel VROC for raid 5 or raid 1, or just straight disks).

So questions ?
Has anyone done system pool on shared nothing cluster?  How did you set it up?
With default metadata replication set at 3, can you make use of four NSD nodes effectively?
How would one design the location vectors and failure groups so that the system metadata is
spread evenly across the four servers?

Thanks,
? ddj
Dave Johnson
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Ellei edell? ole toisin mainittu: / Unless stated otherwise above:
Oy IBM Finland Ab
PL 265, 00101 Helsinki, Finland
Business ID, Y-tunnus: 0195876-3
Registered in Finland
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190730/d782a1c1/attachment-0002.htm>

From luis.bolinches at fi.ibm.com  Tue Jul 30 15:54:42 2019
From: luis.bolinches at fi.ibm.com (Luis Bolinches)
Date: Tue, 30 Jul 2019 14:54:42 +0000
Subject: [gpfsug-discuss] Building GPFS filesystem system data pool on
 shared nothing NVMe drives
In-Reply-To: <b227c5eadc5c46cd9e63b19e51416502@deshaw.com>
References: <b227c5eadc5c46cd9e63b19e51416502@deshaw.com>,
	<OFC1ED3470.4BE6E16D-ON00258446.006095F7-1564421698664@notes.na.collabserv.com><D53515DA-94D8-400B-80E0-8033633A5F21@brown.edu><5664c4aaea5e4fbfac1f8a72acb080dc@deshaw.com><261C855E-FC23-43C6-8A76-FEAE5B48221C@brown.edu>
Message-ID: <OF31F22F72.D63B7538-ON00258447.00510DA2-00258447.0051E9DD@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190730/7157fa12/attachment-0002.htm>