From sfadden at us.ibm.com Mon Oct 3 13:39:00 2016 From: sfadden at us.ibm.com (Scott Fadden) Date: Mon, 3 Oct 2016 05:39:00 -0700 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? In-Reply-To: <0D55AB1D-DB9D-45CF-AB27-157CDA1172D9@nuance.com> References: <0D55AB1D-DB9D-45CF-AB27-157CDA1172D9@nuance.com> Message-ID: About 3.5 KiB if the inode is 4k and you are not using encryption (uses EA space) or large custom extended attributes, Scott Fadden Spectrum Scale - Technical Marketing Phone: (503) 880-5833 sfadden at us.ibm.com http://www.ibm.com/systems/storage/spectrum/scale From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 09/28/2016 11:14 AM Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? Sent by: gpfsug-discuss-bounces at spectrumscale.org What the largest file that will fit inside a 1K, 2K, or 4K inode? Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Mon Oct 3 13:54:47 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Mon, 3 Oct 2016 12:54:47 +0000 Subject: [gpfsug-discuss] License question Message-ID: I have a question about licensing. If I have a system which is NFS mounting (from a CES node), that is then serving data via HTTP, does this system need to have a Server license? Reading the FAQ (in the context of VMs): https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html "The IBM Spectrum Scale Client may not be used for virtual servers to share IBM Spectrum Scale data directly through any application, service protocol or method, such as NFS, CIFS, FTP, HTTP, or OpenStack Swift" My reading of this is that if the VM is mounting via NFS, then as it is not a Spectrum Scale client (rather an NFS client), then it doesn't require a Scale license. Secondly. If the VM is mounting from the hypervisor via NFS (Manilla), is the VM then allowed to serve data out by e.g. Web server, or does the hypervisor require a server license to do this. Assuming my hypervisor is licensed for Spectrum Scale of course. I think my interpretation is that if a VM is using NFS mounting from either a CES node or using Manilla to the hypervisor, then this is all OK to "re-export" data as the VM isn't talking GPFS protocol. Is this correct? Simon From daniel.kidger at uk.ibm.com Mon Oct 3 14:23:35 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Mon, 3 Oct 2016 13:23:35 +0000 Subject: [gpfsug-discuss] License question In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Mon Oct 3 15:53:51 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 3 Oct 2016 10:53:51 -0400 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? 3876! In-Reply-To: References: <0D55AB1D-DB9D-45CF-AB27-157CDA1172D9@nuance.com> Message-ID: Pretty easy to determine experimentally, with a few trials.... And the answer is 3876 for an simple file with just a selinux EA which is apparently created by default on my test system. So apparently there are actually only 220 bytes of metadata in this case (4096-3876=220). As they say YMMV ;-) [root at bog-wifi cmvc]# mmlsfs yy -i flag value description ------------------- ------------------------ ----------------------------------- -i 4096 Inode size in bytes [root at bog-wifi isize]# stat i387[67] File: ?i3876? Size: 3876 Blocks: 0 IO Block: 262144 regular file File: ?i3877? Size: 3877 Blocks: 16 IO Block: 262144 regular file [root at bog-wifi isize]# ls -ild i3876 19406 -rw-r--r--. 1 root root 3876 Oct 3 10:39 i3876 tsdbfs yy inode 19406 Inode 19406 [19406] snap 0 (index 14 in block 303): Inode address: 1:511600 size 4096 nAddrs 323 indirectionLevel=INODE status=USERFILE objectVersion=2 generation=0x7C5599C4 nlink=1 owner uid=0 gid=0 mode=0200100644: -rw-r--r-- ... Data [3876]: 0000000000000000: 136D7B1F 1D29CA0C B59BC917 D9AE0220 *.m{..)..........* 0000000000000010: F18512CF 998AC02E DC6926FA 3FDC3EAF *.........i&.?.>.* ... 0000000000000F20: EFF02209 00000000 05077365 6C696E75 *..".......selinu* trailer: 0x20005993FD8 diskAddrs 323 xattrLen 48 (free 3856/3904) res 0,0,0 overflow pri 0 vers 0 nsub 0 repl 1/2 DA Null [0] id 0x5 prefLen 8 keyLen 7 key security.selinux len 37 val '756E636F6E66696E65645F753A6F626A6563745F723A756E6C6162656C65645F743 A733000' -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 21994 bytes Desc: not available URL: From makaplan at us.ibm.com Mon Oct 3 16:46:09 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 3 Oct 2016 11:46:09 -0400 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? 3968!! In-Reply-To: References: <0D55AB1D-DB9D-45CF-AB27-157CDA1172D9@nuance.com> Message-ID: On a non-SELINUX system the answer is 3968 of data in a 4K inode, just 128 bytes of metadata. Caution: it's possible in some future release, this could change ... I don't know of any plans, I'm just saying ... Inode 16346892 [16346892] snap 0 (index 12 in block 255420): Inode address: 6:123049056 size 4096 nAddrs 330 indirectionLevel=INODE status=USERFILE objectVersion=1 generation=0xC0156CB nlink=1 owner uid=0 gid=0 mode=0200100644: -rw-r--r-- blocksize code=5 (32 subblocks) lastBlockSubblocks=0 checksum=0xAD8E0B4B is Valid fileSize=3968 nFullBlocks=0 currentMetadataReplicas=1 maxMetadataReplicas=2 currentDataReplicas=1 maxDataReplicas=2 ... Data [3968]: 0000000000000000: BCA91252 2B64BEDC A7D7BA9D D5BE8C30 *...R+d.........0* ... 0000000000000F70: DA925E2F 16A68C01 03CA5E37 08D72B7F *..^/......^7..+.* trailer: is NULL -------------- next part -------------- An HTML attachment was scrubbed... URL: From A.K.Ghumra at bham.ac.uk Mon Oct 3 16:52:52 2016 From: A.K.Ghumra at bham.ac.uk (Aslam Ghumra (IT Services, Facilities Management)) Date: Mon, 3 Oct 2016 15:52:52 +0000 Subject: [gpfsug-discuss] WebDAV protocol access support Message-ID: Does anyone know whether WebDAV protocol access support will ( if ever ) be implemented within GPFS ? Regards, Aslam Ghumra Research Data Management ____________________________ IT Services Elms Road Data Centre Building G5 Edgbaston Birmingham B15 2TT T: 0121 414 5877 F; 0121 414 3952 Skype : JanitorX Twitter : @aslamghumra in : https://uk.linkedin.com/in/aslam-ghumra-13907993 http://intranet.bham.ac.uk/bear -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Mon Oct 3 16:56:52 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 3 Oct 2016 15:56:52 +0000 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? Message-ID: <1BDA1763-A6FE-4A52-B2ED-22324C9AA16F@nuance.com> What's going be taken away if you use Encryption or Transparent Cloud Tiering? Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: on behalf of Marc A Kaplan Reply-To: gpfsug main discussion list Date: Monday, October 3, 2016 at 10:46 AM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Biggest file that will fit inside an inode? 3968!! On a non-SELINUX system the answer is 3968 of data in a 4K inode, just 128 bytes of metadata. Caution: it's possible in some future release, this could change ... I don't know of any plans, I'm just saying ... Inode 16346892 [16346892] snap 0 (index 12 in block 255420): Inode address: 6:123049056 size 4096 nAddrs 330 indirectionLevel=INODE status=USERFILE objectVersion=1 generation=0xC0156CB nlink=1 owner uid=0 gid=0 mode=0200100644: -rw-r--r-- blocksize code=5 (32 subblocks) lastBlockSubblocks=0 checksum=0xAD8E0B4B is Valid fileSize=3968 nFullBlocks=0 currentMetadataReplicas=1 maxMetadataReplicas=2 currentDataReplicas=1 maxDataReplicas=2 ... Data [3968]: 0000000000000000: BCA91252 2B64BEDC A7D7BA9D D5BE8C30 *...R+d.........0* ... 0000000000000F70: DA925E2F 16A68C01 03CA5E37 08D72B7F *..^/......^7..+.* trailer: is NULL -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Mon Oct 3 16:59:54 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Mon, 3 Oct 2016 15:59:54 +0000 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? In-Reply-To: <1BDA1763-A6FE-4A52-B2ED-22324C9AA16F@nuance.com> References: <1BDA1763-A6FE-4A52-B2ED-22324C9AA16F@nuance.com> Message-ID: Would you tier an in-inode file to the cloud? I mean, I wouldn't tier an in-inode file out to tape? Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Oesterlin, Robert [Robert.Oesterlin at nuance.com] Sent: 03 October 2016 16:56 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? What's going be taken away if you use Encryption or Transparent Cloud Tiering? Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: on behalf of Marc A Kaplan Reply-To: gpfsug main discussion list Date: Monday, October 3, 2016 at 10:46 AM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Biggest file that will fit inside an inode? 3968!! On a non-SELINUX system the answer is 3968 of data in a 4K inode, just 128 bytes of metadata. Caution: it's possible in some future release, this could change ... I don't know of any plans, I'm just saying ... Inode 16346892 [16346892] snap 0 (index 12 in block 255420): Inode address: 6:123049056 size 4096 nAddrs 330 indirectionLevel=INODE status=USERFILE objectVersion=1 generation=0xC0156CB nlink=1 owner uid=0 gid=0 mode=0200100644: -rw-r--r-- blocksize code=5 (32 subblocks) lastBlockSubblocks=0 checksum=0xAD8E0B4B is Valid fileSize=3968 nFullBlocks=0 currentMetadataReplicas=1 maxMetadataReplicas=2 currentDataReplicas=1 maxDataReplicas=2 ... Data [3968]: 0000000000000000: BCA91252 2B64BEDC A7D7BA9D D5BE8C30 *...R+d.........0* ... 0000000000000F70: DA925E2F 16A68C01 03CA5E37 08D72B7F *..^/......^7..+.* trailer: is NULL From Robert.Oesterlin at nuance.com Mon Oct 3 17:06:41 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 3 Oct 2016 16:06:41 +0000 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? In-Reply-To: References: <1BDA1763-A6FE-4A52-B2ED-22324C9AA16F@nuance.com> Message-ID: <37F799FA-7200-4011-81F0-6FF0DF3A4006@nuance.com> No, but (I think) TCT uses some of the EA space in the inode to indicate that the data is in the cloud tier. And yes, it does copy the data blocks in the inode to the clod tier if you migrate it. Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: on behalf of "Simon Thompson (Research Computing - IT Services)" Reply-To: gpfsug main discussion list Date: Monday, October 3, 2016 at 10:59 AM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Biggest file that will fit inside an inode? Would you tier an in-inode file to the cloud? I mean, I wouldn't tier an in-inode file out to tape? Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Oesterlin, Robert [Robert.Oesterlin at nuance.com] Sent: 03 October 2016 16:56 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? What's going be taken away if you use Encryption or Transparent Cloud Tiering? Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: > on behalf of Marc A Kaplan > Reply-To: gpfsug main discussion list > Date: Monday, October 3, 2016 at 10:46 AM To: gpfsug main discussion list > Subject: [EXTERNAL] Re: [gpfsug-discuss] Biggest file that will fit inside an inode? 3968!! On a non-SELINUX system the answer is 3968 of data in a 4K inode, just 128 bytes of metadata. Caution: it's possible in some future release, this could change ... I don't know of any plans, I'm just saying ... Inode 16346892 [16346892] snap 0 (index 12 in block 255420): Inode address: 6:123049056 size 4096 nAddrs 330 indirectionLevel=INODE status=USERFILE objectVersion=1 generation=0xC0156CB nlink=1 owner uid=0 gid=0 mode=0200100644: -rw-r--r-- blocksize code=5 (32 subblocks) lastBlockSubblocks=0 checksum=0xAD8E0B4B is Valid fileSize=3968 nFullBlocks=0 currentMetadataReplicas=1 maxMetadataReplicas=2 currentDataReplicas=1 maxDataReplicas=2 ... Data [3968]: 0000000000000000: BCA91252 2B64BEDC A7D7BA9D D5BE8C30 *...R+d.........0* ... 0000000000000F70: DA925E2F 16A68C01 03CA5E37 08D72B7F *..^/......^7..+.* trailer: is NULL _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=1C_5OQ7M5ibcRAY6xnDfRpKRJiw3qUawXpXihmGoQcs&s=skn6tFln6MeuKACIoxnUL_kuxHIC7AkskTni8IjzLj0&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From luke.raimbach at googlemail.com Mon Oct 3 17:07:39 2016 From: luke.raimbach at googlemail.com (Luke Raimbach) Date: Mon, 03 Oct 2016 16:07:39 +0000 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? In-Reply-To: References: <1BDA1763-A6FE-4A52-B2ED-22324C9AA16F@nuance.com> Message-ID: Surely it wouldn't go? Maybe the data would get copied out rather than stubbed... DMAPI can't be stupid enough to stub data out of an inode? Can it? Interesting question. Maybe I'll test that one. On Mon, 3 Oct 2016 at 17:00 Simon Thompson (Research Computing - IT Services) wrote: > > Would you tier an in-inode file to the cloud? > > I mean, I wouldn't tier an in-inode file out to tape? > > Simon > ________________________________________ > From: gpfsug-discuss-bounces at spectrumscale.org [ > gpfsug-discuss-bounces at spectrumscale.org] on behalf of Oesterlin, Robert [ > Robert.Oesterlin at nuance.com] > Sent: 03 October 2016 16:56 > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? > > What's going be taken away if you use Encryption or Transparent Cloud > Tiering? > > > Bob Oesterlin > Sr Storage Engineer, Nuance HPC Grid > > > From: on behalf of Marc A > Kaplan > Reply-To: gpfsug main discussion list > Date: Monday, October 3, 2016 at 10:46 AM > To: gpfsug main discussion list > Subject: [EXTERNAL] Re: [gpfsug-discuss] Biggest file that will fit inside > an inode? 3968!! > > On a non-SELINUX system the answer is 3968 of data in a 4K inode, just 128 > bytes of metadata. > > Caution: it's possible in some future release, this could change ... I > don't know of any plans, I'm just saying ... > > Inode 16346892 [16346892] snap 0 (index 12 in block 255420): > Inode address: 6:123049056 size 4096 nAddrs 330 > indirectionLevel=INODE status=USERFILE > objectVersion=1 generation=0xC0156CB nlink=1 > owner uid=0 gid=0 mode=0200100644: -rw-r--r-- > blocksize code=5 (32 subblocks) > lastBlockSubblocks=0 > checksum=0xAD8E0B4B is Valid > fileSize=3968 nFullBlocks=0 > currentMetadataReplicas=1 maxMetadataReplicas=2 > currentDataReplicas=1 maxDataReplicas=2 > ... > Data [3968]: > 0000000000000000: BCA91252 2B64BEDC A7D7BA9D D5BE8C30 *...R+d.........0* > ... > 0000000000000F70: DA925E2F 16A68C01 03CA5E37 08D72B7F *..^/......^7..+.* > trailer: is NULL > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Mon Oct 3 17:10:34 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Mon, 3 Oct 2016 16:10:34 +0000 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? In-Reply-To: References: <1BDA1763-A6FE-4A52-B2ED-22324C9AA16F@nuance.com> , Message-ID: TCT doesn't use dmapi though I thought? ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Luke Raimbach [luke.raimbach at googlemail.com] Sent: 03 October 2016 17:07 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? Surely it wouldn't go? Maybe the data would get copied out rather than stubbed... DMAPI can't be stupid enough to stub data out of an inode? Can it? Interesting question. Maybe I'll test that one. On Mon, 3 Oct 2016 at 17:00 Simon Thompson (Research Computing - IT Services) > wrote: Would you tier an in-inode file to the cloud? I mean, I wouldn't tier an in-inode file out to tape? Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Oesterlin, Robert [Robert.Oesterlin at nuance.com] Sent: 03 October 2016 16:56 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? What's going be taken away if you use Encryption or Transparent Cloud Tiering? Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: > on behalf of Marc A Kaplan > Reply-To: gpfsug main discussion list > Date: Monday, October 3, 2016 at 10:46 AM To: gpfsug main discussion list > Subject: [EXTERNAL] Re: [gpfsug-discuss] Biggest file that will fit inside an inode? 3968!! On a non-SELINUX system the answer is 3968 of data in a 4K inode, just 128 bytes of metadata. Caution: it's possible in some future release, this could change ... I don't know of any plans, I'm just saying ... Inode 16346892 [16346892] snap 0 (index 12 in block 255420): Inode address: 6:123049056 size 4096 nAddrs 330 indirectionLevel=INODE status=USERFILE objectVersion=1 generation=0xC0156CB nlink=1 owner uid=0 gid=0 mode=0200100644: -rw-r--r-- blocksize code=5 (32 subblocks) lastBlockSubblocks=0 checksum=0xAD8E0B4B is Valid fileSize=3968 nFullBlocks=0 currentMetadataReplicas=1 maxMetadataReplicas=2 currentDataReplicas=1 maxDataReplicas=2 ... Data [3968]: 0000000000000000: BCA91252 2B64BEDC A7D7BA9D D5BE8C30 *...R+d.........0* ... 0000000000000F70: DA925E2F 16A68C01 03CA5E37 08D72B7F *..^/......^7..+.* trailer: is NULL _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From luke.raimbach at googlemail.com Mon Oct 3 17:16:05 2016 From: luke.raimbach at googlemail.com (Luke Raimbach) Date: Mon, 03 Oct 2016 16:16:05 +0000 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? In-Reply-To: References: <1BDA1763-A6FE-4A52-B2ED-22324C9AA16F@nuance.com> Message-ID: It doesn't, but the end result is the same... data shipped off 'somewhere else' with a stub file? I have in my mind that DMAPI support was around before data-in-inode (or at least 4K inodes) was introduced, so it had to be made a bit cleverer to cope, but I may be misremembering that. On Mon, 3 Oct 2016 at 17:10 Simon Thompson (Research Computing - IT Services) wrote: > > TCT doesn't use dmapi though I thought? > ________________________________________ > From: gpfsug-discuss-bounces at spectrumscale.org [ > gpfsug-discuss-bounces at spectrumscale.org] on behalf of Luke Raimbach [ > luke.raimbach at googlemail.com] > Sent: 03 October 2016 17:07 > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? > > Surely it wouldn't go? Maybe the data would get copied out rather than > stubbed... DMAPI can't be stupid enough to stub data out of an inode? Can > it? Interesting question. > > Maybe I'll test that one. > > On Mon, 3 Oct 2016 at 17:00 Simon Thompson (Research Computing - IT > Services) > wrote: > > Would you tier an in-inode file to the cloud? > > I mean, I wouldn't tier an in-inode file out to tape? > > Simon > ________________________________________ > From: gpfsug-discuss-bounces at spectrumscale.org gpfsug-discuss-bounces at spectrumscale.org> [ > gpfsug-discuss-bounces at spectrumscale.org gpfsug-discuss-bounces at spectrumscale.org>] on behalf of Oesterlin, Robert > [Robert.Oesterlin at nuance.com] > Sent: 03 October 2016 16:56 > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? > > What's going be taken away if you use Encryption or Transparent Cloud > Tiering? > > > Bob Oesterlin > Sr Storage Engineer, Nuance HPC Grid > > > From: gpfsug-discuss-bounces at spectrumscale.org>> on behalf of Marc A Kaplan < > makaplan at us.ibm.com> > Reply-To: gpfsug main discussion list > > Date: Monday, October 3, 2016 at 10:46 AM > To: gpfsug main discussion list gpfsug-discuss at spectrumscale.org>> > Subject: [EXTERNAL] Re: [gpfsug-discuss] Biggest file that will fit inside > an inode? 3968!! > > On a non-SELINUX system the answer is 3968 of data in a 4K inode, just 128 > bytes of metadata. > > Caution: it's possible in some future release, this could change ... I > don't know of any plans, I'm just saying ... > > Inode 16346892 [16346892] snap 0 (index 12 in block 255420): > Inode address: 6:123049056 size 4096 nAddrs 330 > indirectionLevel=INODE status=USERFILE > objectVersion=1 generation=0xC0156CB nlink=1 > owner uid=0 gid=0 mode=0200100644: -rw-r--r-- > blocksize code=5 (32 subblocks) > lastBlockSubblocks=0 > checksum=0xAD8E0B4B is Valid > fileSize=3968 nFullBlocks=0 > currentMetadataReplicas=1 maxMetadataReplicas=2 > currentDataReplicas=1 maxDataReplicas=2 > ... > Data [3968]: > 0000000000000000: BCA91252 2B64BEDC A7D7BA9D D5BE8C30 *...R+d.........0* > ... > 0000000000000F70: DA925E2F 16A68C01 03CA5E37 08D72B7F *..^/......^7..+.* > trailer: is NULL > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Mon Oct 3 17:26:08 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 3 Oct 2016 12:26:08 -0400 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? What it says! In-Reply-To: References: <1BDA1763-A6FE-4A52-B2ED-22324C9AA16F@nuance.com>, Message-ID: I just noticed that tsdbfs shows you how much data fits in the inode, even if you don't fill it... So a few commands will give you the answer: dd, sync, ls, tsdbfs, inode Encryption uses EAs, so you lose some space in inode to that. I don't have an encrypted FS handy for testing, if you have one, go ahead and use tsdbfs to look at an inode and see whether or not encryption pushes the data out of the inode, I wouldn't be surprised either way. [root at n2 isizes]# dd if=/dev/urandom bs=1 count=100 of=i100 100+0 records in 100+0 records out 100 bytes (100 B) copied, 0.00226643 s, 44.1 kB/s [root at n2 isizes]# sync [root at n2 isizes]# ls -ild i100 16346894 -rw-r--r-- 1 root root 100 Oct 3 09:14 i100 [root at n2 isizes]# tsdbfs mak Enter command or null to read next sector. Type ? for help. inode 16346894 Inode 16346894 [16346894] snap 0 (index 14 in block 255420): Inode address: 6:123049072 size 4096 nAddrs 330 indirectionLevel=INODE status=USERFILE objectVersion=1 generation=0x9D45DC1 nlink=1 owner uid=0 gid=0 mode=0200100644: -rw-r--r-- blocksize code=5 (32 subblocks) lastBlockSubblocks=0 checksum=0x2496E008 is Valid fileSize=100 nFullBlocks=0 ... Data [3968]: 0000000000000000: 17699B6F 5F9ACD28 70C05242 9268F44E *.i.o_..(p.RB.h.N* 0000000000000010: 8FFCDCC1 C2ACE0EC 69C1FE2A 986B752C *........i..*.ku,* 0000000000000020: 8E0434E6 1904B7FC 8B0C5709 2243343C *..4.......W."C4<* 0000000000000030: BD317374 FB5C322D 8E257B69 FF573283 *.1st.\2-.%{i.W2.* 0000000000000040: 515B333B EDFAE930 9B6F8712 0814BF65 *Q[3;...0.o.....e* 0000000000000050: DBDCC25E 87EB2C16 77AE672D 45FB6BE3 *...^..,.w.g-E.k.* 0000000000000060: 65D52D17 00000000 00000000 00000000 *e.-.............* 0000000000000070: 00000000 00000000 00000000 00000000 *................* ... trailer: is NULL From: "Simon Thompson (Research Computing - IT Services)" To: gpfsug main discussion list Date: 10/03/2016 12:10 PM Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? Sent by: gpfsug-discuss-bounces at spectrumscale.org TCT doesn't use dmapi though I thought? ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Luke Raimbach [luke.raimbach at googlemail.com] Sent: 03 October 2016 17:07 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? Surely it wouldn't go? Maybe the data would get copied out rather than stubbed... DMAPI can't be stupid enough to stub data out of an inode? Can it? Interesting question. Maybe I'll test that one. On Mon, 3 Oct 2016 at 17:00 Simon Thompson (Research Computing - IT Services) > wrote: Would you tier an in-inode file to the cloud? I mean, I wouldn't tier an in-inode file out to tape? Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org< mailto:gpfsug-discuss-bounces at spectrumscale.org> [gpfsug-discuss-bounces at spectrumscale.org< mailto:gpfsug-discuss-bounces at spectrumscale.org>] on behalf of Oesterlin, Robert [Robert.Oesterlin at nuance.com] Sent: 03 October 2016 16:56 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? What's going be taken away if you use Encryption or Transparent Cloud Tiering? Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: > on behalf of Marc A Kaplan > Reply-To: gpfsug main discussion list > Date: Monday, October 3, 2016 at 10:46 AM To: gpfsug main discussion list > Subject: [EXTERNAL] Re: [gpfsug-discuss] Biggest file that will fit inside an inode? 3968!! On a non-SELINUX system the answer is 3968 of data in a 4K inode, just 128 bytes of metadata. Caution: it's possible in some future release, this could change ... I don't know of any plans, I'm just saying ... Inode 16346892 [16346892] snap 0 (index 12 in block 255420): Inode address: 6:123049056 size 4096 nAddrs 330 indirectionLevel=INODE status=USERFILE objectVersion=1 generation=0xC0156CB nlink=1 owner uid=0 gid=0 mode=0200100644: -rw-r--r-- blocksize code=5 (32 subblocks) lastBlockSubblocks=0 checksum=0xAD8E0B4B is Valid fileSize=3968 nFullBlocks=0 currentMetadataReplicas=1 maxMetadataReplicas=2 currentDataReplicas=1 maxDataReplicas=2 ... Data [3968]: 0000000000000000: BCA91252 2B64BEDC A7D7BA9D D5BE8C30 *...R+d.........0* ... 0000000000000F70: DA925E2F 16A68C01 03CA5E37 08D72B7F *..^/......^7..+.* trailer: is NULL _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan at buzzard.me.uk Mon Oct 3 17:34:33 2016 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Mon, 03 Oct 2016 17:34:33 +0100 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? In-Reply-To: References: <1BDA1763-A6FE-4A52-B2ED-22324C9AA16F@nuance.com> Message-ID: <1475512473.3748.14.camel@buzzard.phy.strath.ac.uk> On Mon, 2016-10-03 at 16:16 +0000, Luke Raimbach wrote: > It doesn't, but the end result is the same... data shipped off > 'somewhere else' with a stub file? > > I have in my mind that DMAPI support was around before data-in-inode > (or at least 4K inodes) was introduced, so it had to be made a bit > cleverer to cope, but I may be misremembering that. > It did but you can always specify that files below a certain size don't get migrated, be it tape cloud or elsewhere. Never made sense migrating small files to tape anyway, and it certainly makes no sense to migrate a file in an inode anywhere. Perhaps that is the source of a request for enhancement? Stop files existing in inodes being migrated. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From luis.bolinches at fi.ibm.com Mon Oct 3 18:33:04 2016 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Mon, 3 Oct 2016 17:33:04 +0000 Subject: [gpfsug-discuss] WebDAV protocol access support In-Reply-To: Message-ID: Hi You might want to take a look to https://owncloud.com/ibm/ -- Cheers > On 3 Oct 2016, at 18.53, Aslam Ghumra (IT Services, Facilities Management) wrote: > > Does anyone know whether WebDAV protocol access support will ( if ever ) be implemented within GPFS ? > > Regards, > > Aslam Ghumra > Research Data Management > ____________________________ > IT Services > Elms Road Data Centre > Building G5 > Edgbaston > Birmingham B15 2TT > T: 0121 414 5877 > F; 0121 414 3952 > Skype : JanitorX > Twitter : @aslamghumra > in : https://uk.linkedin.com/in/aslam-ghumra-13907993 > http://intranet.bham.ac.uk/bear > Ellei edell?? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.kidger at uk.ibm.com Tue Oct 4 11:43:45 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Tue, 4 Oct 2016 10:43:45 +0000 Subject: [gpfsug-discuss] File_heat for GPFS File Systems In-Reply-To: References: , <8c7fd395-1efc-7197-4a98-763ba784cafd@stanford.edu> Message-ID: An HTML attachment was scrubbed... URL: From alandhae at gmx.de Tue Oct 4 12:25:52 2016 From: alandhae at gmx.de (=?ISO-8859-15?Q?Andreas_Landh=E4u=DFer?=) Date: Tue, 4 Oct 2016 13:25:52 +0200 (CEST) Subject: [gpfsug-discuss] File_heat for GPFS File Systems In-Reply-To: References: , <8c7fd395-1efc-7197-4a98-763ba784cafd@stanford.edu> Message-ID: On Tue, 4 Oct 2016, Daniel Kidger wrote: Daniel, ahh this might also help in my HPC Environment. We are already using AFM, just on HDD, for distributing HPC Data for usage on remote scientific sites. Using the AFM for a temp space seems to be a good idea, will the individual file heat be increased by using ILM policy and AFM? Using a large GPFS Filesystem for scientific results, it seems to be very unlikely all users starting using their cold files at the same point of time, so filesystem space should be sufficient. Thanks for the hint Andreas T-Systems SfR GmbH > If you need functionality like you describe, then as well as using tiering > with the ILM Policy Engine, also consider using AFM to cache onto say SSDs.? > That way you would have dynamic caching: as soon as a file gets accessed, it > goes into the cache and further I/O will be do the SSD copy so faster until > such time as the file ages and newer files replace it in the AFM cache. The > cost of this is perhaps a little more complexity and also a 'hot' file > consumes space on both SSD and disk. > ? > Daniel > /spectrum_storage-banne > ?????????????????????????????? v > ? > Spectrum Scale Logo > ? > ? > Dr Daniel Kidger > IBM?Technical Sales Specialist > Software Defined Solution Sales > > +44-(0)7818 522 266 > daniel.kidger at uk.ibm.com > ? > ? > ? > ----- Original message ----- > From: Eric Horst > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug main discussion list > > Cc: > Subject: Re: [gpfsug-discuss] File_heat for GPFS File Systems > Date: Tue, Sep 27, 2016 9:56 PM > ? >> > >> if a file gets hot again, there is no rule for putting the > file back > >> into a faster storage device? > > > > > > The file will get moved when you run the policy again. ?You > can run the > > policy as often as you like. > > I think its worth stating clearly that if a file is in the > Thrifty > slow pool and a user opens and reads/writes the file there is > nothing > that moves this file to a different tier. A policy run is the > only > action that relocates files. So if you apply the policy daily > and over > the course of the day users access many cold files, the > performance > accessing those cold files may not be ideal until the next day > when > they are repacked by heat. A file is not automatically moved to > the > fast tier on access read or write. I mention this because this > aspect > of tiering was not immediately clear from the docs when I was a > neophyte GPFS admin and I had to learn by observation. It is > easy for > one to make an assumption that it is a more dynamic tiering > system > than it is. > > -Eric > > -- > Eric Horst > University of Washington > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > ? > > ? > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > -- Andreas Landh?u?er +49 151 12133027 (mobile) alandhae at gmx.de From alandhae at gmx.de Tue Oct 4 13:54:16 2016 From: alandhae at gmx.de (=?ISO-8859-15?Q?Andreas_Landh=E4u=DFer?=) Date: Tue, 4 Oct 2016 14:54:16 +0200 (CEST) Subject: [gpfsug-discuss] Fileheat reporting Message-ID: Customer needs a report of filenames being Hot, warm and cold. As far as I'm understanding fileheat can be any value larger than zero. A value of zero or almost zero equals to COLD How am I classifying warm and hot? I'm needing a standardized value between 0 and 1 for fileheat. Is it possible creating such a number when when activating fileheat and creating reports and finally using ILM policies according to this classification? Best regards Andreas -- Andreas Landh?u?er +49 151 12133027 (mobile) alandhae at gmx.de From eric.wonderley at vt.edu Tue Oct 4 14:39:02 2016 From: eric.wonderley at vt.edu (J. Eric Wonderley) Date: Tue, 4 Oct 2016 09:39:02 -0400 Subject: [gpfsug-discuss] migrate policy vs restripe Message-ID: We have the need to move data from one set of spindles to another. Are there any performance or availability considerations when choosing to do either a migration policy or a restripe to make this move? I did discover that a restripe only works within the same pool...even though you setup two pools in different failure groups. -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Tue Oct 4 15:32:55 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 4 Oct 2016 10:32:55 -0400 Subject: [gpfsug-discuss] migrate policy vs restripe In-Reply-To: References: Message-ID: Doubtless there are subtleties and special cases, but my should-work-well advice is: A) Use mmapplypolicy ... -I defer -P policy-to-set-pool-assignments-and-replication-factor ... where the policy rules file contains rules like RULE 'adjust' MIGRATE [FROM POOL 'x'] TO POOL 'y' REPLICATE({1|2}) WHERE decide to which file this rule shall apply The '-I defer' option says just mark the inodes, do not actually move the data B) Use mmrestripefs ... -b ... -N nodeclass The move data to correct pool assignments and rebalance the entire file system, also fix ill-replicated and ill-placed files. (keep reading...) C) "restripe only works within the same pool" -- I believe there is a misunderstanding. For any given file, all the data blocks are supposed to be in the one pool indicated by pool assignment which is recorded in the file's inode. If the pool assignment changes but not all of the blocks are migrated to the new pool, then the file is considered "ill-placed". I believe mmrestripefs endeavors to move blocks so as to "fix" ill-placed files. Failure groups are a different concept that is associated with replication. The replicas of a given data or meta data block should be in different failure groups. If not, the file is considered "ill-replicated" and mmrestripefs endeavors to fix that also. From: "J. Eric Wonderley" To: gpfsug main discussion list Date: 10/04/2016 09:39 AM Subject: [gpfsug-discuss] migrate policy vs restripe Sent by: gpfsug-discuss-bounces at spectrumscale.org We have the need to move data from one set of spindles to another. Are there any performance or availability considerations when choosing to do either a migration policy or a restripe to make this move? I did discover that a restripe only works within the same pool...even though you setup two pools in different failure groups. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From aspalazz at us.ibm.com Tue Oct 4 15:44:33 2016 From: aspalazz at us.ibm.com (Aaron S Palazzolo) Date: Tue, 4 Oct 2016 14:44:33 +0000 Subject: [gpfsug-discuss] Toolkit Message-ID: An HTML attachment was scrubbed... URL: From stef.coene at docum.org Tue Oct 4 15:53:48 2016 From: stef.coene at docum.org (Stef Coene) Date: Tue, 4 Oct 2016 16:53:48 +0200 Subject: [gpfsug-discuss] Toolkit In-Reply-To: References: Message-ID: <6192009e-afea-36f1-d79a-9ec78194fe6d@docum.org> On 10/04/2016 04:44 PM, Aaron S Palazzolo wrote: > *Hi Stef,* > > Thanks for your Install Toolkit feature request: > -------------------- > /Is it possible to recreate the clusterdefinition.txt based on the > current configuration?/ > -------------------- > > A couple questions if you don't mind: > *1) *Are you using the Install Toolkit for base gpfs installation or > solely protocol deployment? I used the spectrum toolkit for basic setup. Then I used some mm commands to add disks and change other stuff. But the toolkit is unaware of theses changes. > *2) *If using it for base gpfs installation, do you create both NSDs and > file systems with it or do you do this manually? Like said, I used the toolkit for the first few NSDs and later used mm commands for other NSDs. I have no problems using the mm commands, but I don't know what will happen if I want to use the toolkit to create proto nodes. Stef From makaplan at us.ibm.com Tue Oct 4 15:56:46 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 4 Oct 2016 10:56:46 -0400 Subject: [gpfsug-discuss] Fileheat reporting In-Reply-To: References: Message-ID: FILE_HEAT value is a non-negative floating point value. The value can grow quite large -- in theory it can grow to the IEEE 64 bit maximum floating point value and maybe even to "infinity". Each time a file is read completely from disk it's FILE_HEAT value increases by 1. If a fraction, f, of the file is read then FILE_HEAT += f. (f<=1) On the other hand, as time passes the FILE_HEAT "decays" at the rate given by the (mmchconfig) parameters fileHeatPeriodMinutes and fileHeatLossPercent. In fact I recently corresponding with a customer (who is no stranger to this forum and may wish to comment!) who found that a bunch of hot little script files had accumulated heat values greater than 100,000,000 ! We were both surprised. Apparently these files are read and re-read many times every day by many nodes and so their heat builds up! So what would you call "Hot", "Warm", "Cold"? It depends... To get a good idea, I suggest that a "survey" of FILE_HEAT values be done by using an mmapplypolicy command with a rule like. rule 'y' list 'fhn0' weight(FILE_HEAT) SHOW(HEX(XATTR('gpfs.FileHeat')) || ' A=' || varchar(ACCESS_TIME) || ' K=' || varchar(KB_ALLOCATED) || ' H=' || varchar(FILE_HEAT)) where FILE_HEAT != 0.0 Using `mmapplypolicy FS -P policy-rules-file -I defer -f /tmp/whatever [other options such as... -N nodeclass -g /shared-temp ... ] This will produce a file list sorted by FILE_HEAT, which you can then peruse or otherwise process.... Perhaps a histogram or other display of (number of files) vs (heat values) would be interesting... From: Andreas Landh?u?er To: gpfsug-discuss at spectrumscale.org Date: 10/04/2016 08:54 AM Subject: [gpfsug-discuss] Fileheat reporting Sent by: gpfsug-discuss-bounces at spectrumscale.org Customer needs a report of filenames being Hot, warm and cold. As far as I'm understanding fileheat can be any value larger than zero. A value of zero or almost zero equals to COLD How am I classifying warm and hot? I'm needing a standardized value between 0 and 1 for fileheat. Is it possible creating such a number when when activating fileheat and creating reports and finally using ILM policies according to this classification? Best regards Andreas -- Andreas Landh?u?er +49 151 12133027 (mobile) alandhae at gmx.de_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From kronberg at nsc.liu.se Tue Oct 4 16:43:51 2016 From: kronberg at nsc.liu.se (Mats Kronberg) Date: Tue, 4 Oct 2016 17:43:51 +0200 Subject: [gpfsug-discuss] GSS 3.0 (GPFS 4.2.1) upgrade experiences? Message-ID: Hi, We are in the planning phase for upgrading our GSS system from GSS 2.0.7 (GPFS 4.1.0-8) to something more recent. Lenovo recommends going directly to the recently released GSS 3.0 (GPFS 4.2.1-0, released late September 2016). I'm inclined to agree with them, but the conservative, let-others-betatest-stuff part of me is suggesting something a little more mature like GSS 2.5.9 (GPFS 4.1.1-2, released late-2015). Has anyone got any success or horror stories to share regarding either of these releases? :) -- Mats Kronberg, Systems Expert NSC, National Supercomputer Centre -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Tue Oct 4 16:48:31 2016 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Tue, 4 Oct 2016 15:48:31 +0000 Subject: [gpfsug-discuss] GSS 3.0 (GPFS 4.2.1) upgrade experiences? In-Reply-To: References: Message-ID: We?re not huge, and not running GSS, but we are running CES for 3000+ concurrent users. We went from 3.5 to 4.2.1 via 4.1.1 and our experience was outstanding. We had some enforced downtime during the upgrad and a few hiccups with CES specifically, but overall very happy. That may help you in some shape or form ? From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Mats Kronberg Sent: 04 October 2016 16:44 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] GSS 3.0 (GPFS 4.2.1) upgrade experiences? Hi, We are in the planning phase for upgrading our GSS system from GSS 2.0.7 (GPFS 4.1.0-8) to something more recent. Lenovo recommends going directly to the recently released GSS 3.0 (GPFS 4.2.1-0, released late September 2016). I'm inclined to agree with them, but the conservative, let-others-betatest-stuff part of me is suggesting something a little more mature like GSS 2.5.9 (GPFS 4.1.1-2, released late-2015). Has anyone got any success or horror stories to share regarding either of these releases? :) -- Mats Kronberg, Systems Expert NSC, National Supercomputer Centre -------------- next part -------------- An HTML attachment was scrubbed... URL: From mimarsh2 at vt.edu Tue Oct 4 18:04:59 2016 From: mimarsh2 at vt.edu (Brian Marshall) Date: Tue, 4 Oct 2016 13:04:59 -0400 Subject: [gpfsug-discuss] testing commands Message-ID: All, Is there a way to "test" GPFS commands and see what the output or result would be before running them? For example, I'd like to verify a command string before actually running it on a production system. Does IBM offer "test" licenses for setting up a small debug/devel environment? I'd be interested to hear everyone's best practices on verifying commands/actions. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Tue Oct 4 18:30:03 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 4 Oct 2016 13:30:03 -0400 Subject: [gpfsug-discuss] testing commands In-Reply-To: References: Message-ID: Simple functional testing can be done on a small system with as little as one Linux node and one disk. Personally, I do a lot of testing on a several years old laptop running Linux. My test gpfs file system doesn't even use a real disk partition! Just some data files I initialized with dd if=/dev/zero ... (This is not officially supported, blah, blah, blah...) [root at bog-wifi cmvc]# mmlsnsd -X Disk name NSD volume ID Device Devtype Node name Remarks --------------------------------------------------------------------------------------------------- y1 C0A8015D57586A5E /vds/y1 file bog-wifi.lan.makaplan.us server node y2 C0A8015D57586A5F /vds/y2 file bog-wifi.lan.makaplan.us server node y3 C0A8015D575EF0A3 /vds/y3 file bog-wifi.lan.makaplan.us server node [root at bog-wifi cmvc]# ls -ld /vds/y? -rw-r--r--. 1 root root 536870912 Oct 4 12:16 /vds/y1 -rw-r--r--. 1 root root 536870912 Oct 3 10:39 /vds/y2 -rw-r--r--. 1 root root 801000000 Sep 29 18:54 /vds/y3 [root at bog-wifi cmvc]# df -T /vds Filesystem Type 1K-blocks Used Available Use% Mounted on /dev/mapper/rhelw_bog--wifi-root ext4 183862728 42118120 132381768 25% / -------------- next part -------------- An HTML attachment was scrubbed... URL: From mimarsh2 at vt.edu Tue Oct 4 21:13:05 2016 From: mimarsh2 at vt.edu (Brian Marshall) Date: Tue, 4 Oct 2016 16:13:05 -0400 Subject: [gpfsug-discuss] testing commands In-Reply-To: References: Message-ID: Do I require additional NSD Server licenses to do this? i.e. What triggers the need to purchase a license for a NSD Server? Brian On Tue, Oct 4, 2016 at 1:30 PM, Marc A Kaplan wrote: > Simple functional testing can be done on a small system with as little as > one Linux node and one disk. > > Personally, I do a lot of testing on a several years old laptop running > Linux. My test gpfs file system doesn't even use a real disk partition! > Just some data files > I initialized with dd if=/dev/zero ... > > (This is not officially supported, blah, blah, blah...) > > [root at bog-wifi cmvc]# mmlsnsd -X > > Disk name NSD volume ID Device Devtype Node name > Remarks > ------------------------------------------------------------ > --------------------------------------- > y1 C0A8015D57586A5E /vds/y1 file > bog-wifi.lan.makaplan.us server node > y2 C0A8015D57586A5F /vds/y2 file > bog-wifi.lan.makaplan.us server node > y3 C0A8015D575EF0A3 /vds/y3 file > bog-wifi.lan.makaplan.us server node > > [root at bog-wifi cmvc]# ls -ld /vds/y? > -rw-r--r--. 1 root root 536870912 Oct 4 12:16 /vds/y1 > -rw-r--r--. 1 root root 536870912 Oct 3 10:39 /vds/y2 > -rw-r--r--. 1 root root 801000000 Sep 29 18:54 /vds/y3 > > [root at bog-wifi cmvc]# df -T /vds > Filesystem Type 1K-blocks Used Available Use% > Mounted on > /dev/mapper/rhelw_bog--wifi-root ext4 183862728 42118120 132381768 25% / > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From swinglet at us.ibm.com Tue Oct 4 21:24:43 2016 From: swinglet at us.ibm.com (Teresa S Swingler) Date: Tue, 4 Oct 2016 20:24:43 +0000 Subject: [gpfsug-discuss] Survey - IBM Spectrum Scale Everyday Management Tasks Message-ID: An HTML attachment was scrubbed... URL: From jtucker at pixitmedia.com Tue Oct 4 21:37:42 2016 From: jtucker at pixitmedia.com (Jez Tucker) Date: Tue, 4 Oct 2016 21:37:42 +0100 Subject: [gpfsug-discuss] testing commands In-Reply-To: References: Message-ID: Hi So we were chatting a while ago about dry-run requirements, but more with respect to mmapplypolicy. For relevance to those on the list who run our Python API, you'll find dry-run functionality here: http://arcapix.com/gpfsapi/execute.html#dry-run Cheers, Jez On 04/10/16 18:04, Brian Marshall wrote: > All, > > Is there a way to "test" GPFS commands and see what the output or > result would be before running them? For example, I'd like to verify > a command string before actually running it on a production system. > > Does IBM offer "test" licenses for setting up a small debug/devel > environment? > > I'd be interested to hear everyone's best practices on verifying > commands/actions. > > > Thanks, > Brian > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Jez Tucker Head of Research & Product Development Pixit Media | ArcaStream www.pixitmedia.com www.arcastream.com -- This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From swinglet at us.ibm.com Tue Oct 4 21:37:39 2016 From: swinglet at us.ibm.com (Teresa S Swingler) Date: Tue, 4 Oct 2016 13:37:39 -0700 Subject: [gpfsug-discuss] Survey - Spectrum Scale Everyday Management Message-ID: Hi, We in IBM Spectrum Scale design and development are interested in learning about the frequency and usability of your everyday management tasks. Your input will help us with our upcoming priorities. Please take 5 minutes to complete this brief survey: https://www.surveygizmo.com/s3/3090036/IBM-Spectrum-Scale-Everyday-Management Thank you, Teresa Swingler Teresa S. Swingler IBM Systems Senior Designer | Lead UX Architect Spectrum Scale and Spectrum Control 2929 N. Central Ave. Phoenix, AZ 85012 Phone: (602) 217-2763 Tieline: 667-2763 E-mail: swinglet at us.ibm.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Tue Oct 4 22:49:01 2016 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 4 Oct 2016 17:49:01 -0400 Subject: [gpfsug-discuss] testing commands In-Reply-To: References: Message-ID: <797ae533-edee-102a-3f3e-f53687cfcf83@nasa.gov> Not to get off-topic here, but WHOA! I had no idea somebody had created a python API for GPFS. How does one get access to said API? On 10/4/16 4:37 PM, Jez Tucker wrote: > Hi > > So we were chatting a while ago about dry-run requirements, but more > with respect to mmapplypolicy. > For relevance to those on the list who run our Python API, you'll find > dry-run functionality here: > > http://arcapix.com/gpfsapi/execute.html#dry-run > > Cheers, > > Jez > > > > On 04/10/16 18:04, Brian Marshall wrote: >> All, >> >> Is there a way to "test" GPFS commands and see what the output or >> result would be before running them? For example, I'd like to verify >> a command string before actually running it on a production system. >> >> Does IBM offer "test" licenses for setting up a small debug/devel >> environment? >> >> I'd be interested to hear everyone's best practices on verifying >> commands/actions. >> >> >> Thanks, >> Brian >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- > Jez Tucker > Head of Research & Product Development > Pixit Media | ArcaStream > www.pixitmedia.com > www.arcastream.com > > > This email is confidential in that it is intended for the exclusive > attention of the addressee(s) indicated. If you are not the intended > recipient, this email should not be read or disclosed to any other > person. Please notify the sender immediately and delete this email from > your computer system. Any opinions expressed are not necessarily those > of the company from which this email was sent and, whilst to the best of > our knowledge no viruses or defects exist, no responsibility can be > accepted for any loss or damage arising from its receipt or subsequent > use of this email. > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From makaplan at us.ibm.com Tue Oct 4 23:15:32 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 4 Oct 2016 18:15:32 -0400 Subject: [gpfsug-discuss] testing commands - licensing In-Reply-To: References: Message-ID: http://www.ibm.com/developerworks/servicemanagement/sdi/spectrum-scale/resources.html ... Please contact your IBM sales representative for more information. ... http://www.ibm.com/developerworks/servicemanagement/tc/gpfs/evaluate.html https://www-01.ibm.com/marketing/iwm/iwm/web/preLogin.do?source=swerpzsw-scale42-3 The 90 day trial license includes these terms... If the Program is designated as "Non-Production", the Program can only be deployed as part of the Licensee's internal development and test environment for internal non-production activities, including but not limited to testing, performance tuning, fault diagnosis, internal benchmarking, staging, quality assurance activity and/or developing internally used additions or extensions to the Program using published application programming interfaces. Licensee is not authorized to use any part of the Program for any other purposes without acquiring the appropriate production entitlements. ... This is posting does not constitute or include the full terms of a license ... It just provides some pointers and lets you know that IBM does offer "Trial" licenses for Spectrum Scale. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aspalazz at us.ibm.com Wed Oct 5 02:22:35 2016 From: aspalazz at us.ibm.com (Aaron S Palazzolo) Date: Wed, 5 Oct 2016 01:22:35 +0000 Subject: [gpfsug-discuss] Toolkit Message-ID: An HTML attachment was scrubbed... URL: From stef.coene at docum.org Wed Oct 5 07:30:21 2016 From: stef.coene at docum.org (Stef Coene) Date: Wed, 5 Oct 2016 08:30:21 +0200 Subject: [gpfsug-discuss] Toolkit In-Reply-To: References: Message-ID: On 10/05/2016 03:22 AM, Aaron S Palazzolo wrote: > *Hi Stef,* > > Good info and thanks for the scenario specifics. I will include your > input in our upcoming design sessions and we can let you know as soon as > any functional improvements are on their way. I just realized that this also a good way to document a GPFS cluster ;) Stef From daniel.kidger at uk.ibm.com Wed Oct 5 14:27:39 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Wed, 5 Oct 2016 13:27:39 +0000 Subject: [gpfsug-discuss] testing commands In-Reply-To: References: , Message-ID: An HTML attachment was scrubbed... URL: From Richard.E.Powell at boeing.com Thu Oct 6 22:32:31 2016 From: Richard.E.Powell at boeing.com (Powell, Richard E) Date: Thu, 6 Oct 2016 21:32:31 +0000 Subject: [gpfsug-discuss] Will there be a UG meeting at SC16? Message-ID: Has there been any word on a Spectrum Scale User Group meeting in conjunction with SC16 in November? -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Thu Oct 6 22:36:18 2016 From: bbanister at jumptrading.com (Bryan Banister) Date: Thu, 6 Oct 2016 21:36:18 +0000 Subject: [gpfsug-discuss] Will there be a UG meeting at SC16? In-Reply-To: References: Message-ID: <21BC488F0AEA2245B2C3E83FC0B33DBB0634F863@CHI-EXCHANGEW1.w2k.jumptrading.com> Word on the street is that the SS UG meeting will be on Sunday, Nov 13th, in the afternoon, -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Powell, Richard E Sent: Thursday, October 06, 2016 4:33 PM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Will there be a UG meeting at SC16? Has there been any word on a Spectrum Scale User Group meeting in conjunction with SC16 in November? ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From swinglet at us.ibm.com Thu Oct 6 22:38:13 2016 From: swinglet at us.ibm.com (Teresa S Swingler) Date: Thu, 6 Oct 2016 14:38:13 -0700 Subject: [gpfsug-discuss] Thank you and Last Call - Everyday Management 5 minute Survey Message-ID: Thank you to all of you who have responded to our short survey of everyday management tasks. We value your feedback as it helps to better inform our upcoming priorities. For those of you who have not had a chance to respond yet, please take 5 minutes to fill out this quick survey: https://www.surveygizmo.com/s3/3090036/IBM-Spectrum-Scale-Everyday-Management The survey will be closed on Sunday. We appreciate your input and thank you for your time. Teresa Teresa S. Swingler IBM Systems Senior Designer | Lead UX Architect Spectrum Scale and Spectrum Control 2929 N. Central Ave. Phoenix, AZ 85012 Phone: (602) 217-2763 Tieline: 667-2763 E-mail: swinglet at us.ibm.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Thu Oct 6 22:40:33 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Thu, 6 Oct 2016 21:40:33 +0000 Subject: [gpfsug-discuss] Will there be a UG meeting at SC16? In-Reply-To: <21BC488F0AEA2245B2C3E83FC0B33DBB0634F863@CHI-EXCHANGEW1.w2k.jumptrading.com> References: , <21BC488F0AEA2245B2C3E83FC0B33DBB0634F863@CHI-EXCHANGEW1.w2k.jumptrading.com> Message-ID: That's correct Sunday (13th?). Agenda is just being finished up at the moment. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Bryan Banister [bbanister at jumptrading.com] Sent: 06 October 2016 22:36 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Will there be a UG meeting at SC16? Word on the street is that the SS UG meeting will be on Sunday, Nov 13th, in the afternoon, -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Powell, Richard E Sent: Thursday, October 06, 2016 4:33 PM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Will there be a UG meeting at SC16? Has there been any word on a Spectrum Scale User Group meeting in conjunction with SC16 in November? ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. From kraemerf at de.ibm.com Fri Oct 7 13:08:19 2016 From: kraemerf at de.ibm.com (Frank Kraemer) Date: Fri, 7 Oct 2016 14:08:19 +0200 Subject: [gpfsug-discuss] Spectrum Scale for Linux on z Systems now supports native Spectrum Scale encryption Message-ID: >Peter Muench1/Germany/IBM wrote on 07.10.2016 12:11:52: Since September 20th, Spectrum Scale for Linux on z Systems 4.2.1.1 code is available. (Download is via fixcentral) Spectrum Scale for Linux on z Systems now supports native GPFS encryption: >From the latest FAQ (see Q2.28: What are the current requirements / limitations for IBM Spectrum Scale for Linux on z Systems?) If you have further questions, please let me know. Thank you. Peter Muench Spectrum Scale for Linux on z Systems Development mailto:muenchp at de.ibm.com IBM Deutschland Research & Development GmbH -------------- next part -------------- An HTML attachment was scrubbed... URL: From volobuev at us.ibm.com Fri Oct 7 17:20:01 2016 From: volobuev at us.ibm.com (Yuri L Volobuev) Date: Fri, 7 Oct 2016 09:20:01 -0700 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? In-Reply-To: References: <1BDA1763-A6FE-4A52-B2ED-22324C9AA16F@nuance.com> Message-ID: Exactly how much data can fit into an inode depends on whether any extended attributes (EAs) are present, which makes this complicated, as EAs may be set explicitly by the user or implicitly by the system. In any inode, there's a fixed footprint of the inode header, 128 bytes on recent GPFS streams. So if there's no possibility to fit more than (inodeSize - 128) bytes into an inode. If some EAs are present, this becomes more complicated. GPFS can store EAs either in the inode, or in a special "EA overflow block". Certain EAs used by GPFS internally (for encryption, AFM, DMAPI, clones) must reside in the inode, due to the need to have those set at certain junctures when it's tricky to accommodate EA overflow allocation with proper atomicity guarantees. Other subsystems, e.g. SELinux, may implicitly use EAs as well. So a given inode may have less space for data than described above. This is another big part of the reason why 4K is the current default inode size. And of course this is not an external spec set in stone, but rather the way the current implementation works. A future version of GPFS may have a bigger inode header, for example. So it would be imprudent to bank on file/directory data fitting into an inode with a great deal of precision. A better way to view this is as a kind of a performance optimization. DMAPI won't "punch out" data that resides in inode. Such a file has zero blocks allocated, so there's nothing for an HSM app to migrate. TCT can copy in-inode data to cloud for co-resident mode. And yes, TCT doesn't rely on the same DMAPI API as HSM, per se, but naturally the underlying code shares some of the infrastructure. yuri From: Luke Raimbach To: gpfsug main discussion list , Date: 10/03/2016 09:16 AM Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? Sent by: gpfsug-discuss-bounces at spectrumscale.org It doesn't, but the end result is the same... data shipped off 'somewhere else' with a stub file? I have in my mind that DMAPI support was around before data-in-inode (or at least 4K inodes) was introduced, so it had to be made a bit cleverer to cope, but I may be misremembering that. On Mon, 3 Oct 2016 at 17:10 Simon Thompson (Research Computing - IT Services) wrote: TCT doesn't use dmapi though I thought? ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [ gpfsug-discuss-bounces at spectrumscale.org] on behalf of Luke Raimbach [ luke.raimbach at googlemail.com] Sent: 03 October 2016 17:07 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? Surely it wouldn't go? Maybe the data would get copied out rather than stubbed... DMAPI can't be stupid enough to stub data out of an inode? Can it? Interesting question. Maybe I'll test that one. On Mon, 3 Oct 2016 at 17:00 Simon Thompson (Research Computing - IT Services) > wrote: Would you tier an in-inode file to the cloud? I mean, I wouldn't tier an in-inode file out to tape? Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [ gpfsug-discuss-bounces at spectrumscale.org] on behalf of Oesterlin, Robert [Robert.Oesterlin at nuance.com] Sent: 03 October 2016 16:56 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? What's going be taken away if you use Encryption or Transparent Cloud Tiering? Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: > on behalf of Marc A Kaplan < makaplan at us.ibm.com> Reply-To: gpfsug main discussion list > Date: Monday, October 3, 2016 at 10:46 AM To: gpfsug main discussion list > Subject: [EXTERNAL] Re: [gpfsug-discuss] Biggest file that will fit inside an inode? 3968!! On a non-SELINUX system the answer is 3968 of data in a 4K inode, just 128 bytes of metadata. Caution: it's possible in some future release, this could change ... I don't know of any plans, I'm just saying ... Inode 16346892 [16346892] snap 0 (index 12 in block 255420): ? Inode address: 6:123049056 size 4096 nAddrs 330 ? indirectionLevel=INODE status=USERFILE ? objectVersion=1 generation=0xC0156CB nlink=1 ? owner uid=0 gid=0 mode=0200100644: -rw-r--r-- ? blocksize code=5 (32 subblocks) ? lastBlockSubblocks=0 ? checksum=0xAD8E0B4B is Valid ? fileSize=3968 nFullBlocks=0 ? currentMetadataReplicas=1 maxMetadataReplicas=2 ? currentDataReplicas=1 maxDataReplicas=2 ? ... ? Data [3968]: 0000000000000000: BCA91252 2B64BEDC A7D7BA9D D5BE8C30? *...R+d.........0* ... 0000000000000F70: DA925E2F 16A68C01 03CA5E37 08D72B7F? *..^/......^7..+.* ? trailer: is NULL _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From Richard.E.Powell at boeing.com Sat Oct 8 00:55:28 2016 From: Richard.E.Powell at boeing.com (Powell, Richard E) Date: Fri, 7 Oct 2016 23:55:28 +0000 Subject: [gpfsug-discuss] Will there be a UG meeting at SC16? Message-ID: Thanks for the info! Can you tell us when and where, yet? We're trying to make our travel plans early, (for a change:)) -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Sat Oct 8 22:57:56 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Sat, 8 Oct 2016 21:57:56 +0000 Subject: [gpfsug-discuss] Will there be a UG meeting at SC16? Message-ID: <82021AF7-4908-4D7B-B3B9-557954A692C8@nuance.com> Sunday November 13th, from about 1PM - 5:30 PM. Not sure of the location, but it will be at one of the hotels near the convention center in downtown SLC. More details soon! Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid GPFS-UG Co-Principal From: on behalf of "Powell, Richard E" Reply-To: gpfsug main discussion list Date: Friday, October 7, 2016 at 6:55 PM To: "gpfsug-discuss at spectrumscale.org" Subject: [EXTERNAL] Re: [gpfsug-discuss] Will there be a UG meeting at SC16? Thanks for the info! Can you tell us when and where, yet? We?re trying to make our travel plans early, (for a change?) -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.childs at qmul.ac.uk Mon Oct 10 13:32:26 2016 From: p.childs at qmul.ac.uk (Peter Childs) Date: Mon, 10 Oct 2016 12:32:26 +0000 Subject: [gpfsug-discuss] GPFS Upgrade 3.5 -> 4.1 Message-ID: We are finishing upgrading our GPFS cluster of around 250 (client) nodes from GPFS 3.5.0.31 to Spectrum Scale 4.1.1.8, and have just about upgraded all the computers. We are looking at running the "mmchfs -V LATEST" step and where wondering how much io this takes and if it was likely to interrupt service? We are looking at upgrading to 4.2 but plan to do that via Multi-cluster and AFM as we are integrating new hardware and wish to increase the block and inode size at the same time. Peter Childs Research Storage Expert ITS Research Infrastructure Queen Mary, University of London From robert at strubi.ox.ac.uk Mon Oct 10 14:21:30 2016 From: robert at strubi.ox.ac.uk (Robert Esnouf) Date: Mon, 10 Oct 2016 14:21:30 +0100 (BST) Subject: [gpfsug-discuss] NEW DEPARTMENT/NEW POST: Head of Research Computing at the Oxford Big Data Institute Message-ID: <201610101321.061282@mail.strubi.ox.ac.uk> Dear All, PLEASE FEEL FREE TO PASS THIS EMAIL ON TO OTHER RELEVANT MAILING LISTS AND ANYONE WHO MIGHT BE INTERESTED The University of Oxford Big Data Institute is looking for a highly skilled research computing manager to establish a significant computing capability in Oxford over the next few years. The job advertisement can be found here: http://www.jobs.ac.uk/job/AUT085/head-of-bdi-research-computing/ The post is not just traditional HPC as it involves creating a large infrastructure for distributed, collaborative processing of sensitive data. If anyone wants an informal chat about how this role could develop then please drop me an email and I'll try to help. Regards, Robert -- Dr. Robert Esnouf, University Research Lecturer, Head of Research Computing Core, NDM Research Computing Strategy Officer Room 10/028, Wellcome Trust Centre for Human Genetics, Old Road Campus, Roosevelt Drive, Oxford OX3 7BN, UK Email: robert at strubi.ox.ac.uk / robert at well.ox.ac.uk Tel: (+44) - 1865 - 287783 From janfrode at tanso.net Mon Oct 10 15:35:02 2016 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Mon, 10 Oct 2016 14:35:02 +0000 Subject: [gpfsug-discuss] GPFS Upgrade 3.5 -> 4.1 In-Reply-To: References: Message-ID: I've also always been worried about that one, but never experienced it taking any time, I/O or interruption. I've the interpreted it to just start using new features, but not really changing anything with the existing metadata. Things needing on disk changes are probably put in mmmigratefs I have not heard about anything needing mmmigratefs since GPFS v3.3 (fs version 11.03) added fast extended attributes. Would be great to hear otherwize, or confirmations. -jf man. 10. okt. 2016 kl. 14.32 skrev Peter Childs : > We are finishing upgrading our GPFS cluster of around 250 (client) nodes > from GPFS 3.5.0.31 to Spectrum Scale 4.1.1.8, and have just about upgraded > all the computers. > > We are looking at running the "mmchfs -V LATEST" step and where wondering > how much io this takes and if it was likely to interrupt service? > > We are looking at upgrading to 4.2 but plan to do that via Multi-cluster > and AFM as we are integrating new hardware and wish to increase the block > and inode size at the same time. > > Peter Childs > Research Storage Expert > ITS Research Infrastructure > Queen Mary, University of London > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From volobuev at us.ibm.com Mon Oct 10 16:39:44 2016 From: volobuev at us.ibm.com (Yuri L Volobuev) Date: Mon, 10 Oct 2016 08:39:44 -0700 Subject: [gpfsug-discuss] GPFS Upgrade 3.5 -> 4.1 In-Reply-To: References: Message-ID: Correct. mmchfs -V only done quick operations (that can be easily undone if something goes wrong). Essentially the big task here is to increase on-disk file system descriptor version number, to allow using those features that require a higher version. Bigger "conversion"-style tasks belong in mmmigratefs. The only way to increase the inode size and the data block size is to format a new file system. This cannot be done on an existing file system. yuri From: Jan-Frode Myklebust To: gpfsug main discussion list , Date: 10/10/2016 07:35 AM Subject: Re: [gpfsug-discuss] GPFS Upgrade 3.5 -> 4.1 Sent by: gpfsug-discuss-bounces at spectrumscale.org I've also always been worried about that one, but never experienced it taking any time, I/O or interruption. I've the interpreted it to just start using new features, but not really changing anything with the existing metadata. Things needing on disk changes are probably put in mmmigratefs I have not heard about anything needing mmmigratefs since GPFS v3.3 (fs version 11.03) added fast extended attributes. Would be great to hear otherwize, or confirmations. -jf man. 10. okt. 2016 kl. 14.32 skrev Peter Childs : We are finishing upgrading our GPFS cluster of around 250 (client) nodes from GPFS 3.5.0.31 to Spectrum Scale 4.1.1.8, and have just about upgraded all the computers. We are looking at running the "mmchfs -V LATEST" step and where wondering how much io this takes and if it was likely to interrupt service? We are looking at upgrading to 4.2 but plan to do that via Multi-cluster and AFM as we are integrating new hardware and wish to increase the block and inode size at the same time. Peter Childs Research Storage Expert ITS Research Infrastructure Queen Mary, University of London _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From p.childs at qmul.ac.uk Mon Oct 10 19:11:31 2016 From: p.childs at qmul.ac.uk (Peter Childs) Date: Mon, 10 Oct 2016 18:11:31 +0000 Subject: [gpfsug-discuss] GPFS Upgrade 3.5 -> 4.1 In-Reply-To: References: , Message-ID: <4i99d985lghdmb5sptsfjp2n.1476123091191@email.android.com> So in short we're saying, "mmchfs -V LATEST" increments a version number and allows new features to become possible, it does not start using them straight away. Hence Directories will shrink in 4.1 but you need to run "mmchattr --compact" on all the old ones before anything actually changes. (new ones are fine) increasing the version number makes this possible but it does not actually do it, as doing it would mean walking every directory and updating stuff. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ---- Yuri L Volobuev wrote ---- Correct. mmchfs -V only done quick operations (that can be easily undone if something goes wrong). Essentially the big task here is to increase on-disk file system descriptor version number, to allow using those features that require a higher version. Bigger "conversion"-style tasks belong in mmmigratefs. The only way to increase the inode size and the data block size is to format a new file system. This cannot be done on an existing file system. yuri [Inactive hide details for Jan-Frode Myklebust ---10/10/2016 07:35:48 AM---I've also always been worried about that one, but nev]Jan-Frode Myklebust ---10/10/2016 07:35:48 AM---I've also always been worried about that one, but never experienced it taking any time, I/O or inter From: Jan-Frode Myklebust To: gpfsug main discussion list , Date: 10/10/2016 07:35 AM Subject: Re: [gpfsug-discuss] GPFS Upgrade 3.5 -> 4.1 Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ I've also always been worried about that one, but never experienced it taking any time, I/O or interruption. I've the interpreted it to just start using new features, but not really changing anything with the existing metadata. Things needing on disk changes are probably put in mmmigratefs I have not heard about anything needing mmmigratefs since GPFS v3.3 (fs version 11.03) added fast extended attributes. Would be great to hear otherwize, or confirmations. -jf man. 10. okt. 2016 kl. 14.32 skrev Peter Childs >: We are finishing upgrading our GPFS cluster of around 250 (client) nodes from GPFS 3.5.0.31 to Spectrum Scale 4.1.1.8, and have just about upgraded all the computers. We are looking at running the "mmchfs -V LATEST" step and where wondering how much io this takes and if it was likely to interrupt service? We are looking at upgrading to 4.2 but plan to do that via Multi-cluster and AFM as we are integrating new hardware and wish to increase the block and inode size at the same time. Peter Childs Research Storage Expert ITS Research Infrastructure Queen Mary, University of London _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: graycol.gif URL: From Mark.Bush at siriuscom.com Mon Oct 10 20:56:22 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Mon, 10 Oct 2016 19:56:22 +0000 Subject: [gpfsug-discuss] Hardware refresh Message-ID: <0663AFB2-7F70-4167-A2D1-F96B5394EBD2@siriuscom.com> Have a very old cluster built on IBM X3650?s and DS3500. Need to refresh hardware. Any lessons learned in this process? Is it easiest to just build new cluster and then use AFM? Add to existing cluster then decommission nodes? What is the recommended process for this? Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Mon Oct 10 21:04:36 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Mon, 10 Oct 2016 20:04:36 +0000 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: <0663AFB2-7F70-4167-A2D1-F96B5394EBD2@siriuscom.com> References: <0663AFB2-7F70-4167-A2D1-F96B5394EBD2@siriuscom.com> Message-ID: Hi Mark, The last time we did something like this was 2010 (we?re doing rolling refreshes now), so there are probably lots of better ways to do this than what we did, but we: 1) set up the new hardware 2) created new filesystems (so that we could make adjustments we wanted to make that can only be made at FS creation time) 3) used rsync to make a 1st pass copy of everything 4) coordinated a time with users / groups to do a 2nd rsync when they weren?t active 5) used symbolic links during the transition (i.e. rm -rvf /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) 6) once everybody was migrated, updated the symlinks (i.e. /home became a symlink to /gpfs2/home) HTHAL? Kevin On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com wrote: Have a very old cluster built on IBM X3650?s and DS3500. Need to refresh hardware. Any lessons learned in this process? Is it easiest to just build new cluster and then use AFM? Add to existing cluster then decommission nodes? What is the recommended process for this? Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From usa-principal at gpfsug.org Mon Oct 10 23:59:46 2016 From: usa-principal at gpfsug.org (GPFS UG USA Principal) Date: Mon, 10 Oct 2016 18:59:46 -0400 Subject: [gpfsug-discuss] SC16 Draft Agenda [Save the date: Sunday, November 13th] Message-ID: Hello all, There have been some questions about the Spectrum Scale Users Group event for SC16 and we thought it best to publish our draft agenda and continue to fill it in as details get settled. That way you can make your travel plans and schedules. The date and rough times are set, we may adjust the time range a bit depending upon the number of user talks that people would like to give. So note, yes! we are asking for user talks. Please consider doing this. We always receive positive feedback about talks given by users that describe real world experiences. If any of the user talk slots below are of interest to you, please let us know as soon as you can. We may turn at least one user talk into a panel if we can get enough participants. As always, your feedback is welcome. This is a users group after all. So, here?s what we have planned so far: Date: Sunday November 13th Venue: TBD, but near the convention center in downtown SLC Time: ~12:30p - ~17:30p Agenda: 12:30 - 12:50 Welcome / Overview 12:50 - 13:50 The latest in IBM Spectrum Scale and ESS 13:50 - 14:20 User Talk: Big Data in HPC 14:20 - 14:50 User Talk: Tiering to Object Storage 14:50 - 15:20 - break - 15:20 - 15:50 User Talk: TBD ? volunteers! please! 15:50 - 16:35 Spectrum Scale Enhancements for CORAL 16:35 - 17:20 News from IBM Research 17:20 - 17:30 Closing Best, Kristy & Bob Kristy Kallback-Rose Manager, Research Storage Indiana University -------------- next part -------------- An HTML attachment was scrubbed... URL: From luis.bolinches at fi.ibm.com Tue Oct 11 00:40:54 2016 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Mon, 10 Oct 2016 23:40:54 +0000 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: Message-ID: Hi Creating a new FS sounds like a best way to go. NSDv2 being a very good reason to do so. AFM for migrations is quite good, latest versions allows to use NSD protocol for mounts as well. Olaf did a great job explaining this scenario on the redbook chapter 6 http://www.redbooks.ibm.com/abstracts/sg248254.html?Open -- Cheers > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L wrote: > > Hi Mark, > > The last time we did something like this was 2010 (we?re doing rolling refreshes now), so there are probably lots of better ways to do this than what we did, but we: > > 1) set up the new hardware > 2) created new filesystems (so that we could make adjustments we wanted to make that can only be made at FS creation time) > 3) used rsync to make a 1st pass copy of everything > 4) coordinated a time with users / groups to do a 2nd rsync when they weren?t active > 5) used symbolic links during the transition (i.e. rm -rvf /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) > 6) once everybody was migrated, updated the symlinks (i.e. /home became a symlink to /gpfs2/home) > > HTHAL? > > Kevin > >> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com wrote: >> >> Have a very old cluster built on IBM X3650?s and DS3500. Need to refresh hardware. Any lessons learned in this process? Is it easiest to just build new cluster and then use AFM? Add to existing cluster then decommission nodes? What is the recommended process for this? >> >> >> Mark >> This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. >> >> Sirius Computer Solutions >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and Education > Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 > > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Tue Oct 11 00:45:04 2016 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Mon, 10 Oct 2016 19:45:04 -0400 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: References: Message-ID: Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron On 10/10/16 7:40 PM, Luis Bolinches wrote: > Hi > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > reason to do so. > > AFM for migrations is quite good, latest versions allows to use NSD > protocol for mounts as well. Olaf did a great job explaining this > scenario on the redbook chapter 6 > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > -- > Cheers > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > wrote: > >> Hi Mark, >> >> The last time we did something like this was 2010 (we?re doing rolling >> refreshes now), so there are probably lots of better ways to do this >> than what we did, but we: >> >> 1) set up the new hardware >> 2) created new filesystems (so that we could make adjustments we >> wanted to make that can only be made at FS creation time) >> 3) used rsync to make a 1st pass copy of everything >> 4) coordinated a time with users / groups to do a 2nd rsync when they >> weren?t active >> 5) used symbolic links during the transition (i.e. rm -rvf >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >> 6) once everybody was migrated, updated the symlinks (i.e. /home >> became a symlink to /gpfs2/home) >> >> HTHAL? >> >> Kevin >> >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>> wrote: >>> >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>> refresh hardware. Any lessons learned in this process? Is it >>> easiest to just build new cluster and then use AFM? Add to existing >>> cluster then decommission nodes? What is the recommended process for >>> this? >>> >>> >>> Mark >>> >>> This message (including any attachments) is intended only for the use >>> of the individual or entity to which it is addressed and may contain >>> information that is non-public, proprietary, privileged, >>> confidential, and exempt from disclosure under applicable law. If you >>> are not the intended recipient, you are hereby notified that any use, >>> dissemination, distribution, or copying of this communication is >>> strictly prohibited. This message may be viewed by parties at Sirius >>> Computer Solutions other than those named in the message header. This >>> message does not contain an official representation of Sirius >>> Computer Solutions. If you have received this communication in error, >>> notify Sirius Computer Solutions immediately and (i) destroy this >>> message if a facsimile or (ii) delete this message immediately if >>> this is an electronic communication. Thank you. >>> >>> Sirius Computer Solutions >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and >> Education >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From volobuev at us.ibm.com Tue Oct 11 18:22:17 2016 From: volobuev at us.ibm.com (Yuri L Volobuev) Date: Tue, 11 Oct 2016 10:22:17 -0700 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: References: Message-ID: This depends on the committed cluster version level (minReleaseLevel) and file system format. Since NFSv2 is an on-disk format change, older code wouldn't be able to understand what it is, and thus if there's a possibility of a downlevel node looking at the NSD, the NFSv1 format is going to be used. The code does NSDv1<->NSDv2 conversions under the covers as needed when adding an empty NSD to a file system. I'd strongly recommend getting a fresh start by formatting a new file system. Many things have changed over the course of the last few years. In particular, having a 4K-aligned file system can be a pretty big deal, depending on what hardware one is going to deploy in the future, and this is something that can't be bolted onto an existing file system. Having 4K inodes is very handy for many reasons. New directory format and NSD format changes are attractive, too. And disks generally tend to get larger with time, and at some point you may want to add a disk to an existing storage pool that's larger than the existing allocation map format allows. Obviously, it's more hassle to migrate data to a new file system, as opposed to extending an existing one. In a perfect world, GPFS would offer a conversion tool that seamlessly and robustly converts old file systems, making them as good as new, but in the real world such a tool doesn't exist. Getting a clean slate by formatting a new file system every few years is a good long-term investment of time, although it comes front-loaded with extra work. yuri From: Aaron Knister To: , Date: 10/10/2016 04:45 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron On 10/10/16 7:40 PM, Luis Bolinches wrote: > Hi > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > reason to do so. > > AFM for migrations is quite good, latest versions allows to use NSD > protocol for mounts as well. Olaf did a great job explaining this > scenario on the redbook chapter 6 > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > -- > Cheers > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > wrote: > >> Hi Mark, >> >> The last time we did something like this was 2010 (we?re doing rolling >> refreshes now), so there are probably lots of better ways to do this >> than what we did, but we: >> >> 1) set up the new hardware >> 2) created new filesystems (so that we could make adjustments we >> wanted to make that can only be made at FS creation time) >> 3) used rsync to make a 1st pass copy of everything >> 4) coordinated a time with users / groups to do a 2nd rsync when they >> weren?t active >> 5) used symbolic links during the transition (i.e. rm -rvf >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >> 6) once everybody was migrated, updated the symlinks (i.e. /home >> became a symlink to /gpfs2/home) >> >> HTHAL? >> >> Kevin >> >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>> wrote: >>> >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>> refresh hardware. Any lessons learned in this process? Is it >>> easiest to just build new cluster and then use AFM? Add to existing >>> cluster then decommission nodes? What is the recommended process for >>> this? >>> >>> >>> Mark >>> >>> This message (including any attachments) is intended only for the use >>> of the individual or entity to which it is addressed and may contain >>> information that is non-public, proprietary, privileged, >>> confidential, and exempt from disclosure under applicable law. If you >>> are not the intended recipient, you are hereby notified that any use, >>> dissemination, distribution, or copying of this communication is >>> strictly prohibited. This message may be viewed by parties at Sirius >>> Computer Solutions other than those named in the message header. This >>> message does not contain an official representation of Sirius >>> Computer Solutions. If you have received this communication in error, >>> notify Sirius Computer Solutions immediately and (i) destroy this >>> message if a facsimile or (ii) delete this message immediately if >>> this is an electronic communication. Thank you. >>> >>> Sirius Computer Solutions >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and >> Education >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From Mark.Bush at siriuscom.com Tue Oct 11 20:34:27 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Tue, 11 Oct 2016 19:34:27 +0000 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: References: Message-ID: <9E337B56-A274-41B6-84C2-7084AAB771F4@siriuscom.com> Ok. I think I am hearing that a new cluster with a new FS and copying data from old to new cluster is the best way forward. Thanks everyone for your input. From: on behalf of Yuri L Volobuev Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 12:22 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh This depends on the committed cluster version level (minReleaseLevel) and file system format. Since NFSv2 is an on-disk format change, older code wouldn't be able to understand what it is, and thus if there's a possibility of a downlevel node looking at the NSD, the NFSv1 format is going to be used. The code does NSDv1<->NSDv2 conversions under the covers as needed when adding an empty NSD to a file system. I'd strongly recommend getting a fresh start by formatting a new file system. Many things have changed over the course of the last few years. In particular, having a 4K-aligned file system can be a pretty big deal, depending on what hardware one is going to deploy in the future, and this is something that can't be bolted onto an existing file system. Having 4K inodes is very handy for many reasons. New directory format and NSD format changes are attractive, too. And disks generally tend to get larger with time, and at some point you may want to add a disk to an existing storage pool that's larger than the existing allocation map format allows. Obviously, it's more hassle to migrate data to a new file system, as opposed to extending an existing one. In a perfect world, GPFS would offer a conversion tool that seamlessly and robustly converts old file systems, making them as good as new, but in the real world such a tool doesn't exist. Getting a clean slate by formatting a new file system every few years is a good long-term investment of time, although it comes front-loaded with extra work. yuri [nactive hide details for Aaron Knister ---10/10/2016 04:45:31 PM---Can on]Aaron Knister ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron From: Aaron Knister To: , Date: 10/10/2016 04:45 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron On 10/10/16 7:40 PM, Luis Bolinches wrote: > Hi > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > reason to do so. > > AFM for migrations is quite good, latest versions allows to use NSD > protocol for mounts as well. Olaf did a great job explaining this > scenario on the redbook chapter 6 > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > -- > Cheers > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > wrote: > >> Hi Mark, >> >> The last time we did something like this was 2010 (we?re doing rolling >> refreshes now), so there are probably lots of better ways to do this >> than what we did, but we: >> >> 1) set up the new hardware >> 2) created new filesystems (so that we could make adjustments we >> wanted to make that can only be made at FS creation time) >> 3) used rsync to make a 1st pass copy of everything >> 4) coordinated a time with users / groups to do a 2nd rsync when they >> weren?t active >> 5) used symbolic links during the transition (i.e. rm -rvf >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >> 6) once everybody was migrated, updated the symlinks (i.e. /home >> became a symlink to /gpfs2/home) >> >> HTHAL? >> >> Kevin >> >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>> wrote: >>> >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>> refresh hardware. Any lessons learned in this process? Is it >>> easiest to just build new cluster and then use AFM? Add to existing >>> cluster then decommission nodes? What is the recommended process for >>> this? >>> >>> >>> Mark >>> >>> This message (including any attachments) is intended only for the use >>> of the individual or entity to which it is addressed and may contain >>> information that is non-public, proprietary, privileged, >>> confidential, and exempt from disclosure under applicable law. If you >>> are not the intended recipient, you are hereby notified that any use, >>> dissemination, distribution, or copying of this communication is >>> strictly prohibited. This message may be viewed by parties at Sirius >>> Computer Solutions other than those named in the message header. This >>> message does not contain an official representation of Sirius >>> Computer Solutions. If you have received this communication in error, >>> notify Sirius Computer Solutions immediately and (i) destroy this >>> message if a facsimile or (ii) delete this message immediately if >>> this is an electronic communication. Thank you. >>> >>> Sirius Computer Solutions >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and >> Education >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 106 bytes Desc: image001.gif URL: From makaplan at us.ibm.com Tue Oct 11 20:58:53 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 11 Oct 2016 15:58:53 -0400 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: <9E337B56-A274-41B6-84C2-7084AAB771F4@siriuscom.com> References: <9E337B56-A274-41B6-84C2-7084AAB771F4@siriuscom.com> Message-ID: New FS? Yes there are some good reasons. New cluster? I did not see a compelling argument either way. From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Date: 10/11/2016 03:34 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org Ok. I think I am hearing that a new cluster with a new FS and copying data from old to new cluster is the best way forward. Thanks everyone for your input. From: on behalf of Yuri L Volobuev Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 12:22 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh This depends on the committed cluster version level (minReleaseLevel) and file system format. Since NFSv2 is an on-disk format change, older code wouldn't be able to understand what it is, and thus if there's a possibility of a downlevel node looking at the NSD, the NFSv1 format is going to be used. The code does NSDv1<->NSDv2 conversions under the covers as needed when adding an empty NSD to a file system. I'd strongly recommend getting a fresh start by formatting a new file system. Many things have changed over the course of the last few years. In particular, having a 4K-aligned file system can be a pretty big deal, depending on what hardware one is going to deploy in the future, and this is something that can't be bolted onto an existing file system. Having 4K inodes is very handy for many reasons. New directory format and NSD format changes are attractive, too. And disks generally tend to get larger with time, and at some point you may want to add a disk to an existing storage pool that's larger than the existing allocation map format allows. Obviously, it's more hassle to migrate data to a new file system, as opposed to extending an existing one. In a perfect world, GPFS would offer a conversion tool that seamlessly and robustly converts old file systems, making them as good as new, but in the real world such a tool doesn't exist. Getting a clean slate by formatting a new file system every few years is a good long-term investment of time, although it comes front-loaded with extra work. yuri Aaron Knister ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron From: Aaron Knister To: , Date: 10/10/2016 04:45 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron On 10/10/16 7:40 PM, Luis Bolinches wrote: > Hi > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > reason to do so. > > AFM for migrations is quite good, latest versions allows to use NSD > protocol for mounts as well. Olaf did a great job explaining this > scenario on the redbook chapter 6 > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > -- > Cheers > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > wrote: > >> Hi Mark, >> >> The last time we did something like this was 2010 (we?re doing rolling >> refreshes now), so there are probably lots of better ways to do this >> than what we did, but we: >> >> 1) set up the new hardware >> 2) created new filesystems (so that we could make adjustments we >> wanted to make that can only be made at FS creation time) >> 3) used rsync to make a 1st pass copy of everything >> 4) coordinated a time with users / groups to do a 2nd rsync when they >> weren?t active >> 5) used symbolic links during the transition (i.e. rm -rvf >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >> 6) once everybody was migrated, updated the symlinks (i.e. /home >> became a symlink to /gpfs2/home) >> >> HTHAL? >> >> Kevin >> >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>> wrote: >>> >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>> refresh hardware. Any lessons learned in this process? Is it >>> easiest to just build new cluster and then use AFM? Add to existing >>> cluster then decommission nodes? What is the recommended process for >>> this? >>> >>> >>> Mark >>> >>> This message (including any attachments) is intended only for the use >>> of the individual or entity to which it is addressed and may contain >>> information that is non-public, proprietary, privileged, >>> confidential, and exempt from disclosure under applicable law. If you >>> are not the intended recipient, you are hereby notified that any use, >>> dissemination, distribution, or copying of this communication is >>> strictly prohibited. This message may be viewed by parties at Sirius >>> Computer Solutions other than those named in the message header. This >>> message does not contain an official representation of Sirius >>> Computer Solutions. If you have received this communication in error, >>> notify Sirius Computer Solutions immediately and (i) destroy this >>> message if a facsimile or (ii) delete this message immediately if >>> this is an electronic communication. Thank you. >>> >>> Sirius Computer Solutions >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and >> Education >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 106 bytes Desc: not available URL: From p.childs at qmul.ac.uk Tue Oct 11 22:25:50 2016 From: p.childs at qmul.ac.uk (Peter Childs) Date: Tue, 11 Oct 2016 21:25:50 +0000 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: References: <9E337B56-A274-41B6-84C2-7084AAB771F4@siriuscom.com>, Message-ID: <7lb12taeugiu5fh1vd718dp5.1476220357109@email.android.com> My reading is, If you are running a small cluster with tie-breaker disks and your wanting to change the manager servers, or you want to switch to using the new config management method in v4 then new cluster, and use multicluster to upgrade. Otherwise just use a new Filesystem within the old cluster. But I'm interested to hear otherwise, as I'm about to embark on this myself. I note you can switch an old cluster but need to shutdown to do so. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ---- Marc A Kaplan wrote ---- New FS? Yes there are some good reasons. New cluster? I did not see a compelling argument either way. From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Date: 10/11/2016 03:34 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Ok. I think I am hearing that a new cluster with a new FS and copying data from old to new cluster is the best way forward. Thanks everyone for your input. From: on behalf of Yuri L Volobuev Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 12:22 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh This depends on the committed cluster version level (minReleaseLevel) and file system format. Since NFSv2 is an on-disk format change, older code wouldn't be able to understand what it is, and thus if there's a possibility of a downlevel node looking at the NSD, the NFSv1 format is going to be used. The code does NSDv1<->NSDv2 conversions under the covers as needed when adding an empty NSD to a file system. I'd strongly recommend getting a fresh start by formatting a new file system. Many things have changed over the course of the last few years. In particular, having a 4K-aligned file system can be a pretty big deal, depending on what hardware one is going to deploy in the future, and this is something that can't be bolted onto an existing file system. Having 4K inodes is very handy for many reasons. New directory format and NSD format changes are attractive, too. And disks generally tend to get larger with time, and at some point you may want to add a disk to an existing storage pool that's larger than the existing allocation map format allows. Obviously, it's more hassle to migrate data to a new file system, as opposed to extending an existing one. In a perfect world, GPFS would offer a conversion tool that seamlessly and robustly converts old file systems, making them as good as new, but in the real world such a tool doesn't exist. Getting a clean slate by formatting a new file system every few years is a good long-term investment of time, although it comes front-loaded with extra work. yuri [nactive hide details for Aaron Knister ---10/10/2016 04:45:31 PM---Can on]Aaron Knister ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron From: Aaron Knister To: , Date: 10/10/2016 04:45 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron On 10/10/16 7:40 PM, Luis Bolinches wrote: > Hi > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > reason to do so. > > AFM for migrations is quite good, latest versions allows to use NSD > protocol for mounts as well. Olaf did a great job explaining this > scenario on the redbook chapter 6 > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > -- > Cheers > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > wrote: > >> Hi Mark, >> >> The last time we did something like this was 2010 (we?re doing rolling >> refreshes now), so there are probably lots of better ways to do this >> than what we did, but we: >> >> 1) set up the new hardware >> 2) created new filesystems (so that we could make adjustments we >> wanted to make that can only be made at FS creation time) >> 3) used rsync to make a 1st pass copy of everything >> 4) coordinated a time with users / groups to do a 2nd rsync when they >> weren?t active >> 5) used symbolic links during the transition (i.e. rm -rvf >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >> 6) once everybody was migrated, updated the symlinks (i.e. /home >> became a symlink to /gpfs2/home) >> >> HTHAL? >> >> Kevin >> >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>> wrote: >>> >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>> refresh hardware. Any lessons learned in this process? Is it >>> easiest to just build new cluster and then use AFM? Add to existing >>> cluster then decommission nodes? What is the recommended process for >>> this? >>> >>> >>> Mark >>> >>> This message (including any attachments) is intended only for the use >>> of the individual or entity to which it is addressed and may contain >>> information that is non-public, proprietary, privileged, >>> confidential, and exempt from disclosure under applicable law. If you >>> are not the intended recipient, you are hereby notified that any use, >>> dissemination, distribution, or copying of this communication is >>> strictly prohibited. This message may be viewed by parties at Sirius >>> Computer Solutions other than those named in the message header. This >>> message does not contain an official representation of Sirius >>> Computer Solutions. If you have received this communication in error, >>> notify Sirius Computer Solutions immediately and (i) destroy this >>> message if a facsimile or (ii) delete this message immediately if >>> this is an electronic communication. Thank you. >>> >>> Sirius Computer Solutions >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and >> Education >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00001.gif Type: image/gif Size: 106 bytes Desc: ATT00001.gif URL: From aaron.s.knister at nasa.gov Tue Oct 11 23:41:36 2016 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 11 Oct 2016 18:41:36 -0400 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: References: Message-ID: Thanks Yuri. I'm asking for my own purposes but I think it's still relevant here: we're still at GPFS 3.5 and will be adding dataOnly NSDs with 4K sectors in the near future. We're planning to update to 4.1 before we format these NSDs, though. If I understand you correctly we can't bring these 4K NSDv2 NSDs into a filesystem with 512b-based NSDv1 NSDs? That's a pretty big deal :( Reformatting every few years with 10's of petabytes of data is not realistic for us (it would take years to move the data around). It also goes against my personal preachings about GPFS's storage virtualization capabilities: the ability to perform upgrades/make underlying storage infrastructure changes with behind-the-scenes data migration, eliminating much of the manual hassle of storage administrators doing rsync dances. I guess it's RFE time? It also seems as though AFM could help with automating the migration, although many of our filesystems do not have filesets on them so we would have to re-think how we lay out our filesystems. This is also curious to me with IBM pitching GPFS as a filesystem for cloud services (the cloud *never* goes down, right?). Granted I believe this pitch started after the NSDv2 format was defined, but if somebody is building a large cloud with GPFS as the underlying filesystem for an object or an image store one might think the idea of having to re-format the filesystem to gain access to critical new features is inconsistent with this pitch. It would be hugely impactful. Just my $.02. As you can tell, I'm frustrated there's no online conversion tool :) Not that there couldn't be... you all are brilliant developers. -Aaron On 10/11/16 1:22 PM, Yuri L Volobuev wrote: > This depends on the committed cluster version level (minReleaseLevel) > and file system format. Since NFSv2 is an on-disk format change, older > code wouldn't be able to understand what it is, and thus if there's a > possibility of a downlevel node looking at the NSD, the NFSv1 format is > going to be used. The code does NSDv1<->NSDv2 conversions under the > covers as needed when adding an empty NSD to a file system. > > I'd strongly recommend getting a fresh start by formatting a new file > system. Many things have changed over the course of the last few years. > In particular, having a 4K-aligned file system can be a pretty big deal, > depending on what hardware one is going to deploy in the future, and > this is something that can't be bolted onto an existing file system. > Having 4K inodes is very handy for many reasons. New directory format > and NSD format changes are attractive, too. And disks generally tend to > get larger with time, and at some point you may want to add a disk to an > existing storage pool that's larger than the existing allocation map > format allows. Obviously, it's more hassle to migrate data to a new file > system, as opposed to extending an existing one. In a perfect world, > GPFS would offer a conversion tool that seamlessly and robustly converts > old file systems, making them as good as new, but in the real world such > a tool doesn't exist. Getting a clean slate by formatting a new file > system every few years is a good long-term investment of time, although > it comes front-loaded with extra work. > > yuri > > Inactive hide details for Aaron Knister ---10/10/2016 04:45:31 PM---Can > one format NSDv2 NSDs and put them in a filesystem withAaron Knister > ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a > filesystem with NSDv1 NSD's? -Aaron > > From: Aaron Knister > To: , > Date: 10/10/2016 04:45 PM > Subject: Re: [gpfsug-discuss] Hardware refresh > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > ------------------------------------------------------------------------ > > > > Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? > > -Aaron > > On 10/10/16 7:40 PM, Luis Bolinches wrote: >> Hi >> >> Creating a new FS sounds like a best way to go. NSDv2 being a very good >> reason to do so. >> >> AFM for migrations is quite good, latest versions allows to use NSD >> protocol for mounts as well. Olaf did a great job explaining this >> scenario on the redbook chapter 6 >> >> http://www.redbooks.ibm.com/abstracts/sg248254.html?Open >> >> -- >> Cheers >> >> On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L >> > > wrote: >> >>> Hi Mark, >>> >>> The last time we did something like this was 2010 (we?re doing rolling >>> refreshes now), so there are probably lots of better ways to do this >>> than what we did, but we: >>> >>> 1) set up the new hardware >>> 2) created new filesystems (so that we could make adjustments we >>> wanted to make that can only be made at FS creation time) >>> 3) used rsync to make a 1st pass copy of everything >>> 4) coordinated a time with users / groups to do a 2nd rsync when they >>> weren?t active >>> 5) used symbolic links during the transition (i.e. rm -rvf >>> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >>> 6) once everybody was migrated, updated the symlinks (i.e. /home >>> became a symlink to /gpfs2/home) >>> >>> HTHAL? >>> >>> Kevin >>> >>>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>>> wrote: >>>> >>>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>>> refresh hardware. Any lessons learned in this process? Is it >>>> easiest to just build new cluster and then use AFM? Add to existing >>>> cluster then decommission nodes? What is the recommended process for >>>> this? >>>> >>>> >>>> Mark >>>> >>>> This message (including any attachments) is intended only for the use >>>> of the individual or entity to which it is addressed and may contain >>>> information that is non-public, proprietary, privileged, >>>> confidential, and exempt from disclosure under applicable law. If you >>>> are not the intended recipient, you are hereby notified that any use, >>>> dissemination, distribution, or copying of this communication is >>>> strictly prohibited. This message may be viewed by parties at Sirius >>>> Computer Solutions other than those named in the message header. This >>>> message does not contain an official representation of Sirius >>>> Computer Solutions. If you have received this communication in error, >>>> notify Sirius Computer Solutions immediately and (i) destroy this >>>> message if a facsimile or (ii) delete this message immediately if >>>> this is an electronic communication. Thank you. >>>> >>>> Sirius Computer Solutions >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> ? >>> Kevin Buterbaugh - Senior System Administrator >>> Vanderbilt University - Advanced Computing Center for Research and >>> Education >>> Kevin.Buterbaugh at vanderbilt.edu >>> - (615)875-9633 >>> >>> >>> >> >> Ellei edell? ole toisin mainittu: / Unless stated otherwise above: >> Oy IBM Finland Ab >> PL 265, 00101 Helsinki, Finland >> Business ID, Y-tunnus: 0195876-3 >> Registered in Finland >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From aaron.s.knister at nasa.gov Wed Oct 12 00:57:38 2016 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 11 Oct 2016 19:57:38 -0400 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: References: Message-ID: <63bd99ad-f684-d1bb-371f-e2c21849ff29@nasa.gov> I think I was a little quick to the trigger. I re-read your last mail after doing some testing and understand it differently. I was wrong about my interpretation-- you can add 4K NSDv2 formatted NSDs to a filesystem previously created with NSDv1 NSDs assuming, as you say, the minReleaseLevel and filesystem version are high enough. That negates about half of my last e-mail. The fs still doesn't show as 4K aligned: loressd01:~ # /usr/lpp/mmfs/bin/mmlsfs tnb4k --is4KAligned flag value description ------------------- ------------------------ ----------------------------------- --is4KAligned No is4KAligned? but *shrug* most of the I/O to these disks should be 1MB anyway. If somebody is pounding the FS with smaller than 4K I/O they're gonna get a talkin' to. -Aaron On 10/11/16 6:41 PM, Aaron Knister wrote: > Thanks Yuri. > > I'm asking for my own purposes but I think it's still relevant here: > we're still at GPFS 3.5 and will be adding dataOnly NSDs with 4K sectors > in the near future. We're planning to update to 4.1 before we format > these NSDs, though. If I understand you correctly we can't bring these > 4K NSDv2 NSDs into a filesystem with 512b-based NSDv1 NSDs? That's a > pretty big deal :( > > Reformatting every few years with 10's of petabytes of data is not > realistic for us (it would take years to move the data around). It also > goes against my personal preachings about GPFS's storage virtualization > capabilities: the ability to perform upgrades/make underlying storage > infrastructure changes with behind-the-scenes data migration, > eliminating much of the manual hassle of storage administrators doing > rsync dances. I guess it's RFE time? It also seems as though AFM could > help with automating the migration, although many of our filesystems do > not have filesets on them so we would have to re-think how we lay out > our filesystems. > > This is also curious to me with IBM pitching GPFS as a filesystem for > cloud services (the cloud *never* goes down, right?). Granted I believe > this pitch started after the NSDv2 format was defined, but if somebody > is building a large cloud with GPFS as the underlying filesystem for an > object or an image store one might think the idea of having to re-format > the filesystem to gain access to critical new features is inconsistent > with this pitch. It would be hugely impactful. Just my $.02. > > As you can tell, I'm frustrated there's no online conversion tool :) Not > that there couldn't be... you all are brilliant developers. > > -Aaron > > On 10/11/16 1:22 PM, Yuri L Volobuev wrote: >> This depends on the committed cluster version level (minReleaseLevel) >> and file system format. Since NFSv2 is an on-disk format change, older >> code wouldn't be able to understand what it is, and thus if there's a >> possibility of a downlevel node looking at the NSD, the NFSv1 format is >> going to be used. The code does NSDv1<->NSDv2 conversions under the >> covers as needed when adding an empty NSD to a file system. >> >> I'd strongly recommend getting a fresh start by formatting a new file >> system. Many things have changed over the course of the last few years. >> In particular, having a 4K-aligned file system can be a pretty big deal, >> depending on what hardware one is going to deploy in the future, and >> this is something that can't be bolted onto an existing file system. >> Having 4K inodes is very handy for many reasons. New directory format >> and NSD format changes are attractive, too. And disks generally tend to >> get larger with time, and at some point you may want to add a disk to an >> existing storage pool that's larger than the existing allocation map >> format allows. Obviously, it's more hassle to migrate data to a new file >> system, as opposed to extending an existing one. In a perfect world, >> GPFS would offer a conversion tool that seamlessly and robustly converts >> old file systems, making them as good as new, but in the real world such >> a tool doesn't exist. Getting a clean slate by formatting a new file >> system every few years is a good long-term investment of time, although >> it comes front-loaded with extra work. >> >> yuri >> >> Inactive hide details for Aaron Knister ---10/10/2016 04:45:31 PM---Can >> one format NSDv2 NSDs and put them in a filesystem withAaron Knister >> ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a >> filesystem with NSDv1 NSD's? -Aaron >> >> From: Aaron Knister >> To: , >> Date: 10/10/2016 04:45 PM >> Subject: Re: [gpfsug-discuss] Hardware refresh >> Sent by: gpfsug-discuss-bounces at spectrumscale.org >> >> ------------------------------------------------------------------------ >> >> >> >> Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? >> >> -Aaron >> >> On 10/10/16 7:40 PM, Luis Bolinches wrote: >>> Hi >>> >>> Creating a new FS sounds like a best way to go. NSDv2 being a very good >>> reason to do so. >>> >>> AFM for migrations is quite good, latest versions allows to use NSD >>> protocol for mounts as well. Olaf did a great job explaining this >>> scenario on the redbook chapter 6 >>> >>> http://www.redbooks.ibm.com/abstracts/sg248254.html?Open >>> >>> -- >>> Cheers >>> >>> On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L >>> >> > wrote: >>> >>>> Hi Mark, >>>> >>>> The last time we did something like this was 2010 (we?re doing rolling >>>> refreshes now), so there are probably lots of better ways to do this >>>> than what we did, but we: >>>> >>>> 1) set up the new hardware >>>> 2) created new filesystems (so that we could make adjustments we >>>> wanted to make that can only be made at FS creation time) >>>> 3) used rsync to make a 1st pass copy of everything >>>> 4) coordinated a time with users / groups to do a 2nd rsync when they >>>> weren?t active >>>> 5) used symbolic links during the transition (i.e. rm -rvf >>>> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >>>> 6) once everybody was migrated, updated the symlinks (i.e. /home >>>> became a symlink to /gpfs2/home) >>>> >>>> HTHAL? >>>> >>>> Kevin >>>> >>>>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>>>> wrote: >>>>> >>>>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>>>> refresh hardware. Any lessons learned in this process? Is it >>>>> easiest to just build new cluster and then use AFM? Add to existing >>>>> cluster then decommission nodes? What is the recommended process for >>>>> this? >>>>> >>>>> >>>>> Mark >>>>> >>>>> This message (including any attachments) is intended only for the use >>>>> of the individual or entity to which it is addressed and may contain >>>>> information that is non-public, proprietary, privileged, >>>>> confidential, and exempt from disclosure under applicable law. If you >>>>> are not the intended recipient, you are hereby notified that any use, >>>>> dissemination, distribution, or copying of this communication is >>>>> strictly prohibited. This message may be viewed by parties at Sirius >>>>> Computer Solutions other than those named in the message header. This >>>>> message does not contain an official representation of Sirius >>>>> Computer Solutions. If you have received this communication in error, >>>>> notify Sirius Computer Solutions immediately and (i) destroy this >>>>> message if a facsimile or (ii) delete this message immediately if >>>>> this is an electronic communication. Thank you. >>>>> >>>>> Sirius Computer Solutions >>>>> _______________________________________________ >>>>> gpfsug-discuss mailing list >>>>> gpfsug-discuss at spectrumscale.org >>>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>>> >>>> ? >>>> Kevin Buterbaugh - Senior System Administrator >>>> Vanderbilt University - Advanced Computing Center for Research and >>>> Education >>>> Kevin.Buterbaugh at vanderbilt.edu >>>> - (615)875-9633 >>>> >>>> >>>> >>> >>> Ellei edell? ole toisin mainittu: / Unless stated otherwise above: >>> Oy IBM Finland Ab >>> PL 265, 00101 Helsinki, Finland >>> Business ID, Y-tunnus: 0195876-3 >>> Registered in Finland >>> >>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Mark.Bush at siriuscom.com Wed Oct 12 01:40:39 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Wed, 12 Oct 2016 00:40:39 +0000 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: References: <9E337B56-A274-41B6-84C2-7084AAB771F4@siriuscom.com> Message-ID: Only compelling reason for new cluster would be old hardware is EOL or no longer want to pay maintenance on it. From: on behalf of Marc A Kaplan Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 2:58 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh New FS? Yes there are some good reasons. New cluster? I did not see a compelling argument either way. From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Date: 10/11/2016 03:34 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Ok. I think I am hearing that a new cluster with a new FS and copying data from old to new cluster is the best way forward. Thanks everyone for your input. From: on behalf of Yuri L Volobuev Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 12:22 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh This depends on the committed cluster version level (minReleaseLevel) and file system format. Since NFSv2 is an on-disk format change, older code wouldn't be able to understand what it is, and thus if there's a possibility of a downlevel node looking at the NSD, the NFSv1 format is going to be used. The code does NSDv1<->NSDv2 conversions under the covers as needed when adding an empty NSD to a file system. I'd strongly recommend getting a fresh start by formatting a new file system. Many things have changed over the course of the last few years. In particular, having a 4K-aligned file system can be a pretty big deal, depending on what hardware one is going to deploy in the future, and this is something that can't be bolted onto an existing file system. Having 4K inodes is very handy for many reasons. New directory format and NSD format changes are attractive, too. And disks generally tend to get larger with time, and at some point you may want to add a disk to an existing storage pool that's larger than the existing allocation map format allows. Obviously, it's more hassle to migrate data to a new file system, as opposed to extending an existing one. In a perfect world, GPFS would offer a conversion tool that seamlessly and robustly converts old file systems, making them as good as new, but in the real world such a tool doesn't exist. Getting a clean slate by formatting a new file system every few years is a good long-term investment of time, although it comes front-loaded with extra work. yuri [active hide details for Aaron Knister ---10/10/2016 04:45:31 PM---Can on]Aaron Knister ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron From: Aaron Knister To: , Date: 10/10/2016 04:45 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron On 10/10/16 7:40 PM, Luis Bolinches wrote: > Hi > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > reason to do so. > > AFM for migrations is quite good, latest versions allows to use NSD > protocol for mounts as well. Olaf did a great job explaining this > scenario on the redbook chapter 6 > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > -- > Cheers > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > wrote: > >> Hi Mark, >> >> The last time we did something like this was 2010 (we?re doing rolling >> refreshes now), so there are probably lots of better ways to do this >> than what we did, but we: >> >> 1) set up the new hardware >> 2) created new filesystems (so that we could make adjustments we >> wanted to make that can only be made at FS creation time) >> 3) used rsync to make a 1st pass copy of everything >> 4) coordinated a time with users / groups to do a 2nd rsync when they >> weren?t active >> 5) used symbolic links during the transition (i.e. rm -rvf >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >> 6) once everybody was migrated, updated the symlinks (i.e. /home >> became a symlink to /gpfs2/home) >> >> HTHAL? >> >> Kevin >> >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>> wrote: >>> >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>> refresh hardware. Any lessons learned in this process? Is it >>> easiest to just build new cluster and then use AFM? Add to existing >>> cluster then decommission nodes? What is the recommended process for >>> this? >>> >>> >>> Mark >>> >>> This message (including any attachments) is intended only for the use >>> of the individual or entity to which it is addressed and may contain >>> information that is non-public, proprietary, privileged, >>> confidential, and exempt from disclosure under applicable law. If you >>> are not the intended recipient, you are hereby notified that any use, >>> dissemination, distribution, or copying of this communication is >>> strictly prohibited. This message may be viewed by parties at Sirius >>> Computer Solutions other than those named in the message header. This >>> message does not contain an official representation of Sirius >>> Computer Solutions. If you have received this communication in error, >>> notify Sirius Computer Solutions immediately and (i) destroy this >>> message if a facsimile or (ii) delete this message immediately if >>> this is an electronic communication. Thank you. >>> >>> Sirius Computer Solutions >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and >> Education >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 107 bytes Desc: image001.gif URL: From aaron.s.knister at nasa.gov Wed Oct 12 01:50:38 2016 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 11 Oct 2016 20:50:38 -0400 Subject: [gpfsug-discuss] 4K sector NSD support (was: Hardware refresh) In-Reply-To: <63bd99ad-f684-d1bb-371f-e2c21849ff29@nasa.gov> References: <63bd99ad-f684-d1bb-371f-e2c21849ff29@nasa.gov> Message-ID: Yuri, (Sorry for being somewhat spammy) I now understand the limitation after some more testing (I'm a hands-on learner, can you tell?). Given the right code/cluster/fs version levels I can add 4K dataOnly NSDv2 NSDs to a filesystem created with NSDv1 NSDs. What I can't do is seemingly add any metadataOnly or dataAndMetadata 4K luns to an fs that is not 4K aligned which I assume would be any fs originally created with NSDv1 LUNs. It seems possible to move all data away from NSDv1 LUNS in a filesystem behind-the-scenes using GPFS migration tools, and move the data to NSDv2 LUNs. In this case I believe what's missing is a tool to convert just the metadata structures to be 4K aligned since the data would already on 4k-based NSDv2 LUNS, is that the case? I'm trying to figure out what exactly I'm asking for in an RFE. -Aaron On 10/11/16 7:57 PM, Aaron Knister wrote: > I think I was a little quick to the trigger. I re-read your last mail > after doing some testing and understand it differently. I was wrong > about my interpretation-- you can add 4K NSDv2 formatted NSDs to a > filesystem previously created with NSDv1 NSDs assuming, as you say, the. > minReleaseLevel and filesystem version are high enough. That negates > about half of my last e-mail. The fs still doesn't show as 4K aligned: > > loressd01:~ # /usr/lpp/mmfs/bin/mmlsfs tnb4k --is4KAligned > flag value description > ------------------- ------------------------ > ----------------------------------- > --is4KAligned No is4KAligned? > > but *shrug* most of the I/O to these disks should be 1MB anyway. If > somebody is pounding the FS with smaller than 4K I/O they're gonna get a > talkin' to. > > -Aaron > > On 10/11/16 6:41 PM, Aaron Knister wrote: >> Thanks Yuri. >> >> I'm asking for my own purposes but I think it's still relevant here: >> we're still at GPFS 3.5 and will be adding dataOnly NSDs with 4K sectors >> in the near future. We're planning to update to 4.1 before we format >> these NSDs, though. If I understand you correctly we can't bring these >> 4K NSDv2 NSDs into a filesystem with 512b-based NSDv1 NSDs? That's a >> pretty big deal :( >> >> Reformatting every few years with 10's of petabytes of data is not >> realistic for us (it would take years to move the data around). It also >> goes against my personal preachings about GPFS's storage virtualization >> capabilities: the ability to perform upgrades/make underlying storage >> infrastructure changes with behind-the-scenes data migration, >> eliminating much of the manual hassle of storage administrators doing >> rsync dances. I guess it's RFE time? It also seems as though AFM could >> help with automating the migration, although many of our filesystems do >> not have filesets on them so we would have to re-think how we lay out >> our filesystems. >> >> This is also curious to me with IBM pitching GPFS as a filesystem for >> cloud services (the cloud *never* goes down, right?). Granted I believe >> this pitch started after the NSDv2 format was defined, but if somebody >> is building a large cloud with GPFS as the underlying filesystem for an >> object or an image store one might think the idea of having to re-format >> the filesystem to gain access to critical new features is inconsistent >> with this pitch. It would be hugely impactful. Just my $.02. >> >> As you can tell, I'm frustrated there's no online conversion tool :) Not >> that there couldn't be... you all are brilliant developers. >> >> -Aaron >> >> On 10/11/16 1:22 PM, Yuri L Volobuev wrote: >>> This depends on the committed cluster version level (minReleaseLevel) >>> and file system format. Since NFSv2 is an on-disk format change, older >>> code wouldn't be able to understand what it is, and thus if there's a >>> possibility of a downlevel node looking at the NSD, the NFSv1 format is >>> going to be used. The code does NSDv1<->NSDv2 conversions under the >>> covers as needed when adding an empty NSD to a file system. >>> >>> I'd strongly recommend getting a fresh start by formatting a new file >>> system. Many things have changed over the course of the last few years. >>> In particular, having a 4K-aligned file system can be a pretty big deal, >>> depending on what hardware one is going to deploy in the future, and >>> this is something that can't be bolted onto an existing file system. >>> Having 4K inodes is very handy for many reasons. New directory format >>> and NSD format changes are attractive, too. And disks generally tend to >>> get larger with time, and at some point you may want to add a disk to an >>> existing storage pool that's larger than the existing allocation map >>> format allows. Obviously, it's more hassle to migrate data to a new file >>> system, as opposed to extending an existing one. In a perfect world, >>> GPFS would offer a conversion tool that seamlessly and robustly converts >>> old file systems, making them as good as new, but in the real world such >>> a tool doesn't exist. Getting a clean slate by formatting a new file >>> system every few years is a good long-term investment of time, although >>> it comes front-loaded with extra work. >>> >>> yuri >>> >>> Inactive hide details for Aaron Knister ---10/10/2016 04:45:31 PM---Can >>> one format NSDv2 NSDs and put them in a filesystem withAaron Knister >>> ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a >>> filesystem with NSDv1 NSD's? -Aaron >>> >>> From: Aaron Knister >>> To: , >>> Date: 10/10/2016 04:45 PM >>> Subject: Re: [gpfsug-discuss] Hardware refresh >>> Sent by: gpfsug-discuss-bounces at spectrumscale.org >>> >>> ------------------------------------------------------------------------ >>> >>> >>> >>> Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? >>> >>> -Aaron >>> >>> On 10/10/16 7:40 PM, Luis Bolinches wrote: >>>> Hi >>>> >>>> Creating a new FS sounds like a best way to go. NSDv2 being a very good >>>> reason to do so. >>>> >>>> AFM for migrations is quite good, latest versions allows to use NSD >>>> protocol for mounts as well. Olaf did a great job explaining this >>>> scenario on the redbook chapter 6 >>>> >>>> http://www.redbooks.ibm.com/abstracts/sg248254.html?Open >>>> >>>> -- >>>> Cheers >>>> >>>> On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L >>>> >>> > wrote: >>>> >>>>> Hi Mark, >>>>> >>>>> The last time we did something like this was 2010 (we?re doing rolling >>>>> refreshes now), so there are probably lots of better ways to do this >>>>> than what we did, but we: >>>>> >>>>> 1) set up the new hardware >>>>> 2) created new filesystems (so that we could make adjustments we >>>>> wanted to make that can only be made at FS creation time) >>>>> 3) used rsync to make a 1st pass copy of everything >>>>> 4) coordinated a time with users / groups to do a 2nd rsync when they >>>>> weren?t active >>>>> 5) used symbolic links during the transition (i.e. rm -rvf >>>>> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >>>>> 6) once everybody was migrated, updated the symlinks (i.e. /home >>>>> became a symlink to /gpfs2/home) >>>>> >>>>> HTHAL? >>>>> >>>>> Kevin >>>>> >>>>>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>>>>> wrote: >>>>>> >>>>>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>>>>> refresh hardware. Any lessons learned in this process? Is it >>>>>> easiest to just build new cluster and then use AFM? Add to existing >>>>>> cluster then decommission nodes? What is the recommended process for >>>>>> this? >>>>>> >>>>>> >>>>>> Mark >>>>>> >>>>>> This message (including any attachments) is intended only for the use >>>>>> of the individual or entity to which it is addressed and may contain >>>>>> information that is non-public, proprietary, privileged, >>>>>> confidential, and exempt from disclosure under applicable law. If you >>>>>> are not the intended recipient, you are hereby notified that any use, >>>>>> dissemination, distribution, or copying of this communication is >>>>>> strictly prohibited. This message may be viewed by parties at Sirius >>>>>> Computer Solutions other than those named in the message header. This >>>>>> message does not contain an official representation of Sirius >>>>>> Computer Solutions. If you have received this communication in error, >>>>>> notify Sirius Computer Solutions immediately and (i) destroy this >>>>>> message if a facsimile or (ii) delete this message immediately if >>>>>> this is an electronic communication. Thank you. >>>>>> >>>>>> Sirius Computer Solutions >>>>>> _______________________________________________ >>>>>> gpfsug-discuss mailing list >>>>>> gpfsug-discuss at spectrumscale.org >>>>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>>>> >>>>> ? >>>>> Kevin Buterbaugh - Senior System Administrator >>>>> Vanderbilt University - Advanced Computing Center for Research and >>>>> Education >>>>> Kevin.Buterbaugh at vanderbilt.edu >>>>> - (615)875-9633 >>>>> >>>>> >>>>> >>>> >>>> Ellei edell? ole toisin mainittu: / Unless stated otherwise above: >>>> Oy IBM Finland Ab >>>> PL 265, 00101 Helsinki, Finland >>>> Business ID, Y-tunnus: 0195876-3 >>>> Registered in Finland >>>> >>>> >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> >>> >>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From ulmer at ulmer.org Wed Oct 12 02:29:36 2016 From: ulmer at ulmer.org (Stephen Ulmer) Date: Tue, 11 Oct 2016 21:29:36 -0400 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: References: <9E337B56-A274-41B6-84C2-7084AAB771F4@siriuscom.com> Message-ID: <279BAB6D-21E6-4EC1-98CD-CA409A24F8F6@ulmer.org> I think that the OP was asking why not expand the existing cluster with the new hardware, and just make a new FS? I?ve not tried to make a cluster talk AFM to itself yet. If that?s impossible, then there?s one good reason to make a new cluster (to use AFM for migration). Liberty, -- Stephen > On Oct 11, 2016, at 8:40 PM, Mark.Bush at siriuscom.com wrote: > > Only compelling reason for new cluster would be old hardware is EOL or no longer want to pay maintenance on it. > > From: > on behalf of Marc A Kaplan > > Reply-To: gpfsug main discussion list > > Date: Tuesday, October 11, 2016 at 2:58 PM > To: gpfsug main discussion list > > Subject: Re: [gpfsug-discuss] Hardware refresh > > New FS? Yes there are some good reasons. > New cluster? I did not see a compelling argument either way. > > > > From: "Mark.Bush at siriuscom.com " > > To: gpfsug main discussion list > > Date: 10/11/2016 03:34 PM > Subject: Re: [gpfsug-discuss] Hardware refresh > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > Ok. I think I am hearing that a new cluster with a new FS and copying data from old to new cluster is the best way forward. Thanks everyone for your input. > > From: > on behalf of Yuri L Volobuev > > Reply-To: gpfsug main discussion list > > Date: Tuesday, October 11, 2016 at 12:22 PM > To: gpfsug main discussion list > > Subject: Re: [gpfsug-discuss] Hardware refresh > > This depends on the committed cluster version level (minReleaseLevel) and file system format. Since NFSv2 is an on-disk format change, older code wouldn't be able to understand what it is, and thus if there's a possibility of a downlevel node looking at the NSD, the NFSv1 format is going to be used. The code does NSDv1<->NSDv2 conversions under the covers as needed when adding an empty NSD to a file system. > > I'd strongly recommend getting a fresh start by formatting a new file system. Many things have changed over the course of the last few years. In particular, having a 4K-aligned file system can be a pretty big deal, depending on what hardware one is going to deploy in the future, and this is something that can't be bolted onto an existing file system. Having 4K inodes is very handy for many reasons. New directory format and NSD format changes are attractive, too. And disks generally tend to get larger with time, and at some point you may want to add a disk to an existing storage pool that's larger than the existing allocation map format allows. Obviously, it's more hassle to migrate data to a new file system, as opposed to extending an existing one. In a perfect world, GPFS would offer a conversion tool that seamlessly and robustly converts old file systems, making them as good as new, but in the real world such a tool doesn't exist. Getting a clean slate by formatting a new file system every few years is a good long-term investment of time, although it comes front-loaded with extra work. > > yuri > > Aaron Knister ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron > > From: Aaron Knister > > To: >, > Date: 10/10/2016 04:45 PM > Subject: Re: [gpfsug-discuss] Hardware refresh > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > > Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? > > -Aaron > > On 10/10/16 7:40 PM, Luis Bolinches wrote: > > Hi > > > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > > reason to do so. > > > > AFM for migrations is quite good, latest versions allows to use NSD > > protocol for mounts as well. Olaf did a great job explaining this > > scenario on the redbook chapter 6 > > > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > > > -- > > Cheers > > > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > > > >> wrote: > > > >> Hi Mark, > >> > >> The last time we did something like this was 2010 (we?re doing rolling > >> refreshes now), so there are probably lots of better ways to do this > >> than what we did, but we: > >> > >> 1) set up the new hardware > >> 2) created new filesystems (so that we could make adjustments we > >> wanted to make that can only be made at FS creation time) > >> 3) used rsync to make a 1st pass copy of everything > >> 4) coordinated a time with users / groups to do a 2nd rsync when they > >> weren?t active > >> 5) used symbolic links during the transition (i.e. rm -rvf > >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) > >> 6) once everybody was migrated, updated the symlinks (i.e. /home > >> became a symlink to /gpfs2/home) > >> > >> HTHAL? > >> > >> Kevin > >> > >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com > >>> > wrote: > >>> > >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to > >>> refresh hardware. Any lessons learned in this process? Is it > >>> easiest to just build new cluster and then use AFM? Add to existing > >>> cluster then decommission nodes? What is the recommended process for > >>> this? > >>> > >>> > >>> Mark > >>> > >>> This message (including any attachments) is intended only for the use > >>> of the individual or entity to which it is addressed and may contain > >>> information that is non-public, proprietary, privileged, > >>> confidential, and exempt from disclosure under applicable law. If you > >>> are not the intended recipient, you are hereby notified that any use, > >>> dissemination, distribution, or copying of this communication is > >>> strictly prohibited. This message may be viewed by parties at Sirius > >>> Computer Solutions other than those named in the message header. This > >>> message does not contain an official representation of Sirius > >>> Computer Solutions. If you have received this communication in error, > >>> notify Sirius Computer Solutions immediately and (i) destroy this > >>> message if a facsimile or (ii) delete this message immediately if > >>> this is an electronic communication. Thank you. > >>> > >>> Sirius Computer Solutions > > >>> _______________________________________________ > >>> gpfsug-discuss mailing list > >>> gpfsug-discuss at spectrumscale.org > > >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > >> ? > >> Kevin Buterbaugh - Senior System Administrator > >> Vanderbilt University - Advanced Computing Center for Research and > >> Education > >> Kevin.Buterbaugh at vanderbilt.edu > >> > - (615)875-9633 > >> > >> > >> > > > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > > Oy IBM Finland Ab > > PL 265, 00101 Helsinki, Finland > > Business ID, Y-tunnus: 0195876-3 > > Registered in Finland > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. > > Sirius Computer Solutions _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Wed Oct 12 02:33:32 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Wed, 12 Oct 2016 01:33:32 +0000 Subject: [gpfsug-discuss] 4K sector NSD support (was: Hardware refresh) In-Reply-To: Message-ID: If you File System was created with i=512 you wont benefit from 4k Disk technologies ... some backend emulate it by Controller Software but most likely you'll get in trouble, when trying to add 4k Disk into your filesystem ... Gesendet von IBM Verse Aaron Knister --- Re: [gpfsug-discuss] 4K sector NSD support (was: Hardware refresh) --- Von:"Aaron Knister" An:gpfsug-discuss at spectrumscale.orgDatum:Di. 11.10.2016 17:59Betreff:Re: [gpfsug-discuss] 4K sector NSD support (was: Hardware refresh) Yuri,(Sorry for being somewhat spammy) I now understand the limitation after some more testing (I'm a hands-on learner, can you tell?). Given the right code/cluster/fs version levels I can add 4K dataOnly NSDv2 NSDs to a filesystem created with NSDv1 NSDs. What I can't do is seemingly add any metadataOnly or dataAndMetadata 4K luns to an fs that is not 4K aligned which I assume would be any fs originally created with NSDv1 LUNs. It seems possible to move all data away from NSDv1 LUNS in a filesystem behind-the-scenes using GPFS migration tools, and move the data to NSDv2 LUNs. In this case I believe what's missing is a tool to convert just the metadata structures to be 4K aligned since the data would already on 4k-based NSDv2 LUNS, is that the case? I'm trying to figure out what exactly I'm asking for in an RFE.-AaronOn 10/11/16 7:57 PM, Aaron Knister wrote:> I think I was a little quick to the trigger. I re-read your last mail> after doing some testing and understand it differently. I was wrong> about my interpretation-- you can add 4K NSDv2 formatted NSDs to a> filesystem previously created with NSDv1 NSDs assuming, as you say, the.> minReleaseLevel and filesystem version are high enough. That negates> about half of my last e-mail. The fs still doesn't show as 4K aligned:>> loressd01:~ # /usr/lpp/mmfs/bin/mmlsfs tnb4k --is4KAligned> flag value description> ------------------- ------------------------> -----------------------------------> --is4KAligned No is4KAligned?>> but *shrug* most of the I/O to these disks should be 1MB anyway. If> somebody is pounding the FS with smaller than 4K I/O they're gonna get a> talkin' to.>> -Aaron>> On 10/11/16 6:41 PM, Aaron Knister wrote:>> Thanks Yuri.>>>> I'm asking for my own purposes but I think it's still relevant here:>> we're still at GPFS 3.5 and will be adding dataOnly NSDs with 4K sectors>> in the near future. We're planning to update to 4.1 before we format>> these NSDs, though. If I understand you correctly we can't bring these>> 4K NSDv2 NSDs into a filesystem with 512b-based NSDv1 NSDs? That's a>> pretty big deal :(>>>> Reformatting every few years with 10's of petabytes of data is not>> realistic for us (it would take years to move the data around). It also>> goes against my personal preachings about GPFS's storage virtualization>> capabilities: the ability to perform upgrades/make underlying storage>> infrastructure changes with behind-the-scenes data migration,>> eliminating much of the manual hassle of storage administrators doing>> rsync dances. I guess it's RFE time? It also seems as though AFM could>> help with automating the migration, although many of our filesystems do>> not have filesets on them so we would have to re-think how we lay out>> our filesystems.>>>> This is also curious to me with IBM pitching GPFS as a filesystem for>> cloud services (the cloud *never* goes down, right?). Granted I believe>> this pitch started after the NSDv2 format was defined, but if somebody>> is building a large cloud with GPFS as the underlying filesystem for an>> object or an image store one might think the idea of having to re-format>> the filesystem to gain access to critical new features is inconsistent>> with this pitch. It would be hugely impactful. Just my $.02.>>>> As you can tell, I'm frustrated there's no online conversion tool :) Not>> that there couldn't be... you all are brilliant developers.>>>> -Aaron>>>> On 10/11/16 1:22 PM, Yuri L Volobuev wrote:>>> This depends on the committed cluster version level (minReleaseLevel)>>> and file system format. Since NFSv2 is an on-disk format change, older>>> code wouldn't be able to understand what it is, and thus if there's a>>> possibility of a downlevel node looking at the NSD, the NFSv1 format is>>> going to be used. The code does NSDv1<->NSDv2 conversions under the>>> covers as needed when adding an empty NSD to a file system.>>>>>> I'd strongly recommend getting a fresh start by formatting a new file>>> system. Many things have changed over the course of the last few years.>>> In particular, having a 4K-aligned file system can be a pretty big deal,>>> depending on what hardware one is going to deploy in the future, and>>> this is something that can't be bolted onto an existing file system.>>> Having 4K inodes is very handy for many reasons. New directory format>>> and NSD format changes are attractive, too. And disks generally tend to>>> get larger with time, and at some point you may want to add a disk to an>>> existing storage pool that's larger than the existing allocation map>>> format allows. Obviously, it's more hassle to migrate data to a new file>>> system, as opposed to extending an existing one. In a perfect world,>>> GPFS would offer a conversion tool that seamlessly and robustly converts>>> old file systems, making them as good as new, but in the real world such>>> a tool doesn't exist. Getting a clean slate by formatting a new file>>> system every few years is a good long-term investment of time, although>>> it comes front-loaded with extra work.>>>>>> yuri>>>>>> Inactive hide details for Aaron Knister ---10/10/2016 04:45:31 PM---Can>>> one format NSDv2 NSDs and put them in a filesystem withAaron Knister>>> ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a>>> filesystem with NSDv1 NSD's? -Aaron>>>>>> From: Aaron Knister >>> To: ,>>> Date: 10/10/2016 04:45 PM>>> Subject: Re: [gpfsug-discuss] Hardware refresh>>> Sent by: gpfsug-discuss-bounces at spectrumscale.org>>>>>> ------------------------------------------------------------------------>>>>>>>>>>>> Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's?>>>>>> -Aaron>>>>>> On 10/10/16 7:40 PM, Luis Bolinches wrote:>>>> Hi>>>>>>>> Creating a new FS sounds like a best way to go. NSDv2 being a very good>>>> reason to do so.>>>>>>>> AFM for migrations is quite good, latest versions allows to use NSD>>>> protocol for mounts as well. Olaf did a great job explaining this>>>> scenario on the redbook chapter 6>>>>>>>> http://www.redbooks.ibm.com/abstracts/sg248254.html?Open>>>>>>>> -->>>> Cheers>>>>>>>> On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L>>>> >>> > wrote:>>>>>>>>> Hi Mark,>>>>>>>>>> The last time we did something like this was 2010 (we?re doing rolling>>>>> refreshes now), so there are probably lots of better ways to do this>>>>> than what we did, but we:>>>>>>>>>> 1) set up the new hardware>>>>> 2) created new filesystems (so that we could make adjustments we>>>>> wanted to make that can only be made at FS creation time)>>>>> 3) used rsync to make a 1st pass copy of everything>>>>> 4) coordinated a time with users / groups to do a 2nd rsync when they>>>>> weren?t active>>>>> 5) used symbolic links during the transition (i.e. rm -rvf>>>>> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser)>>>>> 6) once everybody was migrated, updated the symlinks (i.e. /home>>>>> became a symlink to /gpfs2/home)>>>>>>>>>> HTHAL?>>>>>>>>>> Kevin>>>>>>>>>>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com>>>>>> wrote:>>>>>>>>>>>> Have a very old cluster built on IBM X3650?s and DS3500. Need to>>>>>> refresh hardware. Any lessons learned in this process? Is it>>>>>> easiest to just build new cluster and then use AFM? Add to existing>>>>>> cluster then decommission nodes? What is the recommended process for>>>>>> this?>>>>>>>>>>>>>>>>>> Mark>>>>>>>>>>>> This message (including any attachments) is intended only for the use>>>>>> of the individual or entity to which it is addressed and may contain>>>>>> information that is non-public, proprietary, privileged,>>>>>> confidential, and exempt from disclosure under applicable law. If you>>>>>> are not the intended recipient, you are hereby notified that any use,>>>>>> dissemination, distribution, or copying of this communication is>>>>>> strictly prohibited. This message may be viewed by parties at Sirius>>>>>> Computer Solutions other than those named in the message header. This>>>>>> message does not contain an official representation of Sirius>>>>>> Computer Solutions. If you have received this communication in error,>>>>>> notify Sirius Computer Solutions immediately and (i) destroy this>>>>>> message if a facsimile or (ii) delete this message immediately if>>>>>> this is an electronic communication. Thank you.>>>>>>>>>>>> Sirius Computer Solutions >>>>>> _______________________________________________>>>>>> gpfsug-discuss mailing list>>>>>> gpfsug-discuss at spectrumscale.org >>>>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss>>>>>>>>>> ?>>>>> Kevin Buterbaugh - Senior System Administrator>>>>> Vanderbilt University - Advanced Computing Center for Research and>>>>> Education>>>>> Kevin.Buterbaugh at vanderbilt.edu>>>>> - (615)875-9633>>>>>>>>>>>>>>>>>>>>>>> Ellei edell? ole toisin mainittu: / Unless stated otherwise above:>>>> Oy IBM Finland Ab>>>> PL 265, 00101 Helsinki, Finland>>>> Business ID, Y-tunnus: 0195876-3>>>> Registered in Finland>>>>>>>>>>>>>>>> _______________________________________________>>>> gpfsug-discuss mailing list>>>> gpfsug-discuss at spectrumscale.org>>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss>>>>>>> _______________________________________________>>> gpfsug-discuss mailing list>>> gpfsug-discuss at spectrumscale.org>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss>>>>>>>>>>>>>>>>>> _______________________________________________>>> gpfsug-discuss mailing list>>> gpfsug-discuss at spectrumscale.org>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss>>>>> _______________________________________________>> gpfsug-discuss mailing list>> gpfsug-discuss at spectrumscale.org>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss> _______________________________________________> gpfsug-discuss mailing list> gpfsug-discuss at spectrumscale.org> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_______________________________________________gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From volobuev at us.ibm.com Wed Oct 12 06:44:40 2016 From: volobuev at us.ibm.com (Yuri L Volobuev) Date: Tue, 11 Oct 2016 22:44:40 -0700 Subject: [gpfsug-discuss] 4K sector NSD support (was: Hardware refresh) In-Reply-To: References: <63bd99ad-f684-d1bb-371f-e2c21849ff29@nasa.gov> Message-ID: Yes, it is possible to add a 4KN dataOnly NSD to a non-4K-aligned file system, as you figured out. This is something we didn't plan on doing originally, but then had to implement based on the feedback from the field. There's clearly a need for this. However, this statement is exactly it -- dataOnly NSDs. The only way to put metadata on a 4KN disk is to use a 4K-aligned file system. There are several kinds of metadata present in non-4K-aligned file system that generate non-4K IOs (512 byte inodes being the biggest problem), and there's no way to work around this short of using the new format, and there's no way to perform a conversion to the new format in-place. You're welcome to submit an RFE, of course, but I'd recommend being pragmatic about the chances of such an RFE being implemented. As you can imagine, the main reason why an all-encompassing file system conversion tool doesn't exist is not GPFS developers having no idea that such a tool is wanted. There are several considerations that conspire to make this an unlikely candidate to ever be implemented: 1) The task is hard and has no finish line. In most GPFS releases, something changes, necessitating an added piece of work for the hypothetical conversion tool, and the matrix of from-to format version combinations gets to be big quite quickly. 2) A file system conversion is something that is needed very infrequently, but when this code does run, it absolutely has to run and run perfectly, else the result would be a half-converted file system, i.e. a royal mess. This is a tester's nightmare. 3) The failure scenarios are all unpalatable. What should the conversion tool do if it runs out of space replacing smaller metadata structures with bigger ones? Undoing a partially finished conversion is even harder than doing it in the first place. 4) Doing an on-disk conversion on-line is simply very difficult. Consider the task of converting an inode file to use a different inode size. The file can be huge (billions of records), and it would take a fair chunk of time to rewrite it, but the file is changing while it's being converted (can't simply lock the whole thing down for so long), simultaneously on multiple nodes. Orchestrating the processing of updates in the presence of two inode files, with proper atomicity guarantees (to guard against a node failure) is a task of considerable complexity. None of this means the task is impossible, of course. It is, however, a very big chunk of very complex work, all towards a tool that on an average cluster may run somewhere between zero and one times, not something that benefits day-to-day operations. Where the complexity of the task allows for a reasonably affordable implementation, e.g. conversion from an old-style EA file to the FASTEA format, a conversion tool has been implemented (mmmigratefs). However, doing this for every single changed aspect of the file system format is simply too expensive to justify, given other tasks in front of us. On the other hand, a well-implemented migration mechanism solves the file system reformatting scenario (which covers all aspects of file system format changes) as well as a number of other scenarios. This is a cleaner, more general solution. Migration doesn't have to mean an outage. A simple rsync-based migration requires downtime for a cutover, while an AFM-based migration doesn't necessarily require one. I'm not saying that GPFS has a particularly strong migration story at the moment, but this is a much more productive direction for applying resources than a mythical all-encompassing conversion tool. yuri From: Aaron Knister To: , Date: 10/11/2016 05:59 PM Subject: Re: [gpfsug-discuss] 4K sector NSD support (was: Hardware refresh) Sent by: gpfsug-discuss-bounces at spectrumscale.org Yuri, (Sorry for being somewhat spammy) I now understand the limitation after some more testing (I'm a hands-on learner, can you tell?). Given the right code/cluster/fs version levels I can add 4K dataOnly NSDv2 NSDs to a filesystem created with NSDv1 NSDs. What I can't do is seemingly add any metadataOnly or dataAndMetadata 4K luns to an fs that is not 4K aligned which I assume would be any fs originally created with NSDv1 LUNs. It seems possible to move all data away from NSDv1 LUNS in a filesystem behind-the-scenes using GPFS migration tools, and move the data to NSDv2 LUNs. In this case I believe what's missing is a tool to convert just the metadata structures to be 4K aligned since the data would already on 4k-based NSDv2 LUNS, is that the case? I'm trying to figure out what exactly I'm asking for in an RFE. -Aaron On 10/11/16 7:57 PM, Aaron Knister wrote: > I think I was a little quick to the trigger. I re-read your last mail > after doing some testing and understand it differently. I was wrong > about my interpretation-- you can add 4K NSDv2 formatted NSDs to a > filesystem previously created with NSDv1 NSDs assuming, as you say, the. > minReleaseLevel and filesystem version are high enough. That negates > about half of my last e-mail. The fs still doesn't show as 4K aligned: > > loressd01:~ # /usr/lpp/mmfs/bin/mmlsfs tnb4k --is4KAligned > flag value description > ------------------- ------------------------ > ----------------------------------- > --is4KAligned No is4KAligned? > > but *shrug* most of the I/O to these disks should be 1MB anyway. If > somebody is pounding the FS with smaller than 4K I/O they're gonna get a > talkin' to. > > -Aaron > > On 10/11/16 6:41 PM, Aaron Knister wrote: >> Thanks Yuri. >> >> I'm asking for my own purposes but I think it's still relevant here: >> we're still at GPFS 3.5 and will be adding dataOnly NSDs with 4K sectors >> in the near future. We're planning to update to 4.1 before we format >> these NSDs, though. If I understand you correctly we can't bring these >> 4K NSDv2 NSDs into a filesystem with 512b-based NSDv1 NSDs? That's a >> pretty big deal :( >> >> Reformatting every few years with 10's of petabytes of data is not >> realistic for us (it would take years to move the data around). It also >> goes against my personal preachings about GPFS's storage virtualization >> capabilities: the ability to perform upgrades/make underlying storage >> infrastructure changes with behind-the-scenes data migration, >> eliminating much of the manual hassle of storage administrators doing >> rsync dances. I guess it's RFE time? It also seems as though AFM could >> help with automating the migration, although many of our filesystems do >> not have filesets on them so we would have to re-think how we lay out >> our filesystems. >> >> This is also curious to me with IBM pitching GPFS as a filesystem for >> cloud services (the cloud *never* goes down, right?). Granted I believe >> this pitch started after the NSDv2 format was defined, but if somebody >> is building a large cloud with GPFS as the underlying filesystem for an >> object or an image store one might think the idea of having to re-format >> the filesystem to gain access to critical new features is inconsistent >> with this pitch. It would be hugely impactful. Just my $.02. >> >> As you can tell, I'm frustrated there's no online conversion tool :) Not >> that there couldn't be... you all are brilliant developers. >> >> -Aaron >> >> On 10/11/16 1:22 PM, Yuri L Volobuev wrote: >>> This depends on the committed cluster version level (minReleaseLevel) >>> and file system format. Since NFSv2 is an on-disk format change, older >>> code wouldn't be able to understand what it is, and thus if there's a >>> possibility of a downlevel node looking at the NSD, the NFSv1 format is >>> going to be used. The code does NSDv1<->NSDv2 conversions under the >>> covers as needed when adding an empty NSD to a file system. >>> >>> I'd strongly recommend getting a fresh start by formatting a new file >>> system. Many things have changed over the course of the last few years. >>> In particular, having a 4K-aligned file system can be a pretty big deal, >>> depending on what hardware one is going to deploy in the future, and >>> this is something that can't be bolted onto an existing file system. >>> Having 4K inodes is very handy for many reasons. New directory format >>> and NSD format changes are attractive, too. And disks generally tend to >>> get larger with time, and at some point you may want to add a disk to an >>> existing storage pool that's larger than the existing allocation map >>> format allows. Obviously, it's more hassle to migrate data to a new file >>> system, as opposed to extending an existing one. In a perfect world, >>> GPFS would offer a conversion tool that seamlessly and robustly converts >>> old file systems, making them as good as new, but in the real world such >>> a tool doesn't exist. Getting a clean slate by formatting a new file >>> system every few years is a good long-term investment of time, although >>> it comes front-loaded with extra work. >>> >>> yuri >>> >>> Inactive hide details for Aaron Knister ---10/10/2016 04:45:31 PM---Can >>> one format NSDv2 NSDs and put them in a filesystem withAaron Knister >>> ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a >>> filesystem with NSDv1 NSD's? -Aaron >>> >>> From: Aaron Knister >>> To: , >>> Date: 10/10/2016 04:45 PM >>> Subject: Re: [gpfsug-discuss] Hardware refresh >>> Sent by: gpfsug-discuss-bounces at spectrumscale.org >>> >>> ------------------------------------------------------------------------ >>> >>> >>> >>> Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? >>> >>> -Aaron >>> >>> On 10/10/16 7:40 PM, Luis Bolinches wrote: >>>> Hi >>>> >>>> Creating a new FS sounds like a best way to go. NSDv2 being a very good >>>> reason to do so. >>>> >>>> AFM for migrations is quite good, latest versions allows to use NSD >>>> protocol for mounts as well. Olaf did a great job explaining this >>>> scenario on the redbook chapter 6 >>>> >>>> http://www.redbooks.ibm.com/abstracts/sg248254.html?Open >>>> >>>> -- >>>> Cheers >>>> >>>> On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L >>>> >>> > wrote: >>>> >>>>> Hi Mark, >>>>> >>>>> The last time we did something like this was 2010 (we?re doing rolling >>>>> refreshes now), so there are probably lots of better ways to do this >>>>> than what we did, but we: >>>>> >>>>> 1) set up the new hardware >>>>> 2) created new filesystems (so that we could make adjustments we >>>>> wanted to make that can only be made at FS creation time) >>>>> 3) used rsync to make a 1st pass copy of everything >>>>> 4) coordinated a time with users / groups to do a 2nd rsync when they >>>>> weren?t active >>>>> 5) used symbolic links during the transition (i.e. rm -rvf >>>>> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >>>>> 6) once everybody was migrated, updated the symlinks (i.e. /home >>>>> became a symlink to /gpfs2/home) >>>>> >>>>> HTHAL? >>>>> >>>>> Kevin >>>>> >>>>>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>>>>> wrote: >>>>>> >>>>>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>>>>> refresh hardware. Any lessons learned in this process? Is it >>>>>> easiest to just build new cluster and then use AFM? Add to existing >>>>>> cluster then decommission nodes? What is the recommended process for >>>>>> this? >>>>>> >>>>>> >>>>>> Mark >>>>>> >>>>>> This message (including any attachments) is intended only for the use >>>>>> of the individual or entity to which it is addressed and may contain >>>>>> information that is non-public, proprietary, privileged, >>>>>> confidential, and exempt from disclosure under applicable law. If you >>>>>> are not the intended recipient, you are hereby notified that any use, >>>>>> dissemination, distribution, or copying of this communication is >>>>>> strictly prohibited. This message may be viewed by parties at Sirius >>>>>> Computer Solutions other than those named in the message header. This >>>>>> message does not contain an official representation of Sirius >>>>>> Computer Solutions. If you have received this communication in error, >>>>>> notify Sirius Computer Solutions immediately and (i) destroy this >>>>>> message if a facsimile or (ii) delete this message immediately if >>>>>> this is an electronic communication. Thank you. >>>>>> >>>>>> Sirius Computer Solutions >>>>>> _______________________________________________ >>>>>> gpfsug-discuss mailing list >>>>>> gpfsug-discuss at spectrumscale.org >>>>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>>>> >>>>> ? >>>>> Kevin Buterbaugh - Senior System Administrator >>>>> Vanderbilt University - Advanced Computing Center for Research and >>>>> Education >>>>> Kevin.Buterbaugh at vanderbilt.edu >>>>> - (615)875-9633 >>>>> >>>>> >>>>> >>>> >>>> Ellei edell? ole toisin mainittu: / Unless stated otherwise above: >>>> Oy IBM Finland Ab >>>> PL 265, 00101 Helsinki, Finland >>>> Business ID, Y-tunnus: 0195876-3 >>>> Registered in Finland >>>> >>>> >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> >>> >>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From makaplan at us.ibm.com Wed Oct 12 18:54:52 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Wed, 12 Oct 2016 13:54:52 -0400 Subject: [gpfsug-discuss] Hardware refresh - In-Reply-To: <279BAB6D-21E6-4EC1-98CD-CA409A24F8F6@ulmer.org> References: <9E337B56-A274-41B6-84C2-7084AAB771F4@siriuscom.com> <279BAB6D-21E6-4EC1-98CD-CA409A24F8F6@ulmer.org> Message-ID: Yes, you can AFM within a single cluster, in fact with just a single node. I just set this up on my toy system: [root at bog-wifi cmvc]# mmlsfileset yy afmlu --afm Filesets in file system 'yy': Name Status Path afmTarget afmlu Linked /yy/afmlu gpfs:///xx [root at bog-wifi cmvc]# mount ... yy on /yy type gpfs (rw,relatime,seclabel) xx on /xx type gpfs (rw,relatime,seclabel) [root at bog-wifi cmvc]# mmafmctl yy getstate Fileset Name Fileset Target Cache State Gateway Node Queue Length Queue numExec ------------ -------------- ------------- ------------ ------------ ------------- afmlu gpfs:///xx Active bog-wifi 0 7 So, you may add nodes, add disks to an existing cluster, upgrade your software, define a new FS, migrate data from an old FS to a new FS then delete nodes and disks that are no longer needed... From: Stephen Ulmer To: gpfsug main discussion list Date: 10/11/2016 09:30 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org I think that the OP was asking why not expand the existing cluster with the new hardware, and just make a new FS? I?ve not tried to make a cluster talk AFM to itself yet. If that?s impossible, then there?s one good reason to make a new cluster (to use AFM for migration). Liberty, -- Stephen On Oct 11, 2016, at 8:40 PM, Mark.Bush at siriuscom.com wrote: Only compelling reason for new cluster would be old hardware is EOL or no longer want to pay maintenance on it. From: on behalf of Marc A Kaplan Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 2:58 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh New FS? Yes there are some good reasons. New cluster? I did not see a compelling argument either way. From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Date: 10/11/2016 03:34 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org Ok. I think I am hearing that a new cluster with a new FS and copying data from old to new cluster is the best way forward. Thanks everyone for your input. From: on behalf of Yuri L Volobuev Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 12:22 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh This depends on the committed cluster version level (minReleaseLevel) and file system format. Since NFSv2 is an on-disk format change, older code wouldn't be able to understand what it is, and thus if there's a possibility of a downlevel node looking at the NSD, the NFSv1 format is going to be used. The code does NSDv1<->NSDv2 conversions under the covers as needed when adding an empty NSD to a file system. I'd strongly recommend getting a fresh start by formatting a new file system. Many things have changed over the course of the last few years. In particular, having a 4K-aligned file system can be a pretty big deal, depending on what hardware one is going to deploy in the future, and this is something that can't be bolted onto an existing file system. Having 4K inodes is very handy for many reasons. New directory format and NSD format changes are attractive, too. And disks generally tend to get larger with time, and at some point you may want to add a disk to an existing storage pool that's larger than the existing allocation map format allows. Obviously, it's more hassle to migrate data to a new file system, as opposed to extending an existing one. In a perfect world, GPFS would offer a conversion tool that seamlessly and robustly converts old file systems, making them as good as new, but in the real world such a tool doesn't exist. Getting a clean slate by formatting a new file system every few years is a good long-term investment of time, although it comes front-loaded with extra work. yuri Aaron Knister ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron From: Aaron Knister To: , Date: 10/10/2016 04:45 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron On 10/10/16 7:40 PM, Luis Bolinches wrote: > Hi > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > reason to do so. > > AFM for migrations is quite good, latest versions allows to use NSD > protocol for mounts as well. Olaf did a great job explaining this > scenario on the redbook chapter 6 > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > -- > Cheers > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > wrote: > >> Hi Mark, >> >> The last time we did something like this was 2010 (we?re doing rolling >> refreshes now), so there are probably lots of better ways to do this >> than what we did, but we: >> >> 1) set up the new hardware >> 2) created new filesystems (so that we could make adjustments we >> wanted to make that can only be made at FS creation time) >> 3) used rsync to make a 1st pass copy of everything >> 4) coordinated a time with users / groups to do a 2nd rsync when they >> weren?t active >> 5) used symbolic links during the transition (i.e. rm -rvf >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >> 6) once everybody was migrated, updated the symlinks (i.e. /home >> became a symlink to /gpfs2/home) >> >> HTHAL? >> >> Kevin >> >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>> wrote: >>> >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>> refresh hardware. Any lessons learned in this process? Is it >>> easiest to just build new cluster and then use AFM? Add to existing >>> cluster then decommission nodes? What is the recommended process for >>> this? >>> >>> >>> Mark >>> >>> This message (including any attachments) is intended only for the use >>> of the individual or entity to which it is addressed and may contain >>> information that is non-public, proprietary, privileged, >>> confidential, and exempt from disclosure under applicable law. If you >>> are not the intended recipient, you are hereby notified that any use, >>> dissemination, distribution, or copying of this communication is >>> strictly prohibited. This message may be viewed by parties at Sirius >>> Computer Solutions other than those named in the message header. This >>> message does not contain an official representation of Sirius >>> Computer Solutions. If you have received this communication in error, >>> notify Sirius Computer Solutions immediately and (i) destroy this >>> message if a facsimile or (ii) delete this message immediately if >>> this is an electronic communication. Thank you. >>> >>> Sirius Computer Solutions >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and >> Education >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mark.Bush at siriuscom.com Wed Oct 12 21:19:19 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Wed, 12 Oct 2016 20:19:19 +0000 Subject: [gpfsug-discuss] Spectrum Scale RAID usable space Message-ID: <5B378555-7956-40A1-8840-8090B43CAC7F@siriuscom.com> Anyone know how to calculate this? I realize this is an ESS thing now but I wonder if someone on here knows how to extrapolate usable space. Thanks Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From abeattie at au1.ibm.com Wed Oct 12 22:58:04 2016 From: abeattie at au1.ibm.com (Andrew Beattie) Date: Wed, 12 Oct 2016 21:58:04 +0000 Subject: [gpfsug-discuss] Spectrum Scale RAID usable space In-Reply-To: <5B378555-7956-40A1-8840-8090B43CAC7F@siriuscom.com> Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.14763088875930.png Type: image/png Size: 177376 bytes Desc: not available URL: From Mark.Bush at siriuscom.com Wed Oct 12 23:25:48 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Wed, 12 Oct 2016 22:25:48 +0000 Subject: [gpfsug-discuss] Spectrum Scale RAID usable space In-Reply-To: References: <5B378555-7956-40A1-8840-8090B43CAC7F@siriuscom.com> Message-ID: <7407A861-BC0F-40EA-97F9-24FAEA00C838@siriuscom.com> GL2 with 6TB drives. From: on behalf of Andrew Beattie Reply-To: gpfsug main discussion list Date: Wednesday, October 12, 2016 at 4:58 PM To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] Spectrum Scale RAID usable space Hi Mark, If you let me know which model ESS and what drive size I can give you the possible capacities. there are effectively 4 options 8+2P 8+3P 3-way replication 4-way replication for example: GL6 - 8TB drives Possible capacity options: 8+2P - 2,036TB 8+3P - 1,851TB 3 Way - 848TB 4 Way - 636TB [cid:image002.png at 01D224AD.AB668310] Business Partners: http://www.ibm.com/partnerworld/wps/servlet/ContentHandler/SSPQ048068H83479I86 There is a ESS capacity calculator in the same set of BP links with Capacity Magic and Disk magic Andrew Beattie Software Defined Storage - IT Specialist Phone: 614-2133-7927 E-mail: abeattie at au1.ibm.com ----- Original message ----- From: "Mark.Bush at siriuscom.com" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list Cc: Subject: [gpfsug-discuss] Spectrum Scale RAID usable space Date: Thu, Oct 13, 2016 7:19 AM Anyone know how to calculate this? I realize this is an ESS thing now but I wonder if someone on here knows how to extrapolate usable space. Thanks Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.png Type: image/png Size: 170988 bytes Desc: image002.png URL: From ulmer at ulmer.org Thu Oct 13 01:48:23 2016 From: ulmer at ulmer.org (Stephen Ulmer) Date: Wed, 12 Oct 2016 20:48:23 -0400 Subject: [gpfsug-discuss] Hardware refresh - In-Reply-To: References: <9E337B56-A274-41B6-84C2-7084AAB771F4@siriuscom.com> <279BAB6D-21E6-4EC1-98CD-CA409A24F8F6@ulmer.org> Message-ID: <1F6BD2DF-10F4-456B-9061-67D6D6B90B3D@ulmer.org> Nice! -- Stephen Ulmer Sent from a mobile device; please excuse auto-correct silliness. > On Oct 12, 2016, at 1:54 PM, Marc A Kaplan wrote: > > Yes, you can AFM within a single cluster, in fact with just a single node. I just set this up on my toy system: > > [root at bog-wifi cmvc]# mmlsfileset yy afmlu --afm > Filesets in file system 'yy': > Name Status Path afmTarget > afmlu Linked /yy/afmlu gpfs:///xx > > [root at bog-wifi cmvc]# mount > ... > yy on /yy type gpfs (rw,relatime,seclabel) > xx on /xx type gpfs (rw,relatime,seclabel) > > [root at bog-wifi cmvc]# mmafmctl yy getstate > Fileset Name Fileset Target Cache State Gateway Node Queue Length Queue numExec > ------------ -------------- ------------- ------------ ------------ ------------- > afmlu gpfs:///xx Active bog-wifi 0 7 > > So, you may add nodes, add disks to an existing cluster, upgrade your software, define a new FS, > migrate data from an old FS to a new FS > then delete nodes and disks that are no longer needed... > > > > From: Stephen Ulmer > To: gpfsug main discussion list > Date: 10/11/2016 09:30 PM > Subject: Re: [gpfsug-discuss] Hardware refresh > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > I think that the OP was asking why not expand the existing cluster with the new hardware, and just make a new FS? > > I?ve not tried to make a cluster talk AFM to itself yet. If that?s impossible, then there?s one good reason to make a new cluster (to use AFM for migration). > > Liberty, > > -- > Stephen > > > > On Oct 11, 2016, at 8:40 PM, Mark.Bush at siriuscom.comwrote: > > Only compelling reason for new cluster would be old hardware is EOL or no longer want to pay maintenance on it. > > From: on behalf of Marc A Kaplan > Reply-To: gpfsug main discussion list > Date: Tuesday, October 11, 2016 at 2:58 PM > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Hardware refresh > > New FS? Yes there are some good reasons. > New cluster? I did not see a compelling argument either way. > > > > From: "Mark.Bush at siriuscom.com" > To: gpfsug main discussion list > Date: 10/11/2016 03:34 PM > Subject: Re: [gpfsug-discuss] Hardware refresh > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > > Ok. I think I am hearing that a new cluster with a new FS and copying data from old to new cluster is the best way forward. Thanks everyone for your input. > > From: on behalf of Yuri L Volobuev > Reply-To: gpfsug main discussion list > Date: Tuesday, October 11, 2016 at 12:22 PM > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Hardware refresh > > This depends on the committed cluster version level (minReleaseLevel) and file system format. Since NFSv2 is an on-disk format change, older code wouldn't be able to understand what it is, and thus if there's a possibility of a downlevel node looking at the NSD, the NFSv1 format is going to be used. The code does NSDv1<->NSDv2 conversions under the covers as needed when adding an empty NSD to a file system. > > I'd strongly recommend getting a fresh start by formatting a new file system. Many things have changed over the course of the last few years. In particular, having a 4K-aligned file system can be a pretty big deal, depending on what hardware one is going to deploy in the future, and this is something that can't be bolted onto an existing file system. Having 4K inodes is very handy for many reasons. New directory format and NSD format changes are attractive, too. And disks generally tend to get larger with time, and at some point you may want to add a disk to an existing storage pool that's larger than the existing allocation map format allows. Obviously, it's more hassle to migrate data to a new file system, as opposed to extending an existing one. In a perfect world, GPFS would offer a conversion tool that seamlessly and robustly converts old file systems, making them as good as new, but in the real world such a tool doesn't exist. Getting a clean slate by formatting a new file system every few years is a good long-term investment of time, although it comes front-loaded with extra work. > > yuri > > Aaron Knister ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron > > From: Aaron Knister > To: , > Date: 10/10/2016 04:45 PM > Subject: Re: [gpfsug-discuss] Hardware refresh > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > > > > Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? > > -Aaron > > On 10/10/16 7:40 PM, Luis Bolinches wrote: > > Hi > > > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > > reason to do so. > > > > AFM for migrations is quite good, latest versions allows to use NSD > > protocol for mounts as well. Olaf did a great job explaining this > > scenario on the redbook chapter 6 > > > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > > > -- > > Cheers > > > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > > > wrote: > > > >> Hi Mark, > >> > >> The last time we did something like this was 2010 (we?re doing rolling > >> refreshes now), so there are probably lots of better ways to do this > >> than what we did, but we: > >> > >> 1) set up the new hardware > >> 2) created new filesystems (so that we could make adjustments we > >> wanted to make that can only be made at FS creation time) > >> 3) used rsync to make a 1st pass copy of everything > >> 4) coordinated a time with users / groups to do a 2nd rsync when they > >> weren?t active > >> 5) used symbolic links during the transition (i.e. rm -rvf > >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) > >> 6) once everybody was migrated, updated the symlinks (i.e. /home > >> became a symlink to /gpfs2/home) > >> > >> HTHAL? > >> > >> Kevin > >> > >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com > >>> wrote: > >>> > >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to > >>> refresh hardware. Any lessons learned in this process? Is it > >>> easiest to just build new cluster and then use AFM? Add to existing > >>> cluster then decommission nodes? What is the recommended process for > >>> this? > >>> > >>> > >>> Mark > >>> > >>> This message (including any attachments) is intended only for the use > >>> of the individual or entity to which it is addressed and may contain > >>> information that is non-public, proprietary, privileged, > >>> confidential, and exempt from disclosure under applicable law. If you > >>> are not the intended recipient, you are hereby notified that any use, > >>> dissemination, distribution, or copying of this communication is > >>> strictly prohibited. This message may be viewed by parties at Sirius > >>> Computer Solutions other than those named in the message header. This > >>> message does not contain an official representation of Sirius > >>> Computer Solutions. If you have received this communication in error, > >>> notify Sirius Computer Solutions immediately and (i) destroy this > >>> message if a facsimile or (ii) delete this message immediately if > >>> this is an electronic communication. Thank you. > >>> > >>> Sirius Computer Solutions > >>> _______________________________________________ > >>> gpfsug-discuss mailing list > >>> gpfsug-discuss at spectrumscale.org > >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > >> ? > >> Kevin Buterbaugh - Senior System Administrator > >> Vanderbilt University - Advanced Computing Center for Research and > >> Education > >> Kevin.Buterbaugh at vanderbilt.edu > >> - (615)875-9633 > >> > >> > >> > > > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > > Oy IBM Finland Ab > > PL 265, 00101 Helsinki, Finland > > Business ID, Y-tunnus: 0195876-3 > > Registered in Finland > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. > > Sirius Computer Solutions_______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From shankbal at in.ibm.com Thu Oct 13 11:43:23 2016 From: shankbal at in.ibm.com (Shankar Balasubramanian) Date: Thu, 13 Oct 2016 16:13:23 +0530 Subject: [gpfsug-discuss] Hardware refresh - In-Reply-To: References: <9E337B56-A274-41B6-84C2-7084AAB771F4@siriuscom.com><279BAB6D-21E6-4EC1-98CD-CA409A24F8F6@ulmer.org> Message-ID: Please note - though one of the supported use case of AFM is such migration scenarios, infact AFM does particularly well in faithfully migrating the data when the source and destination cluster are GPFS, the scalability of this solution for multi million/multi terabyte file systems has its own set of challenges. These have to carefully understood and checked if AFM will fit the bill. Best Regards, Shankar Balasubramanian STSM, AFM & Async DR Development IBM Systems Bangalore - Embassy Golf Links India From: "Marc A Kaplan" To: gpfsug main discussion list Date: 10/12/2016 11:25 PM Subject: Re: [gpfsug-discuss] Hardware refresh - Sent by: gpfsug-discuss-bounces at spectrumscale.org Yes, you can AFM within a single cluster, in fact with just a single node. I just set this up on my toy system: [root at bog-wifi cmvc]# mmlsfileset yy afmlu --afm Filesets in file system 'yy': Name Status Path afmTarget afmlu Linked /yy/afmlu gpfs:///xx [root at bog-wifi cmvc]# mount ... yy on /yy type gpfs (rw,relatime,seclabel) xx on /xx type gpfs (rw,relatime,seclabel) [root at bog-wifi cmvc]# mmafmctl yy getstate Fileset Name Fileset Target Cache State Gateway Node Queue Length Queue numExec ------------ -------------- ------------- ------------ ------------ ------------- afmlu gpfs:///xx Active bog-wifi 0 7 So, you may add nodes, add disks to an existing cluster, upgrade your software, define a new FS, migrate data from an old FS to a new FS then delete nodes and disks that are no longer needed... From: Stephen Ulmer To: gpfsug main discussion list Date: 10/11/2016 09:30 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org I think that the OP was asking why not expand the existing cluster with the new hardware, and just make a new FS? I?ve not tried to make a cluster talk AFM to itself yet. If that?s impossible, then there?s one good reason to make a new cluster (to use AFM for migration). Liberty, -- Stephen On Oct 11, 2016, at 8:40 PM, Mark.Bush at siriuscom.comwrote: Only compelling reason for new cluster would be old hardware is EOL or no longer want to pay maintenance on it. From: on behalf of Marc A Kaplan Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 2:58 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh New FS? Yes there are some good reasons. New cluster? I did not see a compelling argument either way. From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Date: 10/11/2016 03:34 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org Ok. I think I am hearing that a new cluster with a new FS and copying data from old to new cluster is the best way forward. Thanks everyone for your input. From: on behalf of Yuri L Volobuev Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 12:22 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh This depends on the committed cluster version level (minReleaseLevel) and file system format. Since NFSv2 is an on-disk format change, older code wouldn't be able to understand what it is, and thus if there's a possibility of a downlevel node looking at the NSD, the NFSv1 format is going to be used. The code does NSDv1<->NSDv2 conversions under the covers as needed when adding an empty NSD to a file system. I'd strongly recommend getting a fresh start by formatting a new file system. Many things have changed over the course of the last few years. In particular, having a 4K-aligned file system can be a pretty big deal, depending on what hardware one is going to deploy in the future, and this is something that can't be bolted onto an existing file system. Having 4K inodes is very handy for many reasons. New directory format and NSD format changes are attractive, too. And disks generally tend to get larger with time, and at some point you may want to add a disk to an existing storage pool that's larger than the existing allocation map format allows. Obviously, it's more hassle to migrate data to a new file system, as opposed to extending an existing one. In a perfect world, GPFS would offer a conversion tool that seamlessly and robustly converts old file systems, making them as good as new, but in the real world such a tool doesn't exist. Getting a clean slate by formatting a new file system every few years is a good long-term investment of time, although it comes front-loaded with extra work. yuri Aaron Knister ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron From: Aaron Knister To: , Date: 10/10/2016 04:45 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron On 10/10/16 7:40 PM, Luis Bolinches wrote: > Hi > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > reason to do so. > > AFM for migrations is quite good, latest versions allows to use NSD > protocol for mounts as well. Olaf did a great job explaining this > scenario on the redbook chapter 6 > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > -- > Cheers > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > wrote: > >> Hi Mark, >> >> The last time we did something like this was 2010 (we?re doing rolling >> refreshes now), so there are probably lots of better ways to do this >> than what we did, but we: >> >> 1) set up the new hardware >> 2) created new filesystems (so that we could make adjustments we >> wanted to make that can only be made at FS creation time) >> 3) used rsync to make a 1st pass copy of everything >> 4) coordinated a time with users / groups to do a 2nd rsync when they >> weren?t active >> 5) used symbolic links during the transition (i.e. rm -rvf >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >> 6) once everybody was migrated, updated the symlinks (i.e. /home >> became a symlink to /gpfs2/home) >> >> HTHAL? >> >> Kevin >> >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>> wrote: >>> >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>> refresh hardware. Any lessons learned in this process? Is it >>> easiest to just build new cluster and then use AFM? Add to existing >>> cluster then decommission nodes? What is the recommended process for >>> this? >>> >>> >>> Mark >>> >>> This message (including any attachments) is intended only for the use >>> of the individual or entity to which it is addressed and may contain >>> information that is non-public, proprietary, privileged, >>> confidential, and exempt from disclosure under applicable law. If you >>> are not the intended recipient, you are hereby notified that any use, >>> dissemination, distribution, or copying of this communication is >>> strictly prohibited. This message may be viewed by parties at Sirius >>> Computer Solutions other than those named in the message header. This >>> message does not contain an official representation of Sirius >>> Computer Solutions. If you have received this communication in error, >>> notify Sirius Computer Solutions immediately and (i) destroy this >>> message if a facsimile or (ii) delete this message immediately if >>> this is an electronic communication. Thank you. >>> >>> Sirius Computer Solutions >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and >> Education >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From mimarsh2 at vt.edu Thu Oct 13 14:30:03 2016 From: mimarsh2 at vt.edu (Brian Marshall) Date: Thu, 13 Oct 2016 09:30:03 -0400 Subject: [gpfsug-discuss] SC16 Draft Agenda [Save the date: Sunday, November 13th] In-Reply-To: References: Message-ID: Virginia Tech ARC purchased a SANDisk IF150 (IBM DeepFlash 150) this Summer and are just getting it into production (maybe next week). I have some dev benchmarking numbers and may have production/user feedback by SC16. I'm not sure I can fill an entire 30 minutes with worthwhile information, but I can give at least 15 minutes on: DeepFlash: a computational scientist's first impressions Best, Brian Marshall On Mon, Oct 10, 2016 at 6:59 PM, GPFS UG USA Principal < usa-principal at gpfsug.org> wrote: > Hello all, > > There have been some questions about the Spectrum Scale Users Group event > for SC16 and we thought it best to publish our draft agenda and continue to > fill it in as details get settled. That way you can make your travel plans > and schedules. > > The date and rough times are set, we may adjust the time range a bit > depending upon the number of user talks that people would like to give. So > note, yes! *we are asking for user talks. Please consider doing this. *We > always receive positive feedback about talks given by users that describe > real world experiences. If *any* of the user talk slots below are of > interest to you, please let us know as soon as you can. We may turn at > least one user talk into a panel if we can get enough participants. > > As always, your feedback is welcome. This is a *users* group after all. > > So, here?s what we have planned so far: > > *Date: Sunday November 13th* > *Venue: TBD, but near the convention center in downtown SLC* > *Time: ~12:30p - ~17:30p* > > *Agenda:* > > > > > > > > > *12:30 - 12:50 Welcome / Overview12:50 - 13:50 The latest in IBM Spectrum > Scale and ESS13:50 - 14:20 User Talk: Big Data in HPC14:20 - 14:50 User > Talk: Tiering to Object Storage14:50 - 15:20 - break -15:20 - 15:50 > User Talk: TBD ? volunteers! please! 15:50 - 16:35 Spectrum Scale > Enhancements for CORAL16:35 - 17:20 News from IBM Research17:20 - 17:30 > Closing* > > Best, > > Kristy & Bob > > Kristy Kallback-Rose > Manager, Research Storage > Indiana University > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Thu Oct 13 14:55:14 2016 From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]) Date: Thu, 13 Oct 2016 13:55:14 +0000 Subject: [gpfsug-discuss] SC16 Draft Agenda [Save the date: Sunday, November 13th] Message-ID: <5F910253243E6A47B81A9A2EB424BBA101D90737@NDMSMBX404.ndc.nasa.gov> I'd be quite interested in hearing about that Brian. I have to double check this with a couple folks here first but is there interest in hear about some of the stuff NASA has been up to with GPFS? I'd be tickled pink to share details on our GPFS activities: - FPO/Hadoop efforts - a site report (we're not *huge* but we're a modest Asize - 3600 nodes and almost 50PB at this point) - demoing the monitoring dashboard we've created with influxdb and grafana. - talk about the tools we've developed to deal with mass uid number changes on GPFS filesystems -Aaron From: Brian Marshall Sent: 10/13/16, 9:30 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] SC16 Draft Agenda [Save the date: Sunday, November 13th] Virginia Tech ARC purchased a SANDisk IF150 (IBM DeepFlash 150) this Summer and are just getting it into production (maybe next week). I have some dev benchmarking numbers and may have production/user feedback by SC16. I'm not sure I can fill an entire 30 minutes with worthwhile information, but I can give at least 15 minutes on: DeepFlash: a computational scientist's first impressions Best, Brian Marshall On Mon, Oct 10, 2016 at 6:59 PM, GPFS UG USA Principal > wrote: Hello all, There have been some questions about the Spectrum Scale Users Group event for SC16 and we thought it best to publish our draft agenda and continue to fill it in as details get settled. That way you can make your travel plans and schedules. The date and rough times are set, we may adjust the time range a bit depending upon the number of user talks that people would like to give. So note, yes! we are asking for user talks. Please consider doing this. We always receive positive feedback about talks given by users that describe real world experiences. If any of the user talk slots below are of interest to you, please let us know as soon as you can. We may turn at least one user talk into a panel if we can get enough participants. As always, your feedback is welcome. This is a users group after all. So, here?s what we have planned so far: Date: Sunday November 13th Venue: TBD, but near the convention center in downtown SLC Time: ~12:30p - ~17:30p Agenda: 12:30 - 12:50 Welcome / Overview 12:50 - 13:50 The latest in IBM Spectrum Scale and ESS 13:50 - 14:20 User Talk: Big Data in HPC 14:20 - 14:50 User Talk: Tiering to Object Storage 14:50 - 15:20 - break - 15:20 - 15:50 User Talk: TBD ? volunteers! please! 15:50 - 16:35 Spectrum Scale Enhancements for CORAL 16:35 - 17:20 News from IBM Research 17:20 - 17:30 Closing Best, Kristy & Bob Kristy Kallback-Rose Manager, Research Storage Indiana University _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew at ellexus.com Thu Oct 13 16:00:54 2016 From: matthew at ellexus.com (Matthew Harris) Date: Thu, 13 Oct 2016 16:00:54 +0100 Subject: [gpfsug-discuss] SC16 Draft Agenda [Save the date: Sunday, November 13th] In-Reply-To: References: Message-ID: Who is delivering the 12:50 talk please? Regards, Matthew Harris Account Manager & Business Development - Ellexus Ltd *www.ellexus.com * *Ellexus Ltd is a limited company registered in England & Wales* *Company registration no. 07166034* *Registered address: 198 High Street, Tonbridge, Kent TN9 1BE, UK* *Operating address: St John's Innovation Centre, Cowley Road, Cambridge CB4 0WS, UK* On Mon, Oct 10, 2016 at 11:59 PM, GPFS UG USA Principal < usa-principal at gpfsug.org> wrote: > Hello all, > > There have been some questions about the Spectrum Scale Users Group event > for SC16 and we thought it best to publish our draft agenda and continue to > fill it in as details get settled. That way you can make your travel plans > and schedules. > > The date and rough times are set, we may adjust the time range a bit > depending upon the number of user talks that people would like to give. So > note, yes! *we are asking for user talks. Please consider doing this. *We > always receive positive feedback about talks given by users that describe > real world experiences. If *any* of the user talk slots below are of > interest to you, please let us know as soon as you can. We may turn at > least one user talk into a panel if we can get enough participants. > > As always, your feedback is welcome. This is a *users* group after all. > > So, here?s what we have planned so far: > > *Date: Sunday November 13th* > *Venue: TBD, but near the convention center in downtown SLC* > *Time: ~12:30p - ~17:30p* > > *Agenda:* > > > > > > > > > *12:30 - 12:50 Welcome / Overview12:50 - 13:50 The latest in IBM Spectrum > Scale and ESS13:50 - 14:20 User Talk: Big Data in HPC14:20 - 14:50 User > Talk: Tiering to Object Storage14:50 - 15:20 - break -15:20 - 15:50 > User Talk: TBD ? volunteers! please! 15:50 - 16:35 Spectrum Scale > Enhancements for CORAL16:35 - 17:20 News from IBM Research17:20 - 17:30 > Closing* > > Best, > > Kristy & Bob > > Kristy Kallback-Rose > Manager, Research Storage > Indiana University > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Thu Oct 13 16:27:14 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Thu, 13 Oct 2016 11:27:14 -0400 Subject: [gpfsug-discuss] Hardware refresh - In-Reply-To: References: <9E337B56-A274-41B6-84C2-7084AAB771F4@siriuscom.com><279BAB6D-21E6-4EC1-98CD-CA409A24F8F6@ulmer.org> Message-ID: IMO, it is simplest to have both the old and new file systems both mounted on the same node(s). Then you can use AFM and/or any other utilities to migrate/copy your files from the old file system to the new. From: "Shankar Balasubramanian" To: gpfsug main discussion list Date: 10/13/2016 06:46 AM Subject: Re: [gpfsug-discuss] Hardware refresh - Sent by: gpfsug-discuss-bounces at spectrumscale.org Please note - though one of the supported use case of AFM is such migration scenarios, infact AFM does particularly well in faithfully migrating the data when the source and destination cluster are GPFS, the scalability of this solution for multi million/multi terabyte file systems has its own set of challenges. These have to carefully understood and checked if AFM will fit the bill. Best Regards, Shankar Balasubramanian STSM, AFM & Async DR Development IBM Systems Bangalore - Embassy Golf Links India "Marc A Kaplan" ---10/12/2016 11:25:20 PM---Yes, you can AFM within a single cluster, in fact with just a single node. I just set this up on m From: "Marc A Kaplan" To: gpfsug main discussion list Date: 10/12/2016 11:25 PM Subject: Re: [gpfsug-discuss] Hardware refresh - Sent by: gpfsug-discuss-bounces at spectrumscale.org Yes, you can AFM within a single cluster, in fact with just a single node. I just set this up on my toy system: [root at bog-wifi cmvc]# mmlsfileset yy afmlu --afm Filesets in file system 'yy': Name Status Path afmTarget afmlu Linked /yy/afmlu gpfs:///xx [root at bog-wifi cmvc]# mount ... yy on /yy type gpfs (rw,relatime,seclabel) xx on /xx type gpfs (rw,relatime,seclabel) [root at bog-wifi cmvc]# mmafmctl yy getstate Fileset Name Fileset Target Cache State Gateway Node Queue Length Queue numExec ------------ -------------- ------------- ------------ ------------ ------------- afmlu gpfs:///xx Active bog-wifi 0 7 So, you may add nodes, add disks to an existing cluster, upgrade your software, define a new FS, migrate data from an old FS to a new FS then delete nodes and disks that are no longer needed... From: Stephen Ulmer To: gpfsug main discussion list Date: 10/11/2016 09:30 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org I think that the OP was asking why not expand the existing cluster with the new hardware, and just make a new FS? I?ve not tried to make a cluster talk AFM to itself yet. If that?s impossible, then there?s one good reason to make a new cluster (to use AFM for migration). Liberty, -- Stephen On Oct 11, 2016, at 8:40 PM, Mark.Bush at siriuscom.comwrote: Only compelling reason for new cluster would be old hardware is EOL or no longer want to pay maintenance on it. From: on behalf of Marc A Kaplan Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 2:58 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh New FS? Yes there are some good reasons. New cluster? I did not see a compelling argument either way. From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Date: 10/11/2016 03:34 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org Ok. I think I am hearing that a new cluster with a new FS and copying data from old to new cluster is the best way forward. Thanks everyone for your input. From: on behalf of Yuri L Volobuev Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 12:22 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh This depends on the committed cluster version level (minReleaseLevel) and file system format. Since NFSv2 is an on-disk format change, older code wouldn't be able to understand what it is, and thus if there's a possibility of a downlevel node looking at the NSD, the NFSv1 format is going to be used. The code does NSDv1<->NSDv2 conversions under the covers as needed when adding an empty NSD to a file system. I'd strongly recommend getting a fresh start by formatting a new file system. Many things have changed over the course of the last few years. In particular, having a 4K-aligned file system can be a pretty big deal, depending on what hardware one is going to deploy in the future, and this is something that can't be bolted onto an existing file system. Having 4K inodes is very handy for many reasons. New directory format and NSD format changes are attractive, too. And disks generally tend to get larger with time, and at some point you may want to add a disk to an existing storage pool that's larger than the existing allocation map format allows. Obviously, it's more hassle to migrate data to a new file system, as opposed to extending an existing one. In a perfect world, GPFS would offer a conversion tool that seamlessly and robustly converts old file systems, making them as good as new, but in the real world such a tool doesn't exist. Getting a clean slate by formatting a new file system every few years is a good long-term investment of time, although it comes front-loaded with extra work. yuri Aaron Knister ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron From: Aaron Knister To: , Date: 10/10/2016 04:45 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron On 10/10/16 7:40 PM, Luis Bolinches wrote: > Hi > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > reason to do so. > > AFM for migrations is quite good, latest versions allows to use NSD > protocol for mounts as well. Olaf did a great job explaining this > scenario on the redbook chapter 6 > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > -- > Cheers > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > wrote: > >> Hi Mark, >> >> The last time we did something like this was 2010 (we?re doing rolling >> refreshes now), so there are probably lots of better ways to do this >> than what we did, but we: >> >> 1) set up the new hardware >> 2) created new filesystems (so that we could make adjustments we >> wanted to make that can only be made at FS creation time) >> 3) used rsync to make a 1st pass copy of everything >> 4) coordinated a time with users / groups to do a 2nd rsync when they >> weren?t active >> 5) used symbolic links during the transition (i.e. rm -rvf >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >> 6) once everybody was migrated, updated the symlinks (i.e. /home >> became a symlink to /gpfs2/home) >> >> HTHAL? >> >> Kevin >> >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>> wrote: >>> >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>> refresh hardware. Any lessons learned in this process? Is it >>> easiest to just build new cluster and then use AFM? Add to existing >>> cluster then decommission nodes? What is the recommended process for >>> this? >>> >>> >>> Mark >>> >>> This message (including any attachments) is intended only for the use >>> of the individual or entity to which it is addressed and may contain >>> information that is non-public, proprietary, privileged, >>> confidential, and exempt from disclosure under applicable law. If you >>> are not the intended recipient, you are hereby notified that any use, >>> dissemination, distribution, or copying of this communication is >>> strictly prohibited. This message may be viewed by parties at Sirius >>> Computer Solutions other than those named in the message header. This >>> message does not contain an official representation of Sirius >>> Computer Solutions. If you have received this communication in error, >>> notify Sirius Computer Solutions immediately and (i) destroy this >>> message if a facsimile or (ii) delete this message immediately if >>> this is an electronic communication. Thank you. >>> >>> Sirius Computer Solutions >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and >> Education >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 105 bytes Desc: not available URL: From aaron.s.knister at nasa.gov Fri Oct 14 03:01:32 2016 From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]) Date: Fri, 14 Oct 2016 02:01:32 +0000 Subject: [gpfsug-discuss] 4K sector NSD support (was: Hardware refresh) In-Reply-To: References: <63bd99ad-f684-d1bb-371f-e2c21849ff29@nasa.gov> , Message-ID: <5F910253243E6A47B81A9A2EB424BBA101D91022@NDMSMBX404.ndc.nasa.gov> Hi Yuri, Thank you for your excellent reply. The immediate issue ahead of us is getting the 4K LUNs formatted as NSDs into the filesystem and since we can do that life is good for a while. The next hurdle for me won't be for another 3 years when I'll likely need to bring into existing filesystems metadata NSDs that have 4k sectors. I wonder if in that time frame a migration tool could be developed to convert existing filesystems to 4K metadata? I do recognize what you're saying, though, about return on investment of effort from a developing a tool to migrate the on-disk format vs migrating the entire filesystem. What were you thinking as far as ways to strengthen the migration story? In an ideal world I'd like to be able to pull the trigger on a migration and have it have no visible impact to end-users other than an understandable performance impact. If a mechanism existed to move data to a new filesystem (a hypothetical mmreplacefs comes to mind) in-place, that would be quite wonderful. At a high level, perhaps one would create a new filesystem that wouldn't be directly mounted but would be an "internal mount". An admin would then issue an mmreplacefs $OLD_FS $NEW_FS command. All activity would be quiesced on the old filesystem and in the background an AFM-like process would be initiated. Files not on the new fs would be migrated in the background. Once the migration process is complete the old fs would perhaps be renamed and the new fs would effectively take its place. This raises more questions in my mind than it does provide answers, such as how quotas would be handled during the migration, how filesets would get migrated/created, how ILM/DMAPI would be affected during the migration. I'll give it some more thought, but understanding a little more about what you were thinking would help me craft the RFE. -Aaron ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Yuri L Volobuev [volobuev at us.ibm.com] Sent: Wednesday, October 12, 2016 1:44 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] 4K sector NSD support (was: Hardware refresh) Yes, it is possible to add a 4KN dataOnly NSD to a non-4K-aligned file system, as you figured out. This is something we didn't plan on doing originally, but then had to implement based on the feedback from the field. There's clearly a need for this. However, this statement is exactly it -- dataOnly NSDs. The only way to put metadata on a 4KN disk is to use a 4K-aligned file system. There are several kinds of metadata present in non-4K-aligned file system that generate non-4K IOs (512 byte inodes being the biggest problem), and there's no way to work around this short of using the new format, and there's no way to perform a conversion to the new format in-place. You're welcome to submit an RFE, of course, but I'd recommend being pragmatic about the chances of such an RFE being implemented. As you can imagine, the main reason why an all-encompassing file system conversion tool doesn't exist is not GPFS developers having no idea that such a tool is wanted. There are several considerations that conspire to make this an unlikely candidate to ever be implemented: 1) The task is hard and has no finish line. In most GPFS releases, something changes, necessitating an added piece of work for the hypothetical conversion tool, and the matrix of from-to format version combinations gets to be big quite quickly. 2) A file system conversion is something that is needed very infrequently, but when this code does run, it absolutely has to run and run perfectly, else the result would be a half-converted file system, i.e. a royal mess. This is a tester's nightmare. 3) The failure scenarios are all unpalatable. What should the conversion tool do if it runs out of space replacing smaller metadata structures with bigger ones? Undoing a partially finished conversion is even harder than doing it in the first place. 4) Doing an on-disk conversion on-line is simply very difficult. Consider the task of converting an inode file to use a different inode size. The file can be huge (billions of records), and it would take a fair chunk of time to rewrite it, but the file is changing while it's being converted (can't simply lock the whole thing down for so long), simultaneously on multiple nodes. Orchestrating the processing of updates in the presence of two inode files, with proper atomicity guarantees (to guard against a node failure) is a task of considerable complexity. None of this means the task is impossible, of course. It is, however, a very big chunk of very complex work, all towards a tool that on an average cluster may run somewhere between zero and one times, not something that benefits day-to-day operations. Where the complexity of the task allows for a reasonably affordable implementation, e.g. conversion from an old-style EA file to the FASTEA format, a conversion tool has been implemented (mmmigratefs). However, doing this for every single changed aspect of the file system format is simply too expensive to justify, given other tasks in front of us. On the other hand, a well-implemented migration mechanism solves the file system reformatting scenario (which covers all aspects of file system format changes) as well as a number of other scenarios. This is a cleaner, more general solution. Migration doesn't have to mean an outage. A simple rsync-based migration requires downtime for a cutover, while an AFM-based migration doesn't necessarily require one. I'm not saying that GPFS has a particularly strong migration story at the moment, but this is a much more productive direction for applying resources than a mythical all-encompassing conversion tool. yuri [Inactive hide details for Aaron Knister ---10/11/2016 05:59:25 PM---Yuri, (Sorry for being somewhat spammy) I now understand th]Aaron Knister ---10/11/2016 05:59:25 PM---Yuri, (Sorry for being somewhat spammy) I now understand the limitation after From: Aaron Knister To: , Date: 10/11/2016 05:59 PM Subject: Re: [gpfsug-discuss] 4K sector NSD support (was: Hardware refresh) Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Yuri, (Sorry for being somewhat spammy) I now understand the limitation after some more testing (I'm a hands-on learner, can you tell?). Given the right code/cluster/fs version levels I can add 4K dataOnly NSDv2 NSDs to a filesystem created with NSDv1 NSDs. What I can't do is seemingly add any metadataOnly or dataAndMetadata 4K luns to an fs that is not 4K aligned which I assume would be any fs originally created with NSDv1 LUNs. It seems possible to move all data away from NSDv1 LUNS in a filesystem behind-the-scenes using GPFS migration tools, and move the data to NSDv2 LUNs. In this case I believe what's missing is a tool to convert just the metadata structures to be 4K aligned since the data would already on 4k-based NSDv2 LUNS, is that the case? I'm trying to figure out what exactly I'm asking for in an RFE. -Aaron On 10/11/16 7:57 PM, Aaron Knister wrote: > I think I was a little quick to the trigger. I re-read your last mail > after doing some testing and understand it differently. I was wrong > about my interpretation-- you can add 4K NSDv2 formatted NSDs to a > filesystem previously created with NSDv1 NSDs assuming, as you say, the. > minReleaseLevel and filesystem version are high enough. That negates > about half of my last e-mail. The fs still doesn't show as 4K aligned: > > loressd01:~ # /usr/lpp/mmfs/bin/mmlsfs tnb4k --is4KAligned > flag value description > ------------------- ------------------------ > ----------------------------------- > --is4KAligned No is4KAligned? > > but *shrug* most of the I/O to these disks should be 1MB anyway. If > somebody is pounding the FS with smaller than 4K I/O they're gonna get a > talkin' to. > > -Aaron > > On 10/11/16 6:41 PM, Aaron Knister wrote: >> Thanks Yuri. >> >> I'm asking for my own purposes but I think it's still relevant here: >> we're still at GPFS 3.5 and will be adding dataOnly NSDs with 4K sectors >> in the near future. We're planning to update to 4.1 before we format >> these NSDs, though. If I understand you correctly we can't bring these >> 4K NSDv2 NSDs into a filesystem with 512b-based NSDv1 NSDs? That's a >> pretty big deal :( >> >> Reformatting every few years with 10's of petabytes of data is not >> realistic for us (it would take years to move the data around). It also >> goes against my personal preachings about GPFS's storage virtualization >> capabilities: the ability to perform upgrades/make underlying storage >> infrastructure changes with behind-the-scenes data migration, >> eliminating much of the manual hassle of storage administrators doing >> rsync dances. I guess it's RFE time? It also seems as though AFM could >> help with automating the migration, although many of our filesystems do >> not have filesets on them so we would have to re-think how we lay out >> our filesystems. >> >> This is also curious to me with IBM pitching GPFS as a filesystem for >> cloud services (the cloud *never* goes down, right?). Granted I believe >> this pitch started after the NSDv2 format was defined, but if somebody >> is building a large cloud with GPFS as the underlying filesystem for an >> object or an image store one might think the idea of having to re-format >> the filesystem to gain access to critical new features is inconsistent >> with this pitch. It would be hugely impactful. Just my $.02. >> >> As you can tell, I'm frustrated there's no online conversion tool :) Not >> that there couldn't be... you all are brilliant developers. >> >> -Aaron >> >> On 10/11/16 1:22 PM, Yuri L Volobuev wrote: >>> This depends on the committed cluster version level (minReleaseLevel) >>> and file system format. Since NFSv2 is an on-disk format change, older >>> code wouldn't be able to understand what it is, and thus if there's a >>> possibility of a downlevel node looking at the NSD, the NFSv1 format is >>> going to be used. The code does NSDv1<->NSDv2 conversions under the >>> covers as needed when adding an empty NSD to a file system. >>> >>> I'd strongly recommend getting a fresh start by formatting a new file >>> system. Many things have changed over the course of the last few years. >>> In particular, having a 4K-aligned file system can be a pretty big deal, >>> depending on what hardware one is going to deploy in the future, and >>> this is something that can't be bolted onto an existing file system. >>> Having 4K inodes is very handy for many reasons. New directory format >>> and NSD format changes are attractive, too. And disks generally tend to >>> get larger with time, and at some point you may want to add a disk to an >>> existing storage pool that's larger than the existing allocation map >>> format allows. Obviously, it's more hassle to migrate data to a new file >>> system, as opposed to extending an existing one. In a perfect world, >>> GPFS would offer a conversion tool that seamlessly and robustly converts >>> old file systems, making them as good as new, but in the real world such >>> a tool doesn't exist. Getting a clean slate by formatting a new file >>> system every few years is a good long-term investment of time, although >>> it comes front-loaded with extra work. >>> >>> yuri >>> >>> Inactive hide details for Aaron Knister ---10/10/2016 04:45:31 PM---Can >>> one format NSDv2 NSDs and put them in a filesystem withAaron Knister >>> ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a >>> filesystem with NSDv1 NSD's? -Aaron >>> >>> From: Aaron Knister >>> To: , >>> Date: 10/10/2016 04:45 PM >>> Subject: Re: [gpfsug-discuss] Hardware refresh >>> Sent by: gpfsug-discuss-bounces at spectrumscale.org >>> >>> ------------------------------------------------------------------------ >>> >>> >>> >>> Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? >>> >>> -Aaron >>> >>> On 10/10/16 7:40 PM, Luis Bolinches wrote: >>>> Hi >>>> >>>> Creating a new FS sounds like a best way to go. NSDv2 being a very good >>>> reason to do so. >>>> >>>> AFM for migrations is quite good, latest versions allows to use NSD >>>> protocol for mounts as well. Olaf did a great job explaining this >>>> scenario on the redbook chapter 6 >>>> >>>> http://www.redbooks.ibm.com/abstracts/sg248254.html?Open >>>> >>>> -- >>>> Cheers >>>> >>>> On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L >>>> >>> > wrote: >>>> >>>>> Hi Mark, >>>>> >>>>> The last time we did something like this was 2010 (we?re doing rolling >>>>> refreshes now), so there are probably lots of better ways to do this >>>>> than what we did, but we: >>>>> >>>>> 1) set up the new hardware >>>>> 2) created new filesystems (so that we could make adjustments we >>>>> wanted to make that can only be made at FS creation time) >>>>> 3) used rsync to make a 1st pass copy of everything >>>>> 4) coordinated a time with users / groups to do a 2nd rsync when they >>>>> weren?t active >>>>> 5) used symbolic links during the transition (i.e. rm -rvf >>>>> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >>>>> 6) once everybody was migrated, updated the symlinks (i.e. /home >>>>> became a symlink to /gpfs2/home) >>>>> >>>>> HTHAL? >>>>> >>>>> Kevin >>>>> >>>>>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>>>>> wrote: >>>>>> >>>>>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>>>>> refresh hardware. Any lessons learned in this process? Is it >>>>>> easiest to just build new cluster and then use AFM? Add to existing >>>>>> cluster then decommission nodes? What is the recommended process for >>>>>> this? >>>>>> >>>>>> >>>>>> Mark >>>>>> >>>>>> This message (including any attachments) is intended only for the use >>>>>> of the individual or entity to which it is addressed and may contain >>>>>> information that is non-public, proprietary, privileged, >>>>>> confidential, and exempt from disclosure under applicable law. If you >>>>>> are not the intended recipient, you are hereby notified that any use, >>>>>> dissemination, distribution, or copying of this communication is >>>>>> strictly prohibited. This message may be viewed by parties at Sirius >>>>>> Computer Solutions other than those named in the message header. This >>>>>> message does not contain an official representation of Sirius >>>>>> Computer Solutions. If you have received this communication in error, >>>>>> notify Sirius Computer Solutions immediately and (i) destroy this >>>>>> message if a facsimile or (ii) delete this message immediately if >>>>>> this is an electronic communication. Thank you. >>>>>> >>>>>> Sirius Computer Solutions >>>>>> _______________________________________________ >>>>>> gpfsug-discuss mailing list >>>>>> gpfsug-discuss at spectrumscale.org >>>>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>>>> >>>>> ? >>>>> Kevin Buterbaugh - Senior System Administrator >>>>> Vanderbilt University - Advanced Computing Center for Research and >>>>> Education >>>>> Kevin.Buterbaugh at vanderbilt.edu >>>>> - (615)875-9633 >>>>> >>>>> >>>>> >>>> >>>> Ellei edell? ole toisin mainittu: / Unless stated otherwise above: >>>> Oy IBM Finland Ab >>>> PL 265, 00101 Helsinki, Finland >>>> Business ID, Y-tunnus: 0195876-3 >>>> Registered in Finland >>>> >>>> >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> >>> >>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: graycol.gif URL: From daniel.kidger at uk.ibm.com Fri Oct 14 11:09:18 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Fri, 14 Oct 2016 10:09:18 +0000 Subject: [gpfsug-discuss] testing commands In-Reply-To: <797ae533-edee-102a-3f3e-f53687cfcf83@nasa.gov> Message-ID: An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Fri Oct 14 14:30:55 2016 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Fri, 14 Oct 2016 13:30:55 +0000 Subject: [gpfsug-discuss] LROC benefits Message-ID: Hi all, Is there any way to gauge the performance benefits that could be realised from installing some LROCs in our CES cluster nodes? Or just suck it and see as it were? Cheers Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Sat Oct 15 15:23:01 2016 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Sat, 15 Oct 2016 10:23:01 -0400 Subject: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter Message-ID: <1c7562cf-d61a-adda-66b4-5fd00d75ae75@nasa.gov> I've got a node that's got some curious waiters on it (see below). Could someone explain what the "SGExceptionLogBufferFullThread" waiter means? Thanks! -Aaron === mmdiag: waiters === 0x7FFFF040D600 waiting 0.038822715 seconds, SGExceptionLogBufferFullThread: on ThCond 0x7FFFDBB07628 (0x7FFFDBB07628) (parallelWaitCond), reason 'wait for parallel write' for NSD I/O completion on node 10.1.53.5 0x7FFFE83F3D60 waiting 0.039629116 seconds, CleanBufferThread: on ThCond 0x17B1488 (0x17B1488) (MsgRecordCondvar), reason 'RPC wait' for NSD I/O completion on node 10.1.53.7 0x7FFFE8373A90 waiting 0.038921480 seconds, CleanBufferThread: on ThCond 0x7FFFCD2B4E30 (0x7FFFCD2B4E30) (LogFileBufferDescriptorCondvar), reason 'force wait on force active buffer write' 0x42CD9B0 waiting 0.028227004 seconds, CleanBufferThread: on ThCond 0x7FFFCD2B4E30 (0x7FFFCD2B4E30) (LogFileBufferDescriptorCondvar), reason 'force wait for buffer write to complete' 0x7FFFE0F0EAD0 waiting 0.027864343 seconds, CleanBufferThread: on ThCond 0x7FFFDC0EEA88 (0x7FFFDC0EEA88) (MsgRecordCondvar), reason 'RPC wait' for NSD I/O completion on node 10.1.53.7 0x1575560 waiting 0.028045975 seconds, RemoveHandlerThread: on ThCond 0x18020CE4E08 (0xFFFFC90020CE4E08) (LkObjCondvar), reason 'waiting for LX lock' 0x1570560 waiting 0.038724949 seconds, CreateHandlerThread: on ThCond 0x18020CE50A0 (0xFFFFC90020CE50A0) (LkObjCondvar), reason 'waiting for LX lock' 0x1563D60 waiting 0.073919918 seconds, RemoveHandlerThread: on ThCond 0x180235F6440 (0xFFFFC900235F6440) (LkObjCondvar), reason 'waiting for LX lock' 0x1561560 waiting 0.054854513 seconds, RemoveHandlerThread: on ThCond 0x1802292D200 (0xFFFFC9002292D200) (LkObjCondvar), reason 'waiting for LX lock' From oehmes at gmail.com Sat Oct 15 15:50:13 2016 From: oehmes at gmail.com (Sven Oehme) Date: Sat, 15 Oct 2016 14:50:13 +0000 Subject: [gpfsug-discuss] LROC benefits In-Reply-To: References: Message-ID: all depends on workload. we will publish a paper comparing mixed workload with and without LROC pretty soon. most numbers i have seen show anywhere between 30% - 1000% (very rare case) improvements, so its for sure worth a test. sven On Fri, Oct 14, 2016 at 6:31 AM Sobey, Richard A wrote: > Hi all, > > > > Is there any way to gauge the performance benefits that could be realised > from installing some LROCs in our CES cluster nodes? Or just suck it and > see as it were? > > > > Cheers > > Richard > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Sat Oct 15 16:23:57 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Sat, 15 Oct 2016 08:23:57 -0700 Subject: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter In-Reply-To: <1c7562cf-d61a-adda-66b4-5fd00d75ae75@nasa.gov> References: <1c7562cf-d61a-adda-66b4-5fd00d75ae75@nasa.gov> Message-ID: An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Sat Oct 15 16:27:42 2016 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Sat, 15 Oct 2016 11:27:42 -0400 Subject: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter In-Reply-To: References: <1c7562cf-d61a-adda-66b4-5fd00d75ae75@nasa.gov> Message-ID: <96434600-7c7a-d835-e7b5-9d6a80d9add5@nasa.gov> It absolutely does, thanks Olaf! The tasks running on these nodes are running on 63 other nodes and generating ~60K iop/s of metadata writes and I *think* about the same in reads. Do you think that could be contributing to the higher waiter times? I'm not sure quite what the job is up to. It's seemingly doing very little data movement, the cpu %used is very low but the load is rather high. -Aaron On 10/15/16 11:23 AM, Olaf Weiser wrote: > from your file system configuration .. mmfs -L you'll find the > size of the LOG > since release 4.x ..you can change it, but you need to re-mount the FS > on every client , to make the change effective ... > > when a clients initiate writes/changes to GPFS it needs to update its > changes to the log - if this narrows a certain filling degree, GPFS > triggers so called logWrapThreads to write content to disk and so free > space > > with your given numbers ... double digit [ms] waiter times .. you fs > get's probably slowed down.. and there's something suspect with the > storage, because LOG-IOs are rather small and should not take that long > > to give you an example from a healthy environment... the IO times are so > small, that you usually don't see waiters for this.. > > I/O start time RW Buf type disk:sectorNum nSec time ms tag1 > tag2 Disk UID typ NSD node context thread > --------------- -- ----------- ----------------- ----- ------- > --------- --------- ------------------ --- --------------- --------- > ---------- > 06:23:32.358851 W logData 2:524306424 8 0.439 > 0 0 C0A70D08:57CF40D1 cli 192.167.20.17 LogData > SGExceptionLogBufferFullThread > 06:23:33.576367 W logData 1:524257280 8 0.646 > 0 0 C0A70D08:57CF40D0 cli 192.167.20.16 LogData > SGExceptionLogBufferFullThread > 06:23:32.358851 W logData 2:524306424 8 0.439 > 0 0 C0A70D08:57CF40D1 cli 192.167.20.17 LogData > SGExceptionLogBufferFullThread > 06:23:33.576367 W logData 1:524257280 8 0.646 > 0 0 C0A70D08:57CF40D0 cli 192.167.20.16 LogData > SGExceptionLogBufferFullThread > 06:23:32.212426 W iallocSeg 1:524490048 64 0.733 > 2 245 C0A70D08:57CF40D0 cli 192.167.20.16 Logwrap > LogWrapHelperThread > 06:23:32.212412 W logWrap 2:524552192 8 0.755 > 0 179200 C0A70D08:57CF40D1 cli 192.167.20.17 Logwrap > LogWrapHelperThread > 06:23:32.212432 W logWrap 2:525162760 8 0.737 > 0 125473 C0A70D08:57CF40D1 cli 192.167.20.17 Logwrap > LogWrapHelperThread > 06:23:32.212416 W iallocSeg 2:524488384 64 0.763 > 2 347 C0A70D08:57CF40D1 cli 192.167.20.17 Logwrap > LogWrapHelperThread > 06:23:32.212414 W logWrap 2:525266944 8 2.160 > 0 177664 C0A70D08:57CF40D1 cli 192.167.20.17 Logwrap > LogWrapHelperThread > > > hope this helps .. > > > Mit freundlichen Gr??en / Kind regards > > > Olaf Weiser > > EMEA Storage Competence Center Mainz, German / IBM Systems, Storage > Platform, > ------------------------------------------------------------------------------------------------------------------------------------------- > IBM Deutschland > IBM Allee 1 > 71139 Ehningen > Phone: +49-170-579-44-66 > E-Mail: olaf.weiser at de.ibm.com > ------------------------------------------------------------------------------------------------------------------------------------------- > IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter > Gesch?ftsf?hrung: Martina Koederitz (Vorsitzende), Susanne Peter, > Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Markus Koerner > Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht > Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 > > > > From: Aaron Knister > To: gpfsug main discussion list > Date: 10/15/2016 07:23 AM > Subject: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------------------------------------------------ > > > > I've got a node that's got some curious waiters on it (see below). Could > someone explain what the "SGExceptionLogBufferFullThread" waiter means? > > Thanks! > > -Aaron > > === mmdiag: waiters === > 0x7FFFF040D600 waiting 0.038822715 seconds, > SGExceptionLogBufferFullThread: on ThCond 0x7FFFDBB07628 > (0x7FFFDBB07628) (parallelWaitCond), reason 'wait for parallel write' > for NSD I/O completion on node 10.1.53.5 > 0x7FFFE83F3D60 waiting 0.039629116 seconds, CleanBufferThread: on ThCond > 0x17B1488 (0x17B1488) (MsgRecordCondvar), reason 'RPC wait' for NSD I/O > completion on node 10.1.53.7 > 0x7FFFE8373A90 waiting 0.038921480 seconds, CleanBufferThread: on ThCond > 0x7FFFCD2B4E30 (0x7FFFCD2B4E30) (LogFileBufferDescriptorCondvar), reason > 'force wait on force active buffer write' > 0x42CD9B0 waiting 0.028227004 seconds, CleanBufferThread: on ThCond > 0x7FFFCD2B4E30 (0x7FFFCD2B4E30) (LogFileBufferDescriptorCondvar), reason > 'force wait for buffer write to complete' > 0x7FFFE0F0EAD0 waiting 0.027864343 seconds, CleanBufferThread: on ThCond > 0x7FFFDC0EEA88 (0x7FFFDC0EEA88) (MsgRecordCondvar), reason 'RPC wait' > for NSD I/O completion on node 10.1.53.7 > 0x1575560 waiting 0.028045975 seconds, RemoveHandlerThread: on ThCond > 0x18020CE4E08 (0xFFFFC90020CE4E08) (LkObjCondvar), reason 'waiting for > LX lock' > 0x1570560 waiting 0.038724949 seconds, CreateHandlerThread: on ThCond > 0x18020CE50A0 (0xFFFFC90020CE50A0) (LkObjCondvar), reason 'waiting for > LX lock' > 0x1563D60 waiting 0.073919918 seconds, RemoveHandlerThread: on ThCond > 0x180235F6440 (0xFFFFC900235F6440) (LkObjCondvar), reason 'waiting for > LX lock' > 0x1561560 waiting 0.054854513 seconds, RemoveHandlerThread: on ThCond > 0x1802292D200 (0xFFFFC9002292D200) (LkObjCondvar), reason 'waiting for > LX lock' > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From olaf.weiser at de.ibm.com Sat Oct 15 17:46:19 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Sat, 15 Oct 2016 09:46:19 -0700 Subject: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter In-Reply-To: <96434600-7c7a-d835-e7b5-9d6a80d9add5@nasa.gov> References: <1c7562cf-d61a-adda-66b4-5fd00d75ae75@nasa.gov> <96434600-7c7a-d835-e7b5-9d6a80d9add5@nasa.gov> Message-ID: An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Sat Oct 15 18:07:42 2016 From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]) Date: Sat, 15 Oct 2016 17:07:42 +0000 Subject: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter References: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter Message-ID: <5F910253243E6A47B81A9A2EB424BBA101D9277B@NDMSMBX404.ndc.nasa.gov> Understood. Thank you for your help. By the way, I was able to figure out by poking mmpmon gfis that the job is performing 20k a second each of inode creations, updates and deletions across 64 nodes. There's my 60k iops on the backend. While I'm impressed and not surprised GPFS can keep up with this...that's a pretty hefty workload. From: Olaf Weiser Sent: 10/15/16, 12:47 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter well - hard to say.. 60K IO may or may not be a problem... it depends on your storage backends.. check the response times to the physical disk on the NSD server... concerning the output you provided ... check particularly 10.1.53.5 and 10.1.53.7 .... if they are in the same (bad/ poor) range .. then your storage back end is in trouble or maybe just too heavily utilized ... if the response times to physical disks on the NSD server are ok... .. than maybe the network from client <--> NSD server is somehow in trouble .. From: Aaron Knister To: Date: 10/15/2016 08:28 AM Subject: Re: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ It absolutely does, thanks Olaf! The tasks running on these nodes are running on 63 other nodes and generating ~60K iop/s of metadata writes and I *think* about the same in reads. Do you think that could be contributing to the higher waiter times? I'm not sure quite what the job is up to. It's seemingly doing very little data movement, the cpu %used is very low but the load is rather high. -Aaron On 10/15/16 11:23 AM, Olaf Weiser wrote: > from your file system configuration .. mmfs -L you'll find the > size of the LOG > since release 4.x ..you can change it, but you need to re-mount the FS > on every client , to make the change effective ... > > when a clients initiate writes/changes to GPFS it needs to update its > changes to the log - if this narrows a certain filling degree, GPFS > triggers so called logWrapThreads to write content to disk and so free > space > > with your given numbers ... double digit [ms] waiter times .. you fs > get's probably slowed down.. and there's something suspect with the > storage, because LOG-IOs are rather small and should not take that long > > to give you an example from a healthy environment... the IO times are so > small, that you usually don't see waiters for this.. > > I/O start time RW Buf type disk:sectorNum nSec time ms tag1 > tag2 Disk UID typ NSD node context thread > --------------- -- ----------- ----------------- ----- ------- > --------- --------- ------------------ --- --------------- --------- > ---------- > 06:23:32.358851 W logData 2:524306424 8 0.439 > 0 0 C0A70D08:57CF40D1 cli 192.167.20.17 LogData > SGExceptionLogBufferFullThread > 06:23:33.576367 W logData 1:524257280 8 0.646 > 0 0 C0A70D08:57CF40D0 cli 192.167.20.16 LogData > SGExceptionLogBufferFullThread > 06:23:32.358851 W logData 2:524306424 8 0.439 > 0 0 C0A70D08:57CF40D1 cli 192.167.20.17 LogData > SGExceptionLogBufferFullThread > 06:23:33.576367 W logData 1:524257280 8 0.646 > 0 0 C0A70D08:57CF40D0 cli 192.167.20.16 LogData > SGExceptionLogBufferFullThread > 06:23:32.212426 W iallocSeg 1:524490048 64 0.733 > 2 245 C0A70D08:57CF40D0 cli 192.167.20.16 Logwrap > LogWrapHelperThread > 06:23:32.212412 W logWrap 2:524552192 8 0.755 > 0 179200 C0A70D08:57CF40D1 cli 192.167.20.17 Logwrap > LogWrapHelperThread > 06:23:32.212432 W logWrap 2:525162760 8 0.737 > 0 125473 C0A70D08:57CF40D1 cli 192.167.20.17 Logwrap > LogWrapHelperThread > 06:23:32.212416 W iallocSeg 2:524488384 64 0.763 > 2 347 C0A70D08:57CF40D1 cli 192.167.20.17 Logwrap > LogWrapHelperThread > 06:23:32.212414 W logWrap 2:525266944 8 2.160 > 0 177664 C0A70D08:57CF40D1 cli 192.167.20.17 Logwrap > LogWrapHelperThread > > > hope this helps .. > > > Mit freundlichen Gr??en / Kind regards > > > Olaf Weiser > > EMEA Storage Competence Center Mainz, German / IBM Systems, Storage > Platform, > ------------------------------------------------------------------------------------------------------------------------------------------- > IBM Deutschland > IBM Allee 1 > 71139 Ehningen > Phone: +49-170-579-44-66 > E-Mail: olaf.weiser at de.ibm.com > ------------------------------------------------------------------------------------------------------------------------------------------- > IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter > Gesch?ftsf?hrung: Martina Koederitz (Vorsitzende), Susanne Peter, > Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Markus Koerner > Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht > Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 > > > > From: Aaron Knister > To: gpfsug main discussion list > Date: 10/15/2016 07:23 AM > Subject: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------------------------------------------------ > > > > I've got a node that's got some curious waiters on it (see below). Could > someone explain what the "SGExceptionLogBufferFullThread" waiter means? > > Thanks! > > -Aaron > > === mmdiag: waiters === > 0x7FFFF040D600 waiting 0.038822715 seconds, > SGExceptionLogBufferFullThread: on ThCond 0x7FFFDBB07628 > (0x7FFFDBB07628) (parallelWaitCond), reason 'wait for parallel write' > for NSD I/O completion on node 10.1.53.5 > 0x7FFFE83F3D60 waiting 0.039629116 seconds, CleanBufferThread: on ThCond > 0x17B1488 (0x17B1488) (MsgRecordCondvar), reason 'RPC wait' for NSD I/O > completion on node 10.1.53.7 > 0x7FFFE8373A90 waiting 0.038921480 seconds, CleanBufferThread: on ThCond > 0x7FFFCD2B4E30 (0x7FFFCD2B4E30) (LogFileBufferDescriptorCondvar), reason > 'force wait on force active buffer write' > 0x42CD9B0 waiting 0.028227004 seconds, CleanBufferThread: on ThCond > 0x7FFFCD2B4E30 (0x7FFFCD2B4E30) (LogFileBufferDescriptorCondvar), reason > 'force wait for buffer write to complete' > 0x7FFFE0F0EAD0 waiting 0.027864343 seconds, CleanBufferThread: on ThCond > 0x7FFFDC0EEA88 (0x7FFFDC0EEA88) (MsgRecordCondvar), reason 'RPC wait' > for NSD I/O completion on node 10.1.53.7 > 0x1575560 waiting 0.028045975 seconds, RemoveHandlerThread: on ThCond > 0x18020CE4E08 (0xFFFFC90020CE4E08) (LkObjCondvar), reason 'waiting for > LX lock' > 0x1570560 waiting 0.038724949 seconds, CreateHandlerThread: on ThCond > 0x18020CE50A0 (0xFFFFC90020CE50A0) (LkObjCondvar), reason 'waiting for > LX lock' > 0x1563D60 waiting 0.073919918 seconds, RemoveHandlerThread: on ThCond > 0x180235F6440 (0xFFFFC900235F6440) (LkObjCondvar), reason 'waiting for > LX lock' > 0x1561560 waiting 0.054854513 seconds, RemoveHandlerThread: on ThCond > 0x1802292D200 (0xFFFFC9002292D200) (LkObjCondvar), reason 'waiting for > LX lock' > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Sat Oct 15 18:43:34 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Sat, 15 Oct 2016 10:43:34 -0700 Subject: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter In-Reply-To: <5F910253243E6A47B81A9A2EB424BBA101D9277B@NDMSMBX404.ndc.nasa.gov> References: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter <5F910253243E6A47B81A9A2EB424BBA101D9277B@NDMSMBX404.ndc.nasa.gov> Message-ID: An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Mon Oct 17 01:06:09 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 17 Oct 2016 00:06:09 +0000 Subject: [gpfsug-discuss] CES and NFS Tuning suggestions Message-ID: Looking for some pointers or suggestions on what I should look at changing in Linux and/or GPFS "mmchconfg" settings to help boost NFS performance. Out of the box it seems "poor". Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid 507-269-0413 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ckrafft at de.ibm.com Mon Oct 17 14:59:39 2016 From: ckrafft at de.ibm.com (Christoph Krafft) Date: Mon, 17 Oct 2016 15:59:39 +0200 Subject: [gpfsug-discuss] Looking for experiences with Huawei Oceanstore and GPFS / Spectrum Scale Message-ID: Hi folks, has anyone made experiences with Huawei Oceanstore and GPFS - and would be willing to share some details with me? Any helpful hints are deeply appreciated - THANK you in advance! Mit freundlichen Gr??en / Sincerely Christoph Krafft Client Technical Specialist - Power Systems, IBM Systems Certified IT Specialist @ The Open Group Phone: +49 (0) 7034 643 2171 IBM Deutschland GmbH Mobile: +49 (0) 160 97 81 86 12 Am Weiher 24 Email: ckrafft at de.ibm.com 65451 Kelsterbach Germany IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter Gesch?ftsf?hrung: Martina Koederitz (Vorsitzende), Nicole Reimer, Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Markus Koerner Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecblank.gif Type: image/gif Size: 45 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0B092252.gif Type: image/gif Size: 1851 bytes Desc: not available URL: From Robert.Oesterlin at nuance.com Mon Oct 17 16:00:20 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 17 Oct 2016 15:00:20 +0000 Subject: [gpfsug-discuss] CES: IP address won't assign: "handleNetworkProblem with lock held" Message-ID: <0669A3C9-4F5E-4834-A1C4-D8D124B0A4F3@nuance.com> Can anyone help me pinpoint the issue here? These message repeat and the IP addresses never get assigned. [root at tct-gw01 ~]# tail /var/mmfs/gen/mmfslog Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: Found unassigned address 10.30.22.178 Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: Found unassigned address 10.30.22.179 Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: Found unassigned address 10.30.22.177 Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: Found unassigned address 10.30.22.176 Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: handleNetworkProblem with lock held: assignIP 10.30.22.178_0-_+,10.30.22.179_0-_+,10.30.22.177_0-_+,10.30.22.176_0-_+ 1 Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: Assigning addresses: 10.30.22.178_0-_+,10.30.22.179_0-_+,10.30.22.177_0-_+,10.30.22.176_0-_+ Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: moveCesIPs: 10.30.22.178_0-_+,10.30.22.179_0-_+,10.30.22.177_0-_+,10.30.22.176_0-_+ [root at tct-gw01 ~]# mmces state show -a NODE AUTH AUTH_OBJ NETWORK NFS OBJ SMB CES tct-gw01.infra.us.grid.nuance.com DISABLED DISABLED HEALTHY HEALTHY DISABLED DISABLED HEALTHY tct-gw02.infra.us.grid.nuance.com DISABLED DISABLED HEALTHY HEALTHY DISABLED DISABLED HEALTHY tct-gw03.infra.us.grid.nuance.com DISABLED DISABLED HEALTHY HEALTHY DISABLED DISABLED HEALTHY tct-gw04.infra.us.grid.nuance.com DISABLED DISABLED HEALTHY HEALTHY DISABLED DISABLED HEALTHY [root at tct-gw01 ~]# mmces address list Address Node Group Attribute ------------------------------------------------------------------------- 10.30.22.178 unassigned none none 10.30.22.179 unassigned none none 10.30.22.177 unassigned none none 10.30.22.176 unassigned none none Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid 507-269-0413 -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenneth.waegeman at ugent.be Mon Oct 17 16:16:05 2016 From: kenneth.waegeman at ugent.be (Kenneth Waegeman) Date: Mon, 17 Oct 2016 17:16:05 +0200 Subject: [gpfsug-discuss] Disk can't be recovered due to uncorrectable read error in vdisk (GSS) Message-ID: Hi, Currently our file system is down due to down/unrecovered disks. We try to start the disks again with mmchdisk, but when we do this, we see this error in our mmfs.log: Mon Oct 17 15:28:18.122 2016: [E] smallRead VIO failed due to uncorrectable read error in vdisk nsd11_MetaData_8M_3p_2 vtrack 33630 vsector 34437120 number of vsectors 1024 segments 0 to 16 startBufOffset 0 endB ufOffset 1023 vtrkOffset 0 vioLen 524288 This is a 3-way replicated vdisk, and not one of the recovering disks, but this disk is in 'up' state.. Does someone has a clue what we could to recover our fs? Thanks! Kenneth From bbanister at jumptrading.com Mon Oct 17 17:00:22 2016 From: bbanister at jumptrading.com (Bryan Banister) Date: Mon, 17 Oct 2016 16:00:22 +0000 Subject: [gpfsug-discuss] CES and NFS Tuning suggestions In-Reply-To: References: Message-ID: <21BC488F0AEA2245B2C3E83FC0B33DBB06364770@CHI-EXCHANGEW1.w2k.jumptrading.com> One major issue is the maxFilesToCache and maybe the maxStatCache (though I hear that Linux negates the use of this parameter now? I don?t quite remember). Ganesha apparently likes to hold open a large number of files and this means that it will quickly fill up the maxFilesToCache. When this happens the [gpfsSwapdKproc] process will start to eat up CPU time. This is the daemon that tries to find a file to evict from the cache when a new file is opened. This overhead will also hurt performance. IBM in a PMR we opened suggested setting this to something like 5 Million for the protocol nodes. I think we started with 1.5 Million. You have to be mindful of memory requirements on the token servers to handle the total sum of all maxFilesToCache settings from all nodes that mount the file system. Of course the other, standard NFS tuning parameters for number of threads and NFS client mount options still should be adjusted too. Hope that helps, -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Oesterlin, Robert Sent: Sunday, October 16, 2016 7:06 PM To: gpfsug main discussion list Subject: [gpfsug-discuss] CES and NFS Tuning suggestions Looking for some pointers or suggestions on what I should look at changing in Linux and/or GPFS "mmchconfg" settings to help boost NFS performance. Out of the box it seems "poor". Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid 507-269-0413 ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralphbsz at us.ibm.com Mon Oct 17 17:39:18 2016 From: ralphbsz at us.ibm.com (Ralph A Becker-szendy) Date: Mon, 17 Oct 2016 16:39:18 +0000 Subject: [gpfsug-discuss] Disk can't be recovered due to uncorrectable read error in vdisk (GSS) Message-ID: An HTML attachment was scrubbed... URL: From stijn.deweirdt at ugent.be Mon Oct 17 18:35:37 2016 From: stijn.deweirdt at ugent.be (Stijn De Weirdt) Date: Mon, 17 Oct 2016 19:35:37 +0200 Subject: [gpfsug-discuss] Disk can't be recovered due to uncorrectable read error in vdisk (GSS) In-Reply-To: References: Message-ID: <8f93f6bf-6b42-c3bf-11b4-39dc53b4451f@ugent.be> hi ralph, >>Currently our file system is down due to down/unrecovered disks. We >> try to start the disks again with mmchdisk, but when we do this, we >> see this error in our mmfs.log: >> ... >> This is a 3-way replicated vdisk, and not one of the recovering disks,but >> this disk is in 'up' state.. > First, please open a PMR through your normal support organization, and make it > clear in the PMR that the problem is GPFS and GNR (a.k.a. ESS). Like that, it > will be assigned to the correct support group. Support will request that you > upload a snap. PMR is created, but we are in particular puzzled by the IO read error. and getting some details about what is gone here (with details as you provided) is somethign we usually do not get from support ;) > There seems to be a combination of two problems here: > One, a NSD (which is also a GNR vdisk) is down, which is usually caused by an IO > error on the vdisk, or by both servers for the recovery group that contains the > vdisk being down simultaneously. Usually, that is easily fixed by running > mmchdisk with a start option, but you tried that and it didn't work. This > problem is at the NSD layer (meaning in the GPFS client that accesses the GNR > vdisk), not in the GNR layer. it's actually more than disk down, all on same recoverygroup. the vdisk with the read error is on the other recoverygroup (only 2, this is/was a GSS24) > Second, another vdisk has an internal error, caused by read error from the > physical disks (which is what "uncorrectable read error" means). what does physical mean here? we have no IO error anywhere (OS/kernel/scsi, also the error counters on the mmlspdisk output do not increase). Now, give that > you say that this vdisk is 3-way replicated, that probably means that there are > multiple problems. This error is purely in the GNR layer, and the error message > you quote "smallRead VIO..." comes from the GNR layer. Now, an error from one > vdisk can't prevent mmchdisk on a different vdisk from working, so these two > problems seem unrelated. well, they can: the disks with all metadata replica on the one recoverygroup are down. starting those forces the read of the ones on the other group, and this runs into the IOread error, and everything stops (well, that's how we understand it ;) > Furthermore, I'm going to bet that the two problems (which at first seem > unrelated) must in reality have a common root cause; it would be too weird a > coincidence to get two problems that are unrelated at the same time. To debug > this requires looking at way more information than a single line from the > mmfs.log file, which is why the support organization needs a complete PMR > opened, and then have the usual snap (with logs, dumps, ...) uploaded, so it can > see what the cause of the problem is. yep, trace and snap uploaded. > Good luck! thanks (and thanks again for some insights, much appreciated !) > Ralph Becker-Szendy > IBM Almaden Research Center - Computer Science -Storage Systems > ralphbsz at us.ibm.com > 408-927-2752 > 650 Harry Road, K56-B3, San Jose, CA 95120 > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From olaf.weiser at de.ibm.com Mon Oct 17 18:40:13 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Mon, 17 Oct 2016 10:40:13 -0700 Subject: [gpfsug-discuss] CES and NFS Tuning suggestions In-Reply-To: <21BC488F0AEA2245B2C3E83FC0B33DBB06364770@CHI-EXCHANGEW1.w2k.jumptrading.com> References: <21BC488F0AEA2245B2C3E83FC0B33DBB06364770@CHI-EXCHANGEW1.w2k.jumptrading.com> Message-ID: An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Mon Oct 17 19:27:01 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Mon, 17 Oct 2016 11:27:01 -0700 Subject: [gpfsug-discuss] CES: IP address won't assign: "handleNetworkProblem with lock held" In-Reply-To: <0669A3C9-4F5E-4834-A1C4-D8D124B0A4F3@nuance.com> References: <0669A3C9-4F5E-4834-A1C4-D8D124B0A4F3@nuance.com> Message-ID: An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Mon Oct 17 19:42:58 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 17 Oct 2016 18:42:58 +0000 Subject: [gpfsug-discuss] CES: IP address won't assign: "handleNetworkProblem with lock held" Message-ID: <99C9D37E-3387-48A6-88BB-DFE711C7CC8F@nuance.com> Yes - so interesting - it looks like the nodes have the addresses assigned but CES doesn?t know that. [root at tct-gw01 ~]# mmlscluster --ces GPFS cluster information ======================== GPFS cluster name: nrg1-tct.nrg1.us.grid.nuance.com GPFS cluster id: 17869514639699411874 Cluster Export Services global parameters ----------------------------------------- Shared root directory: /gpfs/fs1/ces Enabled Services: NFS Log level: 0 Address distribution policy: even-coverage Node Daemon node name IP address CES IP address list ----------------------------------------------------------------------- 4 tct-gw01.infra.us.grid.nuance.com 10.30.22.160 None 5 tct-gw02.infra.us.grid.nuance.com 10.30.22.161 None 6 tct-gw03.infra.us.grid.nuance.com 10.30.22.162 None 7 tct-gw04.infra.us.grid.nuance.com 10.30.22.163 None [root at tct-gw01 ~]# mmdsh -N cesnodes "ip a | grep 10.30.22." | sort -k 1 tct-gw01.infra.us.grid.nuance.com: inet 10.30.22.160/24 brd 10.30.22.255 scope global bond0 tct-gw01.infra.us.grid.nuance.com: inet 10.30.22.176/24 brd 10.30.22.255 scope global secondary bond0:0 tct-gw02.infra.us.grid.nuance.com: inet 10.30.22.161/24 brd 10.30.22.255 scope global bond0 tct-gw02.infra.us.grid.nuance.com: inet 10.30.22.177/24 brd 10.30.22.255 scope global secondary bond0:0 tct-gw02.infra.us.grid.nuance.com: inet 10.30.22.178/24 brd 10.30.22.255 scope global secondary bond0:1 tct-gw03.infra.us.grid.nuance.com: inet 10.30.22.162/24 brd 10.30.22.255 scope global bond0 tct-gw03.infra.us.grid.nuance.com: inet 10.30.22.178/24 brd 10.30.22.255 scope global secondary bond0:0 tct-gw04.infra.us.grid.nuance.com: inet 10.30.22.163/24 brd 10.30.22.255 scope global bond0 tct-gw04.infra.us.grid.nuance.com: inet 10.30.22.179/24 brd 10.30.22.255 scope global secondary bond0:0 Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: on behalf of Olaf Weiser Reply-To: gpfsug main discussion list Date: Monday, October 17, 2016 at 1:27 PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] CES: IP address won't assign: "handleNetworkProblem with lock held" ip a | grep 10.30.22. -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Mon Oct 17 19:53:01 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Mon, 17 Oct 2016 18:53:01 +0000 Subject: [gpfsug-discuss] CES: IP address won't assign: "handleNetworkProblem with lock held" In-Reply-To: <99C9D37E-3387-48A6-88BB-DFE711C7CC8F@nuance.com> References: <99C9D37E-3387-48A6-88BB-DFE711C7CC8F@nuance.com> Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 37817 bytes Desc: not available URL: From S.J.Thompson at bham.ac.uk Mon Oct 17 19:57:15 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Mon, 17 Oct 2016 18:57:15 +0000 Subject: [gpfsug-discuss] CES: IP address won't assign: "handleNetworkProblem with lock held" In-Reply-To: References: <0669A3C9-4F5E-4834-A1C4-D8D124B0A4F3@nuance.com>, Message-ID: Does it strictly follow this? We were doing some testing with tap interfaces into vxlan networks and found that if we simulated taking down the vxlan interface (which appears in ifconfig as a physical int really), then it moved the ces ip onto the box's primary Nic which was a different subnet. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Olaf Weiser [olaf.weiser at de.ibm.com] Sent: 17 October 2016 19:27 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] CES: IP address won't assign: "handleNetworkProblem with lock held" simple question -sorry for that - your Nodes.. do they have an IP address in the same subnet as your IP address listed here ? and if, is this network up n running so that GPFS can find/detect it ? what tells mmlscluster --ces ? from each node - assuming class C /24 network , do a ip a | grep 10.30.22. cheers From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 10/17/2016 08:00 AM Subject: [gpfsug-discuss] CES: IP address won't assign: "handleNetworkProblem with lock held" Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Can anyone help me pinpoint the issue here? These message repeat and the IP addresses never get assigned. [root at tct-gw01 ~]# tail /var/mmfs/gen/mmfslog Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: Found unassigned address 10.30.22.178 Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: Found unassigned address 10.30.22.179 Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: Found unassigned address 10.30.22.177 Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: Found unassigned address 10.30.22.176 Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: handleNetworkProblem with lock held: assignIP 10.30.22.178_0-_+,10.30.22.179_0-_+,10.30.22.177_0-_+,10.30.22.176_0-_+ 1 Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: Assigning addresses: 10.30.22.178_0-_+,10.30.22.179_0-_+,10.30.22.177_0-_+,10.30.22.176_0-_+ Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: moveCesIPs: 10.30.22.178_0-_+,10.30.22.179_0-_+,10.30.22.177_0-_+,10.30.22.176_0-_+ [root at tct-gw01 ~]# mmces state show -a NODE AUTH AUTH_OBJ NETWORK NFS OBJ SMB CES tct-gw01.infra.us.grid.nuance.com DISABLED DISABLED HEALTHY HEALTHY DISABLED DISABLED HEALTHY tct-gw02.infra.us.grid.nuance.com DISABLED DISABLED HEALTHY HEALTHY DISABLED DISABLED HEALTHY tct-gw03.infra.us.grid.nuance.com DISABLED DISABLED HEALTHY HEALTHY DISABLED DISABLED HEALTHY tct-gw04.infra.us.grid.nuance.com DISABLED DISABLED HEALTHY HEALTHY DISABLED DISABLED HEALTHY [root at tct-gw01 ~]# mmces address list Address Node Group Attribute ------------------------------------------------------------------------- 10.30.22.178 unassigned none none 10.30.22.179 unassigned none none 10.30.22.177 unassigned none none 10.30.22.176 unassigned none none Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid 507-269-0413 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Robert.Oesterlin at nuance.com Mon Oct 17 20:01:48 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 17 Oct 2016 19:01:48 +0000 Subject: [gpfsug-discuss] [EXTERNAL] Re: CES: IP address won't assign: "handleNetworkProblem with lock held" In-Reply-To: References: <99C9D37E-3387-48A6-88BB-DFE711C7CC8F@nuance.com> Message-ID: <0E4224B9-F266-4BB2-9B6A-F580976C2207@nuance.com> No - the :0 and :1 address are floating addresses *assigned by CES* - it created those interfaces. The issue seems to be that these are assigned and CES doesn't know it. Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: on behalf of Olaf Weiser Reply-To: gpfsug main discussion list Date: Monday, October 17, 2016 at 1:53 PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] CES: IP address won't assign: "handleNetworkProblem with lock held" ah .. I see.. seems, that you already has IP aliases around .. GPFS don't like it... eg. your node tct-gw01.infra.us.grid.nuance.com: inet 10.30.22.160/24 has already an alias - 10.30.22.176 ... if I understand you answers correctly... from the doc'... [...] you need to provide a static IP adress, that is not already[...] as an alias [...] -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Mon Oct 17 21:13:28 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Mon, 17 Oct 2016 13:13:28 -0700 Subject: [gpfsug-discuss] CES: IP address won't assign: "handleNetworkProblem with lock held" In-Reply-To: References: <0669A3C9-4F5E-4834-A1C4-D8D124B0A4F3@nuance.com>, Message-ID: An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Mon Oct 17 21:13:28 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Mon, 17 Oct 2016 13:13:28 -0700 Subject: [gpfsug-discuss] [EXTERNAL] Re: CES: IP address won't assign: "handleNetworkProblem with lock held" In-Reply-To: <0E4224B9-F266-4BB2-9B6A-F580976C2207@nuance.com> References: <99C9D37E-3387-48A6-88BB-DFE711C7CC8F@nuance.com> <0E4224B9-F266-4BB2-9B6A-F580976C2207@nuance.com> Message-ID: An HTML attachment was scrubbed... URL: From jake.carroll at uq.edu.au Tue Oct 18 05:21:12 2016 From: jake.carroll at uq.edu.au (Jake Carroll) Date: Tue, 18 Oct 2016 04:21:12 +0000 Subject: [gpfsug-discuss] Any spaces left for a user-presentation at the UG, SC16? Message-ID: <1EECF1D7-DD5E-47BD-A99F-57A4E0294442@uq.edu.au> Hi, I have something interesting that I think the user group might find novel, or at least, hopefully interesting, at the UG meetup at SC16. It would entail an unusual use-case for AFM and some of the unusual things we are doing with it. All good if no spots left ? but I?d be happy to present something if the group thinks it would be beneficial. Thanks! -jc From Robert.Oesterlin at nuance.com Tue Oct 18 12:37:17 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Tue, 18 Oct 2016 11:37:17 +0000 Subject: [gpfsug-discuss] Any spaces left for a user-presentation at the UG, SC16? Message-ID: <7A87F87D-8345-4254-B39A-DE38B61D63AB@nuance.com> Hi Jake At this point, we're pretty well filled up. And yes, you should be seeing a final agenda "soon" - still shuffling things around a bit, trying to get as much great content as we can into one afternoon Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid GPFS UG Co-Principal From: on behalf of Jake Carroll Reply-To: gpfsug main discussion list Date: Monday, October 17, 2016 at 11:21 PM To: "gpfsug-discuss at spectrumscale.org" Subject: [EXTERNAL] [gpfsug-discuss] Any spaces left for a user-presentation at the UG, SC16? Hi, I have something interesting that I think the user group might find novel, or at least, hopefully interesting, at the UG meetup at SC16. It would entail an unusual use-case for AFM and some of the unusual things we are doing with it. All good if no spots left ? but I?d be happy to present something if the group thinks it would be beneficial. Thanks! -jc _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwIGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=SI4sTXt3Q32Lv6WKkeOdNHvhsvEAdzOYPmJMU0cqe38&s=UiXMs0ix00_xIWkODVa9o_km3OQ5mSVKXkE_w6lC8ls&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From kallbac at iu.edu Tue Oct 18 14:26:20 2016 From: kallbac at iu.edu (Kallback-Rose, Kristy A) Date: Tue, 18 Oct 2016 13:26:20 +0000 Subject: [gpfsug-discuss] Any spaces left for a user-presentation at the UG, SC16? In-Reply-To: <7A87F87D-8345-4254-B39A-DE38B61D63AB@nuance.com> References: <7A87F87D-8345-4254-B39A-DE38B61D63AB@nuance.com> Message-ID: JC, can you email a little more on the AFM topic to me and Bob? For future reference if nothing else. Thanks, Kristy On Oct 18, 2016, at 7:37 AM, Oesterlin, Robert > wrote: Hi Jake At this point, we're pretty well filled up. And yes, you should be seeing a final agenda "soon" - still shuffling things around a bit, trying to get as much great content as we can into one afternoon Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid GPFS UG Co-Principal From: > on behalf of Jake Carroll > Reply-To: gpfsug main discussion list > Date: Monday, October 17, 2016 at 11:21 PM To: "gpfsug-discuss at spectrumscale.org" > Subject: [EXTERNAL] [gpfsug-discuss] Any spaces left for a user-presentation at the UG, SC16? Hi, I have something interesting that I think the user group might find novel, or at least, hopefully interesting, at the UG meetup at SC16. It would entail an unusual use-case for AFM and some of the unusual things we are doing with it. All good if no spots left ? but I?d be happy to present something if the group thinks it would be beneficial. Thanks! -jc _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwIGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=SI4sTXt3Q32Lv6WKkeOdNHvhsvEAdzOYPmJMU0cqe38&s=UiXMs0ix00_xIWkODVa9o_km3OQ5mSVKXkE_w6lC8ls&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.childs at qmul.ac.uk Wed Oct 19 15:12:41 2016 From: p.childs at qmul.ac.uk (Peter Childs) Date: Wed, 19 Oct 2016 14:12:41 +0000 Subject: [gpfsug-discuss] Using AFM to migrate files. In-Reply-To: References: <7j173edatp91hqocoto2bau8.1476818437765@email.android.com>, Message-ID: We are planning to use AFM to migrate our old GPFS file store to a new GPFS file store. This will give us the advantages of Spectrum Scale (GPFS) 4.2, such as larger block and inode size. I would like to attempt to gain some insight on my plans before I start. The old file store was running GPFS 3.5 with 512 byte inodes and 1MB block size. We have now upgraded it to 4.1 and are working towards 4.2 with 300TB of files. (385TB max space) this is so we can use both the old and new storage via multi-cluster. We are moving to a new GPFS cluster so we can use the new protocol nodes eventually and also put the new storage machines as cluster managers, as this should be faster and future proof The new hardware has 1PB of space running GPFS 4.2 We have multiple filesets, and would like to maintain our namespace as far as possible. My plan was to. 1. Create a read-only (RO) AFM cache on the new storage (ro) 2a. Move old fileset and replace with SymLink to new. 2b. Convert RO AFM to Local Update (LU) AFM pointing to new parking area of old files. 2c. move user access to new location in cache. 3. Flush everything into cache and disconnect. I've read the docs including the ones on migration but it's not clear if it's safe to move the home of a cache and update the target. It looks like it should be possible and my tests say it works. An alternative plan is to use a Independent Writer (IW) AFM Cache to move the home directories which are pointed to by LDAP. Hence we can move users one at a time and only have to drain the HPC cluster at the end to disconnect the cache. I assume that migrating users over an Independent Writer is safe so long as the users don't use both sides of the cache at once (ie home and target) I'm also interested in any recipe people have on GPFS policies to preseed and flush the cache. We plan to do all the migration using AFM over GPFS we're not currently using NFS and have no plans to start. I believe using GPFS is the faster method to preform the migration. Any suggestions and experience of doing similar migration jobs would be helpful. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London From mimarsh2 at vt.edu Wed Oct 19 18:46:02 2016 From: mimarsh2 at vt.edu (Brian Marshall) Date: Wed, 19 Oct 2016 13:46:02 -0400 Subject: [gpfsug-discuss] subnets Message-ID: All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Wed Oct 19 19:10:38 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Wed, 19 Oct 2016 18:10:38 +0000 Subject: [gpfsug-discuss] subnets In-Reply-To: References: Message-ID: mmdiag --network Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Brian Marshall [mimarsh2 at vt.edu] Sent: 19 October 2016 18:46 To: gpfsug main discussion list Subject: [gpfsug-discuss] subnets All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall From UWEFALKE at de.ibm.com Wed Oct 19 19:15:52 2016 From: UWEFALKE at de.ibm.com (Uwe Falke) Date: Wed, 19 Oct 2016 20:15:52 +0200 Subject: [gpfsug-discuss] subnets In-Reply-To: References: Message-ID: Hi Brian, you might use mmfsadm saferdump tscomm to check on which route peer cluster members are reached. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Frank Hammer, Thorsten Moehring Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: Brian Marshall To: gpfsug main discussion list Date: 10/19/2016 07:46 PM Subject: [gpfsug-discuss] subnets Sent by: gpfsug-discuss-bounces at spectrumscale.org All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Kevin.Buterbaugh at Vanderbilt.Edu Wed Oct 19 21:11:57 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Wed, 19 Oct 2016 20:11:57 +0000 Subject: [gpfsug-discuss] Will there be any more GPFS 4.2.0-x releases? Message-ID: <142FECE0-E157-42D9-BC10-4C48E78FA065@vanderbilt.edu> Hi All, We?re currently running GPFS 4.2.0-4 with an efix installed and now we need a 2nd efix. I?m not a big fan of adding efix to efix and would prefer to go to a new PTF that contains both efixes. So ? is there going to be a GPFS 4.2.0-5 (it?s been a longer than normal interval since PTF 4 came out) or do we need to go to GPFS 4.2.1-x? If the latter, any major changes to watch out for? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bpappas at dstonline.com Wed Oct 19 21:44:39 2016 From: bpappas at dstonline.com (Bill Pappas) Date: Wed, 19 Oct 2016 20:44:39 +0000 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) In-Reply-To: References: Message-ID: I have some ideas to suggest given some of my experiences. First, I have some questions: How many files are you migrating? Will you be creating multiple file sets on the target system based off of business or project needs? Like, file set a is for "department a" and file set b is for "large scale project a" Thanks. Bill Pappas 901-619-0585 bpappas at dstonline.com [1466780990050_DSTlogo.png] [http://www.prweb.com/releases/2016/06/prweb13504050.htm] http://www.prweb.com/releases/2016/06/prweb13504050.htm ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org Sent: Wednesday, October 19, 2016 3:12 PM To: gpfsug-discuss at spectrumscale.org Subject: gpfsug-discuss Digest, Vol 57, Issue 49 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Using AFM to migrate files. (Peter Childs) 2. subnets (Brian Marshall) 3. Re: subnets (Simon Thompson (Research Computing - IT Services)) 4. Re: subnets (Uwe Falke) 5. Will there be any more GPFS 4.2.0-x releases? (Buterbaugh, Kevin L) ---------------------------------------------------------------------- Message: 1 Date: Wed, 19 Oct 2016 14:12:41 +0000 From: Peter Childs To: gpfsug main discussion list Subject: [gpfsug-discuss] Using AFM to migrate files. Message-ID: Content-Type: text/plain; charset="iso-8859-1" We are planning to use AFM to migrate our old GPFS file store to a new GPFS file store. This will give us the advantages of Spectrum Scale (GPFS) 4.2, such as larger block and inode size. I would like to attempt to gain some insight on my plans before I start. The old file store was running GPFS 3.5 with 512 byte inodes and 1MB block size. We have now upgraded it to 4.1 and are working towards 4.2 with 300TB of files. (385TB max space) this is so we can use both the old and new storage via multi-cluster. We are moving to a new GPFS cluster so we can use the new protocol nodes eventually and also put the new storage machines as cluster managers, as this should be faster and future proof The new hardware has 1PB of space running GPFS 4.2 We have multiple filesets, and would like to maintain our namespace as far as possible. My plan was to. 1. Create a read-only (RO) AFM cache on the new storage (ro) 2a. Move old fileset and replace with SymLink to new. 2b. Convert RO AFM to Local Update (LU) AFM pointing to new parking area of old files. 2c. move user access to new location in cache. 3. Flush everything into cache and disconnect. I've read the docs including the ones on migration but it's not clear if it's safe to move the home of a cache and update the target. It looks like it should be possible and my tests say it works. An alternative plan is to use a Independent Writer (IW) AFM Cache to move the home directories which are pointed to by LDAP. Hence we can move users one at a time and only have to drain the HPC cluster at the end to disconnect the cache. I assume that migrating users over an Independent Writer is safe so long as the users don't use both sides of the cache at once (ie home and target) I'm also interested in any recipe people have on GPFS policies to preseed and flush the cache. We plan to do all the migration using AFM over GPFS we're not currently using NFS and have no plans to start. I believe using GPFS is the faster method to preform the migration. Any suggestions and experience of doing similar migration jobs would be helpful. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ------------------------------ Message: 2 Date: Wed, 19 Oct 2016 13:46:02 -0400 From: Brian Marshall To: gpfsug main discussion list Subject: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="utf-8" All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 3 Date: Wed, 19 Oct 2016 18:10:38 +0000 From: "Simon Thompson (Research Computing - IT Services)" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="us-ascii" mmdiag --network Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Brian Marshall [mimarsh2 at vt.edu] Sent: 19 October 2016 18:46 To: gpfsug main discussion list Subject: [gpfsug-discuss] subnets All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall ------------------------------ Message: 4 Date: Wed, 19 Oct 2016 20:15:52 +0200 From: "Uwe Falke" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="ISO-8859-1" Hi Brian, you might use mmfsadm saferdump tscomm to check on which route peer cluster members are reached. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Frank Hammer, Thorsten Moehring Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: Brian Marshall To: gpfsug main discussion list Date: 10/19/2016 07:46 PM Subject: [gpfsug-discuss] subnets Sent by: gpfsug-discuss-bounces at spectrumscale.org All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ------------------------------ Message: 5 Date: Wed, 19 Oct 2016 20:11:57 +0000 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Subject: [gpfsug-discuss] Will there be any more GPFS 4.2.0-x releases? Message-ID: <142FECE0-E157-42D9-BC10-4C48E78FA065 at vanderbilt.edu> Content-Type: text/plain; charset="utf-8" Hi All, We?re currently running GPFS 4.2.0-4 with an efix installed and now we need a 2nd efix. I?m not a big fan of adding efix to efix and would prefer to go to a new PTF that contains both efixes. So ? is there going to be a GPFS 4.2.0-5 (it?s been a longer than normal interval since PTF 4 came out) or do we need to go to GPFS 4.2.1-x? If the latter, any major changes to watch out for? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 49 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-1466780990050_DSTlogo.png.png Type: image/png Size: 6282 bytes Desc: OutlookEmoji-1466780990050_DSTlogo.png.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg Type: image/jpeg Size: 14887 bytes Desc: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg URL: From knop at us.ibm.com Wed Oct 19 22:35:43 2016 From: knop at us.ibm.com (Felipe Knop) Date: Wed, 19 Oct 2016 17:35:43 -0400 Subject: [gpfsug-discuss] Will there be any more GPFS 4.2.0-x releases? In-Reply-To: <142FECE0-E157-42D9-BC10-4C48E78FA065@vanderbilt.edu> References: <142FECE0-E157-42D9-BC10-4C48E78FA065@vanderbilt.edu> Message-ID: Kevin, There will no longer be further PTFs on the 4.2.0 stream. Fixes are now provided on 4.2.1. See https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.1/com.ibm.spectrum.scale.v4r21.doc/bl1xx_soc.htm for the functional enhancements for 4.2.1. See https://www.ibm.com/developerworks/community/forums/html/topic?id=14de7136-e7da-4f93-9f50-5981af1b3f54&ps=50 for announcements on 4.2.1, including the changelog for each PTF. Felipe ---- Felipe Knop knop at us.ibm.com GPFS Development and Security IBM Systems IBM Building 008 2455 South Rd, Poughkeepsie, NY 12601 (845) 433-9314 T/L 293-9314 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 10/19/2016 04:12 PM Subject: [gpfsug-discuss] Will there be any more GPFS 4.2.0-x releases? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, We?re currently running GPFS 4.2.0-4 with an efix installed and now we need a 2nd efix. I?m not a big fan of adding efix to efix and would prefer to go to a new PTF that contains both efixes. So ? is there going to be a GPFS 4.2.0-5 (it?s been a longer than normal interval since PTF 4 came out) or do we need to go to GPFS 4.2.1-x? If the latter, any major changes to watch out for? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From douglasof at us.ibm.com Thu Oct 20 01:07:17 2016 From: douglasof at us.ibm.com (Douglas O'flaherty) Date: Wed, 19 Oct 2016 20:07:17 -0400 Subject: [gpfsug-discuss] SC16 & U/G Registration Message-ID: All: Posting... or re-posting, the URLs for registration to the IBM Spectrum Scale User Group and other IBM briefings IBM summary page: http://www-03.ibm.com/systems/technicalcomputing/supercomputing/index.html Link to IBM event and registrations https://www-01.ibm.com/events/wwe/grp/grp305.nsf/Agenda.xsp?openform&seminar=357M7UES&locale=en_US&S_TACT=000001CP&S_OFF_CD=10001235 We are ramping up for a fantastic event at SC16. Keep an ear out for our next announcements in November, which we will cover in both the user group and more deeply in the IBM seminars and briefings. For those not attending SC16, we will have a briefing webinar on the latest advances in IBM Spectrum Scale and ESS on 11/14. Registration link to follow. doug IBM Spectrum Storage PMM -------------- next part -------------- An HTML attachment was scrubbed... URL: From YARD at il.ibm.com Thu Oct 20 07:15:54 2016 From: YARD at il.ibm.com (Yaron Daniel) Date: Thu, 20 Oct 2016 09:15:54 +0300 Subject: [gpfsug-discuss] Using AFM to migrate files. In-Reply-To: References: <7j173edatp91hqocoto2bau8.1476818437765@email.android.com>, Message-ID: Hi Does you use NFSv4 acls in your old cluster ? Regards Yaron Daniel 94 Em Ha'Moshavot Rd Server, Storage and Data Services - Team Leader Petach Tiqva, 49527 Global Technology Services Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com IBM Israel From: Peter Childs To: gpfsug main discussion list Date: 10/19/2016 05:34 PM Subject: [gpfsug-discuss] Using AFM to migrate files. Sent by: gpfsug-discuss-bounces at spectrumscale.org We are planning to use AFM to migrate our old GPFS file store to a new GPFS file store. This will give us the advantages of Spectrum Scale (GPFS) 4.2, such as larger block and inode size. I would like to attempt to gain some insight on my plans before I start. The old file store was running GPFS 3.5 with 512 byte inodes and 1MB block size. We have now upgraded it to 4.1 and are working towards 4.2 with 300TB of files. (385TB max space) this is so we can use both the old and new storage via multi-cluster. We are moving to a new GPFS cluster so we can use the new protocol nodes eventually and also put the new storage machines as cluster managers, as this should be faster and future proof The new hardware has 1PB of space running GPFS 4.2 We have multiple filesets, and would like to maintain our namespace as far as possible. My plan was to. 1. Create a read-only (RO) AFM cache on the new storage (ro) 2a. Move old fileset and replace with SymLink to new. 2b. Convert RO AFM to Local Update (LU) AFM pointing to new parking area of old files. 2c. move user access to new location in cache. 3. Flush everything into cache and disconnect. I've read the docs including the ones on migration but it's not clear if it's safe to move the home of a cache and update the target. It looks like it should be possible and my tests say it works. An alternative plan is to use a Independent Writer (IW) AFM Cache to move the home directories which are pointed to by LDAP. Hence we can move users one at a time and only have to drain the HPC cluster at the end to disconnect the cache. I assume that migrating users over an Independent Writer is safe so long as the users don't use both sides of the cache at once (ie home and target) I'm also interested in any recipe people have on GPFS policies to preseed and flush the cache. We plan to do all the migration using AFM over GPFS we're not currently using NFS and have no plans to start. I believe using GPFS is the faster method to preform the migration. Any suggestions and experience of doing similar migration jobs would be helpful. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1851 bytes Desc: not available URL: From p.childs at qmul.ac.uk Thu Oct 20 11:12:36 2016 From: p.childs at qmul.ac.uk (Peter Childs) Date: Thu, 20 Oct 2016 10:12:36 +0000 Subject: [gpfsug-discuss] Using AFM to migrate files. In-Reply-To: References: <7j173edatp91hqocoto2bau8.1476818437765@email.android.com>, , Message-ID: Yes but not a great deal, Peter Childs Research Storage Expert ITS Research Infrastructure Queen Mary, University of London ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Yaron Daniel Sent: Thursday, October 20, 2016 7:15:54 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Using AFM to migrate files. Hi Does you use NFSv4 acls in your old cluster ? Regards ________________________________ Yaron Daniel 94 Em Ha'Moshavot Rd [cid:_1_09E5055809E4FFC4002269E8C2258052] Server, Storage and Data Services- Team Leader Petach Tiqva, 49527 Global Technology Services Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com IBM Israel From: Peter Childs To: gpfsug main discussion list Date: 10/19/2016 05:34 PM Subject: [gpfsug-discuss] Using AFM to migrate files. Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ We are planning to use AFM to migrate our old GPFS file store to a new GPFS file store. This will give us the advantages of Spectrum Scale (GPFS) 4.2, such as larger block and inode size. I would like to attempt to gain some insight on my plans before I start. The old file store was running GPFS 3.5 with 512 byte inodes and 1MB block size. We have now upgraded it to 4.1 and are working towards 4.2 with 300TB of files. (385TB max space) this is so we can use both the old and new storage via multi-cluster. We are moving to a new GPFS cluster so we can use the new protocol nodes eventually and also put the new storage machines as cluster managers, as this should be faster and future proof The new hardware has 1PB of space running GPFS 4.2 We have multiple filesets, and would like to maintain our namespace as far as possible. My plan was to. 1. Create a read-only (RO) AFM cache on the new storage (ro) 2a. Move old fileset and replace with SymLink to new. 2b. Convert RO AFM to Local Update (LU) AFM pointing to new parking area of old files. 2c. move user access to new location in cache. 3. Flush everything into cache and disconnect. I've read the docs including the ones on migration but it's not clear if it's safe to move the home of a cache and update the target. It looks like it should be possible and my tests say it works. An alternative plan is to use a Independent Writer (IW) AFM Cache to move the home directories which are pointed to by LDAP. Hence we can move users one at a time and only have to drain the HPC cluster at the end to disconnect the cache. I assume that migrating users over an Independent Writer is safe so long as the users don't use both sides of the cache at once (ie home and target) I'm also interested in any recipe people have on GPFS policies to preseed and flush the cache. We plan to do all the migration using AFM over GPFS we're not currently using NFS and have no plans to start. I believe using GPFS is the faster method to preform the migration. Any suggestions and experience of doing similar migration jobs would be helpful. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00001.gif Type: image/gif Size: 1851 bytes Desc: ATT00001.gif URL: From p.childs at qmul.ac.uk Thu Oct 20 20:07:44 2016 From: p.childs at qmul.ac.uk (Peter Childs) Date: Thu, 20 Oct 2016 19:07:44 +0000 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) In-Reply-To: References: , Message-ID: <5qv6d7inj2j1pa94kqamk2uf.1476989646711@email.android.com> Yes, most of the filesets are based on research groups, projects or departments, with the exception of scratch and home, hence the idea to use a different method for these filesets. There are approximately 230 million files, the largest of the filesets has 52TB and 63 million files. 300TB in total. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ---- Bill Pappas wrote ---- I have some ideas to suggest given some of my experiences. First, I have some questions: How many files are you migrating? Will you be creating multiple file sets on the target system based off of business or project needs? Like, file set a is for "department a" and file set b is for "large scale project a" Thanks. Bill Pappas 901-619-0585 bpappas at dstonline.com [1466780990050_DSTlogo.png] [http://www.prweb.com/releases/2016/06/prweb13504050.htm] http://www.prweb.com/releases/2016/06/prweb13504050.htm ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org Sent: Wednesday, October 19, 2016 3:12 PM To: gpfsug-discuss at spectrumscale.org Subject: gpfsug-discuss Digest, Vol 57, Issue 49 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Using AFM to migrate files. (Peter Childs) 2. subnets (Brian Marshall) 3. Re: subnets (Simon Thompson (Research Computing - IT Services)) 4. Re: subnets (Uwe Falke) 5. Will there be any more GPFS 4.2.0-x releases? (Buterbaugh, Kevin L) ---------------------------------------------------------------------- Message: 1 Date: Wed, 19 Oct 2016 14:12:41 +0000 From: Peter Childs To: gpfsug main discussion list Subject: [gpfsug-discuss] Using AFM to migrate files. Message-ID: Content-Type: text/plain; charset="iso-8859-1" We are planning to use AFM to migrate our old GPFS file store to a new GPFS file store. This will give us the advantages of Spectrum Scale (GPFS) 4.2, such as larger block and inode size. I would like to attempt to gain some insight on my plans before I start. The old file store was running GPFS 3.5 with 512 byte inodes and 1MB block size. We have now upgraded it to 4.1 and are working towards 4.2 with 300TB of files. (385TB max space) this is so we can use both the old and new storage via multi-cluster. We are moving to a new GPFS cluster so we can use the new protocol nodes eventually and also put the new storage machines as cluster managers, as this should be faster and future proof The new hardware has 1PB of space running GPFS 4.2 We have multiple filesets, and would like to maintain our namespace as far as possible. My plan was to. 1. Create a read-only (RO) AFM cache on the new storage (ro) 2a. Move old fileset and replace with SymLink to new. 2b. Convert RO AFM to Local Update (LU) AFM pointing to new parking area of old files. 2c. move user access to new location in cache. 3. Flush everything into cache and disconnect. I've read the docs including the ones on migration but it's not clear if it's safe to move the home of a cache and update the target. It looks like it should be possible and my tests say it works. An alternative plan is to use a Independent Writer (IW) AFM Cache to move the home directories which are pointed to by LDAP. Hence we can move users one at a time and only have to drain the HPC cluster at the end to disconnect the cache. I assume that migrating users over an Independent Writer is safe so long as the users don't use both sides of the cache at once (ie home and target) I'm also interested in any recipe people have on GPFS policies to preseed and flush the cache. We plan to do all the migration using AFM over GPFS we're not currently using NFS and have no plans to start. I believe using GPFS is the faster method to preform the migration. Any suggestions and experience of doing similar migration jobs would be helpful. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ------------------------------ Message: 2 Date: Wed, 19 Oct 2016 13:46:02 -0400 From: Brian Marshall To: gpfsug main discussion list Subject: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="utf-8" All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 3 Date: Wed, 19 Oct 2016 18:10:38 +0000 From: "Simon Thompson (Research Computing - IT Services)" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="us-ascii" mmdiag --network Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Brian Marshall [mimarsh2 at vt.edu] Sent: 19 October 2016 18:46 To: gpfsug main discussion list Subject: [gpfsug-discuss] subnets All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall ------------------------------ Message: 4 Date: Wed, 19 Oct 2016 20:15:52 +0200 From: "Uwe Falke" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="ISO-8859-1" Hi Brian, you might use mmfsadm saferdump tscomm to check on which route peer cluster members are reached. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Frank Hammer, Thorsten Moehring Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: Brian Marshall To: gpfsug main discussion list Date: 10/19/2016 07:46 PM Subject: [gpfsug-discuss] subnets Sent by: gpfsug-discuss-bounces at spectrumscale.org All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ------------------------------ Message: 5 Date: Wed, 19 Oct 2016 20:11:57 +0000 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Subject: [gpfsug-discuss] Will there be any more GPFS 4.2.0-x releases? Message-ID: <142FECE0-E157-42D9-BC10-4C48E78FA065 at vanderbilt.edu> Content-Type: text/plain; charset="utf-8" Hi All, We?re currently running GPFS 4.2.0-4 with an efix installed and now we need a 2nd efix. I?m not a big fan of adding efix to efix and would prefer to go to a new PTF that contains both efixes. So ? is there going to be a GPFS 4.2.0-5 (it?s been a longer than normal interval since PTF 4 came out) or do we need to go to GPFS 4.2.1-x? If the latter, any major changes to watch out for? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 49 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-1466780990050_DSTlogo.png.png Type: image/png Size: 6282 bytes Desc: OutlookEmoji-1466780990050_DSTlogo.png.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-httpwww.prweb.comreleases201606prweb13504050.htm.jpg Type: image/jpeg Size: 14887 bytes Desc: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg URL: From volobuev at us.ibm.com Fri Oct 21 02:58:43 2016 From: volobuev at us.ibm.com (Yuri L Volobuev) Date: Thu, 20 Oct 2016 18:58:43 -0700 Subject: [gpfsug-discuss] Two new whitepapers published Message-ID: Esteemed GPFS and Spectrum Scale users, For your reading enjoyment, two new whitepapers have been posted to the Spectrum Scale Wiki: Spectrum?Scale:?Replication?in?GPFS Spectrum?Scale:?GPFS?Metadata The URL for the parent page is https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20 (GPFS)/page/White%20Papers%20%26%20Media The two .pdf documents are accessible through the Attachment section at the bottom of the page. ?Unfortunately, dW "spam prevention engine" does a very good job preventing me from "spamming" the page to actually add links. Best regards, Yuri -------------- next part -------------- An HTML attachment was scrubbed... URL: From alandhae at gmx.de Fri Oct 21 07:43:40 2016 From: alandhae at gmx.de (=?ISO-8859-15?Q?Andreas_Landh=E4u=DFer?=) Date: Fri, 21 Oct 2016 08:43:40 +0200 (CEST) Subject: [gpfsug-discuss] Two new whitepapers published In-Reply-To: References: Message-ID: Hi Yuri, Arrg, can't find them, page has been last updated on Aug, 18 by JohnTOlson, maybe its internal and not open for the public? Ciao Andreas On Fri, 21 Oct 2016, Yuri L Volobuev wrote: > > > Esteemed GPFS and Spectrum Scale users, > > For your reading enjoyment, two new whitepapers have been posted to the > Spectrum Scale Wiki: > > Spectrum?Scale:?Replication?in?GPFS > Spectrum?Scale:?GPFS?Metadata > > The URL for the parent page is > https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20 > (GPFS)/page/White%20Papers%20%26%20Media > > The two .pdf documents are accessible through the Attachment section at the > bottom of the page. ?Unfortunately, dW "spam prevention engine" does a very > good job preventing me from "spamming" the page to actually add links. > > Best regards, > > Yuri > -- Andreas Landh?u?er +49 151 12133027 (mobile) alandhae at gmx.de From dave at milk-vfx.com Fri Oct 21 07:51:51 2016 From: dave at milk-vfx.com (Dave Goodbourn) Date: Fri, 21 Oct 2016 07:51:51 +0100 Subject: [gpfsug-discuss] Two new whitepapers published In-Reply-To: References: Message-ID: I can find them. They're in the attachments tab in the bottom set of tabs. And a very good read. Thanks Yuri! Dave. > On 21 Oct 2016, at 07:43, Andreas Landh?u?er wrote: > > > Hi Yuri, > > Arrg, can't find them, page has been last updated on Aug, 18 by JohnTOlson, maybe its internal and not open for the public? > > Ciao > > Andreas > >> On Fri, 21 Oct 2016, Yuri L Volobuev wrote: >> >> >> >> Esteemed GPFS and Spectrum Scale users, >> >> For your reading enjoyment, two new whitepapers have been posted to the >> Spectrum Scale Wiki: >> >> Spectrum Scale: Replication in GPFS >> Spectrum Scale: GPFS Metadata >> >> The URL for the parent page is >> https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20 >> (GPFS)/page/White%20Papers%20%26%20Media >> >> The two .pdf documents are accessible through the Attachment section at the >> bottom of the page. Unfortunately, dW "spam prevention engine" does a very >> good job preventing me from "spamming" the page to actually add links. >> >> Best regards, >> >> Yuri >> > > -- > Andreas Landh?u?er +49 151 12133027 (mobile) > alandhae at gmx.de > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From laurence at qsplace.co.uk Fri Oct 21 08:10:56 2016 From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow) Date: Fri, 21 Oct 2016 08:10:56 +0100 Subject: [gpfsug-discuss] Two new whitepapers published In-Reply-To: References: Message-ID: <36EC1962-F71F-4492-A175-4BA3E9D97A8D@qsplace.co.uk> Right down the bottom of the page under attachments. -- Lauz On 21 October 2016 07:43:40 BST, "Andreas Landh?u?er" wrote: > >Hi Yuri, > >Arrg, can't find them, page has been last updated on Aug, 18 by >JohnTOlson, maybe its internal and not open for the public? > >Ciao > > Andreas > >On Fri, 21 Oct 2016, Yuri L Volobuev wrote: > >> >> >> Esteemed GPFS and Spectrum Scale users, >> >> For your reading enjoyment, two new whitepapers have been posted to >the >> Spectrum Scale Wiki: >> >> Spectrum?Scale:?Replication?in?GPFS >> Spectrum?Scale:?GPFS?Metadata >> >> The URL for the parent page is >> >https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20 >> (GPFS)/page/White%20Papers%20%26%20Media >> >> The two .pdf documents are accessible through the Attachment section >at the >> bottom of the page. ?Unfortunately, dW "spam prevention engine" does >a very >> good job preventing me from "spamming" the page to actually add >links. >> >> Best regards, >> >> Yuri >> > >-- >Andreas Landh?u?er +49 151 12133027 (mobile) >alandhae at gmx.de > >------------------------------------------------------------------------ > >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alandhae at gmx.de Fri Oct 21 08:31:43 2016 From: alandhae at gmx.de (=?ISO-8859-15?Q?Andreas_Landh=E4u=DFer?=) Date: Fri, 21 Oct 2016 09:31:43 +0200 (CEST) Subject: [gpfsug-discuss] Two new whitepapers published In-Reply-To: <36EC1962-F71F-4492-A175-4BA3E9D97A8D@qsplace.co.uk> References: <36EC1962-F71F-4492-A175-4BA3E9D97A8D@qsplace.co.uk> Message-ID: On Fri, 21 Oct 2016, Laurence Horrocks-Barlow wrote: Found it! thanks Yuri, for the two very interesting papers about the Metadata and replication. We always used the rule of thumb 5% for metadata, since we are having separate devices for metadata, and tiny fast disks aren't available anymore, we are getting a bunch of larger fast disks. We never experienced a problem with the (more or less) amount of metadata ... Andreas > Right down the bottom of the page under attachments. > > -- Lauz > > On 21 October 2016 07:43:40 BST, "Andreas Landh?u?er" wrote: >> >> Hi Yuri, >> >> Arrg, can't find them, page has been last updated on Aug, 18 by >> JohnTOlson, maybe its internal and not open for the public? >> >> Ciao >> >> Andreas >> >> On Fri, 21 Oct 2016, Yuri L Volobuev wrote: >> >>> >>> >>> Esteemed GPFS and Spectrum Scale users, >>> >>> For your reading enjoyment, two new whitepapers have been posted to >> the >>> Spectrum Scale Wiki: >>> >>> Spectrum?Scale:?Replication?in?GPFS >>> Spectrum?Scale:?GPFS?Metadata >>> >>> The URL for the parent page is >>> >> https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20 >>> (GPFS)/page/White%20Papers%20%26%20Media >>> >>> The two .pdf documents are accessible through the Attachment section >> at the >>> bottom of the page. ?Unfortunately, dW "spam prevention engine" does >> a very >>> good job preventing me from "spamming" the page to actually add >> links. >>> >>> Best regards, >>> >>> Yuri >>> >> >> -- >> Andreas Landh?u?er +49 151 12133027 (mobile) >> alandhae at gmx.de >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- Andreas Landh?u?er +49 151 12133027 (mobile) alandhae at gmx.de From daniel.kidger at uk.ibm.com Fri Oct 21 14:48:10 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Fri, 21 Oct 2016 13:48:10 +0000 Subject: [gpfsug-discuss] Two new whitepapers published In-Reply-To: <36EC1962-F71F-4492-A175-4BA3E9D97A8D@qsplace.co.uk> Message-ID: Great papers ! @Yuri: do you want to add entries at the top of that page to: a) help people find those papers, and b) make it more obvious that this page is not obsolete and no longer maintained. Daniel IBM Spectrum Storage Software +44 (0)7818 522266 Sent from my iPad using IBM Verse On 21 Oct 2016, 08:11:23, laurence at qsplace.co.uk wrote: From: laurence at qsplace.co.uk To: alandhae at gmx.de, gpfsug-discuss at spectrumscale.org Cc: Date: 21 Oct 2016 08:11:23 Subject: Re: [gpfsug-discuss] Two new whitepapers published Right down the bottom of the page under attachments. -- Lauz On 21 October 2016 07:43:40 BST, "Andreas Landh?u?er" wrote: Hi Yuri,Arrg, can't find them, page has been last updated on Aug, 18 by JohnTOlson, maybe its internal and not open for the public?Ciao AndreasOn Fri, 21 Oct 2016, Yuri L Volobuev wrote: Esteemed GPFS and Spectrum Scale users, For your reading enjoyment, two new whitepapers have been posted to the Spectrum Scale Wiki: Spectrum Scale: Replication in GPFS Spectrum Scale: GPFS Metadata The URL for the parent page is https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20 (GPFS)/page/White%20Papers%20%26%20Media The two .pdf documents are accessible through the Attachment section at the bottom of the page. Unfortunately, dW "spam prevention engine" does a very good job preventing me from "spamming" the page to actually add links. Best regards, Yuri -- Sent from my Android device with K-9 Mail. Please excuse my brevity.Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From volobuev at us.ibm.com Fri Oct 21 16:45:42 2016 From: volobuev at us.ibm.com (Yuri L Volobuev) Date: Fri, 21 Oct 2016 08:45:42 -0700 Subject: [gpfsug-discuss] Two new whitepapers published In-Reply-To: References: <36EC1962-F71F-4492-A175-4BA3E9D97A8D@qsplace.co.uk> Message-ID: I tried to do just that... the dW "spam prevention engine" wouldn't let me update the page. It may take some time to do this right. yuri From: "Daniel Kidger" To: "gpfsug main discussion list" , Date: 10/21/2016 06:48 AM Subject: Re: [gpfsug-discuss] Two new whitepapers published Sent by: gpfsug-discuss-bounces at spectrumscale.org Great papers ! @Yuri: do you want to add entries at the top of that page to: a) help people find those papers, and b) make it more obvious that this page is not obsolete and no longer maintained. Daniel IBM Spectrum Storage Software +44 (0)7818 522266 Sent from my iPad using IBM Verse On 21 Oct 2016, 08:11:23, laurence at qsplace.co.uk wrote: From: laurence at qsplace.co.uk To: alandhae at gmx.de, gpfsug-discuss at spectrumscale.org Cc: Date: 21 Oct 2016 08:11:23 Subject: Re: [gpfsug-discuss] Two new whitepapers published Right down the bottom of the page under attachments. -- Lauz On 21 October 2016 07:43:40 BST, "Andreas Landh?u?er" wrote: Hi Yuri, Arrg, can't find them, page has been last updated on Aug, 18 by JohnTOlson, maybe its internal and not open for the public? Ciao Andreas On Fri, 21 Oct 2016, Yuri L Volobuev wrote: Esteemed GPFS and Spectrum Scale users, For your reading enjoyment, two new whitepapers have been posted to the Spectrum Scale Wiki: Spectrum Scale: Replication in GPFS Spectrum Scale: GPFS Metadata The URL for the parent page is https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20 (GPFS)/page/White%20Papers%20%26%20Media The two .pdf documents are accessible through the Attachment section at the bottom of the page. Unfortunately, dW "spam prevention engine" does a very good job preventing me from "spamming" the page to actually add links. Best regards, Yuri -- Sent from my Android device with K-9 Mail. Please excuse my brevity. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From bpappas at dstonline.com Fri Oct 21 19:46:09 2016 From: bpappas at dstonline.com (Bill Pappas) Date: Fri, 21 Oct 2016 18:46:09 +0000 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) In-Reply-To: References: Message-ID: >>the largest of the filesets has 52TB and 63 million files Are you using NFS as the transport path between the home and cache? If you are using NFS, how are you producing the list of files to migrate? mmafmctl with the prefetch option? If so, I would measure the time it takes for that command (with that option) to produce the list of files it intends to prefetch. From my experience, this is very important as a) it can take a long time if you have >10 million of files and b) I've seen this operation crash when the list grew large. Does anyone else on this thread have any experiences? I would love to hear positive experiences as well. I tried so hard and for so long to make AFM work with one customer, but we gave up as it was not reliable and stable for large scale (many files) migrations. If you are using GPFS as the conduit between the home and cache (i.e. no NFS), I would still ask the same question, more with respect to stability for large file lists during the initial prefetch stages. As far as I could tell, from GPFS 3.5 to 4.2, the phases of prefetch where the home and cache are compared (i.e. let's make a list of what is ot be migrated over) before the data transfer begins only runs on the GW node managing that cache. It does not leverage multiple gw nodes and multiple home nodes to speed up this 'list and find' stage of prefetch. I hope some AFM developers can clarify or correct my findings. This was a huge impediment for large file migrations where it is difficult (organizationally, not technically) to split a folder structure into multiple file sets. The lack of stability under these large scans was the real failing for us. Bill Pappas 901-619-0585 bpappas at dstonline.com [1466780990050_DSTlogo.png] [http://www.prweb.com/releases/2016/06/prweb13504050.htm] http://www.prweb.com/releases/2016/06/prweb13504050.htm ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org Sent: Thursday, October 20, 2016 2:07 PM To: gpfsug-discuss at spectrumscale.org Subject: gpfsug-discuss Digest, Vol 57, Issue 53 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Using AFM to migrate files. (Peter Childs) (Peter Childs) ---------------------------------------------------------------------- Message: 1 Date: Thu, 20 Oct 2016 19:07:44 +0000 From: Peter Childs To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) Message-ID: <5qv6d7inj2j1pa94kqamk2uf.1476989646711 at email.android.com> Content-Type: text/plain; charset="iso-8859-1" Yes, most of the filesets are based on research groups, projects or departments, with the exception of scratch and home, hence the idea to use a different method for these filesets. There are approximately 230 million files, the largest of the filesets has 52TB and 63 million files. 300TB in total. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ---- Bill Pappas wrote ---- I have some ideas to suggest given some of my experiences. First, I have some questions: How many files are you migrating? Will you be creating multiple file sets on the target system based off of business or project needs? Like, file set a is for "department a" and file set b is for "large scale project a" Thanks. Bill Pappas 901-619-0585 bpappas at dstonline.com [1466780990050_DSTlogo.png] [http://www.prweb.com/releases/2016/06/prweb13504050.htm] http://www.prweb.com/releases/2016/06/prweb13504050.htm ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org Sent: Wednesday, October 19, 2016 3:12 PM To: gpfsug-discuss at spectrumscale.org Subject: gpfsug-discuss Digest, Vol 57, Issue 49 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Using AFM to migrate files. (Peter Childs) 2. subnets (Brian Marshall) 3. Re: subnets (Simon Thompson (Research Computing - IT Services)) 4. Re: subnets (Uwe Falke) 5. Will there be any more GPFS 4.2.0-x releases? (Buterbaugh, Kevin L) ---------------------------------------------------------------------- Message: 1 Date: Wed, 19 Oct 2016 14:12:41 +0000 From: Peter Childs To: gpfsug main discussion list Subject: [gpfsug-discuss] Using AFM to migrate files. Message-ID: Content-Type: text/plain; charset="iso-8859-1" We are planning to use AFM to migrate our old GPFS file store to a new GPFS file store. This will give us the advantages of Spectrum Scale (GPFS) 4.2, such as larger block and inode size. I would like to attempt to gain some insight on my plans before I start. The old file store was running GPFS 3.5 with 512 byte inodes and 1MB block size. We have now upgraded it to 4.1 and are working towards 4.2 with 300TB of files. (385TB max space) this is so we can use both the old and new storage via multi-cluster. We are moving to a new GPFS cluster so we can use the new protocol nodes eventually and also put the new storage machines as cluster managers, as this should be faster and future proof The new hardware has 1PB of space running GPFS 4.2 We have multiple filesets, and would like to maintain our namespace as far as possible. My plan was to. 1. Create a read-only (RO) AFM cache on the new storage (ro) 2a. Move old fileset and replace with SymLink to new. 2b. Convert RO AFM to Local Update (LU) AFM pointing to new parking area of old files. 2c. move user access to new location in cache. 3. Flush everything into cache and disconnect. I've read the docs including the ones on migration but it's not clear if it's safe to move the home of a cache and update the target. It looks like it should be possible and my tests say it works. An alternative plan is to use a Independent Writer (IW) AFM Cache to move the home directories which are pointed to by LDAP. Hence we can move users one at a time and only have to drain the HPC cluster at the end to disconnect the cache. I assume that migrating users over an Independent Writer is safe so long as the users don't use both sides of the cache at once (ie home and target) I'm also interested in any recipe people have on GPFS policies to preseed and flush the cache. We plan to do all the migration using AFM over GPFS we're not currently using NFS and have no plans to start. I believe using GPFS is the faster method to preform the migration. Any suggestions and experience of doing similar migration jobs would be helpful. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ------------------------------ Message: 2 Date: Wed, 19 Oct 2016 13:46:02 -0400 From: Brian Marshall To: gpfsug main discussion list Subject: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="utf-8" All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 3 Date: Wed, 19 Oct 2016 18:10:38 +0000 From: "Simon Thompson (Research Computing - IT Services)" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="us-ascii" mmdiag --network Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Brian Marshall [mimarsh2 at vt.edu] Sent: 19 October 2016 18:46 To: gpfsug main discussion list Subject: [gpfsug-discuss] subnets All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall ------------------------------ Message: 4 Date: Wed, 19 Oct 2016 20:15:52 +0200 From: "Uwe Falke" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="ISO-8859-1" Hi Brian, you might use mmfsadm saferdump tscomm to check on which route peer cluster members are reached. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Frank Hammer, Thorsten Moehring Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: Brian Marshall To: gpfsug main discussion list Date: 10/19/2016 07:46 PM Subject: [gpfsug-discuss] subnets Sent by: gpfsug-discuss-bounces at spectrumscale.org All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ------------------------------ Message: 5 Date: Wed, 19 Oct 2016 20:11:57 +0000 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Subject: [gpfsug-discuss] Will there be any more GPFS 4.2.0-x releases? Message-ID: <142FECE0-E157-42D9-BC10-4C48E78FA065 at vanderbilt.edu> Content-Type: text/plain; charset="utf-8" Hi All, We?re currently running GPFS 4.2.0-4 with an efix installed and now we need a 2nd efix. I?m not a big fan of adding efix to efix and would prefer to go to a new PTF that contains both efixes. So ? is there going to be a GPFS 4.2.0-5 (it?s been a longer than normal interval since PTF 4 came out) or do we need to go to GPFS 4.2.1-x? If the latter, any major changes to watch out for? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 49 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-1466780990050_DSTlogo.png.png Type: image/png Size: 6282 bytes Desc: OutlookEmoji-1466780990050_DSTlogo.png.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-httpwww.prweb.comreleases201606prweb13504050.htm.jpg Type: image/jpeg Size: 14887 bytes Desc: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 53 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-1466780990050_DSTlogo.png.png Type: image/png Size: 6282 bytes Desc: OutlookEmoji-1466780990050_DSTlogo.png.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg Type: image/jpeg Size: 14887 bytes Desc: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg URL: From p.childs at qmul.ac.uk Fri Oct 21 21:35:15 2016 From: p.childs at qmul.ac.uk (Peter Childs) Date: Fri, 21 Oct 2016 20:35:15 +0000 Subject: [gpfsug-discuss] Two new whitepapers published In-Reply-To: References: <36EC1962-F71F-4492-A175-4BA3E9D97A8D@qsplace.co.uk>, Message-ID: <7fifaf55s6bq10jlpkvl5he9.1477081370569@email.android.com> Reading through them and we'll worth read. How important is setting the correct cluster size at file system creation time? Ie with "mmchfs -n 256" ie how much band of error can you get away with? We have a cluster of 240 nodes that was setup with the default 32 setting, it's now about to grow to ~300 nodes. Is this likly to causing as an issue, which is difficult to fix. The manual says it can be adjusted but this white paper suggests not. Fortunately were also migrating our storage to new hardware, so have a good opportunity to get the setting right, this time around. Has anyone got any stats on the benefits of getting it "right" vs getting it wrong. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ---- Andreas Landh?u?er wrote ---- On Fri, 21 Oct 2016, Laurence Horrocks-Barlow wrote: Found it! thanks Yuri, for the two very interesting papers about the Metadata and replication. We always used the rule of thumb 5% for metadata, since we are having separate devices for metadata, and tiny fast disks aren't available anymore, we are getting a bunch of larger fast disks. We never experienced a problem with the (more or less) amount of metadata ... Andreas > Right down the bottom of the page under attachments. > > -- Lauz > > On 21 October 2016 07:43:40 BST, "Andreas Landh?u?er" wrote: >> >> Hi Yuri, >> >> Arrg, can't find them, page has been last updated on Aug, 18 by >> JohnTOlson, maybe its internal and not open for the public? >> >> Ciao >> >> Andreas >> >> On Fri, 21 Oct 2016, Yuri L Volobuev wrote: >> >>> >>> >>> Esteemed GPFS and Spectrum Scale users, >>> >>> For your reading enjoyment, two new whitepapers have been posted to >> the >>> Spectrum Scale Wiki: >>> >>> Spectrum Scale: Replication in GPFS >>> Spectrum Scale: GPFS Metadata >>> >>> The URL for the parent page is >>> >> https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20 >>> (GPFS)/page/White%20Papers%20%26%20Media >>> >>> The two .pdf documents are accessible through the Attachment section >> at the >>> bottom of the page. Unfortunately, dW "spam prevention engine" does >> a very >>> good job preventing me from "spamming" the page to actually add >> links. >>> >>> Best regards, >>> >>> Yuri >>> >> >> -- >> Andreas Landh?u?er +49 151 12133027 (mobile) >> alandhae at gmx.de >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- Andreas Landh?u?er +49 151 12133027 (mobile) alandhae at gmx.de -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.childs at qmul.ac.uk Fri Oct 21 21:44:18 2016 From: p.childs at qmul.ac.uk (Peter Childs) Date: Fri, 21 Oct 2016 20:44:18 +0000 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) In-Reply-To: References: , Message-ID: <9msuk69qtso8etpj6c9mq2pk.1477082656580@email.android.com> ---- Bill Pappas wrote ---- > >>the largest of the filesets has 52TB and 63 million files > > > Are you using NFS as the transport path between the home and cache? No plans to was planning to use gpfs multi-cluster, as transport. > If you are using NFS, how are you producing the list of files to migrate? mmafmctl with the prefetch option? If so, I would measure the time it takes for that command (with that option) to produce the list of files it intends to prefetch. From my experience, this is very important as a) it can take a long time if you have >10 million of files and b) I've seen this operation crash when the list grew large. Does anyone else on this thread have any experiences? I would love to hear positive experiences as well. I tried so hard and for so long to make AFM work with one customer, but we gave up as it was not reliable and stable for large scale (many files) migrations. > If you are using GPFS as the conduit between the home and cache (i.e. no NFS), I would still ask the same question, more with respect to stability for large file lists during the initial prefetch stages. I was planning to use a gpfs policy to create the list, but I guess a find should work, I'm guessing your saying don't migrate the files in bulk by using a find onto cache. It would be nice to see some examples recipes to prefetch files into afm. > > > As far as I could tell, from GPFS 3.5 to 4.2, the phases of prefetch where the home and cache are compared (i.e. let's make a list of what is ot be migrated over) before the data transfer begins only runs on the GW node managing that cache. It does not leverage multiple gw nodes and multiple home nodes to speed up this 'list and find' stage of prefetch. I hope some AFM developers can clarify or correct my findings. This was a huge impediment for large file migrations where it is difficult (organizationally, not technically) to split a folder structure into multiple file sets. The lack of stability under these large scans was the real failing for us. Interesting. > > > Bill Pappas > > 901-619-0585 > > bpappas at dstonline.com > > Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London > > > > > > http://www.prweb.com/releases/2016/06/prweb13504050.htm > > > > From: gpfsug-discuss-bounces at spectrumscale.org > on behalf of gpfsug-discuss-request at spectrumscale.org > > Sent: Thursday, October 20, 2016 2:07 PM > To: gpfsug-discuss at spectrumscale.org > Subject: gpfsug-discuss Digest, Vol 57, Issue 53 > > Send gpfsug-discuss mailing list submissions to > gpfsug-discuss at spectrumscale.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > or, via email, send a message with subject or body 'help' to > gpfsug-discuss-request at spectrumscale.org > > You can reach the person managing the list at > gpfsug-discuss-owner at spectrumscale.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of gpfsug-discuss digest..." > > > Today's Topics: > > 1. Re: Using AFM to migrate files. (Peter Childs) (Peter Childs) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 20 Oct 2016 19:07:44 +0000 > From: Peter Childs > > To: gpfsug main discussion list > > Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter > Childs) > Message-ID: <5qv6d7inj2j1pa94kqamk2uf.1476989646711 at email.android.com> > Content-Type: text/plain; charset="iso-8859-1" > > Yes, most of the filesets are based on research groups, projects or departments, with the exception of scratch and home, hence the idea to use a different method for these filesets. > > There are approximately 230 million files, the largest of the filesets has 52TB and 63 million files. 300TB in total. > > Peter Childs > Research Storage > ITS Research and Teaching Support > Queen Mary, University of London > > > ---- Bill Pappas wrote ---- > > > I have some ideas to suggest given some of my experiences. First, I have some questions: > > > How many files are you migrating? > > Will you be creating multiple file sets on the target system based off of business or project needs? Like, file set a is for "department a" and file set b is for "large scale project a" > > > Thanks. > > > Bill Pappas > > 901-619-0585 > > bpappas at dstonline.com > > > [1466780990050_DSTlogo.png] > > > [http://www.prweb.com/releases/2016/06/prweb13504050.htm] > > http://www.prweb.com/releases/2016/06/prweb13504050.htm > > > ________________________________ > From: gpfsug-discuss-bounces at spectrumscale.org > on behalf of gpfsug-discuss-request at spectrumscale.org > > Sent: Wednesday, October 19, 2016 3:12 PM > To: gpfsug-discuss at spectrumscale.org > Subject: gpfsug-discuss Digest, Vol 57, Issue 49 > > Send gpfsug-discuss mailing list submissions to > gpfsug-discuss at spectrumscale.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > or, via email, send a message with subject or body 'help' to > gpfsug-discuss-request at spectrumscale.org > > You can reach the person managing the list at > gpfsug-discuss-owner at spectrumscale.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of gpfsug-discuss digest..." > > > Today's Topics: > > 1. Using AFM to migrate files. (Peter Childs) > 2. subnets (Brian Marshall) > 3. Re: subnets (Simon Thompson (Research Computing - IT Services)) > 4. Re: subnets (Uwe Falke) > 5. Will there be any more GPFS 4.2.0-x releases? > (Buterbaugh, Kevin L) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 19 Oct 2016 14:12:41 +0000 > From: Peter Childs > > To: gpfsug main discussion list > > Subject: [gpfsug-discuss] Using AFM to migrate files. > Message-ID: > > > > Content-Type: text/plain; charset="iso-8859-1" > > > We are planning to use AFM to migrate our old GPFS file store to a new GPFS file store. This will give us the advantages of Spectrum Scale (GPFS) 4.2, such as larger block and inode size. I would like to attempt to gain some insight on my plans before I start. > > The old file store was running GPFS 3.5 with 512 byte inodes and 1MB block size. We have now upgraded it to 4.1 and are working towards 4.2 with 300TB of files. (385TB max space) this is so we can use both the old and new storage via multi-cluster. > > We are moving to a new GPFS cluster so we can use the new protocol nodes eventually and also put the new storage machines as cluster managers, as this should be faster and future proof > > The new hardware has 1PB of space running GPFS 4.2 > > We have multiple filesets, and would like to maintain our namespace as far as possible. > > My plan was to. > > 1. Create a read-only (RO) AFM cache on the new storage (ro) > 2a. Move old fileset and replace with SymLink to new. > 2b. Convert RO AFM to Local Update (LU) AFM pointing to new parking area of old files. > 2c. move user access to new location in cache. > 3. Flush everything into cache and disconnect. > > I've read the docs including the ones on migration but it's not clear if it's safe to move the home of a cache and update the target. It looks like it should be possible and my tests say it works. > > An alternative plan is to use a Independent Writer (IW) AFM Cache to move the home directories which are pointed to by LDAP. Hence we can move users one at a time and only have to drain the HPC cluster at the end to disconnect the cache. I assume that migrating users over an Independent Writer is safe so long as the users don't use both sides of the cache at once (ie home and target) > > I'm also interested in any recipe people have on GPFS policies to preseed and flush the cache. > > We plan to do all the migration using AFM over GPFS we're not currently using NFS and have no plans to start. I believe using GPFS is the faster method to preform the migration. > > Any suggestions and experience of doing similar migration jobs would be helpful. > > Peter Childs > Research Storage > ITS Research and Teaching Support > Queen Mary, University of London > > > > ------------------------------ > > Message: 2 > Date: Wed, 19 Oct 2016 13:46:02 -0400 > From: Brian Marshall > > To: gpfsug main discussion list > > Subject: [gpfsug-discuss] subnets > Message-ID: > > > Content-Type: text/plain; charset="utf-8" > > All, > > We are setting up communication between 2 clusters using ethernet and > IPoFabric. > > The Daemon interface is running on ethernet, so all admin traffic will use > it. > > We are still getting the subnets setting correct. > > Question: > > Does GPFS have a way to query how it is connecting to a given cluster/node? > i.e. once we have subnets setup how can we tell GPFS is actually using > them. Currently we just do a large transfer and check tcpdump for any > packets flowing on the high-speed/data/non-admin subnet. > > > Thank you, > Brian Marshall > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: ; > > ------------------------------ > > Message: 3 > Date: Wed, 19 Oct 2016 18:10:38 +0000 > From: "Simon Thompson (Research Computing - IT Services)" > > > To: gpfsug main discussion list > > Subject: Re: [gpfsug-discuss] subnets > Message-ID: > > > Content-Type: text/plain; charset="us-ascii" > > > mmdiag --network > > Simon > ________________________________________ > From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Brian Marshall [mimarsh2 at vt.edu] > Sent: 19 October 2016 18:46 > To: gpfsug main discussion list > Subject: [gpfsug-discuss] subnets > > All, > > We are setting up communication between 2 clusters using ethernet and IPoFabric. > > The Daemon interface is running on ethernet, so all admin traffic will use it. > > We are still getting the subnets setting correct. > > Question: > > Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. > > > Thank you, > Brian Marshall > > > ------------------------------ > > Message: 4 > Date: Wed, 19 Oct 2016 20:15:52 +0200 > From: "Uwe Falke" > > To: gpfsug main discussion list > > Subject: Re: [gpfsug-discuss] subnets > Message-ID: > > > > Content-Type: text/plain; charset="ISO-8859-1" > > Hi Brian, > you might use > > mmfsadm saferdump tscomm > > to check on which route peer cluster members are reached. > > > Mit freundlichen Gr??en / Kind regards > > > Dr. Uwe Falke > > IT Specialist > High Performance Computing Services / Integrated Technology Services / > Data Center Services > ------------------------------------------------------------------------------------------------------------------------------------------- > IBM Deutschland > Rathausstr. 7 > 09111 Chemnitz > Phone: +49 371 6978 2165 > Mobile: +49 175 575 2877 > E-Mail: uwefalke at de.ibm.com > ------------------------------------------------------------------------------------------------------------------------------------------- > IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: > Frank Hammer, Thorsten Moehring > Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, > HRB 17122 > > > > > From: Brian Marshall > > To: gpfsug main discussion list > > Date: 10/19/2016 07:46 PM > Subject: [gpfsug-discuss] subnets > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > All, > > We are setting up communication between 2 clusters using ethernet and > IPoFabric. > > The Daemon interface is running on ethernet, so all admin traffic will use > it. > > We are still getting the subnets setting correct. > > Question: > > Does GPFS have a way to query how it is connecting to a given > cluster/node? i.e. once we have subnets setup how can we tell GPFS is > actually using them. Currently we just do a large transfer and check > tcpdump for any packets flowing on the high-speed/data/non-admin subnet. > > > Thank you, > Brian Marshall_______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > ------------------------------ > > Message: 5 > Date: Wed, 19 Oct 2016 20:11:57 +0000 > From: "Buterbaugh, Kevin L" > > To: gpfsug main discussion list > > Subject: [gpfsug-discuss] Will there be any more GPFS 4.2.0-x > releases? > Message-ID: <142FECE0-E157-42D9-BC10-4C48E78FA065 at vanderbilt.edu> > Content-Type: text/plain; charset="utf-8" > > Hi All, > > We?re currently running GPFS 4.2.0-4 with an efix installed and now we need a 2nd efix. I?m not a big fan of adding efix to efix and would prefer to go to a new PTF that contains both efixes. > > So ? is there going to be a GPFS 4.2.0-5 (it?s been a longer than normal interval since PTF 4 came out) or do we need to go to GPFS 4.2.1-x? If the latter, any major changes to watch out for? Thanks? > > Kevin > > ? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and Education > Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633 > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: ; > > ------------------------------ > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > End of gpfsug-discuss Digest, Vol 57, Issue 49 > ********************************************** > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: ; > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: OutlookEmoji-1466780990050_DSTlogo.png.png > Type: image/png > Size: 6282 bytes > Desc: OutlookEmoji-1466780990050_DSTlogo.png.png > URL: ; > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: OutlookEmoji-httpwww.prweb.comreleases201606prweb13504050.htm.jpg > Type: image/jpeg > Size: 14887 bytes > Desc: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg > URL: ; > > ------------------------------ > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > End of gpfsug-discuss Digest, Vol 57, Issue 53 > ********************************************** >>the largest of the filesets has 52TB and 63 million files Are you using NFS as the transport path between the home and cache? If you are using NFS, how are you producing the list of files to migrate? mmafmctl with the prefetch option? If so, I would measure the time it takes for that command (with that option) to produce the list of files it intends to prefetch. From my experience, this is very important as a) it can take a long time if you have >10 million of files and b) I've seen this operation crash when the list grew large. Does anyone else on this thread have any experiences? I would love to hear positive experiences as well. I tried so hard and for so long to make AFM work with one customer, but we gave up as it was not reliable and stable for large scale (many files) migrations. If you are using GPFS as the conduit between the home and cache (i.e. no NFS), I would still ask the same question, more with respect to stability for large file lists during the initial prefetch stages. As far as I could tell, from GPFS 3.5 to 4.2, the phases of prefetch where the home and cache are compared (i.e. let's make a list of what is ot be migrated over) before the data transfer begins only runs on the GW node managing that cache. It does not leverage multiple gw nodes and multiple home nodes to speed up this 'list and find' stage of prefetch. I hope some AFM developers can clarify or correct my findings. This was a huge impediment for large file migrations where it is difficult (organizationally, not technically) to split a folder structure into multiple file sets. The lack of stability under these large scans was the real failing for us. Bill Pappas 901-619-0585 bpappas at dstonline.com [1466780990050_DSTlogo.png] [http://www.prweb.com/releases/2016/06/prweb13504050.htm] http://www.prweb.com/releases/2016/06/prweb13504050.htm ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org Sent: Thursday, October 20, 2016 2:07 PM To: gpfsug-discuss at spectrumscale.org Subject: gpfsug-discuss Digest, Vol 57, Issue 53 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Using AFM to migrate files. (Peter Childs) (Peter Childs) ---------------------------------------------------------------------- Message: 1 Date: Thu, 20 Oct 2016 19:07:44 +0000 From: Peter Childs To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) Message-ID: <5qv6d7inj2j1pa94kqamk2uf.1476989646711 at email.android.com> Content-Type: text/plain; charset="iso-8859-1" Yes, most of the filesets are based on research groups, projects or departments, with the exception of scratch and home, hence the idea to use a different method for these filesets. There are approximately 230 million files, the largest of the filesets has 52TB and 63 million files. 300TB in total. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ---- Bill Pappas wrote ---- I have some ideas to suggest given some of my experiences. First, I have some questions: How many files are you migrating? Will you be creating multiple file sets on the target system based off of business or project needs? Like, file set a is for "department a" and file set b is for "large scale project a" Thanks. Bill Pappas 901-619-0585 bpappas at dstonline.com [1466780990050_DSTlogo.png] [http://www.prweb.com/releases/2016/06/prweb13504050.htm] http://www.prweb.com/releases/2016/06/prweb13504050.htm ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org Sent: Wednesday, October 19, 2016 3:12 PM To: gpfsug-discuss at spectrumscale.org Subject: gpfsug-discuss Digest, Vol 57, Issue 49 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Using AFM to migrate files. (Peter Childs) 2. subnets (Brian Marshall) 3. Re: subnets (Simon Thompson (Research Computing - IT Services)) 4. Re: subnets (Uwe Falke) 5. Will there be any more GPFS 4.2.0-x releases? (Buterbaugh, Kevin L) ---------------------------------------------------------------------- Message: 1 Date: Wed, 19 Oct 2016 14:12:41 +0000 From: Peter Childs To: gpfsug main discussion list Subject: [gpfsug-discuss] Using AFM to migrate files. Message-ID: Content-Type: text/plain; charset="iso-8859-1" We are planning to use AFM to migrate our old GPFS file store to a new GPFS file store. This will give us the advantages of Spectrum Scale (GPFS) 4.2, such as larger block and inode size. I would like to attempt to gain some insight on my plans before I start. The old file store was running GPFS 3.5 with 512 byte inodes and 1MB block size. We have now upgraded it to 4.1 and are working towards 4.2 with 300TB of files. (385TB max space) this is so we can use both the old and new storage via multi-cluster. We are moving to a new GPFS cluster so we can use the new protocol nodes eventually and also put the new storage machines as cluster managers, as this should be faster and future proof The new hardware has 1PB of space running GPFS 4.2 We have multiple filesets, and would like to maintain our namespace as far as possible. My plan was to. 1. Create a read-only (RO) AFM cache on the new storage (ro) 2a. Move old fileset and replace with SymLink to new. 2b. Convert RO AFM to Local Update (LU) AFM pointing to new parking area of old files. 2c. move user access to new location in cache. 3. Flush everything into cache and disconnect. I've read the docs including the ones on migration but it's not clear if it's safe to move the home of a cache and update the target. It looks like it should be possible and my tests say it works. An alternative plan is to use a Independent Writer (IW) AFM Cache to move the home directories which are pointed to by LDAP. Hence we can move users one at a time and only have to drain the HPC cluster at the end to disconnect the cache. I assume that migrating users over an Independent Writer is safe so long as the users don't use both sides of the cache at once (ie home and target) I'm also interested in any recipe people have on GPFS policies to preseed and flush the cache. We plan to do all the migration using AFM over GPFS we're not currently using NFS and have no plans to start. I believe using GPFS is the faster method to preform the migration. Any suggestions and experience of doing similar migration jobs would be helpful. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ------------------------------ Message: 2 Date: Wed, 19 Oct 2016 13:46:02 -0400 From: Brian Marshall To: gpfsug main discussion list Subject: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="utf-8" All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 3 Date: Wed, 19 Oct 2016 18:10:38 +0000 From: "Simon Thompson (Research Computing - IT Services)" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="us-ascii" mmdiag --network Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Brian Marshall [mimarsh2 at vt.edu] Sent: 19 October 2016 18:46 To: gpfsug main discussion list Subject: [gpfsug-discuss] subnets All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall ------------------------------ Message: 4 Date: Wed, 19 Oct 2016 20:15:52 +0200 From: "Uwe Falke" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="ISO-8859-1" Hi Brian, you might use mmfsadm saferdump tscomm to check on which route peer cluster members are reached. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Frank Hammer, Thorsten Moehring Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: Brian Marshall To: gpfsug main discussion list Date: 10/19/2016 07:46 PM Subject: [gpfsug-discuss] subnets Sent by: gpfsug-discuss-bounces at spectrumscale.org All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ------------------------------ Message: 5 Date: Wed, 19 Oct 2016 20:11:57 +0000 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Subject: [gpfsug-discuss] Will there be any more GPFS 4.2.0-x releases? Message-ID: <142FECE0-E157-42D9-BC10-4C48E78FA065 at vanderbilt.edu> Content-Type: text/plain; charset="utf-8" Hi All, We?re currently running GPFS 4.2.0-4 with an efix installed and now we need a 2nd efix. I?m not a big fan of adding efix to efix and would prefer to go to a new PTF that contains both efixes. So ? is there going to be a GPFS 4.2.0-5 (it?s been a longer than normal interval since PTF 4 came out) or do we need to go to GPFS 4.2.1-x? If the latter, any major changes to watch out for? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 49 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-1466780990050_DSTlogo.png.png Type: image/png Size: 6282 bytes Desc: OutlookEmoji-1466780990050_DSTlogo.png.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-httpwww.prweb.comreleases201606prweb13504050.htm.jpg Type: image/jpeg Size: 14887 bytes Desc: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 53 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-1466780990050_DSTlogo.png.png Type: image/png Size: 6282 bytes Desc: OutlookEmoji-1466780990050_DSTlogo.png.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-httpwww.prweb.comreleases201606prweb13504050.htm.jpg Type: image/jpeg Size: 14887 bytes Desc: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg URL: From tortay at cc.in2p3.fr Sat Oct 22 11:45:23 2016 From: tortay at cc.in2p3.fr (Loic Tortay) Date: Sat, 22 Oct 2016 12:45:23 +0200 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) In-Reply-To: References: Message-ID: <580B4343.5020700@cc.in2p3.fr> On 21/10/2016 20:46, Bill Pappas wrote: [...] > > If you are using GPFS as the conduit between the home and cache > (i.e. no NFS), I would still ask the same question, more with respect to > stability for large file lists during the initial prefetch stages. > Hello, I'm in the final stage of what the AFM documentation calls an "incremental migration", for a filesystem with 100 million files. (GPFS 4.1.1, single cluster migration, "hardware/filesystem refresh" use case) I initially tried to use the NFS transport but found it too unreliable (and, in my opinion, very poorly documented). As I was about to give up on AFM, I tried using the GPFS transport (after seeing a trivially simple example on slides by someone from ANL) and things just started to work (almost) as I expected. For the files lists, I use data produced for our monitoring system that relies on snapshots fast scans (we do daily statistics on all objects in our GPFS filesystems). Our data gathering tool encodes object names in the RFC3986 (URL encoding) format which is what I found "mmafmctl prefetch" expects for "special" filenames. I understand that the policy engine does this too which, I guess, is what the documentation means by "generate a file list using policy" (sic), yet "mmafmctl prefetch" does not seem to accept/like files produced by a simple "LIST" policy (and the documentation lacks an example). As you did, I found that trying to prefetch large lists of files does not work reliably. I remember reading on that list someone (from IBM Germany, I think) recommanding to limit the number of a files in a single prefetch to 2 millions. This appears to be the sweet spot for my needs, as I can split the files list in 2 millions parts (the largest fileset in the "home" filesystem has 26 million files) and at the same time manage the issues I mention below. To keep up with the updates on the "home" filesystem (modified files), I rely on the "gpfs_winflags" GPFS extended attribute (the GPFS_WINATTR_OFFLINE bit is on for modified files, see "mmlsattr -L /cachefs/objectname" output). By chance, this attribute is included in the files produced for our statistics. This allows us to avoid doing a prefetch of all the files "continuously", since the file scan indeed appears to use only the (single) gateway node for the fileset being prefetched. In my specific configuration/environment, there are still several issues: . There is a significant memory and "slab" leak on the gateway nodes which can easily lead to a completely unreachable gateway node. These leaks appear directly related to the number of files prefetched. Stoping GPFS on the gateway node only releases some of the memory but none of the "slab", which requires a node reboot. . There is also a need to increase the "fs.file-max" sysctl on the gateway nodes to a value larger than the default (I use 10M), to avoid the kernel running out of file descriptors, since this leads to node being unreachable too. . Sometimes, an AFM association will go to the "Unmounted" state (for no apparent reason). The only reliable way to bring it back to "Active" state I found is to : unmount the "cache" filesystem from all nodes mouting it, unmounting/remounting the "home" filesystem on the gateway nodes, then remounting the "cache" filesystem where it is needed (gateway nodes, etc.) and finally using "ls -la" in the "cache" filesystem to bring the AFM associations back into the Active state. As I'm doing an "incremental migration", the "home" fileystem is still actively used and unmounting it on all nodes (as suggested by the documentation) is not an option. . I include directories and symlinks in the file lists for the prefetch. This ensures symlinks targets are there without needing a "ls" or "stat" in the "cache" filesystem ("incremental migration"). . The only reliable way I found to have "mmafmctl prefetch" accept files lists is to use the "--home-list-file" & "--home-fs-path" options. In my experience, in my environment, using AFM for migrations requires a significant amount of work and hand-holding to get a useful result. Since this migration is actually only the first step in a extensive multiple filesystems migration/merge plan, I'm pondering wether I'll use AFM for the rest of the operations. Sorry, this mail is too long, Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2931 bytes Desc: S/MIME Cryptographic Signature URL: From makaplan at us.ibm.com Sat Oct 22 14:05:34 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Sat, 22 Oct 2016 09:05:34 -0400 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames In-Reply-To: <580B4343.5020700@cc.in2p3.fr> References: <580B4343.5020700@cc.in2p3.fr> Message-ID: RULE .... EXTERNAL LIST ... ESCAPE '%' will direct mmapplypolicy to encode pathnames following RFC3986. Use ESCAPE '%/@' or similar for a more "relaxed" encoding. See the "Information lifecycle management" chapter of the official pubs for more details. Read the section that begins ... ESCAPE '%SpecialCharacters' Specifies that path names and the SHOW('string') expressions within the associated file lists are encoded with a scheme based on RFC3986 URI percent encoding. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tortay at cc.in2p3.fr Sat Oct 22 14:15:25 2016 From: tortay at cc.in2p3.fr (Loic Tortay) Date: Sat, 22 Oct 2016 15:15:25 +0200 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames In-Reply-To: References: <580B4343.5020700@cc.in2p3.fr> Message-ID: <580B666D.3090200@cc.in2p3.fr> On 22/10/2016 15:05, Marc A Kaplan wrote: > RULE .... EXTERNAL LIST ... ESCAPE '%' > will direct mmapplypolicy to encode pathnames following RFC3986. > Use ESCAPE '%/@' or similar for a more "relaxed" encoding. > > See the "Information lifecycle management" chapter of the official pubs > for more details. > Read the section that begins ... > > ESCAPE '%SpecialCharacters' > Specifies that path names and the SHOW('string') expressions within the > associated file lists are > encoded with a scheme based on RFC3986 URI percent encoding. > Hello, I've read (and used many times) that part of the fine documentation. My issue is with the documentation of what "mmafcmtl prefetch" expects. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2931 bytes Desc: S/MIME Cryptographic Signature URL: From makaplan at us.ibm.com Sat Oct 22 20:24:22 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Sat, 22 Oct 2016 15:24:22 -0400 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames In-Reply-To: <580B666D.3090200@cc.in2p3.fr> References: <580B4343.5020700@cc.in2p3.fr> <580B666D.3090200@cc.in2p3.fr> Message-ID: Hmmm.... Peeking into the script, it looks like some of AFM uses the simple \\ and \n file list escaping convention (the mmapplypolicy default with no ESCAPE specified) ... And other parts use ESCAPE '%'. Is this inconsistency hidden or does the CLI user have to deal with it? From: Loic Tortay To: gpfsug main discussion list Date: 10/22/2016 09:15 AM Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Sent by: gpfsug-discuss-bounces at spectrumscale.org On 22/10/2016 15:05, Marc A Kaplan wrote: > RULE .... EXTERNAL LIST ... ESCAPE '%' > will direct mmapplypolicy to encode pathnames following RFC3986. > Use ESCAPE '%/@' or similar for a more "relaxed" encoding. > > See the "Information lifecycle management" chapter of the official pubs > for more details. > Read the section that begins ... > > ESCAPE '%SpecialCharacters' > Specifies that path names and the SHOW('string') expressions within the > associated file lists are > encoded with a scheme based on RFC3986 URI percent encoding. > Hello, I've read (and used many times) that part of the fine documentation. My issue is with the documentation of what "mmafcmtl prefetch" expects. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | [attachment "smime.p7s" deleted by Marc A Kaplan/Watson/IBM] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtolson at us.ibm.com Mon Oct 24 01:41:07 2016 From: jtolson at us.ibm.com (John T Olson) Date: Sun, 23 Oct 2016 17:41:07 -0700 Subject: [gpfsug-discuss] New whitepaper published In-Reply-To: References: Message-ID: A new white paper has been published which shows integration of the Varonis UNIX agent in Spectrum Scale for audit logging. Here is a link to the paper: https://www.ibm.com/developerworks/community/wikis/form/api/wiki/fa32927c-e904-49cc-a4cc-870bcc8e307c/page/f0cc9b82-a133-41b4-83fe-3f560e95b35a/attachment/0ab62645-e0ab-4377-81e7-abd11879bb75/media/Spectrum_Scale_Varonis_Audit_Logging.pdf Thanks, John John T. Olson, Ph.D., MI.C., K.EY. Master Inventor, Software Defined Storage 957/9032-1 Tucson, AZ, 85744 (520) 799-5185, tie 321-5185 (FAX: 520-799-4237) Email: jtolson at us.ibm.com "Do or do not. There is no try." - Yoda Olson's Razor: Any situation that we, as humans, can encounter in life can be modeled by either an episode of The Simpsons or Seinfeld. -------------- next part -------------- An HTML attachment was scrubbed... URL: From laurence at qsplace.co.uk Mon Oct 24 08:10:12 2016 From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow) Date: Mon, 24 Oct 2016 08:10:12 +0100 Subject: [gpfsug-discuss] Two new whitepapers published In-Reply-To: <7fifaf55s6bq10jlpkvl5he9.1477081370569@email.android.com> References: <36EC1962-F71F-4492-A175-4BA3E9D97A8D@qsplace.co.uk>, <7fifaf55s6bq10jlpkvl5he9.1477081370569@email.android.com> Message-ID: <0386E88D-DC69-42DA-ACF7-BF0B0F45CD4E@qsplace.co.uk> Hi Peter, I've always been under the impression that this is a ball park figure that changes some of the on disk data structures to help parallel access to the filesystem I try to estimate the number of clients and then add ~10% (depending on the cluster size ofc). I have tested the default 32 vs 200 on a previous cluster however didn't find a difference in performance when testing up to 50 clients concurrently with IOR random and sequential. Maybe the difference is more subtle that just throughput? What I've always found strange is that the value can be changed on the filesystem however I don't believe this change effects an already created filesystem. -- Lauz On 21 October 2016 21:35:15 BST, Peter Childs wrote: > >Reading through them and we'll worth read. > >How important is setting the correct cluster size at file system >creation time? Ie with "mmchfs -n 256" ie how much band of error can >you get away with? > >We have a cluster of 240 nodes that was setup with the default 32 >setting, it's now about to grow to ~300 nodes. Is this likly to causing >as an issue, which is difficult to fix. The manual says it can be >adjusted but this white paper suggests not. > >Fortunately were also migrating our storage to new hardware, so have a >good opportunity to get the setting right, this time around. > >Has anyone got any stats on the benefits of getting it "right" vs >getting it wrong. > > > > >Peter Childs >Research Storage >ITS Research and Teaching Support >Queen Mary, University of London > > >---- Andreas Landh?u?er wrote ---- > >On Fri, 21 Oct 2016, Laurence Horrocks-Barlow >wrote: > >Found it! > >thanks Yuri, for the two very interesting papers about the Metadata and >replication. > >We always used the rule of thumb 5% for metadata, since we are having >separate devices for metadata, and tiny fast disks aren't available >anymore, we are getting a bunch of larger fast disks. We never >experienced >a problem with the (more or less) amount of metadata ... > > Andreas > > >> Right down the bottom of the page under attachments. >> >> -- Lauz >> >> On 21 October 2016 07:43:40 BST, "Andreas Landh?u?er" > wrote: >>> >>> Hi Yuri, >>> >>> Arrg, can't find them, page has been last updated on Aug, 18 by >>> JohnTOlson, maybe its internal and not open for the public? >>> >>> Ciao >>> >>> Andreas >>> >>> On Fri, 21 Oct 2016, Yuri L Volobuev wrote: >>> >>>> >>>> >>>> Esteemed GPFS and Spectrum Scale users, >>>> >>>> For your reading enjoyment, two new whitepapers have been posted to >>> the >>>> Spectrum Scale Wiki: >>>> >>>> Spectrum Scale: Replication in GPFS >>>> Spectrum Scale: GPFS Metadata >>>> >>>> The URL for the parent page is >>>> >>> >https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20 >>>> (GPFS)/page/White%20Papers%20%26%20Media >>>> >>>> The two .pdf documents are accessible through the Attachment >section >>> at the >>>> bottom of the page. Unfortunately, dW "spam prevention engine" >does >>> a very >>>> good job preventing me from "spamming" the page to actually add >>> links. >>>> >>>> Best regards, >>>> >>>> Yuri >>>> >>> >>> -- >>> Andreas Landh?u?er +49 151 12133027 >(mobile) >>> alandhae at gmx.de >>> >>> >------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> > >-- >Andreas Landh?u?er +49 151 12133027 >(mobile) >alandhae at gmx.de > > >------------------------------------------------------------------------ > >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vpuvvada at in.ibm.com Mon Oct 24 10:44:19 2016 From: vpuvvada at in.ibm.com (Venkateswara R Puvvada) Date: Mon, 24 Oct 2016 15:14:19 +0530 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames In-Reply-To: References: <580B4343.5020700@cc.in2p3.fr><580B666D.3090200@cc.in2p3.fr> Message-ID: Loic, mmafmctl prefecth expects encoded list file, and it is not documented correctly. Issues like memory leak, file descriptor leak, and fileset going into Unmounted state were fixed in later releases (4.2.1/4.2.2). All your points are correct with respect to AFM migration. There is manual intervention required. Also prefetch does not give list of files which were failed during data read. Users need to run policy to find all uncached files today. ~Venkat (vpuvvada at in.ibm.com) From: "Marc A Kaplan" To: gpfsug main discussion list Date: 10/23/2016 12:54 AM Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Sent by: gpfsug-discuss-bounces at spectrumscale.org Hmmm.... Peeking into the script, it looks like some of AFM uses the simple \\ and \n file list escaping convention (the mmapplypolicy default with no ESCAPE specified) ... And other parts use ESCAPE '%'. Is this inconsistency hidden or does the CLI user have to deal with it? From: Loic Tortay To: gpfsug main discussion list Date: 10/22/2016 09:15 AM Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Sent by: gpfsug-discuss-bounces at spectrumscale.org On 22/10/2016 15:05, Marc A Kaplan wrote: > RULE .... EXTERNAL LIST ... ESCAPE '%' > will direct mmapplypolicy to encode pathnames following RFC3986. > Use ESCAPE '%/@' or similar for a more "relaxed" encoding. > > See the "Information lifecycle management" chapter of the official pubs > for more details. > Read the section that begins ... > > ESCAPE '%SpecialCharacters' > Specifies that path names and the SHOW('string') expressions within the > associated file lists are > encoded with a scheme based on RFC3986 URI percent encoding. > Hello, I've read (and used many times) that part of the fine documentation. My issue is with the documentation of what "mmafcmtl prefetch" expects. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | [attachment "smime.p7s" deleted by Marc A Kaplan/Watson/IBM] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From erich at uw.edu Mon Oct 24 17:16:56 2016 From: erich at uw.edu (Eric Horst) Date: Mon, 24 Oct 2016 09:16:56 -0700 Subject: [gpfsug-discuss] Using AFM to migrate within the same cluster Message-ID: The recent conversation about AFM has been interesting. I've read the documentation several times and this is my question. Can AFM be used to migrate between two filesystems in the same cluster? There are examples of moving between clusters with NFS or native protocol but I've got a simple situation of needing to transparently move 100M files between two existing filesystems. Thanks, -Eric -- Eric Horst University of Washington From tortay at cc.in2p3.fr Mon Oct 24 17:50:51 2016 From: tortay at cc.in2p3.fr (Loic Tortay) Date: Mon, 24 Oct 2016 18:50:51 +0200 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames In-Reply-To: References: <580B4343.5020700@cc.in2p3.fr> <580B666D.3090200@cc.in2p3.fr> Message-ID: On 10/24/2016 11:44 AM, Venkateswara R Puvvada wrote: > > mmafmctl prefecth expects encoded list file, and it is not documented > correctly. Issues like memory leak, file descriptor leak, and fileset > going into Unmounted state were fixed in later releases (4.2.1/4.2.2). All > your points are correct with respect to AFM migration. There is manual > intervention required. Also prefetch does not give list of files which > were failed during data read. Users need to run policy to find all > uncached files today. > Hello, For the record, I have completed today my AFM migration of a filesystem with 100 million files. Users are now accessing the new filesystem. After disabling user access and a last "prefetch", the AFM filesets were converted to independent filesets. Less than 600 files were then found to be different between the "home" and the "cache" filesystems with a metadata comparison (I just copied the files from the old filesystem to the new one). I have compared the MD5 of a few thousand randomly selected files and found no differences between the "home" and the "cache" filesystems. I expect the users to let us know if they find something different (they have been instructed to do so). We'll keep the "home" filesystem around for some time, just in case there is a problem. Maybe something else that should be mentionned in the documentation is what to do with the ".ptrash" directories after the AFM filesets have been converted. I removed them since they contained files that had clearly been deleted by the users. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | From sfadden at us.ibm.com Mon Oct 24 17:57:33 2016 From: sfadden at us.ibm.com (Scott Fadden) Date: Mon, 24 Oct 2016 09:57:33 -0700 Subject: [gpfsug-discuss] Using AFM to migrate within the same cluster In-Reply-To: References: Message-ID: Yes you can use AFM to move data within a cluster. If you are using the NSD protocol the target needs to be a separate file system, if you are using NFS it needs to be an NFS export. Scott Fadden Spectrum Scale - Technical Marketing Phone: (503) 880-5833 sfadden at us.ibm.com http://www.ibm.com/systems/storage/spectrum/scale From: Eric Horst To: gpfsug main discussion list Date: 10/24/2016 09:17 AM Subject: [gpfsug-discuss] Using AFM to migrate within the same cluster Sent by: gpfsug-discuss-bounces at spectrumscale.org The recent conversation about AFM has been interesting. I've read the documentation several times and this is my question. Can AFM be used to migrate between two filesystems in the same cluster? There are examples of moving between clusters with NFS or native protocol but I've got a simple situation of needing to transparently move 100M files between two existing filesystems. Thanks, -Eric -- Eric Horst University of Washington _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From YARD at il.ibm.com Mon Oct 24 18:05:00 2016 From: YARD at il.ibm.com (Yaron Daniel) Date: Mon, 24 Oct 2016 20:05:00 +0300 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames In-Reply-To: References: <580B4343.5020700@cc.in2p3.fr><580B666D.3090200@cc.in2p3.fr> Message-ID: Hi Maybe worth also to check if there are any orphan files in the NEW fs ? Regards Yaron Daniel 94 Em Ha'Moshavot Rd Server, Storage and Data Services - Team Leader Petach Tiqva, 49527 Global Technology Services Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com IBM Israel From: Loic Tortay To: gpfsug main discussion list Date: 10/24/2016 07:50 PM Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Sent by: gpfsug-discuss-bounces at spectrumscale.org On 10/24/2016 11:44 AM, Venkateswara R Puvvada wrote: > > mmafmctl prefecth expects encoded list file, and it is not documented > correctly. Issues like memory leak, file descriptor leak, and fileset > going into Unmounted state were fixed in later releases (4.2.1/4.2.2). All > your points are correct with respect to AFM migration. There is manual > intervention required. Also prefetch does not give list of files which > were failed during data read. Users need to run policy to find all > uncached files today. > Hello, For the record, I have completed today my AFM migration of a filesystem with 100 million files. Users are now accessing the new filesystem. After disabling user access and a last "prefetch", the AFM filesets were converted to independent filesets. Less than 600 files were then found to be different between the "home" and the "cache" filesystems with a metadata comparison (I just copied the files from the old filesystem to the new one). I have compared the MD5 of a few thousand randomly selected files and found no differences between the "home" and the "cache" filesystems. I expect the users to let us know if they find something different (they have been instructed to do so). We'll keep the "home" filesystem around for some time, just in case there is a problem. Maybe something else that should be mentionned in the documentation is what to do with the ".ptrash" directories after the AFM filesets have been converted. I removed them since they contained files that had clearly been deleted by the users. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1851 bytes Desc: not available URL: From bpappas at dstonline.com Mon Oct 24 19:03:07 2016 From: bpappas at dstonline.com (Bill Pappas) Date: Mon, 24 Oct 2016 18:03:07 +0000 Subject: [gpfsug-discuss] Using AFM to migrate files. (Bill Pappas to Loric Totay In-Reply-To: References: Message-ID: >>For the record, I have completed today my AFM migration of a filesystem with 100 million files. Users are now accessing the new filesystem. Loric-> Hi. I was wondering, what version of GPFS where you running on the home and cache clusters? I take it you broke up the prefetch list into smaller (for example <2 million) file lists? If not, how? How much capacity did you migfrate over and how long did this process take? Thanks. Bill Pappas 901-619-0585 bpappas at dstonline.com [1466780990050_DSTlogo.png] [http://www.prweb.com/releases/2016/06/prweb13504050.htm] http://www.prweb.com/releases/2016/06/prweb13504050.htm ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org Sent: Monday, October 24, 2016 12:05 PM To: gpfsug-discuss at spectrumscale.org Subject: gpfsug-discuss Digest, Vol 57, Issue 60 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Using AFM to migrate within the same cluster (Eric Horst) 2. Re: Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames (Loic Tortay) 3. Re: Using AFM to migrate within the same cluster (Scott Fadden) 4. Re: Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames (Yaron Daniel) ---------------------------------------------------------------------- Message: 1 Date: Mon, 24 Oct 2016 09:16:56 -0700 From: Eric Horst To: gpfsug main discussion list Subject: [gpfsug-discuss] Using AFM to migrate within the same cluster Message-ID: Content-Type: text/plain; charset=UTF-8 The recent conversation about AFM has been interesting. I've read the documentation several times and this is my question. Can AFM be used to migrate between two filesystems in the same cluster? There are examples of moving between clusters with NFS or native protocol but I've got a simple situation of needing to transparently move 100M files between two existing filesystems. Thanks, -Eric -- Eric Horst University of Washington ------------------------------ Message: 2 Date: Mon, 24 Oct 2016 18:50:51 +0200 From: Loic Tortay To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Message-ID: Content-Type: text/plain; charset=utf-8 On 10/24/2016 11:44 AM, Venkateswara R Puvvada wrote: > > mmafmctl prefecth expects encoded list file, and it is not documented > correctly. Issues like memory leak, file descriptor leak, and fileset > going into Unmounted state were fixed in later releases (4.2.1/4.2.2). All > your points are correct with respect to AFM migration. There is manual > intervention required. Also prefetch does not give list of files which > were failed during data read. Users need to run policy to find all > uncached files today. > Hello, For the record, I have completed today my AFM migration of a filesystem with 100 million files. Users are now accessing the new filesystem. After disabling user access and a last "prefetch", the AFM filesets were converted to independent filesets. Less than 600 files were then found to be different between the "home" and the "cache" filesystems with a metadata comparison (I just copied the files from the old filesystem to the new one). I have compared the MD5 of a few thousand randomly selected files and found no differences between the "home" and the "cache" filesystems. I expect the users to let us know if they find something different (they have been instructed to do so). We'll keep the "home" filesystem around for some time, just in case there is a problem. Maybe something else that should be mentionned in the documentation is what to do with the ".ptrash" directories after the AFM filesets have been converted. I removed them since they contained files that had clearly been deleted by the users. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | ------------------------------ Message: 3 Date: Mon, 24 Oct 2016 09:57:33 -0700 From: "Scott Fadden" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Using AFM to migrate within the same cluster Message-ID: Content-Type: text/plain; charset="us-ascii" Yes you can use AFM to move data within a cluster. If you are using the NSD protocol the target needs to be a separate file system, if you are using NFS it needs to be an NFS export. Scott Fadden Spectrum Scale - Technical Marketing Phone: (503) 880-5833 sfadden at us.ibm.com http://www.ibm.com/systems/storage/spectrum/scale From: Eric Horst To: gpfsug main discussion list Date: 10/24/2016 09:17 AM Subject: [gpfsug-discuss] Using AFM to migrate within the same cluster Sent by: gpfsug-discuss-bounces at spectrumscale.org The recent conversation about AFM has been interesting. I've read the documentation several times and this is my question. Can AFM be used to migrate between two filesystems in the same cluster? There are examples of moving between clusters with NFS or native protocol but I've got a simple situation of needing to transparently move 100M files between two existing filesystems. Thanks, -Eric -- Eric Horst University of Washington _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 4 Date: Mon, 24 Oct 2016 20:05:00 +0300 From: "Yaron Daniel" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Message-ID: Content-Type: text/plain; charset="iso-8859-1" Hi Maybe worth also to check if there are any orphan files in the NEW fs ? Regards Yaron Daniel 94 Em Ha'Moshavot Rd Server, Storage and Data Services - Team Leader Petach Tiqva, 49527 Global Technology Services Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com IBM Israel From: Loic Tortay To: gpfsug main discussion list Date: 10/24/2016 07:50 PM Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Sent by: gpfsug-discuss-bounces at spectrumscale.org On 10/24/2016 11:44 AM, Venkateswara R Puvvada wrote: > > mmafmctl prefecth expects encoded list file, and it is not documented > correctly. Issues like memory leak, file descriptor leak, and fileset > going into Unmounted state were fixed in later releases (4.2.1/4.2.2). All > your points are correct with respect to AFM migration. There is manual > intervention required. Also prefetch does not give list of files which > were failed during data read. Users need to run policy to find all > uncached files today. > Hello, For the record, I have completed today my AFM migration of a filesystem with 100 million files. Users are now accessing the new filesystem. After disabling user access and a last "prefetch", the AFM filesets were converted to independent filesets. Less than 600 files were then found to be different between the "home" and the "cache" filesystems with a metadata comparison (I just copied the files from the old filesystem to the new one). I have compared the MD5 of a few thousand randomly selected files and found no differences between the "home" and the "cache" filesystems. I expect the users to let us know if they find something different (they have been instructed to do so). We'll keep the "home" filesystem around for some time, just in case there is a problem. Maybe something else that should be mentionned in the documentation is what to do with the ".ptrash" directories after the AFM filesets have been converted. I removed them since they contained files that had clearly been deleted by the users. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1851 bytes Desc: not available URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 60 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-1466780990050_DSTlogo.png.png Type: image/png Size: 6282 bytes Desc: OutlookEmoji-1466780990050_DSTlogo.png.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg Type: image/jpeg Size: 14887 bytes Desc: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg URL: From vpuvvada at in.ibm.com Tue Oct 25 05:53:10 2016 From: vpuvvada at in.ibm.com (Venkateswara R Puvvada) Date: Tue, 25 Oct 2016 10:23:10 +0530 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames In-Reply-To: References: <580B4343.5020700@cc.in2p3.fr><580B666D.3090200@cc.in2p3.fr> Message-ID: Loic, It is good to hear that migration is completed. Did you find any prefetch errors in mmfs.log for the files which are different between cache and home ? Not all errors are logged, there is a plan to generate the list of read failed file names after prefetch completion. Informaion about files moving to .ptrash is documented @ http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.1/com.ibm.spectrum.scale.v4r21.doc/bl1ins_internaldirectoriesAFM.htm ~Venkat (vpuvvada at in.ibm.com) From: Loic Tortay To: gpfsug main discussion list Date: 10/24/2016 10:20 PM Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Sent by: gpfsug-discuss-bounces at spectrumscale.org On 10/24/2016 11:44 AM, Venkateswara R Puvvada wrote: > > mmafmctl prefecth expects encoded list file, and it is not documented > correctly. Issues like memory leak, file descriptor leak, and fileset > going into Unmounted state were fixed in later releases (4.2.1/4.2.2). All > your points are correct with respect to AFM migration. There is manual > intervention required. Also prefetch does not give list of files which > were failed during data read. Users need to run policy to find all > uncached files today. > Hello, For the record, I have completed today my AFM migration of a filesystem with 100 million files. Users are now accessing the new filesystem. After disabling user access and a last "prefetch", the AFM filesets were converted to independent filesets. Less than 600 files were then found to be different between the "home" and the "cache" filesystems with a metadata comparison (I just copied the files from the old filesystem to the new one). I have compared the MD5 of a few thousand randomly selected files and found no differences between the "home" and the "cache" filesystems. I expect the users to let us know if they find something different (they have been instructed to do so). We'll keep the "home" filesystem around for some time, just in case there is a problem. Maybe something else that should be mentionned in the documentation is what to do with the ".ptrash" directories after the AFM filesets have been converted. I removed them since they contained files that had clearly been deleted by the users. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From radhika.p at in.ibm.com Tue Oct 25 08:24:56 2016 From: radhika.p at in.ibm.com (Radhika A Parameswaran) Date: Tue, 25 Oct 2016 12:54:56 +0530 Subject: [gpfsug-discuss] Using AFM to migrate files In-Reply-To: References: Message-ID: Bill, The issues/limitation with preparing a listfile with >2M files in the file list has been fixed in 4.2.1. It has also been internally checked with 45M file list. Peter, The 4.2.1 man pages and prefetch section has added some examples of policy and listfile options for prefetch. Please take a look at them and let us know if those help. Loic, We will add about removing .ptrash to the migration usecase documentation. Can you please share some details about your dataset and performance for migrating the 100M (time for listfile processing and actual transfer) ? Thanks and Regards Radhika From: gpfsug-discuss-request at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Date: 10/24/2016 11:33 PM Subject: gpfsug-discuss Digest, Vol 57, Issue 61 Sent by: gpfsug-discuss-bounces at spectrumscale.org Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Using AFM to migrate files. (Bill Pappas to Loric Totay (Bill Pappas) ---------------------------------------------------------------------- Message: 1 Date: Mon, 24 Oct 2016 18:03:07 +0000 From: Bill Pappas To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Bill Pappas to Loric Totay Message-ID: Content-Type: text/plain; charset="iso-8859-1" >>For the record, I have completed today my AFM migration of a filesystem with 100 million files. Users are now accessing the new filesystem. Loric-> Hi. I was wondering, what version of GPFS where you running on the home and cache clusters? I take it you broke up the prefetch list into smaller (for example <2 million) file lists? If not, how? How much capacity did you migfrate over and how long did this process take? Thanks. Bill Pappas 901-619-0585 bpappas at dstonline.com [1466780990050_DSTlogo.png] [http://www.prweb.com/releases/2016/06/prweb13504050.htm] http://www.prweb.com/releases/2016/06/prweb13504050.htm ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org Sent: Monday, October 24, 2016 12:05 PM To: gpfsug-discuss at spectrumscale.org Subject: gpfsug-discuss Digest, Vol 57, Issue 60 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Using AFM to migrate within the same cluster (Eric Horst) 2. Re: Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames (Loic Tortay) 3. Re: Using AFM to migrate within the same cluster (Scott Fadden) 4. Re: Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames (Yaron Daniel) ---------------------------------------------------------------------- Message: 1 Date: Mon, 24 Oct 2016 09:16:56 -0700 From: Eric Horst To: gpfsug main discussion list Subject: [gpfsug-discuss] Using AFM to migrate within the same cluster Message-ID: Content-Type: text/plain; charset=UTF-8 The recent conversation about AFM has been interesting. I've read the documentation several times and this is my question. Can AFM be used to migrate between two filesystems in the same cluster? There are examples of moving between clusters with NFS or native protocol but I've got a simple situation of needing to transparently move 100M files between two existing filesystems. Thanks, -Eric -- Eric Horst University of Washington ------------------------------ Message: 2 Date: Mon, 24 Oct 2016 18:50:51 +0200 From: Loic Tortay To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Message-ID: Content-Type: text/plain; charset=utf-8 On 10/24/2016 11:44 AM, Venkateswara R Puvvada wrote: > > mmafmctl prefecth expects encoded list file, and it is not documented > correctly. Issues like memory leak, file descriptor leak, and fileset > going into Unmounted state were fixed in later releases (4.2.1/4.2.2). All > your points are correct with respect to AFM migration. There is manual > intervention required. Also prefetch does not give list of files which > were failed during data read. Users need to run policy to find all > uncached files today. > Hello, For the record, I have completed today my AFM migration of a filesystem with 100 million files. Users are now accessing the new filesystem. After disabling user access and a last "prefetch", the AFM filesets were converted to independent filesets. Less than 600 files were then found to be different between the "home" and the "cache" filesystems with a metadata comparison (I just copied the files from the old filesystem to the new one). I have compared the MD5 of a few thousand randomly selected files and found no differences between the "home" and the "cache" filesystems. I expect the users to let us know if they find something different (they have been instructed to do so). We'll keep the "home" filesystem around for some time, just in case there is a problem. Maybe something else that should be mentionned in the documentation is what to do with the ".ptrash" directories after the AFM filesets have been converted. I removed them since they contained files that had clearly been deleted by the users. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | ------------------------------ Message: 3 Date: Mon, 24 Oct 2016 09:57:33 -0700 From: "Scott Fadden" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Using AFM to migrate within the same cluster Message-ID: Content-Type: text/plain; charset="us-ascii" Yes you can use AFM to move data within a cluster. If you are using the NSD protocol the target needs to be a separate file system, if you are using NFS it needs to be an NFS export. Scott Fadden Spectrum Scale - Technical Marketing Phone: (503) 880-5833 sfadden at us.ibm.com http://www.ibm.com/systems/storage/spectrum/scale From: Eric Horst To: gpfsug main discussion list Date: 10/24/2016 09:17 AM Subject: [gpfsug-discuss] Using AFM to migrate within the same cluster Sent by: gpfsug-discuss-bounces at spectrumscale.org The recent conversation about AFM has been interesting. I've read the documentation several times and this is my question. Can AFM be used to migrate between two filesystems in the same cluster? There are examples of moving between clusters with NFS or native protocol but I've got a simple situation of needing to transparently move 100M files between two existing filesystems. Thanks, -Eric -- Eric Horst University of Washington _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161024/866c55cf/attachment-0001.html > ------------------------------ Message: 4 Date: Mon, 24 Oct 2016 20:05:00 +0300 From: "Yaron Daniel" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Message-ID: Content-Type: text/plain; charset="iso-8859-1" Hi Maybe worth also to check if there are any orphan files in the NEW fs ? Regards Yaron Daniel 94 Em Ha'Moshavot Rd Server, Storage and Data Services - Team Leader Petach Tiqva, 49527 Global Technology Services Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com IBM Israel From: Loic Tortay To: gpfsug main discussion list Date: 10/24/2016 07:50 PM Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Sent by: gpfsug-discuss-bounces at spectrumscale.org On 10/24/2016 11:44 AM, Venkateswara R Puvvada wrote: > > mmafmctl prefecth expects encoded list file, and it is not documented > correctly. Issues like memory leak, file descriptor leak, and fileset > going into Unmounted state were fixed in later releases (4.2.1/4.2.2). All > your points are correct with respect to AFM migration. There is manual > intervention required. Also prefetch does not give list of files which > were failed during data read. Users need to run policy to find all > uncached files today. > Hello, For the record, I have completed today my AFM migration of a filesystem with 100 million files. Users are now accessing the new filesystem. After disabling user access and a last "prefetch", the AFM filesets were converted to independent filesets. Less than 600 files were then found to be different between the "home" and the "cache" filesystems with a metadata comparison (I just copied the files from the old filesystem to the new one). I have compared the MD5 of a few thousand randomly selected files and found no differences between the "home" and the "cache" filesystems. I expect the users to let us know if they find something different (they have been instructed to do so). We'll keep the "home" filesystem around for some time, just in case there is a problem. Maybe something else that should be mentionned in the documentation is what to do with the ".ptrash" directories after the AFM filesets have been converted. I removed them since they contained files that had clearly been deleted by the users. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161024/449a0cf1/attachment.html > -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1851 bytes Desc: not available URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161024/449a0cf1/attachment.gif > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 60 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161024/fff3f370/attachment.html > -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-1466780990050_DSTlogo.png.png Type: image/png Size: 6282 bytes Desc: OutlookEmoji-1466780990050_DSTlogo.png.png URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161024/fff3f370/attachment.png > -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg Type: image/jpeg Size: 14887 bytes Desc: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161024/fff3f370/attachment.jpg > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 61 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: From tortay at cc.in2p3.fr Tue Oct 25 13:01:01 2016 From: tortay at cc.in2p3.fr (Loic Tortay) Date: Tue, 25 Oct 2016 14:01:01 +0200 Subject: [gpfsug-discuss] Using AFM to migrate files. (Bill Pappas to Loric Totay In-Reply-To: References: Message-ID: <05eb9917-0b3a-1bde-fc04-9acaed06fe29@cc.in2p3.fr> On 10/24/2016 08:03 PM, Bill Pappas wrote: > > Loric-> Hi. I was wondering, what version of GPFS where you running > on the home and cache clusters? I take it you broke up the prefetch list > into smaller (for example <2 million) file lists? If not, how? How much > capacity did you migfrate over and how long did this process take? Thanks. > Hello, We use GPFS 4.1.1-8. The migration was within a single cluster. The files lists were split in 2M files parts. The amount of data migrated is rather small (~70 TB), but there are ~100M files, including many extremely small files: the median (not average) file size is 2705 bytes. The migration itself took about 3 weeks: I initially did the first "mmafmctl prefetches" one fileset at a time (& in slices of 2M files), to manage the various issues mentionned previously and minimize the impact on the "home" filesystem since it was still being used by the end users. Following "prefetches" were done a few filesets at a time (filesets chosen to spread the load on the gateway nodes). We chose to do an "incremental migration" instead of a "progressive migration", since we were unable to get a good understanding of the performance impact of the AFM migration during our tests on a toy cluster running on VMs. Lo?c (Loic w/ a diaresis on the i, not Loric :-) -- | Lo?c Tortay - IN2P3 Computing Centre | From matt.thorpe at bodleian.ox.ac.uk Tue Oct 25 13:05:36 2016 From: matt.thorpe at bodleian.ox.ac.uk (Matt Thorpe) Date: Tue, 25 Oct 2016 12:05:36 +0000 Subject: [gpfsug-discuss] Forcing which node gets expelled? Message-ID: <63906C4AF5FFD0429565A743F6AD5BEDDAF29F@MBX04.ad.oak.ox.ac.uk> Hi, We are in the process of diagnosing a networking issue that is causing 2 of our 6 node GPFS cluster to expel each other (it appears they experience a temporary network connection outage and lose contact with each other). At present it's not consistent which gets expelled by the cluster manager, and I wondered if there was any way to always force a specific node to be expelled in this situation? Thanks and best regards, Matt -------- Matt Thorpe | BDLSS Systems Administrator Bodleian Libraries Osney One Building, Osney Mead, Oxford, OX2 0EW matt.thorpe at bodleian.ox.ac.uk | 01865 (2)80027 From tortay at cc.in2p3.fr Tue Oct 25 13:17:56 2016 From: tortay at cc.in2p3.fr (Loic Tortay) Date: Tue, 25 Oct 2016 14:17:56 +0200 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames In-Reply-To: References: <580B4343.5020700@cc.in2p3.fr> <580B666D.3090200@cc.in2p3.fr> Message-ID: On 10/24/2016 07:05 PM, Yaron Daniel wrote: > > Maybe worth also to check if there are any orphan files in the NEW fs ? > Hello, I ran a online "dry run" (-n) mmfsck which found a few lost blocks, but I'm not sure orphan files are detected with an online mmfsck. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | From Robert.Oesterlin at nuance.com Tue Oct 25 13:22:30 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Tue, 25 Oct 2016 12:22:30 +0000 Subject: [gpfsug-discuss] Forcing which node gets expelled? In-Reply-To: <63906C4AF5FFD0429565A743F6AD5BEDDAF29F@MBX04.ad.oak.ox.ac.uk> References: <63906C4AF5FFD0429565A743F6AD5BEDDAF29F@MBX04.ad.oak.ox.ac.uk> Message-ID: <4403E9DB-88B4-4281-91B5-E171D1284899@nuance.com> If you look at /usr/lpp/mmfs/samples/expelnode.sample you can use this as a base and install this in /var/mmfs/etc on the cluster manager. You can control which of the two nodes get expelled. We use it here to send an alert when a node is expelled. There is also "mmexpelnode" which you can force a particular node to be expelled. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Matt Thorpe Reply-To: gpfsug main discussion list Date: Tuesday, October 25, 2016 at 7:05 AM To: gpfsug main discussion list Subject: [EXTERNAL] [gpfsug-discuss] Forcing which node gets expelled? Hi, We are in the process of diagnosing a networking issue that is causing 2 of our 6 node GPFS cluster to expel each other (it appears they experience a temporary network connection outage and lose contact with each other). At present it's not consistent which gets expelled by the cluster manager, and I wondered if there was any way to always force a specific node to be expelled in this situation? Thanks and best regards, Matt -------- Matt Thorpe | BDLSS Systems Administrator Bodleian Libraries Osney One Building, Osney Mead, Oxford, OX2 0EW matt.thorpe at bodleian.ox.ac.uk | 01865 (2)80027 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Q_f6z64tvENxA9ac7rqWFj2jNd5IrpWEXcynJzHFjz4&s=jqso6xVB-V_zgLba-xjlWwiw3fNRan9NspsVq4PY4nA&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From tortay at cc.in2p3.fr Tue Oct 25 13:31:59 2016 From: tortay at cc.in2p3.fr (Loic Tortay) Date: Tue, 25 Oct 2016 14:31:59 +0200 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames In-Reply-To: References: <580B4343.5020700@cc.in2p3.fr> <580B666D.3090200@cc.in2p3.fr> Message-ID: On 25/10/2016 06:53, Venkateswara R Puvvada wrote: > Loic, > > It is good to hear that migration is completed. Did you find any prefetch > errors in mmfs.log for the files which are different between cache and > home ? Not all errors are logged, there is a plan to generate the list of > read failed file names after prefetch completion. > Hello, Indeed, I had missed those messages: $GATEWAY1: Mon Oct 24 10:44:05.725 2016: [E] AFM: Read file system $FS fileset juno file IDs [469826991.470473596.-1.-1,N] name CMakeFiles local error 21 $GATEWAY1: Mon Oct 24 10:50:05.655 2016: [E] AFM: Read file system $FS fileset mnm file IDs [637555582.-1.-1.-1,N] name Mn_G4_SansTube local error 21 >From what I can see, for the previous runs of "mmafmctl prefetch", there are much less error messages logged than the values of the "asyncReadFailed" counters displayed by "mmafmctl prefetch -j $FILESET -Y". Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | From UWEFALKE at de.ibm.com Tue Oct 25 13:32:11 2016 From: UWEFALKE at de.ibm.com (Uwe Falke) Date: Tue, 25 Oct 2016 14:32:11 +0200 Subject: [gpfsug-discuss] Forcing which node gets expelled? In-Reply-To: <63906C4AF5FFD0429565A743F6AD5BEDDAF29F@MBX04.ad.oak.ox.ac.uk> References: <63906C4AF5FFD0429565A743F6AD5BEDDAF29F@MBX04.ad.oak.ox.ac.uk> Message-ID: Usually, the cluster mgr, receiving a complaint from a node about another node being gone, checks its own connection to that other node. If that is positive it expells the requester, if not it follows the request and expells the other node. AFAIK, there are some more subtle algorithms in place if managers or quorum nodes are affected. Maybe that can be used to protect certain nodes from getting expelled by assigning some role in the cluster to them. I do however not know these exactly. That means: it is not easily controllable which one gets expelled. It is better to concentrate on fixing your connectivity issues, as GPFS will not feel comfortable in such a unreliable environment anyway. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Frank Hammer, Thorsten Moehring Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: Matt Thorpe To: gpfsug main discussion list Date: 10/25/2016 02:05 PM Subject: [gpfsug-discuss] Forcing which node gets expelled? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi, We are in the process of diagnosing a networking issue that is causing 2 of our 6 node GPFS cluster to expel each other (it appears they experience a temporary network connection outage and lose contact with each other). At present it's not consistent which gets expelled by the cluster manager, and I wondered if there was any way to always force a specific node to be expelled in this situation? Thanks and best regards, Matt -------- Matt Thorpe | BDLSS Systems Administrator Bodleian Libraries Osney One Building, Osney Mead, Oxford, OX2 0EW matt.thorpe at bodleian.ox.ac.uk | 01865 (2)80027 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From tortay at cc.in2p3.fr Tue Oct 25 14:01:20 2016 From: tortay at cc.in2p3.fr (Loic Tortay) Date: Tue, 25 Oct 2016 15:01:20 +0200 Subject: [gpfsug-discuss] Using AFM to migrate files In-Reply-To: References: Message-ID: On 10/25/2016 09:24 AM, Radhika A Parameswaran wrote: > > Loic, > We will add about removing .ptrash to the migration usecase documentation. > Can you please share some details about your dataset and performance for > migrating the 100M (time for listfile processing and actual transfer) ? > Hello, According to my logs, the files lists processing by "mmafmctl prefetch" with 2M files took between ~30 minutes and ~70 minutes. >From what I can see, the longer processing times were for filesets with many directories (less files per directory). For the actual data transfers, it varied widely from a few minutes (small files stored in the inode on a Flash-based storage unit) to a few hours (1 or 2 TB of files not large enough to trigger the parallel migration, while many batch jobs were accessing the "home" filesystem). Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | From knop at us.ibm.com Tue Oct 25 14:02:39 2016 From: knop at us.ibm.com (Felipe Knop) Date: Tue, 25 Oct 2016 09:02:39 -0400 Subject: [gpfsug-discuss] Forcing which node gets expelled? In-Reply-To: References: <63906C4AF5FFD0429565A743F6AD5BEDDAF29F@MBX04.ad.oak.ox.ac.uk> Message-ID: All, As Bob Oesterlin indicated, it is possible to define an expel script (see /usr/lpp/mmfs/samples/expelnode.sample) to control which of the two nodes to get expelled. The script can also be used to issue alerts, etc. The current policy used (before the script is invoked) when deciding which node to expel is the following: 1. quorum nodes over non-quorum nodes 2. local nodes over remote nodes 3. manager-capable nodes over non-manager-capable nodes 4. nodes managing more FSs over nodes managing fewer FSs 5. NSD server over non-NSD server Otherwise, expel whoever joined the cluster more recently. The statement below from Dr. Uwe Falke is also correct: addressing the network connectivity is the better long-term approach, but the callback script could be used to control which node to expel. Felipe ---- Felipe Knop knop at us.ibm.com GPFS Development and Security IBM Systems IBM Building 008 2455 South Rd, Poughkeepsie, NY 12601 (845) 433-9314 T/L 293-9314 From: "Uwe Falke" To: gpfsug main discussion list Date: 10/25/2016 08:32 AM Subject: Re: [gpfsug-discuss] Forcing which node gets expelled? Sent by: gpfsug-discuss-bounces at spectrumscale.org Usually, the cluster mgr, receiving a complaint from a node about another node being gone, checks its own connection to that other node. If that is positive it expells the requester, if not it follows the request and expells the other node. AFAIK, there are some more subtle algorithms in place if managers or quorum nodes are affected. Maybe that can be used to protect certain nodes from getting expelled by assigning some role in the cluster to them. I do however not know these exactly. That means: it is not easily controllable which one gets expelled. It is better to concentrate on fixing your connectivity issues, as GPFS will not feel comfortable in such a unreliable environment anyway. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Frank Hammer, Thorsten Moehring Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: Matt Thorpe To: gpfsug main discussion list Date: 10/25/2016 02:05 PM Subject: [gpfsug-discuss] Forcing which node gets expelled? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi, We are in the process of diagnosing a networking issue that is causing 2 of our 6 node GPFS cluster to expel each other (it appears they experience a temporary network connection outage and lose contact with each other). At present it's not consistent which gets expelled by the cluster manager, and I wondered if there was any way to always force a specific node to be expelled in this situation? Thanks and best regards, Matt -------- Matt Thorpe | BDLSS Systems Administrator Bodleian Libraries Osney One Building, Osney Mead, Oxford, OX2 0EW matt.thorpe at bodleian.ox.ac.uk | 01865 (2)80027 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From matt.thorpe at bodleian.ox.ac.uk Tue Oct 25 14:06:12 2016 From: matt.thorpe at bodleian.ox.ac.uk (Matt Thorpe) Date: Tue, 25 Oct 2016 13:06:12 +0000 Subject: [gpfsug-discuss] Forcing which node gets expelled? In-Reply-To: <4403E9DB-88B4-4281-91B5-E171D1284899@nuance.com> References: <63906C4AF5FFD0429565A743F6AD5BEDDAF29F@MBX04.ad.oak.ox.ac.uk> <4403E9DB-88B4-4281-91B5-E171D1284899@nuance.com> Message-ID: <63906C4AF5FFD0429565A743F6AD5BEDDAF314@MBX04.ad.oak.ox.ac.uk> Hi Bob, That is exactly what I was after, thanks very much! Should buy us a little time so we can resolve our networking issue. Thanks again Matt. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Oesterlin, Robert Sent: 25 October 2016 13:23 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Forcing which node gets expelled? If you look at /usr/lpp/mmfs/samples/expelnode.sample you can use this as a base and install this in /var/mmfs/etc on the cluster manager. You can control which of the two nodes get expelled. We use it here to send an alert when a node is expelled. There is also "mmexpelnode" which you can force a particular node to be expelled. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: > on behalf of Matt Thorpe > Reply-To: gpfsug main discussion list > Date: Tuesday, October 25, 2016 at 7:05 AM To: gpfsug main discussion list > Subject: [EXTERNAL] [gpfsug-discuss] Forcing which node gets expelled? Hi, We are in the process of diagnosing a networking issue that is causing 2 of our 6 node GPFS cluster to expel each other (it appears they experience a temporary network connection outage and lose contact with each other). At present it's not consistent which gets expelled by the cluster manager, and I wondered if there was any way to always force a specific node to be expelled in this situation? Thanks and best regards, Matt -------- Matt Thorpe | BDLSS Systems Administrator Bodleian Libraries Osney One Building, Osney Mead, Oxford, OX2 0EW matt.thorpe at bodleian.ox.ac.uk | 01865 (2)80027 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Q_f6z64tvENxA9ac7rqWFj2jNd5IrpWEXcynJzHFjz4&s=jqso6xVB-V_zgLba-xjlWwiw3fNRan9NspsVq4PY4nA&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Tue Oct 25 16:33:16 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Tue, 25 Oct 2016 15:33:16 +0000 Subject: [gpfsug-discuss] October meet the devs report Message-ID: The October meet the devs workshop on cloud was last week. Thanks to Dean Hildebrand and John Lewars for flying in from the US to support us, Ulf Troppens and Dan Kidger also from IBM. And finally thanks to OCF for buying the pizza (one day we'll manage to get the pizza co to deliver on time!). The event report is now up on the group website at: http://www.spectrumscale.org/meet-the-devs-cloud-workshop-birmingham-uk/ Simon From Mark.Bush at siriuscom.com Tue Oct 25 19:46:26 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Tue, 25 Oct 2016 18:46:26 +0000 Subject: [gpfsug-discuss] Virtualized Spectrum Scale Message-ID: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> Anyone running SpectrumScale on Virtual Machines (intel)? I?m curious how you manage disks? Do you use RDM?s? Does this even make sense to do? If you have a 2-3 node cluster how do you share the disks across? Do you have VM?s with their own VMDK?s (if not RDM) in each node or is there some way to share access to the same VMDK?s? What are the advantages doing this other than existing HW use? Seems to me for a lab environment or very small nonperformance focused implementation this may be a viable option. Thanks Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From luis.bolinches at fi.ibm.com Tue Oct 25 19:58:21 2016 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Tue, 25 Oct 2016 18:58:21 +0000 Subject: [gpfsug-discuss] Virtualized Spectrum Scale In-Reply-To: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> Message-ID: Hi You must use RDM. Otherwise is not supported. SCSI commands is the reason. Furthermore on some versions I managed to crash the ESXi as well. -- Cheers > On 25 Oct 2016, at 19.46, "Mark.Bush at siriuscom.com" wrote: > > Anyone running SpectrumScale on Virtual Machines (intel)? I?m curious how you manage disks? Do you use RDM?s? Does this even make sense to do? If you have a 2-3 node cluster how do you share the disks across? Do you have VM?s with their own VMDK?s (if not RDM) in each node or is there some way to share access to the same VMDK?s? What are the advantages doing this other than existing HW use? Seems to me for a lab environment or very small nonperformance focused implementation this may be a viable option. > > Thanks > > Mark > This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. > > Sirius Computer Solutions Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: From kevindjo at us.ibm.com Tue Oct 25 20:05:12 2016 From: kevindjo at us.ibm.com (Kevin D Johnson) Date: Tue, 25 Oct 2016 19:05:12 +0000 Subject: [gpfsug-discuss] Virtualized Spectrum Scale In-Reply-To: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> References: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> Message-ID: An HTML attachment was scrubbed... URL: From Mark.Bush at siriuscom.com Tue Oct 25 20:34:29 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Tue, 25 Oct 2016 19:34:29 +0000 Subject: [gpfsug-discuss] Virtualized Spectrum Scale In-Reply-To: References: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> Message-ID: <5482A551-873C-4F9F-89BF-0CF0FB926D92@siriuscom.com> This is interesting since the SS FAQ leads me to believe that I have some options here. Does GPFS nodes with no direct disk access still mean RDMs? [cid:image001.png at 01D22ECC.E20EC2F0] From: on behalf of Luis Bolinches Reply-To: gpfsug main discussion list Date: Tuesday, October 25, 2016 at 1:58 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Virtualized Spectrum Scale Hi You must use RDM. Otherwise is not supported. SCSI commands is the reason. Furthermore on some versions I managed to crash the ESXi as well. -- Cheers On 25 Oct 2016, at 19.46, "Mark.Bush at siriuscom.com" > wrote: Anyone running SpectrumScale on Virtual Machines (intel)? I?m curious how you manage disks? Do you use RDM?s? Does this even make sense to do? If you have a 2-3 node cluster how do you share the disks across? Do you have VM?s with their own VMDK?s (if not RDM) in each node or is there some way to share access to the same VMDK?s? What are the advantages doing this other than existing HW use? Seems to me for a lab environment or very small nonperformance focused implementation this may be a viable option. Thanks Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 169073 bytes Desc: image001.png URL: From gsanjay at us.ibm.com Tue Oct 25 20:48:58 2016 From: gsanjay at us.ibm.com (Sanjay Gandhi) Date: Tue, 25 Oct 2016 12:48:58 -0700 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 57, Issue 66 In-Reply-To: References: Message-ID: VMDK is not supported due to http://kb.vmware.com/kb/2032940 It can cause GPFS deadlock due to hung IO. Thanks, Sanjay Gandhi GPFS FVT IBM, Beaverton Phone/FAX : 503-578-4141 T/L 775-4141 gsanjay at us.ibm.com From: gpfsug-discuss-request at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Date: 10/25/2016 12:35 PM Subject: gpfsug-discuss Digest, Vol 57, Issue 66 Sent by: gpfsug-discuss-bounces at spectrumscale.org Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Virtualized Spectrum Scale (Mark.Bush at siriuscom.com) ---------------------------------------------------------------------- Message: 1 Date: Tue, 25 Oct 2016 19:34:29 +0000 From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Virtualized Spectrum Scale Message-ID: <5482A551-873C-4F9F-89BF-0CF0FB926D92 at siriuscom.com> Content-Type: text/plain; charset="utf-8" This is interesting since the SS FAQ leads me to believe that I have some options here. Does GPFS nodes with no direct disk access still mean RDMs? [cid:image001.png at 01D22ECC.E20EC2F0] From: on behalf of Luis Bolinches Reply-To: gpfsug main discussion list Date: Tuesday, October 25, 2016 at 1:58 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Virtualized Spectrum Scale Hi You must use RDM. Otherwise is not supported. SCSI commands is the reason. Furthermore on some versions I managed to crash the ESXi as well. -- Cheers On 25 Oct 2016, at 19.46, "Mark.Bush at siriuscom.com< mailto:Mark.Bush at siriuscom.com>" > wrote: Anyone running SpectrumScale on Virtual Machines (intel)? I?m curious how you manage disks? Do you use RDM?s? Does this even make sense to do? If you have a 2-3 node cluster how do you share the disks across? Do you have VM?s with their own VMDK?s (if not RDM) in each node or is there some way to share access to the same VMDK?s? What are the advantages doing this other than existing HW use? Seems to me for a lab environment or very small nonperformance focused implementation this may be a viable option. Thanks Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161025/fc9a8bf7/attachment.html > -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 169073 bytes Desc: image001.png URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161025/fc9a8bf7/attachment.png > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 66 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From Mark.Bush at siriuscom.com Tue Oct 25 20:50:51 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Tue, 25 Oct 2016 19:50:51 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 57, Issue 66 In-Reply-To: References: Message-ID: <0A2BC512-5EB7-4531-91DD-3C862A89C2E2@siriuscom.com> Ok. I re-read the FAQ and I think the option in the Table is a bit misleading. I think it means it?s supported without direct disk access and that means GPFS client not an NSD. Sound right? From: on behalf of Sanjay Gandhi Reply-To: gpfsug main discussion list Date: Tuesday, October 25, 2016 at 2:48 PM To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 57, Issue 66 VMDK is not supported due to http://kb.vmware.com/kb/2032940 It can cause GPFS deadlock due to hung IO. Thanks, Sanjay Gandhi GPFS FVT IBM, Beaverton Phone/FAX : 503-578-4141 T/L 775-4141 gsanjay at us.ibm.com [nactive hide details for gpfsug-discuss-request---10/25/2016 12:35:04 PM-]gpfsug-discuss-request---10/25/2016 12:35:04 PM---Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org From: gpfsug-discuss-request at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Date: 10/25/2016 12:35 PM Subject: gpfsug-discuss Digest, Vol 57, Issue 66 Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Virtualized Spectrum Scale (Mark.Bush at siriuscom.com) ---------------------------------------------------------------------- Message: 1 Date: Tue, 25 Oct 2016 19:34:29 +0000 From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Virtualized Spectrum Scale Message-ID: <5482A551-873C-4F9F-89BF-0CF0FB926D92 at siriuscom.com> Content-Type: text/plain; charset="utf-8" This is interesting since the SS FAQ leads me to believe that I have some options here. Does GPFS nodes with no direct disk access still mean RDMs? [cid:image001.png at 01D22ECC.E20EC2F0] From: on behalf of Luis Bolinches Reply-To: gpfsug main discussion list Date: Tuesday, October 25, 2016 at 1:58 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Virtualized Spectrum Scale Hi You must use RDM. Otherwise is not supported. SCSI commands is the reason. Furthermore on some versions I managed to crash the ESXi as well. -- Cheers On 25 Oct 2016, at 19.46, "Mark.Bush at siriuscom.com" > wrote: Anyone running SpectrumScale on Virtual Machines (intel)? I?m curious how you manage disks? Do you use RDM?s? Does this even make sense to do? If you have a 2-3 node cluster how do you share the disks across? Do you have VM?s with their own VMDK?s (if not RDM) in each node or is there some way to share access to the same VMDK?s? What are the advantages doing this other than existing HW use? Seems to me for a lab environment or very small nonperformance focused implementation this may be a viable option. Thanks Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 169073 bytes Desc: image001.png URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 66 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 106 bytes Desc: image001.gif URL: From laurence at qsplace.co.uk Tue Oct 25 20:53:55 2016 From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow) Date: Tue, 25 Oct 2016 20:53:55 +0100 Subject: [gpfsug-discuss] Virtualized Spectrum Scale In-Reply-To: References: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> Message-ID: Kevin, This is how I run test systems, I let iovirt devices to be attached to multiple kvm systems. It works well. -- Lauz On 25 October 2016 20:05:12 BST, Kevin D Johnson wrote: >Mark, > > > >You can run Spectrum Scale with virtual machines. As long as the >virtual disks present to the file system as devices, you should be good >to go (for example, "cat /proc/partitions" should show your virtual >disks as devices). I typically use scsi raw devices with virtual >machines and that seems to work well. KVM allows you to share the >disks as well between hosts and that's important to emulate more >production level uses. It is good for a lab or to try something >quickly without provisioning actual machines. We have sometimes used >SS with virtual machines in production but we typically recommend bare >metal if/when possible. > > > >Kevin D. Johnson, MBA, MAFM >Spectrum Computing, Senior Managing Consultant > >IBM Certified Deployment Professional - Spectrum Scale V4.1.1 >IBM Certified Deployment Professional - Cloud Object Storage V3.8 > >IBM Certified Solution Advisor - Spectrum Computing V1 > > > >720.349.6199 - kevindjo at us.ibm.com > > > > > > > >----- Original message ----- >From: "Mark.Bush at siriuscom.com" >Sent by: gpfsug-discuss-bounces at spectrumscale.org >To: "gpfsug-discuss at spectrumscale.org" > >Cc: >Subject: [gpfsug-discuss] Virtualized Spectrum Scale >Date: Tue, Oct 25, 2016 2:47 PM > > >Anyone running SpectrumScale on Virtual Machines (intel)? I?m curious >how you manage disks? Do you use RDM?s? Does this even make sense to >do? If you have a 2-3 node cluster how do you share the disks across? >Do you have VM?s with their own VMDK?s (if not RDM) in each node or is >there some way to share access to the same VMDK?s? What are the >advantages doing this other than existing HW use? Seems to me for a >lab environment or very small nonperformance focused implementation >this may be a viable option. > > > >Thanks > > > >Mark > >This message (including any attachments) is intended only for the use >of the individual or entity to which it is addressed and may contain >information that is non-public, proprietary, privileged, confidential, >and exempt from disclosure under applicable law. If you are not the >intended recipient, you are hereby notified that any use, >dissemination, distribution, or copying of this communication is >strictly prohibited. This message may be viewed by parties at Sirius >Computer Solutions other than those named in the message header. This >message does not contain an official representation of Sirius Computer >Solutions. If you have received this communication in error, notify >Sirius Computer Solutions immediately and (i) destroy this message if a >facsimile or (ii) delete this message immediately if this is an >electronic communication. Thank you. > >Sirius Computer Solutions > > > >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > >------------------------------------------------------------------------ > >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aspalazz at us.ibm.com Tue Oct 25 20:59:31 2016 From: aspalazz at us.ibm.com (Aaron S Palazzolo) Date: Tue, 25 Oct 2016 19:59:31 +0000 Subject: [gpfsug-discuss] Virtualized Spectrum Scale In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From kevindjo at us.ibm.com Tue Oct 25 21:02:42 2016 From: kevindjo at us.ibm.com (Kevin D Johnson) Date: Tue, 25 Oct 2016 20:02:42 +0000 Subject: [gpfsug-discuss] Virtualized Spectrum Scale In-Reply-To: References: , <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> Message-ID: An HTML attachment was scrubbed... URL: From Mark.Bush at siriuscom.com Tue Oct 25 21:36:51 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Tue, 25 Oct 2016 20:36:51 +0000 Subject: [gpfsug-discuss] Virtualized Spectrum Scale In-Reply-To: References: Message-ID: <82AD0334-B515-48FD-8C30-FC8F309AD00F@siriuscom.com> Excellent info Aaron. Thanks for this. I may reach out via PM to explore this more with you. From: on behalf of Aaron S Palazzolo Reply-To: gpfsug main discussion list Date: Tuesday, October 25, 2016 at 2:59 PM To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] Virtualized Spectrum Scale Hi Mark, Great questions on the VM side. Within our Spectrum Scale Test labs at IBM, we run within both VMware and KVM. It sounds as though your questions are more towards the VMware side so I'll give a few pointers. For more detailed info, feel free to grab me on the side or continue the discussion within the user group so that others can learn from this as well. #1) As Luis suggests, using vmdks will not give you full support of the scsi protocol. Specifically persistent reserve. EIO errors are also not passed correctly in some path down situations. - For certain test/dev environments, in which data protection and performance are not high priorities, then this may be fine. Some of our test environments do run like this. - Note that vmdks can be shared among virtual Spectrum Scale nodes using the multi-writer flag in VMware. To do this, you'll first need to create your vmdks using 'Thick Provision Eager Zeroed'. You'll then need to configure the multi-writer flag for scsi-sharing such as this, for each vmdk: scsi1:0.sharing = "multi-writer" scsi1:1.sharing = "multi-writer" scsi1:2.sharing = "multi-writer" You'll find this in the Advanced configuration parameters. Finally, you'll need to set all of these vmdks to a separate scsi adapter which is configured for either virtual or physical sharing. Downsides are lack of support, some degradation of performance, lack of failover support, and inability to use VMware snapshots of VMs with scsi sharing/multi-writer enabled. Upsides are ease of setup both with virtually and physically and ability to store all VMs on a single datastore that can itself be snapshotted if the underlying physical storage supports this. In our testlabs, we create 4node -> 50node virtual Spectrum Scale clusters, each node is a VM, some of these VMs have extra vmdks for NSDs and some do not. All VMs belonging to a cluster reside on a single datastore which ends up being a single XIV Volume. We then can snapshot this XIV volume and in essence, snapshot the entire Spectrum Scale cluster back and forth in time. I'm sure this is overly complicated for what you want to do, but it may get you thinking of use cases. #2) RDM will give you both performance and the piece of mind that you're using a fully supported config. Officially, I will always recommend RDM due to this. The downside to RDM is complexity in setup unless you have a fairly static config or unless you can automate the zoning. #3) No matter what type of VM infrastructure you use, make sure you investigate the memory/cpu requirements. - Check our FAQ, specifically 6.1 for tuning info regarding vm.min_free_kbytes: https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#gpfsclustersfaqAugust2016-gen4__lintunq - We have run some of our test clusters with as little as 4GB of memory but I would definitely recommend quite a bit more memory in each VM for production use. If you use additional functions other than core file system, pay attention to the memory requirements of these. #4) KVM is a viable alternative if needed..... Regards, Aaron Palazzolo IBM Spectrum Scale Deployment, Infrastructure, Virtualization 9042 S Rita Road, Tucson AZ 85744 Phone: 520-799-5161, T/L: 321-5161 E-mail: aspalazz at us.ibm.com ----- Original message ----- From: gpfsug-discuss-request at spectrumscale.org Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Cc: Subject: gpfsug-discuss Digest, Vol 57, Issue 65 Date: Tue, Oct 25, 2016 12:05 PM Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. October meet the devs report (Simon Thompson (Research Computing - IT Services)) 2. Virtualized Spectrum Scale (Mark.Bush at siriuscom.com) 3. Re: Virtualized Spectrum Scale (Luis Bolinches) 4. Re: Virtualized Spectrum Scale (Kevin D Johnson) ---------------------------------------------------------------------- Message: 1 Date: Tue, 25 Oct 2016 15:33:16 +0000 From: "Simon Thompson (Research Computing - IT Services)" To: "gpfsug-discuss at spectrumscale.org" Subject: [gpfsug-discuss] October meet the devs report Message-ID: Content-Type: text/plain; charset="us-ascii" The October meet the devs workshop on cloud was last week. Thanks to Dean Hildebrand and John Lewars for flying in from the US to support us, Ulf Troppens and Dan Kidger also from IBM. And finally thanks to OCF for buying the pizza (one day we'll manage to get the pizza co to deliver on time!). The event report is now up on the group website at: http://www.spectrumscale.org/meet-the-devs-cloud-workshop-birmingham-uk/ Simon ------------------------------ Message: 2 Date: Tue, 25 Oct 2016 18:46:26 +0000 From: "Mark.Bush at siriuscom.com" To: "gpfsug-discuss at spectrumscale.org" Subject: [gpfsug-discuss] Virtualized Spectrum Scale Message-ID: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD at siriuscom.com> Content-Type: text/plain; charset="utf-8" Anyone running SpectrumScale on Virtual Machines (intel)? I?m curious how you manage disks? Do you use RDM?s? Does this even make sense to do? If you have a 2-3 node cluster how do you share the disks across? Do you have VM?s with their own VMDK?s (if not RDM) in each node or is there some way to share access to the same VMDK?s? What are the advantages doing this other than existing HW use? Seems to me for a lab environment or very small nonperformance focused implementation this may be a viable option. Thanks Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 3 Date: Tue, 25 Oct 2016 18:58:21 +0000 From: "Luis Bolinches" To: "gpfsug main discussion list" Subject: Re: [gpfsug-discuss] Virtualized Spectrum Scale Message-ID: Content-Type: text/plain; charset="utf-8" Hi You must use RDM. Otherwise is not supported. SCSI commands is the reason. Furthermore on some versions I managed to crash the ESXi as well. -- Cheers > On 25 Oct 2016, at 19.46, "Mark.Bush at siriuscom.com" wrote: > > Anyone running SpectrumScale on Virtual Machines (intel)? I?m curious how you manage disks? Do you use RDM?s? Does this even make sense to do? If you have a 2-3 node cluster how do you share the disks across? Do you have VM?s with their own VMDK?s (if not RDM) in each node or is there some way to share access to the same VMDK?s? What are the advantages doing this other than existing HW use? Seems to me for a lab environment or very small nonperformance focused implementation this may be a viable option. > > Thanks > > Mark > This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. > > Sirius Computer Solutions Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 4 Date: Tue, 25 Oct 2016 19:05:12 +0000 From: "Kevin D Johnson" To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Virtualized Spectrum Scale Message-ID: Content-Type: text/plain; charset="us-ascii" An HTML attachment was scrubbed... URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 65 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: From leslie.james.elliott at gmail.com Tue Oct 25 23:01:25 2016 From: leslie.james.elliott at gmail.com (leslie elliott) Date: Wed, 26 Oct 2016 08:01:25 +1000 Subject: [gpfsug-discuss] unified file and object Message-ID: Hi We are in the process of trying to configure unified file and object storage in unified_mode and have a few problems We are running 4.2.1 and do not have any current issues the file access protocols setting up the authentication First issue we do have is binding the object data to our Active Directory, we seem to be hitting a road block due to the fact that the bind DN has spaces in it, if we enclose the DN in quotes it still fails, if we escape them with the appropriate RFC value we can get the mmuserauth to complete but the lookups from the local keystone fail for the authentication of the users The DN for the swift user and swift admin also have quotes in them, so just doing it on the command line is not enough to get the mmuserauth command to complete Second problem is OBJ:openstack-swift-object-sof is not running This seems to be due to the config file not having bind_ip and bind_port values, if these are added then the error turns to pipeline of other settings in the config file missing This particular issue occurs no matter what the auth type is set to be for object Hopefully this make some sense to someone Thanks leslie Leslie Elliott, Infrastructure Support Specialist, Faculty Infrastructure and Applications Support Information Technology Services, The University of Queensland -------------- next part -------------- An HTML attachment was scrubbed... URL: From Greg.Lehmann at csiro.au Wed Oct 26 01:51:16 2016 From: Greg.Lehmann at csiro.au (Greg.Lehmann at csiro.au) Date: Wed, 26 Oct 2016 00:51:16 +0000 Subject: [gpfsug-discuss] Forcing which node gets expelled? In-Reply-To: <63906C4AF5FFD0429565A743F6AD5BEDDAF314@MBX04.ad.oak.ox.ac.uk> References: <63906C4AF5FFD0429565A743F6AD5BEDDAF29F@MBX04.ad.oak.ox.ac.uk> <4403E9DB-88B4-4281-91B5-E171D1284899@nuance.com> <63906C4AF5FFD0429565A743F6AD5BEDDAF314@MBX04.ad.oak.ox.ac.uk> Message-ID: We hit something like this due to a bug in gskit. We all thought it was networking at first and it took me a fair bit of time to check all that. We have 7 nsd servers and around 400 clients running 4.2.0.4. We are just trying a workaround now that looks promising. The bug will be fixed at some point. Cheers, Greg From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Thorpe Sent: Tuesday, 25 October 2016 11:06 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Forcing which node gets expelled? Hi Bob, That is exactly what I was after, thanks very much! Should buy us a little time so we can resolve our networking issue. Thanks again Matt. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Oesterlin, Robert Sent: 25 October 2016 13:23 To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Forcing which node gets expelled? If you look at /usr/lpp/mmfs/samples/expelnode.sample you can use this as a base and install this in /var/mmfs/etc on the cluster manager. You can control which of the two nodes get expelled. We use it here to send an alert when a node is expelled. There is also "mmexpelnode" which you can force a particular node to be expelled. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: > on behalf of Matt Thorpe > Reply-To: gpfsug main discussion list > Date: Tuesday, October 25, 2016 at 7:05 AM To: gpfsug main discussion list > Subject: [EXTERNAL] [gpfsug-discuss] Forcing which node gets expelled? Hi, We are in the process of diagnosing a networking issue that is causing 2 of our 6 node GPFS cluster to expel each other (it appears they experience a temporary network connection outage and lose contact with each other). At present it's not consistent which gets expelled by the cluster manager, and I wondered if there was any way to always force a specific node to be expelled in this situation? Thanks and best regards, Matt -------- Matt Thorpe | BDLSS Systems Administrator Bodleian Libraries Osney One Building, Osney Mead, Oxford, OX2 0EW matt.thorpe at bodleian.ox.ac.uk | 01865 (2)80027 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Q_f6z64tvENxA9ac7rqWFj2jNd5IrpWEXcynJzHFjz4&s=jqso6xVB-V_zgLba-xjlWwiw3fNRan9NspsVq4PY4nA&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From Greg.Lehmann at csiro.au Wed Oct 26 02:04:35 2016 From: Greg.Lehmann at csiro.au (Greg.Lehmann at csiro.au) Date: Wed, 26 Oct 2016 01:04:35 +0000 Subject: [gpfsug-discuss] Virtualized Spectrum Scale In-Reply-To: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> References: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> Message-ID: <72fda8bf40634ad38e72689756a444cf@exch1-cdc.nexus.csiro.au> We use KVM running on a Debian host, with CentOS guests. Storage is zoned from our DDN Infiniband array to the host and then passed through to the guests. We would like to zone it directly to the guests SRIOV IB HCA, but srp seems to be a bit of dead code tree. We had to do a bit of work to get it working with Debian and haven?t as yet spent the time on getting it going with CentOS. We also run Spectrum Archive on the guest with tape drives and libraries zoned to the guest?s PCIe HBAs which are passed through from the host. We are working towards going production with this setup. Xen was a bit a failure for us so we switched to KVM. Cheers, Greg From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Mark.Bush at siriuscom.com Sent: Wednesday, 26 October 2016 4:46 AM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Virtualized Spectrum Scale Anyone running SpectrumScale on Virtual Machines (intel)? I?m curious how you manage disks? Do you use RDM?s? Does this even make sense to do? If you have a 2-3 node cluster how do you share the disks across? Do you have VM?s with their own VMDK?s (if not RDM) in each node or is there some way to share access to the same VMDK?s? What are the advantages doing this other than existing HW use? Seems to me for a lab environment or very small nonperformance focused implementation this may be a viable option. Thanks Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Wed Oct 26 03:48:38 2016 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 25 Oct 2016 22:48:38 -0400 Subject: [gpfsug-discuss] Virtualized Spectrum Scale In-Reply-To: <72fda8bf40634ad38e72689756a444cf@exch1-cdc.nexus.csiro.au> References: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> <72fda8bf40634ad38e72689756a444cf@exch1-cdc.nexus.csiro.au> Message-ID: <2e759489-a38f-58d2-4e20-b9c090c21b4a@nasa.gov> Hi Greg, I'm rather curious about your srp difficulties (not to get too off topic). Is it an issue with srp by itself or the interaction of srp with the SRIOV IB HCA? We've used SRP quite a bit both virtualized and not and have seen good results both in terms of stability and performance. -Aaron On 10/25/16 9:04 PM, Greg.Lehmann at csiro.au wrote: > We use KVM running on a Debian host, with CentOS guests. Storage is > zoned from our DDN Infiniband array to the host and then passed through > to the guests. We would like to zone it directly to the guests SRIOV IB > HCA, but srp seems to be a bit of dead code tree. We had to do a bit of > work to get it working with Debian and haven?t as yet spent the time on > getting it going with CentOS. > > > > We also run Spectrum Archive on the guest with tape drives and libraries > zoned to the guest?s PCIe HBAs which are passed through from the host. > We are working towards going production with this setup. > > > > Xen was a bit a failure for us so we switched to KVM. > > > > Cheers, > > > > Greg > > > > *From:*gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of > *Mark.Bush at siriuscom.com > *Sent:* Wednesday, 26 October 2016 4:46 AM > *To:* gpfsug-discuss at spectrumscale.org > *Subject:* [gpfsug-discuss] Virtualized Spectrum Scale > > > > Anyone running SpectrumScale on Virtual Machines (intel)? I?m curious > how you manage disks? Do you use RDM?s? Does this even make sense to > do? If you have a 2-3 node cluster how do you share the disks across? > Do you have VM?s with their own VMDK?s (if not RDM) in each node or is > there some way to share access to the same VMDK?s? What are the > advantages doing this other than existing HW use? Seems to me for a lab > environment or very small nonperformance focused implementation this may > be a viable option. > > > > Thanks > > > > Mark > > This message (including any attachments) is intended only for the use of > the individual or entity to which it is addressed and may contain > information that is non-public, proprietary, privileged, confidential, > and exempt from disclosure under applicable law. If you are not the > intended recipient, you are hereby notified that any use, dissemination, > distribution, or copying of this communication is strictly prohibited. > This message may be viewed by parties at Sirius Computer Solutions other > than those named in the message header. This message does not contain an > official representation of Sirius Computer Solutions. If you have > received this communication in error, notify Sirius Computer Solutions > immediately and (i) destroy this message if a facsimile or (ii) delete > this message immediately if this is an electronic communication. Thank you. > > *Sirius Computer Solutions * > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From Greg.Lehmann at csiro.au Wed Oct 26 04:07:19 2016 From: Greg.Lehmann at csiro.au (Greg.Lehmann at csiro.au) Date: Wed, 26 Oct 2016 03:07:19 +0000 Subject: [gpfsug-discuss] Virtualized Spectrum Scale In-Reply-To: <2e759489-a38f-58d2-4e20-b9c090c21b4a@nasa.gov> References: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> <72fda8bf40634ad38e72689756a444cf@exch1-cdc.nexus.csiro.au> <2e759489-a38f-58d2-4e20-b9c090c21b4a@nasa.gov> Message-ID: The srp work was done a few years ago now. We use the same srp code for both physical and virtual, so I am guessing it has nothing to do with the SRIOV side of things. Somebody else did the work, so I will try and get an answer for you. I agree performance and stability is good with physical and virtual. Cheers, Greg -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aaron Knister Sent: Wednesday, 26 October 2016 12:49 PM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Virtualized Spectrum Scale Hi Greg, I'm rather curious about your srp difficulties (not to get too off topic). Is it an issue with srp by itself or the interaction of srp with the SRIOV IB HCA? We've used SRP quite a bit both virtualized and not and have seen good results both in terms of stability and performance. -Aaron On 10/25/16 9:04 PM, Greg.Lehmann at csiro.au wrote: > We use KVM running on a Debian host, with CentOS guests. Storage is > zoned from our DDN Infiniband array to the host and then passed > through to the guests. We would like to zone it directly to the guests > SRIOV IB HCA, but srp seems to be a bit of dead code tree. We had to > do a bit of work to get it working with Debian and haven't as yet > spent the time on getting it going with CentOS. > > > > We also run Spectrum Archive on the guest with tape drives and > libraries zoned to the guest's PCIe HBAs which are passed through from the host. > We are working towards going production with this setup. > > > > Xen was a bit a failure for us so we switched to KVM. > > > > Cheers, > > > > Greg > > > > *From:*gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of > *Mark.Bush at siriuscom.com > *Sent:* Wednesday, 26 October 2016 4:46 AM > *To:* gpfsug-discuss at spectrumscale.org > *Subject:* [gpfsug-discuss] Virtualized Spectrum Scale > > > > Anyone running SpectrumScale on Virtual Machines (intel)? I'm curious > how you manage disks? Do you use RDM's? Does this even make sense to > do? If you have a 2-3 node cluster how do you share the disks across? > Do you have VM's with their own VMDK's (if not RDM) in each node or is > there some way to share access to the same VMDK's? What are the > advantages doing this other than existing HW use? Seems to me for a > lab environment or very small nonperformance focused implementation > this may be a viable option. > > > > Thanks > > > > Mark > > This message (including any attachments) is intended only for the use > of the individual or entity to which it is addressed and may contain > information that is non-public, proprietary, privileged, confidential, > and exempt from disclosure under applicable law. If you are not the > intended recipient, you are hereby notified that any use, > dissemination, distribution, or copying of this communication is strictly prohibited. > This message may be viewed by parties at Sirius Computer Solutions > other than those named in the message header. This message does not > contain an official representation of Sirius Computer Solutions. If > you have received this communication in error, notify Sirius Computer > Solutions immediately and (i) destroy this message if a facsimile or > (ii) delete this message immediately if this is an electronic communication. Thank you. > > *Sirius Computer Solutions * > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From janfrode at tanso.net Wed Oct 26 12:53:21 2016 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Wed, 26 Oct 2016 13:53:21 +0200 Subject: [gpfsug-discuss] filesystem thresholds in gui alerting Message-ID: Does anybody know if there are any way to define what thresholds are to be used for alterting in the GUI? F.ex. we have some filesystems that are very full, but won't be getting any more data added.. we'd like to turn off monitoring of these, are raise the threshold to allow them to be ~100% full. -jf -------------- next part -------------- An HTML attachment was scrubbed... URL: From stschmid at de.ibm.com Wed Oct 26 16:14:27 2016 From: stschmid at de.ibm.com (Stefan Schmidt) Date: Wed, 26 Oct 2016 17:14:27 +0200 Subject: [gpfsug-discuss] filesystem thresholds in gui alerting In-Reply-To: References: Message-ID: Hi Jan-Frode Myklebust, The current GUI-internal logic has a hard coded warning level of 80% and an hard-coded error level of 90%. With 4.2.2 the system health monitoring will have thresholds set and monitored via ZIMON and the system health component will provide events. With 4.2.2 those can be changed with an mm command. In general it is not a good idea to let a filesystem get loaded 100%. It could be, if this is done with the filesystem used by the system to store some configuration data, that the system can run into an error (usually this is the filesystem we call gpfs0). I suggest you get the 4.2.2 version which will GA quite soon. I have to apologize that I'm not allowed to talk about GA dates. If you want to get this feature earlier you may contact IBM and ask for participation on the beta program which starts very soon but the beta code is usually limited to non productive systems. As said , ask your IBM contact for the 4.2.2 GA update ( hopefully GA is soon) and you can get this feature you are looking for. Mit freundlichen Gr??en / Kind regards Stefan Schmidt Scrum Master IBM Spectrum Scale GUI / Senior IT Architect/PMP?-Dept. M069 / IBM Spectrum Scale Software Development IBM Systems Group IBM Deutschland Phone: +49-703 4274 1966 IBM Deutschland Am Weiher 24 E-Mail: stschmid at de.ibm.com 65421 Kelsterbach Germany IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: Jan-Frode Myklebust To: gpfsug main discussion list Date: 26.10.2016 13:53 Subject: [gpfsug-discuss] filesystem thresholds in gui alerting Sent by: gpfsug-discuss-bounces at spectrumscale.org Does anybody know if there are any way to define what thresholds are to be used for alterting in the GUI? F.ex. we have some filesystems that are very full, but won't be getting any more data added.. we'd like to turn off monitoring of these, are raise the threshold to allow them to be ~100% full. -jf_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulmer at ulmer.org Thu Oct 27 01:48:50 2016 From: ulmer at ulmer.org (Stephen Ulmer) Date: Wed, 26 Oct 2016 20:48:50 -0400 Subject: [gpfsug-discuss] filesystem thresholds in gui alerting In-Reply-To: References: Message-ID: <76513F28-0646-4282-A595-C69A9F32A945@ulmer.org> It seems prudent to note here that a file system that reports 100% full can have quite a bit of free space. Take for example a 10TB file system that is no longer written to ? no one wants to use 100GB just to make the alert color green. This gets silly very quickly at even ?medium? sizes... What would be really cool would be to decide how much reserved space you want, look at the velocity[1] with which the filesystem is filling, and then alert when the time until left until full is less than an operations shift (or something useful). Maybe that is just silly. Hey, Jan-Frode, can you mount the no-more-data file systems read only? Maybe not alerting on the full-ness of read only filesystems is easier and more sane? I mean, they can?t fill up any more. Liberty, -- Stephen > On Oct 26, 2016, at 11:14 AM, Stefan Schmidt wrote: > > Hi Jan-Frode Myklebust, > > The current GUI-internal logic has a hard coded warning level of 80% and an hard-coded error level of 90%. > > With 4.2.2 the system health monitoring will have thresholds set and monitored via ZIMON and the system health > component will provide events.With 4.2.2 those can be changed with an mm command. > > In general it is not a good idea to let a filesystem get loaded 100%. It could be, if this is done with the filesystem used by the system to store some configuration data, that the system can run into an error (usually this is the filesystem we call gpfs0). > > I suggest you get the 4.2.2 version which will GA quite soon. I have to apologize that I'm not allowed to talk about GA dates. If you want to get this feature earlier you may contact IBM and ask for participation on the beta program which starts very soon but the beta code is usually limited to non productive systems. > > As said , ask your IBM contact for the 4.2.2 GA update ( hopefully GA is soon) and you can get this feature you are looking for. > Mit freundlichen Gr??en / Kind regards > > > Stefan Schmidt > > > Scrum Master IBM Spectrum Scale GUI / Senior IT Architect/PMP?-Dept. M069 / IBM Spectrum Scale Software Development > > IBM Systems Group > > IBM Deutschland > > Phone: +49-703 4274 1966 IBM Deutschland > Am Weiher 24 > E-Mail: stschmid at de.ibm.com 65421 Kelsterbach > Germany > IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Martina Koederitz > Gesch?ftsf?hrung: Dirk Wittkopp > Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 > > > > > > > > From: Jan-Frode Myklebust > To: gpfsug main discussion list > Date: 26.10.2016 13:53 > Subject: [gpfsug-discuss] filesystem thresholds in gui alerting > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > > > Does anybody know if there are any way to define what thresholds are to be used for alterting in the GUI? F.ex. we have some filesystems that are very full, but won't be getting any more data added.. we'd like to turn off monitoring of these, are raise the threshold to allow them to be ~100% full. > > > > -jf_______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From usa-principal at gpfsug.org Thu Oct 27 02:06:06 2016 From: usa-principal at gpfsug.org (Spectrum Scale Users Group - USA Principal Kristy Kallback-Rose) Date: Wed, 26 Oct 2016 21:06:06 -0400 Subject: [gpfsug-discuss] [UPDATE] SC16 Draft Agenda [Save the date: Sunday, November 13th] In-Reply-To: References: Message-ID: <10BF7862-5EB0-412E-9B1D-9531D89B87C2@gpfsug.org> All, Here is where we?re at with the agenda. Still finalizing a few things, but wanted to keep you all in the loop. Registration Web Site [Register for the SC16 UG event here]: https://www-01.ibm.com/events/wwe/grp/grp305.nsf/Agenda.xsp?openform&seminar=357M7UES&locale=en_US&S_TACT=000001CP&S_OFF_CD=10001235 IBM summary page: http://www-03.ibm.com/systems/technicalcomputing/supercomputing/index.html Location: Salt Lake City Downtown Marriott [Room Number TBD] 75 SW Temple Street, Salt Lake City UT 84101 Draft Agenda: 12:30 12:40 Welcome 12:40 12:50 Overview 12:50 13:35 The latest in IBM Spectrum Scale and ESS 13:55 14:20 NASA Goddard: Big Data in HPC 14:20 14:45 Nuance: Tiering to Object Storage 14:45 15:15 - break - 15:15 15:30 Sponsor Talk: TOPIC TBD (NetApp) 13:35 13:55 Virginia Tech ARC: SANDisk IF150 (IBM DeepFlash 150) 15:30 15:50 NASA Goddard: Monitoring with Grafana and InfluxDB 15:50 16:35 Spectrum Scale Enhancements for CORAL 16:35 17:20 News from IBM Research 17:20 17:30 Closing [Reception to follow closing, details TBD] Questions? Comments? Let us know. Hope to see many of you next month. Final agenda will be made available before the event. Cheers, Kristy & Bob Kristy Kallback-Rose Manager, Research Storage Indiana University > On Oct 10, 2016, at 6:59 PM, GPFS UG USA Principal > wrote: > > Hello all, > > There have been some questions about the Spectrum Scale Users Group event for SC16 and we thought it best to publish our draft agenda and continue to fill it in as details get settled. That way you can make your travel plans and schedules. > > The date and rough times are set, we may adjust the time range a bit depending upon the number of user talks that people would like to give. So note, yes! we are asking for user talks. Please consider doing this. We always receive positive feedback about talks given by users that describe real world experiences. If any of the user talk slots below are of interest to you, please let us know as soon as you can. We may turn at least one user talk into a panel if we can get enough participants. > > As always, your feedback is welcome. This is a users group after all. > > So, here?s what we have planned so far: > > Date: Sunday November 13th > Venue: TBD, but near the convention center in downtown SLC > Time: ~12:30p - ~17:30p > > Agenda: > 12:30 - 12:50 Welcome / Overview > 12:50 - 13:50 The latest in IBM Spectrum Scale and ESS > 13:50 - 14:20 User Talk: Big Data in HPC > 14:20 - 14:50 User Talk: Tiering to Object Storage > 14:50 - 15:20 - break - > 15:20 - 15:50 User Talk: TBD ? volunteers! please! > 15:50 - 16:35 Spectrum Scale Enhancements for CORAL > 16:35 - 17:20 News from IBM Research > 17:20 - 17:30 Closing > > Best, > > Kristy & Bob > > Kristy Kallback-Rose > Manager, Research Storage > Indiana University > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Ola.Pontusson at kb.se Thu Oct 27 07:49:24 2016 From: Ola.Pontusson at kb.se (Ola Pontusson) Date: Thu, 27 Oct 2016 06:49:24 +0000 Subject: [gpfsug-discuss] filesystem thresholds in gui alerting In-Reply-To: <76513F28-0646-4282-A595-C69A9F32A945@ulmer.org> References: <76513F28-0646-4282-A595-C69A9F32A945@ulmer.org> Message-ID: Hi I am really looking forward to the 4.2.2 where I hopefully can adjust the thresholds of alarm levels. We just as Jan-Frode have filesystems that is not growing so much and waste a large amount of disk just to keep it green is never going to happend. Below is an example of one of our filesystems. /dev/smdb04 942T 900T 42T 96% /gpfs/smdb04 /Ola Fr?n: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] F?r Stephen Ulmer Skickat: den 27 oktober 2016 02:49 Till: gpfsug main discussion list ?mne: Re: [gpfsug-discuss] filesystem thresholds in gui alerting It seems prudent to note here that a file system that reports 100% full can have quite a bit of free space. Take for example a 10TB file system that is no longer written to ? no one wants to use 100GB just to make the alert color green. This gets silly very quickly at even ?medium? sizes... What would be really cool would be to decide how much reserved space you want, look at the velocity[1] with which the filesystem is filling, and then alert when the time until left until full is less than an operations shift (or something useful). Maybe that is just silly. Hey, Jan-Frode, can you mount the no-more-data file systems read only? Maybe not alerting on the full-ness of read only filesystems is easier and more sane? I mean, they can?t fill up any more. Liberty, -- Stephen On Oct 26, 2016, at 11:14 AM, Stefan Schmidt > wrote: Hi Jan-Frode Myklebust, The current GUI-internal logic has a hard coded warning level of 80% and an hard-coded error level of 90%. With 4.2.2 the system health monitoring will have thresholds set and monitored via ZIMON and the system health component will provide events.With 4.2.2 those can be changed with an mm command. In general it is not a good idea to let a filesystem get loaded 100%. It could be, if this is done with the filesystem used by the system to store some configuration data, that the system can run into an error (usually this is the filesystem we call gpfs0). I suggest you get the 4.2.2 version which will GA quite soon. I have to apologize that I'm not allowed to talk about GA dates. If you want to get this feature earlier you may contact IBM and ask for participation on the beta program which starts very soon but the beta code is usually limited to non productive systems. As said , ask your IBM contact for the 4.2.2 GA update ( hopefully GA is soon) and you can get this feature you are looking for. Mit freundlichen Gr??en / Kind regards Stefan Schmidt Scrum Master IBM Spectrum Scale GUI / Senior IT Architect/PMP?-Dept. M069 / IBM Spectrum Scale Software Development IBM Systems Group IBM Deutschland ________________________________ Phone: +49-703 4274 1966 IBM Deutschland Am Weiher 24 E-Mail: stschmid at de.ibm.com 65421 Kelsterbach Germany ________________________________ IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: Jan-Frode Myklebust > To: gpfsug main discussion list > Date: 26.10.2016 13:53 Subject: [gpfsug-discuss] filesystem thresholds in gui alerting Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Does anybody know if there are any way to define what thresholds are to be used for alterting in the GUI? F.ex. we have some filesystems that are very full, but won't be getting any more data added.. we'd like to turn off monitoring of these, are raise the threshold to allow them to be ~100% full. -jf_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From stijn.deweirdt at ugent.be Thu Oct 27 18:27:44 2016 From: stijn.deweirdt at ugent.be (Stijn De Weirdt) Date: Thu, 27 Oct 2016 19:27:44 +0200 Subject: [gpfsug-discuss] workaround gpfs 4.2.1-0 rpm issue Message-ID: <86d24598-7af8-38f0-fe80-998294c7bef9@ugent.be> hi all, gpfs.base 4.2.1-0 rpm has following postuninstall snippet it will disable the gpfs unit always (when previously enabled), whether this is a removal or an upgrade. this however prevents an update to 4.2.1-1, as it will remove the unit that is added during install (post has 'systemctl reenable /usr/lpp/mmfs/lib/systemd/gpfs.service') so after the upgrade, we are left with nodes that have no gpfs service unit (and thus no gpfs). it would have been better if the rpm symlinked teh service to the /usr/lib/systemd/... units, and enabled/disabled it. i'll probably rpmrebuild the 4.2.1-1 rpms to make a more permanent unit in a proper system location. other tips are welcome. stijn > postuninstall scriptlet (using /bin/sh): > if test -n "$DEBUG" || test -n "$DEBUGpostun"; then > set -x > fi > packageCnt=$1 > debian_release="/etc/debian_version" > > # set the system utilities if they are in different path on different systems > if [ -f "$debian_release" ] > then > AWK=/usr/bin/awk > else > AWK=/bin/awk > fi > > if /usr/bin/systemctl -q is-enabled gpfs.service 2>/dev/null > then > /usr/bin/systemctl -q disable gpfs.service > fi > From douglasof at us.ibm.com Fri Oct 28 14:37:53 2016 From: douglasof at us.ibm.com (Douglas O'flaherty) Date: Fri, 28 Oct 2016 09:37:53 -0400 Subject: [gpfsug-discuss] SC16 Registration Message-ID: Greetings: If you haven't yet registered, planned a 1:1 meeting, or interested in a specific topic ... Here you go https://www-01.ibm.com/events/wwe/grp/grp305.nsf/Registration.xsp?openform&seminar=357M7UES&locale=en_US&auth=anonymous Sign up so we can plan the cocktail hour on Sunday evening -- Thanks to NetApp for sponsoring again. doug IBM Spectrum Storage PMM -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mark.Bush at siriuscom.com Fri Oct 28 14:52:47 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Fri, 28 Oct 2016 13:52:47 +0000 Subject: [gpfsug-discuss] GUI and pmsensors/pmcollector Message-ID: <9566FC70-85D3-4D9B-B28B-58C4535B8972@siriuscom.com> I am able to do everything I need just fine when I have a single node cluster and both pmsensors and pmcollector are installed all on the same node. When I try and get pmsensors on my NSD nodes and a separate pmcollector elsewhere I never can get anything to show up in the graphs. I?ve configured the pmsensors with mmperfmon config generate ?collectors=(my node) and I still get nothing. The logs show that I get connections but the GUI never show and I?m unable to get mmperfmon query to work either as it fails saying it can?t find the collector. Anyone had this much trouble just getting this to work? Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From stschmid at de.ibm.com Fri Oct 28 14:59:02 2016 From: stschmid at de.ibm.com (Stefan Schmidt) Date: Fri, 28 Oct 2016 15:59:02 +0200 Subject: [gpfsug-discuss] GUI and pmsensors/pmcollector In-Reply-To: <9566FC70-85D3-4D9B-B28B-58C4535B8972@siriuscom.com> References: <9566FC70-85D3-4D9B-B28B-58C4535B8972@siriuscom.com> Message-ID: Hi, the PMCollector must be installed on the same node as the GUI. The collectors can be installed on any node you want to monitor. Hope that helps. Mit freundlichen Gr??en / Kind regards Stefan Schmidt Scrum Master IBM Spectrum Scale GUI / Senior IT Architect/PMP?-Dept. M069 / IBM Spectrum Scale Software Development IBM Systems Group IBM Deutschland Phone: +49-703 4274 1966 IBM Deutschland Am Weiher 24 E-Mail: stschmid at de.ibm.com 65421 Kelsterbach Germany IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Mark.Bush at siriuscom.com" To: "gpfsug-discuss at spectrumscale.org" Date: 28.10.2016 15:53 Subject: [gpfsug-discuss] GUI and pmsensors/pmcollector Sent by: gpfsug-discuss-bounces at spectrumscale.org I am able to do everything I need just fine when I have a single node cluster and both pmsensors and pmcollector are installed all on the same node. When I try and get pmsensors on my NSD nodes and a separate pmcollector elsewhere I never can get anything to show up in the graphs. I?ve configured the pmsensors with mmperfmon config generate ?collectors=(my node) and I still get nothing. The logs show that I get connections but the GUI never show and I?m unable to get mmperfmon query to work either as it fails saying it can?t find the collector. Anyone had this much trouble just getting this to work? Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mark.Bush at siriuscom.com Fri Oct 28 15:12:25 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Fri, 28 Oct 2016 14:12:25 +0000 Subject: [gpfsug-discuss] GUI and pmsensors/pmcollector In-Reply-To: References: <9566FC70-85D3-4D9B-B28B-58C4535B8972@siriuscom.com> Message-ID: <66CEE4CD-BD01-4F50-839A-529062DC38D3@siriuscom.com> Perhaps I needed more description I have a 3 node cluster 2 NDS?s (with tiebreaker) 1 gui node The NDS?s all have pmsensors installed The gui node had pmcollector installed I?ve modified the /opt/IBM/zimon/ZIMSensors.cfg file on the NSD?s to point to my gui node Systemctl start pmsensors on NSD?s Systemclt start pmcollector on gui Systemctl start gpfsgui on gui The log even shows connection from the two NSD?s but for some reason the GUI always has a red x in the performance graphs and claims it can?t connect to the collector (which is running on the same node). Not sure what I?m missing here. It all works fine when it?s all on one node. Mark From: on behalf of Stefan Schmidt Reply-To: gpfsug main discussion list Date: Friday, October 28, 2016 at 8:59 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GUI and pmsensors/pmcollector Hi, the PMCollector must be installed on the same node as the GUI. The collectors can be installed on any node you want to monitor. Hope that helps. Mit freundlichen Gr??en / Kind regards Stefan Schmidt Scrum Master IBM Spectrum Scale GUI / Senior IT Architect/PMP?-Dept. M069 / IBM Spectrum Scale Software Development IBM Systems Group IBM Deutschland ________________________________ Phone: +49-703 4274 1966 IBM Deutschland Am Weiher 24 E-Mail: stschmid at de.ibm.com 65421 Kelsterbach Germany ________________________________ IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Mark.Bush at siriuscom.com" To: "gpfsug-discuss at spectrumscale.org" Date: 28.10.2016 15:53 Subject: [gpfsug-discuss] GUI and pmsensors/pmcollector Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ I am able to do everything I need just fine when I have a single node cluster and both pmsensors and pmcollector are installed all on the same node. When I try and get pmsensors on my NSD nodes and a separate pmcollector elsewhere I never can get anything to show up in the graphs. I?ve configured the pmsensors with mmperfmon config generate ?collectors=(my node) and I still get nothing. The logs show that I get connections but the GUI never show and I?m unable to get mmperfmon query to work either as it fails saying it can?t find the collector. Anyone had this much trouble just getting this to work? Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From akers at vt.edu Fri Oct 28 15:22:49 2016 From: akers at vt.edu (Joshua Akers) Date: Fri, 28 Oct 2016 14:22:49 +0000 Subject: [gpfsug-discuss] GPFS Log Levels Message-ID: Hi all, I am trying to find more information on GPFS log levels. Here is what I have so far: [D] - Detail info [I] - Info [N] - Notice [W] - Warning [E] - Error [X] - Critical Error [A] - Deadlock related? Any corrections or additional information would be greatly appreciated. Thanks, Josh -- *Joshua D. Akers* *HPC Systems Specialist* NI&S Systems Support (MC0214) 1700 Pratt Drive Blacksburg, VA 24061 540-231-9506 -------------- next part -------------- An HTML attachment was scrubbed... URL: From billowen at us.ibm.com Fri Oct 28 15:37:27 2016 From: billowen at us.ibm.com (Bill Owen) Date: Fri, 28 Oct 2016 07:37:27 -0700 Subject: [gpfsug-discuss] unified file and object In-Reply-To: References: Message-ID: Hi Leslie, For the issues you list below: 1. AD configuration - there is a known issue with quoted names passed to mmuserauth - we are investigating how to correct this. 2. Can you provide more details on how you configured file access? The normal procedure is to use "mmobj file-access enable", and this will set up the required settings in the config file. Can you send us: - the steps used to configure file access - the resulting /etc/swift/object-server-sof.conf - log files from /var/log/swift or output of "systemctl status openstack-swift-object-sof" We can schedule a short call to help debug if needed. Here is a more detailed description of enabling file access to object data: Enable unified file and object access See: http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.1/com.ibm.spectrum.scale.v4r21.doc/bl1adm_enablefileaccess.htm 1. enable & configure: # mmobj file-access enable # mmobj config list --ccrfile spectrum-scale-object.conf --section capabilities --property file-access-enabled file-access-enabled = true Different user id mapping approaches are supporting. The simplest is id_mgmt = local_mode. This example leaves that default in place. Verify id_mgmt mode: # mmobj config list --ccrfile object-server-sof.conf --section DEFAULT --property id_mgmt id_mgmt = local_mode At this point, you can create a storage policy that will be used for file & object access containers. We'll create a new fileset for this step (in 4.2.2 we will support objectizing existing filesets). In this example, we create the fileset with 1M inodes initially allocated: # mmobj policy create fileaccess --enable-file-access -i 1000000 [I] Getting latest configuration from ccr [I] Creating fileset fs2-1m-03:obj_fileaccess [I] Creating new unique index and building the object rings [I] Updating the configuration [I] Uploading the changed configuration The new storage policy is listed: # mmobj policy list Index Name Default Deprecated Fileset Functions -------------------------------------------------------------------------------- 0 policy-0 yes Object_Fileset default 87811608170 fileaccess obj_fileaccess file-and-object-access The new fileset called obj_fileaccess is shown below: # mmlsfileset fs2-1m-03 obj_fileaccess -L Filesets in file system 'fs2-1m-03': Name Id RootInode ParentId Created InodeSpace MaxInodes AllocInodes Comment obj_fileaccess 6 134217731 0 Wed Aug 17 10:08:51 2016 4 1000192 1000192 2. Create one or more containers that are file-access enabled: Next, you can create one or more containers that use this storage policy (and have file access enabled): # source openrc # swift post fileaccess_container --header "x-storage-policy: fileaccess" # swift stat fileaccess_container Account: AUTH_acad33df8cdf402ebab6bfb87b1674af Container: fileaccess_container Objects: 0 Bytes: 0 Read ACL: Write ACL: Sync To: Sync Key: Accept-Ranges: bytes X-Storage-Policy: fileaccess X-Timestamp: 1471456287.21609 X-Trans-Id: tx0f305353a20a4ac1ab927-0057b4a428 Content-Type: text/plain; charset=utf-8 Objects can be added to the container from the object or file interface, the file path is structured like this: ///// In our example it would be: /ibm/fs2-1m-03/obj_fileaccess/s87811608170z1device1/AUTH_acad33df8cdf402ebab6bfb87b1674af/fileaccess_container/ For convenience, create a soft link: ln -s /ibm/fs2-1m-03/obj_fileaccess/s87811608170z1device1/AUTH_acad33df8cdf402ebab6bfb87b1674af/fileaccess_container/ /ibm/fs2-1m-03/unified_file_object 3. Verify access from both file & object interface: Upload a file from object interface, it will be readable from file interface, owned by swift user. For example: # date > test1.obj # swift upload fileaccess_container test1.obj test1.obj # ls -l /ibm/fs2-1m-03/unified_file_object/ total 0 -rwxr-xr-x 1 swift swift 29 Aug 17 11:03 test1.obj Similarly, create an example object (or set of objects) from file interface: # date > /ibm/fs2-1m-03/unified_file_object/test2.obj The object must be readable by swift user - either by setting file ownership or using file acls: # chown swift:swift /ibm/fs2-1m-03/unified_file_object/test2.obj # ls -l /ibm/fs2-1m-03/unified_file_object/ -rwxr-xr-x 1 swift swift 29 Aug 17 11:03 test1.obj -rw-r--r-- 1 swift swift 29 Aug 17 11:05 test2.obj New files will be "objectized" (made visible to the object interface) by a periodic process. The default period for this is 30 min, but can be changed using mmob config change command. This process can be resource intensive, so don't see to run too often if there are many containers & files. # mmobj config list --ccrfile spectrum-scale-objectizer.conf --section DEFAULT --property objectization_interval objectization_interval = 1800 After waiting the objectizer interval, the new object shows up from the object interface, and can be accessed like any other object: # swift list fileaccess_container test1.obj test2.obj # swift download fileaccess_container test2.obj test2.obj [auth 0.229s, headers 0.862s, total 0.862s, 0.000 MB/s] Verify the contents match # cat test2.obj Wed Aug 17 11:05:00 PDT 2016 # cat /ibm/fs2-1m-03/unified_file_object/test2.obj Wed Aug 17 11:05:00 PDT 2016 And update from file interface: # echo "bill was here" >> /ibm/fs2-1m-03/unified_file_object/test2.obj Redownload from object interface and confirm: # swift download fileaccess_container test2.obj test2.obj [auth 0.230s, headers 0.729s, total 0.729s, 0.000 MB/s] [ # cat test2.obj Wed Aug 17 11:05:00 PDT 2016 bill was here Use Case Example: One usecase is adding/expanding an archive file in file interface, and results can be accessed from file interface. For example, create a set of test files: # mkdir test [root at client22 ~]# for i in {1..10}; do date > test/smallobj$i ; done [root at client22 ~]# ll test total 40 -rw-r--r-- 1 root root 29 Aug 17 11:31 smallobj1 -rw-r--r-- 1 root root 29 Aug 17 11:31 smallobj10 -rw-r--r-- 1 root root 29 Aug 17 11:31 smallobj2 -rw-r--r-- 1 root root 29 Aug 17 11:31 smallobj3 -rw-r--r-- 1 root root 29 Aug 17 11:31 smallobj4 -rw-r--r-- 1 root root 29 Aug 17 11:31 smallobj5 -rw-r--r-- 1 root root 29 Aug 17 11:31 smallobj6 -rw-r--r-- 1 root root 29 Aug 17 11:31 smallobj7 -rw-r--r-- 1 root root 29 Aug 17 11:31 smallobj8 -rw-r--r-- 1 root root 29 Aug 17 11:31 smallobj9 # tar -cf databag.tar test Expand the file into file interface and set ownership to swift: # cd /ibm/fs2-1m-03/unified_file_object/ # tar -xf ~/databag.tar # chown -R swift:swift test After objectizer runs, all files are available from object interface: # swift list fileaccess_container test/smallobj1 test/smallobj10 test/smallobj2 test/smallobj3 test/smallobj4 test/smallobj5 test/smallobj6 test/smallobj7 test/smallobj8 test/smallobj9 test1.obj test2.obj Regards, Bill Owen billowen at us.ibm.com Spectrum Scale Object Storage 520-799-4829 From: leslie elliott To: gpfsug-discuss at spectrumscale.org Date: 10/25/2016 03:01 PM Subject: [gpfsug-discuss] unified file and object Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi We are in the process of trying to configure unified file and object storage in unified_mode and have a few problems We are running 4.2.1 and do not have any current issues the file access protocols setting up the authentication First issue we do have is binding the object data to our Active Directory, we seem to be hitting a road block due to the fact that the bind DN has spaces in it, if we enclose the DN in quotes it still fails, if we escape them with the appropriate RFC ?value we can get the mmuserauth to complete but the lookups from the local keystone fail for the authentication of the users The DN for the swift user and swift admin also have quotes in them, so just doing it on the command line is not enough to get the mmuserauth command to complete Second problem is OBJ:openstack-swift-object-sof ? ? ? ? ? is not running This seems to be due to the config file not having ?bind_ip and bind_port values, if these are added then the error turns to pipeline of other settings in the config file missing This particular issue occurs no matter what the auth type is set to be for object Hopefully this make some sense to someone Thanks leslie Leslie Elliott, Infrastructure Support Specialist, ?Faculty Infrastructure and Applications Support Information Technology Services, The University of Queensland _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From taylorm at us.ibm.com Fri Oct 28 15:56:30 2016 From: taylorm at us.ibm.com (Michael L Taylor) Date: Fri, 28 Oct 2016 07:56:30 -0700 Subject: [gpfsug-discuss] GUI and pmsensors/pmcollector In-Reply-To: References: Message-ID: Have a quick read of this link and see if it helps: https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.1/com.ibm.spectrum.scale.v4r21.doc/bl1ins_manualinstallofgui.htm From: gpfsug-discuss-request at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Date: 10/28/2016 07:23 AM Subject: gpfsug-discuss Digest, Vol 57, Issue 76 Sent by: gpfsug-discuss-bounces at spectrumscale.org Message: 1 Date: Fri, 28 Oct 2016 14:12:25 +0000 From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GUI and pmsensors/pmcollector Message-ID: <66CEE4CD-BD01-4F50-839A-529062DC38D3 at siriuscom.com> Content-Type: text/plain; charset="utf-8" Perhaps I needed more description I have a 3 node cluster 2 NDS?s (with tiebreaker) 1 gui node The NDS?s all have pmsensors installed The gui node had pmcollector installed I?ve modified the /opt/IBM/zimon/ZIMSensors.cfg file on the NSD?s to point to my gui node Systemctl start pmsensors on NSD?s Systemclt start pmcollector on gui Systemctl start gpfsgui on gui The log even shows connection from the two NSD?s but for some reason the GUI always has a red x in the performance graphs and claims it can?t connect to the collector (which is running on the same node). Not sure what I?m missing here. It all works fine when it?s all on one node. Mark From: on behalf of Stefan Schmidt Reply-To: gpfsug main discussion list Date: Friday, October 28, 2016 at 8:59 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GUI and pmsensors/pmcollector Hi, the PMCollector must be installed on the same node as the GUI. The collectors can be installed on any node you want to monitor. Hope that helps. Mit freundlichen Gr??en / Kind regards Stefan Schmidt Scrum Master IBM Spectrum Scale GUI / Senior IT Architect/PMP?-Dept. M069 / IBM Spectrum Scale Software Development IBM Systems Group IBM Deutschland ________________________________ Phone: +49-703 4274 1966 IBM Deutschland Am Weiher 24 E-Mail: stschmid at de.ibm.com 65421 Kelsterbach Germany ________________________________ IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Mark.Bush at siriuscom.com" To: "gpfsug-discuss at spectrumscale.org" Date: 28.10.2016 15:53 Subject: [gpfsug-discuss] GUI and pmsensors/pmcollector Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ I am able to do everything I need just fine when I have a single node cluster and both pmsensors and pmcollector are installed all on the same node. When I try and get pmsensors on my NSD nodes and a separate pmcollector elsewhere I never can get anything to show up in the graphs. I?ve configured the pmsensors with mmperfmon config generate ?collectors= (my node) and I still get nothing. The logs show that I get connections but the GUI never show and I?m unable to get mmperfmon query to work either as it fails saying it can?t find the collector. Anyone had this much trouble just getting this to work? Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161028/b4552796/attachment-0001.html > ------------------------------ Message: 2 Date: Fri, 28 Oct 2016 14:22:49 +0000 From: Joshua Akers To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] GPFS Log Levels Message-ID: Content-Type: text/plain; charset="utf-8" Hi all, I am trying to find more information on GPFS log levels. Here is what I have so far: [D] - Detail info [I] - Info [N] - Notice [W] - Warning [E] - Error [X] - Critical Error [A] - Deadlock related? Any corrections or additional information would be greatly appreciated. Thanks, Josh -- *Joshua D. Akers* *HPC Systems Specialist* NI&S Systems Support (MC0214) 1700 Pratt Drive Blacksburg, VA 24061 540-231-9506 -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161028/7ebac661/attachment.html > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 76 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: From rohwedder at de.ibm.com Fri Oct 28 16:08:17 2016 From: rohwedder at de.ibm.com (Markus Rohwedder) Date: Fri, 28 Oct 2016 17:08:17 +0200 Subject: [gpfsug-discuss] GUI and pmsensors/pmcollector In-Reply-To: References: Message-ID: Hi, Some more questions and things to look for: 1. Are there several collectors running, potentially at different code levels? All collectors should run at the same code level. Simplest case, there is only one collector. Sensors and collectors can have different code levels. 2. Is the sensor config updated manually or per mmperfmon config update? Manual update might be overwritten if the system thinks that the config should be distr?buted automatically. See Knowledge center and mmperfmon CLI command for info. 3. How does the sensor config look like (output of mmperfmon config show)? 4. Are only NSD charts showing that there is no data, or all charts? Please note; In a SAN environment (Every node sees every NSD), there is no NSD server side data reported. GPFS is smart and knows to use the local device and circumvents the NSD code stack. However, we dont get notified on traffic to the disks. 5. Which code level? Thanks, Mit freundlichen Gr??en / Kind regards Dr. Markus Rohwedder Spectrum Scale GUI Development Phone: +49 7034 6430190 IBM Deutschland E-Mail: rohwedder at de.ibm.com Am Weiher 24 65451 Kelsterbach Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Martina K?deritz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: gpfsug-discuss-request at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Date: 28.10.2016 16:23 Subject: gpfsug-discuss Digest, Vol 57, Issue 76 Sent by: gpfsug-discuss-bounces at spectrumscale.org Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: GUI and pmsensors/pmcollector (Mark.Bush at siriuscom.com) 2. GPFS Log Levels (Joshua Akers) ---------------------------------------------------------------------- Message: 1 Date: Fri, 28 Oct 2016 14:12:25 +0000 From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GUI and pmsensors/pmcollector Message-ID: <66CEE4CD-BD01-4F50-839A-529062DC38D3 at siriuscom.com> Content-Type: text/plain; charset="utf-8" Perhaps I needed more description I have a 3 node cluster 2 NDS?s (with tiebreaker) 1 gui node The NDS?s all have pmsensors installed The gui node had pmcollector installed I?ve modified the /opt/IBM/zimon/ZIMSensors.cfg file on the NSD?s to point to my gui node Systemctl start pmsensors on NSD?s Systemclt start pmcollector on gui Systemctl start gpfsgui on gui The log even shows connection from the two NSD?s but for some reason the GUI always has a red x in the performance graphs and claims it can?t connect to the collector (which is running on the same node). Not sure what I?m missing here. It all works fine when it?s all on one node. Mark From: on behalf of Stefan Schmidt Reply-To: gpfsug main discussion list Date: Friday, October 28, 2016 at 8:59 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GUI and pmsensors/pmcollector Hi, the PMCollector must be installed on the same node as the GUI. The collectors can be installed on any node you want to monitor. Hope that helps. Mit freundlichen Gr??en / Kind regards Stefan Schmidt Scrum Master IBM Spectrum Scale GUI / Senior IT Architect/PMP?-Dept. M069 / IBM Spectrum Scale Software Development IBM Systems Group IBM Deutschland ________________________________ Phone: +49-703 4274 1966 IBM Deutschland Am Weiher 24 E-Mail: stschmid at de.ibm.com 65421 Kelsterbach Germany ________________________________ IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Mark.Bush at siriuscom.com" To: "gpfsug-discuss at spectrumscale.org" Date: 28.10.2016 15:53 Subject: [gpfsug-discuss] GUI and pmsensors/pmcollector Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ I am able to do everything I need just fine when I have a single node cluster and both pmsensors and pmcollector are installed all on the same node. When I try and get pmsensors on my NSD nodes and a separate pmcollector elsewhere I never can get anything to show up in the graphs. I?ve configured the pmsensors with mmperfmon config generate ?collectors= (my node) and I still get nothing. The logs show that I get connections but the GUI never show and I?m unable to get mmperfmon query to work either as it fails saying it can?t find the collector. Anyone had this much trouble just getting this to work? Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161028/b4552796/attachment-0001.html > ------------------------------ Message: 2 Date: Fri, 28 Oct 2016 14:22:49 +0000 From: Joshua Akers To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] GPFS Log Levels Message-ID: Content-Type: text/plain; charset="utf-8" Hi all, I am trying to find more information on GPFS log levels. Here is what I have so far: [D] - Detail info [I] - Info [N] - Notice [W] - Warning [E] - Error [X] - Critical Error [A] - Deadlock related? Any corrections or additional information would be greatly appreciated. Thanks, Josh -- *Joshua D. Akers* *HPC Systems Specialist* NI&S Systems Support (MC0214) 1700 Pratt Drive Blacksburg, VA 24061 540-231-9506 -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161028/7ebac661/attachment.html > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 76 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecblank.gif Type: image/gif Size: 45 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 06388797.gif Type: image/gif Size: 1851 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From mweil at wustl.edu Fri Oct 28 21:49:37 2016 From: mweil at wustl.edu (Matt Weil) Date: Fri, 28 Oct 2016 15:49:37 -0500 Subject: [gpfsug-discuss] Leave protocol detail info Message-ID: anybody no what this means? Wed Oct 26 20:08:29.619 2016: [D] Leave protocol detail info: LA: 75 LFLG: 24409951 LFLG delta: 75 ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. From mweil at wustl.edu Fri Oct 28 22:02:59 2016 From: mweil at wustl.edu (Matt Weil) Date: Fri, 28 Oct 2016 16:02:59 -0500 Subject: [gpfsug-discuss] Leave protocol detail info In-Reply-To: References: Message-ID: Also is there any document that explains what happens to the cluster when a client node stops responding and get evicted. On 10/28/16 3:49 PM, Matt Weil wrote: > anybody no what this means? > > Wed Oct 26 20:08:29.619 2016: [D] Leave protocol detail info: LA: 75 > LFLG: 24409951 LFLG delta: 75 > ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. From leslie.james.elliott at gmail.com Sat Oct 29 11:53:19 2016 From: leslie.james.elliott at gmail.com (leslie elliott) Date: Sat, 29 Oct 2016 20:53:19 +1000 Subject: [gpfsug-discuss] unified file and object In-Reply-To: References: Message-ID: Bill to be clear the file access I mentioned was in relation to SMB and NFS using mmuserauth rather than the unification with the object store since it is required as well but I did try to do this for object as well using the Administration and Programming Reference from page 142, was using unified_mode rather than local_mode mmobj config change --ccrfile spectrum-scale-object.conf --section capabilities --property file-access-enabled --value true the mmuserauth failed as you are aware, we have created test accounts without spaces in the DN and were successful with this step, so eagerly await a fix to be able to use the correct accounts mmobj config change --ccrfile object-server-sof.conf --section DEFAULT --property id_mgmt --value unified_mode mmobj config change --ccrfile object-server-sof.conf --section DEFAULT --property ad_domain --value DOMAIN we have successfully tested object stores on this cluster with simple auth the output you asked for is as follows [root at pren-gs7k-vm4 ~]# cat /etc/swift/object-server-sof.conf [DEFAULT] devices = /gpfs/pren01/ObjectFileset/o log_level = ERROR [root at pren-gs7k-vm4 ~]# systemctl -l status openstack-swift-object-sof ? openstack-swift-object-sof.service - OpenStack Object Storage (swift) - Object Server Loaded: loaded (/usr/lib/systemd/system/openstack-swift-object-sof.service; disabled; vendor preset: disabled) Active: failed (Result: exit-code) since Sat 2016-10-29 10:30:22 UTC; 27s ago Process: 8086 ExecStart=/usr/bin/swift-object-server-sof /etc/swift/object-server-sof.conf (code=exited, status=1/FAILURE) Main PID: 8086 (code=exited, status=1/FAILURE) Oct 29 10:30:22 pren-gs7k-vm4 systemd[1]: Started OpenStack Object Storage (swift) - Object Server. Oct 29 10:30:22 pren-gs7k-vm4 systemd[1]: Starting OpenStack Object Storage (swift) - Object Server... Oct 29 10:30:22 pren-gs7k-vm4 swift-object-server-sof[8086]: Error trying to load config from /etc/swift/object-server-sof.conf: No section 'object-server' (prefixed by 'app' or 'application' or 'composite' or 'composit' or 'pipeline' or 'filter-app') found in config /etc/swift/object-server-sof.conf Oct 29 10:30:22 pren-gs7k-vm4 systemd[1]: openstack-swift-object-sof.service: main process exited, code=exited, status=1/FAILURE Oct 29 10:30:22 pren-gs7k-vm4 systemd[1]: Unit openstack-swift-object-sof.service entered failed state. Oct 29 10:30:22 pren-gs7k-vm4 systemd[1]: openstack-swift-object-sof.service failed. I am happy to help you or for you to help to debug this problem via a short call thanks leslie On 29 October 2016 at 00:37, Bill Owen wrote: > > 2. Can you provide more details on how you configured file access? The > normal procedure is to use "mmobj file-access enable", and this will set up > the required settings in the config file. Can you send us: > - the steps used to configure file access > - the resulting /etc/swift/object-server-sof.conf > - log files from /var/log/swift or output of "systemctl status > openstack-swift-object-sof" > > We can schedule a short call to help debug if needed. > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Greg.Lehmann at csiro.au Mon Oct 31 07:03:35 2016 From: Greg.Lehmann at csiro.au (Greg.Lehmann at csiro.au) Date: Mon, 31 Oct 2016 07:03:35 +0000 Subject: [gpfsug-discuss] Two new whitepapers published In-Reply-To: References: Message-ID: <33bda80b2ba24bcb996819dbc30f1d75@exch1-cdc.nexus.csiro.au> Thanks Yuri. These were great. I'm not trying to be impertinent, but I have one suggestion - If you can find the time, add some diagrams to help readers visualise the various data structures and layouts. I am thinking along the lines of what was in "The Magic Garden Explained" and "The Design and Implementation of the 4.3BSD UNIX Operating System" where they talked about their filesystems. Cheers, Greg From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Yuri L Volobuev Sent: Friday, 21 October 2016 11:59 AM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Two new whitepapers published Esteemed GPFS and Spectrum Scale users, For your reading enjoyment, two new whitepapers have been posted to the Spectrum Scale Wiki: Spectrum Scale: Replication in GPFS Spectrum Scale: GPFS Metadata The URL for the parent page is https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/White%20Papers%20%26%20Media The two .pdf documents are accessible through the Attachment section at the bottom of the page. Unfortunately, dW "spam prevention engine" does a very good job preventing me from "spamming" the page to actually add links. Best regards, Yuri -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Mon Oct 31 09:53:38 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 31 Oct 2016 09:53:38 +0000 Subject: [gpfsug-discuss] Recent Whitepapers from Yuri Volobuev Message-ID: For those of you who may not know, Yuri Volobuev has left IBM to pursue new challenges. Myself along with many others received so much help and keen insight from Yuri on all things GPFS. He will be missed. Bob Oesterlin Sr Principal Storage Engineer, Nuance -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.knister at gmail.com Mon Oct 31 16:19:11 2016 From: aaron.knister at gmail.com (Aaron Knister) Date: Mon, 31 Oct 2016 12:19:11 -0400 Subject: [gpfsug-discuss] Recent Whitepapers from Yuri Volobuev In-Reply-To: References: Message-ID: <9C6B34F6-2849-4B77-9E8B-3806D634C281@gmail.com> Wow! That's a real shame. His ability to articulate his immense knowledge of GPFS was truly impressive. Sent from my iPhone > On Oct 31, 2016, at 5:53 AM, Oesterlin, Robert wrote: > > For those of you who may not know, Yuri Volobuev has left IBM to pursue new challenges. Myself along with many others received so much help and keen insight from Yuri on all things GPFS. > > He will be missed. > > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.wonderley at vt.edu Mon Oct 31 16:52:43 2016 From: eric.wonderley at vt.edu (J. Eric Wonderley) Date: Mon, 31 Oct 2016 12:52:43 -0400 Subject: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size Message-ID: I wanted to do something like this... [root at cl001 ~]# cat /opt/gpfs/home.ply /*Failsafe migration of old small files back to spinning media pool(fc_8T) */ RULE 'theshold' MIGRATE FROM POOL 'system' THRESHOLD(90,70) WEIGHT(ACCESS_TIME) TO POOL 'fc_8T' /*Write files larger than 16MB to pool called "fc_8T" */ RULE 'bigfiles' SET POOL 'fc_8T' WHERE FILE_SIZE>16777216 /*Move anything else to system pool */ RULE 'default' SET POOL 'system' Apparently there is no happiness using FILE_SIZE in a placement policy: [root at cl001 ~]# mmchpolicy home /opt/gpfs/home.ply Error while validating policy `home.ply': rc=22: PCSQLERR: 'FILE_SIZE' is an unsupported or unknown attribute or variable name in this context. PCSQLCTX: at line 4 of 6: RULE 'bigfiles' SET POOL 'fc_8T' WHERE {{{FILE_SIZE}}}>16777216 runRemoteCommand_v2: cl002.cl.arc.internal: tschpolicy /dev/home /var/mmfs/tmp/tspolicyFile.mmchpolicy.113372 -t home.ply failed. mmchpolicy: Command failed. Examine previous error messages to determine cause. Can anyone suggest a way to accomplish this using policy? -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Mon Oct 31 17:09:55 2016 From: bbanister at jumptrading.com (Bryan Banister) Date: Mon, 31 Oct 2016 17:09:55 +0000 Subject: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size In-Reply-To: References: Message-ID: <21BC488F0AEA2245B2C3E83FC0B33DBB063A1D4A@CHI-EXCHANGEW1.w2k.jumptrading.com> The File Placement Policy that you are trying to set cannot use the size of the file to determine the placement of the file in a GPFS Storage Pool. This is because GPFS has no idea what the file size will be when the file is open()?d for writing. Hope that helps! -Bryan PS. I really wish that we could use a path for specifying data placement in a GPFS Pool, and not just the file name, owner, etc. I?ll submit a RFE for this. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of J. Eric Wonderley Sent: Monday, October 31, 2016 11:53 AM To: gpfsug main discussion list Subject: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size I wanted to do something like this... [root at cl001 ~]# cat /opt/gpfs/home.ply /*Failsafe migration of old small files back to spinning media pool(fc_8T) */ RULE 'theshold' MIGRATE FROM POOL 'system' THRESHOLD(90,70) WEIGHT(ACCESS_TIME) TO POOL 'fc_8T' /*Write files larger than 16MB to pool called "fc_8T" */ RULE 'bigfiles' SET POOL 'fc_8T' WHERE FILE_SIZE>16777216 /*Move anything else to system pool */ RULE 'default' SET POOL 'system' Apparently there is no happiness using FILE_SIZE in a placement policy: [root at cl001 ~]# mmchpolicy home /opt/gpfs/home.ply Error while validating policy `home.ply': rc=22: PCSQLERR: 'FILE_SIZE' is an unsupported or unknown attribute or variable name in this context. PCSQLCTX: at line 4 of 6: RULE 'bigfiles' SET POOL 'fc_8T' WHERE {{{FILE_SIZE}}}>16777216 runRemoteCommand_v2: cl002.cl.arc.internal: tschpolicy /dev/home /var/mmfs/tmp/tspolicyFile.mmchpolicy.113372 -t home.ply failed. mmchpolicy: Command failed. Examine previous error messages to determine cause. Can anyone suggest a way to accomplish this using policy? ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jez.tucker at gpfsug.org Mon Oct 31 17:20:30 2016 From: jez.tucker at gpfsug.org (Jez Tucker) Date: Mon, 31 Oct 2016 17:20:30 +0000 Subject: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size In-Reply-To: <21BC488F0AEA2245B2C3E83FC0B33DBB063A1D4A@CHI-EXCHANGEW1.w2k.jumptrading.com> References: <21BC488F0AEA2245B2C3E83FC0B33DBB063A1D4A@CHI-EXCHANGEW1.w2k.jumptrading.com> Message-ID: Hey Bryan There was a previous RFE for path placement from the UG, but Yuri told me this was not techically possible as an inode has no knowledge about the parent dentry. (IIRC). You can see this in effect in the C API. It is possible to work this out at kernel level, but it's so costly that it becomes non-viable at scale / performance. IBMers please chip in and expand if you will. Jez On 31/10/16 17:09, Bryan Banister wrote: > > The File Placement Policy that you are trying to set cannot use the > size of the file to determine the placement of the file in a GPFS > Storage Pool. This is because GPFS has no idea what the file size > will be when the file is open()?d for writing. > > Hope that helps! > > -Bryan > > PS. I really wish that we could use a path for specifying data > placement in a GPFS Pool, and not just the file name, owner, etc. > I?ll submit a RFE for this. > > *From:*gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of *J. > Eric Wonderley > *Sent:* Monday, October 31, 2016 11:53 AM > *To:* gpfsug main discussion list > *Subject:* [gpfsug-discuss] wanted...gpfs policy that places larger > files onto a pool based on size > > I wanted to do something like this... > > > [root at cl001 ~]# cat /opt/gpfs/home.ply > /*Failsafe migration of old small files back to spinning media > pool(fc_8T) */ > RULE 'theshold' MIGRATE FROM POOL 'system' THRESHOLD(90,70) > WEIGHT(ACCESS_TIME) TO POOL 'fc_8T' > /*Write files larger than 16MB to pool called "fc_8T" */ > RULE 'bigfiles' SET POOL 'fc_8T' WHERE FILE_SIZE>16777216 > /*Move anything else to system pool */ > RULE 'default' SET POOL 'system' > > Apparently there is no happiness using FILE_SIZE in a placement policy: > [root at cl001 ~]# mmchpolicy home /opt/gpfs/home.ply > Error while validating policy `home.ply': rc=22: > PCSQLERR: 'FILE_SIZE' is an unsupported or unknown attribute or > variable name in this context. > PCSQLCTX: at line 4 of 6: RULE 'bigfiles' SET POOL 'fc_8T' WHERE > {{{FILE_SIZE}}}>16777216 > runRemoteCommand_v2: cl002.cl.arc.internal: tschpolicy /dev/home > /var/mmfs/tmp/tspolicyFile.mmchpolicy.113372 -t home.ply failed. > mmchpolicy: Command failed. Examine previous error messages to > determine cause. > > Can anyone suggest a way to accomplish this using policy? > > > ------------------------------------------------------------------------ > > Note: This email is for the confidential use of the named addressee(s) > only and may contain proprietary, confidential or privileged > information. If you are not the intended recipient, you are hereby > notified that any review, dissemination or copying of this email is > strictly prohibited, and to please notify the sender immediately and > destroy this email and any attachments. Email transmission cannot be > guaranteed to be secure or error-free. The Company, therefore, does > not make any guarantees as to the completeness or accuracy of this > email or any attachments. This email is for informational purposes > only and does not constitute a recommendation, offer, request or > solicitation of any kind to buy, sell, subscribe, redeem or perform > any type of transaction of a financial product. > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From mimarsh2 at vt.edu Mon Oct 31 17:23:40 2016 From: mimarsh2 at vt.edu (Brian Marshall) Date: Mon, 31 Oct 2016 13:23:40 -0400 Subject: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size In-Reply-To: References: <21BC488F0AEA2245B2C3E83FC0B33DBB063A1D4A@CHI-EXCHANGEW1.w2k.jumptrading.com> Message-ID: When creating a "fast tier" storage pool in a filesystem is the normal case to create a placement policy that places all files in the fast tier and migrates out old and large files? Brian Marshall On Mon, Oct 31, 2016 at 1:20 PM, Jez Tucker wrote: > Hey Bryan > > There was a previous RFE for path placement from the UG, but Yuri told > me this was not techically possible as an inode has no knowledge about the > parent dentry. (IIRC). You can see this in effect in the C API. It is > possible to work this out at kernel level, but it's so costly that it > becomes non-viable at scale / performance. > > IBMers please chip in and expand if you will. > > Jez > > > On 31/10/16 17:09, Bryan Banister wrote: > > The File Placement Policy that you are trying to set cannot use the size > of the file to determine the placement of the file in a GPFS Storage Pool. > This is because GPFS has no idea what the file size will be when the file > is open()?d for writing. > > > > Hope that helps! > > -Bryan > > > > PS. I really wish that we could use a path for specifying data placement > in a GPFS Pool, and not just the file name, owner, etc. I?ll submit a RFE > for this. > > > > *From:* gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss- > bounces at spectrumscale.org ] *On > Behalf Of *J. Eric Wonderley > *Sent:* Monday, October 31, 2016 11:53 AM > *To:* gpfsug main discussion list > *Subject:* [gpfsug-discuss] wanted...gpfs policy that places larger files > onto a pool based on size > > > > I wanted to do something like this... > > > [root at cl001 ~]# cat /opt/gpfs/home.ply > /*Failsafe migration of old small files back to spinning media pool(fc_8T) > */ > RULE 'theshold' MIGRATE FROM POOL 'system' THRESHOLD(90,70) > WEIGHT(ACCESS_TIME) TO POOL 'fc_8T' > /*Write files larger than 16MB to pool called "fc_8T" */ > RULE 'bigfiles' SET POOL 'fc_8T' WHERE FILE_SIZE>16777216 > /*Move anything else to system pool */ > RULE 'default' SET POOL 'system' > > Apparently there is no happiness using FILE_SIZE in a placement policy: > [root at cl001 ~]# mmchpolicy home /opt/gpfs/home.ply > Error while validating policy `home.ply': rc=22: > PCSQLERR: 'FILE_SIZE' is an unsupported or unknown attribute or variable > name in this context. > PCSQLCTX: at line 4 of 6: RULE 'bigfiles' SET POOL 'fc_8T' WHERE > {{{FILE_SIZE}}}>16777216 > runRemoteCommand_v2: cl002.cl.arc.internal: tschpolicy /dev/home > /var/mmfs/tmp/tspolicyFile.mmchpolicy.113372 -t home.ply failed. > mmchpolicy: Command failed. Examine previous error messages to determine > cause. > > Can anyone suggest a way to accomplish this using policy? > > ------------------------------ > > Note: This email is for the confidential use of the named addressee(s) > only and may contain proprietary, confidential or privileged information. > If you are not the intended recipient, you are hereby notified that any > review, dissemination or copying of this email is strictly prohibited, and > to please notify the sender immediately and destroy this email and any > attachments. Email transmission cannot be guaranteed to be secure or > error-free. The Company, therefore, does not make any guarantees as to the > completeness or accuracy of this email or any attachments. This email is > for informational purposes only and does not constitute a recommendation, > offer, request or solicitation of any kind to buy, sell, subscribe, redeem > or perform any type of transaction of a financial product. > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ewahl at osc.edu Mon Oct 31 17:26:28 2016 From: ewahl at osc.edu (Edward Wahl) Date: Mon, 31 Oct 2016 13:26:28 -0400 Subject: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size In-Reply-To: <21BC488F0AEA2245B2C3E83FC0B33DBB063A1D4A@CHI-EXCHANGEW1.w2k.jumptrading.com> References: <21BC488F0AEA2245B2C3E83FC0B33DBB063A1D4A@CHI-EXCHANGEW1.w2k.jumptrading.com> Message-ID: <20161031132628.5c952a40@osc.edu> On Mon, 31 Oct 2016 17:09:55 +0000 Bryan Banister wrote: > -Bryan > ? > PS. I really wish that we could use a path for specifying data > placement in a GPFS Pool, and not just the file name, owner, etc. > I?ll submit a RFE for this. So... use a fileset with a linked directory and a fileset placement policy to a pool? Might be a bit more rigid for what you really want, and it would be messy, but it would work just fine. -- Ed Wahl Ohio Supercomputer Center 614-292-9302 From bbanister at jumptrading.com Mon Oct 31 17:29:05 2016 From: bbanister at jumptrading.com (Bryan Banister) Date: Mon, 31 Oct 2016 17:29:05 +0000 Subject: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size In-Reply-To: <20161031132628.5c952a40@osc.edu> References: <21BC488F0AEA2245B2C3E83FC0B33DBB063A1D4A@CHI-EXCHANGEW1.w2k.jumptrading.com> <20161031132628.5c952a40@osc.edu> Message-ID: <21BC488F0AEA2245B2C3E83FC0B33DBB063A1F04@CHI-EXCHANGEW1.w2k.jumptrading.com> Thanks for that suggestion, which is something we already do, but as you say is not as flexible. I think Jez is correct about the overhead being too high to support directory path for data placement, -B -----Original Message----- From: Edward Wahl [mailto:ewahl at osc.edu] Sent: Monday, October 31, 2016 12:26 PM To: Bryan Banister Cc: gpfsug main discussion list Subject: Re: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size On Mon, 31 Oct 2016 17:09:55 +0000 Bryan Banister wrote: > -Bryan > > PS. I really wish that we could use a path for specifying data > placement in a GPFS Pool, and not just the file name, owner, etc. > I?ll submit a RFE for this. So... use a fileset with a linked directory and a fileset placement policy to a pool? Might be a bit more rigid for what you really want, and it would be messy, but it would work just fine. -- Ed Wahl Ohio Supercomputer Center 614-292-9302 ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. From chrisjscott at gmail.com Mon Oct 31 17:29:45 2016 From: chrisjscott at gmail.com (Chris Scott) Date: Mon, 31 Oct 2016 17:29:45 +0000 Subject: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size In-Reply-To: References: <21BC488F0AEA2245B2C3E83FC0B33DBB063A1D4A@CHI-EXCHANGEW1.w2k.jumptrading.com> Message-ID: Hi Brian This is exactly what I do with a SSD tier on top of 10K and 7.2K tiers. HAWC is another recent option that might address Eric's requirement but needs further consideration of the read requirements you want from the small files. Cheers Chris On 31 October 2016 at 17:23, Brian Marshall wrote: > When creating a "fast tier" storage pool in a filesystem is the normal > case to create a placement policy that places all files in the fast tier > and migrates out old and large files? > > > Brian Marshall > > On Mon, Oct 31, 2016 at 1:20 PM, Jez Tucker wrote: > >> Hey Bryan >> >> There was a previous RFE for path placement from the UG, but Yuri told >> me this was not techically possible as an inode has no knowledge about the >> parent dentry. (IIRC). You can see this in effect in the C API. It is >> possible to work this out at kernel level, but it's so costly that it >> becomes non-viable at scale / performance. >> >> IBMers please chip in and expand if you will. >> >> Jez >> >> >> On 31/10/16 17:09, Bryan Banister wrote: >> >> The File Placement Policy that you are trying to set cannot use the size >> of the file to determine the placement of the file in a GPFS Storage Pool. >> This is because GPFS has no idea what the file size will be when the file >> is open()?d for writing. >> >> >> >> Hope that helps! >> >> -Bryan >> >> >> >> PS. I really wish that we could use a path for specifying data placement >> in a GPFS Pool, and not just the file name, owner, etc. I?ll submit a RFE >> for this. >> >> >> >> *From:* gpfsug-discuss-bounces at spectrumscale.org [ >> mailto:gpfsug-discuss-bounces at spectrumscale.org >> ] *On Behalf Of *J. Eric >> Wonderley >> *Sent:* Monday, October 31, 2016 11:53 AM >> *To:* gpfsug main discussion list >> *Subject:* [gpfsug-discuss] wanted...gpfs policy that places larger >> files onto a pool based on size >> >> >> >> I wanted to do something like this... >> >> >> [root at cl001 ~]# cat /opt/gpfs/home.ply >> /*Failsafe migration of old small files back to spinning media >> pool(fc_8T) */ >> RULE 'theshold' MIGRATE FROM POOL 'system' THRESHOLD(90,70) >> WEIGHT(ACCESS_TIME) TO POOL 'fc_8T' >> /*Write files larger than 16MB to pool called "fc_8T" */ >> RULE 'bigfiles' SET POOL 'fc_8T' WHERE FILE_SIZE>16777216 >> /*Move anything else to system pool */ >> RULE 'default' SET POOL 'system' >> >> Apparently there is no happiness using FILE_SIZE in a placement policy: >> [root at cl001 ~]# mmchpolicy home /opt/gpfs/home.ply >> Error while validating policy `home.ply': rc=22: >> PCSQLERR: 'FILE_SIZE' is an unsupported or unknown attribute or variable >> name in this context. >> PCSQLCTX: at line 4 of 6: RULE 'bigfiles' SET POOL 'fc_8T' WHERE >> {{{FILE_SIZE}}}>16777216 >> runRemoteCommand_v2: cl002.cl.arc.internal: tschpolicy /dev/home >> /var/mmfs/tmp/tspolicyFile.mmchpolicy.113372 -t home.ply failed. >> mmchpolicy: Command failed. Examine previous error messages to determine >> cause. >> >> Can anyone suggest a way to accomplish this using policy? >> >> ------------------------------ >> >> Note: This email is for the confidential use of the named addressee(s) >> only and may contain proprietary, confidential or privileged information. >> If you are not the intended recipient, you are hereby notified that any >> review, dissemination or copying of this email is strictly prohibited, and >> to please notify the sender immediately and destroy this email and any >> attachments. Email transmission cannot be guaranteed to be secure or >> error-free. The Company, therefore, does not make any guarantees as to the >> completeness or accuracy of this email or any attachments. This email is >> for informational purposes only and does not constitute a recommendation, >> offer, request or solicitation of any kind to buy, sell, subscribe, redeem >> or perform any type of transaction of a financial product. >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chrisjscott at gmail.com Mon Oct 31 17:33:57 2016 From: chrisjscott at gmail.com (Chris Scott) Date: Mon, 31 Oct 2016 17:33:57 +0000 Subject: [gpfsug-discuss] Recent Whitepapers from Yuri Volobuev In-Reply-To: <9C6B34F6-2849-4B77-9E8B-3806D634C281@gmail.com> References: <9C6B34F6-2849-4B77-9E8B-3806D634C281@gmail.com> Message-ID: Hear, hear. My appreciation to Yuri for his great contribution to the user community of important details. On 31 October 2016 at 16:19, Aaron Knister wrote: > Wow! That's a real shame. His ability to articulate his immense knowledge > of GPFS was truly impressive. > > Sent from my iPhone > > On Oct 31, 2016, at 5:53 AM, Oesterlin, Robert < > Robert.Oesterlin at nuance.com> wrote: > > For those of you who may not know, Yuri Volobuev has left IBM to pursue > new challenges. Myself along with many others received so much help and > keen insight from Yuri on all things GPFS. > > > > He will be missed. > > > > > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.wonderley at vt.edu Mon Oct 31 17:45:48 2016 From: eric.wonderley at vt.edu (J. Eric Wonderley) Date: Mon, 31 Oct 2016 13:45:48 -0400 Subject: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size In-Reply-To: References: <21BC488F0AEA2245B2C3E83FC0B33DBB063A1D4A@CHI-EXCHANGEW1.w2k.jumptrading.com> Message-ID: Placement policy only applies to writes and I had thought that gpfs did enough writing to memory "pagepool" to figure out the size before committing the write to pool. I also admit I don't know all of the innards of gpfs. Pehaps being a copy on write type filesystem prevents this for occurring. On Mon, Oct 31, 2016 at 1:29 PM, Chris Scott wrote: > Hi Brian > > This is exactly what I do with a SSD tier on top of 10K and 7.2K tiers. > > HAWC is another recent option that might address Eric's requirement but > needs further consideration of the read requirements you want from the > small files. > > Cheers > Chris > > On 31 October 2016 at 17:23, Brian Marshall wrote: > >> When creating a "fast tier" storage pool in a filesystem is the normal >> case to create a placement policy that places all files in the fast tier >> and migrates out old and large files? >> >> >> Brian Marshall >> >> On Mon, Oct 31, 2016 at 1:20 PM, Jez Tucker >> wrote: >> >>> Hey Bryan >>> >>> There was a previous RFE for path placement from the UG, but Yuri told >>> me this was not techically possible as an inode has no knowledge about the >>> parent dentry. (IIRC). You can see this in effect in the C API. It is >>> possible to work this out at kernel level, but it's so costly that it >>> becomes non-viable at scale / performance. >>> >>> IBMers please chip in and expand if you will. >>> >>> Jez >>> >>> >>> On 31/10/16 17:09, Bryan Banister wrote: >>> >>> The File Placement Policy that you are trying to set cannot use the size >>> of the file to determine the placement of the file in a GPFS Storage Pool. >>> This is because GPFS has no idea what the file size will be when the file >>> is open()?d for writing. >>> >>> >>> >>> Hope that helps! >>> >>> -Bryan >>> >>> >>> >>> PS. I really wish that we could use a path for specifying data placement >>> in a GPFS Pool, and not just the file name, owner, etc. I?ll submit a RFE >>> for this. >>> >>> >>> >>> *From:* gpfsug-discuss-bounces at spectrumscale.org [ >>> mailto:gpfsug-discuss-bounces at spectrumscale.org >>> ] *On Behalf Of *J. Eric >>> Wonderley >>> *Sent:* Monday, October 31, 2016 11:53 AM >>> *To:* gpfsug main discussion list >>> *Subject:* [gpfsug-discuss] wanted...gpfs policy that places larger >>> files onto a pool based on size >>> >>> >>> >>> I wanted to do something like this... >>> >>> >>> [root at cl001 ~]# cat /opt/gpfs/home.ply >>> /*Failsafe migration of old small files back to spinning media >>> pool(fc_8T) */ >>> RULE 'theshold' MIGRATE FROM POOL 'system' THRESHOLD(90,70) >>> WEIGHT(ACCESS_TIME) TO POOL 'fc_8T' >>> /*Write files larger than 16MB to pool called "fc_8T" */ >>> RULE 'bigfiles' SET POOL 'fc_8T' WHERE FILE_SIZE>16777216 >>> /*Move anything else to system pool */ >>> RULE 'default' SET POOL 'system' >>> >>> Apparently there is no happiness using FILE_SIZE in a placement policy: >>> [root at cl001 ~]# mmchpolicy home /opt/gpfs/home.ply >>> Error while validating policy `home.ply': rc=22: >>> PCSQLERR: 'FILE_SIZE' is an unsupported or unknown attribute or variable >>> name in this context. >>> PCSQLCTX: at line 4 of 6: RULE 'bigfiles' SET POOL 'fc_8T' WHERE >>> {{{FILE_SIZE}}}>16777216 >>> runRemoteCommand_v2: cl002.cl.arc.internal: tschpolicy /dev/home >>> /var/mmfs/tmp/tspolicyFile.mmchpolicy.113372 -t home.ply failed. >>> mmchpolicy: Command failed. Examine previous error messages to determine >>> cause. >>> >>> Can anyone suggest a way to accomplish this using policy? >>> >>> ------------------------------ >>> >>> Note: This email is for the confidential use of the named addressee(s) >>> only and may contain proprietary, confidential or privileged information. >>> If you are not the intended recipient, you are hereby notified that any >>> review, dissemination or copying of this email is strictly prohibited, and >>> to please notify the sender immediately and destroy this email and any >>> attachments. Email transmission cannot be guaranteed to be secure or >>> error-free. The Company, therefore, does not make any guarantees as to the >>> completeness or accuracy of this email or any attachments. This email is >>> for informational purposes only and does not constitute a recommendation, >>> offer, request or solicitation of any kind to buy, sell, subscribe, redeem >>> or perform any type of transaction of a financial product. >>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sfadden at us.ibm.com Mon Oct 3 13:39:00 2016 From: sfadden at us.ibm.com (Scott Fadden) Date: Mon, 3 Oct 2016 05:39:00 -0700 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? In-Reply-To: <0D55AB1D-DB9D-45CF-AB27-157CDA1172D9@nuance.com> References: <0D55AB1D-DB9D-45CF-AB27-157CDA1172D9@nuance.com> Message-ID: About 3.5 KiB if the inode is 4k and you are not using encryption (uses EA space) or large custom extended attributes, Scott Fadden Spectrum Scale - Technical Marketing Phone: (503) 880-5833 sfadden at us.ibm.com http://www.ibm.com/systems/storage/spectrum/scale From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 09/28/2016 11:14 AM Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? Sent by: gpfsug-discuss-bounces at spectrumscale.org What the largest file that will fit inside a 1K, 2K, or 4K inode? Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Mon Oct 3 13:54:47 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Mon, 3 Oct 2016 12:54:47 +0000 Subject: [gpfsug-discuss] License question Message-ID: I have a question about licensing. If I have a system which is NFS mounting (from a CES node), that is then serving data via HTTP, does this system need to have a Server license? Reading the FAQ (in the context of VMs): https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html "The IBM Spectrum Scale Client may not be used for virtual servers to share IBM Spectrum Scale data directly through any application, service protocol or method, such as NFS, CIFS, FTP, HTTP, or OpenStack Swift" My reading of this is that if the VM is mounting via NFS, then as it is not a Spectrum Scale client (rather an NFS client), then it doesn't require a Scale license. Secondly. If the VM is mounting from the hypervisor via NFS (Manilla), is the VM then allowed to serve data out by e.g. Web server, or does the hypervisor require a server license to do this. Assuming my hypervisor is licensed for Spectrum Scale of course. I think my interpretation is that if a VM is using NFS mounting from either a CES node or using Manilla to the hypervisor, then this is all OK to "re-export" data as the VM isn't talking GPFS protocol. Is this correct? Simon From daniel.kidger at uk.ibm.com Mon Oct 3 14:23:35 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Mon, 3 Oct 2016 13:23:35 +0000 Subject: [gpfsug-discuss] License question In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Mon Oct 3 15:53:51 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 3 Oct 2016 10:53:51 -0400 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? 3876! In-Reply-To: References: <0D55AB1D-DB9D-45CF-AB27-157CDA1172D9@nuance.com> Message-ID: Pretty easy to determine experimentally, with a few trials.... And the answer is 3876 for an simple file with just a selinux EA which is apparently created by default on my test system. So apparently there are actually only 220 bytes of metadata in this case (4096-3876=220). As they say YMMV ;-) [root at bog-wifi cmvc]# mmlsfs yy -i flag value description ------------------- ------------------------ ----------------------------------- -i 4096 Inode size in bytes [root at bog-wifi isize]# stat i387[67] File: ?i3876? Size: 3876 Blocks: 0 IO Block: 262144 regular file File: ?i3877? Size: 3877 Blocks: 16 IO Block: 262144 regular file [root at bog-wifi isize]# ls -ild i3876 19406 -rw-r--r--. 1 root root 3876 Oct 3 10:39 i3876 tsdbfs yy inode 19406 Inode 19406 [19406] snap 0 (index 14 in block 303): Inode address: 1:511600 size 4096 nAddrs 323 indirectionLevel=INODE status=USERFILE objectVersion=2 generation=0x7C5599C4 nlink=1 owner uid=0 gid=0 mode=0200100644: -rw-r--r-- ... Data [3876]: 0000000000000000: 136D7B1F 1D29CA0C B59BC917 D9AE0220 *.m{..)..........* 0000000000000010: F18512CF 998AC02E DC6926FA 3FDC3EAF *.........i&.?.>.* ... 0000000000000F20: EFF02209 00000000 05077365 6C696E75 *..".......selinu* trailer: 0x20005993FD8 diskAddrs 323 xattrLen 48 (free 3856/3904) res 0,0,0 overflow pri 0 vers 0 nsub 0 repl 1/2 DA Null [0] id 0x5 prefLen 8 keyLen 7 key security.selinux len 37 val '756E636F6E66696E65645F753A6F626A6563745F723A756E6C6162656C65645F743 A733000' -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 21994 bytes Desc: not available URL: From makaplan at us.ibm.com Mon Oct 3 16:46:09 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 3 Oct 2016 11:46:09 -0400 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? 3968!! In-Reply-To: References: <0D55AB1D-DB9D-45CF-AB27-157CDA1172D9@nuance.com> Message-ID: On a non-SELINUX system the answer is 3968 of data in a 4K inode, just 128 bytes of metadata. Caution: it's possible in some future release, this could change ... I don't know of any plans, I'm just saying ... Inode 16346892 [16346892] snap 0 (index 12 in block 255420): Inode address: 6:123049056 size 4096 nAddrs 330 indirectionLevel=INODE status=USERFILE objectVersion=1 generation=0xC0156CB nlink=1 owner uid=0 gid=0 mode=0200100644: -rw-r--r-- blocksize code=5 (32 subblocks) lastBlockSubblocks=0 checksum=0xAD8E0B4B is Valid fileSize=3968 nFullBlocks=0 currentMetadataReplicas=1 maxMetadataReplicas=2 currentDataReplicas=1 maxDataReplicas=2 ... Data [3968]: 0000000000000000: BCA91252 2B64BEDC A7D7BA9D D5BE8C30 *...R+d.........0* ... 0000000000000F70: DA925E2F 16A68C01 03CA5E37 08D72B7F *..^/......^7..+.* trailer: is NULL -------------- next part -------------- An HTML attachment was scrubbed... URL: From A.K.Ghumra at bham.ac.uk Mon Oct 3 16:52:52 2016 From: A.K.Ghumra at bham.ac.uk (Aslam Ghumra (IT Services, Facilities Management)) Date: Mon, 3 Oct 2016 15:52:52 +0000 Subject: [gpfsug-discuss] WebDAV protocol access support Message-ID: Does anyone know whether WebDAV protocol access support will ( if ever ) be implemented within GPFS ? Regards, Aslam Ghumra Research Data Management ____________________________ IT Services Elms Road Data Centre Building G5 Edgbaston Birmingham B15 2TT T: 0121 414 5877 F; 0121 414 3952 Skype : JanitorX Twitter : @aslamghumra in : https://uk.linkedin.com/in/aslam-ghumra-13907993 http://intranet.bham.ac.uk/bear -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Mon Oct 3 16:56:52 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 3 Oct 2016 15:56:52 +0000 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? Message-ID: <1BDA1763-A6FE-4A52-B2ED-22324C9AA16F@nuance.com> What's going be taken away if you use Encryption or Transparent Cloud Tiering? Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: on behalf of Marc A Kaplan Reply-To: gpfsug main discussion list Date: Monday, October 3, 2016 at 10:46 AM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Biggest file that will fit inside an inode? 3968!! On a non-SELINUX system the answer is 3968 of data in a 4K inode, just 128 bytes of metadata. Caution: it's possible in some future release, this could change ... I don't know of any plans, I'm just saying ... Inode 16346892 [16346892] snap 0 (index 12 in block 255420): Inode address: 6:123049056 size 4096 nAddrs 330 indirectionLevel=INODE status=USERFILE objectVersion=1 generation=0xC0156CB nlink=1 owner uid=0 gid=0 mode=0200100644: -rw-r--r-- blocksize code=5 (32 subblocks) lastBlockSubblocks=0 checksum=0xAD8E0B4B is Valid fileSize=3968 nFullBlocks=0 currentMetadataReplicas=1 maxMetadataReplicas=2 currentDataReplicas=1 maxDataReplicas=2 ... Data [3968]: 0000000000000000: BCA91252 2B64BEDC A7D7BA9D D5BE8C30 *...R+d.........0* ... 0000000000000F70: DA925E2F 16A68C01 03CA5E37 08D72B7F *..^/......^7..+.* trailer: is NULL -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Mon Oct 3 16:59:54 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Mon, 3 Oct 2016 15:59:54 +0000 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? In-Reply-To: <1BDA1763-A6FE-4A52-B2ED-22324C9AA16F@nuance.com> References: <1BDA1763-A6FE-4A52-B2ED-22324C9AA16F@nuance.com> Message-ID: Would you tier an in-inode file to the cloud? I mean, I wouldn't tier an in-inode file out to tape? Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Oesterlin, Robert [Robert.Oesterlin at nuance.com] Sent: 03 October 2016 16:56 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? What's going be taken away if you use Encryption or Transparent Cloud Tiering? Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: on behalf of Marc A Kaplan Reply-To: gpfsug main discussion list Date: Monday, October 3, 2016 at 10:46 AM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Biggest file that will fit inside an inode? 3968!! On a non-SELINUX system the answer is 3968 of data in a 4K inode, just 128 bytes of metadata. Caution: it's possible in some future release, this could change ... I don't know of any plans, I'm just saying ... Inode 16346892 [16346892] snap 0 (index 12 in block 255420): Inode address: 6:123049056 size 4096 nAddrs 330 indirectionLevel=INODE status=USERFILE objectVersion=1 generation=0xC0156CB nlink=1 owner uid=0 gid=0 mode=0200100644: -rw-r--r-- blocksize code=5 (32 subblocks) lastBlockSubblocks=0 checksum=0xAD8E0B4B is Valid fileSize=3968 nFullBlocks=0 currentMetadataReplicas=1 maxMetadataReplicas=2 currentDataReplicas=1 maxDataReplicas=2 ... Data [3968]: 0000000000000000: BCA91252 2B64BEDC A7D7BA9D D5BE8C30 *...R+d.........0* ... 0000000000000F70: DA925E2F 16A68C01 03CA5E37 08D72B7F *..^/......^7..+.* trailer: is NULL From Robert.Oesterlin at nuance.com Mon Oct 3 17:06:41 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 3 Oct 2016 16:06:41 +0000 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? In-Reply-To: References: <1BDA1763-A6FE-4A52-B2ED-22324C9AA16F@nuance.com> Message-ID: <37F799FA-7200-4011-81F0-6FF0DF3A4006@nuance.com> No, but (I think) TCT uses some of the EA space in the inode to indicate that the data is in the cloud tier. And yes, it does copy the data blocks in the inode to the clod tier if you migrate it. Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: on behalf of "Simon Thompson (Research Computing - IT Services)" Reply-To: gpfsug main discussion list Date: Monday, October 3, 2016 at 10:59 AM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Biggest file that will fit inside an inode? Would you tier an in-inode file to the cloud? I mean, I wouldn't tier an in-inode file out to tape? Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Oesterlin, Robert [Robert.Oesterlin at nuance.com] Sent: 03 October 2016 16:56 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? What's going be taken away if you use Encryption or Transparent Cloud Tiering? Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: > on behalf of Marc A Kaplan > Reply-To: gpfsug main discussion list > Date: Monday, October 3, 2016 at 10:46 AM To: gpfsug main discussion list > Subject: [EXTERNAL] Re: [gpfsug-discuss] Biggest file that will fit inside an inode? 3968!! On a non-SELINUX system the answer is 3968 of data in a 4K inode, just 128 bytes of metadata. Caution: it's possible in some future release, this could change ... I don't know of any plans, I'm just saying ... Inode 16346892 [16346892] snap 0 (index 12 in block 255420): Inode address: 6:123049056 size 4096 nAddrs 330 indirectionLevel=INODE status=USERFILE objectVersion=1 generation=0xC0156CB nlink=1 owner uid=0 gid=0 mode=0200100644: -rw-r--r-- blocksize code=5 (32 subblocks) lastBlockSubblocks=0 checksum=0xAD8E0B4B is Valid fileSize=3968 nFullBlocks=0 currentMetadataReplicas=1 maxMetadataReplicas=2 currentDataReplicas=1 maxDataReplicas=2 ... Data [3968]: 0000000000000000: BCA91252 2B64BEDC A7D7BA9D D5BE8C30 *...R+d.........0* ... 0000000000000F70: DA925E2F 16A68C01 03CA5E37 08D72B7F *..^/......^7..+.* trailer: is NULL _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=1C_5OQ7M5ibcRAY6xnDfRpKRJiw3qUawXpXihmGoQcs&s=skn6tFln6MeuKACIoxnUL_kuxHIC7AkskTni8IjzLj0&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From luke.raimbach at googlemail.com Mon Oct 3 17:07:39 2016 From: luke.raimbach at googlemail.com (Luke Raimbach) Date: Mon, 03 Oct 2016 16:07:39 +0000 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? In-Reply-To: References: <1BDA1763-A6FE-4A52-B2ED-22324C9AA16F@nuance.com> Message-ID: Surely it wouldn't go? Maybe the data would get copied out rather than stubbed... DMAPI can't be stupid enough to stub data out of an inode? Can it? Interesting question. Maybe I'll test that one. On Mon, 3 Oct 2016 at 17:00 Simon Thompson (Research Computing - IT Services) wrote: > > Would you tier an in-inode file to the cloud? > > I mean, I wouldn't tier an in-inode file out to tape? > > Simon > ________________________________________ > From: gpfsug-discuss-bounces at spectrumscale.org [ > gpfsug-discuss-bounces at spectrumscale.org] on behalf of Oesterlin, Robert [ > Robert.Oesterlin at nuance.com] > Sent: 03 October 2016 16:56 > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? > > What's going be taken away if you use Encryption or Transparent Cloud > Tiering? > > > Bob Oesterlin > Sr Storage Engineer, Nuance HPC Grid > > > From: on behalf of Marc A > Kaplan > Reply-To: gpfsug main discussion list > Date: Monday, October 3, 2016 at 10:46 AM > To: gpfsug main discussion list > Subject: [EXTERNAL] Re: [gpfsug-discuss] Biggest file that will fit inside > an inode? 3968!! > > On a non-SELINUX system the answer is 3968 of data in a 4K inode, just 128 > bytes of metadata. > > Caution: it's possible in some future release, this could change ... I > don't know of any plans, I'm just saying ... > > Inode 16346892 [16346892] snap 0 (index 12 in block 255420): > Inode address: 6:123049056 size 4096 nAddrs 330 > indirectionLevel=INODE status=USERFILE > objectVersion=1 generation=0xC0156CB nlink=1 > owner uid=0 gid=0 mode=0200100644: -rw-r--r-- > blocksize code=5 (32 subblocks) > lastBlockSubblocks=0 > checksum=0xAD8E0B4B is Valid > fileSize=3968 nFullBlocks=0 > currentMetadataReplicas=1 maxMetadataReplicas=2 > currentDataReplicas=1 maxDataReplicas=2 > ... > Data [3968]: > 0000000000000000: BCA91252 2B64BEDC A7D7BA9D D5BE8C30 *...R+d.........0* > ... > 0000000000000F70: DA925E2F 16A68C01 03CA5E37 08D72B7F *..^/......^7..+.* > trailer: is NULL > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Mon Oct 3 17:10:34 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Mon, 3 Oct 2016 16:10:34 +0000 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? In-Reply-To: References: <1BDA1763-A6FE-4A52-B2ED-22324C9AA16F@nuance.com> , Message-ID: TCT doesn't use dmapi though I thought? ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Luke Raimbach [luke.raimbach at googlemail.com] Sent: 03 October 2016 17:07 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? Surely it wouldn't go? Maybe the data would get copied out rather than stubbed... DMAPI can't be stupid enough to stub data out of an inode? Can it? Interesting question. Maybe I'll test that one. On Mon, 3 Oct 2016 at 17:00 Simon Thompson (Research Computing - IT Services) > wrote: Would you tier an in-inode file to the cloud? I mean, I wouldn't tier an in-inode file out to tape? Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Oesterlin, Robert [Robert.Oesterlin at nuance.com] Sent: 03 October 2016 16:56 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? What's going be taken away if you use Encryption or Transparent Cloud Tiering? Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: > on behalf of Marc A Kaplan > Reply-To: gpfsug main discussion list > Date: Monday, October 3, 2016 at 10:46 AM To: gpfsug main discussion list > Subject: [EXTERNAL] Re: [gpfsug-discuss] Biggest file that will fit inside an inode? 3968!! On a non-SELINUX system the answer is 3968 of data in a 4K inode, just 128 bytes of metadata. Caution: it's possible in some future release, this could change ... I don't know of any plans, I'm just saying ... Inode 16346892 [16346892] snap 0 (index 12 in block 255420): Inode address: 6:123049056 size 4096 nAddrs 330 indirectionLevel=INODE status=USERFILE objectVersion=1 generation=0xC0156CB nlink=1 owner uid=0 gid=0 mode=0200100644: -rw-r--r-- blocksize code=5 (32 subblocks) lastBlockSubblocks=0 checksum=0xAD8E0B4B is Valid fileSize=3968 nFullBlocks=0 currentMetadataReplicas=1 maxMetadataReplicas=2 currentDataReplicas=1 maxDataReplicas=2 ... Data [3968]: 0000000000000000: BCA91252 2B64BEDC A7D7BA9D D5BE8C30 *...R+d.........0* ... 0000000000000F70: DA925E2F 16A68C01 03CA5E37 08D72B7F *..^/......^7..+.* trailer: is NULL _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From luke.raimbach at googlemail.com Mon Oct 3 17:16:05 2016 From: luke.raimbach at googlemail.com (Luke Raimbach) Date: Mon, 03 Oct 2016 16:16:05 +0000 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? In-Reply-To: References: <1BDA1763-A6FE-4A52-B2ED-22324C9AA16F@nuance.com> Message-ID: It doesn't, but the end result is the same... data shipped off 'somewhere else' with a stub file? I have in my mind that DMAPI support was around before data-in-inode (or at least 4K inodes) was introduced, so it had to be made a bit cleverer to cope, but I may be misremembering that. On Mon, 3 Oct 2016 at 17:10 Simon Thompson (Research Computing - IT Services) wrote: > > TCT doesn't use dmapi though I thought? > ________________________________________ > From: gpfsug-discuss-bounces at spectrumscale.org [ > gpfsug-discuss-bounces at spectrumscale.org] on behalf of Luke Raimbach [ > luke.raimbach at googlemail.com] > Sent: 03 October 2016 17:07 > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? > > Surely it wouldn't go? Maybe the data would get copied out rather than > stubbed... DMAPI can't be stupid enough to stub data out of an inode? Can > it? Interesting question. > > Maybe I'll test that one. > > On Mon, 3 Oct 2016 at 17:00 Simon Thompson (Research Computing - IT > Services) > wrote: > > Would you tier an in-inode file to the cloud? > > I mean, I wouldn't tier an in-inode file out to tape? > > Simon > ________________________________________ > From: gpfsug-discuss-bounces at spectrumscale.org gpfsug-discuss-bounces at spectrumscale.org> [ > gpfsug-discuss-bounces at spectrumscale.org gpfsug-discuss-bounces at spectrumscale.org>] on behalf of Oesterlin, Robert > [Robert.Oesterlin at nuance.com] > Sent: 03 October 2016 16:56 > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? > > What's going be taken away if you use Encryption or Transparent Cloud > Tiering? > > > Bob Oesterlin > Sr Storage Engineer, Nuance HPC Grid > > > From: gpfsug-discuss-bounces at spectrumscale.org>> on behalf of Marc A Kaplan < > makaplan at us.ibm.com> > Reply-To: gpfsug main discussion list > > Date: Monday, October 3, 2016 at 10:46 AM > To: gpfsug main discussion list gpfsug-discuss at spectrumscale.org>> > Subject: [EXTERNAL] Re: [gpfsug-discuss] Biggest file that will fit inside > an inode? 3968!! > > On a non-SELINUX system the answer is 3968 of data in a 4K inode, just 128 > bytes of metadata. > > Caution: it's possible in some future release, this could change ... I > don't know of any plans, I'm just saying ... > > Inode 16346892 [16346892] snap 0 (index 12 in block 255420): > Inode address: 6:123049056 size 4096 nAddrs 330 > indirectionLevel=INODE status=USERFILE > objectVersion=1 generation=0xC0156CB nlink=1 > owner uid=0 gid=0 mode=0200100644: -rw-r--r-- > blocksize code=5 (32 subblocks) > lastBlockSubblocks=0 > checksum=0xAD8E0B4B is Valid > fileSize=3968 nFullBlocks=0 > currentMetadataReplicas=1 maxMetadataReplicas=2 > currentDataReplicas=1 maxDataReplicas=2 > ... > Data [3968]: > 0000000000000000: BCA91252 2B64BEDC A7D7BA9D D5BE8C30 *...R+d.........0* > ... > 0000000000000F70: DA925E2F 16A68C01 03CA5E37 08D72B7F *..^/......^7..+.* > trailer: is NULL > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Mon Oct 3 17:26:08 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 3 Oct 2016 12:26:08 -0400 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? What it says! In-Reply-To: References: <1BDA1763-A6FE-4A52-B2ED-22324C9AA16F@nuance.com>, Message-ID: I just noticed that tsdbfs shows you how much data fits in the inode, even if you don't fill it... So a few commands will give you the answer: dd, sync, ls, tsdbfs, inode Encryption uses EAs, so you lose some space in inode to that. I don't have an encrypted FS handy for testing, if you have one, go ahead and use tsdbfs to look at an inode and see whether or not encryption pushes the data out of the inode, I wouldn't be surprised either way. [root at n2 isizes]# dd if=/dev/urandom bs=1 count=100 of=i100 100+0 records in 100+0 records out 100 bytes (100 B) copied, 0.00226643 s, 44.1 kB/s [root at n2 isizes]# sync [root at n2 isizes]# ls -ild i100 16346894 -rw-r--r-- 1 root root 100 Oct 3 09:14 i100 [root at n2 isizes]# tsdbfs mak Enter command or null to read next sector. Type ? for help. inode 16346894 Inode 16346894 [16346894] snap 0 (index 14 in block 255420): Inode address: 6:123049072 size 4096 nAddrs 330 indirectionLevel=INODE status=USERFILE objectVersion=1 generation=0x9D45DC1 nlink=1 owner uid=0 gid=0 mode=0200100644: -rw-r--r-- blocksize code=5 (32 subblocks) lastBlockSubblocks=0 checksum=0x2496E008 is Valid fileSize=100 nFullBlocks=0 ... Data [3968]: 0000000000000000: 17699B6F 5F9ACD28 70C05242 9268F44E *.i.o_..(p.RB.h.N* 0000000000000010: 8FFCDCC1 C2ACE0EC 69C1FE2A 986B752C *........i..*.ku,* 0000000000000020: 8E0434E6 1904B7FC 8B0C5709 2243343C *..4.......W."C4<* 0000000000000030: BD317374 FB5C322D 8E257B69 FF573283 *.1st.\2-.%{i.W2.* 0000000000000040: 515B333B EDFAE930 9B6F8712 0814BF65 *Q[3;...0.o.....e* 0000000000000050: DBDCC25E 87EB2C16 77AE672D 45FB6BE3 *...^..,.w.g-E.k.* 0000000000000060: 65D52D17 00000000 00000000 00000000 *e.-.............* 0000000000000070: 00000000 00000000 00000000 00000000 *................* ... trailer: is NULL From: "Simon Thompson (Research Computing - IT Services)" To: gpfsug main discussion list Date: 10/03/2016 12:10 PM Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? Sent by: gpfsug-discuss-bounces at spectrumscale.org TCT doesn't use dmapi though I thought? ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Luke Raimbach [luke.raimbach at googlemail.com] Sent: 03 October 2016 17:07 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? Surely it wouldn't go? Maybe the data would get copied out rather than stubbed... DMAPI can't be stupid enough to stub data out of an inode? Can it? Interesting question. Maybe I'll test that one. On Mon, 3 Oct 2016 at 17:00 Simon Thompson (Research Computing - IT Services) > wrote: Would you tier an in-inode file to the cloud? I mean, I wouldn't tier an in-inode file out to tape? Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org< mailto:gpfsug-discuss-bounces at spectrumscale.org> [gpfsug-discuss-bounces at spectrumscale.org< mailto:gpfsug-discuss-bounces at spectrumscale.org>] on behalf of Oesterlin, Robert [Robert.Oesterlin at nuance.com] Sent: 03 October 2016 16:56 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? What's going be taken away if you use Encryption or Transparent Cloud Tiering? Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: > on behalf of Marc A Kaplan > Reply-To: gpfsug main discussion list > Date: Monday, October 3, 2016 at 10:46 AM To: gpfsug main discussion list > Subject: [EXTERNAL] Re: [gpfsug-discuss] Biggest file that will fit inside an inode? 3968!! On a non-SELINUX system the answer is 3968 of data in a 4K inode, just 128 bytes of metadata. Caution: it's possible in some future release, this could change ... I don't know of any plans, I'm just saying ... Inode 16346892 [16346892] snap 0 (index 12 in block 255420): Inode address: 6:123049056 size 4096 nAddrs 330 indirectionLevel=INODE status=USERFILE objectVersion=1 generation=0xC0156CB nlink=1 owner uid=0 gid=0 mode=0200100644: -rw-r--r-- blocksize code=5 (32 subblocks) lastBlockSubblocks=0 checksum=0xAD8E0B4B is Valid fileSize=3968 nFullBlocks=0 currentMetadataReplicas=1 maxMetadataReplicas=2 currentDataReplicas=1 maxDataReplicas=2 ... Data [3968]: 0000000000000000: BCA91252 2B64BEDC A7D7BA9D D5BE8C30 *...R+d.........0* ... 0000000000000F70: DA925E2F 16A68C01 03CA5E37 08D72B7F *..^/......^7..+.* trailer: is NULL _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan at buzzard.me.uk Mon Oct 3 17:34:33 2016 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Mon, 03 Oct 2016 17:34:33 +0100 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? In-Reply-To: References: <1BDA1763-A6FE-4A52-B2ED-22324C9AA16F@nuance.com> Message-ID: <1475512473.3748.14.camel@buzzard.phy.strath.ac.uk> On Mon, 2016-10-03 at 16:16 +0000, Luke Raimbach wrote: > It doesn't, but the end result is the same... data shipped off > 'somewhere else' with a stub file? > > I have in my mind that DMAPI support was around before data-in-inode > (or at least 4K inodes) was introduced, so it had to be made a bit > cleverer to cope, but I may be misremembering that. > It did but you can always specify that files below a certain size don't get migrated, be it tape cloud or elsewhere. Never made sense migrating small files to tape anyway, and it certainly makes no sense to migrate a file in an inode anywhere. Perhaps that is the source of a request for enhancement? Stop files existing in inodes being migrated. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From luis.bolinches at fi.ibm.com Mon Oct 3 18:33:04 2016 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Mon, 3 Oct 2016 17:33:04 +0000 Subject: [gpfsug-discuss] WebDAV protocol access support In-Reply-To: Message-ID: Hi You might want to take a look to https://owncloud.com/ibm/ -- Cheers > On 3 Oct 2016, at 18.53, Aslam Ghumra (IT Services, Facilities Management) wrote: > > Does anyone know whether WebDAV protocol access support will ( if ever ) be implemented within GPFS ? > > Regards, > > Aslam Ghumra > Research Data Management > ____________________________ > IT Services > Elms Road Data Centre > Building G5 > Edgbaston > Birmingham B15 2TT > T: 0121 414 5877 > F; 0121 414 3952 > Skype : JanitorX > Twitter : @aslamghumra > in : https://uk.linkedin.com/in/aslam-ghumra-13907993 > http://intranet.bham.ac.uk/bear > Ellei edell?? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.kidger at uk.ibm.com Tue Oct 4 11:43:45 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Tue, 4 Oct 2016 10:43:45 +0000 Subject: [gpfsug-discuss] File_heat for GPFS File Systems In-Reply-To: References: , <8c7fd395-1efc-7197-4a98-763ba784cafd@stanford.edu> Message-ID: An HTML attachment was scrubbed... URL: From alandhae at gmx.de Tue Oct 4 12:25:52 2016 From: alandhae at gmx.de (=?ISO-8859-15?Q?Andreas_Landh=E4u=DFer?=) Date: Tue, 4 Oct 2016 13:25:52 +0200 (CEST) Subject: [gpfsug-discuss] File_heat for GPFS File Systems In-Reply-To: References: , <8c7fd395-1efc-7197-4a98-763ba784cafd@stanford.edu> Message-ID: On Tue, 4 Oct 2016, Daniel Kidger wrote: Daniel, ahh this might also help in my HPC Environment. We are already using AFM, just on HDD, for distributing HPC Data for usage on remote scientific sites. Using the AFM for a temp space seems to be a good idea, will the individual file heat be increased by using ILM policy and AFM? Using a large GPFS Filesystem for scientific results, it seems to be very unlikely all users starting using their cold files at the same point of time, so filesystem space should be sufficient. Thanks for the hint Andreas T-Systems SfR GmbH > If you need functionality like you describe, then as well as using tiering > with the ILM Policy Engine, also consider using AFM to cache onto say SSDs.? > That way you would have dynamic caching: as soon as a file gets accessed, it > goes into the cache and further I/O will be do the SSD copy so faster until > such time as the file ages and newer files replace it in the AFM cache. The > cost of this is perhaps a little more complexity and also a 'hot' file > consumes space on both SSD and disk. > ? > Daniel > /spectrum_storage-banne > ?????????????????????????????? v > ? > Spectrum Scale Logo > ? > ? > Dr Daniel Kidger > IBM?Technical Sales Specialist > Software Defined Solution Sales > > +44-(0)7818 522 266 > daniel.kidger at uk.ibm.com > ? > ? > ? > ----- Original message ----- > From: Eric Horst > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug main discussion list > > Cc: > Subject: Re: [gpfsug-discuss] File_heat for GPFS File Systems > Date: Tue, Sep 27, 2016 9:56 PM > ? >> > >> if a file gets hot again, there is no rule for putting the > file back > >> into a faster storage device? > > > > > > The file will get moved when you run the policy again. ?You > can run the > > policy as often as you like. > > I think its worth stating clearly that if a file is in the > Thrifty > slow pool and a user opens and reads/writes the file there is > nothing > that moves this file to a different tier. A policy run is the > only > action that relocates files. So if you apply the policy daily > and over > the course of the day users access many cold files, the > performance > accessing those cold files may not be ideal until the next day > when > they are repacked by heat. A file is not automatically moved to > the > fast tier on access read or write. I mention this because this > aspect > of tiering was not immediately clear from the docs when I was a > neophyte GPFS admin and I had to learn by observation. It is > easy for > one to make an assumption that it is a more dynamic tiering > system > than it is. > > -Eric > > -- > Eric Horst > University of Washington > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > ? > > ? > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > -- Andreas Landh?u?er +49 151 12133027 (mobile) alandhae at gmx.de From alandhae at gmx.de Tue Oct 4 13:54:16 2016 From: alandhae at gmx.de (=?ISO-8859-15?Q?Andreas_Landh=E4u=DFer?=) Date: Tue, 4 Oct 2016 14:54:16 +0200 (CEST) Subject: [gpfsug-discuss] Fileheat reporting Message-ID: Customer needs a report of filenames being Hot, warm and cold. As far as I'm understanding fileheat can be any value larger than zero. A value of zero or almost zero equals to COLD How am I classifying warm and hot? I'm needing a standardized value between 0 and 1 for fileheat. Is it possible creating such a number when when activating fileheat and creating reports and finally using ILM policies according to this classification? Best regards Andreas -- Andreas Landh?u?er +49 151 12133027 (mobile) alandhae at gmx.de From eric.wonderley at vt.edu Tue Oct 4 14:39:02 2016 From: eric.wonderley at vt.edu (J. Eric Wonderley) Date: Tue, 4 Oct 2016 09:39:02 -0400 Subject: [gpfsug-discuss] migrate policy vs restripe Message-ID: We have the need to move data from one set of spindles to another. Are there any performance or availability considerations when choosing to do either a migration policy or a restripe to make this move? I did discover that a restripe only works within the same pool...even though you setup two pools in different failure groups. -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Tue Oct 4 15:32:55 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 4 Oct 2016 10:32:55 -0400 Subject: [gpfsug-discuss] migrate policy vs restripe In-Reply-To: References: Message-ID: Doubtless there are subtleties and special cases, but my should-work-well advice is: A) Use mmapplypolicy ... -I defer -P policy-to-set-pool-assignments-and-replication-factor ... where the policy rules file contains rules like RULE 'adjust' MIGRATE [FROM POOL 'x'] TO POOL 'y' REPLICATE({1|2}) WHERE decide to which file this rule shall apply The '-I defer' option says just mark the inodes, do not actually move the data B) Use mmrestripefs ... -b ... -N nodeclass The move data to correct pool assignments and rebalance the entire file system, also fix ill-replicated and ill-placed files. (keep reading...) C) "restripe only works within the same pool" -- I believe there is a misunderstanding. For any given file, all the data blocks are supposed to be in the one pool indicated by pool assignment which is recorded in the file's inode. If the pool assignment changes but not all of the blocks are migrated to the new pool, then the file is considered "ill-placed". I believe mmrestripefs endeavors to move blocks so as to "fix" ill-placed files. Failure groups are a different concept that is associated with replication. The replicas of a given data or meta data block should be in different failure groups. If not, the file is considered "ill-replicated" and mmrestripefs endeavors to fix that also. From: "J. Eric Wonderley" To: gpfsug main discussion list Date: 10/04/2016 09:39 AM Subject: [gpfsug-discuss] migrate policy vs restripe Sent by: gpfsug-discuss-bounces at spectrumscale.org We have the need to move data from one set of spindles to another. Are there any performance or availability considerations when choosing to do either a migration policy or a restripe to make this move? I did discover that a restripe only works within the same pool...even though you setup two pools in different failure groups. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From aspalazz at us.ibm.com Tue Oct 4 15:44:33 2016 From: aspalazz at us.ibm.com (Aaron S Palazzolo) Date: Tue, 4 Oct 2016 14:44:33 +0000 Subject: [gpfsug-discuss] Toolkit Message-ID: An HTML attachment was scrubbed... URL: From stef.coene at docum.org Tue Oct 4 15:53:48 2016 From: stef.coene at docum.org (Stef Coene) Date: Tue, 4 Oct 2016 16:53:48 +0200 Subject: [gpfsug-discuss] Toolkit In-Reply-To: References: Message-ID: <6192009e-afea-36f1-d79a-9ec78194fe6d@docum.org> On 10/04/2016 04:44 PM, Aaron S Palazzolo wrote: > *Hi Stef,* > > Thanks for your Install Toolkit feature request: > -------------------- > /Is it possible to recreate the clusterdefinition.txt based on the > current configuration?/ > -------------------- > > A couple questions if you don't mind: > *1) *Are you using the Install Toolkit for base gpfs installation or > solely protocol deployment? I used the spectrum toolkit for basic setup. Then I used some mm commands to add disks and change other stuff. But the toolkit is unaware of theses changes. > *2) *If using it for base gpfs installation, do you create both NSDs and > file systems with it or do you do this manually? Like said, I used the toolkit for the first few NSDs and later used mm commands for other NSDs. I have no problems using the mm commands, but I don't know what will happen if I want to use the toolkit to create proto nodes. Stef From makaplan at us.ibm.com Tue Oct 4 15:56:46 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 4 Oct 2016 10:56:46 -0400 Subject: [gpfsug-discuss] Fileheat reporting In-Reply-To: References: Message-ID: FILE_HEAT value is a non-negative floating point value. The value can grow quite large -- in theory it can grow to the IEEE 64 bit maximum floating point value and maybe even to "infinity". Each time a file is read completely from disk it's FILE_HEAT value increases by 1. If a fraction, f, of the file is read then FILE_HEAT += f. (f<=1) On the other hand, as time passes the FILE_HEAT "decays" at the rate given by the (mmchconfig) parameters fileHeatPeriodMinutes and fileHeatLossPercent. In fact I recently corresponding with a customer (who is no stranger to this forum and may wish to comment!) who found that a bunch of hot little script files had accumulated heat values greater than 100,000,000 ! We were both surprised. Apparently these files are read and re-read many times every day by many nodes and so their heat builds up! So what would you call "Hot", "Warm", "Cold"? It depends... To get a good idea, I suggest that a "survey" of FILE_HEAT values be done by using an mmapplypolicy command with a rule like. rule 'y' list 'fhn0' weight(FILE_HEAT) SHOW(HEX(XATTR('gpfs.FileHeat')) || ' A=' || varchar(ACCESS_TIME) || ' K=' || varchar(KB_ALLOCATED) || ' H=' || varchar(FILE_HEAT)) where FILE_HEAT != 0.0 Using `mmapplypolicy FS -P policy-rules-file -I defer -f /tmp/whatever [other options such as... -N nodeclass -g /shared-temp ... ] This will produce a file list sorted by FILE_HEAT, which you can then peruse or otherwise process.... Perhaps a histogram or other display of (number of files) vs (heat values) would be interesting... From: Andreas Landh?u?er To: gpfsug-discuss at spectrumscale.org Date: 10/04/2016 08:54 AM Subject: [gpfsug-discuss] Fileheat reporting Sent by: gpfsug-discuss-bounces at spectrumscale.org Customer needs a report of filenames being Hot, warm and cold. As far as I'm understanding fileheat can be any value larger than zero. A value of zero or almost zero equals to COLD How am I classifying warm and hot? I'm needing a standardized value between 0 and 1 for fileheat. Is it possible creating such a number when when activating fileheat and creating reports and finally using ILM policies according to this classification? Best regards Andreas -- Andreas Landh?u?er +49 151 12133027 (mobile) alandhae at gmx.de_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From kronberg at nsc.liu.se Tue Oct 4 16:43:51 2016 From: kronberg at nsc.liu.se (Mats Kronberg) Date: Tue, 4 Oct 2016 17:43:51 +0200 Subject: [gpfsug-discuss] GSS 3.0 (GPFS 4.2.1) upgrade experiences? Message-ID: Hi, We are in the planning phase for upgrading our GSS system from GSS 2.0.7 (GPFS 4.1.0-8) to something more recent. Lenovo recommends going directly to the recently released GSS 3.0 (GPFS 4.2.1-0, released late September 2016). I'm inclined to agree with them, but the conservative, let-others-betatest-stuff part of me is suggesting something a little more mature like GSS 2.5.9 (GPFS 4.1.1-2, released late-2015). Has anyone got any success or horror stories to share regarding either of these releases? :) -- Mats Kronberg, Systems Expert NSC, National Supercomputer Centre -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Tue Oct 4 16:48:31 2016 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Tue, 4 Oct 2016 15:48:31 +0000 Subject: [gpfsug-discuss] GSS 3.0 (GPFS 4.2.1) upgrade experiences? In-Reply-To: References: Message-ID: We?re not huge, and not running GSS, but we are running CES for 3000+ concurrent users. We went from 3.5 to 4.2.1 via 4.1.1 and our experience was outstanding. We had some enforced downtime during the upgrad and a few hiccups with CES specifically, but overall very happy. That may help you in some shape or form ? From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Mats Kronberg Sent: 04 October 2016 16:44 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] GSS 3.0 (GPFS 4.2.1) upgrade experiences? Hi, We are in the planning phase for upgrading our GSS system from GSS 2.0.7 (GPFS 4.1.0-8) to something more recent. Lenovo recommends going directly to the recently released GSS 3.0 (GPFS 4.2.1-0, released late September 2016). I'm inclined to agree with them, but the conservative, let-others-betatest-stuff part of me is suggesting something a little more mature like GSS 2.5.9 (GPFS 4.1.1-2, released late-2015). Has anyone got any success or horror stories to share regarding either of these releases? :) -- Mats Kronberg, Systems Expert NSC, National Supercomputer Centre -------------- next part -------------- An HTML attachment was scrubbed... URL: From mimarsh2 at vt.edu Tue Oct 4 18:04:59 2016 From: mimarsh2 at vt.edu (Brian Marshall) Date: Tue, 4 Oct 2016 13:04:59 -0400 Subject: [gpfsug-discuss] testing commands Message-ID: All, Is there a way to "test" GPFS commands and see what the output or result would be before running them? For example, I'd like to verify a command string before actually running it on a production system. Does IBM offer "test" licenses for setting up a small debug/devel environment? I'd be interested to hear everyone's best practices on verifying commands/actions. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Tue Oct 4 18:30:03 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 4 Oct 2016 13:30:03 -0400 Subject: [gpfsug-discuss] testing commands In-Reply-To: References: Message-ID: Simple functional testing can be done on a small system with as little as one Linux node and one disk. Personally, I do a lot of testing on a several years old laptop running Linux. My test gpfs file system doesn't even use a real disk partition! Just some data files I initialized with dd if=/dev/zero ... (This is not officially supported, blah, blah, blah...) [root at bog-wifi cmvc]# mmlsnsd -X Disk name NSD volume ID Device Devtype Node name Remarks --------------------------------------------------------------------------------------------------- y1 C0A8015D57586A5E /vds/y1 file bog-wifi.lan.makaplan.us server node y2 C0A8015D57586A5F /vds/y2 file bog-wifi.lan.makaplan.us server node y3 C0A8015D575EF0A3 /vds/y3 file bog-wifi.lan.makaplan.us server node [root at bog-wifi cmvc]# ls -ld /vds/y? -rw-r--r--. 1 root root 536870912 Oct 4 12:16 /vds/y1 -rw-r--r--. 1 root root 536870912 Oct 3 10:39 /vds/y2 -rw-r--r--. 1 root root 801000000 Sep 29 18:54 /vds/y3 [root at bog-wifi cmvc]# df -T /vds Filesystem Type 1K-blocks Used Available Use% Mounted on /dev/mapper/rhelw_bog--wifi-root ext4 183862728 42118120 132381768 25% / -------------- next part -------------- An HTML attachment was scrubbed... URL: From mimarsh2 at vt.edu Tue Oct 4 21:13:05 2016 From: mimarsh2 at vt.edu (Brian Marshall) Date: Tue, 4 Oct 2016 16:13:05 -0400 Subject: [gpfsug-discuss] testing commands In-Reply-To: References: Message-ID: Do I require additional NSD Server licenses to do this? i.e. What triggers the need to purchase a license for a NSD Server? Brian On Tue, Oct 4, 2016 at 1:30 PM, Marc A Kaplan wrote: > Simple functional testing can be done on a small system with as little as > one Linux node and one disk. > > Personally, I do a lot of testing on a several years old laptop running > Linux. My test gpfs file system doesn't even use a real disk partition! > Just some data files > I initialized with dd if=/dev/zero ... > > (This is not officially supported, blah, blah, blah...) > > [root at bog-wifi cmvc]# mmlsnsd -X > > Disk name NSD volume ID Device Devtype Node name > Remarks > ------------------------------------------------------------ > --------------------------------------- > y1 C0A8015D57586A5E /vds/y1 file > bog-wifi.lan.makaplan.us server node > y2 C0A8015D57586A5F /vds/y2 file > bog-wifi.lan.makaplan.us server node > y3 C0A8015D575EF0A3 /vds/y3 file > bog-wifi.lan.makaplan.us server node > > [root at bog-wifi cmvc]# ls -ld /vds/y? > -rw-r--r--. 1 root root 536870912 Oct 4 12:16 /vds/y1 > -rw-r--r--. 1 root root 536870912 Oct 3 10:39 /vds/y2 > -rw-r--r--. 1 root root 801000000 Sep 29 18:54 /vds/y3 > > [root at bog-wifi cmvc]# df -T /vds > Filesystem Type 1K-blocks Used Available Use% > Mounted on > /dev/mapper/rhelw_bog--wifi-root ext4 183862728 42118120 132381768 25% / > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From swinglet at us.ibm.com Tue Oct 4 21:24:43 2016 From: swinglet at us.ibm.com (Teresa S Swingler) Date: Tue, 4 Oct 2016 20:24:43 +0000 Subject: [gpfsug-discuss] Survey - IBM Spectrum Scale Everyday Management Tasks Message-ID: An HTML attachment was scrubbed... URL: From jtucker at pixitmedia.com Tue Oct 4 21:37:42 2016 From: jtucker at pixitmedia.com (Jez Tucker) Date: Tue, 4 Oct 2016 21:37:42 +0100 Subject: [gpfsug-discuss] testing commands In-Reply-To: References: Message-ID: Hi So we were chatting a while ago about dry-run requirements, but more with respect to mmapplypolicy. For relevance to those on the list who run our Python API, you'll find dry-run functionality here: http://arcapix.com/gpfsapi/execute.html#dry-run Cheers, Jez On 04/10/16 18:04, Brian Marshall wrote: > All, > > Is there a way to "test" GPFS commands and see what the output or > result would be before running them? For example, I'd like to verify > a command string before actually running it on a production system. > > Does IBM offer "test" licenses for setting up a small debug/devel > environment? > > I'd be interested to hear everyone's best practices on verifying > commands/actions. > > > Thanks, > Brian > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Jez Tucker Head of Research & Product Development Pixit Media | ArcaStream www.pixitmedia.com www.arcastream.com -- This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From swinglet at us.ibm.com Tue Oct 4 21:37:39 2016 From: swinglet at us.ibm.com (Teresa S Swingler) Date: Tue, 4 Oct 2016 13:37:39 -0700 Subject: [gpfsug-discuss] Survey - Spectrum Scale Everyday Management Message-ID: Hi, We in IBM Spectrum Scale design and development are interested in learning about the frequency and usability of your everyday management tasks. Your input will help us with our upcoming priorities. Please take 5 minutes to complete this brief survey: https://www.surveygizmo.com/s3/3090036/IBM-Spectrum-Scale-Everyday-Management Thank you, Teresa Swingler Teresa S. Swingler IBM Systems Senior Designer | Lead UX Architect Spectrum Scale and Spectrum Control 2929 N. Central Ave. Phoenix, AZ 85012 Phone: (602) 217-2763 Tieline: 667-2763 E-mail: swinglet at us.ibm.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Tue Oct 4 22:49:01 2016 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 4 Oct 2016 17:49:01 -0400 Subject: [gpfsug-discuss] testing commands In-Reply-To: References: Message-ID: <797ae533-edee-102a-3f3e-f53687cfcf83@nasa.gov> Not to get off-topic here, but WHOA! I had no idea somebody had created a python API for GPFS. How does one get access to said API? On 10/4/16 4:37 PM, Jez Tucker wrote: > Hi > > So we were chatting a while ago about dry-run requirements, but more > with respect to mmapplypolicy. > For relevance to those on the list who run our Python API, you'll find > dry-run functionality here: > > http://arcapix.com/gpfsapi/execute.html#dry-run > > Cheers, > > Jez > > > > On 04/10/16 18:04, Brian Marshall wrote: >> All, >> >> Is there a way to "test" GPFS commands and see what the output or >> result would be before running them? For example, I'd like to verify >> a command string before actually running it on a production system. >> >> Does IBM offer "test" licenses for setting up a small debug/devel >> environment? >> >> I'd be interested to hear everyone's best practices on verifying >> commands/actions. >> >> >> Thanks, >> Brian >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- > Jez Tucker > Head of Research & Product Development > Pixit Media | ArcaStream > www.pixitmedia.com > www.arcastream.com > > > This email is confidential in that it is intended for the exclusive > attention of the addressee(s) indicated. If you are not the intended > recipient, this email should not be read or disclosed to any other > person. Please notify the sender immediately and delete this email from > your computer system. Any opinions expressed are not necessarily those > of the company from which this email was sent and, whilst to the best of > our knowledge no viruses or defects exist, no responsibility can be > accepted for any loss or damage arising from its receipt or subsequent > use of this email. > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From makaplan at us.ibm.com Tue Oct 4 23:15:32 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 4 Oct 2016 18:15:32 -0400 Subject: [gpfsug-discuss] testing commands - licensing In-Reply-To: References: Message-ID: http://www.ibm.com/developerworks/servicemanagement/sdi/spectrum-scale/resources.html ... Please contact your IBM sales representative for more information. ... http://www.ibm.com/developerworks/servicemanagement/tc/gpfs/evaluate.html https://www-01.ibm.com/marketing/iwm/iwm/web/preLogin.do?source=swerpzsw-scale42-3 The 90 day trial license includes these terms... If the Program is designated as "Non-Production", the Program can only be deployed as part of the Licensee's internal development and test environment for internal non-production activities, including but not limited to testing, performance tuning, fault diagnosis, internal benchmarking, staging, quality assurance activity and/or developing internally used additions or extensions to the Program using published application programming interfaces. Licensee is not authorized to use any part of the Program for any other purposes without acquiring the appropriate production entitlements. ... This is posting does not constitute or include the full terms of a license ... It just provides some pointers and lets you know that IBM does offer "Trial" licenses for Spectrum Scale. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aspalazz at us.ibm.com Wed Oct 5 02:22:35 2016 From: aspalazz at us.ibm.com (Aaron S Palazzolo) Date: Wed, 5 Oct 2016 01:22:35 +0000 Subject: [gpfsug-discuss] Toolkit Message-ID: An HTML attachment was scrubbed... URL: From stef.coene at docum.org Wed Oct 5 07:30:21 2016 From: stef.coene at docum.org (Stef Coene) Date: Wed, 5 Oct 2016 08:30:21 +0200 Subject: [gpfsug-discuss] Toolkit In-Reply-To: References: Message-ID: On 10/05/2016 03:22 AM, Aaron S Palazzolo wrote: > *Hi Stef,* > > Good info and thanks for the scenario specifics. I will include your > input in our upcoming design sessions and we can let you know as soon as > any functional improvements are on their way. I just realized that this also a good way to document a GPFS cluster ;) Stef From daniel.kidger at uk.ibm.com Wed Oct 5 14:27:39 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Wed, 5 Oct 2016 13:27:39 +0000 Subject: [gpfsug-discuss] testing commands In-Reply-To: References: , Message-ID: An HTML attachment was scrubbed... URL: From Richard.E.Powell at boeing.com Thu Oct 6 22:32:31 2016 From: Richard.E.Powell at boeing.com (Powell, Richard E) Date: Thu, 6 Oct 2016 21:32:31 +0000 Subject: [gpfsug-discuss] Will there be a UG meeting at SC16? Message-ID: Has there been any word on a Spectrum Scale User Group meeting in conjunction with SC16 in November? -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Thu Oct 6 22:36:18 2016 From: bbanister at jumptrading.com (Bryan Banister) Date: Thu, 6 Oct 2016 21:36:18 +0000 Subject: [gpfsug-discuss] Will there be a UG meeting at SC16? In-Reply-To: References: Message-ID: <21BC488F0AEA2245B2C3E83FC0B33DBB0634F863@CHI-EXCHANGEW1.w2k.jumptrading.com> Word on the street is that the SS UG meeting will be on Sunday, Nov 13th, in the afternoon, -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Powell, Richard E Sent: Thursday, October 06, 2016 4:33 PM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Will there be a UG meeting at SC16? Has there been any word on a Spectrum Scale User Group meeting in conjunction with SC16 in November? ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From swinglet at us.ibm.com Thu Oct 6 22:38:13 2016 From: swinglet at us.ibm.com (Teresa S Swingler) Date: Thu, 6 Oct 2016 14:38:13 -0700 Subject: [gpfsug-discuss] Thank you and Last Call - Everyday Management 5 minute Survey Message-ID: Thank you to all of you who have responded to our short survey of everyday management tasks. We value your feedback as it helps to better inform our upcoming priorities. For those of you who have not had a chance to respond yet, please take 5 minutes to fill out this quick survey: https://www.surveygizmo.com/s3/3090036/IBM-Spectrum-Scale-Everyday-Management The survey will be closed on Sunday. We appreciate your input and thank you for your time. Teresa Teresa S. Swingler IBM Systems Senior Designer | Lead UX Architect Spectrum Scale and Spectrum Control 2929 N. Central Ave. Phoenix, AZ 85012 Phone: (602) 217-2763 Tieline: 667-2763 E-mail: swinglet at us.ibm.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Thu Oct 6 22:40:33 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Thu, 6 Oct 2016 21:40:33 +0000 Subject: [gpfsug-discuss] Will there be a UG meeting at SC16? In-Reply-To: <21BC488F0AEA2245B2C3E83FC0B33DBB0634F863@CHI-EXCHANGEW1.w2k.jumptrading.com> References: , <21BC488F0AEA2245B2C3E83FC0B33DBB0634F863@CHI-EXCHANGEW1.w2k.jumptrading.com> Message-ID: That's correct Sunday (13th?). Agenda is just being finished up at the moment. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Bryan Banister [bbanister at jumptrading.com] Sent: 06 October 2016 22:36 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Will there be a UG meeting at SC16? Word on the street is that the SS UG meeting will be on Sunday, Nov 13th, in the afternoon, -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Powell, Richard E Sent: Thursday, October 06, 2016 4:33 PM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Will there be a UG meeting at SC16? Has there been any word on a Spectrum Scale User Group meeting in conjunction with SC16 in November? ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. From kraemerf at de.ibm.com Fri Oct 7 13:08:19 2016 From: kraemerf at de.ibm.com (Frank Kraemer) Date: Fri, 7 Oct 2016 14:08:19 +0200 Subject: [gpfsug-discuss] Spectrum Scale for Linux on z Systems now supports native Spectrum Scale encryption Message-ID: >Peter Muench1/Germany/IBM wrote on 07.10.2016 12:11:52: Since September 20th, Spectrum Scale for Linux on z Systems 4.2.1.1 code is available. (Download is via fixcentral) Spectrum Scale for Linux on z Systems now supports native GPFS encryption: >From the latest FAQ (see Q2.28: What are the current requirements / limitations for IBM Spectrum Scale for Linux on z Systems?) If you have further questions, please let me know. Thank you. Peter Muench Spectrum Scale for Linux on z Systems Development mailto:muenchp at de.ibm.com IBM Deutschland Research & Development GmbH -------------- next part -------------- An HTML attachment was scrubbed... URL: From volobuev at us.ibm.com Fri Oct 7 17:20:01 2016 From: volobuev at us.ibm.com (Yuri L Volobuev) Date: Fri, 7 Oct 2016 09:20:01 -0700 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? In-Reply-To: References: <1BDA1763-A6FE-4A52-B2ED-22324C9AA16F@nuance.com> Message-ID: Exactly how much data can fit into an inode depends on whether any extended attributes (EAs) are present, which makes this complicated, as EAs may be set explicitly by the user or implicitly by the system. In any inode, there's a fixed footprint of the inode header, 128 bytes on recent GPFS streams. So if there's no possibility to fit more than (inodeSize - 128) bytes into an inode. If some EAs are present, this becomes more complicated. GPFS can store EAs either in the inode, or in a special "EA overflow block". Certain EAs used by GPFS internally (for encryption, AFM, DMAPI, clones) must reside in the inode, due to the need to have those set at certain junctures when it's tricky to accommodate EA overflow allocation with proper atomicity guarantees. Other subsystems, e.g. SELinux, may implicitly use EAs as well. So a given inode may have less space for data than described above. This is another big part of the reason why 4K is the current default inode size. And of course this is not an external spec set in stone, but rather the way the current implementation works. A future version of GPFS may have a bigger inode header, for example. So it would be imprudent to bank on file/directory data fitting into an inode with a great deal of precision. A better way to view this is as a kind of a performance optimization. DMAPI won't "punch out" data that resides in inode. Such a file has zero blocks allocated, so there's nothing for an HSM app to migrate. TCT can copy in-inode data to cloud for co-resident mode. And yes, TCT doesn't rely on the same DMAPI API as HSM, per se, but naturally the underlying code shares some of the infrastructure. yuri From: Luke Raimbach To: gpfsug main discussion list , Date: 10/03/2016 09:16 AM Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? Sent by: gpfsug-discuss-bounces at spectrumscale.org It doesn't, but the end result is the same... data shipped off 'somewhere else' with a stub file? I have in my mind that DMAPI support was around before data-in-inode (or at least 4K inodes) was introduced, so it had to be made a bit cleverer to cope, but I may be misremembering that. On Mon, 3 Oct 2016 at 17:10 Simon Thompson (Research Computing - IT Services) wrote: TCT doesn't use dmapi though I thought? ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [ gpfsug-discuss-bounces at spectrumscale.org] on behalf of Luke Raimbach [ luke.raimbach at googlemail.com] Sent: 03 October 2016 17:07 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? Surely it wouldn't go? Maybe the data would get copied out rather than stubbed... DMAPI can't be stupid enough to stub data out of an inode? Can it? Interesting question. Maybe I'll test that one. On Mon, 3 Oct 2016 at 17:00 Simon Thompson (Research Computing - IT Services) > wrote: Would you tier an in-inode file to the cloud? I mean, I wouldn't tier an in-inode file out to tape? Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [ gpfsug-discuss-bounces at spectrumscale.org] on behalf of Oesterlin, Robert [Robert.Oesterlin at nuance.com] Sent: 03 October 2016 16:56 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? What's going be taken away if you use Encryption or Transparent Cloud Tiering? Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: > on behalf of Marc A Kaplan < makaplan at us.ibm.com> Reply-To: gpfsug main discussion list > Date: Monday, October 3, 2016 at 10:46 AM To: gpfsug main discussion list > Subject: [EXTERNAL] Re: [gpfsug-discuss] Biggest file that will fit inside an inode? 3968!! On a non-SELINUX system the answer is 3968 of data in a 4K inode, just 128 bytes of metadata. Caution: it's possible in some future release, this could change ... I don't know of any plans, I'm just saying ... Inode 16346892 [16346892] snap 0 (index 12 in block 255420): ? Inode address: 6:123049056 size 4096 nAddrs 330 ? indirectionLevel=INODE status=USERFILE ? objectVersion=1 generation=0xC0156CB nlink=1 ? owner uid=0 gid=0 mode=0200100644: -rw-r--r-- ? blocksize code=5 (32 subblocks) ? lastBlockSubblocks=0 ? checksum=0xAD8E0B4B is Valid ? fileSize=3968 nFullBlocks=0 ? currentMetadataReplicas=1 maxMetadataReplicas=2 ? currentDataReplicas=1 maxDataReplicas=2 ? ... ? Data [3968]: 0000000000000000: BCA91252 2B64BEDC A7D7BA9D D5BE8C30? *...R+d.........0* ... 0000000000000F70: DA925E2F 16A68C01 03CA5E37 08D72B7F? *..^/......^7..+.* ? trailer: is NULL _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From Richard.E.Powell at boeing.com Sat Oct 8 00:55:28 2016 From: Richard.E.Powell at boeing.com (Powell, Richard E) Date: Fri, 7 Oct 2016 23:55:28 +0000 Subject: [gpfsug-discuss] Will there be a UG meeting at SC16? Message-ID: Thanks for the info! Can you tell us when and where, yet? We're trying to make our travel plans early, (for a change:)) -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Sat Oct 8 22:57:56 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Sat, 8 Oct 2016 21:57:56 +0000 Subject: [gpfsug-discuss] Will there be a UG meeting at SC16? Message-ID: <82021AF7-4908-4D7B-B3B9-557954A692C8@nuance.com> Sunday November 13th, from about 1PM - 5:30 PM. Not sure of the location, but it will be at one of the hotels near the convention center in downtown SLC. More details soon! Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid GPFS-UG Co-Principal From: on behalf of "Powell, Richard E" Reply-To: gpfsug main discussion list Date: Friday, October 7, 2016 at 6:55 PM To: "gpfsug-discuss at spectrumscale.org" Subject: [EXTERNAL] Re: [gpfsug-discuss] Will there be a UG meeting at SC16? Thanks for the info! Can you tell us when and where, yet? We?re trying to make our travel plans early, (for a change?) -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.childs at qmul.ac.uk Mon Oct 10 13:32:26 2016 From: p.childs at qmul.ac.uk (Peter Childs) Date: Mon, 10 Oct 2016 12:32:26 +0000 Subject: [gpfsug-discuss] GPFS Upgrade 3.5 -> 4.1 Message-ID: We are finishing upgrading our GPFS cluster of around 250 (client) nodes from GPFS 3.5.0.31 to Spectrum Scale 4.1.1.8, and have just about upgraded all the computers. We are looking at running the "mmchfs -V LATEST" step and where wondering how much io this takes and if it was likely to interrupt service? We are looking at upgrading to 4.2 but plan to do that via Multi-cluster and AFM as we are integrating new hardware and wish to increase the block and inode size at the same time. Peter Childs Research Storage Expert ITS Research Infrastructure Queen Mary, University of London From robert at strubi.ox.ac.uk Mon Oct 10 14:21:30 2016 From: robert at strubi.ox.ac.uk (Robert Esnouf) Date: Mon, 10 Oct 2016 14:21:30 +0100 (BST) Subject: [gpfsug-discuss] NEW DEPARTMENT/NEW POST: Head of Research Computing at the Oxford Big Data Institute Message-ID: <201610101321.061282@mail.strubi.ox.ac.uk> Dear All, PLEASE FEEL FREE TO PASS THIS EMAIL ON TO OTHER RELEVANT MAILING LISTS AND ANYONE WHO MIGHT BE INTERESTED The University of Oxford Big Data Institute is looking for a highly skilled research computing manager to establish a significant computing capability in Oxford over the next few years. The job advertisement can be found here: http://www.jobs.ac.uk/job/AUT085/head-of-bdi-research-computing/ The post is not just traditional HPC as it involves creating a large infrastructure for distributed, collaborative processing of sensitive data. If anyone wants an informal chat about how this role could develop then please drop me an email and I'll try to help. Regards, Robert -- Dr. Robert Esnouf, University Research Lecturer, Head of Research Computing Core, NDM Research Computing Strategy Officer Room 10/028, Wellcome Trust Centre for Human Genetics, Old Road Campus, Roosevelt Drive, Oxford OX3 7BN, UK Email: robert at strubi.ox.ac.uk / robert at well.ox.ac.uk Tel: (+44) - 1865 - 287783 From janfrode at tanso.net Mon Oct 10 15:35:02 2016 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Mon, 10 Oct 2016 14:35:02 +0000 Subject: [gpfsug-discuss] GPFS Upgrade 3.5 -> 4.1 In-Reply-To: References: Message-ID: I've also always been worried about that one, but never experienced it taking any time, I/O or interruption. I've the interpreted it to just start using new features, but not really changing anything with the existing metadata. Things needing on disk changes are probably put in mmmigratefs I have not heard about anything needing mmmigratefs since GPFS v3.3 (fs version 11.03) added fast extended attributes. Would be great to hear otherwize, or confirmations. -jf man. 10. okt. 2016 kl. 14.32 skrev Peter Childs : > We are finishing upgrading our GPFS cluster of around 250 (client) nodes > from GPFS 3.5.0.31 to Spectrum Scale 4.1.1.8, and have just about upgraded > all the computers. > > We are looking at running the "mmchfs -V LATEST" step and where wondering > how much io this takes and if it was likely to interrupt service? > > We are looking at upgrading to 4.2 but plan to do that via Multi-cluster > and AFM as we are integrating new hardware and wish to increase the block > and inode size at the same time. > > Peter Childs > Research Storage Expert > ITS Research Infrastructure > Queen Mary, University of London > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From volobuev at us.ibm.com Mon Oct 10 16:39:44 2016 From: volobuev at us.ibm.com (Yuri L Volobuev) Date: Mon, 10 Oct 2016 08:39:44 -0700 Subject: [gpfsug-discuss] GPFS Upgrade 3.5 -> 4.1 In-Reply-To: References: Message-ID: Correct. mmchfs -V only done quick operations (that can be easily undone if something goes wrong). Essentially the big task here is to increase on-disk file system descriptor version number, to allow using those features that require a higher version. Bigger "conversion"-style tasks belong in mmmigratefs. The only way to increase the inode size and the data block size is to format a new file system. This cannot be done on an existing file system. yuri From: Jan-Frode Myklebust To: gpfsug main discussion list , Date: 10/10/2016 07:35 AM Subject: Re: [gpfsug-discuss] GPFS Upgrade 3.5 -> 4.1 Sent by: gpfsug-discuss-bounces at spectrumscale.org I've also always been worried about that one, but never experienced it taking any time, I/O or interruption. I've the interpreted it to just start using new features, but not really changing anything with the existing metadata. Things needing on disk changes are probably put in mmmigratefs I have not heard about anything needing mmmigratefs since GPFS v3.3 (fs version 11.03) added fast extended attributes. Would be great to hear otherwize, or confirmations. -jf man. 10. okt. 2016 kl. 14.32 skrev Peter Childs : We are finishing upgrading our GPFS cluster of around 250 (client) nodes from GPFS 3.5.0.31 to Spectrum Scale 4.1.1.8, and have just about upgraded all the computers. We are looking at running the "mmchfs -V LATEST" step and where wondering how much io this takes and if it was likely to interrupt service? We are looking at upgrading to 4.2 but plan to do that via Multi-cluster and AFM as we are integrating new hardware and wish to increase the block and inode size at the same time. Peter Childs Research Storage Expert ITS Research Infrastructure Queen Mary, University of London _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From p.childs at qmul.ac.uk Mon Oct 10 19:11:31 2016 From: p.childs at qmul.ac.uk (Peter Childs) Date: Mon, 10 Oct 2016 18:11:31 +0000 Subject: [gpfsug-discuss] GPFS Upgrade 3.5 -> 4.1 In-Reply-To: References: , Message-ID: <4i99d985lghdmb5sptsfjp2n.1476123091191@email.android.com> So in short we're saying, "mmchfs -V LATEST" increments a version number and allows new features to become possible, it does not start using them straight away. Hence Directories will shrink in 4.1 but you need to run "mmchattr --compact" on all the old ones before anything actually changes. (new ones are fine) increasing the version number makes this possible but it does not actually do it, as doing it would mean walking every directory and updating stuff. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ---- Yuri L Volobuev wrote ---- Correct. mmchfs -V only done quick operations (that can be easily undone if something goes wrong). Essentially the big task here is to increase on-disk file system descriptor version number, to allow using those features that require a higher version. Bigger "conversion"-style tasks belong in mmmigratefs. The only way to increase the inode size and the data block size is to format a new file system. This cannot be done on an existing file system. yuri [Inactive hide details for Jan-Frode Myklebust ---10/10/2016 07:35:48 AM---I've also always been worried about that one, but nev]Jan-Frode Myklebust ---10/10/2016 07:35:48 AM---I've also always been worried about that one, but never experienced it taking any time, I/O or inter From: Jan-Frode Myklebust To: gpfsug main discussion list , Date: 10/10/2016 07:35 AM Subject: Re: [gpfsug-discuss] GPFS Upgrade 3.5 -> 4.1 Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ I've also always been worried about that one, but never experienced it taking any time, I/O or interruption. I've the interpreted it to just start using new features, but not really changing anything with the existing metadata. Things needing on disk changes are probably put in mmmigratefs I have not heard about anything needing mmmigratefs since GPFS v3.3 (fs version 11.03) added fast extended attributes. Would be great to hear otherwize, or confirmations. -jf man. 10. okt. 2016 kl. 14.32 skrev Peter Childs >: We are finishing upgrading our GPFS cluster of around 250 (client) nodes from GPFS 3.5.0.31 to Spectrum Scale 4.1.1.8, and have just about upgraded all the computers. We are looking at running the "mmchfs -V LATEST" step and where wondering how much io this takes and if it was likely to interrupt service? We are looking at upgrading to 4.2 but plan to do that via Multi-cluster and AFM as we are integrating new hardware and wish to increase the block and inode size at the same time. Peter Childs Research Storage Expert ITS Research Infrastructure Queen Mary, University of London _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: graycol.gif URL: From Mark.Bush at siriuscom.com Mon Oct 10 20:56:22 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Mon, 10 Oct 2016 19:56:22 +0000 Subject: [gpfsug-discuss] Hardware refresh Message-ID: <0663AFB2-7F70-4167-A2D1-F96B5394EBD2@siriuscom.com> Have a very old cluster built on IBM X3650?s and DS3500. Need to refresh hardware. Any lessons learned in this process? Is it easiest to just build new cluster and then use AFM? Add to existing cluster then decommission nodes? What is the recommended process for this? Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Mon Oct 10 21:04:36 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Mon, 10 Oct 2016 20:04:36 +0000 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: <0663AFB2-7F70-4167-A2D1-F96B5394EBD2@siriuscom.com> References: <0663AFB2-7F70-4167-A2D1-F96B5394EBD2@siriuscom.com> Message-ID: Hi Mark, The last time we did something like this was 2010 (we?re doing rolling refreshes now), so there are probably lots of better ways to do this than what we did, but we: 1) set up the new hardware 2) created new filesystems (so that we could make adjustments we wanted to make that can only be made at FS creation time) 3) used rsync to make a 1st pass copy of everything 4) coordinated a time with users / groups to do a 2nd rsync when they weren?t active 5) used symbolic links during the transition (i.e. rm -rvf /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) 6) once everybody was migrated, updated the symlinks (i.e. /home became a symlink to /gpfs2/home) HTHAL? Kevin On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com wrote: Have a very old cluster built on IBM X3650?s and DS3500. Need to refresh hardware. Any lessons learned in this process? Is it easiest to just build new cluster and then use AFM? Add to existing cluster then decommission nodes? What is the recommended process for this? Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From usa-principal at gpfsug.org Mon Oct 10 23:59:46 2016 From: usa-principal at gpfsug.org (GPFS UG USA Principal) Date: Mon, 10 Oct 2016 18:59:46 -0400 Subject: [gpfsug-discuss] SC16 Draft Agenda [Save the date: Sunday, November 13th] Message-ID: Hello all, There have been some questions about the Spectrum Scale Users Group event for SC16 and we thought it best to publish our draft agenda and continue to fill it in as details get settled. That way you can make your travel plans and schedules. The date and rough times are set, we may adjust the time range a bit depending upon the number of user talks that people would like to give. So note, yes! we are asking for user talks. Please consider doing this. We always receive positive feedback about talks given by users that describe real world experiences. If any of the user talk slots below are of interest to you, please let us know as soon as you can. We may turn at least one user talk into a panel if we can get enough participants. As always, your feedback is welcome. This is a users group after all. So, here?s what we have planned so far: Date: Sunday November 13th Venue: TBD, but near the convention center in downtown SLC Time: ~12:30p - ~17:30p Agenda: 12:30 - 12:50 Welcome / Overview 12:50 - 13:50 The latest in IBM Spectrum Scale and ESS 13:50 - 14:20 User Talk: Big Data in HPC 14:20 - 14:50 User Talk: Tiering to Object Storage 14:50 - 15:20 - break - 15:20 - 15:50 User Talk: TBD ? volunteers! please! 15:50 - 16:35 Spectrum Scale Enhancements for CORAL 16:35 - 17:20 News from IBM Research 17:20 - 17:30 Closing Best, Kristy & Bob Kristy Kallback-Rose Manager, Research Storage Indiana University -------------- next part -------------- An HTML attachment was scrubbed... URL: From luis.bolinches at fi.ibm.com Tue Oct 11 00:40:54 2016 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Mon, 10 Oct 2016 23:40:54 +0000 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: Message-ID: Hi Creating a new FS sounds like a best way to go. NSDv2 being a very good reason to do so. AFM for migrations is quite good, latest versions allows to use NSD protocol for mounts as well. Olaf did a great job explaining this scenario on the redbook chapter 6 http://www.redbooks.ibm.com/abstracts/sg248254.html?Open -- Cheers > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L wrote: > > Hi Mark, > > The last time we did something like this was 2010 (we?re doing rolling refreshes now), so there are probably lots of better ways to do this than what we did, but we: > > 1) set up the new hardware > 2) created new filesystems (so that we could make adjustments we wanted to make that can only be made at FS creation time) > 3) used rsync to make a 1st pass copy of everything > 4) coordinated a time with users / groups to do a 2nd rsync when they weren?t active > 5) used symbolic links during the transition (i.e. rm -rvf /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) > 6) once everybody was migrated, updated the symlinks (i.e. /home became a symlink to /gpfs2/home) > > HTHAL? > > Kevin > >> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com wrote: >> >> Have a very old cluster built on IBM X3650?s and DS3500. Need to refresh hardware. Any lessons learned in this process? Is it easiest to just build new cluster and then use AFM? Add to existing cluster then decommission nodes? What is the recommended process for this? >> >> >> Mark >> This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. >> >> Sirius Computer Solutions >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and Education > Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 > > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Tue Oct 11 00:45:04 2016 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Mon, 10 Oct 2016 19:45:04 -0400 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: References: Message-ID: Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron On 10/10/16 7:40 PM, Luis Bolinches wrote: > Hi > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > reason to do so. > > AFM for migrations is quite good, latest versions allows to use NSD > protocol for mounts as well. Olaf did a great job explaining this > scenario on the redbook chapter 6 > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > -- > Cheers > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > wrote: > >> Hi Mark, >> >> The last time we did something like this was 2010 (we?re doing rolling >> refreshes now), so there are probably lots of better ways to do this >> than what we did, but we: >> >> 1) set up the new hardware >> 2) created new filesystems (so that we could make adjustments we >> wanted to make that can only be made at FS creation time) >> 3) used rsync to make a 1st pass copy of everything >> 4) coordinated a time with users / groups to do a 2nd rsync when they >> weren?t active >> 5) used symbolic links during the transition (i.e. rm -rvf >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >> 6) once everybody was migrated, updated the symlinks (i.e. /home >> became a symlink to /gpfs2/home) >> >> HTHAL? >> >> Kevin >> >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>> wrote: >>> >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>> refresh hardware. Any lessons learned in this process? Is it >>> easiest to just build new cluster and then use AFM? Add to existing >>> cluster then decommission nodes? What is the recommended process for >>> this? >>> >>> >>> Mark >>> >>> This message (including any attachments) is intended only for the use >>> of the individual or entity to which it is addressed and may contain >>> information that is non-public, proprietary, privileged, >>> confidential, and exempt from disclosure under applicable law. If you >>> are not the intended recipient, you are hereby notified that any use, >>> dissemination, distribution, or copying of this communication is >>> strictly prohibited. This message may be viewed by parties at Sirius >>> Computer Solutions other than those named in the message header. This >>> message does not contain an official representation of Sirius >>> Computer Solutions. If you have received this communication in error, >>> notify Sirius Computer Solutions immediately and (i) destroy this >>> message if a facsimile or (ii) delete this message immediately if >>> this is an electronic communication. Thank you. >>> >>> Sirius Computer Solutions >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and >> Education >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From volobuev at us.ibm.com Tue Oct 11 18:22:17 2016 From: volobuev at us.ibm.com (Yuri L Volobuev) Date: Tue, 11 Oct 2016 10:22:17 -0700 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: References: Message-ID: This depends on the committed cluster version level (minReleaseLevel) and file system format. Since NFSv2 is an on-disk format change, older code wouldn't be able to understand what it is, and thus if there's a possibility of a downlevel node looking at the NSD, the NFSv1 format is going to be used. The code does NSDv1<->NSDv2 conversions under the covers as needed when adding an empty NSD to a file system. I'd strongly recommend getting a fresh start by formatting a new file system. Many things have changed over the course of the last few years. In particular, having a 4K-aligned file system can be a pretty big deal, depending on what hardware one is going to deploy in the future, and this is something that can't be bolted onto an existing file system. Having 4K inodes is very handy for many reasons. New directory format and NSD format changes are attractive, too. And disks generally tend to get larger with time, and at some point you may want to add a disk to an existing storage pool that's larger than the existing allocation map format allows. Obviously, it's more hassle to migrate data to a new file system, as opposed to extending an existing one. In a perfect world, GPFS would offer a conversion tool that seamlessly and robustly converts old file systems, making them as good as new, but in the real world such a tool doesn't exist. Getting a clean slate by formatting a new file system every few years is a good long-term investment of time, although it comes front-loaded with extra work. yuri From: Aaron Knister To: , Date: 10/10/2016 04:45 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron On 10/10/16 7:40 PM, Luis Bolinches wrote: > Hi > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > reason to do so. > > AFM for migrations is quite good, latest versions allows to use NSD > protocol for mounts as well. Olaf did a great job explaining this > scenario on the redbook chapter 6 > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > -- > Cheers > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > wrote: > >> Hi Mark, >> >> The last time we did something like this was 2010 (we?re doing rolling >> refreshes now), so there are probably lots of better ways to do this >> than what we did, but we: >> >> 1) set up the new hardware >> 2) created new filesystems (so that we could make adjustments we >> wanted to make that can only be made at FS creation time) >> 3) used rsync to make a 1st pass copy of everything >> 4) coordinated a time with users / groups to do a 2nd rsync when they >> weren?t active >> 5) used symbolic links during the transition (i.e. rm -rvf >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >> 6) once everybody was migrated, updated the symlinks (i.e. /home >> became a symlink to /gpfs2/home) >> >> HTHAL? >> >> Kevin >> >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>> wrote: >>> >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>> refresh hardware. Any lessons learned in this process? Is it >>> easiest to just build new cluster and then use AFM? Add to existing >>> cluster then decommission nodes? What is the recommended process for >>> this? >>> >>> >>> Mark >>> >>> This message (including any attachments) is intended only for the use >>> of the individual or entity to which it is addressed and may contain >>> information that is non-public, proprietary, privileged, >>> confidential, and exempt from disclosure under applicable law. If you >>> are not the intended recipient, you are hereby notified that any use, >>> dissemination, distribution, or copying of this communication is >>> strictly prohibited. This message may be viewed by parties at Sirius >>> Computer Solutions other than those named in the message header. This >>> message does not contain an official representation of Sirius >>> Computer Solutions. If you have received this communication in error, >>> notify Sirius Computer Solutions immediately and (i) destroy this >>> message if a facsimile or (ii) delete this message immediately if >>> this is an electronic communication. Thank you. >>> >>> Sirius Computer Solutions >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and >> Education >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From Mark.Bush at siriuscom.com Tue Oct 11 20:34:27 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Tue, 11 Oct 2016 19:34:27 +0000 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: References: Message-ID: <9E337B56-A274-41B6-84C2-7084AAB771F4@siriuscom.com> Ok. I think I am hearing that a new cluster with a new FS and copying data from old to new cluster is the best way forward. Thanks everyone for your input. From: on behalf of Yuri L Volobuev Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 12:22 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh This depends on the committed cluster version level (minReleaseLevel) and file system format. Since NFSv2 is an on-disk format change, older code wouldn't be able to understand what it is, and thus if there's a possibility of a downlevel node looking at the NSD, the NFSv1 format is going to be used. The code does NSDv1<->NSDv2 conversions under the covers as needed when adding an empty NSD to a file system. I'd strongly recommend getting a fresh start by formatting a new file system. Many things have changed over the course of the last few years. In particular, having a 4K-aligned file system can be a pretty big deal, depending on what hardware one is going to deploy in the future, and this is something that can't be bolted onto an existing file system. Having 4K inodes is very handy for many reasons. New directory format and NSD format changes are attractive, too. And disks generally tend to get larger with time, and at some point you may want to add a disk to an existing storage pool that's larger than the existing allocation map format allows. Obviously, it's more hassle to migrate data to a new file system, as opposed to extending an existing one. In a perfect world, GPFS would offer a conversion tool that seamlessly and robustly converts old file systems, making them as good as new, but in the real world such a tool doesn't exist. Getting a clean slate by formatting a new file system every few years is a good long-term investment of time, although it comes front-loaded with extra work. yuri [nactive hide details for Aaron Knister ---10/10/2016 04:45:31 PM---Can on]Aaron Knister ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron From: Aaron Knister To: , Date: 10/10/2016 04:45 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron On 10/10/16 7:40 PM, Luis Bolinches wrote: > Hi > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > reason to do so. > > AFM for migrations is quite good, latest versions allows to use NSD > protocol for mounts as well. Olaf did a great job explaining this > scenario on the redbook chapter 6 > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > -- > Cheers > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > wrote: > >> Hi Mark, >> >> The last time we did something like this was 2010 (we?re doing rolling >> refreshes now), so there are probably lots of better ways to do this >> than what we did, but we: >> >> 1) set up the new hardware >> 2) created new filesystems (so that we could make adjustments we >> wanted to make that can only be made at FS creation time) >> 3) used rsync to make a 1st pass copy of everything >> 4) coordinated a time with users / groups to do a 2nd rsync when they >> weren?t active >> 5) used symbolic links during the transition (i.e. rm -rvf >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >> 6) once everybody was migrated, updated the symlinks (i.e. /home >> became a symlink to /gpfs2/home) >> >> HTHAL? >> >> Kevin >> >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>> wrote: >>> >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>> refresh hardware. Any lessons learned in this process? Is it >>> easiest to just build new cluster and then use AFM? Add to existing >>> cluster then decommission nodes? What is the recommended process for >>> this? >>> >>> >>> Mark >>> >>> This message (including any attachments) is intended only for the use >>> of the individual or entity to which it is addressed and may contain >>> information that is non-public, proprietary, privileged, >>> confidential, and exempt from disclosure under applicable law. If you >>> are not the intended recipient, you are hereby notified that any use, >>> dissemination, distribution, or copying of this communication is >>> strictly prohibited. This message may be viewed by parties at Sirius >>> Computer Solutions other than those named in the message header. This >>> message does not contain an official representation of Sirius >>> Computer Solutions. If you have received this communication in error, >>> notify Sirius Computer Solutions immediately and (i) destroy this >>> message if a facsimile or (ii) delete this message immediately if >>> this is an electronic communication. Thank you. >>> >>> Sirius Computer Solutions >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and >> Education >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 106 bytes Desc: image001.gif URL: From makaplan at us.ibm.com Tue Oct 11 20:58:53 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 11 Oct 2016 15:58:53 -0400 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: <9E337B56-A274-41B6-84C2-7084AAB771F4@siriuscom.com> References: <9E337B56-A274-41B6-84C2-7084AAB771F4@siriuscom.com> Message-ID: New FS? Yes there are some good reasons. New cluster? I did not see a compelling argument either way. From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Date: 10/11/2016 03:34 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org Ok. I think I am hearing that a new cluster with a new FS and copying data from old to new cluster is the best way forward. Thanks everyone for your input. From: on behalf of Yuri L Volobuev Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 12:22 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh This depends on the committed cluster version level (minReleaseLevel) and file system format. Since NFSv2 is an on-disk format change, older code wouldn't be able to understand what it is, and thus if there's a possibility of a downlevel node looking at the NSD, the NFSv1 format is going to be used. The code does NSDv1<->NSDv2 conversions under the covers as needed when adding an empty NSD to a file system. I'd strongly recommend getting a fresh start by formatting a new file system. Many things have changed over the course of the last few years. In particular, having a 4K-aligned file system can be a pretty big deal, depending on what hardware one is going to deploy in the future, and this is something that can't be bolted onto an existing file system. Having 4K inodes is very handy for many reasons. New directory format and NSD format changes are attractive, too. And disks generally tend to get larger with time, and at some point you may want to add a disk to an existing storage pool that's larger than the existing allocation map format allows. Obviously, it's more hassle to migrate data to a new file system, as opposed to extending an existing one. In a perfect world, GPFS would offer a conversion tool that seamlessly and robustly converts old file systems, making them as good as new, but in the real world such a tool doesn't exist. Getting a clean slate by formatting a new file system every few years is a good long-term investment of time, although it comes front-loaded with extra work. yuri Aaron Knister ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron From: Aaron Knister To: , Date: 10/10/2016 04:45 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron On 10/10/16 7:40 PM, Luis Bolinches wrote: > Hi > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > reason to do so. > > AFM for migrations is quite good, latest versions allows to use NSD > protocol for mounts as well. Olaf did a great job explaining this > scenario on the redbook chapter 6 > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > -- > Cheers > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > wrote: > >> Hi Mark, >> >> The last time we did something like this was 2010 (we?re doing rolling >> refreshes now), so there are probably lots of better ways to do this >> than what we did, but we: >> >> 1) set up the new hardware >> 2) created new filesystems (so that we could make adjustments we >> wanted to make that can only be made at FS creation time) >> 3) used rsync to make a 1st pass copy of everything >> 4) coordinated a time with users / groups to do a 2nd rsync when they >> weren?t active >> 5) used symbolic links during the transition (i.e. rm -rvf >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >> 6) once everybody was migrated, updated the symlinks (i.e. /home >> became a symlink to /gpfs2/home) >> >> HTHAL? >> >> Kevin >> >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>> wrote: >>> >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>> refresh hardware. Any lessons learned in this process? Is it >>> easiest to just build new cluster and then use AFM? Add to existing >>> cluster then decommission nodes? What is the recommended process for >>> this? >>> >>> >>> Mark >>> >>> This message (including any attachments) is intended only for the use >>> of the individual or entity to which it is addressed and may contain >>> information that is non-public, proprietary, privileged, >>> confidential, and exempt from disclosure under applicable law. If you >>> are not the intended recipient, you are hereby notified that any use, >>> dissemination, distribution, or copying of this communication is >>> strictly prohibited. This message may be viewed by parties at Sirius >>> Computer Solutions other than those named in the message header. This >>> message does not contain an official representation of Sirius >>> Computer Solutions. If you have received this communication in error, >>> notify Sirius Computer Solutions immediately and (i) destroy this >>> message if a facsimile or (ii) delete this message immediately if >>> this is an electronic communication. Thank you. >>> >>> Sirius Computer Solutions >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and >> Education >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 106 bytes Desc: not available URL: From p.childs at qmul.ac.uk Tue Oct 11 22:25:50 2016 From: p.childs at qmul.ac.uk (Peter Childs) Date: Tue, 11 Oct 2016 21:25:50 +0000 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: References: <9E337B56-A274-41B6-84C2-7084AAB771F4@siriuscom.com>, Message-ID: <7lb12taeugiu5fh1vd718dp5.1476220357109@email.android.com> My reading is, If you are running a small cluster with tie-breaker disks and your wanting to change the manager servers, or you want to switch to using the new config management method in v4 then new cluster, and use multicluster to upgrade. Otherwise just use a new Filesystem within the old cluster. But I'm interested to hear otherwise, as I'm about to embark on this myself. I note you can switch an old cluster but need to shutdown to do so. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ---- Marc A Kaplan wrote ---- New FS? Yes there are some good reasons. New cluster? I did not see a compelling argument either way. From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Date: 10/11/2016 03:34 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Ok. I think I am hearing that a new cluster with a new FS and copying data from old to new cluster is the best way forward. Thanks everyone for your input. From: on behalf of Yuri L Volobuev Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 12:22 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh This depends on the committed cluster version level (minReleaseLevel) and file system format. Since NFSv2 is an on-disk format change, older code wouldn't be able to understand what it is, and thus if there's a possibility of a downlevel node looking at the NSD, the NFSv1 format is going to be used. The code does NSDv1<->NSDv2 conversions under the covers as needed when adding an empty NSD to a file system. I'd strongly recommend getting a fresh start by formatting a new file system. Many things have changed over the course of the last few years. In particular, having a 4K-aligned file system can be a pretty big deal, depending on what hardware one is going to deploy in the future, and this is something that can't be bolted onto an existing file system. Having 4K inodes is very handy for many reasons. New directory format and NSD format changes are attractive, too. And disks generally tend to get larger with time, and at some point you may want to add a disk to an existing storage pool that's larger than the existing allocation map format allows. Obviously, it's more hassle to migrate data to a new file system, as opposed to extending an existing one. In a perfect world, GPFS would offer a conversion tool that seamlessly and robustly converts old file systems, making them as good as new, but in the real world such a tool doesn't exist. Getting a clean slate by formatting a new file system every few years is a good long-term investment of time, although it comes front-loaded with extra work. yuri [nactive hide details for Aaron Knister ---10/10/2016 04:45:31 PM---Can on]Aaron Knister ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron From: Aaron Knister To: , Date: 10/10/2016 04:45 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron On 10/10/16 7:40 PM, Luis Bolinches wrote: > Hi > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > reason to do so. > > AFM for migrations is quite good, latest versions allows to use NSD > protocol for mounts as well. Olaf did a great job explaining this > scenario on the redbook chapter 6 > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > -- > Cheers > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > wrote: > >> Hi Mark, >> >> The last time we did something like this was 2010 (we?re doing rolling >> refreshes now), so there are probably lots of better ways to do this >> than what we did, but we: >> >> 1) set up the new hardware >> 2) created new filesystems (so that we could make adjustments we >> wanted to make that can only be made at FS creation time) >> 3) used rsync to make a 1st pass copy of everything >> 4) coordinated a time with users / groups to do a 2nd rsync when they >> weren?t active >> 5) used symbolic links during the transition (i.e. rm -rvf >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >> 6) once everybody was migrated, updated the symlinks (i.e. /home >> became a symlink to /gpfs2/home) >> >> HTHAL? >> >> Kevin >> >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>> wrote: >>> >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>> refresh hardware. Any lessons learned in this process? Is it >>> easiest to just build new cluster and then use AFM? Add to existing >>> cluster then decommission nodes? What is the recommended process for >>> this? >>> >>> >>> Mark >>> >>> This message (including any attachments) is intended only for the use >>> of the individual or entity to which it is addressed and may contain >>> information that is non-public, proprietary, privileged, >>> confidential, and exempt from disclosure under applicable law. If you >>> are not the intended recipient, you are hereby notified that any use, >>> dissemination, distribution, or copying of this communication is >>> strictly prohibited. This message may be viewed by parties at Sirius >>> Computer Solutions other than those named in the message header. This >>> message does not contain an official representation of Sirius >>> Computer Solutions. If you have received this communication in error, >>> notify Sirius Computer Solutions immediately and (i) destroy this >>> message if a facsimile or (ii) delete this message immediately if >>> this is an electronic communication. Thank you. >>> >>> Sirius Computer Solutions >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and >> Education >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00001.gif Type: image/gif Size: 106 bytes Desc: ATT00001.gif URL: From aaron.s.knister at nasa.gov Tue Oct 11 23:41:36 2016 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 11 Oct 2016 18:41:36 -0400 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: References: Message-ID: Thanks Yuri. I'm asking for my own purposes but I think it's still relevant here: we're still at GPFS 3.5 and will be adding dataOnly NSDs with 4K sectors in the near future. We're planning to update to 4.1 before we format these NSDs, though. If I understand you correctly we can't bring these 4K NSDv2 NSDs into a filesystem with 512b-based NSDv1 NSDs? That's a pretty big deal :( Reformatting every few years with 10's of petabytes of data is not realistic for us (it would take years to move the data around). It also goes against my personal preachings about GPFS's storage virtualization capabilities: the ability to perform upgrades/make underlying storage infrastructure changes with behind-the-scenes data migration, eliminating much of the manual hassle of storage administrators doing rsync dances. I guess it's RFE time? It also seems as though AFM could help with automating the migration, although many of our filesystems do not have filesets on them so we would have to re-think how we lay out our filesystems. This is also curious to me with IBM pitching GPFS as a filesystem for cloud services (the cloud *never* goes down, right?). Granted I believe this pitch started after the NSDv2 format was defined, but if somebody is building a large cloud with GPFS as the underlying filesystem for an object or an image store one might think the idea of having to re-format the filesystem to gain access to critical new features is inconsistent with this pitch. It would be hugely impactful. Just my $.02. As you can tell, I'm frustrated there's no online conversion tool :) Not that there couldn't be... you all are brilliant developers. -Aaron On 10/11/16 1:22 PM, Yuri L Volobuev wrote: > This depends on the committed cluster version level (minReleaseLevel) > and file system format. Since NFSv2 is an on-disk format change, older > code wouldn't be able to understand what it is, and thus if there's a > possibility of a downlevel node looking at the NSD, the NFSv1 format is > going to be used. The code does NSDv1<->NSDv2 conversions under the > covers as needed when adding an empty NSD to a file system. > > I'd strongly recommend getting a fresh start by formatting a new file > system. Many things have changed over the course of the last few years. > In particular, having a 4K-aligned file system can be a pretty big deal, > depending on what hardware one is going to deploy in the future, and > this is something that can't be bolted onto an existing file system. > Having 4K inodes is very handy for many reasons. New directory format > and NSD format changes are attractive, too. And disks generally tend to > get larger with time, and at some point you may want to add a disk to an > existing storage pool that's larger than the existing allocation map > format allows. Obviously, it's more hassle to migrate data to a new file > system, as opposed to extending an existing one. In a perfect world, > GPFS would offer a conversion tool that seamlessly and robustly converts > old file systems, making them as good as new, but in the real world such > a tool doesn't exist. Getting a clean slate by formatting a new file > system every few years is a good long-term investment of time, although > it comes front-loaded with extra work. > > yuri > > Inactive hide details for Aaron Knister ---10/10/2016 04:45:31 PM---Can > one format NSDv2 NSDs and put them in a filesystem withAaron Knister > ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a > filesystem with NSDv1 NSD's? -Aaron > > From: Aaron Knister > To: , > Date: 10/10/2016 04:45 PM > Subject: Re: [gpfsug-discuss] Hardware refresh > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > ------------------------------------------------------------------------ > > > > Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? > > -Aaron > > On 10/10/16 7:40 PM, Luis Bolinches wrote: >> Hi >> >> Creating a new FS sounds like a best way to go. NSDv2 being a very good >> reason to do so. >> >> AFM for migrations is quite good, latest versions allows to use NSD >> protocol for mounts as well. Olaf did a great job explaining this >> scenario on the redbook chapter 6 >> >> http://www.redbooks.ibm.com/abstracts/sg248254.html?Open >> >> -- >> Cheers >> >> On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L >> > > wrote: >> >>> Hi Mark, >>> >>> The last time we did something like this was 2010 (we?re doing rolling >>> refreshes now), so there are probably lots of better ways to do this >>> than what we did, but we: >>> >>> 1) set up the new hardware >>> 2) created new filesystems (so that we could make adjustments we >>> wanted to make that can only be made at FS creation time) >>> 3) used rsync to make a 1st pass copy of everything >>> 4) coordinated a time with users / groups to do a 2nd rsync when they >>> weren?t active >>> 5) used symbolic links during the transition (i.e. rm -rvf >>> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >>> 6) once everybody was migrated, updated the symlinks (i.e. /home >>> became a symlink to /gpfs2/home) >>> >>> HTHAL? >>> >>> Kevin >>> >>>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>>> wrote: >>>> >>>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>>> refresh hardware. Any lessons learned in this process? Is it >>>> easiest to just build new cluster and then use AFM? Add to existing >>>> cluster then decommission nodes? What is the recommended process for >>>> this? >>>> >>>> >>>> Mark >>>> >>>> This message (including any attachments) is intended only for the use >>>> of the individual or entity to which it is addressed and may contain >>>> information that is non-public, proprietary, privileged, >>>> confidential, and exempt from disclosure under applicable law. If you >>>> are not the intended recipient, you are hereby notified that any use, >>>> dissemination, distribution, or copying of this communication is >>>> strictly prohibited. This message may be viewed by parties at Sirius >>>> Computer Solutions other than those named in the message header. This >>>> message does not contain an official representation of Sirius >>>> Computer Solutions. If you have received this communication in error, >>>> notify Sirius Computer Solutions immediately and (i) destroy this >>>> message if a facsimile or (ii) delete this message immediately if >>>> this is an electronic communication. Thank you. >>>> >>>> Sirius Computer Solutions >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> ? >>> Kevin Buterbaugh - Senior System Administrator >>> Vanderbilt University - Advanced Computing Center for Research and >>> Education >>> Kevin.Buterbaugh at vanderbilt.edu >>> - (615)875-9633 >>> >>> >>> >> >> Ellei edell? ole toisin mainittu: / Unless stated otherwise above: >> Oy IBM Finland Ab >> PL 265, 00101 Helsinki, Finland >> Business ID, Y-tunnus: 0195876-3 >> Registered in Finland >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From aaron.s.knister at nasa.gov Wed Oct 12 00:57:38 2016 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 11 Oct 2016 19:57:38 -0400 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: References: Message-ID: <63bd99ad-f684-d1bb-371f-e2c21849ff29@nasa.gov> I think I was a little quick to the trigger. I re-read your last mail after doing some testing and understand it differently. I was wrong about my interpretation-- you can add 4K NSDv2 formatted NSDs to a filesystem previously created with NSDv1 NSDs assuming, as you say, the minReleaseLevel and filesystem version are high enough. That negates about half of my last e-mail. The fs still doesn't show as 4K aligned: loressd01:~ # /usr/lpp/mmfs/bin/mmlsfs tnb4k --is4KAligned flag value description ------------------- ------------------------ ----------------------------------- --is4KAligned No is4KAligned? but *shrug* most of the I/O to these disks should be 1MB anyway. If somebody is pounding the FS with smaller than 4K I/O they're gonna get a talkin' to. -Aaron On 10/11/16 6:41 PM, Aaron Knister wrote: > Thanks Yuri. > > I'm asking for my own purposes but I think it's still relevant here: > we're still at GPFS 3.5 and will be adding dataOnly NSDs with 4K sectors > in the near future. We're planning to update to 4.1 before we format > these NSDs, though. If I understand you correctly we can't bring these > 4K NSDv2 NSDs into a filesystem with 512b-based NSDv1 NSDs? That's a > pretty big deal :( > > Reformatting every few years with 10's of petabytes of data is not > realistic for us (it would take years to move the data around). It also > goes against my personal preachings about GPFS's storage virtualization > capabilities: the ability to perform upgrades/make underlying storage > infrastructure changes with behind-the-scenes data migration, > eliminating much of the manual hassle of storage administrators doing > rsync dances. I guess it's RFE time? It also seems as though AFM could > help with automating the migration, although many of our filesystems do > not have filesets on them so we would have to re-think how we lay out > our filesystems. > > This is also curious to me with IBM pitching GPFS as a filesystem for > cloud services (the cloud *never* goes down, right?). Granted I believe > this pitch started after the NSDv2 format was defined, but if somebody > is building a large cloud with GPFS as the underlying filesystem for an > object or an image store one might think the idea of having to re-format > the filesystem to gain access to critical new features is inconsistent > with this pitch. It would be hugely impactful. Just my $.02. > > As you can tell, I'm frustrated there's no online conversion tool :) Not > that there couldn't be... you all are brilliant developers. > > -Aaron > > On 10/11/16 1:22 PM, Yuri L Volobuev wrote: >> This depends on the committed cluster version level (minReleaseLevel) >> and file system format. Since NFSv2 is an on-disk format change, older >> code wouldn't be able to understand what it is, and thus if there's a >> possibility of a downlevel node looking at the NSD, the NFSv1 format is >> going to be used. The code does NSDv1<->NSDv2 conversions under the >> covers as needed when adding an empty NSD to a file system. >> >> I'd strongly recommend getting a fresh start by formatting a new file >> system. Many things have changed over the course of the last few years. >> In particular, having a 4K-aligned file system can be a pretty big deal, >> depending on what hardware one is going to deploy in the future, and >> this is something that can't be bolted onto an existing file system. >> Having 4K inodes is very handy for many reasons. New directory format >> and NSD format changes are attractive, too. And disks generally tend to >> get larger with time, and at some point you may want to add a disk to an >> existing storage pool that's larger than the existing allocation map >> format allows. Obviously, it's more hassle to migrate data to a new file >> system, as opposed to extending an existing one. In a perfect world, >> GPFS would offer a conversion tool that seamlessly and robustly converts >> old file systems, making them as good as new, but in the real world such >> a tool doesn't exist. Getting a clean slate by formatting a new file >> system every few years is a good long-term investment of time, although >> it comes front-loaded with extra work. >> >> yuri >> >> Inactive hide details for Aaron Knister ---10/10/2016 04:45:31 PM---Can >> one format NSDv2 NSDs and put them in a filesystem withAaron Knister >> ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a >> filesystem with NSDv1 NSD's? -Aaron >> >> From: Aaron Knister >> To: , >> Date: 10/10/2016 04:45 PM >> Subject: Re: [gpfsug-discuss] Hardware refresh >> Sent by: gpfsug-discuss-bounces at spectrumscale.org >> >> ------------------------------------------------------------------------ >> >> >> >> Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? >> >> -Aaron >> >> On 10/10/16 7:40 PM, Luis Bolinches wrote: >>> Hi >>> >>> Creating a new FS sounds like a best way to go. NSDv2 being a very good >>> reason to do so. >>> >>> AFM for migrations is quite good, latest versions allows to use NSD >>> protocol for mounts as well. Olaf did a great job explaining this >>> scenario on the redbook chapter 6 >>> >>> http://www.redbooks.ibm.com/abstracts/sg248254.html?Open >>> >>> -- >>> Cheers >>> >>> On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L >>> >> > wrote: >>> >>>> Hi Mark, >>>> >>>> The last time we did something like this was 2010 (we?re doing rolling >>>> refreshes now), so there are probably lots of better ways to do this >>>> than what we did, but we: >>>> >>>> 1) set up the new hardware >>>> 2) created new filesystems (so that we could make adjustments we >>>> wanted to make that can only be made at FS creation time) >>>> 3) used rsync to make a 1st pass copy of everything >>>> 4) coordinated a time with users / groups to do a 2nd rsync when they >>>> weren?t active >>>> 5) used symbolic links during the transition (i.e. rm -rvf >>>> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >>>> 6) once everybody was migrated, updated the symlinks (i.e. /home >>>> became a symlink to /gpfs2/home) >>>> >>>> HTHAL? >>>> >>>> Kevin >>>> >>>>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>>>> wrote: >>>>> >>>>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>>>> refresh hardware. Any lessons learned in this process? Is it >>>>> easiest to just build new cluster and then use AFM? Add to existing >>>>> cluster then decommission nodes? What is the recommended process for >>>>> this? >>>>> >>>>> >>>>> Mark >>>>> >>>>> This message (including any attachments) is intended only for the use >>>>> of the individual or entity to which it is addressed and may contain >>>>> information that is non-public, proprietary, privileged, >>>>> confidential, and exempt from disclosure under applicable law. If you >>>>> are not the intended recipient, you are hereby notified that any use, >>>>> dissemination, distribution, or copying of this communication is >>>>> strictly prohibited. This message may be viewed by parties at Sirius >>>>> Computer Solutions other than those named in the message header. This >>>>> message does not contain an official representation of Sirius >>>>> Computer Solutions. If you have received this communication in error, >>>>> notify Sirius Computer Solutions immediately and (i) destroy this >>>>> message if a facsimile or (ii) delete this message immediately if >>>>> this is an electronic communication. Thank you. >>>>> >>>>> Sirius Computer Solutions >>>>> _______________________________________________ >>>>> gpfsug-discuss mailing list >>>>> gpfsug-discuss at spectrumscale.org >>>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>>> >>>> ? >>>> Kevin Buterbaugh - Senior System Administrator >>>> Vanderbilt University - Advanced Computing Center for Research and >>>> Education >>>> Kevin.Buterbaugh at vanderbilt.edu >>>> - (615)875-9633 >>>> >>>> >>>> >>> >>> Ellei edell? ole toisin mainittu: / Unless stated otherwise above: >>> Oy IBM Finland Ab >>> PL 265, 00101 Helsinki, Finland >>> Business ID, Y-tunnus: 0195876-3 >>> Registered in Finland >>> >>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Mark.Bush at siriuscom.com Wed Oct 12 01:40:39 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Wed, 12 Oct 2016 00:40:39 +0000 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: References: <9E337B56-A274-41B6-84C2-7084AAB771F4@siriuscom.com> Message-ID: Only compelling reason for new cluster would be old hardware is EOL or no longer want to pay maintenance on it. From: on behalf of Marc A Kaplan Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 2:58 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh New FS? Yes there are some good reasons. New cluster? I did not see a compelling argument either way. From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Date: 10/11/2016 03:34 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Ok. I think I am hearing that a new cluster with a new FS and copying data from old to new cluster is the best way forward. Thanks everyone for your input. From: on behalf of Yuri L Volobuev Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 12:22 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh This depends on the committed cluster version level (minReleaseLevel) and file system format. Since NFSv2 is an on-disk format change, older code wouldn't be able to understand what it is, and thus if there's a possibility of a downlevel node looking at the NSD, the NFSv1 format is going to be used. The code does NSDv1<->NSDv2 conversions under the covers as needed when adding an empty NSD to a file system. I'd strongly recommend getting a fresh start by formatting a new file system. Many things have changed over the course of the last few years. In particular, having a 4K-aligned file system can be a pretty big deal, depending on what hardware one is going to deploy in the future, and this is something that can't be bolted onto an existing file system. Having 4K inodes is very handy for many reasons. New directory format and NSD format changes are attractive, too. And disks generally tend to get larger with time, and at some point you may want to add a disk to an existing storage pool that's larger than the existing allocation map format allows. Obviously, it's more hassle to migrate data to a new file system, as opposed to extending an existing one. In a perfect world, GPFS would offer a conversion tool that seamlessly and robustly converts old file systems, making them as good as new, but in the real world such a tool doesn't exist. Getting a clean slate by formatting a new file system every few years is a good long-term investment of time, although it comes front-loaded with extra work. yuri [active hide details for Aaron Knister ---10/10/2016 04:45:31 PM---Can on]Aaron Knister ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron From: Aaron Knister To: , Date: 10/10/2016 04:45 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron On 10/10/16 7:40 PM, Luis Bolinches wrote: > Hi > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > reason to do so. > > AFM for migrations is quite good, latest versions allows to use NSD > protocol for mounts as well. Olaf did a great job explaining this > scenario on the redbook chapter 6 > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > -- > Cheers > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > wrote: > >> Hi Mark, >> >> The last time we did something like this was 2010 (we?re doing rolling >> refreshes now), so there are probably lots of better ways to do this >> than what we did, but we: >> >> 1) set up the new hardware >> 2) created new filesystems (so that we could make adjustments we >> wanted to make that can only be made at FS creation time) >> 3) used rsync to make a 1st pass copy of everything >> 4) coordinated a time with users / groups to do a 2nd rsync when they >> weren?t active >> 5) used symbolic links during the transition (i.e. rm -rvf >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >> 6) once everybody was migrated, updated the symlinks (i.e. /home >> became a symlink to /gpfs2/home) >> >> HTHAL? >> >> Kevin >> >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>> wrote: >>> >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>> refresh hardware. Any lessons learned in this process? Is it >>> easiest to just build new cluster and then use AFM? Add to existing >>> cluster then decommission nodes? What is the recommended process for >>> this? >>> >>> >>> Mark >>> >>> This message (including any attachments) is intended only for the use >>> of the individual or entity to which it is addressed and may contain >>> information that is non-public, proprietary, privileged, >>> confidential, and exempt from disclosure under applicable law. If you >>> are not the intended recipient, you are hereby notified that any use, >>> dissemination, distribution, or copying of this communication is >>> strictly prohibited. This message may be viewed by parties at Sirius >>> Computer Solutions other than those named in the message header. This >>> message does not contain an official representation of Sirius >>> Computer Solutions. If you have received this communication in error, >>> notify Sirius Computer Solutions immediately and (i) destroy this >>> message if a facsimile or (ii) delete this message immediately if >>> this is an electronic communication. Thank you. >>> >>> Sirius Computer Solutions >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and >> Education >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 107 bytes Desc: image001.gif URL: From aaron.s.knister at nasa.gov Wed Oct 12 01:50:38 2016 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 11 Oct 2016 20:50:38 -0400 Subject: [gpfsug-discuss] 4K sector NSD support (was: Hardware refresh) In-Reply-To: <63bd99ad-f684-d1bb-371f-e2c21849ff29@nasa.gov> References: <63bd99ad-f684-d1bb-371f-e2c21849ff29@nasa.gov> Message-ID: Yuri, (Sorry for being somewhat spammy) I now understand the limitation after some more testing (I'm a hands-on learner, can you tell?). Given the right code/cluster/fs version levels I can add 4K dataOnly NSDv2 NSDs to a filesystem created with NSDv1 NSDs. What I can't do is seemingly add any metadataOnly or dataAndMetadata 4K luns to an fs that is not 4K aligned which I assume would be any fs originally created with NSDv1 LUNs. It seems possible to move all data away from NSDv1 LUNS in a filesystem behind-the-scenes using GPFS migration tools, and move the data to NSDv2 LUNs. In this case I believe what's missing is a tool to convert just the metadata structures to be 4K aligned since the data would already on 4k-based NSDv2 LUNS, is that the case? I'm trying to figure out what exactly I'm asking for in an RFE. -Aaron On 10/11/16 7:57 PM, Aaron Knister wrote: > I think I was a little quick to the trigger. I re-read your last mail > after doing some testing and understand it differently. I was wrong > about my interpretation-- you can add 4K NSDv2 formatted NSDs to a > filesystem previously created with NSDv1 NSDs assuming, as you say, the. > minReleaseLevel and filesystem version are high enough. That negates > about half of my last e-mail. The fs still doesn't show as 4K aligned: > > loressd01:~ # /usr/lpp/mmfs/bin/mmlsfs tnb4k --is4KAligned > flag value description > ------------------- ------------------------ > ----------------------------------- > --is4KAligned No is4KAligned? > > but *shrug* most of the I/O to these disks should be 1MB anyway. If > somebody is pounding the FS with smaller than 4K I/O they're gonna get a > talkin' to. > > -Aaron > > On 10/11/16 6:41 PM, Aaron Knister wrote: >> Thanks Yuri. >> >> I'm asking for my own purposes but I think it's still relevant here: >> we're still at GPFS 3.5 and will be adding dataOnly NSDs with 4K sectors >> in the near future. We're planning to update to 4.1 before we format >> these NSDs, though. If I understand you correctly we can't bring these >> 4K NSDv2 NSDs into a filesystem with 512b-based NSDv1 NSDs? That's a >> pretty big deal :( >> >> Reformatting every few years with 10's of petabytes of data is not >> realistic for us (it would take years to move the data around). It also >> goes against my personal preachings about GPFS's storage virtualization >> capabilities: the ability to perform upgrades/make underlying storage >> infrastructure changes with behind-the-scenes data migration, >> eliminating much of the manual hassle of storage administrators doing >> rsync dances. I guess it's RFE time? It also seems as though AFM could >> help with automating the migration, although many of our filesystems do >> not have filesets on them so we would have to re-think how we lay out >> our filesystems. >> >> This is also curious to me with IBM pitching GPFS as a filesystem for >> cloud services (the cloud *never* goes down, right?). Granted I believe >> this pitch started after the NSDv2 format was defined, but if somebody >> is building a large cloud with GPFS as the underlying filesystem for an >> object or an image store one might think the idea of having to re-format >> the filesystem to gain access to critical new features is inconsistent >> with this pitch. It would be hugely impactful. Just my $.02. >> >> As you can tell, I'm frustrated there's no online conversion tool :) Not >> that there couldn't be... you all are brilliant developers. >> >> -Aaron >> >> On 10/11/16 1:22 PM, Yuri L Volobuev wrote: >>> This depends on the committed cluster version level (minReleaseLevel) >>> and file system format. Since NFSv2 is an on-disk format change, older >>> code wouldn't be able to understand what it is, and thus if there's a >>> possibility of a downlevel node looking at the NSD, the NFSv1 format is >>> going to be used. The code does NSDv1<->NSDv2 conversions under the >>> covers as needed when adding an empty NSD to a file system. >>> >>> I'd strongly recommend getting a fresh start by formatting a new file >>> system. Many things have changed over the course of the last few years. >>> In particular, having a 4K-aligned file system can be a pretty big deal, >>> depending on what hardware one is going to deploy in the future, and >>> this is something that can't be bolted onto an existing file system. >>> Having 4K inodes is very handy for many reasons. New directory format >>> and NSD format changes are attractive, too. And disks generally tend to >>> get larger with time, and at some point you may want to add a disk to an >>> existing storage pool that's larger than the existing allocation map >>> format allows. Obviously, it's more hassle to migrate data to a new file >>> system, as opposed to extending an existing one. In a perfect world, >>> GPFS would offer a conversion tool that seamlessly and robustly converts >>> old file systems, making them as good as new, but in the real world such >>> a tool doesn't exist. Getting a clean slate by formatting a new file >>> system every few years is a good long-term investment of time, although >>> it comes front-loaded with extra work. >>> >>> yuri >>> >>> Inactive hide details for Aaron Knister ---10/10/2016 04:45:31 PM---Can >>> one format NSDv2 NSDs and put them in a filesystem withAaron Knister >>> ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a >>> filesystem with NSDv1 NSD's? -Aaron >>> >>> From: Aaron Knister >>> To: , >>> Date: 10/10/2016 04:45 PM >>> Subject: Re: [gpfsug-discuss] Hardware refresh >>> Sent by: gpfsug-discuss-bounces at spectrumscale.org >>> >>> ------------------------------------------------------------------------ >>> >>> >>> >>> Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? >>> >>> -Aaron >>> >>> On 10/10/16 7:40 PM, Luis Bolinches wrote: >>>> Hi >>>> >>>> Creating a new FS sounds like a best way to go. NSDv2 being a very good >>>> reason to do so. >>>> >>>> AFM for migrations is quite good, latest versions allows to use NSD >>>> protocol for mounts as well. Olaf did a great job explaining this >>>> scenario on the redbook chapter 6 >>>> >>>> http://www.redbooks.ibm.com/abstracts/sg248254.html?Open >>>> >>>> -- >>>> Cheers >>>> >>>> On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L >>>> >>> > wrote: >>>> >>>>> Hi Mark, >>>>> >>>>> The last time we did something like this was 2010 (we?re doing rolling >>>>> refreshes now), so there are probably lots of better ways to do this >>>>> than what we did, but we: >>>>> >>>>> 1) set up the new hardware >>>>> 2) created new filesystems (so that we could make adjustments we >>>>> wanted to make that can only be made at FS creation time) >>>>> 3) used rsync to make a 1st pass copy of everything >>>>> 4) coordinated a time with users / groups to do a 2nd rsync when they >>>>> weren?t active >>>>> 5) used symbolic links during the transition (i.e. rm -rvf >>>>> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >>>>> 6) once everybody was migrated, updated the symlinks (i.e. /home >>>>> became a symlink to /gpfs2/home) >>>>> >>>>> HTHAL? >>>>> >>>>> Kevin >>>>> >>>>>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>>>>> wrote: >>>>>> >>>>>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>>>>> refresh hardware. Any lessons learned in this process? Is it >>>>>> easiest to just build new cluster and then use AFM? Add to existing >>>>>> cluster then decommission nodes? What is the recommended process for >>>>>> this? >>>>>> >>>>>> >>>>>> Mark >>>>>> >>>>>> This message (including any attachments) is intended only for the use >>>>>> of the individual or entity to which it is addressed and may contain >>>>>> information that is non-public, proprietary, privileged, >>>>>> confidential, and exempt from disclosure under applicable law. If you >>>>>> are not the intended recipient, you are hereby notified that any use, >>>>>> dissemination, distribution, or copying of this communication is >>>>>> strictly prohibited. This message may be viewed by parties at Sirius >>>>>> Computer Solutions other than those named in the message header. This >>>>>> message does not contain an official representation of Sirius >>>>>> Computer Solutions. If you have received this communication in error, >>>>>> notify Sirius Computer Solutions immediately and (i) destroy this >>>>>> message if a facsimile or (ii) delete this message immediately if >>>>>> this is an electronic communication. Thank you. >>>>>> >>>>>> Sirius Computer Solutions >>>>>> _______________________________________________ >>>>>> gpfsug-discuss mailing list >>>>>> gpfsug-discuss at spectrumscale.org >>>>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>>>> >>>>> ? >>>>> Kevin Buterbaugh - Senior System Administrator >>>>> Vanderbilt University - Advanced Computing Center for Research and >>>>> Education >>>>> Kevin.Buterbaugh at vanderbilt.edu >>>>> - (615)875-9633 >>>>> >>>>> >>>>> >>>> >>>> Ellei edell? ole toisin mainittu: / Unless stated otherwise above: >>>> Oy IBM Finland Ab >>>> PL 265, 00101 Helsinki, Finland >>>> Business ID, Y-tunnus: 0195876-3 >>>> Registered in Finland >>>> >>>> >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> >>> >>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From ulmer at ulmer.org Wed Oct 12 02:29:36 2016 From: ulmer at ulmer.org (Stephen Ulmer) Date: Tue, 11 Oct 2016 21:29:36 -0400 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: References: <9E337B56-A274-41B6-84C2-7084AAB771F4@siriuscom.com> Message-ID: <279BAB6D-21E6-4EC1-98CD-CA409A24F8F6@ulmer.org> I think that the OP was asking why not expand the existing cluster with the new hardware, and just make a new FS? I?ve not tried to make a cluster talk AFM to itself yet. If that?s impossible, then there?s one good reason to make a new cluster (to use AFM for migration). Liberty, -- Stephen > On Oct 11, 2016, at 8:40 PM, Mark.Bush at siriuscom.com wrote: > > Only compelling reason for new cluster would be old hardware is EOL or no longer want to pay maintenance on it. > > From: > on behalf of Marc A Kaplan > > Reply-To: gpfsug main discussion list > > Date: Tuesday, October 11, 2016 at 2:58 PM > To: gpfsug main discussion list > > Subject: Re: [gpfsug-discuss] Hardware refresh > > New FS? Yes there are some good reasons. > New cluster? I did not see a compelling argument either way. > > > > From: "Mark.Bush at siriuscom.com " > > To: gpfsug main discussion list > > Date: 10/11/2016 03:34 PM > Subject: Re: [gpfsug-discuss] Hardware refresh > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > Ok. I think I am hearing that a new cluster with a new FS and copying data from old to new cluster is the best way forward. Thanks everyone for your input. > > From: > on behalf of Yuri L Volobuev > > Reply-To: gpfsug main discussion list > > Date: Tuesday, October 11, 2016 at 12:22 PM > To: gpfsug main discussion list > > Subject: Re: [gpfsug-discuss] Hardware refresh > > This depends on the committed cluster version level (minReleaseLevel) and file system format. Since NFSv2 is an on-disk format change, older code wouldn't be able to understand what it is, and thus if there's a possibility of a downlevel node looking at the NSD, the NFSv1 format is going to be used. The code does NSDv1<->NSDv2 conversions under the covers as needed when adding an empty NSD to a file system. > > I'd strongly recommend getting a fresh start by formatting a new file system. Many things have changed over the course of the last few years. In particular, having a 4K-aligned file system can be a pretty big deal, depending on what hardware one is going to deploy in the future, and this is something that can't be bolted onto an existing file system. Having 4K inodes is very handy for many reasons. New directory format and NSD format changes are attractive, too. And disks generally tend to get larger with time, and at some point you may want to add a disk to an existing storage pool that's larger than the existing allocation map format allows. Obviously, it's more hassle to migrate data to a new file system, as opposed to extending an existing one. In a perfect world, GPFS would offer a conversion tool that seamlessly and robustly converts old file systems, making them as good as new, but in the real world such a tool doesn't exist. Getting a clean slate by formatting a new file system every few years is a good long-term investment of time, although it comes front-loaded with extra work. > > yuri > > Aaron Knister ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron > > From: Aaron Knister > > To: >, > Date: 10/10/2016 04:45 PM > Subject: Re: [gpfsug-discuss] Hardware refresh > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > > Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? > > -Aaron > > On 10/10/16 7:40 PM, Luis Bolinches wrote: > > Hi > > > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > > reason to do so. > > > > AFM for migrations is quite good, latest versions allows to use NSD > > protocol for mounts as well. Olaf did a great job explaining this > > scenario on the redbook chapter 6 > > > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > > > -- > > Cheers > > > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > > > >> wrote: > > > >> Hi Mark, > >> > >> The last time we did something like this was 2010 (we?re doing rolling > >> refreshes now), so there are probably lots of better ways to do this > >> than what we did, but we: > >> > >> 1) set up the new hardware > >> 2) created new filesystems (so that we could make adjustments we > >> wanted to make that can only be made at FS creation time) > >> 3) used rsync to make a 1st pass copy of everything > >> 4) coordinated a time with users / groups to do a 2nd rsync when they > >> weren?t active > >> 5) used symbolic links during the transition (i.e. rm -rvf > >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) > >> 6) once everybody was migrated, updated the symlinks (i.e. /home > >> became a symlink to /gpfs2/home) > >> > >> HTHAL? > >> > >> Kevin > >> > >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com > >>> > wrote: > >>> > >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to > >>> refresh hardware. Any lessons learned in this process? Is it > >>> easiest to just build new cluster and then use AFM? Add to existing > >>> cluster then decommission nodes? What is the recommended process for > >>> this? > >>> > >>> > >>> Mark > >>> > >>> This message (including any attachments) is intended only for the use > >>> of the individual or entity to which it is addressed and may contain > >>> information that is non-public, proprietary, privileged, > >>> confidential, and exempt from disclosure under applicable law. If you > >>> are not the intended recipient, you are hereby notified that any use, > >>> dissemination, distribution, or copying of this communication is > >>> strictly prohibited. This message may be viewed by parties at Sirius > >>> Computer Solutions other than those named in the message header. This > >>> message does not contain an official representation of Sirius > >>> Computer Solutions. If you have received this communication in error, > >>> notify Sirius Computer Solutions immediately and (i) destroy this > >>> message if a facsimile or (ii) delete this message immediately if > >>> this is an electronic communication. Thank you. > >>> > >>> Sirius Computer Solutions > > >>> _______________________________________________ > >>> gpfsug-discuss mailing list > >>> gpfsug-discuss at spectrumscale.org > > >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > >> ? > >> Kevin Buterbaugh - Senior System Administrator > >> Vanderbilt University - Advanced Computing Center for Research and > >> Education > >> Kevin.Buterbaugh at vanderbilt.edu > >> > - (615)875-9633 > >> > >> > >> > > > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > > Oy IBM Finland Ab > > PL 265, 00101 Helsinki, Finland > > Business ID, Y-tunnus: 0195876-3 > > Registered in Finland > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. > > Sirius Computer Solutions _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Wed Oct 12 02:33:32 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Wed, 12 Oct 2016 01:33:32 +0000 Subject: [gpfsug-discuss] 4K sector NSD support (was: Hardware refresh) In-Reply-To: Message-ID: If you File System was created with i=512 you wont benefit from 4k Disk technologies ... some backend emulate it by Controller Software but most likely you'll get in trouble, when trying to add 4k Disk into your filesystem ... Gesendet von IBM Verse Aaron Knister --- Re: [gpfsug-discuss] 4K sector NSD support (was: Hardware refresh) --- Von:"Aaron Knister" An:gpfsug-discuss at spectrumscale.orgDatum:Di. 11.10.2016 17:59Betreff:Re: [gpfsug-discuss] 4K sector NSD support (was: Hardware refresh) Yuri,(Sorry for being somewhat spammy) I now understand the limitation after some more testing (I'm a hands-on learner, can you tell?). Given the right code/cluster/fs version levels I can add 4K dataOnly NSDv2 NSDs to a filesystem created with NSDv1 NSDs. What I can't do is seemingly add any metadataOnly or dataAndMetadata 4K luns to an fs that is not 4K aligned which I assume would be any fs originally created with NSDv1 LUNs. It seems possible to move all data away from NSDv1 LUNS in a filesystem behind-the-scenes using GPFS migration tools, and move the data to NSDv2 LUNs. In this case I believe what's missing is a tool to convert just the metadata structures to be 4K aligned since the data would already on 4k-based NSDv2 LUNS, is that the case? I'm trying to figure out what exactly I'm asking for in an RFE.-AaronOn 10/11/16 7:57 PM, Aaron Knister wrote:> I think I was a little quick to the trigger. I re-read your last mail> after doing some testing and understand it differently. I was wrong> about my interpretation-- you can add 4K NSDv2 formatted NSDs to a> filesystem previously created with NSDv1 NSDs assuming, as you say, the.> minReleaseLevel and filesystem version are high enough. That negates> about half of my last e-mail. The fs still doesn't show as 4K aligned:>> loressd01:~ # /usr/lpp/mmfs/bin/mmlsfs tnb4k --is4KAligned> flag value description> ------------------- ------------------------> -----------------------------------> --is4KAligned No is4KAligned?>> but *shrug* most of the I/O to these disks should be 1MB anyway. If> somebody is pounding the FS with smaller than 4K I/O they're gonna get a> talkin' to.>> -Aaron>> On 10/11/16 6:41 PM, Aaron Knister wrote:>> Thanks Yuri.>>>> I'm asking for my own purposes but I think it's still relevant here:>> we're still at GPFS 3.5 and will be adding dataOnly NSDs with 4K sectors>> in the near future. We're planning to update to 4.1 before we format>> these NSDs, though. If I understand you correctly we can't bring these>> 4K NSDv2 NSDs into a filesystem with 512b-based NSDv1 NSDs? That's a>> pretty big deal :(>>>> Reformatting every few years with 10's of petabytes of data is not>> realistic for us (it would take years to move the data around). It also>> goes against my personal preachings about GPFS's storage virtualization>> capabilities: the ability to perform upgrades/make underlying storage>> infrastructure changes with behind-the-scenes data migration,>> eliminating much of the manual hassle of storage administrators doing>> rsync dances. I guess it's RFE time? It also seems as though AFM could>> help with automating the migration, although many of our filesystems do>> not have filesets on them so we would have to re-think how we lay out>> our filesystems.>>>> This is also curious to me with IBM pitching GPFS as a filesystem for>> cloud services (the cloud *never* goes down, right?). Granted I believe>> this pitch started after the NSDv2 format was defined, but if somebody>> is building a large cloud with GPFS as the underlying filesystem for an>> object or an image store one might think the idea of having to re-format>> the filesystem to gain access to critical new features is inconsistent>> with this pitch. It would be hugely impactful. Just my $.02.>>>> As you can tell, I'm frustrated there's no online conversion tool :) Not>> that there couldn't be... you all are brilliant developers.>>>> -Aaron>>>> On 10/11/16 1:22 PM, Yuri L Volobuev wrote:>>> This depends on the committed cluster version level (minReleaseLevel)>>> and file system format. Since NFSv2 is an on-disk format change, older>>> code wouldn't be able to understand what it is, and thus if there's a>>> possibility of a downlevel node looking at the NSD, the NFSv1 format is>>> going to be used. The code does NSDv1<->NSDv2 conversions under the>>> covers as needed when adding an empty NSD to a file system.>>>>>> I'd strongly recommend getting a fresh start by formatting a new file>>> system. Many things have changed over the course of the last few years.>>> In particular, having a 4K-aligned file system can be a pretty big deal,>>> depending on what hardware one is going to deploy in the future, and>>> this is something that can't be bolted onto an existing file system.>>> Having 4K inodes is very handy for many reasons. New directory format>>> and NSD format changes are attractive, too. And disks generally tend to>>> get larger with time, and at some point you may want to add a disk to an>>> existing storage pool that's larger than the existing allocation map>>> format allows. Obviously, it's more hassle to migrate data to a new file>>> system, as opposed to extending an existing one. In a perfect world,>>> GPFS would offer a conversion tool that seamlessly and robustly converts>>> old file systems, making them as good as new, but in the real world such>>> a tool doesn't exist. Getting a clean slate by formatting a new file>>> system every few years is a good long-term investment of time, although>>> it comes front-loaded with extra work.>>>>>> yuri>>>>>> Inactive hide details for Aaron Knister ---10/10/2016 04:45:31 PM---Can>>> one format NSDv2 NSDs and put them in a filesystem withAaron Knister>>> ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a>>> filesystem with NSDv1 NSD's? -Aaron>>>>>> From: Aaron Knister >>> To: ,>>> Date: 10/10/2016 04:45 PM>>> Subject: Re: [gpfsug-discuss] Hardware refresh>>> Sent by: gpfsug-discuss-bounces at spectrumscale.org>>>>>> ------------------------------------------------------------------------>>>>>>>>>>>> Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's?>>>>>> -Aaron>>>>>> On 10/10/16 7:40 PM, Luis Bolinches wrote:>>>> Hi>>>>>>>> Creating a new FS sounds like a best way to go. NSDv2 being a very good>>>> reason to do so.>>>>>>>> AFM for migrations is quite good, latest versions allows to use NSD>>>> protocol for mounts as well. Olaf did a great job explaining this>>>> scenario on the redbook chapter 6>>>>>>>> http://www.redbooks.ibm.com/abstracts/sg248254.html?Open>>>>>>>> -->>>> Cheers>>>>>>>> On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L>>>> >>> > wrote:>>>>>>>>> Hi Mark,>>>>>>>>>> The last time we did something like this was 2010 (we?re doing rolling>>>>> refreshes now), so there are probably lots of better ways to do this>>>>> than what we did, but we:>>>>>>>>>> 1) set up the new hardware>>>>> 2) created new filesystems (so that we could make adjustments we>>>>> wanted to make that can only be made at FS creation time)>>>>> 3) used rsync to make a 1st pass copy of everything>>>>> 4) coordinated a time with users / groups to do a 2nd rsync when they>>>>> weren?t active>>>>> 5) used symbolic links during the transition (i.e. rm -rvf>>>>> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser)>>>>> 6) once everybody was migrated, updated the symlinks (i.e. /home>>>>> became a symlink to /gpfs2/home)>>>>>>>>>> HTHAL?>>>>>>>>>> Kevin>>>>>>>>>>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com>>>>>> wrote:>>>>>>>>>>>> Have a very old cluster built on IBM X3650?s and DS3500. Need to>>>>>> refresh hardware. Any lessons learned in this process? Is it>>>>>> easiest to just build new cluster and then use AFM? Add to existing>>>>>> cluster then decommission nodes? What is the recommended process for>>>>>> this?>>>>>>>>>>>>>>>>>> Mark>>>>>>>>>>>> This message (including any attachments) is intended only for the use>>>>>> of the individual or entity to which it is addressed and may contain>>>>>> information that is non-public, proprietary, privileged,>>>>>> confidential, and exempt from disclosure under applicable law. If you>>>>>> are not the intended recipient, you are hereby notified that any use,>>>>>> dissemination, distribution, or copying of this communication is>>>>>> strictly prohibited. This message may be viewed by parties at Sirius>>>>>> Computer Solutions other than those named in the message header. This>>>>>> message does not contain an official representation of Sirius>>>>>> Computer Solutions. If you have received this communication in error,>>>>>> notify Sirius Computer Solutions immediately and (i) destroy this>>>>>> message if a facsimile or (ii) delete this message immediately if>>>>>> this is an electronic communication. Thank you.>>>>>>>>>>>> Sirius Computer Solutions >>>>>> _______________________________________________>>>>>> gpfsug-discuss mailing list>>>>>> gpfsug-discuss at spectrumscale.org >>>>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss>>>>>>>>>> ?>>>>> Kevin Buterbaugh - Senior System Administrator>>>>> Vanderbilt University - Advanced Computing Center for Research and>>>>> Education>>>>> Kevin.Buterbaugh at vanderbilt.edu>>>>> - (615)875-9633>>>>>>>>>>>>>>>>>>>>>>> Ellei edell? ole toisin mainittu: / Unless stated otherwise above:>>>> Oy IBM Finland Ab>>>> PL 265, 00101 Helsinki, Finland>>>> Business ID, Y-tunnus: 0195876-3>>>> Registered in Finland>>>>>>>>>>>>>>>> _______________________________________________>>>> gpfsug-discuss mailing list>>>> gpfsug-discuss at spectrumscale.org>>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss>>>>>>> _______________________________________________>>> gpfsug-discuss mailing list>>> gpfsug-discuss at spectrumscale.org>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss>>>>>>>>>>>>>>>>>> _______________________________________________>>> gpfsug-discuss mailing list>>> gpfsug-discuss at spectrumscale.org>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss>>>>> _______________________________________________>> gpfsug-discuss mailing list>> gpfsug-discuss at spectrumscale.org>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss> _______________________________________________> gpfsug-discuss mailing list> gpfsug-discuss at spectrumscale.org> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_______________________________________________gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From volobuev at us.ibm.com Wed Oct 12 06:44:40 2016 From: volobuev at us.ibm.com (Yuri L Volobuev) Date: Tue, 11 Oct 2016 22:44:40 -0700 Subject: [gpfsug-discuss] 4K sector NSD support (was: Hardware refresh) In-Reply-To: References: <63bd99ad-f684-d1bb-371f-e2c21849ff29@nasa.gov> Message-ID: Yes, it is possible to add a 4KN dataOnly NSD to a non-4K-aligned file system, as you figured out. This is something we didn't plan on doing originally, but then had to implement based on the feedback from the field. There's clearly a need for this. However, this statement is exactly it -- dataOnly NSDs. The only way to put metadata on a 4KN disk is to use a 4K-aligned file system. There are several kinds of metadata present in non-4K-aligned file system that generate non-4K IOs (512 byte inodes being the biggest problem), and there's no way to work around this short of using the new format, and there's no way to perform a conversion to the new format in-place. You're welcome to submit an RFE, of course, but I'd recommend being pragmatic about the chances of such an RFE being implemented. As you can imagine, the main reason why an all-encompassing file system conversion tool doesn't exist is not GPFS developers having no idea that such a tool is wanted. There are several considerations that conspire to make this an unlikely candidate to ever be implemented: 1) The task is hard and has no finish line. In most GPFS releases, something changes, necessitating an added piece of work for the hypothetical conversion tool, and the matrix of from-to format version combinations gets to be big quite quickly. 2) A file system conversion is something that is needed very infrequently, but when this code does run, it absolutely has to run and run perfectly, else the result would be a half-converted file system, i.e. a royal mess. This is a tester's nightmare. 3) The failure scenarios are all unpalatable. What should the conversion tool do if it runs out of space replacing smaller metadata structures with bigger ones? Undoing a partially finished conversion is even harder than doing it in the first place. 4) Doing an on-disk conversion on-line is simply very difficult. Consider the task of converting an inode file to use a different inode size. The file can be huge (billions of records), and it would take a fair chunk of time to rewrite it, but the file is changing while it's being converted (can't simply lock the whole thing down for so long), simultaneously on multiple nodes. Orchestrating the processing of updates in the presence of two inode files, with proper atomicity guarantees (to guard against a node failure) is a task of considerable complexity. None of this means the task is impossible, of course. It is, however, a very big chunk of very complex work, all towards a tool that on an average cluster may run somewhere between zero and one times, not something that benefits day-to-day operations. Where the complexity of the task allows for a reasonably affordable implementation, e.g. conversion from an old-style EA file to the FASTEA format, a conversion tool has been implemented (mmmigratefs). However, doing this for every single changed aspect of the file system format is simply too expensive to justify, given other tasks in front of us. On the other hand, a well-implemented migration mechanism solves the file system reformatting scenario (which covers all aspects of file system format changes) as well as a number of other scenarios. This is a cleaner, more general solution. Migration doesn't have to mean an outage. A simple rsync-based migration requires downtime for a cutover, while an AFM-based migration doesn't necessarily require one. I'm not saying that GPFS has a particularly strong migration story at the moment, but this is a much more productive direction for applying resources than a mythical all-encompassing conversion tool. yuri From: Aaron Knister To: , Date: 10/11/2016 05:59 PM Subject: Re: [gpfsug-discuss] 4K sector NSD support (was: Hardware refresh) Sent by: gpfsug-discuss-bounces at spectrumscale.org Yuri, (Sorry for being somewhat spammy) I now understand the limitation after some more testing (I'm a hands-on learner, can you tell?). Given the right code/cluster/fs version levels I can add 4K dataOnly NSDv2 NSDs to a filesystem created with NSDv1 NSDs. What I can't do is seemingly add any metadataOnly or dataAndMetadata 4K luns to an fs that is not 4K aligned which I assume would be any fs originally created with NSDv1 LUNs. It seems possible to move all data away from NSDv1 LUNS in a filesystem behind-the-scenes using GPFS migration tools, and move the data to NSDv2 LUNs. In this case I believe what's missing is a tool to convert just the metadata structures to be 4K aligned since the data would already on 4k-based NSDv2 LUNS, is that the case? I'm trying to figure out what exactly I'm asking for in an RFE. -Aaron On 10/11/16 7:57 PM, Aaron Knister wrote: > I think I was a little quick to the trigger. I re-read your last mail > after doing some testing and understand it differently. I was wrong > about my interpretation-- you can add 4K NSDv2 formatted NSDs to a > filesystem previously created with NSDv1 NSDs assuming, as you say, the. > minReleaseLevel and filesystem version are high enough. That negates > about half of my last e-mail. The fs still doesn't show as 4K aligned: > > loressd01:~ # /usr/lpp/mmfs/bin/mmlsfs tnb4k --is4KAligned > flag value description > ------------------- ------------------------ > ----------------------------------- > --is4KAligned No is4KAligned? > > but *shrug* most of the I/O to these disks should be 1MB anyway. If > somebody is pounding the FS with smaller than 4K I/O they're gonna get a > talkin' to. > > -Aaron > > On 10/11/16 6:41 PM, Aaron Knister wrote: >> Thanks Yuri. >> >> I'm asking for my own purposes but I think it's still relevant here: >> we're still at GPFS 3.5 and will be adding dataOnly NSDs with 4K sectors >> in the near future. We're planning to update to 4.1 before we format >> these NSDs, though. If I understand you correctly we can't bring these >> 4K NSDv2 NSDs into a filesystem with 512b-based NSDv1 NSDs? That's a >> pretty big deal :( >> >> Reformatting every few years with 10's of petabytes of data is not >> realistic for us (it would take years to move the data around). It also >> goes against my personal preachings about GPFS's storage virtualization >> capabilities: the ability to perform upgrades/make underlying storage >> infrastructure changes with behind-the-scenes data migration, >> eliminating much of the manual hassle of storage administrators doing >> rsync dances. I guess it's RFE time? It also seems as though AFM could >> help with automating the migration, although many of our filesystems do >> not have filesets on them so we would have to re-think how we lay out >> our filesystems. >> >> This is also curious to me with IBM pitching GPFS as a filesystem for >> cloud services (the cloud *never* goes down, right?). Granted I believe >> this pitch started after the NSDv2 format was defined, but if somebody >> is building a large cloud with GPFS as the underlying filesystem for an >> object or an image store one might think the idea of having to re-format >> the filesystem to gain access to critical new features is inconsistent >> with this pitch. It would be hugely impactful. Just my $.02. >> >> As you can tell, I'm frustrated there's no online conversion tool :) Not >> that there couldn't be... you all are brilliant developers. >> >> -Aaron >> >> On 10/11/16 1:22 PM, Yuri L Volobuev wrote: >>> This depends on the committed cluster version level (minReleaseLevel) >>> and file system format. Since NFSv2 is an on-disk format change, older >>> code wouldn't be able to understand what it is, and thus if there's a >>> possibility of a downlevel node looking at the NSD, the NFSv1 format is >>> going to be used. The code does NSDv1<->NSDv2 conversions under the >>> covers as needed when adding an empty NSD to a file system. >>> >>> I'd strongly recommend getting a fresh start by formatting a new file >>> system. Many things have changed over the course of the last few years. >>> In particular, having a 4K-aligned file system can be a pretty big deal, >>> depending on what hardware one is going to deploy in the future, and >>> this is something that can't be bolted onto an existing file system. >>> Having 4K inodes is very handy for many reasons. New directory format >>> and NSD format changes are attractive, too. And disks generally tend to >>> get larger with time, and at some point you may want to add a disk to an >>> existing storage pool that's larger than the existing allocation map >>> format allows. Obviously, it's more hassle to migrate data to a new file >>> system, as opposed to extending an existing one. In a perfect world, >>> GPFS would offer a conversion tool that seamlessly and robustly converts >>> old file systems, making them as good as new, but in the real world such >>> a tool doesn't exist. Getting a clean slate by formatting a new file >>> system every few years is a good long-term investment of time, although >>> it comes front-loaded with extra work. >>> >>> yuri >>> >>> Inactive hide details for Aaron Knister ---10/10/2016 04:45:31 PM---Can >>> one format NSDv2 NSDs and put them in a filesystem withAaron Knister >>> ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a >>> filesystem with NSDv1 NSD's? -Aaron >>> >>> From: Aaron Knister >>> To: , >>> Date: 10/10/2016 04:45 PM >>> Subject: Re: [gpfsug-discuss] Hardware refresh >>> Sent by: gpfsug-discuss-bounces at spectrumscale.org >>> >>> ------------------------------------------------------------------------ >>> >>> >>> >>> Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? >>> >>> -Aaron >>> >>> On 10/10/16 7:40 PM, Luis Bolinches wrote: >>>> Hi >>>> >>>> Creating a new FS sounds like a best way to go. NSDv2 being a very good >>>> reason to do so. >>>> >>>> AFM for migrations is quite good, latest versions allows to use NSD >>>> protocol for mounts as well. Olaf did a great job explaining this >>>> scenario on the redbook chapter 6 >>>> >>>> http://www.redbooks.ibm.com/abstracts/sg248254.html?Open >>>> >>>> -- >>>> Cheers >>>> >>>> On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L >>>> >>> > wrote: >>>> >>>>> Hi Mark, >>>>> >>>>> The last time we did something like this was 2010 (we?re doing rolling >>>>> refreshes now), so there are probably lots of better ways to do this >>>>> than what we did, but we: >>>>> >>>>> 1) set up the new hardware >>>>> 2) created new filesystems (so that we could make adjustments we >>>>> wanted to make that can only be made at FS creation time) >>>>> 3) used rsync to make a 1st pass copy of everything >>>>> 4) coordinated a time with users / groups to do a 2nd rsync when they >>>>> weren?t active >>>>> 5) used symbolic links during the transition (i.e. rm -rvf >>>>> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >>>>> 6) once everybody was migrated, updated the symlinks (i.e. /home >>>>> became a symlink to /gpfs2/home) >>>>> >>>>> HTHAL? >>>>> >>>>> Kevin >>>>> >>>>>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>>>>> wrote: >>>>>> >>>>>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>>>>> refresh hardware. Any lessons learned in this process? Is it >>>>>> easiest to just build new cluster and then use AFM? Add to existing >>>>>> cluster then decommission nodes? What is the recommended process for >>>>>> this? >>>>>> >>>>>> >>>>>> Mark >>>>>> >>>>>> This message (including any attachments) is intended only for the use >>>>>> of the individual or entity to which it is addressed and may contain >>>>>> information that is non-public, proprietary, privileged, >>>>>> confidential, and exempt from disclosure under applicable law. If you >>>>>> are not the intended recipient, you are hereby notified that any use, >>>>>> dissemination, distribution, or copying of this communication is >>>>>> strictly prohibited. This message may be viewed by parties at Sirius >>>>>> Computer Solutions other than those named in the message header. This >>>>>> message does not contain an official representation of Sirius >>>>>> Computer Solutions. If you have received this communication in error, >>>>>> notify Sirius Computer Solutions immediately and (i) destroy this >>>>>> message if a facsimile or (ii) delete this message immediately if >>>>>> this is an electronic communication. Thank you. >>>>>> >>>>>> Sirius Computer Solutions >>>>>> _______________________________________________ >>>>>> gpfsug-discuss mailing list >>>>>> gpfsug-discuss at spectrumscale.org >>>>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>>>> >>>>> ? >>>>> Kevin Buterbaugh - Senior System Administrator >>>>> Vanderbilt University - Advanced Computing Center for Research and >>>>> Education >>>>> Kevin.Buterbaugh at vanderbilt.edu >>>>> - (615)875-9633 >>>>> >>>>> >>>>> >>>> >>>> Ellei edell? ole toisin mainittu: / Unless stated otherwise above: >>>> Oy IBM Finland Ab >>>> PL 265, 00101 Helsinki, Finland >>>> Business ID, Y-tunnus: 0195876-3 >>>> Registered in Finland >>>> >>>> >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> >>> >>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From makaplan at us.ibm.com Wed Oct 12 18:54:52 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Wed, 12 Oct 2016 13:54:52 -0400 Subject: [gpfsug-discuss] Hardware refresh - In-Reply-To: <279BAB6D-21E6-4EC1-98CD-CA409A24F8F6@ulmer.org> References: <9E337B56-A274-41B6-84C2-7084AAB771F4@siriuscom.com> <279BAB6D-21E6-4EC1-98CD-CA409A24F8F6@ulmer.org> Message-ID: Yes, you can AFM within a single cluster, in fact with just a single node. I just set this up on my toy system: [root at bog-wifi cmvc]# mmlsfileset yy afmlu --afm Filesets in file system 'yy': Name Status Path afmTarget afmlu Linked /yy/afmlu gpfs:///xx [root at bog-wifi cmvc]# mount ... yy on /yy type gpfs (rw,relatime,seclabel) xx on /xx type gpfs (rw,relatime,seclabel) [root at bog-wifi cmvc]# mmafmctl yy getstate Fileset Name Fileset Target Cache State Gateway Node Queue Length Queue numExec ------------ -------------- ------------- ------------ ------------ ------------- afmlu gpfs:///xx Active bog-wifi 0 7 So, you may add nodes, add disks to an existing cluster, upgrade your software, define a new FS, migrate data from an old FS to a new FS then delete nodes and disks that are no longer needed... From: Stephen Ulmer To: gpfsug main discussion list Date: 10/11/2016 09:30 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org I think that the OP was asking why not expand the existing cluster with the new hardware, and just make a new FS? I?ve not tried to make a cluster talk AFM to itself yet. If that?s impossible, then there?s one good reason to make a new cluster (to use AFM for migration). Liberty, -- Stephen On Oct 11, 2016, at 8:40 PM, Mark.Bush at siriuscom.com wrote: Only compelling reason for new cluster would be old hardware is EOL or no longer want to pay maintenance on it. From: on behalf of Marc A Kaplan Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 2:58 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh New FS? Yes there are some good reasons. New cluster? I did not see a compelling argument either way. From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Date: 10/11/2016 03:34 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org Ok. I think I am hearing that a new cluster with a new FS and copying data from old to new cluster is the best way forward. Thanks everyone for your input. From: on behalf of Yuri L Volobuev Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 12:22 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh This depends on the committed cluster version level (minReleaseLevel) and file system format. Since NFSv2 is an on-disk format change, older code wouldn't be able to understand what it is, and thus if there's a possibility of a downlevel node looking at the NSD, the NFSv1 format is going to be used. The code does NSDv1<->NSDv2 conversions under the covers as needed when adding an empty NSD to a file system. I'd strongly recommend getting a fresh start by formatting a new file system. Many things have changed over the course of the last few years. In particular, having a 4K-aligned file system can be a pretty big deal, depending on what hardware one is going to deploy in the future, and this is something that can't be bolted onto an existing file system. Having 4K inodes is very handy for many reasons. New directory format and NSD format changes are attractive, too. And disks generally tend to get larger with time, and at some point you may want to add a disk to an existing storage pool that's larger than the existing allocation map format allows. Obviously, it's more hassle to migrate data to a new file system, as opposed to extending an existing one. In a perfect world, GPFS would offer a conversion tool that seamlessly and robustly converts old file systems, making them as good as new, but in the real world such a tool doesn't exist. Getting a clean slate by formatting a new file system every few years is a good long-term investment of time, although it comes front-loaded with extra work. yuri Aaron Knister ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron From: Aaron Knister To: , Date: 10/10/2016 04:45 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron On 10/10/16 7:40 PM, Luis Bolinches wrote: > Hi > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > reason to do so. > > AFM for migrations is quite good, latest versions allows to use NSD > protocol for mounts as well. Olaf did a great job explaining this > scenario on the redbook chapter 6 > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > -- > Cheers > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > wrote: > >> Hi Mark, >> >> The last time we did something like this was 2010 (we?re doing rolling >> refreshes now), so there are probably lots of better ways to do this >> than what we did, but we: >> >> 1) set up the new hardware >> 2) created new filesystems (so that we could make adjustments we >> wanted to make that can only be made at FS creation time) >> 3) used rsync to make a 1st pass copy of everything >> 4) coordinated a time with users / groups to do a 2nd rsync when they >> weren?t active >> 5) used symbolic links during the transition (i.e. rm -rvf >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >> 6) once everybody was migrated, updated the symlinks (i.e. /home >> became a symlink to /gpfs2/home) >> >> HTHAL? >> >> Kevin >> >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>> wrote: >>> >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>> refresh hardware. Any lessons learned in this process? Is it >>> easiest to just build new cluster and then use AFM? Add to existing >>> cluster then decommission nodes? What is the recommended process for >>> this? >>> >>> >>> Mark >>> >>> This message (including any attachments) is intended only for the use >>> of the individual or entity to which it is addressed and may contain >>> information that is non-public, proprietary, privileged, >>> confidential, and exempt from disclosure under applicable law. If you >>> are not the intended recipient, you are hereby notified that any use, >>> dissemination, distribution, or copying of this communication is >>> strictly prohibited. This message may be viewed by parties at Sirius >>> Computer Solutions other than those named in the message header. This >>> message does not contain an official representation of Sirius >>> Computer Solutions. If you have received this communication in error, >>> notify Sirius Computer Solutions immediately and (i) destroy this >>> message if a facsimile or (ii) delete this message immediately if >>> this is an electronic communication. Thank you. >>> >>> Sirius Computer Solutions >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and >> Education >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mark.Bush at siriuscom.com Wed Oct 12 21:19:19 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Wed, 12 Oct 2016 20:19:19 +0000 Subject: [gpfsug-discuss] Spectrum Scale RAID usable space Message-ID: <5B378555-7956-40A1-8840-8090B43CAC7F@siriuscom.com> Anyone know how to calculate this? I realize this is an ESS thing now but I wonder if someone on here knows how to extrapolate usable space. Thanks Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From abeattie at au1.ibm.com Wed Oct 12 22:58:04 2016 From: abeattie at au1.ibm.com (Andrew Beattie) Date: Wed, 12 Oct 2016 21:58:04 +0000 Subject: [gpfsug-discuss] Spectrum Scale RAID usable space In-Reply-To: <5B378555-7956-40A1-8840-8090B43CAC7F@siriuscom.com> Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.14763088875930.png Type: image/png Size: 177376 bytes Desc: not available URL: From Mark.Bush at siriuscom.com Wed Oct 12 23:25:48 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Wed, 12 Oct 2016 22:25:48 +0000 Subject: [gpfsug-discuss] Spectrum Scale RAID usable space In-Reply-To: References: <5B378555-7956-40A1-8840-8090B43CAC7F@siriuscom.com> Message-ID: <7407A861-BC0F-40EA-97F9-24FAEA00C838@siriuscom.com> GL2 with 6TB drives. From: on behalf of Andrew Beattie Reply-To: gpfsug main discussion list Date: Wednesday, October 12, 2016 at 4:58 PM To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] Spectrum Scale RAID usable space Hi Mark, If you let me know which model ESS and what drive size I can give you the possible capacities. there are effectively 4 options 8+2P 8+3P 3-way replication 4-way replication for example: GL6 - 8TB drives Possible capacity options: 8+2P - 2,036TB 8+3P - 1,851TB 3 Way - 848TB 4 Way - 636TB [cid:image002.png at 01D224AD.AB668310] Business Partners: http://www.ibm.com/partnerworld/wps/servlet/ContentHandler/SSPQ048068H83479I86 There is a ESS capacity calculator in the same set of BP links with Capacity Magic and Disk magic Andrew Beattie Software Defined Storage - IT Specialist Phone: 614-2133-7927 E-mail: abeattie at au1.ibm.com ----- Original message ----- From: "Mark.Bush at siriuscom.com" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list Cc: Subject: [gpfsug-discuss] Spectrum Scale RAID usable space Date: Thu, Oct 13, 2016 7:19 AM Anyone know how to calculate this? I realize this is an ESS thing now but I wonder if someone on here knows how to extrapolate usable space. Thanks Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.png Type: image/png Size: 170988 bytes Desc: image002.png URL: From ulmer at ulmer.org Thu Oct 13 01:48:23 2016 From: ulmer at ulmer.org (Stephen Ulmer) Date: Wed, 12 Oct 2016 20:48:23 -0400 Subject: [gpfsug-discuss] Hardware refresh - In-Reply-To: References: <9E337B56-A274-41B6-84C2-7084AAB771F4@siriuscom.com> <279BAB6D-21E6-4EC1-98CD-CA409A24F8F6@ulmer.org> Message-ID: <1F6BD2DF-10F4-456B-9061-67D6D6B90B3D@ulmer.org> Nice! -- Stephen Ulmer Sent from a mobile device; please excuse auto-correct silliness. > On Oct 12, 2016, at 1:54 PM, Marc A Kaplan wrote: > > Yes, you can AFM within a single cluster, in fact with just a single node. I just set this up on my toy system: > > [root at bog-wifi cmvc]# mmlsfileset yy afmlu --afm > Filesets in file system 'yy': > Name Status Path afmTarget > afmlu Linked /yy/afmlu gpfs:///xx > > [root at bog-wifi cmvc]# mount > ... > yy on /yy type gpfs (rw,relatime,seclabel) > xx on /xx type gpfs (rw,relatime,seclabel) > > [root at bog-wifi cmvc]# mmafmctl yy getstate > Fileset Name Fileset Target Cache State Gateway Node Queue Length Queue numExec > ------------ -------------- ------------- ------------ ------------ ------------- > afmlu gpfs:///xx Active bog-wifi 0 7 > > So, you may add nodes, add disks to an existing cluster, upgrade your software, define a new FS, > migrate data from an old FS to a new FS > then delete nodes and disks that are no longer needed... > > > > From: Stephen Ulmer > To: gpfsug main discussion list > Date: 10/11/2016 09:30 PM > Subject: Re: [gpfsug-discuss] Hardware refresh > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > I think that the OP was asking why not expand the existing cluster with the new hardware, and just make a new FS? > > I?ve not tried to make a cluster talk AFM to itself yet. If that?s impossible, then there?s one good reason to make a new cluster (to use AFM for migration). > > Liberty, > > -- > Stephen > > > > On Oct 11, 2016, at 8:40 PM, Mark.Bush at siriuscom.comwrote: > > Only compelling reason for new cluster would be old hardware is EOL or no longer want to pay maintenance on it. > > From: on behalf of Marc A Kaplan > Reply-To: gpfsug main discussion list > Date: Tuesday, October 11, 2016 at 2:58 PM > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Hardware refresh > > New FS? Yes there are some good reasons. > New cluster? I did not see a compelling argument either way. > > > > From: "Mark.Bush at siriuscom.com" > To: gpfsug main discussion list > Date: 10/11/2016 03:34 PM > Subject: Re: [gpfsug-discuss] Hardware refresh > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > > Ok. I think I am hearing that a new cluster with a new FS and copying data from old to new cluster is the best way forward. Thanks everyone for your input. > > From: on behalf of Yuri L Volobuev > Reply-To: gpfsug main discussion list > Date: Tuesday, October 11, 2016 at 12:22 PM > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Hardware refresh > > This depends on the committed cluster version level (minReleaseLevel) and file system format. Since NFSv2 is an on-disk format change, older code wouldn't be able to understand what it is, and thus if there's a possibility of a downlevel node looking at the NSD, the NFSv1 format is going to be used. The code does NSDv1<->NSDv2 conversions under the covers as needed when adding an empty NSD to a file system. > > I'd strongly recommend getting a fresh start by formatting a new file system. Many things have changed over the course of the last few years. In particular, having a 4K-aligned file system can be a pretty big deal, depending on what hardware one is going to deploy in the future, and this is something that can't be bolted onto an existing file system. Having 4K inodes is very handy for many reasons. New directory format and NSD format changes are attractive, too. And disks generally tend to get larger with time, and at some point you may want to add a disk to an existing storage pool that's larger than the existing allocation map format allows. Obviously, it's more hassle to migrate data to a new file system, as opposed to extending an existing one. In a perfect world, GPFS would offer a conversion tool that seamlessly and robustly converts old file systems, making them as good as new, but in the real world such a tool doesn't exist. Getting a clean slate by formatting a new file system every few years is a good long-term investment of time, although it comes front-loaded with extra work. > > yuri > > Aaron Knister ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron > > From: Aaron Knister > To: , > Date: 10/10/2016 04:45 PM > Subject: Re: [gpfsug-discuss] Hardware refresh > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > > > > Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? > > -Aaron > > On 10/10/16 7:40 PM, Luis Bolinches wrote: > > Hi > > > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > > reason to do so. > > > > AFM for migrations is quite good, latest versions allows to use NSD > > protocol for mounts as well. Olaf did a great job explaining this > > scenario on the redbook chapter 6 > > > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > > > -- > > Cheers > > > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > > > wrote: > > > >> Hi Mark, > >> > >> The last time we did something like this was 2010 (we?re doing rolling > >> refreshes now), so there are probably lots of better ways to do this > >> than what we did, but we: > >> > >> 1) set up the new hardware > >> 2) created new filesystems (so that we could make adjustments we > >> wanted to make that can only be made at FS creation time) > >> 3) used rsync to make a 1st pass copy of everything > >> 4) coordinated a time with users / groups to do a 2nd rsync when they > >> weren?t active > >> 5) used symbolic links during the transition (i.e. rm -rvf > >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) > >> 6) once everybody was migrated, updated the symlinks (i.e. /home > >> became a symlink to /gpfs2/home) > >> > >> HTHAL? > >> > >> Kevin > >> > >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com > >>> wrote: > >>> > >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to > >>> refresh hardware. Any lessons learned in this process? Is it > >>> easiest to just build new cluster and then use AFM? Add to existing > >>> cluster then decommission nodes? What is the recommended process for > >>> this? > >>> > >>> > >>> Mark > >>> > >>> This message (including any attachments) is intended only for the use > >>> of the individual or entity to which it is addressed and may contain > >>> information that is non-public, proprietary, privileged, > >>> confidential, and exempt from disclosure under applicable law. If you > >>> are not the intended recipient, you are hereby notified that any use, > >>> dissemination, distribution, or copying of this communication is > >>> strictly prohibited. This message may be viewed by parties at Sirius > >>> Computer Solutions other than those named in the message header. This > >>> message does not contain an official representation of Sirius > >>> Computer Solutions. If you have received this communication in error, > >>> notify Sirius Computer Solutions immediately and (i) destroy this > >>> message if a facsimile or (ii) delete this message immediately if > >>> this is an electronic communication. Thank you. > >>> > >>> Sirius Computer Solutions > >>> _______________________________________________ > >>> gpfsug-discuss mailing list > >>> gpfsug-discuss at spectrumscale.org > >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > >> ? > >> Kevin Buterbaugh - Senior System Administrator > >> Vanderbilt University - Advanced Computing Center for Research and > >> Education > >> Kevin.Buterbaugh at vanderbilt.edu > >> - (615)875-9633 > >> > >> > >> > > > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > > Oy IBM Finland Ab > > PL 265, 00101 Helsinki, Finland > > Business ID, Y-tunnus: 0195876-3 > > Registered in Finland > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. > > Sirius Computer Solutions_______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From shankbal at in.ibm.com Thu Oct 13 11:43:23 2016 From: shankbal at in.ibm.com (Shankar Balasubramanian) Date: Thu, 13 Oct 2016 16:13:23 +0530 Subject: [gpfsug-discuss] Hardware refresh - In-Reply-To: References: <9E337B56-A274-41B6-84C2-7084AAB771F4@siriuscom.com><279BAB6D-21E6-4EC1-98CD-CA409A24F8F6@ulmer.org> Message-ID: Please note - though one of the supported use case of AFM is such migration scenarios, infact AFM does particularly well in faithfully migrating the data when the source and destination cluster are GPFS, the scalability of this solution for multi million/multi terabyte file systems has its own set of challenges. These have to carefully understood and checked if AFM will fit the bill. Best Regards, Shankar Balasubramanian STSM, AFM & Async DR Development IBM Systems Bangalore - Embassy Golf Links India From: "Marc A Kaplan" To: gpfsug main discussion list Date: 10/12/2016 11:25 PM Subject: Re: [gpfsug-discuss] Hardware refresh - Sent by: gpfsug-discuss-bounces at spectrumscale.org Yes, you can AFM within a single cluster, in fact with just a single node. I just set this up on my toy system: [root at bog-wifi cmvc]# mmlsfileset yy afmlu --afm Filesets in file system 'yy': Name Status Path afmTarget afmlu Linked /yy/afmlu gpfs:///xx [root at bog-wifi cmvc]# mount ... yy on /yy type gpfs (rw,relatime,seclabel) xx on /xx type gpfs (rw,relatime,seclabel) [root at bog-wifi cmvc]# mmafmctl yy getstate Fileset Name Fileset Target Cache State Gateway Node Queue Length Queue numExec ------------ -------------- ------------- ------------ ------------ ------------- afmlu gpfs:///xx Active bog-wifi 0 7 So, you may add nodes, add disks to an existing cluster, upgrade your software, define a new FS, migrate data from an old FS to a new FS then delete nodes and disks that are no longer needed... From: Stephen Ulmer To: gpfsug main discussion list Date: 10/11/2016 09:30 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org I think that the OP was asking why not expand the existing cluster with the new hardware, and just make a new FS? I?ve not tried to make a cluster talk AFM to itself yet. If that?s impossible, then there?s one good reason to make a new cluster (to use AFM for migration). Liberty, -- Stephen On Oct 11, 2016, at 8:40 PM, Mark.Bush at siriuscom.comwrote: Only compelling reason for new cluster would be old hardware is EOL or no longer want to pay maintenance on it. From: on behalf of Marc A Kaplan Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 2:58 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh New FS? Yes there are some good reasons. New cluster? I did not see a compelling argument either way. From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Date: 10/11/2016 03:34 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org Ok. I think I am hearing that a new cluster with a new FS and copying data from old to new cluster is the best way forward. Thanks everyone for your input. From: on behalf of Yuri L Volobuev Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 12:22 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh This depends on the committed cluster version level (minReleaseLevel) and file system format. Since NFSv2 is an on-disk format change, older code wouldn't be able to understand what it is, and thus if there's a possibility of a downlevel node looking at the NSD, the NFSv1 format is going to be used. The code does NSDv1<->NSDv2 conversions under the covers as needed when adding an empty NSD to a file system. I'd strongly recommend getting a fresh start by formatting a new file system. Many things have changed over the course of the last few years. In particular, having a 4K-aligned file system can be a pretty big deal, depending on what hardware one is going to deploy in the future, and this is something that can't be bolted onto an existing file system. Having 4K inodes is very handy for many reasons. New directory format and NSD format changes are attractive, too. And disks generally tend to get larger with time, and at some point you may want to add a disk to an existing storage pool that's larger than the existing allocation map format allows. Obviously, it's more hassle to migrate data to a new file system, as opposed to extending an existing one. In a perfect world, GPFS would offer a conversion tool that seamlessly and robustly converts old file systems, making them as good as new, but in the real world such a tool doesn't exist. Getting a clean slate by formatting a new file system every few years is a good long-term investment of time, although it comes front-loaded with extra work. yuri Aaron Knister ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron From: Aaron Knister To: , Date: 10/10/2016 04:45 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron On 10/10/16 7:40 PM, Luis Bolinches wrote: > Hi > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > reason to do so. > > AFM for migrations is quite good, latest versions allows to use NSD > protocol for mounts as well. Olaf did a great job explaining this > scenario on the redbook chapter 6 > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > -- > Cheers > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > wrote: > >> Hi Mark, >> >> The last time we did something like this was 2010 (we?re doing rolling >> refreshes now), so there are probably lots of better ways to do this >> than what we did, but we: >> >> 1) set up the new hardware >> 2) created new filesystems (so that we could make adjustments we >> wanted to make that can only be made at FS creation time) >> 3) used rsync to make a 1st pass copy of everything >> 4) coordinated a time with users / groups to do a 2nd rsync when they >> weren?t active >> 5) used symbolic links during the transition (i.e. rm -rvf >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >> 6) once everybody was migrated, updated the symlinks (i.e. /home >> became a symlink to /gpfs2/home) >> >> HTHAL? >> >> Kevin >> >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>> wrote: >>> >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>> refresh hardware. Any lessons learned in this process? Is it >>> easiest to just build new cluster and then use AFM? Add to existing >>> cluster then decommission nodes? What is the recommended process for >>> this? >>> >>> >>> Mark >>> >>> This message (including any attachments) is intended only for the use >>> of the individual or entity to which it is addressed and may contain >>> information that is non-public, proprietary, privileged, >>> confidential, and exempt from disclosure under applicable law. If you >>> are not the intended recipient, you are hereby notified that any use, >>> dissemination, distribution, or copying of this communication is >>> strictly prohibited. This message may be viewed by parties at Sirius >>> Computer Solutions other than those named in the message header. This >>> message does not contain an official representation of Sirius >>> Computer Solutions. If you have received this communication in error, >>> notify Sirius Computer Solutions immediately and (i) destroy this >>> message if a facsimile or (ii) delete this message immediately if >>> this is an electronic communication. Thank you. >>> >>> Sirius Computer Solutions >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and >> Education >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From mimarsh2 at vt.edu Thu Oct 13 14:30:03 2016 From: mimarsh2 at vt.edu (Brian Marshall) Date: Thu, 13 Oct 2016 09:30:03 -0400 Subject: [gpfsug-discuss] SC16 Draft Agenda [Save the date: Sunday, November 13th] In-Reply-To: References: Message-ID: Virginia Tech ARC purchased a SANDisk IF150 (IBM DeepFlash 150) this Summer and are just getting it into production (maybe next week). I have some dev benchmarking numbers and may have production/user feedback by SC16. I'm not sure I can fill an entire 30 minutes with worthwhile information, but I can give at least 15 minutes on: DeepFlash: a computational scientist's first impressions Best, Brian Marshall On Mon, Oct 10, 2016 at 6:59 PM, GPFS UG USA Principal < usa-principal at gpfsug.org> wrote: > Hello all, > > There have been some questions about the Spectrum Scale Users Group event > for SC16 and we thought it best to publish our draft agenda and continue to > fill it in as details get settled. That way you can make your travel plans > and schedules. > > The date and rough times are set, we may adjust the time range a bit > depending upon the number of user talks that people would like to give. So > note, yes! *we are asking for user talks. Please consider doing this. *We > always receive positive feedback about talks given by users that describe > real world experiences. If *any* of the user talk slots below are of > interest to you, please let us know as soon as you can. We may turn at > least one user talk into a panel if we can get enough participants. > > As always, your feedback is welcome. This is a *users* group after all. > > So, here?s what we have planned so far: > > *Date: Sunday November 13th* > *Venue: TBD, but near the convention center in downtown SLC* > *Time: ~12:30p - ~17:30p* > > *Agenda:* > > > > > > > > > *12:30 - 12:50 Welcome / Overview12:50 - 13:50 The latest in IBM Spectrum > Scale and ESS13:50 - 14:20 User Talk: Big Data in HPC14:20 - 14:50 User > Talk: Tiering to Object Storage14:50 - 15:20 - break -15:20 - 15:50 > User Talk: TBD ? volunteers! please! 15:50 - 16:35 Spectrum Scale > Enhancements for CORAL16:35 - 17:20 News from IBM Research17:20 - 17:30 > Closing* > > Best, > > Kristy & Bob > > Kristy Kallback-Rose > Manager, Research Storage > Indiana University > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Thu Oct 13 14:55:14 2016 From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]) Date: Thu, 13 Oct 2016 13:55:14 +0000 Subject: [gpfsug-discuss] SC16 Draft Agenda [Save the date: Sunday, November 13th] Message-ID: <5F910253243E6A47B81A9A2EB424BBA101D90737@NDMSMBX404.ndc.nasa.gov> I'd be quite interested in hearing about that Brian. I have to double check this with a couple folks here first but is there interest in hear about some of the stuff NASA has been up to with GPFS? I'd be tickled pink to share details on our GPFS activities: - FPO/Hadoop efforts - a site report (we're not *huge* but we're a modest Asize - 3600 nodes and almost 50PB at this point) - demoing the monitoring dashboard we've created with influxdb and grafana. - talk about the tools we've developed to deal with mass uid number changes on GPFS filesystems -Aaron From: Brian Marshall Sent: 10/13/16, 9:30 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] SC16 Draft Agenda [Save the date: Sunday, November 13th] Virginia Tech ARC purchased a SANDisk IF150 (IBM DeepFlash 150) this Summer and are just getting it into production (maybe next week). I have some dev benchmarking numbers and may have production/user feedback by SC16. I'm not sure I can fill an entire 30 minutes with worthwhile information, but I can give at least 15 minutes on: DeepFlash: a computational scientist's first impressions Best, Brian Marshall On Mon, Oct 10, 2016 at 6:59 PM, GPFS UG USA Principal > wrote: Hello all, There have been some questions about the Spectrum Scale Users Group event for SC16 and we thought it best to publish our draft agenda and continue to fill it in as details get settled. That way you can make your travel plans and schedules. The date and rough times are set, we may adjust the time range a bit depending upon the number of user talks that people would like to give. So note, yes! we are asking for user talks. Please consider doing this. We always receive positive feedback about talks given by users that describe real world experiences. If any of the user talk slots below are of interest to you, please let us know as soon as you can. We may turn at least one user talk into a panel if we can get enough participants. As always, your feedback is welcome. This is a users group after all. So, here?s what we have planned so far: Date: Sunday November 13th Venue: TBD, but near the convention center in downtown SLC Time: ~12:30p - ~17:30p Agenda: 12:30 - 12:50 Welcome / Overview 12:50 - 13:50 The latest in IBM Spectrum Scale and ESS 13:50 - 14:20 User Talk: Big Data in HPC 14:20 - 14:50 User Talk: Tiering to Object Storage 14:50 - 15:20 - break - 15:20 - 15:50 User Talk: TBD ? volunteers! please! 15:50 - 16:35 Spectrum Scale Enhancements for CORAL 16:35 - 17:20 News from IBM Research 17:20 - 17:30 Closing Best, Kristy & Bob Kristy Kallback-Rose Manager, Research Storage Indiana University _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew at ellexus.com Thu Oct 13 16:00:54 2016 From: matthew at ellexus.com (Matthew Harris) Date: Thu, 13 Oct 2016 16:00:54 +0100 Subject: [gpfsug-discuss] SC16 Draft Agenda [Save the date: Sunday, November 13th] In-Reply-To: References: Message-ID: Who is delivering the 12:50 talk please? Regards, Matthew Harris Account Manager & Business Development - Ellexus Ltd *www.ellexus.com * *Ellexus Ltd is a limited company registered in England & Wales* *Company registration no. 07166034* *Registered address: 198 High Street, Tonbridge, Kent TN9 1BE, UK* *Operating address: St John's Innovation Centre, Cowley Road, Cambridge CB4 0WS, UK* On Mon, Oct 10, 2016 at 11:59 PM, GPFS UG USA Principal < usa-principal at gpfsug.org> wrote: > Hello all, > > There have been some questions about the Spectrum Scale Users Group event > for SC16 and we thought it best to publish our draft agenda and continue to > fill it in as details get settled. That way you can make your travel plans > and schedules. > > The date and rough times are set, we may adjust the time range a bit > depending upon the number of user talks that people would like to give. So > note, yes! *we are asking for user talks. Please consider doing this. *We > always receive positive feedback about talks given by users that describe > real world experiences. If *any* of the user talk slots below are of > interest to you, please let us know as soon as you can. We may turn at > least one user talk into a panel if we can get enough participants. > > As always, your feedback is welcome. This is a *users* group after all. > > So, here?s what we have planned so far: > > *Date: Sunday November 13th* > *Venue: TBD, but near the convention center in downtown SLC* > *Time: ~12:30p - ~17:30p* > > *Agenda:* > > > > > > > > > *12:30 - 12:50 Welcome / Overview12:50 - 13:50 The latest in IBM Spectrum > Scale and ESS13:50 - 14:20 User Talk: Big Data in HPC14:20 - 14:50 User > Talk: Tiering to Object Storage14:50 - 15:20 - break -15:20 - 15:50 > User Talk: TBD ? volunteers! please! 15:50 - 16:35 Spectrum Scale > Enhancements for CORAL16:35 - 17:20 News from IBM Research17:20 - 17:30 > Closing* > > Best, > > Kristy & Bob > > Kristy Kallback-Rose > Manager, Research Storage > Indiana University > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Thu Oct 13 16:27:14 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Thu, 13 Oct 2016 11:27:14 -0400 Subject: [gpfsug-discuss] Hardware refresh - In-Reply-To: References: <9E337B56-A274-41B6-84C2-7084AAB771F4@siriuscom.com><279BAB6D-21E6-4EC1-98CD-CA409A24F8F6@ulmer.org> Message-ID: IMO, it is simplest to have both the old and new file systems both mounted on the same node(s). Then you can use AFM and/or any other utilities to migrate/copy your files from the old file system to the new. From: "Shankar Balasubramanian" To: gpfsug main discussion list Date: 10/13/2016 06:46 AM Subject: Re: [gpfsug-discuss] Hardware refresh - Sent by: gpfsug-discuss-bounces at spectrumscale.org Please note - though one of the supported use case of AFM is such migration scenarios, infact AFM does particularly well in faithfully migrating the data when the source and destination cluster are GPFS, the scalability of this solution for multi million/multi terabyte file systems has its own set of challenges. These have to carefully understood and checked if AFM will fit the bill. Best Regards, Shankar Balasubramanian STSM, AFM & Async DR Development IBM Systems Bangalore - Embassy Golf Links India "Marc A Kaplan" ---10/12/2016 11:25:20 PM---Yes, you can AFM within a single cluster, in fact with just a single node. I just set this up on m From: "Marc A Kaplan" To: gpfsug main discussion list Date: 10/12/2016 11:25 PM Subject: Re: [gpfsug-discuss] Hardware refresh - Sent by: gpfsug-discuss-bounces at spectrumscale.org Yes, you can AFM within a single cluster, in fact with just a single node. I just set this up on my toy system: [root at bog-wifi cmvc]# mmlsfileset yy afmlu --afm Filesets in file system 'yy': Name Status Path afmTarget afmlu Linked /yy/afmlu gpfs:///xx [root at bog-wifi cmvc]# mount ... yy on /yy type gpfs (rw,relatime,seclabel) xx on /xx type gpfs (rw,relatime,seclabel) [root at bog-wifi cmvc]# mmafmctl yy getstate Fileset Name Fileset Target Cache State Gateway Node Queue Length Queue numExec ------------ -------------- ------------- ------------ ------------ ------------- afmlu gpfs:///xx Active bog-wifi 0 7 So, you may add nodes, add disks to an existing cluster, upgrade your software, define a new FS, migrate data from an old FS to a new FS then delete nodes and disks that are no longer needed... From: Stephen Ulmer To: gpfsug main discussion list Date: 10/11/2016 09:30 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org I think that the OP was asking why not expand the existing cluster with the new hardware, and just make a new FS? I?ve not tried to make a cluster talk AFM to itself yet. If that?s impossible, then there?s one good reason to make a new cluster (to use AFM for migration). Liberty, -- Stephen On Oct 11, 2016, at 8:40 PM, Mark.Bush at siriuscom.comwrote: Only compelling reason for new cluster would be old hardware is EOL or no longer want to pay maintenance on it. From: on behalf of Marc A Kaplan Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 2:58 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh New FS? Yes there are some good reasons. New cluster? I did not see a compelling argument either way. From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Date: 10/11/2016 03:34 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org Ok. I think I am hearing that a new cluster with a new FS and copying data from old to new cluster is the best way forward. Thanks everyone for your input. From: on behalf of Yuri L Volobuev Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 12:22 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh This depends on the committed cluster version level (minReleaseLevel) and file system format. Since NFSv2 is an on-disk format change, older code wouldn't be able to understand what it is, and thus if there's a possibility of a downlevel node looking at the NSD, the NFSv1 format is going to be used. The code does NSDv1<->NSDv2 conversions under the covers as needed when adding an empty NSD to a file system. I'd strongly recommend getting a fresh start by formatting a new file system. Many things have changed over the course of the last few years. In particular, having a 4K-aligned file system can be a pretty big deal, depending on what hardware one is going to deploy in the future, and this is something that can't be bolted onto an existing file system. Having 4K inodes is very handy for many reasons. New directory format and NSD format changes are attractive, too. And disks generally tend to get larger with time, and at some point you may want to add a disk to an existing storage pool that's larger than the existing allocation map format allows. Obviously, it's more hassle to migrate data to a new file system, as opposed to extending an existing one. In a perfect world, GPFS would offer a conversion tool that seamlessly and robustly converts old file systems, making them as good as new, but in the real world such a tool doesn't exist. Getting a clean slate by formatting a new file system every few years is a good long-term investment of time, although it comes front-loaded with extra work. yuri Aaron Knister ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron From: Aaron Knister To: , Date: 10/10/2016 04:45 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron On 10/10/16 7:40 PM, Luis Bolinches wrote: > Hi > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > reason to do so. > > AFM for migrations is quite good, latest versions allows to use NSD > protocol for mounts as well. Olaf did a great job explaining this > scenario on the redbook chapter 6 > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > -- > Cheers > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > wrote: > >> Hi Mark, >> >> The last time we did something like this was 2010 (we?re doing rolling >> refreshes now), so there are probably lots of better ways to do this >> than what we did, but we: >> >> 1) set up the new hardware >> 2) created new filesystems (so that we could make adjustments we >> wanted to make that can only be made at FS creation time) >> 3) used rsync to make a 1st pass copy of everything >> 4) coordinated a time with users / groups to do a 2nd rsync when they >> weren?t active >> 5) used symbolic links during the transition (i.e. rm -rvf >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >> 6) once everybody was migrated, updated the symlinks (i.e. /home >> became a symlink to /gpfs2/home) >> >> HTHAL? >> >> Kevin >> >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>> wrote: >>> >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>> refresh hardware. Any lessons learned in this process? Is it >>> easiest to just build new cluster and then use AFM? Add to existing >>> cluster then decommission nodes? What is the recommended process for >>> this? >>> >>> >>> Mark >>> >>> This message (including any attachments) is intended only for the use >>> of the individual or entity to which it is addressed and may contain >>> information that is non-public, proprietary, privileged, >>> confidential, and exempt from disclosure under applicable law. If you >>> are not the intended recipient, you are hereby notified that any use, >>> dissemination, distribution, or copying of this communication is >>> strictly prohibited. This message may be viewed by parties at Sirius >>> Computer Solutions other than those named in the message header. This >>> message does not contain an official representation of Sirius >>> Computer Solutions. If you have received this communication in error, >>> notify Sirius Computer Solutions immediately and (i) destroy this >>> message if a facsimile or (ii) delete this message immediately if >>> this is an electronic communication. Thank you. >>> >>> Sirius Computer Solutions >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and >> Education >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 105 bytes Desc: not available URL: From aaron.s.knister at nasa.gov Fri Oct 14 03:01:32 2016 From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]) Date: Fri, 14 Oct 2016 02:01:32 +0000 Subject: [gpfsug-discuss] 4K sector NSD support (was: Hardware refresh) In-Reply-To: References: <63bd99ad-f684-d1bb-371f-e2c21849ff29@nasa.gov> , Message-ID: <5F910253243E6A47B81A9A2EB424BBA101D91022@NDMSMBX404.ndc.nasa.gov> Hi Yuri, Thank you for your excellent reply. The immediate issue ahead of us is getting the 4K LUNs formatted as NSDs into the filesystem and since we can do that life is good for a while. The next hurdle for me won't be for another 3 years when I'll likely need to bring into existing filesystems metadata NSDs that have 4k sectors. I wonder if in that time frame a migration tool could be developed to convert existing filesystems to 4K metadata? I do recognize what you're saying, though, about return on investment of effort from a developing a tool to migrate the on-disk format vs migrating the entire filesystem. What were you thinking as far as ways to strengthen the migration story? In an ideal world I'd like to be able to pull the trigger on a migration and have it have no visible impact to end-users other than an understandable performance impact. If a mechanism existed to move data to a new filesystem (a hypothetical mmreplacefs comes to mind) in-place, that would be quite wonderful. At a high level, perhaps one would create a new filesystem that wouldn't be directly mounted but would be an "internal mount". An admin would then issue an mmreplacefs $OLD_FS $NEW_FS command. All activity would be quiesced on the old filesystem and in the background an AFM-like process would be initiated. Files not on the new fs would be migrated in the background. Once the migration process is complete the old fs would perhaps be renamed and the new fs would effectively take its place. This raises more questions in my mind than it does provide answers, such as how quotas would be handled during the migration, how filesets would get migrated/created, how ILM/DMAPI would be affected during the migration. I'll give it some more thought, but understanding a little more about what you were thinking would help me craft the RFE. -Aaron ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Yuri L Volobuev [volobuev at us.ibm.com] Sent: Wednesday, October 12, 2016 1:44 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] 4K sector NSD support (was: Hardware refresh) Yes, it is possible to add a 4KN dataOnly NSD to a non-4K-aligned file system, as you figured out. This is something we didn't plan on doing originally, but then had to implement based on the feedback from the field. There's clearly a need for this. However, this statement is exactly it -- dataOnly NSDs. The only way to put metadata on a 4KN disk is to use a 4K-aligned file system. There are several kinds of metadata present in non-4K-aligned file system that generate non-4K IOs (512 byte inodes being the biggest problem), and there's no way to work around this short of using the new format, and there's no way to perform a conversion to the new format in-place. You're welcome to submit an RFE, of course, but I'd recommend being pragmatic about the chances of such an RFE being implemented. As you can imagine, the main reason why an all-encompassing file system conversion tool doesn't exist is not GPFS developers having no idea that such a tool is wanted. There are several considerations that conspire to make this an unlikely candidate to ever be implemented: 1) The task is hard and has no finish line. In most GPFS releases, something changes, necessitating an added piece of work for the hypothetical conversion tool, and the matrix of from-to format version combinations gets to be big quite quickly. 2) A file system conversion is something that is needed very infrequently, but when this code does run, it absolutely has to run and run perfectly, else the result would be a half-converted file system, i.e. a royal mess. This is a tester's nightmare. 3) The failure scenarios are all unpalatable. What should the conversion tool do if it runs out of space replacing smaller metadata structures with bigger ones? Undoing a partially finished conversion is even harder than doing it in the first place. 4) Doing an on-disk conversion on-line is simply very difficult. Consider the task of converting an inode file to use a different inode size. The file can be huge (billions of records), and it would take a fair chunk of time to rewrite it, but the file is changing while it's being converted (can't simply lock the whole thing down for so long), simultaneously on multiple nodes. Orchestrating the processing of updates in the presence of two inode files, with proper atomicity guarantees (to guard against a node failure) is a task of considerable complexity. None of this means the task is impossible, of course. It is, however, a very big chunk of very complex work, all towards a tool that on an average cluster may run somewhere between zero and one times, not something that benefits day-to-day operations. Where the complexity of the task allows for a reasonably affordable implementation, e.g. conversion from an old-style EA file to the FASTEA format, a conversion tool has been implemented (mmmigratefs). However, doing this for every single changed aspect of the file system format is simply too expensive to justify, given other tasks in front of us. On the other hand, a well-implemented migration mechanism solves the file system reformatting scenario (which covers all aspects of file system format changes) as well as a number of other scenarios. This is a cleaner, more general solution. Migration doesn't have to mean an outage. A simple rsync-based migration requires downtime for a cutover, while an AFM-based migration doesn't necessarily require one. I'm not saying that GPFS has a particularly strong migration story at the moment, but this is a much more productive direction for applying resources than a mythical all-encompassing conversion tool. yuri [Inactive hide details for Aaron Knister ---10/11/2016 05:59:25 PM---Yuri, (Sorry for being somewhat spammy) I now understand th]Aaron Knister ---10/11/2016 05:59:25 PM---Yuri, (Sorry for being somewhat spammy) I now understand the limitation after From: Aaron Knister To: , Date: 10/11/2016 05:59 PM Subject: Re: [gpfsug-discuss] 4K sector NSD support (was: Hardware refresh) Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Yuri, (Sorry for being somewhat spammy) I now understand the limitation after some more testing (I'm a hands-on learner, can you tell?). Given the right code/cluster/fs version levels I can add 4K dataOnly NSDv2 NSDs to a filesystem created with NSDv1 NSDs. What I can't do is seemingly add any metadataOnly or dataAndMetadata 4K luns to an fs that is not 4K aligned which I assume would be any fs originally created with NSDv1 LUNs. It seems possible to move all data away from NSDv1 LUNS in a filesystem behind-the-scenes using GPFS migration tools, and move the data to NSDv2 LUNs. In this case I believe what's missing is a tool to convert just the metadata structures to be 4K aligned since the data would already on 4k-based NSDv2 LUNS, is that the case? I'm trying to figure out what exactly I'm asking for in an RFE. -Aaron On 10/11/16 7:57 PM, Aaron Knister wrote: > I think I was a little quick to the trigger. I re-read your last mail > after doing some testing and understand it differently. I was wrong > about my interpretation-- you can add 4K NSDv2 formatted NSDs to a > filesystem previously created with NSDv1 NSDs assuming, as you say, the. > minReleaseLevel and filesystem version are high enough. That negates > about half of my last e-mail. The fs still doesn't show as 4K aligned: > > loressd01:~ # /usr/lpp/mmfs/bin/mmlsfs tnb4k --is4KAligned > flag value description > ------------------- ------------------------ > ----------------------------------- > --is4KAligned No is4KAligned? > > but *shrug* most of the I/O to these disks should be 1MB anyway. If > somebody is pounding the FS with smaller than 4K I/O they're gonna get a > talkin' to. > > -Aaron > > On 10/11/16 6:41 PM, Aaron Knister wrote: >> Thanks Yuri. >> >> I'm asking for my own purposes but I think it's still relevant here: >> we're still at GPFS 3.5 and will be adding dataOnly NSDs with 4K sectors >> in the near future. We're planning to update to 4.1 before we format >> these NSDs, though. If I understand you correctly we can't bring these >> 4K NSDv2 NSDs into a filesystem with 512b-based NSDv1 NSDs? That's a >> pretty big deal :( >> >> Reformatting every few years with 10's of petabytes of data is not >> realistic for us (it would take years to move the data around). It also >> goes against my personal preachings about GPFS's storage virtualization >> capabilities: the ability to perform upgrades/make underlying storage >> infrastructure changes with behind-the-scenes data migration, >> eliminating much of the manual hassle of storage administrators doing >> rsync dances. I guess it's RFE time? It also seems as though AFM could >> help with automating the migration, although many of our filesystems do >> not have filesets on them so we would have to re-think how we lay out >> our filesystems. >> >> This is also curious to me with IBM pitching GPFS as a filesystem for >> cloud services (the cloud *never* goes down, right?). Granted I believe >> this pitch started after the NSDv2 format was defined, but if somebody >> is building a large cloud with GPFS as the underlying filesystem for an >> object or an image store one might think the idea of having to re-format >> the filesystem to gain access to critical new features is inconsistent >> with this pitch. It would be hugely impactful. Just my $.02. >> >> As you can tell, I'm frustrated there's no online conversion tool :) Not >> that there couldn't be... you all are brilliant developers. >> >> -Aaron >> >> On 10/11/16 1:22 PM, Yuri L Volobuev wrote: >>> This depends on the committed cluster version level (minReleaseLevel) >>> and file system format. Since NFSv2 is an on-disk format change, older >>> code wouldn't be able to understand what it is, and thus if there's a >>> possibility of a downlevel node looking at the NSD, the NFSv1 format is >>> going to be used. The code does NSDv1<->NSDv2 conversions under the >>> covers as needed when adding an empty NSD to a file system. >>> >>> I'd strongly recommend getting a fresh start by formatting a new file >>> system. Many things have changed over the course of the last few years. >>> In particular, having a 4K-aligned file system can be a pretty big deal, >>> depending on what hardware one is going to deploy in the future, and >>> this is something that can't be bolted onto an existing file system. >>> Having 4K inodes is very handy for many reasons. New directory format >>> and NSD format changes are attractive, too. And disks generally tend to >>> get larger with time, and at some point you may want to add a disk to an >>> existing storage pool that's larger than the existing allocation map >>> format allows. Obviously, it's more hassle to migrate data to a new file >>> system, as opposed to extending an existing one. In a perfect world, >>> GPFS would offer a conversion tool that seamlessly and robustly converts >>> old file systems, making them as good as new, but in the real world such >>> a tool doesn't exist. Getting a clean slate by formatting a new file >>> system every few years is a good long-term investment of time, although >>> it comes front-loaded with extra work. >>> >>> yuri >>> >>> Inactive hide details for Aaron Knister ---10/10/2016 04:45:31 PM---Can >>> one format NSDv2 NSDs and put them in a filesystem withAaron Knister >>> ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a >>> filesystem with NSDv1 NSD's? -Aaron >>> >>> From: Aaron Knister >>> To: , >>> Date: 10/10/2016 04:45 PM >>> Subject: Re: [gpfsug-discuss] Hardware refresh >>> Sent by: gpfsug-discuss-bounces at spectrumscale.org >>> >>> ------------------------------------------------------------------------ >>> >>> >>> >>> Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? >>> >>> -Aaron >>> >>> On 10/10/16 7:40 PM, Luis Bolinches wrote: >>>> Hi >>>> >>>> Creating a new FS sounds like a best way to go. NSDv2 being a very good >>>> reason to do so. >>>> >>>> AFM for migrations is quite good, latest versions allows to use NSD >>>> protocol for mounts as well. Olaf did a great job explaining this >>>> scenario on the redbook chapter 6 >>>> >>>> http://www.redbooks.ibm.com/abstracts/sg248254.html?Open >>>> >>>> -- >>>> Cheers >>>> >>>> On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L >>>> >>> > wrote: >>>> >>>>> Hi Mark, >>>>> >>>>> The last time we did something like this was 2010 (we?re doing rolling >>>>> refreshes now), so there are probably lots of better ways to do this >>>>> than what we did, but we: >>>>> >>>>> 1) set up the new hardware >>>>> 2) created new filesystems (so that we could make adjustments we >>>>> wanted to make that can only be made at FS creation time) >>>>> 3) used rsync to make a 1st pass copy of everything >>>>> 4) coordinated a time with users / groups to do a 2nd rsync when they >>>>> weren?t active >>>>> 5) used symbolic links during the transition (i.e. rm -rvf >>>>> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >>>>> 6) once everybody was migrated, updated the symlinks (i.e. /home >>>>> became a symlink to /gpfs2/home) >>>>> >>>>> HTHAL? >>>>> >>>>> Kevin >>>>> >>>>>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>>>>> wrote: >>>>>> >>>>>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>>>>> refresh hardware. Any lessons learned in this process? Is it >>>>>> easiest to just build new cluster and then use AFM? Add to existing >>>>>> cluster then decommission nodes? What is the recommended process for >>>>>> this? >>>>>> >>>>>> >>>>>> Mark >>>>>> >>>>>> This message (including any attachments) is intended only for the use >>>>>> of the individual or entity to which it is addressed and may contain >>>>>> information that is non-public, proprietary, privileged, >>>>>> confidential, and exempt from disclosure under applicable law. If you >>>>>> are not the intended recipient, you are hereby notified that any use, >>>>>> dissemination, distribution, or copying of this communication is >>>>>> strictly prohibited. This message may be viewed by parties at Sirius >>>>>> Computer Solutions other than those named in the message header. This >>>>>> message does not contain an official representation of Sirius >>>>>> Computer Solutions. If you have received this communication in error, >>>>>> notify Sirius Computer Solutions immediately and (i) destroy this >>>>>> message if a facsimile or (ii) delete this message immediately if >>>>>> this is an electronic communication. Thank you. >>>>>> >>>>>> Sirius Computer Solutions >>>>>> _______________________________________________ >>>>>> gpfsug-discuss mailing list >>>>>> gpfsug-discuss at spectrumscale.org >>>>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>>>> >>>>> ? >>>>> Kevin Buterbaugh - Senior System Administrator >>>>> Vanderbilt University - Advanced Computing Center for Research and >>>>> Education >>>>> Kevin.Buterbaugh at vanderbilt.edu >>>>> - (615)875-9633 >>>>> >>>>> >>>>> >>>> >>>> Ellei edell? ole toisin mainittu: / Unless stated otherwise above: >>>> Oy IBM Finland Ab >>>> PL 265, 00101 Helsinki, Finland >>>> Business ID, Y-tunnus: 0195876-3 >>>> Registered in Finland >>>> >>>> >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> >>> >>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: graycol.gif URL: From daniel.kidger at uk.ibm.com Fri Oct 14 11:09:18 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Fri, 14 Oct 2016 10:09:18 +0000 Subject: [gpfsug-discuss] testing commands In-Reply-To: <797ae533-edee-102a-3f3e-f53687cfcf83@nasa.gov> Message-ID: An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Fri Oct 14 14:30:55 2016 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Fri, 14 Oct 2016 13:30:55 +0000 Subject: [gpfsug-discuss] LROC benefits Message-ID: Hi all, Is there any way to gauge the performance benefits that could be realised from installing some LROCs in our CES cluster nodes? Or just suck it and see as it were? Cheers Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Sat Oct 15 15:23:01 2016 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Sat, 15 Oct 2016 10:23:01 -0400 Subject: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter Message-ID: <1c7562cf-d61a-adda-66b4-5fd00d75ae75@nasa.gov> I've got a node that's got some curious waiters on it (see below). Could someone explain what the "SGExceptionLogBufferFullThread" waiter means? Thanks! -Aaron === mmdiag: waiters === 0x7FFFF040D600 waiting 0.038822715 seconds, SGExceptionLogBufferFullThread: on ThCond 0x7FFFDBB07628 (0x7FFFDBB07628) (parallelWaitCond), reason 'wait for parallel write' for NSD I/O completion on node 10.1.53.5 0x7FFFE83F3D60 waiting 0.039629116 seconds, CleanBufferThread: on ThCond 0x17B1488 (0x17B1488) (MsgRecordCondvar), reason 'RPC wait' for NSD I/O completion on node 10.1.53.7 0x7FFFE8373A90 waiting 0.038921480 seconds, CleanBufferThread: on ThCond 0x7FFFCD2B4E30 (0x7FFFCD2B4E30) (LogFileBufferDescriptorCondvar), reason 'force wait on force active buffer write' 0x42CD9B0 waiting 0.028227004 seconds, CleanBufferThread: on ThCond 0x7FFFCD2B4E30 (0x7FFFCD2B4E30) (LogFileBufferDescriptorCondvar), reason 'force wait for buffer write to complete' 0x7FFFE0F0EAD0 waiting 0.027864343 seconds, CleanBufferThread: on ThCond 0x7FFFDC0EEA88 (0x7FFFDC0EEA88) (MsgRecordCondvar), reason 'RPC wait' for NSD I/O completion on node 10.1.53.7 0x1575560 waiting 0.028045975 seconds, RemoveHandlerThread: on ThCond 0x18020CE4E08 (0xFFFFC90020CE4E08) (LkObjCondvar), reason 'waiting for LX lock' 0x1570560 waiting 0.038724949 seconds, CreateHandlerThread: on ThCond 0x18020CE50A0 (0xFFFFC90020CE50A0) (LkObjCondvar), reason 'waiting for LX lock' 0x1563D60 waiting 0.073919918 seconds, RemoveHandlerThread: on ThCond 0x180235F6440 (0xFFFFC900235F6440) (LkObjCondvar), reason 'waiting for LX lock' 0x1561560 waiting 0.054854513 seconds, RemoveHandlerThread: on ThCond 0x1802292D200 (0xFFFFC9002292D200) (LkObjCondvar), reason 'waiting for LX lock' From oehmes at gmail.com Sat Oct 15 15:50:13 2016 From: oehmes at gmail.com (Sven Oehme) Date: Sat, 15 Oct 2016 14:50:13 +0000 Subject: [gpfsug-discuss] LROC benefits In-Reply-To: References: Message-ID: all depends on workload. we will publish a paper comparing mixed workload with and without LROC pretty soon. most numbers i have seen show anywhere between 30% - 1000% (very rare case) improvements, so its for sure worth a test. sven On Fri, Oct 14, 2016 at 6:31 AM Sobey, Richard A wrote: > Hi all, > > > > Is there any way to gauge the performance benefits that could be realised > from installing some LROCs in our CES cluster nodes? Or just suck it and > see as it were? > > > > Cheers > > Richard > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Sat Oct 15 16:23:57 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Sat, 15 Oct 2016 08:23:57 -0700 Subject: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter In-Reply-To: <1c7562cf-d61a-adda-66b4-5fd00d75ae75@nasa.gov> References: <1c7562cf-d61a-adda-66b4-5fd00d75ae75@nasa.gov> Message-ID: An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Sat Oct 15 16:27:42 2016 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Sat, 15 Oct 2016 11:27:42 -0400 Subject: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter In-Reply-To: References: <1c7562cf-d61a-adda-66b4-5fd00d75ae75@nasa.gov> Message-ID: <96434600-7c7a-d835-e7b5-9d6a80d9add5@nasa.gov> It absolutely does, thanks Olaf! The tasks running on these nodes are running on 63 other nodes and generating ~60K iop/s of metadata writes and I *think* about the same in reads. Do you think that could be contributing to the higher waiter times? I'm not sure quite what the job is up to. It's seemingly doing very little data movement, the cpu %used is very low but the load is rather high. -Aaron On 10/15/16 11:23 AM, Olaf Weiser wrote: > from your file system configuration .. mmfs -L you'll find the > size of the LOG > since release 4.x ..you can change it, but you need to re-mount the FS > on every client , to make the change effective ... > > when a clients initiate writes/changes to GPFS it needs to update its > changes to the log - if this narrows a certain filling degree, GPFS > triggers so called logWrapThreads to write content to disk and so free > space > > with your given numbers ... double digit [ms] waiter times .. you fs > get's probably slowed down.. and there's something suspect with the > storage, because LOG-IOs are rather small and should not take that long > > to give you an example from a healthy environment... the IO times are so > small, that you usually don't see waiters for this.. > > I/O start time RW Buf type disk:sectorNum nSec time ms tag1 > tag2 Disk UID typ NSD node context thread > --------------- -- ----------- ----------------- ----- ------- > --------- --------- ------------------ --- --------------- --------- > ---------- > 06:23:32.358851 W logData 2:524306424 8 0.439 > 0 0 C0A70D08:57CF40D1 cli 192.167.20.17 LogData > SGExceptionLogBufferFullThread > 06:23:33.576367 W logData 1:524257280 8 0.646 > 0 0 C0A70D08:57CF40D0 cli 192.167.20.16 LogData > SGExceptionLogBufferFullThread > 06:23:32.358851 W logData 2:524306424 8 0.439 > 0 0 C0A70D08:57CF40D1 cli 192.167.20.17 LogData > SGExceptionLogBufferFullThread > 06:23:33.576367 W logData 1:524257280 8 0.646 > 0 0 C0A70D08:57CF40D0 cli 192.167.20.16 LogData > SGExceptionLogBufferFullThread > 06:23:32.212426 W iallocSeg 1:524490048 64 0.733 > 2 245 C0A70D08:57CF40D0 cli 192.167.20.16 Logwrap > LogWrapHelperThread > 06:23:32.212412 W logWrap 2:524552192 8 0.755 > 0 179200 C0A70D08:57CF40D1 cli 192.167.20.17 Logwrap > LogWrapHelperThread > 06:23:32.212432 W logWrap 2:525162760 8 0.737 > 0 125473 C0A70D08:57CF40D1 cli 192.167.20.17 Logwrap > LogWrapHelperThread > 06:23:32.212416 W iallocSeg 2:524488384 64 0.763 > 2 347 C0A70D08:57CF40D1 cli 192.167.20.17 Logwrap > LogWrapHelperThread > 06:23:32.212414 W logWrap 2:525266944 8 2.160 > 0 177664 C0A70D08:57CF40D1 cli 192.167.20.17 Logwrap > LogWrapHelperThread > > > hope this helps .. > > > Mit freundlichen Gr??en / Kind regards > > > Olaf Weiser > > EMEA Storage Competence Center Mainz, German / IBM Systems, Storage > Platform, > ------------------------------------------------------------------------------------------------------------------------------------------- > IBM Deutschland > IBM Allee 1 > 71139 Ehningen > Phone: +49-170-579-44-66 > E-Mail: olaf.weiser at de.ibm.com > ------------------------------------------------------------------------------------------------------------------------------------------- > IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter > Gesch?ftsf?hrung: Martina Koederitz (Vorsitzende), Susanne Peter, > Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Markus Koerner > Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht > Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 > > > > From: Aaron Knister > To: gpfsug main discussion list > Date: 10/15/2016 07:23 AM > Subject: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------------------------------------------------ > > > > I've got a node that's got some curious waiters on it (see below). Could > someone explain what the "SGExceptionLogBufferFullThread" waiter means? > > Thanks! > > -Aaron > > === mmdiag: waiters === > 0x7FFFF040D600 waiting 0.038822715 seconds, > SGExceptionLogBufferFullThread: on ThCond 0x7FFFDBB07628 > (0x7FFFDBB07628) (parallelWaitCond), reason 'wait for parallel write' > for NSD I/O completion on node 10.1.53.5 > 0x7FFFE83F3D60 waiting 0.039629116 seconds, CleanBufferThread: on ThCond > 0x17B1488 (0x17B1488) (MsgRecordCondvar), reason 'RPC wait' for NSD I/O > completion on node 10.1.53.7 > 0x7FFFE8373A90 waiting 0.038921480 seconds, CleanBufferThread: on ThCond > 0x7FFFCD2B4E30 (0x7FFFCD2B4E30) (LogFileBufferDescriptorCondvar), reason > 'force wait on force active buffer write' > 0x42CD9B0 waiting 0.028227004 seconds, CleanBufferThread: on ThCond > 0x7FFFCD2B4E30 (0x7FFFCD2B4E30) (LogFileBufferDescriptorCondvar), reason > 'force wait for buffer write to complete' > 0x7FFFE0F0EAD0 waiting 0.027864343 seconds, CleanBufferThread: on ThCond > 0x7FFFDC0EEA88 (0x7FFFDC0EEA88) (MsgRecordCondvar), reason 'RPC wait' > for NSD I/O completion on node 10.1.53.7 > 0x1575560 waiting 0.028045975 seconds, RemoveHandlerThread: on ThCond > 0x18020CE4E08 (0xFFFFC90020CE4E08) (LkObjCondvar), reason 'waiting for > LX lock' > 0x1570560 waiting 0.038724949 seconds, CreateHandlerThread: on ThCond > 0x18020CE50A0 (0xFFFFC90020CE50A0) (LkObjCondvar), reason 'waiting for > LX lock' > 0x1563D60 waiting 0.073919918 seconds, RemoveHandlerThread: on ThCond > 0x180235F6440 (0xFFFFC900235F6440) (LkObjCondvar), reason 'waiting for > LX lock' > 0x1561560 waiting 0.054854513 seconds, RemoveHandlerThread: on ThCond > 0x1802292D200 (0xFFFFC9002292D200) (LkObjCondvar), reason 'waiting for > LX lock' > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From olaf.weiser at de.ibm.com Sat Oct 15 17:46:19 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Sat, 15 Oct 2016 09:46:19 -0700 Subject: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter In-Reply-To: <96434600-7c7a-d835-e7b5-9d6a80d9add5@nasa.gov> References: <1c7562cf-d61a-adda-66b4-5fd00d75ae75@nasa.gov> <96434600-7c7a-d835-e7b5-9d6a80d9add5@nasa.gov> Message-ID: An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Sat Oct 15 18:07:42 2016 From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]) Date: Sat, 15 Oct 2016 17:07:42 +0000 Subject: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter References: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter Message-ID: <5F910253243E6A47B81A9A2EB424BBA101D9277B@NDMSMBX404.ndc.nasa.gov> Understood. Thank you for your help. By the way, I was able to figure out by poking mmpmon gfis that the job is performing 20k a second each of inode creations, updates and deletions across 64 nodes. There's my 60k iops on the backend. While I'm impressed and not surprised GPFS can keep up with this...that's a pretty hefty workload. From: Olaf Weiser Sent: 10/15/16, 12:47 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter well - hard to say.. 60K IO may or may not be a problem... it depends on your storage backends.. check the response times to the physical disk on the NSD server... concerning the output you provided ... check particularly 10.1.53.5 and 10.1.53.7 .... if they are in the same (bad/ poor) range .. then your storage back end is in trouble or maybe just too heavily utilized ... if the response times to physical disks on the NSD server are ok... .. than maybe the network from client <--> NSD server is somehow in trouble .. From: Aaron Knister To: Date: 10/15/2016 08:28 AM Subject: Re: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ It absolutely does, thanks Olaf! The tasks running on these nodes are running on 63 other nodes and generating ~60K iop/s of metadata writes and I *think* about the same in reads. Do you think that could be contributing to the higher waiter times? I'm not sure quite what the job is up to. It's seemingly doing very little data movement, the cpu %used is very low but the load is rather high. -Aaron On 10/15/16 11:23 AM, Olaf Weiser wrote: > from your file system configuration .. mmfs -L you'll find the > size of the LOG > since release 4.x ..you can change it, but you need to re-mount the FS > on every client , to make the change effective ... > > when a clients initiate writes/changes to GPFS it needs to update its > changes to the log - if this narrows a certain filling degree, GPFS > triggers so called logWrapThreads to write content to disk and so free > space > > with your given numbers ... double digit [ms] waiter times .. you fs > get's probably slowed down.. and there's something suspect with the > storage, because LOG-IOs are rather small and should not take that long > > to give you an example from a healthy environment... the IO times are so > small, that you usually don't see waiters for this.. > > I/O start time RW Buf type disk:sectorNum nSec time ms tag1 > tag2 Disk UID typ NSD node context thread > --------------- -- ----------- ----------------- ----- ------- > --------- --------- ------------------ --- --------------- --------- > ---------- > 06:23:32.358851 W logData 2:524306424 8 0.439 > 0 0 C0A70D08:57CF40D1 cli 192.167.20.17 LogData > SGExceptionLogBufferFullThread > 06:23:33.576367 W logData 1:524257280 8 0.646 > 0 0 C0A70D08:57CF40D0 cli 192.167.20.16 LogData > SGExceptionLogBufferFullThread > 06:23:32.358851 W logData 2:524306424 8 0.439 > 0 0 C0A70D08:57CF40D1 cli 192.167.20.17 LogData > SGExceptionLogBufferFullThread > 06:23:33.576367 W logData 1:524257280 8 0.646 > 0 0 C0A70D08:57CF40D0 cli 192.167.20.16 LogData > SGExceptionLogBufferFullThread > 06:23:32.212426 W iallocSeg 1:524490048 64 0.733 > 2 245 C0A70D08:57CF40D0 cli 192.167.20.16 Logwrap > LogWrapHelperThread > 06:23:32.212412 W logWrap 2:524552192 8 0.755 > 0 179200 C0A70D08:57CF40D1 cli 192.167.20.17 Logwrap > LogWrapHelperThread > 06:23:32.212432 W logWrap 2:525162760 8 0.737 > 0 125473 C0A70D08:57CF40D1 cli 192.167.20.17 Logwrap > LogWrapHelperThread > 06:23:32.212416 W iallocSeg 2:524488384 64 0.763 > 2 347 C0A70D08:57CF40D1 cli 192.167.20.17 Logwrap > LogWrapHelperThread > 06:23:32.212414 W logWrap 2:525266944 8 2.160 > 0 177664 C0A70D08:57CF40D1 cli 192.167.20.17 Logwrap > LogWrapHelperThread > > > hope this helps .. > > > Mit freundlichen Gr??en / Kind regards > > > Olaf Weiser > > EMEA Storage Competence Center Mainz, German / IBM Systems, Storage > Platform, > ------------------------------------------------------------------------------------------------------------------------------------------- > IBM Deutschland > IBM Allee 1 > 71139 Ehningen > Phone: +49-170-579-44-66 > E-Mail: olaf.weiser at de.ibm.com > ------------------------------------------------------------------------------------------------------------------------------------------- > IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter > Gesch?ftsf?hrung: Martina Koederitz (Vorsitzende), Susanne Peter, > Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Markus Koerner > Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht > Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 > > > > From: Aaron Knister > To: gpfsug main discussion list > Date: 10/15/2016 07:23 AM > Subject: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------------------------------------------------ > > > > I've got a node that's got some curious waiters on it (see below). Could > someone explain what the "SGExceptionLogBufferFullThread" waiter means? > > Thanks! > > -Aaron > > === mmdiag: waiters === > 0x7FFFF040D600 waiting 0.038822715 seconds, > SGExceptionLogBufferFullThread: on ThCond 0x7FFFDBB07628 > (0x7FFFDBB07628) (parallelWaitCond), reason 'wait for parallel write' > for NSD I/O completion on node 10.1.53.5 > 0x7FFFE83F3D60 waiting 0.039629116 seconds, CleanBufferThread: on ThCond > 0x17B1488 (0x17B1488) (MsgRecordCondvar), reason 'RPC wait' for NSD I/O > completion on node 10.1.53.7 > 0x7FFFE8373A90 waiting 0.038921480 seconds, CleanBufferThread: on ThCond > 0x7FFFCD2B4E30 (0x7FFFCD2B4E30) (LogFileBufferDescriptorCondvar), reason > 'force wait on force active buffer write' > 0x42CD9B0 waiting 0.028227004 seconds, CleanBufferThread: on ThCond > 0x7FFFCD2B4E30 (0x7FFFCD2B4E30) (LogFileBufferDescriptorCondvar), reason > 'force wait for buffer write to complete' > 0x7FFFE0F0EAD0 waiting 0.027864343 seconds, CleanBufferThread: on ThCond > 0x7FFFDC0EEA88 (0x7FFFDC0EEA88) (MsgRecordCondvar), reason 'RPC wait' > for NSD I/O completion on node 10.1.53.7 > 0x1575560 waiting 0.028045975 seconds, RemoveHandlerThread: on ThCond > 0x18020CE4E08 (0xFFFFC90020CE4E08) (LkObjCondvar), reason 'waiting for > LX lock' > 0x1570560 waiting 0.038724949 seconds, CreateHandlerThread: on ThCond > 0x18020CE50A0 (0xFFFFC90020CE50A0) (LkObjCondvar), reason 'waiting for > LX lock' > 0x1563D60 waiting 0.073919918 seconds, RemoveHandlerThread: on ThCond > 0x180235F6440 (0xFFFFC900235F6440) (LkObjCondvar), reason 'waiting for > LX lock' > 0x1561560 waiting 0.054854513 seconds, RemoveHandlerThread: on ThCond > 0x1802292D200 (0xFFFFC9002292D200) (LkObjCondvar), reason 'waiting for > LX lock' > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Sat Oct 15 18:43:34 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Sat, 15 Oct 2016 10:43:34 -0700 Subject: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter In-Reply-To: <5F910253243E6A47B81A9A2EB424BBA101D9277B@NDMSMBX404.ndc.nasa.gov> References: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter <5F910253243E6A47B81A9A2EB424BBA101D9277B@NDMSMBX404.ndc.nasa.gov> Message-ID: An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Mon Oct 17 01:06:09 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 17 Oct 2016 00:06:09 +0000 Subject: [gpfsug-discuss] CES and NFS Tuning suggestions Message-ID: Looking for some pointers or suggestions on what I should look at changing in Linux and/or GPFS "mmchconfg" settings to help boost NFS performance. Out of the box it seems "poor". Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid 507-269-0413 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ckrafft at de.ibm.com Mon Oct 17 14:59:39 2016 From: ckrafft at de.ibm.com (Christoph Krafft) Date: Mon, 17 Oct 2016 15:59:39 +0200 Subject: [gpfsug-discuss] Looking for experiences with Huawei Oceanstore and GPFS / Spectrum Scale Message-ID: Hi folks, has anyone made experiences with Huawei Oceanstore and GPFS - and would be willing to share some details with me? Any helpful hints are deeply appreciated - THANK you in advance! Mit freundlichen Gr??en / Sincerely Christoph Krafft Client Technical Specialist - Power Systems, IBM Systems Certified IT Specialist @ The Open Group Phone: +49 (0) 7034 643 2171 IBM Deutschland GmbH Mobile: +49 (0) 160 97 81 86 12 Am Weiher 24 Email: ckrafft at de.ibm.com 65451 Kelsterbach Germany IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter Gesch?ftsf?hrung: Martina Koederitz (Vorsitzende), Nicole Reimer, Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Markus Koerner Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecblank.gif Type: image/gif Size: 45 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0B092252.gif Type: image/gif Size: 1851 bytes Desc: not available URL: From Robert.Oesterlin at nuance.com Mon Oct 17 16:00:20 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 17 Oct 2016 15:00:20 +0000 Subject: [gpfsug-discuss] CES: IP address won't assign: "handleNetworkProblem with lock held" Message-ID: <0669A3C9-4F5E-4834-A1C4-D8D124B0A4F3@nuance.com> Can anyone help me pinpoint the issue here? These message repeat and the IP addresses never get assigned. [root at tct-gw01 ~]# tail /var/mmfs/gen/mmfslog Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: Found unassigned address 10.30.22.178 Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: Found unassigned address 10.30.22.179 Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: Found unassigned address 10.30.22.177 Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: Found unassigned address 10.30.22.176 Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: handleNetworkProblem with lock held: assignIP 10.30.22.178_0-_+,10.30.22.179_0-_+,10.30.22.177_0-_+,10.30.22.176_0-_+ 1 Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: Assigning addresses: 10.30.22.178_0-_+,10.30.22.179_0-_+,10.30.22.177_0-_+,10.30.22.176_0-_+ Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: moveCesIPs: 10.30.22.178_0-_+,10.30.22.179_0-_+,10.30.22.177_0-_+,10.30.22.176_0-_+ [root at tct-gw01 ~]# mmces state show -a NODE AUTH AUTH_OBJ NETWORK NFS OBJ SMB CES tct-gw01.infra.us.grid.nuance.com DISABLED DISABLED HEALTHY HEALTHY DISABLED DISABLED HEALTHY tct-gw02.infra.us.grid.nuance.com DISABLED DISABLED HEALTHY HEALTHY DISABLED DISABLED HEALTHY tct-gw03.infra.us.grid.nuance.com DISABLED DISABLED HEALTHY HEALTHY DISABLED DISABLED HEALTHY tct-gw04.infra.us.grid.nuance.com DISABLED DISABLED HEALTHY HEALTHY DISABLED DISABLED HEALTHY [root at tct-gw01 ~]# mmces address list Address Node Group Attribute ------------------------------------------------------------------------- 10.30.22.178 unassigned none none 10.30.22.179 unassigned none none 10.30.22.177 unassigned none none 10.30.22.176 unassigned none none Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid 507-269-0413 -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenneth.waegeman at ugent.be Mon Oct 17 16:16:05 2016 From: kenneth.waegeman at ugent.be (Kenneth Waegeman) Date: Mon, 17 Oct 2016 17:16:05 +0200 Subject: [gpfsug-discuss] Disk can't be recovered due to uncorrectable read error in vdisk (GSS) Message-ID: Hi, Currently our file system is down due to down/unrecovered disks. We try to start the disks again with mmchdisk, but when we do this, we see this error in our mmfs.log: Mon Oct 17 15:28:18.122 2016: [E] smallRead VIO failed due to uncorrectable read error in vdisk nsd11_MetaData_8M_3p_2 vtrack 33630 vsector 34437120 number of vsectors 1024 segments 0 to 16 startBufOffset 0 endB ufOffset 1023 vtrkOffset 0 vioLen 524288 This is a 3-way replicated vdisk, and not one of the recovering disks, but this disk is in 'up' state.. Does someone has a clue what we could to recover our fs? Thanks! Kenneth From bbanister at jumptrading.com Mon Oct 17 17:00:22 2016 From: bbanister at jumptrading.com (Bryan Banister) Date: Mon, 17 Oct 2016 16:00:22 +0000 Subject: [gpfsug-discuss] CES and NFS Tuning suggestions In-Reply-To: References: Message-ID: <21BC488F0AEA2245B2C3E83FC0B33DBB06364770@CHI-EXCHANGEW1.w2k.jumptrading.com> One major issue is the maxFilesToCache and maybe the maxStatCache (though I hear that Linux negates the use of this parameter now? I don?t quite remember). Ganesha apparently likes to hold open a large number of files and this means that it will quickly fill up the maxFilesToCache. When this happens the [gpfsSwapdKproc] process will start to eat up CPU time. This is the daemon that tries to find a file to evict from the cache when a new file is opened. This overhead will also hurt performance. IBM in a PMR we opened suggested setting this to something like 5 Million for the protocol nodes. I think we started with 1.5 Million. You have to be mindful of memory requirements on the token servers to handle the total sum of all maxFilesToCache settings from all nodes that mount the file system. Of course the other, standard NFS tuning parameters for number of threads and NFS client mount options still should be adjusted too. Hope that helps, -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Oesterlin, Robert Sent: Sunday, October 16, 2016 7:06 PM To: gpfsug main discussion list Subject: [gpfsug-discuss] CES and NFS Tuning suggestions Looking for some pointers or suggestions on what I should look at changing in Linux and/or GPFS "mmchconfg" settings to help boost NFS performance. Out of the box it seems "poor". Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid 507-269-0413 ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralphbsz at us.ibm.com Mon Oct 17 17:39:18 2016 From: ralphbsz at us.ibm.com (Ralph A Becker-szendy) Date: Mon, 17 Oct 2016 16:39:18 +0000 Subject: [gpfsug-discuss] Disk can't be recovered due to uncorrectable read error in vdisk (GSS) Message-ID: An HTML attachment was scrubbed... URL: From stijn.deweirdt at ugent.be Mon Oct 17 18:35:37 2016 From: stijn.deweirdt at ugent.be (Stijn De Weirdt) Date: Mon, 17 Oct 2016 19:35:37 +0200 Subject: [gpfsug-discuss] Disk can't be recovered due to uncorrectable read error in vdisk (GSS) In-Reply-To: References: Message-ID: <8f93f6bf-6b42-c3bf-11b4-39dc53b4451f@ugent.be> hi ralph, >>Currently our file system is down due to down/unrecovered disks. We >> try to start the disks again with mmchdisk, but when we do this, we >> see this error in our mmfs.log: >> ... >> This is a 3-way replicated vdisk, and not one of the recovering disks,but >> this disk is in 'up' state.. > First, please open a PMR through your normal support organization, and make it > clear in the PMR that the problem is GPFS and GNR (a.k.a. ESS). Like that, it > will be assigned to the correct support group. Support will request that you > upload a snap. PMR is created, but we are in particular puzzled by the IO read error. and getting some details about what is gone here (with details as you provided) is somethign we usually do not get from support ;) > There seems to be a combination of two problems here: > One, a NSD (which is also a GNR vdisk) is down, which is usually caused by an IO > error on the vdisk, or by both servers for the recovery group that contains the > vdisk being down simultaneously. Usually, that is easily fixed by running > mmchdisk with a start option, but you tried that and it didn't work. This > problem is at the NSD layer (meaning in the GPFS client that accesses the GNR > vdisk), not in the GNR layer. it's actually more than disk down, all on same recoverygroup. the vdisk with the read error is on the other recoverygroup (only 2, this is/was a GSS24) > Second, another vdisk has an internal error, caused by read error from the > physical disks (which is what "uncorrectable read error" means). what does physical mean here? we have no IO error anywhere (OS/kernel/scsi, also the error counters on the mmlspdisk output do not increase). Now, give that > you say that this vdisk is 3-way replicated, that probably means that there are > multiple problems. This error is purely in the GNR layer, and the error message > you quote "smallRead VIO..." comes from the GNR layer. Now, an error from one > vdisk can't prevent mmchdisk on a different vdisk from working, so these two > problems seem unrelated. well, they can: the disks with all metadata replica on the one recoverygroup are down. starting those forces the read of the ones on the other group, and this runs into the IOread error, and everything stops (well, that's how we understand it ;) > Furthermore, I'm going to bet that the two problems (which at first seem > unrelated) must in reality have a common root cause; it would be too weird a > coincidence to get two problems that are unrelated at the same time. To debug > this requires looking at way more information than a single line from the > mmfs.log file, which is why the support organization needs a complete PMR > opened, and then have the usual snap (with logs, dumps, ...) uploaded, so it can > see what the cause of the problem is. yep, trace and snap uploaded. > Good luck! thanks (and thanks again for some insights, much appreciated !) > Ralph Becker-Szendy > IBM Almaden Research Center - Computer Science -Storage Systems > ralphbsz at us.ibm.com > 408-927-2752 > 650 Harry Road, K56-B3, San Jose, CA 95120 > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From olaf.weiser at de.ibm.com Mon Oct 17 18:40:13 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Mon, 17 Oct 2016 10:40:13 -0700 Subject: [gpfsug-discuss] CES and NFS Tuning suggestions In-Reply-To: <21BC488F0AEA2245B2C3E83FC0B33DBB06364770@CHI-EXCHANGEW1.w2k.jumptrading.com> References: <21BC488F0AEA2245B2C3E83FC0B33DBB06364770@CHI-EXCHANGEW1.w2k.jumptrading.com> Message-ID: An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Mon Oct 17 19:27:01 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Mon, 17 Oct 2016 11:27:01 -0700 Subject: [gpfsug-discuss] CES: IP address won't assign: "handleNetworkProblem with lock held" In-Reply-To: <0669A3C9-4F5E-4834-A1C4-D8D124B0A4F3@nuance.com> References: <0669A3C9-4F5E-4834-A1C4-D8D124B0A4F3@nuance.com> Message-ID: An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Mon Oct 17 19:42:58 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 17 Oct 2016 18:42:58 +0000 Subject: [gpfsug-discuss] CES: IP address won't assign: "handleNetworkProblem with lock held" Message-ID: <99C9D37E-3387-48A6-88BB-DFE711C7CC8F@nuance.com> Yes - so interesting - it looks like the nodes have the addresses assigned but CES doesn?t know that. [root at tct-gw01 ~]# mmlscluster --ces GPFS cluster information ======================== GPFS cluster name: nrg1-tct.nrg1.us.grid.nuance.com GPFS cluster id: 17869514639699411874 Cluster Export Services global parameters ----------------------------------------- Shared root directory: /gpfs/fs1/ces Enabled Services: NFS Log level: 0 Address distribution policy: even-coverage Node Daemon node name IP address CES IP address list ----------------------------------------------------------------------- 4 tct-gw01.infra.us.grid.nuance.com 10.30.22.160 None 5 tct-gw02.infra.us.grid.nuance.com 10.30.22.161 None 6 tct-gw03.infra.us.grid.nuance.com 10.30.22.162 None 7 tct-gw04.infra.us.grid.nuance.com 10.30.22.163 None [root at tct-gw01 ~]# mmdsh -N cesnodes "ip a | grep 10.30.22." | sort -k 1 tct-gw01.infra.us.grid.nuance.com: inet 10.30.22.160/24 brd 10.30.22.255 scope global bond0 tct-gw01.infra.us.grid.nuance.com: inet 10.30.22.176/24 brd 10.30.22.255 scope global secondary bond0:0 tct-gw02.infra.us.grid.nuance.com: inet 10.30.22.161/24 brd 10.30.22.255 scope global bond0 tct-gw02.infra.us.grid.nuance.com: inet 10.30.22.177/24 brd 10.30.22.255 scope global secondary bond0:0 tct-gw02.infra.us.grid.nuance.com: inet 10.30.22.178/24 brd 10.30.22.255 scope global secondary bond0:1 tct-gw03.infra.us.grid.nuance.com: inet 10.30.22.162/24 brd 10.30.22.255 scope global bond0 tct-gw03.infra.us.grid.nuance.com: inet 10.30.22.178/24 brd 10.30.22.255 scope global secondary bond0:0 tct-gw04.infra.us.grid.nuance.com: inet 10.30.22.163/24 brd 10.30.22.255 scope global bond0 tct-gw04.infra.us.grid.nuance.com: inet 10.30.22.179/24 brd 10.30.22.255 scope global secondary bond0:0 Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: on behalf of Olaf Weiser Reply-To: gpfsug main discussion list Date: Monday, October 17, 2016 at 1:27 PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] CES: IP address won't assign: "handleNetworkProblem with lock held" ip a | grep 10.30.22. -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Mon Oct 17 19:53:01 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Mon, 17 Oct 2016 18:53:01 +0000 Subject: [gpfsug-discuss] CES: IP address won't assign: "handleNetworkProblem with lock held" In-Reply-To: <99C9D37E-3387-48A6-88BB-DFE711C7CC8F@nuance.com> References: <99C9D37E-3387-48A6-88BB-DFE711C7CC8F@nuance.com> Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 37817 bytes Desc: not available URL: From S.J.Thompson at bham.ac.uk Mon Oct 17 19:57:15 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Mon, 17 Oct 2016 18:57:15 +0000 Subject: [gpfsug-discuss] CES: IP address won't assign: "handleNetworkProblem with lock held" In-Reply-To: References: <0669A3C9-4F5E-4834-A1C4-D8D124B0A4F3@nuance.com>, Message-ID: Does it strictly follow this? We were doing some testing with tap interfaces into vxlan networks and found that if we simulated taking down the vxlan interface (which appears in ifconfig as a physical int really), then it moved the ces ip onto the box's primary Nic which was a different subnet. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Olaf Weiser [olaf.weiser at de.ibm.com] Sent: 17 October 2016 19:27 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] CES: IP address won't assign: "handleNetworkProblem with lock held" simple question -sorry for that - your Nodes.. do they have an IP address in the same subnet as your IP address listed here ? and if, is this network up n running so that GPFS can find/detect it ? what tells mmlscluster --ces ? from each node - assuming class C /24 network , do a ip a | grep 10.30.22. cheers From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 10/17/2016 08:00 AM Subject: [gpfsug-discuss] CES: IP address won't assign: "handleNetworkProblem with lock held" Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Can anyone help me pinpoint the issue here? These message repeat and the IP addresses never get assigned. [root at tct-gw01 ~]# tail /var/mmfs/gen/mmfslog Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: Found unassigned address 10.30.22.178 Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: Found unassigned address 10.30.22.179 Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: Found unassigned address 10.30.22.177 Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: Found unassigned address 10.30.22.176 Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: handleNetworkProblem with lock held: assignIP 10.30.22.178_0-_+,10.30.22.179_0-_+,10.30.22.177_0-_+,10.30.22.176_0-_+ 1 Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: Assigning addresses: 10.30.22.178_0-_+,10.30.22.179_0-_+,10.30.22.177_0-_+,10.30.22.176_0-_+ Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: moveCesIPs: 10.30.22.178_0-_+,10.30.22.179_0-_+,10.30.22.177_0-_+,10.30.22.176_0-_+ [root at tct-gw01 ~]# mmces state show -a NODE AUTH AUTH_OBJ NETWORK NFS OBJ SMB CES tct-gw01.infra.us.grid.nuance.com DISABLED DISABLED HEALTHY HEALTHY DISABLED DISABLED HEALTHY tct-gw02.infra.us.grid.nuance.com DISABLED DISABLED HEALTHY HEALTHY DISABLED DISABLED HEALTHY tct-gw03.infra.us.grid.nuance.com DISABLED DISABLED HEALTHY HEALTHY DISABLED DISABLED HEALTHY tct-gw04.infra.us.grid.nuance.com DISABLED DISABLED HEALTHY HEALTHY DISABLED DISABLED HEALTHY [root at tct-gw01 ~]# mmces address list Address Node Group Attribute ------------------------------------------------------------------------- 10.30.22.178 unassigned none none 10.30.22.179 unassigned none none 10.30.22.177 unassigned none none 10.30.22.176 unassigned none none Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid 507-269-0413 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Robert.Oesterlin at nuance.com Mon Oct 17 20:01:48 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 17 Oct 2016 19:01:48 +0000 Subject: [gpfsug-discuss] [EXTERNAL] Re: CES: IP address won't assign: "handleNetworkProblem with lock held" In-Reply-To: References: <99C9D37E-3387-48A6-88BB-DFE711C7CC8F@nuance.com> Message-ID: <0E4224B9-F266-4BB2-9B6A-F580976C2207@nuance.com> No - the :0 and :1 address are floating addresses *assigned by CES* - it created those interfaces. The issue seems to be that these are assigned and CES doesn't know it. Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: on behalf of Olaf Weiser Reply-To: gpfsug main discussion list Date: Monday, October 17, 2016 at 1:53 PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] CES: IP address won't assign: "handleNetworkProblem with lock held" ah .. I see.. seems, that you already has IP aliases around .. GPFS don't like it... eg. your node tct-gw01.infra.us.grid.nuance.com: inet 10.30.22.160/24 has already an alias - 10.30.22.176 ... if I understand you answers correctly... from the doc'... [...] you need to provide a static IP adress, that is not already[...] as an alias [...] -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Mon Oct 17 21:13:28 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Mon, 17 Oct 2016 13:13:28 -0700 Subject: [gpfsug-discuss] CES: IP address won't assign: "handleNetworkProblem with lock held" In-Reply-To: References: <0669A3C9-4F5E-4834-A1C4-D8D124B0A4F3@nuance.com>, Message-ID: An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Mon Oct 17 21:13:28 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Mon, 17 Oct 2016 13:13:28 -0700 Subject: [gpfsug-discuss] [EXTERNAL] Re: CES: IP address won't assign: "handleNetworkProblem with lock held" In-Reply-To: <0E4224B9-F266-4BB2-9B6A-F580976C2207@nuance.com> References: <99C9D37E-3387-48A6-88BB-DFE711C7CC8F@nuance.com> <0E4224B9-F266-4BB2-9B6A-F580976C2207@nuance.com> Message-ID: An HTML attachment was scrubbed... URL: From jake.carroll at uq.edu.au Tue Oct 18 05:21:12 2016 From: jake.carroll at uq.edu.au (Jake Carroll) Date: Tue, 18 Oct 2016 04:21:12 +0000 Subject: [gpfsug-discuss] Any spaces left for a user-presentation at the UG, SC16? Message-ID: <1EECF1D7-DD5E-47BD-A99F-57A4E0294442@uq.edu.au> Hi, I have something interesting that I think the user group might find novel, or at least, hopefully interesting, at the UG meetup at SC16. It would entail an unusual use-case for AFM and some of the unusual things we are doing with it. All good if no spots left ? but I?d be happy to present something if the group thinks it would be beneficial. Thanks! -jc From Robert.Oesterlin at nuance.com Tue Oct 18 12:37:17 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Tue, 18 Oct 2016 11:37:17 +0000 Subject: [gpfsug-discuss] Any spaces left for a user-presentation at the UG, SC16? Message-ID: <7A87F87D-8345-4254-B39A-DE38B61D63AB@nuance.com> Hi Jake At this point, we're pretty well filled up. And yes, you should be seeing a final agenda "soon" - still shuffling things around a bit, trying to get as much great content as we can into one afternoon Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid GPFS UG Co-Principal From: on behalf of Jake Carroll Reply-To: gpfsug main discussion list Date: Monday, October 17, 2016 at 11:21 PM To: "gpfsug-discuss at spectrumscale.org" Subject: [EXTERNAL] [gpfsug-discuss] Any spaces left for a user-presentation at the UG, SC16? Hi, I have something interesting that I think the user group might find novel, or at least, hopefully interesting, at the UG meetup at SC16. It would entail an unusual use-case for AFM and some of the unusual things we are doing with it. All good if no spots left ? but I?d be happy to present something if the group thinks it would be beneficial. Thanks! -jc _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwIGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=SI4sTXt3Q32Lv6WKkeOdNHvhsvEAdzOYPmJMU0cqe38&s=UiXMs0ix00_xIWkODVa9o_km3OQ5mSVKXkE_w6lC8ls&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From kallbac at iu.edu Tue Oct 18 14:26:20 2016 From: kallbac at iu.edu (Kallback-Rose, Kristy A) Date: Tue, 18 Oct 2016 13:26:20 +0000 Subject: [gpfsug-discuss] Any spaces left for a user-presentation at the UG, SC16? In-Reply-To: <7A87F87D-8345-4254-B39A-DE38B61D63AB@nuance.com> References: <7A87F87D-8345-4254-B39A-DE38B61D63AB@nuance.com> Message-ID: JC, can you email a little more on the AFM topic to me and Bob? For future reference if nothing else. Thanks, Kristy On Oct 18, 2016, at 7:37 AM, Oesterlin, Robert > wrote: Hi Jake At this point, we're pretty well filled up. And yes, you should be seeing a final agenda "soon" - still shuffling things around a bit, trying to get as much great content as we can into one afternoon Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid GPFS UG Co-Principal From: > on behalf of Jake Carroll > Reply-To: gpfsug main discussion list > Date: Monday, October 17, 2016 at 11:21 PM To: "gpfsug-discuss at spectrumscale.org" > Subject: [EXTERNAL] [gpfsug-discuss] Any spaces left for a user-presentation at the UG, SC16? Hi, I have something interesting that I think the user group might find novel, or at least, hopefully interesting, at the UG meetup at SC16. It would entail an unusual use-case for AFM and some of the unusual things we are doing with it. All good if no spots left ? but I?d be happy to present something if the group thinks it would be beneficial. Thanks! -jc _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwIGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=SI4sTXt3Q32Lv6WKkeOdNHvhsvEAdzOYPmJMU0cqe38&s=UiXMs0ix00_xIWkODVa9o_km3OQ5mSVKXkE_w6lC8ls&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.childs at qmul.ac.uk Wed Oct 19 15:12:41 2016 From: p.childs at qmul.ac.uk (Peter Childs) Date: Wed, 19 Oct 2016 14:12:41 +0000 Subject: [gpfsug-discuss] Using AFM to migrate files. In-Reply-To: References: <7j173edatp91hqocoto2bau8.1476818437765@email.android.com>, Message-ID: We are planning to use AFM to migrate our old GPFS file store to a new GPFS file store. This will give us the advantages of Spectrum Scale (GPFS) 4.2, such as larger block and inode size. I would like to attempt to gain some insight on my plans before I start. The old file store was running GPFS 3.5 with 512 byte inodes and 1MB block size. We have now upgraded it to 4.1 and are working towards 4.2 with 300TB of files. (385TB max space) this is so we can use both the old and new storage via multi-cluster. We are moving to a new GPFS cluster so we can use the new protocol nodes eventually and also put the new storage machines as cluster managers, as this should be faster and future proof The new hardware has 1PB of space running GPFS 4.2 We have multiple filesets, and would like to maintain our namespace as far as possible. My plan was to. 1. Create a read-only (RO) AFM cache on the new storage (ro) 2a. Move old fileset and replace with SymLink to new. 2b. Convert RO AFM to Local Update (LU) AFM pointing to new parking area of old files. 2c. move user access to new location in cache. 3. Flush everything into cache and disconnect. I've read the docs including the ones on migration but it's not clear if it's safe to move the home of a cache and update the target. It looks like it should be possible and my tests say it works. An alternative plan is to use a Independent Writer (IW) AFM Cache to move the home directories which are pointed to by LDAP. Hence we can move users one at a time and only have to drain the HPC cluster at the end to disconnect the cache. I assume that migrating users over an Independent Writer is safe so long as the users don't use both sides of the cache at once (ie home and target) I'm also interested in any recipe people have on GPFS policies to preseed and flush the cache. We plan to do all the migration using AFM over GPFS we're not currently using NFS and have no plans to start. I believe using GPFS is the faster method to preform the migration. Any suggestions and experience of doing similar migration jobs would be helpful. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London From mimarsh2 at vt.edu Wed Oct 19 18:46:02 2016 From: mimarsh2 at vt.edu (Brian Marshall) Date: Wed, 19 Oct 2016 13:46:02 -0400 Subject: [gpfsug-discuss] subnets Message-ID: All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Wed Oct 19 19:10:38 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Wed, 19 Oct 2016 18:10:38 +0000 Subject: [gpfsug-discuss] subnets In-Reply-To: References: Message-ID: mmdiag --network Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Brian Marshall [mimarsh2 at vt.edu] Sent: 19 October 2016 18:46 To: gpfsug main discussion list Subject: [gpfsug-discuss] subnets All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall From UWEFALKE at de.ibm.com Wed Oct 19 19:15:52 2016 From: UWEFALKE at de.ibm.com (Uwe Falke) Date: Wed, 19 Oct 2016 20:15:52 +0200 Subject: [gpfsug-discuss] subnets In-Reply-To: References: Message-ID: Hi Brian, you might use mmfsadm saferdump tscomm to check on which route peer cluster members are reached. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Frank Hammer, Thorsten Moehring Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: Brian Marshall To: gpfsug main discussion list Date: 10/19/2016 07:46 PM Subject: [gpfsug-discuss] subnets Sent by: gpfsug-discuss-bounces at spectrumscale.org All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Kevin.Buterbaugh at Vanderbilt.Edu Wed Oct 19 21:11:57 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Wed, 19 Oct 2016 20:11:57 +0000 Subject: [gpfsug-discuss] Will there be any more GPFS 4.2.0-x releases? Message-ID: <142FECE0-E157-42D9-BC10-4C48E78FA065@vanderbilt.edu> Hi All, We?re currently running GPFS 4.2.0-4 with an efix installed and now we need a 2nd efix. I?m not a big fan of adding efix to efix and would prefer to go to a new PTF that contains both efixes. So ? is there going to be a GPFS 4.2.0-5 (it?s been a longer than normal interval since PTF 4 came out) or do we need to go to GPFS 4.2.1-x? If the latter, any major changes to watch out for? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bpappas at dstonline.com Wed Oct 19 21:44:39 2016 From: bpappas at dstonline.com (Bill Pappas) Date: Wed, 19 Oct 2016 20:44:39 +0000 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) In-Reply-To: References: Message-ID: I have some ideas to suggest given some of my experiences. First, I have some questions: How many files are you migrating? Will you be creating multiple file sets on the target system based off of business or project needs? Like, file set a is for "department a" and file set b is for "large scale project a" Thanks. Bill Pappas 901-619-0585 bpappas at dstonline.com [1466780990050_DSTlogo.png] [http://www.prweb.com/releases/2016/06/prweb13504050.htm] http://www.prweb.com/releases/2016/06/prweb13504050.htm ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org Sent: Wednesday, October 19, 2016 3:12 PM To: gpfsug-discuss at spectrumscale.org Subject: gpfsug-discuss Digest, Vol 57, Issue 49 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Using AFM to migrate files. (Peter Childs) 2. subnets (Brian Marshall) 3. Re: subnets (Simon Thompson (Research Computing - IT Services)) 4. Re: subnets (Uwe Falke) 5. Will there be any more GPFS 4.2.0-x releases? (Buterbaugh, Kevin L) ---------------------------------------------------------------------- Message: 1 Date: Wed, 19 Oct 2016 14:12:41 +0000 From: Peter Childs To: gpfsug main discussion list Subject: [gpfsug-discuss] Using AFM to migrate files. Message-ID: Content-Type: text/plain; charset="iso-8859-1" We are planning to use AFM to migrate our old GPFS file store to a new GPFS file store. This will give us the advantages of Spectrum Scale (GPFS) 4.2, such as larger block and inode size. I would like to attempt to gain some insight on my plans before I start. The old file store was running GPFS 3.5 with 512 byte inodes and 1MB block size. We have now upgraded it to 4.1 and are working towards 4.2 with 300TB of files. (385TB max space) this is so we can use both the old and new storage via multi-cluster. We are moving to a new GPFS cluster so we can use the new protocol nodes eventually and also put the new storage machines as cluster managers, as this should be faster and future proof The new hardware has 1PB of space running GPFS 4.2 We have multiple filesets, and would like to maintain our namespace as far as possible. My plan was to. 1. Create a read-only (RO) AFM cache on the new storage (ro) 2a. Move old fileset and replace with SymLink to new. 2b. Convert RO AFM to Local Update (LU) AFM pointing to new parking area of old files. 2c. move user access to new location in cache. 3. Flush everything into cache and disconnect. I've read the docs including the ones on migration but it's not clear if it's safe to move the home of a cache and update the target. It looks like it should be possible and my tests say it works. An alternative plan is to use a Independent Writer (IW) AFM Cache to move the home directories which are pointed to by LDAP. Hence we can move users one at a time and only have to drain the HPC cluster at the end to disconnect the cache. I assume that migrating users over an Independent Writer is safe so long as the users don't use both sides of the cache at once (ie home and target) I'm also interested in any recipe people have on GPFS policies to preseed and flush the cache. We plan to do all the migration using AFM over GPFS we're not currently using NFS and have no plans to start. I believe using GPFS is the faster method to preform the migration. Any suggestions and experience of doing similar migration jobs would be helpful. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ------------------------------ Message: 2 Date: Wed, 19 Oct 2016 13:46:02 -0400 From: Brian Marshall To: gpfsug main discussion list Subject: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="utf-8" All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 3 Date: Wed, 19 Oct 2016 18:10:38 +0000 From: "Simon Thompson (Research Computing - IT Services)" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="us-ascii" mmdiag --network Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Brian Marshall [mimarsh2 at vt.edu] Sent: 19 October 2016 18:46 To: gpfsug main discussion list Subject: [gpfsug-discuss] subnets All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall ------------------------------ Message: 4 Date: Wed, 19 Oct 2016 20:15:52 +0200 From: "Uwe Falke" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="ISO-8859-1" Hi Brian, you might use mmfsadm saferdump tscomm to check on which route peer cluster members are reached. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Frank Hammer, Thorsten Moehring Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: Brian Marshall To: gpfsug main discussion list Date: 10/19/2016 07:46 PM Subject: [gpfsug-discuss] subnets Sent by: gpfsug-discuss-bounces at spectrumscale.org All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ------------------------------ Message: 5 Date: Wed, 19 Oct 2016 20:11:57 +0000 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Subject: [gpfsug-discuss] Will there be any more GPFS 4.2.0-x releases? Message-ID: <142FECE0-E157-42D9-BC10-4C48E78FA065 at vanderbilt.edu> Content-Type: text/plain; charset="utf-8" Hi All, We?re currently running GPFS 4.2.0-4 with an efix installed and now we need a 2nd efix. I?m not a big fan of adding efix to efix and would prefer to go to a new PTF that contains both efixes. So ? is there going to be a GPFS 4.2.0-5 (it?s been a longer than normal interval since PTF 4 came out) or do we need to go to GPFS 4.2.1-x? If the latter, any major changes to watch out for? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 49 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-1466780990050_DSTlogo.png.png Type: image/png Size: 6282 bytes Desc: OutlookEmoji-1466780990050_DSTlogo.png.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg Type: image/jpeg Size: 14887 bytes Desc: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg URL: From knop at us.ibm.com Wed Oct 19 22:35:43 2016 From: knop at us.ibm.com (Felipe Knop) Date: Wed, 19 Oct 2016 17:35:43 -0400 Subject: [gpfsug-discuss] Will there be any more GPFS 4.2.0-x releases? In-Reply-To: <142FECE0-E157-42D9-BC10-4C48E78FA065@vanderbilt.edu> References: <142FECE0-E157-42D9-BC10-4C48E78FA065@vanderbilt.edu> Message-ID: Kevin, There will no longer be further PTFs on the 4.2.0 stream. Fixes are now provided on 4.2.1. See https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.1/com.ibm.spectrum.scale.v4r21.doc/bl1xx_soc.htm for the functional enhancements for 4.2.1. See https://www.ibm.com/developerworks/community/forums/html/topic?id=14de7136-e7da-4f93-9f50-5981af1b3f54&ps=50 for announcements on 4.2.1, including the changelog for each PTF. Felipe ---- Felipe Knop knop at us.ibm.com GPFS Development and Security IBM Systems IBM Building 008 2455 South Rd, Poughkeepsie, NY 12601 (845) 433-9314 T/L 293-9314 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 10/19/2016 04:12 PM Subject: [gpfsug-discuss] Will there be any more GPFS 4.2.0-x releases? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, We?re currently running GPFS 4.2.0-4 with an efix installed and now we need a 2nd efix. I?m not a big fan of adding efix to efix and would prefer to go to a new PTF that contains both efixes. So ? is there going to be a GPFS 4.2.0-5 (it?s been a longer than normal interval since PTF 4 came out) or do we need to go to GPFS 4.2.1-x? If the latter, any major changes to watch out for? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From douglasof at us.ibm.com Thu Oct 20 01:07:17 2016 From: douglasof at us.ibm.com (Douglas O'flaherty) Date: Wed, 19 Oct 2016 20:07:17 -0400 Subject: [gpfsug-discuss] SC16 & U/G Registration Message-ID: All: Posting... or re-posting, the URLs for registration to the IBM Spectrum Scale User Group and other IBM briefings IBM summary page: http://www-03.ibm.com/systems/technicalcomputing/supercomputing/index.html Link to IBM event and registrations https://www-01.ibm.com/events/wwe/grp/grp305.nsf/Agenda.xsp?openform&seminar=357M7UES&locale=en_US&S_TACT=000001CP&S_OFF_CD=10001235 We are ramping up for a fantastic event at SC16. Keep an ear out for our next announcements in November, which we will cover in both the user group and more deeply in the IBM seminars and briefings. For those not attending SC16, we will have a briefing webinar on the latest advances in IBM Spectrum Scale and ESS on 11/14. Registration link to follow. doug IBM Spectrum Storage PMM -------------- next part -------------- An HTML attachment was scrubbed... URL: From YARD at il.ibm.com Thu Oct 20 07:15:54 2016 From: YARD at il.ibm.com (Yaron Daniel) Date: Thu, 20 Oct 2016 09:15:54 +0300 Subject: [gpfsug-discuss] Using AFM to migrate files. In-Reply-To: References: <7j173edatp91hqocoto2bau8.1476818437765@email.android.com>, Message-ID: Hi Does you use NFSv4 acls in your old cluster ? Regards Yaron Daniel 94 Em Ha'Moshavot Rd Server, Storage and Data Services - Team Leader Petach Tiqva, 49527 Global Technology Services Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com IBM Israel From: Peter Childs To: gpfsug main discussion list Date: 10/19/2016 05:34 PM Subject: [gpfsug-discuss] Using AFM to migrate files. Sent by: gpfsug-discuss-bounces at spectrumscale.org We are planning to use AFM to migrate our old GPFS file store to a new GPFS file store. This will give us the advantages of Spectrum Scale (GPFS) 4.2, such as larger block and inode size. I would like to attempt to gain some insight on my plans before I start. The old file store was running GPFS 3.5 with 512 byte inodes and 1MB block size. We have now upgraded it to 4.1 and are working towards 4.2 with 300TB of files. (385TB max space) this is so we can use both the old and new storage via multi-cluster. We are moving to a new GPFS cluster so we can use the new protocol nodes eventually and also put the new storage machines as cluster managers, as this should be faster and future proof The new hardware has 1PB of space running GPFS 4.2 We have multiple filesets, and would like to maintain our namespace as far as possible. My plan was to. 1. Create a read-only (RO) AFM cache on the new storage (ro) 2a. Move old fileset and replace with SymLink to new. 2b. Convert RO AFM to Local Update (LU) AFM pointing to new parking area of old files. 2c. move user access to new location in cache. 3. Flush everything into cache and disconnect. I've read the docs including the ones on migration but it's not clear if it's safe to move the home of a cache and update the target. It looks like it should be possible and my tests say it works. An alternative plan is to use a Independent Writer (IW) AFM Cache to move the home directories which are pointed to by LDAP. Hence we can move users one at a time and only have to drain the HPC cluster at the end to disconnect the cache. I assume that migrating users over an Independent Writer is safe so long as the users don't use both sides of the cache at once (ie home and target) I'm also interested in any recipe people have on GPFS policies to preseed and flush the cache. We plan to do all the migration using AFM over GPFS we're not currently using NFS and have no plans to start. I believe using GPFS is the faster method to preform the migration. Any suggestions and experience of doing similar migration jobs would be helpful. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1851 bytes Desc: not available URL: From p.childs at qmul.ac.uk Thu Oct 20 11:12:36 2016 From: p.childs at qmul.ac.uk (Peter Childs) Date: Thu, 20 Oct 2016 10:12:36 +0000 Subject: [gpfsug-discuss] Using AFM to migrate files. In-Reply-To: References: <7j173edatp91hqocoto2bau8.1476818437765@email.android.com>, , Message-ID: Yes but not a great deal, Peter Childs Research Storage Expert ITS Research Infrastructure Queen Mary, University of London ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Yaron Daniel Sent: Thursday, October 20, 2016 7:15:54 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Using AFM to migrate files. Hi Does you use NFSv4 acls in your old cluster ? Regards ________________________________ Yaron Daniel 94 Em Ha'Moshavot Rd [cid:_1_09E5055809E4FFC4002269E8C2258052] Server, Storage and Data Services- Team Leader Petach Tiqva, 49527 Global Technology Services Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com IBM Israel From: Peter Childs To: gpfsug main discussion list Date: 10/19/2016 05:34 PM Subject: [gpfsug-discuss] Using AFM to migrate files. Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ We are planning to use AFM to migrate our old GPFS file store to a new GPFS file store. This will give us the advantages of Spectrum Scale (GPFS) 4.2, such as larger block and inode size. I would like to attempt to gain some insight on my plans before I start. The old file store was running GPFS 3.5 with 512 byte inodes and 1MB block size. We have now upgraded it to 4.1 and are working towards 4.2 with 300TB of files. (385TB max space) this is so we can use both the old and new storage via multi-cluster. We are moving to a new GPFS cluster so we can use the new protocol nodes eventually and also put the new storage machines as cluster managers, as this should be faster and future proof The new hardware has 1PB of space running GPFS 4.2 We have multiple filesets, and would like to maintain our namespace as far as possible. My plan was to. 1. Create a read-only (RO) AFM cache on the new storage (ro) 2a. Move old fileset and replace with SymLink to new. 2b. Convert RO AFM to Local Update (LU) AFM pointing to new parking area of old files. 2c. move user access to new location in cache. 3. Flush everything into cache and disconnect. I've read the docs including the ones on migration but it's not clear if it's safe to move the home of a cache and update the target. It looks like it should be possible and my tests say it works. An alternative plan is to use a Independent Writer (IW) AFM Cache to move the home directories which are pointed to by LDAP. Hence we can move users one at a time and only have to drain the HPC cluster at the end to disconnect the cache. I assume that migrating users over an Independent Writer is safe so long as the users don't use both sides of the cache at once (ie home and target) I'm also interested in any recipe people have on GPFS policies to preseed and flush the cache. We plan to do all the migration using AFM over GPFS we're not currently using NFS and have no plans to start. I believe using GPFS is the faster method to preform the migration. Any suggestions and experience of doing similar migration jobs would be helpful. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00001.gif Type: image/gif Size: 1851 bytes Desc: ATT00001.gif URL: From p.childs at qmul.ac.uk Thu Oct 20 20:07:44 2016 From: p.childs at qmul.ac.uk (Peter Childs) Date: Thu, 20 Oct 2016 19:07:44 +0000 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) In-Reply-To: References: , Message-ID: <5qv6d7inj2j1pa94kqamk2uf.1476989646711@email.android.com> Yes, most of the filesets are based on research groups, projects or departments, with the exception of scratch and home, hence the idea to use a different method for these filesets. There are approximately 230 million files, the largest of the filesets has 52TB and 63 million files. 300TB in total. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ---- Bill Pappas wrote ---- I have some ideas to suggest given some of my experiences. First, I have some questions: How many files are you migrating? Will you be creating multiple file sets on the target system based off of business or project needs? Like, file set a is for "department a" and file set b is for "large scale project a" Thanks. Bill Pappas 901-619-0585 bpappas at dstonline.com [1466780990050_DSTlogo.png] [http://www.prweb.com/releases/2016/06/prweb13504050.htm] http://www.prweb.com/releases/2016/06/prweb13504050.htm ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org Sent: Wednesday, October 19, 2016 3:12 PM To: gpfsug-discuss at spectrumscale.org Subject: gpfsug-discuss Digest, Vol 57, Issue 49 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Using AFM to migrate files. (Peter Childs) 2. subnets (Brian Marshall) 3. Re: subnets (Simon Thompson (Research Computing - IT Services)) 4. Re: subnets (Uwe Falke) 5. Will there be any more GPFS 4.2.0-x releases? (Buterbaugh, Kevin L) ---------------------------------------------------------------------- Message: 1 Date: Wed, 19 Oct 2016 14:12:41 +0000 From: Peter Childs To: gpfsug main discussion list Subject: [gpfsug-discuss] Using AFM to migrate files. Message-ID: Content-Type: text/plain; charset="iso-8859-1" We are planning to use AFM to migrate our old GPFS file store to a new GPFS file store. This will give us the advantages of Spectrum Scale (GPFS) 4.2, such as larger block and inode size. I would like to attempt to gain some insight on my plans before I start. The old file store was running GPFS 3.5 with 512 byte inodes and 1MB block size. We have now upgraded it to 4.1 and are working towards 4.2 with 300TB of files. (385TB max space) this is so we can use both the old and new storage via multi-cluster. We are moving to a new GPFS cluster so we can use the new protocol nodes eventually and also put the new storage machines as cluster managers, as this should be faster and future proof The new hardware has 1PB of space running GPFS 4.2 We have multiple filesets, and would like to maintain our namespace as far as possible. My plan was to. 1. Create a read-only (RO) AFM cache on the new storage (ro) 2a. Move old fileset and replace with SymLink to new. 2b. Convert RO AFM to Local Update (LU) AFM pointing to new parking area of old files. 2c. move user access to new location in cache. 3. Flush everything into cache and disconnect. I've read the docs including the ones on migration but it's not clear if it's safe to move the home of a cache and update the target. It looks like it should be possible and my tests say it works. An alternative plan is to use a Independent Writer (IW) AFM Cache to move the home directories which are pointed to by LDAP. Hence we can move users one at a time and only have to drain the HPC cluster at the end to disconnect the cache. I assume that migrating users over an Independent Writer is safe so long as the users don't use both sides of the cache at once (ie home and target) I'm also interested in any recipe people have on GPFS policies to preseed and flush the cache. We plan to do all the migration using AFM over GPFS we're not currently using NFS and have no plans to start. I believe using GPFS is the faster method to preform the migration. Any suggestions and experience of doing similar migration jobs would be helpful. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ------------------------------ Message: 2 Date: Wed, 19 Oct 2016 13:46:02 -0400 From: Brian Marshall To: gpfsug main discussion list Subject: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="utf-8" All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 3 Date: Wed, 19 Oct 2016 18:10:38 +0000 From: "Simon Thompson (Research Computing - IT Services)" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="us-ascii" mmdiag --network Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Brian Marshall [mimarsh2 at vt.edu] Sent: 19 October 2016 18:46 To: gpfsug main discussion list Subject: [gpfsug-discuss] subnets All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall ------------------------------ Message: 4 Date: Wed, 19 Oct 2016 20:15:52 +0200 From: "Uwe Falke" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="ISO-8859-1" Hi Brian, you might use mmfsadm saferdump tscomm to check on which route peer cluster members are reached. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Frank Hammer, Thorsten Moehring Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: Brian Marshall To: gpfsug main discussion list Date: 10/19/2016 07:46 PM Subject: [gpfsug-discuss] subnets Sent by: gpfsug-discuss-bounces at spectrumscale.org All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ------------------------------ Message: 5 Date: Wed, 19 Oct 2016 20:11:57 +0000 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Subject: [gpfsug-discuss] Will there be any more GPFS 4.2.0-x releases? Message-ID: <142FECE0-E157-42D9-BC10-4C48E78FA065 at vanderbilt.edu> Content-Type: text/plain; charset="utf-8" Hi All, We?re currently running GPFS 4.2.0-4 with an efix installed and now we need a 2nd efix. I?m not a big fan of adding efix to efix and would prefer to go to a new PTF that contains both efixes. So ? is there going to be a GPFS 4.2.0-5 (it?s been a longer than normal interval since PTF 4 came out) or do we need to go to GPFS 4.2.1-x? If the latter, any major changes to watch out for? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 49 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-1466780990050_DSTlogo.png.png Type: image/png Size: 6282 bytes Desc: OutlookEmoji-1466780990050_DSTlogo.png.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-httpwww.prweb.comreleases201606prweb13504050.htm.jpg Type: image/jpeg Size: 14887 bytes Desc: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg URL: From volobuev at us.ibm.com Fri Oct 21 02:58:43 2016 From: volobuev at us.ibm.com (Yuri L Volobuev) Date: Thu, 20 Oct 2016 18:58:43 -0700 Subject: [gpfsug-discuss] Two new whitepapers published Message-ID: Esteemed GPFS and Spectrum Scale users, For your reading enjoyment, two new whitepapers have been posted to the Spectrum Scale Wiki: Spectrum?Scale:?Replication?in?GPFS Spectrum?Scale:?GPFS?Metadata The URL for the parent page is https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20 (GPFS)/page/White%20Papers%20%26%20Media The two .pdf documents are accessible through the Attachment section at the bottom of the page. ?Unfortunately, dW "spam prevention engine" does a very good job preventing me from "spamming" the page to actually add links. Best regards, Yuri -------------- next part -------------- An HTML attachment was scrubbed... URL: From alandhae at gmx.de Fri Oct 21 07:43:40 2016 From: alandhae at gmx.de (=?ISO-8859-15?Q?Andreas_Landh=E4u=DFer?=) Date: Fri, 21 Oct 2016 08:43:40 +0200 (CEST) Subject: [gpfsug-discuss] Two new whitepapers published In-Reply-To: References: Message-ID: Hi Yuri, Arrg, can't find them, page has been last updated on Aug, 18 by JohnTOlson, maybe its internal and not open for the public? Ciao Andreas On Fri, 21 Oct 2016, Yuri L Volobuev wrote: > > > Esteemed GPFS and Spectrum Scale users, > > For your reading enjoyment, two new whitepapers have been posted to the > Spectrum Scale Wiki: > > Spectrum?Scale:?Replication?in?GPFS > Spectrum?Scale:?GPFS?Metadata > > The URL for the parent page is > https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20 > (GPFS)/page/White%20Papers%20%26%20Media > > The two .pdf documents are accessible through the Attachment section at the > bottom of the page. ?Unfortunately, dW "spam prevention engine" does a very > good job preventing me from "spamming" the page to actually add links. > > Best regards, > > Yuri > -- Andreas Landh?u?er +49 151 12133027 (mobile) alandhae at gmx.de From dave at milk-vfx.com Fri Oct 21 07:51:51 2016 From: dave at milk-vfx.com (Dave Goodbourn) Date: Fri, 21 Oct 2016 07:51:51 +0100 Subject: [gpfsug-discuss] Two new whitepapers published In-Reply-To: References: Message-ID: I can find them. They're in the attachments tab in the bottom set of tabs. And a very good read. Thanks Yuri! Dave. > On 21 Oct 2016, at 07:43, Andreas Landh?u?er wrote: > > > Hi Yuri, > > Arrg, can't find them, page has been last updated on Aug, 18 by JohnTOlson, maybe its internal and not open for the public? > > Ciao > > Andreas > >> On Fri, 21 Oct 2016, Yuri L Volobuev wrote: >> >> >> >> Esteemed GPFS and Spectrum Scale users, >> >> For your reading enjoyment, two new whitepapers have been posted to the >> Spectrum Scale Wiki: >> >> Spectrum Scale: Replication in GPFS >> Spectrum Scale: GPFS Metadata >> >> The URL for the parent page is >> https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20 >> (GPFS)/page/White%20Papers%20%26%20Media >> >> The two .pdf documents are accessible through the Attachment section at the >> bottom of the page. Unfortunately, dW "spam prevention engine" does a very >> good job preventing me from "spamming" the page to actually add links. >> >> Best regards, >> >> Yuri >> > > -- > Andreas Landh?u?er +49 151 12133027 (mobile) > alandhae at gmx.de > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From laurence at qsplace.co.uk Fri Oct 21 08:10:56 2016 From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow) Date: Fri, 21 Oct 2016 08:10:56 +0100 Subject: [gpfsug-discuss] Two new whitepapers published In-Reply-To: References: Message-ID: <36EC1962-F71F-4492-A175-4BA3E9D97A8D@qsplace.co.uk> Right down the bottom of the page under attachments. -- Lauz On 21 October 2016 07:43:40 BST, "Andreas Landh?u?er" wrote: > >Hi Yuri, > >Arrg, can't find them, page has been last updated on Aug, 18 by >JohnTOlson, maybe its internal and not open for the public? > >Ciao > > Andreas > >On Fri, 21 Oct 2016, Yuri L Volobuev wrote: > >> >> >> Esteemed GPFS and Spectrum Scale users, >> >> For your reading enjoyment, two new whitepapers have been posted to >the >> Spectrum Scale Wiki: >> >> Spectrum?Scale:?Replication?in?GPFS >> Spectrum?Scale:?GPFS?Metadata >> >> The URL for the parent page is >> >https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20 >> (GPFS)/page/White%20Papers%20%26%20Media >> >> The two .pdf documents are accessible through the Attachment section >at the >> bottom of the page. ?Unfortunately, dW "spam prevention engine" does >a very >> good job preventing me from "spamming" the page to actually add >links. >> >> Best regards, >> >> Yuri >> > >-- >Andreas Landh?u?er +49 151 12133027 (mobile) >alandhae at gmx.de > >------------------------------------------------------------------------ > >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alandhae at gmx.de Fri Oct 21 08:31:43 2016 From: alandhae at gmx.de (=?ISO-8859-15?Q?Andreas_Landh=E4u=DFer?=) Date: Fri, 21 Oct 2016 09:31:43 +0200 (CEST) Subject: [gpfsug-discuss] Two new whitepapers published In-Reply-To: <36EC1962-F71F-4492-A175-4BA3E9D97A8D@qsplace.co.uk> References: <36EC1962-F71F-4492-A175-4BA3E9D97A8D@qsplace.co.uk> Message-ID: On Fri, 21 Oct 2016, Laurence Horrocks-Barlow wrote: Found it! thanks Yuri, for the two very interesting papers about the Metadata and replication. We always used the rule of thumb 5% for metadata, since we are having separate devices for metadata, and tiny fast disks aren't available anymore, we are getting a bunch of larger fast disks. We never experienced a problem with the (more or less) amount of metadata ... Andreas > Right down the bottom of the page under attachments. > > -- Lauz > > On 21 October 2016 07:43:40 BST, "Andreas Landh?u?er" wrote: >> >> Hi Yuri, >> >> Arrg, can't find them, page has been last updated on Aug, 18 by >> JohnTOlson, maybe its internal and not open for the public? >> >> Ciao >> >> Andreas >> >> On Fri, 21 Oct 2016, Yuri L Volobuev wrote: >> >>> >>> >>> Esteemed GPFS and Spectrum Scale users, >>> >>> For your reading enjoyment, two new whitepapers have been posted to >> the >>> Spectrum Scale Wiki: >>> >>> Spectrum?Scale:?Replication?in?GPFS >>> Spectrum?Scale:?GPFS?Metadata >>> >>> The URL for the parent page is >>> >> https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20 >>> (GPFS)/page/White%20Papers%20%26%20Media >>> >>> The two .pdf documents are accessible through the Attachment section >> at the >>> bottom of the page. ?Unfortunately, dW "spam prevention engine" does >> a very >>> good job preventing me from "spamming" the page to actually add >> links. >>> >>> Best regards, >>> >>> Yuri >>> >> >> -- >> Andreas Landh?u?er +49 151 12133027 (mobile) >> alandhae at gmx.de >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- Andreas Landh?u?er +49 151 12133027 (mobile) alandhae at gmx.de From daniel.kidger at uk.ibm.com Fri Oct 21 14:48:10 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Fri, 21 Oct 2016 13:48:10 +0000 Subject: [gpfsug-discuss] Two new whitepapers published In-Reply-To: <36EC1962-F71F-4492-A175-4BA3E9D97A8D@qsplace.co.uk> Message-ID: Great papers ! @Yuri: do you want to add entries at the top of that page to: a) help people find those papers, and b) make it more obvious that this page is not obsolete and no longer maintained. Daniel IBM Spectrum Storage Software +44 (0)7818 522266 Sent from my iPad using IBM Verse On 21 Oct 2016, 08:11:23, laurence at qsplace.co.uk wrote: From: laurence at qsplace.co.uk To: alandhae at gmx.de, gpfsug-discuss at spectrumscale.org Cc: Date: 21 Oct 2016 08:11:23 Subject: Re: [gpfsug-discuss] Two new whitepapers published Right down the bottom of the page under attachments. -- Lauz On 21 October 2016 07:43:40 BST, "Andreas Landh?u?er" wrote: Hi Yuri,Arrg, can't find them, page has been last updated on Aug, 18 by JohnTOlson, maybe its internal and not open for the public?Ciao AndreasOn Fri, 21 Oct 2016, Yuri L Volobuev wrote: Esteemed GPFS and Spectrum Scale users, For your reading enjoyment, two new whitepapers have been posted to the Spectrum Scale Wiki: Spectrum Scale: Replication in GPFS Spectrum Scale: GPFS Metadata The URL for the parent page is https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20 (GPFS)/page/White%20Papers%20%26%20Media The two .pdf documents are accessible through the Attachment section at the bottom of the page. Unfortunately, dW "spam prevention engine" does a very good job preventing me from "spamming" the page to actually add links. Best regards, Yuri -- Sent from my Android device with K-9 Mail. Please excuse my brevity.Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From volobuev at us.ibm.com Fri Oct 21 16:45:42 2016 From: volobuev at us.ibm.com (Yuri L Volobuev) Date: Fri, 21 Oct 2016 08:45:42 -0700 Subject: [gpfsug-discuss] Two new whitepapers published In-Reply-To: References: <36EC1962-F71F-4492-A175-4BA3E9D97A8D@qsplace.co.uk> Message-ID: I tried to do just that... the dW "spam prevention engine" wouldn't let me update the page. It may take some time to do this right. yuri From: "Daniel Kidger" To: "gpfsug main discussion list" , Date: 10/21/2016 06:48 AM Subject: Re: [gpfsug-discuss] Two new whitepapers published Sent by: gpfsug-discuss-bounces at spectrumscale.org Great papers ! @Yuri: do you want to add entries at the top of that page to: a) help people find those papers, and b) make it more obvious that this page is not obsolete and no longer maintained. Daniel IBM Spectrum Storage Software +44 (0)7818 522266 Sent from my iPad using IBM Verse On 21 Oct 2016, 08:11:23, laurence at qsplace.co.uk wrote: From: laurence at qsplace.co.uk To: alandhae at gmx.de, gpfsug-discuss at spectrumscale.org Cc: Date: 21 Oct 2016 08:11:23 Subject: Re: [gpfsug-discuss] Two new whitepapers published Right down the bottom of the page under attachments. -- Lauz On 21 October 2016 07:43:40 BST, "Andreas Landh?u?er" wrote: Hi Yuri, Arrg, can't find them, page has been last updated on Aug, 18 by JohnTOlson, maybe its internal and not open for the public? Ciao Andreas On Fri, 21 Oct 2016, Yuri L Volobuev wrote: Esteemed GPFS and Spectrum Scale users, For your reading enjoyment, two new whitepapers have been posted to the Spectrum Scale Wiki: Spectrum Scale: Replication in GPFS Spectrum Scale: GPFS Metadata The URL for the parent page is https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20 (GPFS)/page/White%20Papers%20%26%20Media The two .pdf documents are accessible through the Attachment section at the bottom of the page. Unfortunately, dW "spam prevention engine" does a very good job preventing me from "spamming" the page to actually add links. Best regards, Yuri -- Sent from my Android device with K-9 Mail. Please excuse my brevity. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From bpappas at dstonline.com Fri Oct 21 19:46:09 2016 From: bpappas at dstonline.com (Bill Pappas) Date: Fri, 21 Oct 2016 18:46:09 +0000 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) In-Reply-To: References: Message-ID: >>the largest of the filesets has 52TB and 63 million files Are you using NFS as the transport path between the home and cache? If you are using NFS, how are you producing the list of files to migrate? mmafmctl with the prefetch option? If so, I would measure the time it takes for that command (with that option) to produce the list of files it intends to prefetch. From my experience, this is very important as a) it can take a long time if you have >10 million of files and b) I've seen this operation crash when the list grew large. Does anyone else on this thread have any experiences? I would love to hear positive experiences as well. I tried so hard and for so long to make AFM work with one customer, but we gave up as it was not reliable and stable for large scale (many files) migrations. If you are using GPFS as the conduit between the home and cache (i.e. no NFS), I would still ask the same question, more with respect to stability for large file lists during the initial prefetch stages. As far as I could tell, from GPFS 3.5 to 4.2, the phases of prefetch where the home and cache are compared (i.e. let's make a list of what is ot be migrated over) before the data transfer begins only runs on the GW node managing that cache. It does not leverage multiple gw nodes and multiple home nodes to speed up this 'list and find' stage of prefetch. I hope some AFM developers can clarify or correct my findings. This was a huge impediment for large file migrations where it is difficult (organizationally, not technically) to split a folder structure into multiple file sets. The lack of stability under these large scans was the real failing for us. Bill Pappas 901-619-0585 bpappas at dstonline.com [1466780990050_DSTlogo.png] [http://www.prweb.com/releases/2016/06/prweb13504050.htm] http://www.prweb.com/releases/2016/06/prweb13504050.htm ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org Sent: Thursday, October 20, 2016 2:07 PM To: gpfsug-discuss at spectrumscale.org Subject: gpfsug-discuss Digest, Vol 57, Issue 53 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Using AFM to migrate files. (Peter Childs) (Peter Childs) ---------------------------------------------------------------------- Message: 1 Date: Thu, 20 Oct 2016 19:07:44 +0000 From: Peter Childs To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) Message-ID: <5qv6d7inj2j1pa94kqamk2uf.1476989646711 at email.android.com> Content-Type: text/plain; charset="iso-8859-1" Yes, most of the filesets are based on research groups, projects or departments, with the exception of scratch and home, hence the idea to use a different method for these filesets. There are approximately 230 million files, the largest of the filesets has 52TB and 63 million files. 300TB in total. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ---- Bill Pappas wrote ---- I have some ideas to suggest given some of my experiences. First, I have some questions: How many files are you migrating? Will you be creating multiple file sets on the target system based off of business or project needs? Like, file set a is for "department a" and file set b is for "large scale project a" Thanks. Bill Pappas 901-619-0585 bpappas at dstonline.com [1466780990050_DSTlogo.png] [http://www.prweb.com/releases/2016/06/prweb13504050.htm] http://www.prweb.com/releases/2016/06/prweb13504050.htm ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org Sent: Wednesday, October 19, 2016 3:12 PM To: gpfsug-discuss at spectrumscale.org Subject: gpfsug-discuss Digest, Vol 57, Issue 49 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Using AFM to migrate files. (Peter Childs) 2. subnets (Brian Marshall) 3. Re: subnets (Simon Thompson (Research Computing - IT Services)) 4. Re: subnets (Uwe Falke) 5. Will there be any more GPFS 4.2.0-x releases? (Buterbaugh, Kevin L) ---------------------------------------------------------------------- Message: 1 Date: Wed, 19 Oct 2016 14:12:41 +0000 From: Peter Childs To: gpfsug main discussion list Subject: [gpfsug-discuss] Using AFM to migrate files. Message-ID: Content-Type: text/plain; charset="iso-8859-1" We are planning to use AFM to migrate our old GPFS file store to a new GPFS file store. This will give us the advantages of Spectrum Scale (GPFS) 4.2, such as larger block and inode size. I would like to attempt to gain some insight on my plans before I start. The old file store was running GPFS 3.5 with 512 byte inodes and 1MB block size. We have now upgraded it to 4.1 and are working towards 4.2 with 300TB of files. (385TB max space) this is so we can use both the old and new storage via multi-cluster. We are moving to a new GPFS cluster so we can use the new protocol nodes eventually and also put the new storage machines as cluster managers, as this should be faster and future proof The new hardware has 1PB of space running GPFS 4.2 We have multiple filesets, and would like to maintain our namespace as far as possible. My plan was to. 1. Create a read-only (RO) AFM cache on the new storage (ro) 2a. Move old fileset and replace with SymLink to new. 2b. Convert RO AFM to Local Update (LU) AFM pointing to new parking area of old files. 2c. move user access to new location in cache. 3. Flush everything into cache and disconnect. I've read the docs including the ones on migration but it's not clear if it's safe to move the home of a cache and update the target. It looks like it should be possible and my tests say it works. An alternative plan is to use a Independent Writer (IW) AFM Cache to move the home directories which are pointed to by LDAP. Hence we can move users one at a time and only have to drain the HPC cluster at the end to disconnect the cache. I assume that migrating users over an Independent Writer is safe so long as the users don't use both sides of the cache at once (ie home and target) I'm also interested in any recipe people have on GPFS policies to preseed and flush the cache. We plan to do all the migration using AFM over GPFS we're not currently using NFS and have no plans to start. I believe using GPFS is the faster method to preform the migration. Any suggestions and experience of doing similar migration jobs would be helpful. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ------------------------------ Message: 2 Date: Wed, 19 Oct 2016 13:46:02 -0400 From: Brian Marshall To: gpfsug main discussion list Subject: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="utf-8" All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 3 Date: Wed, 19 Oct 2016 18:10:38 +0000 From: "Simon Thompson (Research Computing - IT Services)" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="us-ascii" mmdiag --network Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Brian Marshall [mimarsh2 at vt.edu] Sent: 19 October 2016 18:46 To: gpfsug main discussion list Subject: [gpfsug-discuss] subnets All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall ------------------------------ Message: 4 Date: Wed, 19 Oct 2016 20:15:52 +0200 From: "Uwe Falke" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="ISO-8859-1" Hi Brian, you might use mmfsadm saferdump tscomm to check on which route peer cluster members are reached. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Frank Hammer, Thorsten Moehring Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: Brian Marshall To: gpfsug main discussion list Date: 10/19/2016 07:46 PM Subject: [gpfsug-discuss] subnets Sent by: gpfsug-discuss-bounces at spectrumscale.org All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ------------------------------ Message: 5 Date: Wed, 19 Oct 2016 20:11:57 +0000 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Subject: [gpfsug-discuss] Will there be any more GPFS 4.2.0-x releases? Message-ID: <142FECE0-E157-42D9-BC10-4C48E78FA065 at vanderbilt.edu> Content-Type: text/plain; charset="utf-8" Hi All, We?re currently running GPFS 4.2.0-4 with an efix installed and now we need a 2nd efix. I?m not a big fan of adding efix to efix and would prefer to go to a new PTF that contains both efixes. So ? is there going to be a GPFS 4.2.0-5 (it?s been a longer than normal interval since PTF 4 came out) or do we need to go to GPFS 4.2.1-x? If the latter, any major changes to watch out for? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 49 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-1466780990050_DSTlogo.png.png Type: image/png Size: 6282 bytes Desc: OutlookEmoji-1466780990050_DSTlogo.png.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-httpwww.prweb.comreleases201606prweb13504050.htm.jpg Type: image/jpeg Size: 14887 bytes Desc: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 53 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-1466780990050_DSTlogo.png.png Type: image/png Size: 6282 bytes Desc: OutlookEmoji-1466780990050_DSTlogo.png.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg Type: image/jpeg Size: 14887 bytes Desc: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg URL: From p.childs at qmul.ac.uk Fri Oct 21 21:35:15 2016 From: p.childs at qmul.ac.uk (Peter Childs) Date: Fri, 21 Oct 2016 20:35:15 +0000 Subject: [gpfsug-discuss] Two new whitepapers published In-Reply-To: References: <36EC1962-F71F-4492-A175-4BA3E9D97A8D@qsplace.co.uk>, Message-ID: <7fifaf55s6bq10jlpkvl5he9.1477081370569@email.android.com> Reading through them and we'll worth read. How important is setting the correct cluster size at file system creation time? Ie with "mmchfs -n 256" ie how much band of error can you get away with? We have a cluster of 240 nodes that was setup with the default 32 setting, it's now about to grow to ~300 nodes. Is this likly to causing as an issue, which is difficult to fix. The manual says it can be adjusted but this white paper suggests not. Fortunately were also migrating our storage to new hardware, so have a good opportunity to get the setting right, this time around. Has anyone got any stats on the benefits of getting it "right" vs getting it wrong. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ---- Andreas Landh?u?er wrote ---- On Fri, 21 Oct 2016, Laurence Horrocks-Barlow wrote: Found it! thanks Yuri, for the two very interesting papers about the Metadata and replication. We always used the rule of thumb 5% for metadata, since we are having separate devices for metadata, and tiny fast disks aren't available anymore, we are getting a bunch of larger fast disks. We never experienced a problem with the (more or less) amount of metadata ... Andreas > Right down the bottom of the page under attachments. > > -- Lauz > > On 21 October 2016 07:43:40 BST, "Andreas Landh?u?er" wrote: >> >> Hi Yuri, >> >> Arrg, can't find them, page has been last updated on Aug, 18 by >> JohnTOlson, maybe its internal and not open for the public? >> >> Ciao >> >> Andreas >> >> On Fri, 21 Oct 2016, Yuri L Volobuev wrote: >> >>> >>> >>> Esteemed GPFS and Spectrum Scale users, >>> >>> For your reading enjoyment, two new whitepapers have been posted to >> the >>> Spectrum Scale Wiki: >>> >>> Spectrum Scale: Replication in GPFS >>> Spectrum Scale: GPFS Metadata >>> >>> The URL for the parent page is >>> >> https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20 >>> (GPFS)/page/White%20Papers%20%26%20Media >>> >>> The two .pdf documents are accessible through the Attachment section >> at the >>> bottom of the page. Unfortunately, dW "spam prevention engine" does >> a very >>> good job preventing me from "spamming" the page to actually add >> links. >>> >>> Best regards, >>> >>> Yuri >>> >> >> -- >> Andreas Landh?u?er +49 151 12133027 (mobile) >> alandhae at gmx.de >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- Andreas Landh?u?er +49 151 12133027 (mobile) alandhae at gmx.de -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.childs at qmul.ac.uk Fri Oct 21 21:44:18 2016 From: p.childs at qmul.ac.uk (Peter Childs) Date: Fri, 21 Oct 2016 20:44:18 +0000 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) In-Reply-To: References: , Message-ID: <9msuk69qtso8etpj6c9mq2pk.1477082656580@email.android.com> ---- Bill Pappas wrote ---- > >>the largest of the filesets has 52TB and 63 million files > > > Are you using NFS as the transport path between the home and cache? No plans to was planning to use gpfs multi-cluster, as transport. > If you are using NFS, how are you producing the list of files to migrate? mmafmctl with the prefetch option? If so, I would measure the time it takes for that command (with that option) to produce the list of files it intends to prefetch. From my experience, this is very important as a) it can take a long time if you have >10 million of files and b) I've seen this operation crash when the list grew large. Does anyone else on this thread have any experiences? I would love to hear positive experiences as well. I tried so hard and for so long to make AFM work with one customer, but we gave up as it was not reliable and stable for large scale (many files) migrations. > If you are using GPFS as the conduit between the home and cache (i.e. no NFS), I would still ask the same question, more with respect to stability for large file lists during the initial prefetch stages. I was planning to use a gpfs policy to create the list, but I guess a find should work, I'm guessing your saying don't migrate the files in bulk by using a find onto cache. It would be nice to see some examples recipes to prefetch files into afm. > > > As far as I could tell, from GPFS 3.5 to 4.2, the phases of prefetch where the home and cache are compared (i.e. let's make a list of what is ot be migrated over) before the data transfer begins only runs on the GW node managing that cache. It does not leverage multiple gw nodes and multiple home nodes to speed up this 'list and find' stage of prefetch. I hope some AFM developers can clarify or correct my findings. This was a huge impediment for large file migrations where it is difficult (organizationally, not technically) to split a folder structure into multiple file sets. The lack of stability under these large scans was the real failing for us. Interesting. > > > Bill Pappas > > 901-619-0585 > > bpappas at dstonline.com > > Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London > > > > > > http://www.prweb.com/releases/2016/06/prweb13504050.htm > > > > From: gpfsug-discuss-bounces at spectrumscale.org > on behalf of gpfsug-discuss-request at spectrumscale.org > > Sent: Thursday, October 20, 2016 2:07 PM > To: gpfsug-discuss at spectrumscale.org > Subject: gpfsug-discuss Digest, Vol 57, Issue 53 > > Send gpfsug-discuss mailing list submissions to > gpfsug-discuss at spectrumscale.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > or, via email, send a message with subject or body 'help' to > gpfsug-discuss-request at spectrumscale.org > > You can reach the person managing the list at > gpfsug-discuss-owner at spectrumscale.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of gpfsug-discuss digest..." > > > Today's Topics: > > 1. Re: Using AFM to migrate files. (Peter Childs) (Peter Childs) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 20 Oct 2016 19:07:44 +0000 > From: Peter Childs > > To: gpfsug main discussion list > > Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter > Childs) > Message-ID: <5qv6d7inj2j1pa94kqamk2uf.1476989646711 at email.android.com> > Content-Type: text/plain; charset="iso-8859-1" > > Yes, most of the filesets are based on research groups, projects or departments, with the exception of scratch and home, hence the idea to use a different method for these filesets. > > There are approximately 230 million files, the largest of the filesets has 52TB and 63 million files. 300TB in total. > > Peter Childs > Research Storage > ITS Research and Teaching Support > Queen Mary, University of London > > > ---- Bill Pappas wrote ---- > > > I have some ideas to suggest given some of my experiences. First, I have some questions: > > > How many files are you migrating? > > Will you be creating multiple file sets on the target system based off of business or project needs? Like, file set a is for "department a" and file set b is for "large scale project a" > > > Thanks. > > > Bill Pappas > > 901-619-0585 > > bpappas at dstonline.com > > > [1466780990050_DSTlogo.png] > > > [http://www.prweb.com/releases/2016/06/prweb13504050.htm] > > http://www.prweb.com/releases/2016/06/prweb13504050.htm > > > ________________________________ > From: gpfsug-discuss-bounces at spectrumscale.org > on behalf of gpfsug-discuss-request at spectrumscale.org > > Sent: Wednesday, October 19, 2016 3:12 PM > To: gpfsug-discuss at spectrumscale.org > Subject: gpfsug-discuss Digest, Vol 57, Issue 49 > > Send gpfsug-discuss mailing list submissions to > gpfsug-discuss at spectrumscale.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > or, via email, send a message with subject or body 'help' to > gpfsug-discuss-request at spectrumscale.org > > You can reach the person managing the list at > gpfsug-discuss-owner at spectrumscale.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of gpfsug-discuss digest..." > > > Today's Topics: > > 1. Using AFM to migrate files. (Peter Childs) > 2. subnets (Brian Marshall) > 3. Re: subnets (Simon Thompson (Research Computing - IT Services)) > 4. Re: subnets (Uwe Falke) > 5. Will there be any more GPFS 4.2.0-x releases? > (Buterbaugh, Kevin L) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 19 Oct 2016 14:12:41 +0000 > From: Peter Childs > > To: gpfsug main discussion list > > Subject: [gpfsug-discuss] Using AFM to migrate files. > Message-ID: > > > > Content-Type: text/plain; charset="iso-8859-1" > > > We are planning to use AFM to migrate our old GPFS file store to a new GPFS file store. This will give us the advantages of Spectrum Scale (GPFS) 4.2, such as larger block and inode size. I would like to attempt to gain some insight on my plans before I start. > > The old file store was running GPFS 3.5 with 512 byte inodes and 1MB block size. We have now upgraded it to 4.1 and are working towards 4.2 with 300TB of files. (385TB max space) this is so we can use both the old and new storage via multi-cluster. > > We are moving to a new GPFS cluster so we can use the new protocol nodes eventually and also put the new storage machines as cluster managers, as this should be faster and future proof > > The new hardware has 1PB of space running GPFS 4.2 > > We have multiple filesets, and would like to maintain our namespace as far as possible. > > My plan was to. > > 1. Create a read-only (RO) AFM cache on the new storage (ro) > 2a. Move old fileset and replace with SymLink to new. > 2b. Convert RO AFM to Local Update (LU) AFM pointing to new parking area of old files. > 2c. move user access to new location in cache. > 3. Flush everything into cache and disconnect. > > I've read the docs including the ones on migration but it's not clear if it's safe to move the home of a cache and update the target. It looks like it should be possible and my tests say it works. > > An alternative plan is to use a Independent Writer (IW) AFM Cache to move the home directories which are pointed to by LDAP. Hence we can move users one at a time and only have to drain the HPC cluster at the end to disconnect the cache. I assume that migrating users over an Independent Writer is safe so long as the users don't use both sides of the cache at once (ie home and target) > > I'm also interested in any recipe people have on GPFS policies to preseed and flush the cache. > > We plan to do all the migration using AFM over GPFS we're not currently using NFS and have no plans to start. I believe using GPFS is the faster method to preform the migration. > > Any suggestions and experience of doing similar migration jobs would be helpful. > > Peter Childs > Research Storage > ITS Research and Teaching Support > Queen Mary, University of London > > > > ------------------------------ > > Message: 2 > Date: Wed, 19 Oct 2016 13:46:02 -0400 > From: Brian Marshall > > To: gpfsug main discussion list > > Subject: [gpfsug-discuss] subnets > Message-ID: > > > Content-Type: text/plain; charset="utf-8" > > All, > > We are setting up communication between 2 clusters using ethernet and > IPoFabric. > > The Daemon interface is running on ethernet, so all admin traffic will use > it. > > We are still getting the subnets setting correct. > > Question: > > Does GPFS have a way to query how it is connecting to a given cluster/node? > i.e. once we have subnets setup how can we tell GPFS is actually using > them. Currently we just do a large transfer and check tcpdump for any > packets flowing on the high-speed/data/non-admin subnet. > > > Thank you, > Brian Marshall > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: ; > > ------------------------------ > > Message: 3 > Date: Wed, 19 Oct 2016 18:10:38 +0000 > From: "Simon Thompson (Research Computing - IT Services)" > > > To: gpfsug main discussion list > > Subject: Re: [gpfsug-discuss] subnets > Message-ID: > > > Content-Type: text/plain; charset="us-ascii" > > > mmdiag --network > > Simon > ________________________________________ > From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Brian Marshall [mimarsh2 at vt.edu] > Sent: 19 October 2016 18:46 > To: gpfsug main discussion list > Subject: [gpfsug-discuss] subnets > > All, > > We are setting up communication between 2 clusters using ethernet and IPoFabric. > > The Daemon interface is running on ethernet, so all admin traffic will use it. > > We are still getting the subnets setting correct. > > Question: > > Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. > > > Thank you, > Brian Marshall > > > ------------------------------ > > Message: 4 > Date: Wed, 19 Oct 2016 20:15:52 +0200 > From: "Uwe Falke" > > To: gpfsug main discussion list > > Subject: Re: [gpfsug-discuss] subnets > Message-ID: > > > > Content-Type: text/plain; charset="ISO-8859-1" > > Hi Brian, > you might use > > mmfsadm saferdump tscomm > > to check on which route peer cluster members are reached. > > > Mit freundlichen Gr??en / Kind regards > > > Dr. Uwe Falke > > IT Specialist > High Performance Computing Services / Integrated Technology Services / > Data Center Services > ------------------------------------------------------------------------------------------------------------------------------------------- > IBM Deutschland > Rathausstr. 7 > 09111 Chemnitz > Phone: +49 371 6978 2165 > Mobile: +49 175 575 2877 > E-Mail: uwefalke at de.ibm.com > ------------------------------------------------------------------------------------------------------------------------------------------- > IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: > Frank Hammer, Thorsten Moehring > Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, > HRB 17122 > > > > > From: Brian Marshall > > To: gpfsug main discussion list > > Date: 10/19/2016 07:46 PM > Subject: [gpfsug-discuss] subnets > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > All, > > We are setting up communication between 2 clusters using ethernet and > IPoFabric. > > The Daemon interface is running on ethernet, so all admin traffic will use > it. > > We are still getting the subnets setting correct. > > Question: > > Does GPFS have a way to query how it is connecting to a given > cluster/node? i.e. once we have subnets setup how can we tell GPFS is > actually using them. Currently we just do a large transfer and check > tcpdump for any packets flowing on the high-speed/data/non-admin subnet. > > > Thank you, > Brian Marshall_______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > ------------------------------ > > Message: 5 > Date: Wed, 19 Oct 2016 20:11:57 +0000 > From: "Buterbaugh, Kevin L" > > To: gpfsug main discussion list > > Subject: [gpfsug-discuss] Will there be any more GPFS 4.2.0-x > releases? > Message-ID: <142FECE0-E157-42D9-BC10-4C48E78FA065 at vanderbilt.edu> > Content-Type: text/plain; charset="utf-8" > > Hi All, > > We?re currently running GPFS 4.2.0-4 with an efix installed and now we need a 2nd efix. I?m not a big fan of adding efix to efix and would prefer to go to a new PTF that contains both efixes. > > So ? is there going to be a GPFS 4.2.0-5 (it?s been a longer than normal interval since PTF 4 came out) or do we need to go to GPFS 4.2.1-x? If the latter, any major changes to watch out for? Thanks? > > Kevin > > ? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and Education > Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633 > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: ; > > ------------------------------ > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > End of gpfsug-discuss Digest, Vol 57, Issue 49 > ********************************************** > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: ; > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: OutlookEmoji-1466780990050_DSTlogo.png.png > Type: image/png > Size: 6282 bytes > Desc: OutlookEmoji-1466780990050_DSTlogo.png.png > URL: ; > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: OutlookEmoji-httpwww.prweb.comreleases201606prweb13504050.htm.jpg > Type: image/jpeg > Size: 14887 bytes > Desc: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg > URL: ; > > ------------------------------ > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > End of gpfsug-discuss Digest, Vol 57, Issue 53 > ********************************************** >>the largest of the filesets has 52TB and 63 million files Are you using NFS as the transport path between the home and cache? If you are using NFS, how are you producing the list of files to migrate? mmafmctl with the prefetch option? If so, I would measure the time it takes for that command (with that option) to produce the list of files it intends to prefetch. From my experience, this is very important as a) it can take a long time if you have >10 million of files and b) I've seen this operation crash when the list grew large. Does anyone else on this thread have any experiences? I would love to hear positive experiences as well. I tried so hard and for so long to make AFM work with one customer, but we gave up as it was not reliable and stable for large scale (many files) migrations. If you are using GPFS as the conduit between the home and cache (i.e. no NFS), I would still ask the same question, more with respect to stability for large file lists during the initial prefetch stages. As far as I could tell, from GPFS 3.5 to 4.2, the phases of prefetch where the home and cache are compared (i.e. let's make a list of what is ot be migrated over) before the data transfer begins only runs on the GW node managing that cache. It does not leverage multiple gw nodes and multiple home nodes to speed up this 'list and find' stage of prefetch. I hope some AFM developers can clarify or correct my findings. This was a huge impediment for large file migrations where it is difficult (organizationally, not technically) to split a folder structure into multiple file sets. The lack of stability under these large scans was the real failing for us. Bill Pappas 901-619-0585 bpappas at dstonline.com [1466780990050_DSTlogo.png] [http://www.prweb.com/releases/2016/06/prweb13504050.htm] http://www.prweb.com/releases/2016/06/prweb13504050.htm ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org Sent: Thursday, October 20, 2016 2:07 PM To: gpfsug-discuss at spectrumscale.org Subject: gpfsug-discuss Digest, Vol 57, Issue 53 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Using AFM to migrate files. (Peter Childs) (Peter Childs) ---------------------------------------------------------------------- Message: 1 Date: Thu, 20 Oct 2016 19:07:44 +0000 From: Peter Childs To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) Message-ID: <5qv6d7inj2j1pa94kqamk2uf.1476989646711 at email.android.com> Content-Type: text/plain; charset="iso-8859-1" Yes, most of the filesets are based on research groups, projects or departments, with the exception of scratch and home, hence the idea to use a different method for these filesets. There are approximately 230 million files, the largest of the filesets has 52TB and 63 million files. 300TB in total. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ---- Bill Pappas wrote ---- I have some ideas to suggest given some of my experiences. First, I have some questions: How many files are you migrating? Will you be creating multiple file sets on the target system based off of business or project needs? Like, file set a is for "department a" and file set b is for "large scale project a" Thanks. Bill Pappas 901-619-0585 bpappas at dstonline.com [1466780990050_DSTlogo.png] [http://www.prweb.com/releases/2016/06/prweb13504050.htm] http://www.prweb.com/releases/2016/06/prweb13504050.htm ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org Sent: Wednesday, October 19, 2016 3:12 PM To: gpfsug-discuss at spectrumscale.org Subject: gpfsug-discuss Digest, Vol 57, Issue 49 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Using AFM to migrate files. (Peter Childs) 2. subnets (Brian Marshall) 3. Re: subnets (Simon Thompson (Research Computing - IT Services)) 4. Re: subnets (Uwe Falke) 5. Will there be any more GPFS 4.2.0-x releases? (Buterbaugh, Kevin L) ---------------------------------------------------------------------- Message: 1 Date: Wed, 19 Oct 2016 14:12:41 +0000 From: Peter Childs To: gpfsug main discussion list Subject: [gpfsug-discuss] Using AFM to migrate files. Message-ID: Content-Type: text/plain; charset="iso-8859-1" We are planning to use AFM to migrate our old GPFS file store to a new GPFS file store. This will give us the advantages of Spectrum Scale (GPFS) 4.2, such as larger block and inode size. I would like to attempt to gain some insight on my plans before I start. The old file store was running GPFS 3.5 with 512 byte inodes and 1MB block size. We have now upgraded it to 4.1 and are working towards 4.2 with 300TB of files. (385TB max space) this is so we can use both the old and new storage via multi-cluster. We are moving to a new GPFS cluster so we can use the new protocol nodes eventually and also put the new storage machines as cluster managers, as this should be faster and future proof The new hardware has 1PB of space running GPFS 4.2 We have multiple filesets, and would like to maintain our namespace as far as possible. My plan was to. 1. Create a read-only (RO) AFM cache on the new storage (ro) 2a. Move old fileset and replace with SymLink to new. 2b. Convert RO AFM to Local Update (LU) AFM pointing to new parking area of old files. 2c. move user access to new location in cache. 3. Flush everything into cache and disconnect. I've read the docs including the ones on migration but it's not clear if it's safe to move the home of a cache and update the target. It looks like it should be possible and my tests say it works. An alternative plan is to use a Independent Writer (IW) AFM Cache to move the home directories which are pointed to by LDAP. Hence we can move users one at a time and only have to drain the HPC cluster at the end to disconnect the cache. I assume that migrating users over an Independent Writer is safe so long as the users don't use both sides of the cache at once (ie home and target) I'm also interested in any recipe people have on GPFS policies to preseed and flush the cache. We plan to do all the migration using AFM over GPFS we're not currently using NFS and have no plans to start. I believe using GPFS is the faster method to preform the migration. Any suggestions and experience of doing similar migration jobs would be helpful. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ------------------------------ Message: 2 Date: Wed, 19 Oct 2016 13:46:02 -0400 From: Brian Marshall To: gpfsug main discussion list Subject: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="utf-8" All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 3 Date: Wed, 19 Oct 2016 18:10:38 +0000 From: "Simon Thompson (Research Computing - IT Services)" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="us-ascii" mmdiag --network Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Brian Marshall [mimarsh2 at vt.edu] Sent: 19 October 2016 18:46 To: gpfsug main discussion list Subject: [gpfsug-discuss] subnets All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall ------------------------------ Message: 4 Date: Wed, 19 Oct 2016 20:15:52 +0200 From: "Uwe Falke" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="ISO-8859-1" Hi Brian, you might use mmfsadm saferdump tscomm to check on which route peer cluster members are reached. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Frank Hammer, Thorsten Moehring Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: Brian Marshall To: gpfsug main discussion list Date: 10/19/2016 07:46 PM Subject: [gpfsug-discuss] subnets Sent by: gpfsug-discuss-bounces at spectrumscale.org All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ------------------------------ Message: 5 Date: Wed, 19 Oct 2016 20:11:57 +0000 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Subject: [gpfsug-discuss] Will there be any more GPFS 4.2.0-x releases? Message-ID: <142FECE0-E157-42D9-BC10-4C48E78FA065 at vanderbilt.edu> Content-Type: text/plain; charset="utf-8" Hi All, We?re currently running GPFS 4.2.0-4 with an efix installed and now we need a 2nd efix. I?m not a big fan of adding efix to efix and would prefer to go to a new PTF that contains both efixes. So ? is there going to be a GPFS 4.2.0-5 (it?s been a longer than normal interval since PTF 4 came out) or do we need to go to GPFS 4.2.1-x? If the latter, any major changes to watch out for? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 49 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-1466780990050_DSTlogo.png.png Type: image/png Size: 6282 bytes Desc: OutlookEmoji-1466780990050_DSTlogo.png.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-httpwww.prweb.comreleases201606prweb13504050.htm.jpg Type: image/jpeg Size: 14887 bytes Desc: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 53 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-1466780990050_DSTlogo.png.png Type: image/png Size: 6282 bytes Desc: OutlookEmoji-1466780990050_DSTlogo.png.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-httpwww.prweb.comreleases201606prweb13504050.htm.jpg Type: image/jpeg Size: 14887 bytes Desc: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg URL: From tortay at cc.in2p3.fr Sat Oct 22 11:45:23 2016 From: tortay at cc.in2p3.fr (Loic Tortay) Date: Sat, 22 Oct 2016 12:45:23 +0200 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) In-Reply-To: References: Message-ID: <580B4343.5020700@cc.in2p3.fr> On 21/10/2016 20:46, Bill Pappas wrote: [...] > > If you are using GPFS as the conduit between the home and cache > (i.e. no NFS), I would still ask the same question, more with respect to > stability for large file lists during the initial prefetch stages. > Hello, I'm in the final stage of what the AFM documentation calls an "incremental migration", for a filesystem with 100 million files. (GPFS 4.1.1, single cluster migration, "hardware/filesystem refresh" use case) I initially tried to use the NFS transport but found it too unreliable (and, in my opinion, very poorly documented). As I was about to give up on AFM, I tried using the GPFS transport (after seeing a trivially simple example on slides by someone from ANL) and things just started to work (almost) as I expected. For the files lists, I use data produced for our monitoring system that relies on snapshots fast scans (we do daily statistics on all objects in our GPFS filesystems). Our data gathering tool encodes object names in the RFC3986 (URL encoding) format which is what I found "mmafmctl prefetch" expects for "special" filenames. I understand that the policy engine does this too which, I guess, is what the documentation means by "generate a file list using policy" (sic), yet "mmafmctl prefetch" does not seem to accept/like files produced by a simple "LIST" policy (and the documentation lacks an example). As you did, I found that trying to prefetch large lists of files does not work reliably. I remember reading on that list someone (from IBM Germany, I think) recommanding to limit the number of a files in a single prefetch to 2 millions. This appears to be the sweet spot for my needs, as I can split the files list in 2 millions parts (the largest fileset in the "home" filesystem has 26 million files) and at the same time manage the issues I mention below. To keep up with the updates on the "home" filesystem (modified files), I rely on the "gpfs_winflags" GPFS extended attribute (the GPFS_WINATTR_OFFLINE bit is on for modified files, see "mmlsattr -L /cachefs/objectname" output). By chance, this attribute is included in the files produced for our statistics. This allows us to avoid doing a prefetch of all the files "continuously", since the file scan indeed appears to use only the (single) gateway node for the fileset being prefetched. In my specific configuration/environment, there are still several issues: . There is a significant memory and "slab" leak on the gateway nodes which can easily lead to a completely unreachable gateway node. These leaks appear directly related to the number of files prefetched. Stoping GPFS on the gateway node only releases some of the memory but none of the "slab", which requires a node reboot. . There is also a need to increase the "fs.file-max" sysctl on the gateway nodes to a value larger than the default (I use 10M), to avoid the kernel running out of file descriptors, since this leads to node being unreachable too. . Sometimes, an AFM association will go to the "Unmounted" state (for no apparent reason). The only reliable way to bring it back to "Active" state I found is to : unmount the "cache" filesystem from all nodes mouting it, unmounting/remounting the "home" filesystem on the gateway nodes, then remounting the "cache" filesystem where it is needed (gateway nodes, etc.) and finally using "ls -la" in the "cache" filesystem to bring the AFM associations back into the Active state. As I'm doing an "incremental migration", the "home" fileystem is still actively used and unmounting it on all nodes (as suggested by the documentation) is not an option. . I include directories and symlinks in the file lists for the prefetch. This ensures symlinks targets are there without needing a "ls" or "stat" in the "cache" filesystem ("incremental migration"). . The only reliable way I found to have "mmafmctl prefetch" accept files lists is to use the "--home-list-file" & "--home-fs-path" options. In my experience, in my environment, using AFM for migrations requires a significant amount of work and hand-holding to get a useful result. Since this migration is actually only the first step in a extensive multiple filesystems migration/merge plan, I'm pondering wether I'll use AFM for the rest of the operations. Sorry, this mail is too long, Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2931 bytes Desc: S/MIME Cryptographic Signature URL: From makaplan at us.ibm.com Sat Oct 22 14:05:34 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Sat, 22 Oct 2016 09:05:34 -0400 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames In-Reply-To: <580B4343.5020700@cc.in2p3.fr> References: <580B4343.5020700@cc.in2p3.fr> Message-ID: RULE .... EXTERNAL LIST ... ESCAPE '%' will direct mmapplypolicy to encode pathnames following RFC3986. Use ESCAPE '%/@' or similar for a more "relaxed" encoding. See the "Information lifecycle management" chapter of the official pubs for more details. Read the section that begins ... ESCAPE '%SpecialCharacters' Specifies that path names and the SHOW('string') expressions within the associated file lists are encoded with a scheme based on RFC3986 URI percent encoding. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tortay at cc.in2p3.fr Sat Oct 22 14:15:25 2016 From: tortay at cc.in2p3.fr (Loic Tortay) Date: Sat, 22 Oct 2016 15:15:25 +0200 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames In-Reply-To: References: <580B4343.5020700@cc.in2p3.fr> Message-ID: <580B666D.3090200@cc.in2p3.fr> On 22/10/2016 15:05, Marc A Kaplan wrote: > RULE .... EXTERNAL LIST ... ESCAPE '%' > will direct mmapplypolicy to encode pathnames following RFC3986. > Use ESCAPE '%/@' or similar for a more "relaxed" encoding. > > See the "Information lifecycle management" chapter of the official pubs > for more details. > Read the section that begins ... > > ESCAPE '%SpecialCharacters' > Specifies that path names and the SHOW('string') expressions within the > associated file lists are > encoded with a scheme based on RFC3986 URI percent encoding. > Hello, I've read (and used many times) that part of the fine documentation. My issue is with the documentation of what "mmafcmtl prefetch" expects. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2931 bytes Desc: S/MIME Cryptographic Signature URL: From makaplan at us.ibm.com Sat Oct 22 20:24:22 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Sat, 22 Oct 2016 15:24:22 -0400 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames In-Reply-To: <580B666D.3090200@cc.in2p3.fr> References: <580B4343.5020700@cc.in2p3.fr> <580B666D.3090200@cc.in2p3.fr> Message-ID: Hmmm.... Peeking into the script, it looks like some of AFM uses the simple \\ and \n file list escaping convention (the mmapplypolicy default with no ESCAPE specified) ... And other parts use ESCAPE '%'. Is this inconsistency hidden or does the CLI user have to deal with it? From: Loic Tortay To: gpfsug main discussion list Date: 10/22/2016 09:15 AM Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Sent by: gpfsug-discuss-bounces at spectrumscale.org On 22/10/2016 15:05, Marc A Kaplan wrote: > RULE .... EXTERNAL LIST ... ESCAPE '%' > will direct mmapplypolicy to encode pathnames following RFC3986. > Use ESCAPE '%/@' or similar for a more "relaxed" encoding. > > See the "Information lifecycle management" chapter of the official pubs > for more details. > Read the section that begins ... > > ESCAPE '%SpecialCharacters' > Specifies that path names and the SHOW('string') expressions within the > associated file lists are > encoded with a scheme based on RFC3986 URI percent encoding. > Hello, I've read (and used many times) that part of the fine documentation. My issue is with the documentation of what "mmafcmtl prefetch" expects. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | [attachment "smime.p7s" deleted by Marc A Kaplan/Watson/IBM] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtolson at us.ibm.com Mon Oct 24 01:41:07 2016 From: jtolson at us.ibm.com (John T Olson) Date: Sun, 23 Oct 2016 17:41:07 -0700 Subject: [gpfsug-discuss] New whitepaper published In-Reply-To: References: Message-ID: A new white paper has been published which shows integration of the Varonis UNIX agent in Spectrum Scale for audit logging. Here is a link to the paper: https://www.ibm.com/developerworks/community/wikis/form/api/wiki/fa32927c-e904-49cc-a4cc-870bcc8e307c/page/f0cc9b82-a133-41b4-83fe-3f560e95b35a/attachment/0ab62645-e0ab-4377-81e7-abd11879bb75/media/Spectrum_Scale_Varonis_Audit_Logging.pdf Thanks, John John T. Olson, Ph.D., MI.C., K.EY. Master Inventor, Software Defined Storage 957/9032-1 Tucson, AZ, 85744 (520) 799-5185, tie 321-5185 (FAX: 520-799-4237) Email: jtolson at us.ibm.com "Do or do not. There is no try." - Yoda Olson's Razor: Any situation that we, as humans, can encounter in life can be modeled by either an episode of The Simpsons or Seinfeld. -------------- next part -------------- An HTML attachment was scrubbed... URL: From laurence at qsplace.co.uk Mon Oct 24 08:10:12 2016 From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow) Date: Mon, 24 Oct 2016 08:10:12 +0100 Subject: [gpfsug-discuss] Two new whitepapers published In-Reply-To: <7fifaf55s6bq10jlpkvl5he9.1477081370569@email.android.com> References: <36EC1962-F71F-4492-A175-4BA3E9D97A8D@qsplace.co.uk>, <7fifaf55s6bq10jlpkvl5he9.1477081370569@email.android.com> Message-ID: <0386E88D-DC69-42DA-ACF7-BF0B0F45CD4E@qsplace.co.uk> Hi Peter, I've always been under the impression that this is a ball park figure that changes some of the on disk data structures to help parallel access to the filesystem I try to estimate the number of clients and then add ~10% (depending on the cluster size ofc). I have tested the default 32 vs 200 on a previous cluster however didn't find a difference in performance when testing up to 50 clients concurrently with IOR random and sequential. Maybe the difference is more subtle that just throughput? What I've always found strange is that the value can be changed on the filesystem however I don't believe this change effects an already created filesystem. -- Lauz On 21 October 2016 21:35:15 BST, Peter Childs wrote: > >Reading through them and we'll worth read. > >How important is setting the correct cluster size at file system >creation time? Ie with "mmchfs -n 256" ie how much band of error can >you get away with? > >We have a cluster of 240 nodes that was setup with the default 32 >setting, it's now about to grow to ~300 nodes. Is this likly to causing >as an issue, which is difficult to fix. The manual says it can be >adjusted but this white paper suggests not. > >Fortunately were also migrating our storage to new hardware, so have a >good opportunity to get the setting right, this time around. > >Has anyone got any stats on the benefits of getting it "right" vs >getting it wrong. > > > > >Peter Childs >Research Storage >ITS Research and Teaching Support >Queen Mary, University of London > > >---- Andreas Landh?u?er wrote ---- > >On Fri, 21 Oct 2016, Laurence Horrocks-Barlow >wrote: > >Found it! > >thanks Yuri, for the two very interesting papers about the Metadata and >replication. > >We always used the rule of thumb 5% for metadata, since we are having >separate devices for metadata, and tiny fast disks aren't available >anymore, we are getting a bunch of larger fast disks. We never >experienced >a problem with the (more or less) amount of metadata ... > > Andreas > > >> Right down the bottom of the page under attachments. >> >> -- Lauz >> >> On 21 October 2016 07:43:40 BST, "Andreas Landh?u?er" > wrote: >>> >>> Hi Yuri, >>> >>> Arrg, can't find them, page has been last updated on Aug, 18 by >>> JohnTOlson, maybe its internal and not open for the public? >>> >>> Ciao >>> >>> Andreas >>> >>> On Fri, 21 Oct 2016, Yuri L Volobuev wrote: >>> >>>> >>>> >>>> Esteemed GPFS and Spectrum Scale users, >>>> >>>> For your reading enjoyment, two new whitepapers have been posted to >>> the >>>> Spectrum Scale Wiki: >>>> >>>> Spectrum Scale: Replication in GPFS >>>> Spectrum Scale: GPFS Metadata >>>> >>>> The URL for the parent page is >>>> >>> >https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20 >>>> (GPFS)/page/White%20Papers%20%26%20Media >>>> >>>> The two .pdf documents are accessible through the Attachment >section >>> at the >>>> bottom of the page. Unfortunately, dW "spam prevention engine" >does >>> a very >>>> good job preventing me from "spamming" the page to actually add >>> links. >>>> >>>> Best regards, >>>> >>>> Yuri >>>> >>> >>> -- >>> Andreas Landh?u?er +49 151 12133027 >(mobile) >>> alandhae at gmx.de >>> >>> >------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> > >-- >Andreas Landh?u?er +49 151 12133027 >(mobile) >alandhae at gmx.de > > >------------------------------------------------------------------------ > >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vpuvvada at in.ibm.com Mon Oct 24 10:44:19 2016 From: vpuvvada at in.ibm.com (Venkateswara R Puvvada) Date: Mon, 24 Oct 2016 15:14:19 +0530 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames In-Reply-To: References: <580B4343.5020700@cc.in2p3.fr><580B666D.3090200@cc.in2p3.fr> Message-ID: Loic, mmafmctl prefecth expects encoded list file, and it is not documented correctly. Issues like memory leak, file descriptor leak, and fileset going into Unmounted state were fixed in later releases (4.2.1/4.2.2). All your points are correct with respect to AFM migration. There is manual intervention required. Also prefetch does not give list of files which were failed during data read. Users need to run policy to find all uncached files today. ~Venkat (vpuvvada at in.ibm.com) From: "Marc A Kaplan" To: gpfsug main discussion list Date: 10/23/2016 12:54 AM Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Sent by: gpfsug-discuss-bounces at spectrumscale.org Hmmm.... Peeking into the script, it looks like some of AFM uses the simple \\ and \n file list escaping convention (the mmapplypolicy default with no ESCAPE specified) ... And other parts use ESCAPE '%'. Is this inconsistency hidden or does the CLI user have to deal with it? From: Loic Tortay To: gpfsug main discussion list Date: 10/22/2016 09:15 AM Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Sent by: gpfsug-discuss-bounces at spectrumscale.org On 22/10/2016 15:05, Marc A Kaplan wrote: > RULE .... EXTERNAL LIST ... ESCAPE '%' > will direct mmapplypolicy to encode pathnames following RFC3986. > Use ESCAPE '%/@' or similar for a more "relaxed" encoding. > > See the "Information lifecycle management" chapter of the official pubs > for more details. > Read the section that begins ... > > ESCAPE '%SpecialCharacters' > Specifies that path names and the SHOW('string') expressions within the > associated file lists are > encoded with a scheme based on RFC3986 URI percent encoding. > Hello, I've read (and used many times) that part of the fine documentation. My issue is with the documentation of what "mmafcmtl prefetch" expects. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | [attachment "smime.p7s" deleted by Marc A Kaplan/Watson/IBM] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From erich at uw.edu Mon Oct 24 17:16:56 2016 From: erich at uw.edu (Eric Horst) Date: Mon, 24 Oct 2016 09:16:56 -0700 Subject: [gpfsug-discuss] Using AFM to migrate within the same cluster Message-ID: The recent conversation about AFM has been interesting. I've read the documentation several times and this is my question. Can AFM be used to migrate between two filesystems in the same cluster? There are examples of moving between clusters with NFS or native protocol but I've got a simple situation of needing to transparently move 100M files between two existing filesystems. Thanks, -Eric -- Eric Horst University of Washington From tortay at cc.in2p3.fr Mon Oct 24 17:50:51 2016 From: tortay at cc.in2p3.fr (Loic Tortay) Date: Mon, 24 Oct 2016 18:50:51 +0200 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames In-Reply-To: References: <580B4343.5020700@cc.in2p3.fr> <580B666D.3090200@cc.in2p3.fr> Message-ID: On 10/24/2016 11:44 AM, Venkateswara R Puvvada wrote: > > mmafmctl prefecth expects encoded list file, and it is not documented > correctly. Issues like memory leak, file descriptor leak, and fileset > going into Unmounted state were fixed in later releases (4.2.1/4.2.2). All > your points are correct with respect to AFM migration. There is manual > intervention required. Also prefetch does not give list of files which > were failed during data read. Users need to run policy to find all > uncached files today. > Hello, For the record, I have completed today my AFM migration of a filesystem with 100 million files. Users are now accessing the new filesystem. After disabling user access and a last "prefetch", the AFM filesets were converted to independent filesets. Less than 600 files were then found to be different between the "home" and the "cache" filesystems with a metadata comparison (I just copied the files from the old filesystem to the new one). I have compared the MD5 of a few thousand randomly selected files and found no differences between the "home" and the "cache" filesystems. I expect the users to let us know if they find something different (they have been instructed to do so). We'll keep the "home" filesystem around for some time, just in case there is a problem. Maybe something else that should be mentionned in the documentation is what to do with the ".ptrash" directories after the AFM filesets have been converted. I removed them since they contained files that had clearly been deleted by the users. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | From sfadden at us.ibm.com Mon Oct 24 17:57:33 2016 From: sfadden at us.ibm.com (Scott Fadden) Date: Mon, 24 Oct 2016 09:57:33 -0700 Subject: [gpfsug-discuss] Using AFM to migrate within the same cluster In-Reply-To: References: Message-ID: Yes you can use AFM to move data within a cluster. If you are using the NSD protocol the target needs to be a separate file system, if you are using NFS it needs to be an NFS export. Scott Fadden Spectrum Scale - Technical Marketing Phone: (503) 880-5833 sfadden at us.ibm.com http://www.ibm.com/systems/storage/spectrum/scale From: Eric Horst To: gpfsug main discussion list Date: 10/24/2016 09:17 AM Subject: [gpfsug-discuss] Using AFM to migrate within the same cluster Sent by: gpfsug-discuss-bounces at spectrumscale.org The recent conversation about AFM has been interesting. I've read the documentation several times and this is my question. Can AFM be used to migrate between two filesystems in the same cluster? There are examples of moving between clusters with NFS or native protocol but I've got a simple situation of needing to transparently move 100M files between two existing filesystems. Thanks, -Eric -- Eric Horst University of Washington _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From YARD at il.ibm.com Mon Oct 24 18:05:00 2016 From: YARD at il.ibm.com (Yaron Daniel) Date: Mon, 24 Oct 2016 20:05:00 +0300 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames In-Reply-To: References: <580B4343.5020700@cc.in2p3.fr><580B666D.3090200@cc.in2p3.fr> Message-ID: Hi Maybe worth also to check if there are any orphan files in the NEW fs ? Regards Yaron Daniel 94 Em Ha'Moshavot Rd Server, Storage and Data Services - Team Leader Petach Tiqva, 49527 Global Technology Services Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com IBM Israel From: Loic Tortay To: gpfsug main discussion list Date: 10/24/2016 07:50 PM Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Sent by: gpfsug-discuss-bounces at spectrumscale.org On 10/24/2016 11:44 AM, Venkateswara R Puvvada wrote: > > mmafmctl prefecth expects encoded list file, and it is not documented > correctly. Issues like memory leak, file descriptor leak, and fileset > going into Unmounted state were fixed in later releases (4.2.1/4.2.2). All > your points are correct with respect to AFM migration. There is manual > intervention required. Also prefetch does not give list of files which > were failed during data read. Users need to run policy to find all > uncached files today. > Hello, For the record, I have completed today my AFM migration of a filesystem with 100 million files. Users are now accessing the new filesystem. After disabling user access and a last "prefetch", the AFM filesets were converted to independent filesets. Less than 600 files were then found to be different between the "home" and the "cache" filesystems with a metadata comparison (I just copied the files from the old filesystem to the new one). I have compared the MD5 of a few thousand randomly selected files and found no differences between the "home" and the "cache" filesystems. I expect the users to let us know if they find something different (they have been instructed to do so). We'll keep the "home" filesystem around for some time, just in case there is a problem. Maybe something else that should be mentionned in the documentation is what to do with the ".ptrash" directories after the AFM filesets have been converted. I removed them since they contained files that had clearly been deleted by the users. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1851 bytes Desc: not available URL: From bpappas at dstonline.com Mon Oct 24 19:03:07 2016 From: bpappas at dstonline.com (Bill Pappas) Date: Mon, 24 Oct 2016 18:03:07 +0000 Subject: [gpfsug-discuss] Using AFM to migrate files. (Bill Pappas to Loric Totay In-Reply-To: References: Message-ID: >>For the record, I have completed today my AFM migration of a filesystem with 100 million files. Users are now accessing the new filesystem. Loric-> Hi. I was wondering, what version of GPFS where you running on the home and cache clusters? I take it you broke up the prefetch list into smaller (for example <2 million) file lists? If not, how? How much capacity did you migfrate over and how long did this process take? Thanks. Bill Pappas 901-619-0585 bpappas at dstonline.com [1466780990050_DSTlogo.png] [http://www.prweb.com/releases/2016/06/prweb13504050.htm] http://www.prweb.com/releases/2016/06/prweb13504050.htm ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org Sent: Monday, October 24, 2016 12:05 PM To: gpfsug-discuss at spectrumscale.org Subject: gpfsug-discuss Digest, Vol 57, Issue 60 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Using AFM to migrate within the same cluster (Eric Horst) 2. Re: Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames (Loic Tortay) 3. Re: Using AFM to migrate within the same cluster (Scott Fadden) 4. Re: Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames (Yaron Daniel) ---------------------------------------------------------------------- Message: 1 Date: Mon, 24 Oct 2016 09:16:56 -0700 From: Eric Horst To: gpfsug main discussion list Subject: [gpfsug-discuss] Using AFM to migrate within the same cluster Message-ID: Content-Type: text/plain; charset=UTF-8 The recent conversation about AFM has been interesting. I've read the documentation several times and this is my question. Can AFM be used to migrate between two filesystems in the same cluster? There are examples of moving between clusters with NFS or native protocol but I've got a simple situation of needing to transparently move 100M files between two existing filesystems. Thanks, -Eric -- Eric Horst University of Washington ------------------------------ Message: 2 Date: Mon, 24 Oct 2016 18:50:51 +0200 From: Loic Tortay To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Message-ID: Content-Type: text/plain; charset=utf-8 On 10/24/2016 11:44 AM, Venkateswara R Puvvada wrote: > > mmafmctl prefecth expects encoded list file, and it is not documented > correctly. Issues like memory leak, file descriptor leak, and fileset > going into Unmounted state were fixed in later releases (4.2.1/4.2.2). All > your points are correct with respect to AFM migration. There is manual > intervention required. Also prefetch does not give list of files which > were failed during data read. Users need to run policy to find all > uncached files today. > Hello, For the record, I have completed today my AFM migration of a filesystem with 100 million files. Users are now accessing the new filesystem. After disabling user access and a last "prefetch", the AFM filesets were converted to independent filesets. Less than 600 files were then found to be different between the "home" and the "cache" filesystems with a metadata comparison (I just copied the files from the old filesystem to the new one). I have compared the MD5 of a few thousand randomly selected files and found no differences between the "home" and the "cache" filesystems. I expect the users to let us know if they find something different (they have been instructed to do so). We'll keep the "home" filesystem around for some time, just in case there is a problem. Maybe something else that should be mentionned in the documentation is what to do with the ".ptrash" directories after the AFM filesets have been converted. I removed them since they contained files that had clearly been deleted by the users. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | ------------------------------ Message: 3 Date: Mon, 24 Oct 2016 09:57:33 -0700 From: "Scott Fadden" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Using AFM to migrate within the same cluster Message-ID: Content-Type: text/plain; charset="us-ascii" Yes you can use AFM to move data within a cluster. If you are using the NSD protocol the target needs to be a separate file system, if you are using NFS it needs to be an NFS export. Scott Fadden Spectrum Scale - Technical Marketing Phone: (503) 880-5833 sfadden at us.ibm.com http://www.ibm.com/systems/storage/spectrum/scale From: Eric Horst To: gpfsug main discussion list Date: 10/24/2016 09:17 AM Subject: [gpfsug-discuss] Using AFM to migrate within the same cluster Sent by: gpfsug-discuss-bounces at spectrumscale.org The recent conversation about AFM has been interesting. I've read the documentation several times and this is my question. Can AFM be used to migrate between two filesystems in the same cluster? There are examples of moving between clusters with NFS or native protocol but I've got a simple situation of needing to transparently move 100M files between two existing filesystems. Thanks, -Eric -- Eric Horst University of Washington _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 4 Date: Mon, 24 Oct 2016 20:05:00 +0300 From: "Yaron Daniel" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Message-ID: Content-Type: text/plain; charset="iso-8859-1" Hi Maybe worth also to check if there are any orphan files in the NEW fs ? Regards Yaron Daniel 94 Em Ha'Moshavot Rd Server, Storage and Data Services - Team Leader Petach Tiqva, 49527 Global Technology Services Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com IBM Israel From: Loic Tortay To: gpfsug main discussion list Date: 10/24/2016 07:50 PM Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Sent by: gpfsug-discuss-bounces at spectrumscale.org On 10/24/2016 11:44 AM, Venkateswara R Puvvada wrote: > > mmafmctl prefecth expects encoded list file, and it is not documented > correctly. Issues like memory leak, file descriptor leak, and fileset > going into Unmounted state were fixed in later releases (4.2.1/4.2.2). All > your points are correct with respect to AFM migration. There is manual > intervention required. Also prefetch does not give list of files which > were failed during data read. Users need to run policy to find all > uncached files today. > Hello, For the record, I have completed today my AFM migration of a filesystem with 100 million files. Users are now accessing the new filesystem. After disabling user access and a last "prefetch", the AFM filesets were converted to independent filesets. Less than 600 files were then found to be different between the "home" and the "cache" filesystems with a metadata comparison (I just copied the files from the old filesystem to the new one). I have compared the MD5 of a few thousand randomly selected files and found no differences between the "home" and the "cache" filesystems. I expect the users to let us know if they find something different (they have been instructed to do so). We'll keep the "home" filesystem around for some time, just in case there is a problem. Maybe something else that should be mentionned in the documentation is what to do with the ".ptrash" directories after the AFM filesets have been converted. I removed them since they contained files that had clearly been deleted by the users. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1851 bytes Desc: not available URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 60 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-1466780990050_DSTlogo.png.png Type: image/png Size: 6282 bytes Desc: OutlookEmoji-1466780990050_DSTlogo.png.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg Type: image/jpeg Size: 14887 bytes Desc: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg URL: From vpuvvada at in.ibm.com Tue Oct 25 05:53:10 2016 From: vpuvvada at in.ibm.com (Venkateswara R Puvvada) Date: Tue, 25 Oct 2016 10:23:10 +0530 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames In-Reply-To: References: <580B4343.5020700@cc.in2p3.fr><580B666D.3090200@cc.in2p3.fr> Message-ID: Loic, It is good to hear that migration is completed. Did you find any prefetch errors in mmfs.log for the files which are different between cache and home ? Not all errors are logged, there is a plan to generate the list of read failed file names after prefetch completion. Informaion about files moving to .ptrash is documented @ http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.1/com.ibm.spectrum.scale.v4r21.doc/bl1ins_internaldirectoriesAFM.htm ~Venkat (vpuvvada at in.ibm.com) From: Loic Tortay To: gpfsug main discussion list Date: 10/24/2016 10:20 PM Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Sent by: gpfsug-discuss-bounces at spectrumscale.org On 10/24/2016 11:44 AM, Venkateswara R Puvvada wrote: > > mmafmctl prefecth expects encoded list file, and it is not documented > correctly. Issues like memory leak, file descriptor leak, and fileset > going into Unmounted state were fixed in later releases (4.2.1/4.2.2). All > your points are correct with respect to AFM migration. There is manual > intervention required. Also prefetch does not give list of files which > were failed during data read. Users need to run policy to find all > uncached files today. > Hello, For the record, I have completed today my AFM migration of a filesystem with 100 million files. Users are now accessing the new filesystem. After disabling user access and a last "prefetch", the AFM filesets were converted to independent filesets. Less than 600 files were then found to be different between the "home" and the "cache" filesystems with a metadata comparison (I just copied the files from the old filesystem to the new one). I have compared the MD5 of a few thousand randomly selected files and found no differences between the "home" and the "cache" filesystems. I expect the users to let us know if they find something different (they have been instructed to do so). We'll keep the "home" filesystem around for some time, just in case there is a problem. Maybe something else that should be mentionned in the documentation is what to do with the ".ptrash" directories after the AFM filesets have been converted. I removed them since they contained files that had clearly been deleted by the users. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From radhika.p at in.ibm.com Tue Oct 25 08:24:56 2016 From: radhika.p at in.ibm.com (Radhika A Parameswaran) Date: Tue, 25 Oct 2016 12:54:56 +0530 Subject: [gpfsug-discuss] Using AFM to migrate files In-Reply-To: References: Message-ID: Bill, The issues/limitation with preparing a listfile with >2M files in the file list has been fixed in 4.2.1. It has also been internally checked with 45M file list. Peter, The 4.2.1 man pages and prefetch section has added some examples of policy and listfile options for prefetch. Please take a look at them and let us know if those help. Loic, We will add about removing .ptrash to the migration usecase documentation. Can you please share some details about your dataset and performance for migrating the 100M (time for listfile processing and actual transfer) ? Thanks and Regards Radhika From: gpfsug-discuss-request at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Date: 10/24/2016 11:33 PM Subject: gpfsug-discuss Digest, Vol 57, Issue 61 Sent by: gpfsug-discuss-bounces at spectrumscale.org Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Using AFM to migrate files. (Bill Pappas to Loric Totay (Bill Pappas) ---------------------------------------------------------------------- Message: 1 Date: Mon, 24 Oct 2016 18:03:07 +0000 From: Bill Pappas To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Bill Pappas to Loric Totay Message-ID: Content-Type: text/plain; charset="iso-8859-1" >>For the record, I have completed today my AFM migration of a filesystem with 100 million files. Users are now accessing the new filesystem. Loric-> Hi. I was wondering, what version of GPFS where you running on the home and cache clusters? I take it you broke up the prefetch list into smaller (for example <2 million) file lists? If not, how? How much capacity did you migfrate over and how long did this process take? Thanks. Bill Pappas 901-619-0585 bpappas at dstonline.com [1466780990050_DSTlogo.png] [http://www.prweb.com/releases/2016/06/prweb13504050.htm] http://www.prweb.com/releases/2016/06/prweb13504050.htm ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org Sent: Monday, October 24, 2016 12:05 PM To: gpfsug-discuss at spectrumscale.org Subject: gpfsug-discuss Digest, Vol 57, Issue 60 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Using AFM to migrate within the same cluster (Eric Horst) 2. Re: Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames (Loic Tortay) 3. Re: Using AFM to migrate within the same cluster (Scott Fadden) 4. Re: Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames (Yaron Daniel) ---------------------------------------------------------------------- Message: 1 Date: Mon, 24 Oct 2016 09:16:56 -0700 From: Eric Horst To: gpfsug main discussion list Subject: [gpfsug-discuss] Using AFM to migrate within the same cluster Message-ID: Content-Type: text/plain; charset=UTF-8 The recent conversation about AFM has been interesting. I've read the documentation several times and this is my question. Can AFM be used to migrate between two filesystems in the same cluster? There are examples of moving between clusters with NFS or native protocol but I've got a simple situation of needing to transparently move 100M files between two existing filesystems. Thanks, -Eric -- Eric Horst University of Washington ------------------------------ Message: 2 Date: Mon, 24 Oct 2016 18:50:51 +0200 From: Loic Tortay To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Message-ID: Content-Type: text/plain; charset=utf-8 On 10/24/2016 11:44 AM, Venkateswara R Puvvada wrote: > > mmafmctl prefecth expects encoded list file, and it is not documented > correctly. Issues like memory leak, file descriptor leak, and fileset > going into Unmounted state were fixed in later releases (4.2.1/4.2.2). All > your points are correct with respect to AFM migration. There is manual > intervention required. Also prefetch does not give list of files which > were failed during data read. Users need to run policy to find all > uncached files today. > Hello, For the record, I have completed today my AFM migration of a filesystem with 100 million files. Users are now accessing the new filesystem. After disabling user access and a last "prefetch", the AFM filesets were converted to independent filesets. Less than 600 files were then found to be different between the "home" and the "cache" filesystems with a metadata comparison (I just copied the files from the old filesystem to the new one). I have compared the MD5 of a few thousand randomly selected files and found no differences between the "home" and the "cache" filesystems. I expect the users to let us know if they find something different (they have been instructed to do so). We'll keep the "home" filesystem around for some time, just in case there is a problem. Maybe something else that should be mentionned in the documentation is what to do with the ".ptrash" directories after the AFM filesets have been converted. I removed them since they contained files that had clearly been deleted by the users. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | ------------------------------ Message: 3 Date: Mon, 24 Oct 2016 09:57:33 -0700 From: "Scott Fadden" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Using AFM to migrate within the same cluster Message-ID: Content-Type: text/plain; charset="us-ascii" Yes you can use AFM to move data within a cluster. If you are using the NSD protocol the target needs to be a separate file system, if you are using NFS it needs to be an NFS export. Scott Fadden Spectrum Scale - Technical Marketing Phone: (503) 880-5833 sfadden at us.ibm.com http://www.ibm.com/systems/storage/spectrum/scale From: Eric Horst To: gpfsug main discussion list Date: 10/24/2016 09:17 AM Subject: [gpfsug-discuss] Using AFM to migrate within the same cluster Sent by: gpfsug-discuss-bounces at spectrumscale.org The recent conversation about AFM has been interesting. I've read the documentation several times and this is my question. Can AFM be used to migrate between two filesystems in the same cluster? There are examples of moving between clusters with NFS or native protocol but I've got a simple situation of needing to transparently move 100M files between two existing filesystems. Thanks, -Eric -- Eric Horst University of Washington _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161024/866c55cf/attachment-0001.html > ------------------------------ Message: 4 Date: Mon, 24 Oct 2016 20:05:00 +0300 From: "Yaron Daniel" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Message-ID: Content-Type: text/plain; charset="iso-8859-1" Hi Maybe worth also to check if there are any orphan files in the NEW fs ? Regards Yaron Daniel 94 Em Ha'Moshavot Rd Server, Storage and Data Services - Team Leader Petach Tiqva, 49527 Global Technology Services Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com IBM Israel From: Loic Tortay To: gpfsug main discussion list Date: 10/24/2016 07:50 PM Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Sent by: gpfsug-discuss-bounces at spectrumscale.org On 10/24/2016 11:44 AM, Venkateswara R Puvvada wrote: > > mmafmctl prefecth expects encoded list file, and it is not documented > correctly. Issues like memory leak, file descriptor leak, and fileset > going into Unmounted state were fixed in later releases (4.2.1/4.2.2). All > your points are correct with respect to AFM migration. There is manual > intervention required. Also prefetch does not give list of files which > were failed during data read. Users need to run policy to find all > uncached files today. > Hello, For the record, I have completed today my AFM migration of a filesystem with 100 million files. Users are now accessing the new filesystem. After disabling user access and a last "prefetch", the AFM filesets were converted to independent filesets. Less than 600 files were then found to be different between the "home" and the "cache" filesystems with a metadata comparison (I just copied the files from the old filesystem to the new one). I have compared the MD5 of a few thousand randomly selected files and found no differences between the "home" and the "cache" filesystems. I expect the users to let us know if they find something different (they have been instructed to do so). We'll keep the "home" filesystem around for some time, just in case there is a problem. Maybe something else that should be mentionned in the documentation is what to do with the ".ptrash" directories after the AFM filesets have been converted. I removed them since they contained files that had clearly been deleted by the users. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161024/449a0cf1/attachment.html > -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1851 bytes Desc: not available URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161024/449a0cf1/attachment.gif > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 60 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161024/fff3f370/attachment.html > -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-1466780990050_DSTlogo.png.png Type: image/png Size: 6282 bytes Desc: OutlookEmoji-1466780990050_DSTlogo.png.png URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161024/fff3f370/attachment.png > -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg Type: image/jpeg Size: 14887 bytes Desc: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161024/fff3f370/attachment.jpg > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 61 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: From tortay at cc.in2p3.fr Tue Oct 25 13:01:01 2016 From: tortay at cc.in2p3.fr (Loic Tortay) Date: Tue, 25 Oct 2016 14:01:01 +0200 Subject: [gpfsug-discuss] Using AFM to migrate files. (Bill Pappas to Loric Totay In-Reply-To: References: Message-ID: <05eb9917-0b3a-1bde-fc04-9acaed06fe29@cc.in2p3.fr> On 10/24/2016 08:03 PM, Bill Pappas wrote: > > Loric-> Hi. I was wondering, what version of GPFS where you running > on the home and cache clusters? I take it you broke up the prefetch list > into smaller (for example <2 million) file lists? If not, how? How much > capacity did you migfrate over and how long did this process take? Thanks. > Hello, We use GPFS 4.1.1-8. The migration was within a single cluster. The files lists were split in 2M files parts. The amount of data migrated is rather small (~70 TB), but there are ~100M files, including many extremely small files: the median (not average) file size is 2705 bytes. The migration itself took about 3 weeks: I initially did the first "mmafmctl prefetches" one fileset at a time (& in slices of 2M files), to manage the various issues mentionned previously and minimize the impact on the "home" filesystem since it was still being used by the end users. Following "prefetches" were done a few filesets at a time (filesets chosen to spread the load on the gateway nodes). We chose to do an "incremental migration" instead of a "progressive migration", since we were unable to get a good understanding of the performance impact of the AFM migration during our tests on a toy cluster running on VMs. Lo?c (Loic w/ a diaresis on the i, not Loric :-) -- | Lo?c Tortay - IN2P3 Computing Centre | From matt.thorpe at bodleian.ox.ac.uk Tue Oct 25 13:05:36 2016 From: matt.thorpe at bodleian.ox.ac.uk (Matt Thorpe) Date: Tue, 25 Oct 2016 12:05:36 +0000 Subject: [gpfsug-discuss] Forcing which node gets expelled? Message-ID: <63906C4AF5FFD0429565A743F6AD5BEDDAF29F@MBX04.ad.oak.ox.ac.uk> Hi, We are in the process of diagnosing a networking issue that is causing 2 of our 6 node GPFS cluster to expel each other (it appears they experience a temporary network connection outage and lose contact with each other). At present it's not consistent which gets expelled by the cluster manager, and I wondered if there was any way to always force a specific node to be expelled in this situation? Thanks and best regards, Matt -------- Matt Thorpe | BDLSS Systems Administrator Bodleian Libraries Osney One Building, Osney Mead, Oxford, OX2 0EW matt.thorpe at bodleian.ox.ac.uk | 01865 (2)80027 From tortay at cc.in2p3.fr Tue Oct 25 13:17:56 2016 From: tortay at cc.in2p3.fr (Loic Tortay) Date: Tue, 25 Oct 2016 14:17:56 +0200 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames In-Reply-To: References: <580B4343.5020700@cc.in2p3.fr> <580B666D.3090200@cc.in2p3.fr> Message-ID: On 10/24/2016 07:05 PM, Yaron Daniel wrote: > > Maybe worth also to check if there are any orphan files in the NEW fs ? > Hello, I ran a online "dry run" (-n) mmfsck which found a few lost blocks, but I'm not sure orphan files are detected with an online mmfsck. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | From Robert.Oesterlin at nuance.com Tue Oct 25 13:22:30 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Tue, 25 Oct 2016 12:22:30 +0000 Subject: [gpfsug-discuss] Forcing which node gets expelled? In-Reply-To: <63906C4AF5FFD0429565A743F6AD5BEDDAF29F@MBX04.ad.oak.ox.ac.uk> References: <63906C4AF5FFD0429565A743F6AD5BEDDAF29F@MBX04.ad.oak.ox.ac.uk> Message-ID: <4403E9DB-88B4-4281-91B5-E171D1284899@nuance.com> If you look at /usr/lpp/mmfs/samples/expelnode.sample you can use this as a base and install this in /var/mmfs/etc on the cluster manager. You can control which of the two nodes get expelled. We use it here to send an alert when a node is expelled. There is also "mmexpelnode" which you can force a particular node to be expelled. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Matt Thorpe Reply-To: gpfsug main discussion list Date: Tuesday, October 25, 2016 at 7:05 AM To: gpfsug main discussion list Subject: [EXTERNAL] [gpfsug-discuss] Forcing which node gets expelled? Hi, We are in the process of diagnosing a networking issue that is causing 2 of our 6 node GPFS cluster to expel each other (it appears they experience a temporary network connection outage and lose contact with each other). At present it's not consistent which gets expelled by the cluster manager, and I wondered if there was any way to always force a specific node to be expelled in this situation? Thanks and best regards, Matt -------- Matt Thorpe | BDLSS Systems Administrator Bodleian Libraries Osney One Building, Osney Mead, Oxford, OX2 0EW matt.thorpe at bodleian.ox.ac.uk | 01865 (2)80027 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Q_f6z64tvENxA9ac7rqWFj2jNd5IrpWEXcynJzHFjz4&s=jqso6xVB-V_zgLba-xjlWwiw3fNRan9NspsVq4PY4nA&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From tortay at cc.in2p3.fr Tue Oct 25 13:31:59 2016 From: tortay at cc.in2p3.fr (Loic Tortay) Date: Tue, 25 Oct 2016 14:31:59 +0200 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames In-Reply-To: References: <580B4343.5020700@cc.in2p3.fr> <580B666D.3090200@cc.in2p3.fr> Message-ID: On 25/10/2016 06:53, Venkateswara R Puvvada wrote: > Loic, > > It is good to hear that migration is completed. Did you find any prefetch > errors in mmfs.log for the files which are different between cache and > home ? Not all errors are logged, there is a plan to generate the list of > read failed file names after prefetch completion. > Hello, Indeed, I had missed those messages: $GATEWAY1: Mon Oct 24 10:44:05.725 2016: [E] AFM: Read file system $FS fileset juno file IDs [469826991.470473596.-1.-1,N] name CMakeFiles local error 21 $GATEWAY1: Mon Oct 24 10:50:05.655 2016: [E] AFM: Read file system $FS fileset mnm file IDs [637555582.-1.-1.-1,N] name Mn_G4_SansTube local error 21 >From what I can see, for the previous runs of "mmafmctl prefetch", there are much less error messages logged than the values of the "asyncReadFailed" counters displayed by "mmafmctl prefetch -j $FILESET -Y". Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | From UWEFALKE at de.ibm.com Tue Oct 25 13:32:11 2016 From: UWEFALKE at de.ibm.com (Uwe Falke) Date: Tue, 25 Oct 2016 14:32:11 +0200 Subject: [gpfsug-discuss] Forcing which node gets expelled? In-Reply-To: <63906C4AF5FFD0429565A743F6AD5BEDDAF29F@MBX04.ad.oak.ox.ac.uk> References: <63906C4AF5FFD0429565A743F6AD5BEDDAF29F@MBX04.ad.oak.ox.ac.uk> Message-ID: Usually, the cluster mgr, receiving a complaint from a node about another node being gone, checks its own connection to that other node. If that is positive it expells the requester, if not it follows the request and expells the other node. AFAIK, there are some more subtle algorithms in place if managers or quorum nodes are affected. Maybe that can be used to protect certain nodes from getting expelled by assigning some role in the cluster to them. I do however not know these exactly. That means: it is not easily controllable which one gets expelled. It is better to concentrate on fixing your connectivity issues, as GPFS will not feel comfortable in such a unreliable environment anyway. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Frank Hammer, Thorsten Moehring Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: Matt Thorpe To: gpfsug main discussion list Date: 10/25/2016 02:05 PM Subject: [gpfsug-discuss] Forcing which node gets expelled? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi, We are in the process of diagnosing a networking issue that is causing 2 of our 6 node GPFS cluster to expel each other (it appears they experience a temporary network connection outage and lose contact with each other). At present it's not consistent which gets expelled by the cluster manager, and I wondered if there was any way to always force a specific node to be expelled in this situation? Thanks and best regards, Matt -------- Matt Thorpe | BDLSS Systems Administrator Bodleian Libraries Osney One Building, Osney Mead, Oxford, OX2 0EW matt.thorpe at bodleian.ox.ac.uk | 01865 (2)80027 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From tortay at cc.in2p3.fr Tue Oct 25 14:01:20 2016 From: tortay at cc.in2p3.fr (Loic Tortay) Date: Tue, 25 Oct 2016 15:01:20 +0200 Subject: [gpfsug-discuss] Using AFM to migrate files In-Reply-To: References: Message-ID: On 10/25/2016 09:24 AM, Radhika A Parameswaran wrote: > > Loic, > We will add about removing .ptrash to the migration usecase documentation. > Can you please share some details about your dataset and performance for > migrating the 100M (time for listfile processing and actual transfer) ? > Hello, According to my logs, the files lists processing by "mmafmctl prefetch" with 2M files took between ~30 minutes and ~70 minutes. >From what I can see, the longer processing times were for filesets with many directories (less files per directory). For the actual data transfers, it varied widely from a few minutes (small files stored in the inode on a Flash-based storage unit) to a few hours (1 or 2 TB of files not large enough to trigger the parallel migration, while many batch jobs were accessing the "home" filesystem). Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | From knop at us.ibm.com Tue Oct 25 14:02:39 2016 From: knop at us.ibm.com (Felipe Knop) Date: Tue, 25 Oct 2016 09:02:39 -0400 Subject: [gpfsug-discuss] Forcing which node gets expelled? In-Reply-To: References: <63906C4AF5FFD0429565A743F6AD5BEDDAF29F@MBX04.ad.oak.ox.ac.uk> Message-ID: All, As Bob Oesterlin indicated, it is possible to define an expel script (see /usr/lpp/mmfs/samples/expelnode.sample) to control which of the two nodes to get expelled. The script can also be used to issue alerts, etc. The current policy used (before the script is invoked) when deciding which node to expel is the following: 1. quorum nodes over non-quorum nodes 2. local nodes over remote nodes 3. manager-capable nodes over non-manager-capable nodes 4. nodes managing more FSs over nodes managing fewer FSs 5. NSD server over non-NSD server Otherwise, expel whoever joined the cluster more recently. The statement below from Dr. Uwe Falke is also correct: addressing the network connectivity is the better long-term approach, but the callback script could be used to control which node to expel. Felipe ---- Felipe Knop knop at us.ibm.com GPFS Development and Security IBM Systems IBM Building 008 2455 South Rd, Poughkeepsie, NY 12601 (845) 433-9314 T/L 293-9314 From: "Uwe Falke" To: gpfsug main discussion list Date: 10/25/2016 08:32 AM Subject: Re: [gpfsug-discuss] Forcing which node gets expelled? Sent by: gpfsug-discuss-bounces at spectrumscale.org Usually, the cluster mgr, receiving a complaint from a node about another node being gone, checks its own connection to that other node. If that is positive it expells the requester, if not it follows the request and expells the other node. AFAIK, there are some more subtle algorithms in place if managers or quorum nodes are affected. Maybe that can be used to protect certain nodes from getting expelled by assigning some role in the cluster to them. I do however not know these exactly. That means: it is not easily controllable which one gets expelled. It is better to concentrate on fixing your connectivity issues, as GPFS will not feel comfortable in such a unreliable environment anyway. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Frank Hammer, Thorsten Moehring Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: Matt Thorpe To: gpfsug main discussion list Date: 10/25/2016 02:05 PM Subject: [gpfsug-discuss] Forcing which node gets expelled? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi, We are in the process of diagnosing a networking issue that is causing 2 of our 6 node GPFS cluster to expel each other (it appears they experience a temporary network connection outage and lose contact with each other). At present it's not consistent which gets expelled by the cluster manager, and I wondered if there was any way to always force a specific node to be expelled in this situation? Thanks and best regards, Matt -------- Matt Thorpe | BDLSS Systems Administrator Bodleian Libraries Osney One Building, Osney Mead, Oxford, OX2 0EW matt.thorpe at bodleian.ox.ac.uk | 01865 (2)80027 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From matt.thorpe at bodleian.ox.ac.uk Tue Oct 25 14:06:12 2016 From: matt.thorpe at bodleian.ox.ac.uk (Matt Thorpe) Date: Tue, 25 Oct 2016 13:06:12 +0000 Subject: [gpfsug-discuss] Forcing which node gets expelled? In-Reply-To: <4403E9DB-88B4-4281-91B5-E171D1284899@nuance.com> References: <63906C4AF5FFD0429565A743F6AD5BEDDAF29F@MBX04.ad.oak.ox.ac.uk> <4403E9DB-88B4-4281-91B5-E171D1284899@nuance.com> Message-ID: <63906C4AF5FFD0429565A743F6AD5BEDDAF314@MBX04.ad.oak.ox.ac.uk> Hi Bob, That is exactly what I was after, thanks very much! Should buy us a little time so we can resolve our networking issue. Thanks again Matt. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Oesterlin, Robert Sent: 25 October 2016 13:23 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Forcing which node gets expelled? If you look at /usr/lpp/mmfs/samples/expelnode.sample you can use this as a base and install this in /var/mmfs/etc on the cluster manager. You can control which of the two nodes get expelled. We use it here to send an alert when a node is expelled. There is also "mmexpelnode" which you can force a particular node to be expelled. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: > on behalf of Matt Thorpe > Reply-To: gpfsug main discussion list > Date: Tuesday, October 25, 2016 at 7:05 AM To: gpfsug main discussion list > Subject: [EXTERNAL] [gpfsug-discuss] Forcing which node gets expelled? Hi, We are in the process of diagnosing a networking issue that is causing 2 of our 6 node GPFS cluster to expel each other (it appears they experience a temporary network connection outage and lose contact with each other). At present it's not consistent which gets expelled by the cluster manager, and I wondered if there was any way to always force a specific node to be expelled in this situation? Thanks and best regards, Matt -------- Matt Thorpe | BDLSS Systems Administrator Bodleian Libraries Osney One Building, Osney Mead, Oxford, OX2 0EW matt.thorpe at bodleian.ox.ac.uk | 01865 (2)80027 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Q_f6z64tvENxA9ac7rqWFj2jNd5IrpWEXcynJzHFjz4&s=jqso6xVB-V_zgLba-xjlWwiw3fNRan9NspsVq4PY4nA&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Tue Oct 25 16:33:16 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Tue, 25 Oct 2016 15:33:16 +0000 Subject: [gpfsug-discuss] October meet the devs report Message-ID: The October meet the devs workshop on cloud was last week. Thanks to Dean Hildebrand and John Lewars for flying in from the US to support us, Ulf Troppens and Dan Kidger also from IBM. And finally thanks to OCF for buying the pizza (one day we'll manage to get the pizza co to deliver on time!). The event report is now up on the group website at: http://www.spectrumscale.org/meet-the-devs-cloud-workshop-birmingham-uk/ Simon From Mark.Bush at siriuscom.com Tue Oct 25 19:46:26 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Tue, 25 Oct 2016 18:46:26 +0000 Subject: [gpfsug-discuss] Virtualized Spectrum Scale Message-ID: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> Anyone running SpectrumScale on Virtual Machines (intel)? I?m curious how you manage disks? Do you use RDM?s? Does this even make sense to do? If you have a 2-3 node cluster how do you share the disks across? Do you have VM?s with their own VMDK?s (if not RDM) in each node or is there some way to share access to the same VMDK?s? What are the advantages doing this other than existing HW use? Seems to me for a lab environment or very small nonperformance focused implementation this may be a viable option. Thanks Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From luis.bolinches at fi.ibm.com Tue Oct 25 19:58:21 2016 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Tue, 25 Oct 2016 18:58:21 +0000 Subject: [gpfsug-discuss] Virtualized Spectrum Scale In-Reply-To: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> Message-ID: Hi You must use RDM. Otherwise is not supported. SCSI commands is the reason. Furthermore on some versions I managed to crash the ESXi as well. -- Cheers > On 25 Oct 2016, at 19.46, "Mark.Bush at siriuscom.com" wrote: > > Anyone running SpectrumScale on Virtual Machines (intel)? I?m curious how you manage disks? Do you use RDM?s? Does this even make sense to do? If you have a 2-3 node cluster how do you share the disks across? Do you have VM?s with their own VMDK?s (if not RDM) in each node or is there some way to share access to the same VMDK?s? What are the advantages doing this other than existing HW use? Seems to me for a lab environment or very small nonperformance focused implementation this may be a viable option. > > Thanks > > Mark > This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. > > Sirius Computer Solutions Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: From kevindjo at us.ibm.com Tue Oct 25 20:05:12 2016 From: kevindjo at us.ibm.com (Kevin D Johnson) Date: Tue, 25 Oct 2016 19:05:12 +0000 Subject: [gpfsug-discuss] Virtualized Spectrum Scale In-Reply-To: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> References: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> Message-ID: An HTML attachment was scrubbed... URL: From Mark.Bush at siriuscom.com Tue Oct 25 20:34:29 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Tue, 25 Oct 2016 19:34:29 +0000 Subject: [gpfsug-discuss] Virtualized Spectrum Scale In-Reply-To: References: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> Message-ID: <5482A551-873C-4F9F-89BF-0CF0FB926D92@siriuscom.com> This is interesting since the SS FAQ leads me to believe that I have some options here. Does GPFS nodes with no direct disk access still mean RDMs? [cid:image001.png at 01D22ECC.E20EC2F0] From: on behalf of Luis Bolinches Reply-To: gpfsug main discussion list Date: Tuesday, October 25, 2016 at 1:58 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Virtualized Spectrum Scale Hi You must use RDM. Otherwise is not supported. SCSI commands is the reason. Furthermore on some versions I managed to crash the ESXi as well. -- Cheers On 25 Oct 2016, at 19.46, "Mark.Bush at siriuscom.com" > wrote: Anyone running SpectrumScale on Virtual Machines (intel)? I?m curious how you manage disks? Do you use RDM?s? Does this even make sense to do? If you have a 2-3 node cluster how do you share the disks across? Do you have VM?s with their own VMDK?s (if not RDM) in each node or is there some way to share access to the same VMDK?s? What are the advantages doing this other than existing HW use? Seems to me for a lab environment or very small nonperformance focused implementation this may be a viable option. Thanks Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 169073 bytes Desc: image001.png URL: From gsanjay at us.ibm.com Tue Oct 25 20:48:58 2016 From: gsanjay at us.ibm.com (Sanjay Gandhi) Date: Tue, 25 Oct 2016 12:48:58 -0700 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 57, Issue 66 In-Reply-To: References: Message-ID: VMDK is not supported due to http://kb.vmware.com/kb/2032940 It can cause GPFS deadlock due to hung IO. Thanks, Sanjay Gandhi GPFS FVT IBM, Beaverton Phone/FAX : 503-578-4141 T/L 775-4141 gsanjay at us.ibm.com From: gpfsug-discuss-request at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Date: 10/25/2016 12:35 PM Subject: gpfsug-discuss Digest, Vol 57, Issue 66 Sent by: gpfsug-discuss-bounces at spectrumscale.org Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Virtualized Spectrum Scale (Mark.Bush at siriuscom.com) ---------------------------------------------------------------------- Message: 1 Date: Tue, 25 Oct 2016 19:34:29 +0000 From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Virtualized Spectrum Scale Message-ID: <5482A551-873C-4F9F-89BF-0CF0FB926D92 at siriuscom.com> Content-Type: text/plain; charset="utf-8" This is interesting since the SS FAQ leads me to believe that I have some options here. Does GPFS nodes with no direct disk access still mean RDMs? [cid:image001.png at 01D22ECC.E20EC2F0] From: on behalf of Luis Bolinches Reply-To: gpfsug main discussion list Date: Tuesday, October 25, 2016 at 1:58 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Virtualized Spectrum Scale Hi You must use RDM. Otherwise is not supported. SCSI commands is the reason. Furthermore on some versions I managed to crash the ESXi as well. -- Cheers On 25 Oct 2016, at 19.46, "Mark.Bush at siriuscom.com< mailto:Mark.Bush at siriuscom.com>" > wrote: Anyone running SpectrumScale on Virtual Machines (intel)? I?m curious how you manage disks? Do you use RDM?s? Does this even make sense to do? If you have a 2-3 node cluster how do you share the disks across? Do you have VM?s with their own VMDK?s (if not RDM) in each node or is there some way to share access to the same VMDK?s? What are the advantages doing this other than existing HW use? Seems to me for a lab environment or very small nonperformance focused implementation this may be a viable option. Thanks Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161025/fc9a8bf7/attachment.html > -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 169073 bytes Desc: image001.png URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161025/fc9a8bf7/attachment.png > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 66 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From Mark.Bush at siriuscom.com Tue Oct 25 20:50:51 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Tue, 25 Oct 2016 19:50:51 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 57, Issue 66 In-Reply-To: References: Message-ID: <0A2BC512-5EB7-4531-91DD-3C862A89C2E2@siriuscom.com> Ok. I re-read the FAQ and I think the option in the Table is a bit misleading. I think it means it?s supported without direct disk access and that means GPFS client not an NSD. Sound right? From: on behalf of Sanjay Gandhi Reply-To: gpfsug main discussion list Date: Tuesday, October 25, 2016 at 2:48 PM To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 57, Issue 66 VMDK is not supported due to http://kb.vmware.com/kb/2032940 It can cause GPFS deadlock due to hung IO. Thanks, Sanjay Gandhi GPFS FVT IBM, Beaverton Phone/FAX : 503-578-4141 T/L 775-4141 gsanjay at us.ibm.com [nactive hide details for gpfsug-discuss-request---10/25/2016 12:35:04 PM-]gpfsug-discuss-request---10/25/2016 12:35:04 PM---Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org From: gpfsug-discuss-request at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Date: 10/25/2016 12:35 PM Subject: gpfsug-discuss Digest, Vol 57, Issue 66 Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Virtualized Spectrum Scale (Mark.Bush at siriuscom.com) ---------------------------------------------------------------------- Message: 1 Date: Tue, 25 Oct 2016 19:34:29 +0000 From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Virtualized Spectrum Scale Message-ID: <5482A551-873C-4F9F-89BF-0CF0FB926D92 at siriuscom.com> Content-Type: text/plain; charset="utf-8" This is interesting since the SS FAQ leads me to believe that I have some options here. Does GPFS nodes with no direct disk access still mean RDMs? [cid:image001.png at 01D22ECC.E20EC2F0] From: on behalf of Luis Bolinches Reply-To: gpfsug main discussion list Date: Tuesday, October 25, 2016 at 1:58 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Virtualized Spectrum Scale Hi You must use RDM. Otherwise is not supported. SCSI commands is the reason. Furthermore on some versions I managed to crash the ESXi as well. -- Cheers On 25 Oct 2016, at 19.46, "Mark.Bush at siriuscom.com" > wrote: Anyone running SpectrumScale on Virtual Machines (intel)? I?m curious how you manage disks? Do you use RDM?s? Does this even make sense to do? If you have a 2-3 node cluster how do you share the disks across? Do you have VM?s with their own VMDK?s (if not RDM) in each node or is there some way to share access to the same VMDK?s? What are the advantages doing this other than existing HW use? Seems to me for a lab environment or very small nonperformance focused implementation this may be a viable option. Thanks Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 169073 bytes Desc: image001.png URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 66 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 106 bytes Desc: image001.gif URL: From laurence at qsplace.co.uk Tue Oct 25 20:53:55 2016 From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow) Date: Tue, 25 Oct 2016 20:53:55 +0100 Subject: [gpfsug-discuss] Virtualized Spectrum Scale In-Reply-To: References: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> Message-ID: Kevin, This is how I run test systems, I let iovirt devices to be attached to multiple kvm systems. It works well. -- Lauz On 25 October 2016 20:05:12 BST, Kevin D Johnson wrote: >Mark, > > > >You can run Spectrum Scale with virtual machines. As long as the >virtual disks present to the file system as devices, you should be good >to go (for example, "cat /proc/partitions" should show your virtual >disks as devices). I typically use scsi raw devices with virtual >machines and that seems to work well. KVM allows you to share the >disks as well between hosts and that's important to emulate more >production level uses. It is good for a lab or to try something >quickly without provisioning actual machines. We have sometimes used >SS with virtual machines in production but we typically recommend bare >metal if/when possible. > > > >Kevin D. Johnson, MBA, MAFM >Spectrum Computing, Senior Managing Consultant > >IBM Certified Deployment Professional - Spectrum Scale V4.1.1 >IBM Certified Deployment Professional - Cloud Object Storage V3.8 > >IBM Certified Solution Advisor - Spectrum Computing V1 > > > >720.349.6199 - kevindjo at us.ibm.com > > > > > > > >----- Original message ----- >From: "Mark.Bush at siriuscom.com" >Sent by: gpfsug-discuss-bounces at spectrumscale.org >To: "gpfsug-discuss at spectrumscale.org" > >Cc: >Subject: [gpfsug-discuss] Virtualized Spectrum Scale >Date: Tue, Oct 25, 2016 2:47 PM > > >Anyone running SpectrumScale on Virtual Machines (intel)? I?m curious >how you manage disks? Do you use RDM?s? Does this even make sense to >do? If you have a 2-3 node cluster how do you share the disks across? >Do you have VM?s with their own VMDK?s (if not RDM) in each node or is >there some way to share access to the same VMDK?s? What are the >advantages doing this other than existing HW use? Seems to me for a >lab environment or very small nonperformance focused implementation >this may be a viable option. > > > >Thanks > > > >Mark > >This message (including any attachments) is intended only for the use >of the individual or entity to which it is addressed and may contain >information that is non-public, proprietary, privileged, confidential, >and exempt from disclosure under applicable law. If you are not the >intended recipient, you are hereby notified that any use, >dissemination, distribution, or copying of this communication is >strictly prohibited. This message may be viewed by parties at Sirius >Computer Solutions other than those named in the message header. This >message does not contain an official representation of Sirius Computer >Solutions. If you have received this communication in error, notify >Sirius Computer Solutions immediately and (i) destroy this message if a >facsimile or (ii) delete this message immediately if this is an >electronic communication. Thank you. > >Sirius Computer Solutions > > > >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > >------------------------------------------------------------------------ > >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aspalazz at us.ibm.com Tue Oct 25 20:59:31 2016 From: aspalazz at us.ibm.com (Aaron S Palazzolo) Date: Tue, 25 Oct 2016 19:59:31 +0000 Subject: [gpfsug-discuss] Virtualized Spectrum Scale In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From kevindjo at us.ibm.com Tue Oct 25 21:02:42 2016 From: kevindjo at us.ibm.com (Kevin D Johnson) Date: Tue, 25 Oct 2016 20:02:42 +0000 Subject: [gpfsug-discuss] Virtualized Spectrum Scale In-Reply-To: References: , <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> Message-ID: An HTML attachment was scrubbed... URL: From Mark.Bush at siriuscom.com Tue Oct 25 21:36:51 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Tue, 25 Oct 2016 20:36:51 +0000 Subject: [gpfsug-discuss] Virtualized Spectrum Scale In-Reply-To: References: Message-ID: <82AD0334-B515-48FD-8C30-FC8F309AD00F@siriuscom.com> Excellent info Aaron. Thanks for this. I may reach out via PM to explore this more with you. From: on behalf of Aaron S Palazzolo Reply-To: gpfsug main discussion list Date: Tuesday, October 25, 2016 at 2:59 PM To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] Virtualized Spectrum Scale Hi Mark, Great questions on the VM side. Within our Spectrum Scale Test labs at IBM, we run within both VMware and KVM. It sounds as though your questions are more towards the VMware side so I'll give a few pointers. For more detailed info, feel free to grab me on the side or continue the discussion within the user group so that others can learn from this as well. #1) As Luis suggests, using vmdks will not give you full support of the scsi protocol. Specifically persistent reserve. EIO errors are also not passed correctly in some path down situations. - For certain test/dev environments, in which data protection and performance are not high priorities, then this may be fine. Some of our test environments do run like this. - Note that vmdks can be shared among virtual Spectrum Scale nodes using the multi-writer flag in VMware. To do this, you'll first need to create your vmdks using 'Thick Provision Eager Zeroed'. You'll then need to configure the multi-writer flag for scsi-sharing such as this, for each vmdk: scsi1:0.sharing = "multi-writer" scsi1:1.sharing = "multi-writer" scsi1:2.sharing = "multi-writer" You'll find this in the Advanced configuration parameters. Finally, you'll need to set all of these vmdks to a separate scsi adapter which is configured for either virtual or physical sharing. Downsides are lack of support, some degradation of performance, lack of failover support, and inability to use VMware snapshots of VMs with scsi sharing/multi-writer enabled. Upsides are ease of setup both with virtually and physically and ability to store all VMs on a single datastore that can itself be snapshotted if the underlying physical storage supports this. In our testlabs, we create 4node -> 50node virtual Spectrum Scale clusters, each node is a VM, some of these VMs have extra vmdks for NSDs and some do not. All VMs belonging to a cluster reside on a single datastore which ends up being a single XIV Volume. We then can snapshot this XIV volume and in essence, snapshot the entire Spectrum Scale cluster back and forth in time. I'm sure this is overly complicated for what you want to do, but it may get you thinking of use cases. #2) RDM will give you both performance and the piece of mind that you're using a fully supported config. Officially, I will always recommend RDM due to this. The downside to RDM is complexity in setup unless you have a fairly static config or unless you can automate the zoning. #3) No matter what type of VM infrastructure you use, make sure you investigate the memory/cpu requirements. - Check our FAQ, specifically 6.1 for tuning info regarding vm.min_free_kbytes: https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#gpfsclustersfaqAugust2016-gen4__lintunq - We have run some of our test clusters with as little as 4GB of memory but I would definitely recommend quite a bit more memory in each VM for production use. If you use additional functions other than core file system, pay attention to the memory requirements of these. #4) KVM is a viable alternative if needed..... Regards, Aaron Palazzolo IBM Spectrum Scale Deployment, Infrastructure, Virtualization 9042 S Rita Road, Tucson AZ 85744 Phone: 520-799-5161, T/L: 321-5161 E-mail: aspalazz at us.ibm.com ----- Original message ----- From: gpfsug-discuss-request at spectrumscale.org Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Cc: Subject: gpfsug-discuss Digest, Vol 57, Issue 65 Date: Tue, Oct 25, 2016 12:05 PM Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. October meet the devs report (Simon Thompson (Research Computing - IT Services)) 2. Virtualized Spectrum Scale (Mark.Bush at siriuscom.com) 3. Re: Virtualized Spectrum Scale (Luis Bolinches) 4. Re: Virtualized Spectrum Scale (Kevin D Johnson) ---------------------------------------------------------------------- Message: 1 Date: Tue, 25 Oct 2016 15:33:16 +0000 From: "Simon Thompson (Research Computing - IT Services)" To: "gpfsug-discuss at spectrumscale.org" Subject: [gpfsug-discuss] October meet the devs report Message-ID: Content-Type: text/plain; charset="us-ascii" The October meet the devs workshop on cloud was last week. Thanks to Dean Hildebrand and John Lewars for flying in from the US to support us, Ulf Troppens and Dan Kidger also from IBM. And finally thanks to OCF for buying the pizza (one day we'll manage to get the pizza co to deliver on time!). The event report is now up on the group website at: http://www.spectrumscale.org/meet-the-devs-cloud-workshop-birmingham-uk/ Simon ------------------------------ Message: 2 Date: Tue, 25 Oct 2016 18:46:26 +0000 From: "Mark.Bush at siriuscom.com" To: "gpfsug-discuss at spectrumscale.org" Subject: [gpfsug-discuss] Virtualized Spectrum Scale Message-ID: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD at siriuscom.com> Content-Type: text/plain; charset="utf-8" Anyone running SpectrumScale on Virtual Machines (intel)? I?m curious how you manage disks? Do you use RDM?s? Does this even make sense to do? If you have a 2-3 node cluster how do you share the disks across? Do you have VM?s with their own VMDK?s (if not RDM) in each node or is there some way to share access to the same VMDK?s? What are the advantages doing this other than existing HW use? Seems to me for a lab environment or very small nonperformance focused implementation this may be a viable option. Thanks Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 3 Date: Tue, 25 Oct 2016 18:58:21 +0000 From: "Luis Bolinches" To: "gpfsug main discussion list" Subject: Re: [gpfsug-discuss] Virtualized Spectrum Scale Message-ID: Content-Type: text/plain; charset="utf-8" Hi You must use RDM. Otherwise is not supported. SCSI commands is the reason. Furthermore on some versions I managed to crash the ESXi as well. -- Cheers > On 25 Oct 2016, at 19.46, "Mark.Bush at siriuscom.com" wrote: > > Anyone running SpectrumScale on Virtual Machines (intel)? I?m curious how you manage disks? Do you use RDM?s? Does this even make sense to do? If you have a 2-3 node cluster how do you share the disks across? Do you have VM?s with their own VMDK?s (if not RDM) in each node or is there some way to share access to the same VMDK?s? What are the advantages doing this other than existing HW use? Seems to me for a lab environment or very small nonperformance focused implementation this may be a viable option. > > Thanks > > Mark > This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. > > Sirius Computer Solutions Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 4 Date: Tue, 25 Oct 2016 19:05:12 +0000 From: "Kevin D Johnson" To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Virtualized Spectrum Scale Message-ID: Content-Type: text/plain; charset="us-ascii" An HTML attachment was scrubbed... URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 65 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: From leslie.james.elliott at gmail.com Tue Oct 25 23:01:25 2016 From: leslie.james.elliott at gmail.com (leslie elliott) Date: Wed, 26 Oct 2016 08:01:25 +1000 Subject: [gpfsug-discuss] unified file and object Message-ID: Hi We are in the process of trying to configure unified file and object storage in unified_mode and have a few problems We are running 4.2.1 and do not have any current issues the file access protocols setting up the authentication First issue we do have is binding the object data to our Active Directory, we seem to be hitting a road block due to the fact that the bind DN has spaces in it, if we enclose the DN in quotes it still fails, if we escape them with the appropriate RFC value we can get the mmuserauth to complete but the lookups from the local keystone fail for the authentication of the users The DN for the swift user and swift admin also have quotes in them, so just doing it on the command line is not enough to get the mmuserauth command to complete Second problem is OBJ:openstack-swift-object-sof is not running This seems to be due to the config file not having bind_ip and bind_port values, if these are added then the error turns to pipeline of other settings in the config file missing This particular issue occurs no matter what the auth type is set to be for object Hopefully this make some sense to someone Thanks leslie Leslie Elliott, Infrastructure Support Specialist, Faculty Infrastructure and Applications Support Information Technology Services, The University of Queensland -------------- next part -------------- An HTML attachment was scrubbed... URL: From Greg.Lehmann at csiro.au Wed Oct 26 01:51:16 2016 From: Greg.Lehmann at csiro.au (Greg.Lehmann at csiro.au) Date: Wed, 26 Oct 2016 00:51:16 +0000 Subject: [gpfsug-discuss] Forcing which node gets expelled? In-Reply-To: <63906C4AF5FFD0429565A743F6AD5BEDDAF314@MBX04.ad.oak.ox.ac.uk> References: <63906C4AF5FFD0429565A743F6AD5BEDDAF29F@MBX04.ad.oak.ox.ac.uk> <4403E9DB-88B4-4281-91B5-E171D1284899@nuance.com> <63906C4AF5FFD0429565A743F6AD5BEDDAF314@MBX04.ad.oak.ox.ac.uk> Message-ID: We hit something like this due to a bug in gskit. We all thought it was networking at first and it took me a fair bit of time to check all that. We have 7 nsd servers and around 400 clients running 4.2.0.4. We are just trying a workaround now that looks promising. The bug will be fixed at some point. Cheers, Greg From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Thorpe Sent: Tuesday, 25 October 2016 11:06 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Forcing which node gets expelled? Hi Bob, That is exactly what I was after, thanks very much! Should buy us a little time so we can resolve our networking issue. Thanks again Matt. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Oesterlin, Robert Sent: 25 October 2016 13:23 To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Forcing which node gets expelled? If you look at /usr/lpp/mmfs/samples/expelnode.sample you can use this as a base and install this in /var/mmfs/etc on the cluster manager. You can control which of the two nodes get expelled. We use it here to send an alert when a node is expelled. There is also "mmexpelnode" which you can force a particular node to be expelled. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: > on behalf of Matt Thorpe > Reply-To: gpfsug main discussion list > Date: Tuesday, October 25, 2016 at 7:05 AM To: gpfsug main discussion list > Subject: [EXTERNAL] [gpfsug-discuss] Forcing which node gets expelled? Hi, We are in the process of diagnosing a networking issue that is causing 2 of our 6 node GPFS cluster to expel each other (it appears they experience a temporary network connection outage and lose contact with each other). At present it's not consistent which gets expelled by the cluster manager, and I wondered if there was any way to always force a specific node to be expelled in this situation? Thanks and best regards, Matt -------- Matt Thorpe | BDLSS Systems Administrator Bodleian Libraries Osney One Building, Osney Mead, Oxford, OX2 0EW matt.thorpe at bodleian.ox.ac.uk | 01865 (2)80027 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Q_f6z64tvENxA9ac7rqWFj2jNd5IrpWEXcynJzHFjz4&s=jqso6xVB-V_zgLba-xjlWwiw3fNRan9NspsVq4PY4nA&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From Greg.Lehmann at csiro.au Wed Oct 26 02:04:35 2016 From: Greg.Lehmann at csiro.au (Greg.Lehmann at csiro.au) Date: Wed, 26 Oct 2016 01:04:35 +0000 Subject: [gpfsug-discuss] Virtualized Spectrum Scale In-Reply-To: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> References: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> Message-ID: <72fda8bf40634ad38e72689756a444cf@exch1-cdc.nexus.csiro.au> We use KVM running on a Debian host, with CentOS guests. Storage is zoned from our DDN Infiniband array to the host and then passed through to the guests. We would like to zone it directly to the guests SRIOV IB HCA, but srp seems to be a bit of dead code tree. We had to do a bit of work to get it working with Debian and haven?t as yet spent the time on getting it going with CentOS. We also run Spectrum Archive on the guest with tape drives and libraries zoned to the guest?s PCIe HBAs which are passed through from the host. We are working towards going production with this setup. Xen was a bit a failure for us so we switched to KVM. Cheers, Greg From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Mark.Bush at siriuscom.com Sent: Wednesday, 26 October 2016 4:46 AM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Virtualized Spectrum Scale Anyone running SpectrumScale on Virtual Machines (intel)? I?m curious how you manage disks? Do you use RDM?s? Does this even make sense to do? If you have a 2-3 node cluster how do you share the disks across? Do you have VM?s with their own VMDK?s (if not RDM) in each node or is there some way to share access to the same VMDK?s? What are the advantages doing this other than existing HW use? Seems to me for a lab environment or very small nonperformance focused implementation this may be a viable option. Thanks Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Wed Oct 26 03:48:38 2016 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 25 Oct 2016 22:48:38 -0400 Subject: [gpfsug-discuss] Virtualized Spectrum Scale In-Reply-To: <72fda8bf40634ad38e72689756a444cf@exch1-cdc.nexus.csiro.au> References: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> <72fda8bf40634ad38e72689756a444cf@exch1-cdc.nexus.csiro.au> Message-ID: <2e759489-a38f-58d2-4e20-b9c090c21b4a@nasa.gov> Hi Greg, I'm rather curious about your srp difficulties (not to get too off topic). Is it an issue with srp by itself or the interaction of srp with the SRIOV IB HCA? We've used SRP quite a bit both virtualized and not and have seen good results both in terms of stability and performance. -Aaron On 10/25/16 9:04 PM, Greg.Lehmann at csiro.au wrote: > We use KVM running on a Debian host, with CentOS guests. Storage is > zoned from our DDN Infiniband array to the host and then passed through > to the guests. We would like to zone it directly to the guests SRIOV IB > HCA, but srp seems to be a bit of dead code tree. We had to do a bit of > work to get it working with Debian and haven?t as yet spent the time on > getting it going with CentOS. > > > > We also run Spectrum Archive on the guest with tape drives and libraries > zoned to the guest?s PCIe HBAs which are passed through from the host. > We are working towards going production with this setup. > > > > Xen was a bit a failure for us so we switched to KVM. > > > > Cheers, > > > > Greg > > > > *From:*gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of > *Mark.Bush at siriuscom.com > *Sent:* Wednesday, 26 October 2016 4:46 AM > *To:* gpfsug-discuss at spectrumscale.org > *Subject:* [gpfsug-discuss] Virtualized Spectrum Scale > > > > Anyone running SpectrumScale on Virtual Machines (intel)? I?m curious > how you manage disks? Do you use RDM?s? Does this even make sense to > do? If you have a 2-3 node cluster how do you share the disks across? > Do you have VM?s with their own VMDK?s (if not RDM) in each node or is > there some way to share access to the same VMDK?s? What are the > advantages doing this other than existing HW use? Seems to me for a lab > environment or very small nonperformance focused implementation this may > be a viable option. > > > > Thanks > > > > Mark > > This message (including any attachments) is intended only for the use of > the individual or entity to which it is addressed and may contain > information that is non-public, proprietary, privileged, confidential, > and exempt from disclosure under applicable law. If you are not the > intended recipient, you are hereby notified that any use, dissemination, > distribution, or copying of this communication is strictly prohibited. > This message may be viewed by parties at Sirius Computer Solutions other > than those named in the message header. This message does not contain an > official representation of Sirius Computer Solutions. If you have > received this communication in error, notify Sirius Computer Solutions > immediately and (i) destroy this message if a facsimile or (ii) delete > this message immediately if this is an electronic communication. Thank you. > > *Sirius Computer Solutions * > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From Greg.Lehmann at csiro.au Wed Oct 26 04:07:19 2016 From: Greg.Lehmann at csiro.au (Greg.Lehmann at csiro.au) Date: Wed, 26 Oct 2016 03:07:19 +0000 Subject: [gpfsug-discuss] Virtualized Spectrum Scale In-Reply-To: <2e759489-a38f-58d2-4e20-b9c090c21b4a@nasa.gov> References: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> <72fda8bf40634ad38e72689756a444cf@exch1-cdc.nexus.csiro.au> <2e759489-a38f-58d2-4e20-b9c090c21b4a@nasa.gov> Message-ID: The srp work was done a few years ago now. We use the same srp code for both physical and virtual, so I am guessing it has nothing to do with the SRIOV side of things. Somebody else did the work, so I will try and get an answer for you. I agree performance and stability is good with physical and virtual. Cheers, Greg -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aaron Knister Sent: Wednesday, 26 October 2016 12:49 PM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Virtualized Spectrum Scale Hi Greg, I'm rather curious about your srp difficulties (not to get too off topic). Is it an issue with srp by itself or the interaction of srp with the SRIOV IB HCA? We've used SRP quite a bit both virtualized and not and have seen good results both in terms of stability and performance. -Aaron On 10/25/16 9:04 PM, Greg.Lehmann at csiro.au wrote: > We use KVM running on a Debian host, with CentOS guests. Storage is > zoned from our DDN Infiniband array to the host and then passed > through to the guests. We would like to zone it directly to the guests > SRIOV IB HCA, but srp seems to be a bit of dead code tree. We had to > do a bit of work to get it working with Debian and haven't as yet > spent the time on getting it going with CentOS. > > > > We also run Spectrum Archive on the guest with tape drives and > libraries zoned to the guest's PCIe HBAs which are passed through from the host. > We are working towards going production with this setup. > > > > Xen was a bit a failure for us so we switched to KVM. > > > > Cheers, > > > > Greg > > > > *From:*gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of > *Mark.Bush at siriuscom.com > *Sent:* Wednesday, 26 October 2016 4:46 AM > *To:* gpfsug-discuss at spectrumscale.org > *Subject:* [gpfsug-discuss] Virtualized Spectrum Scale > > > > Anyone running SpectrumScale on Virtual Machines (intel)? I'm curious > how you manage disks? Do you use RDM's? Does this even make sense to > do? If you have a 2-3 node cluster how do you share the disks across? > Do you have VM's with their own VMDK's (if not RDM) in each node or is > there some way to share access to the same VMDK's? What are the > advantages doing this other than existing HW use? Seems to me for a > lab environment or very small nonperformance focused implementation > this may be a viable option. > > > > Thanks > > > > Mark > > This message (including any attachments) is intended only for the use > of the individual or entity to which it is addressed and may contain > information that is non-public, proprietary, privileged, confidential, > and exempt from disclosure under applicable law. If you are not the > intended recipient, you are hereby notified that any use, > dissemination, distribution, or copying of this communication is strictly prohibited. > This message may be viewed by parties at Sirius Computer Solutions > other than those named in the message header. This message does not > contain an official representation of Sirius Computer Solutions. If > you have received this communication in error, notify Sirius Computer > Solutions immediately and (i) destroy this message if a facsimile or > (ii) delete this message immediately if this is an electronic communication. Thank you. > > *Sirius Computer Solutions * > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From janfrode at tanso.net Wed Oct 26 12:53:21 2016 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Wed, 26 Oct 2016 13:53:21 +0200 Subject: [gpfsug-discuss] filesystem thresholds in gui alerting Message-ID: Does anybody know if there are any way to define what thresholds are to be used for alterting in the GUI? F.ex. we have some filesystems that are very full, but won't be getting any more data added.. we'd like to turn off monitoring of these, are raise the threshold to allow them to be ~100% full. -jf -------------- next part -------------- An HTML attachment was scrubbed... URL: From stschmid at de.ibm.com Wed Oct 26 16:14:27 2016 From: stschmid at de.ibm.com (Stefan Schmidt) Date: Wed, 26 Oct 2016 17:14:27 +0200 Subject: [gpfsug-discuss] filesystem thresholds in gui alerting In-Reply-To: References: Message-ID: Hi Jan-Frode Myklebust, The current GUI-internal logic has a hard coded warning level of 80% and an hard-coded error level of 90%. With 4.2.2 the system health monitoring will have thresholds set and monitored via ZIMON and the system health component will provide events. With 4.2.2 those can be changed with an mm command. In general it is not a good idea to let a filesystem get loaded 100%. It could be, if this is done with the filesystem used by the system to store some configuration data, that the system can run into an error (usually this is the filesystem we call gpfs0). I suggest you get the 4.2.2 version which will GA quite soon. I have to apologize that I'm not allowed to talk about GA dates. If you want to get this feature earlier you may contact IBM and ask for participation on the beta program which starts very soon but the beta code is usually limited to non productive systems. As said , ask your IBM contact for the 4.2.2 GA update ( hopefully GA is soon) and you can get this feature you are looking for. Mit freundlichen Gr??en / Kind regards Stefan Schmidt Scrum Master IBM Spectrum Scale GUI / Senior IT Architect/PMP?-Dept. M069 / IBM Spectrum Scale Software Development IBM Systems Group IBM Deutschland Phone: +49-703 4274 1966 IBM Deutschland Am Weiher 24 E-Mail: stschmid at de.ibm.com 65421 Kelsterbach Germany IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: Jan-Frode Myklebust To: gpfsug main discussion list Date: 26.10.2016 13:53 Subject: [gpfsug-discuss] filesystem thresholds in gui alerting Sent by: gpfsug-discuss-bounces at spectrumscale.org Does anybody know if there are any way to define what thresholds are to be used for alterting in the GUI? F.ex. we have some filesystems that are very full, but won't be getting any more data added.. we'd like to turn off monitoring of these, are raise the threshold to allow them to be ~100% full. -jf_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulmer at ulmer.org Thu Oct 27 01:48:50 2016 From: ulmer at ulmer.org (Stephen Ulmer) Date: Wed, 26 Oct 2016 20:48:50 -0400 Subject: [gpfsug-discuss] filesystem thresholds in gui alerting In-Reply-To: References: Message-ID: <76513F28-0646-4282-A595-C69A9F32A945@ulmer.org> It seems prudent to note here that a file system that reports 100% full can have quite a bit of free space. Take for example a 10TB file system that is no longer written to ? no one wants to use 100GB just to make the alert color green. This gets silly very quickly at even ?medium? sizes... What would be really cool would be to decide how much reserved space you want, look at the velocity[1] with which the filesystem is filling, and then alert when the time until left until full is less than an operations shift (or something useful). Maybe that is just silly. Hey, Jan-Frode, can you mount the no-more-data file systems read only? Maybe not alerting on the full-ness of read only filesystems is easier and more sane? I mean, they can?t fill up any more. Liberty, -- Stephen > On Oct 26, 2016, at 11:14 AM, Stefan Schmidt wrote: > > Hi Jan-Frode Myklebust, > > The current GUI-internal logic has a hard coded warning level of 80% and an hard-coded error level of 90%. > > With 4.2.2 the system health monitoring will have thresholds set and monitored via ZIMON and the system health > component will provide events.With 4.2.2 those can be changed with an mm command. > > In general it is not a good idea to let a filesystem get loaded 100%. It could be, if this is done with the filesystem used by the system to store some configuration data, that the system can run into an error (usually this is the filesystem we call gpfs0). > > I suggest you get the 4.2.2 version which will GA quite soon. I have to apologize that I'm not allowed to talk about GA dates. If you want to get this feature earlier you may contact IBM and ask for participation on the beta program which starts very soon but the beta code is usually limited to non productive systems. > > As said , ask your IBM contact for the 4.2.2 GA update ( hopefully GA is soon) and you can get this feature you are looking for. > Mit freundlichen Gr??en / Kind regards > > > Stefan Schmidt > > > Scrum Master IBM Spectrum Scale GUI / Senior IT Architect/PMP?-Dept. M069 / IBM Spectrum Scale Software Development > > IBM Systems Group > > IBM Deutschland > > Phone: +49-703 4274 1966 IBM Deutschland > Am Weiher 24 > E-Mail: stschmid at de.ibm.com 65421 Kelsterbach > Germany > IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Martina Koederitz > Gesch?ftsf?hrung: Dirk Wittkopp > Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 > > > > > > > > From: Jan-Frode Myklebust > To: gpfsug main discussion list > Date: 26.10.2016 13:53 > Subject: [gpfsug-discuss] filesystem thresholds in gui alerting > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > > > Does anybody know if there are any way to define what thresholds are to be used for alterting in the GUI? F.ex. we have some filesystems that are very full, but won't be getting any more data added.. we'd like to turn off monitoring of these, are raise the threshold to allow them to be ~100% full. > > > > -jf_______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From usa-principal at gpfsug.org Thu Oct 27 02:06:06 2016 From: usa-principal at gpfsug.org (Spectrum Scale Users Group - USA Principal Kristy Kallback-Rose) Date: Wed, 26 Oct 2016 21:06:06 -0400 Subject: [gpfsug-discuss] [UPDATE] SC16 Draft Agenda [Save the date: Sunday, November 13th] In-Reply-To: References: Message-ID: <10BF7862-5EB0-412E-9B1D-9531D89B87C2@gpfsug.org> All, Here is where we?re at with the agenda. Still finalizing a few things, but wanted to keep you all in the loop. Registration Web Site [Register for the SC16 UG event here]: https://www-01.ibm.com/events/wwe/grp/grp305.nsf/Agenda.xsp?openform&seminar=357M7UES&locale=en_US&S_TACT=000001CP&S_OFF_CD=10001235 IBM summary page: http://www-03.ibm.com/systems/technicalcomputing/supercomputing/index.html Location: Salt Lake City Downtown Marriott [Room Number TBD] 75 SW Temple Street, Salt Lake City UT 84101 Draft Agenda: 12:30 12:40 Welcome 12:40 12:50 Overview 12:50 13:35 The latest in IBM Spectrum Scale and ESS 13:55 14:20 NASA Goddard: Big Data in HPC 14:20 14:45 Nuance: Tiering to Object Storage 14:45 15:15 - break - 15:15 15:30 Sponsor Talk: TOPIC TBD (NetApp) 13:35 13:55 Virginia Tech ARC: SANDisk IF150 (IBM DeepFlash 150) 15:30 15:50 NASA Goddard: Monitoring with Grafana and InfluxDB 15:50 16:35 Spectrum Scale Enhancements for CORAL 16:35 17:20 News from IBM Research 17:20 17:30 Closing [Reception to follow closing, details TBD] Questions? Comments? Let us know. Hope to see many of you next month. Final agenda will be made available before the event. Cheers, Kristy & Bob Kristy Kallback-Rose Manager, Research Storage Indiana University > On Oct 10, 2016, at 6:59 PM, GPFS UG USA Principal > wrote: > > Hello all, > > There have been some questions about the Spectrum Scale Users Group event for SC16 and we thought it best to publish our draft agenda and continue to fill it in as details get settled. That way you can make your travel plans and schedules. > > The date and rough times are set, we may adjust the time range a bit depending upon the number of user talks that people would like to give. So note, yes! we are asking for user talks. Please consider doing this. We always receive positive feedback about talks given by users that describe real world experiences. If any of the user talk slots below are of interest to you, please let us know as soon as you can. We may turn at least one user talk into a panel if we can get enough participants. > > As always, your feedback is welcome. This is a users group after all. > > So, here?s what we have planned so far: > > Date: Sunday November 13th > Venue: TBD, but near the convention center in downtown SLC > Time: ~12:30p - ~17:30p > > Agenda: > 12:30 - 12:50 Welcome / Overview > 12:50 - 13:50 The latest in IBM Spectrum Scale and ESS > 13:50 - 14:20 User Talk: Big Data in HPC > 14:20 - 14:50 User Talk: Tiering to Object Storage > 14:50 - 15:20 - break - > 15:20 - 15:50 User Talk: TBD ? volunteers! please! > 15:50 - 16:35 Spectrum Scale Enhancements for CORAL > 16:35 - 17:20 News from IBM Research > 17:20 - 17:30 Closing > > Best, > > Kristy & Bob > > Kristy Kallback-Rose > Manager, Research Storage > Indiana University > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Ola.Pontusson at kb.se Thu Oct 27 07:49:24 2016 From: Ola.Pontusson at kb.se (Ola Pontusson) Date: Thu, 27 Oct 2016 06:49:24 +0000 Subject: [gpfsug-discuss] filesystem thresholds in gui alerting In-Reply-To: <76513F28-0646-4282-A595-C69A9F32A945@ulmer.org> References: <76513F28-0646-4282-A595-C69A9F32A945@ulmer.org> Message-ID: Hi I am really looking forward to the 4.2.2 where I hopefully can adjust the thresholds of alarm levels. We just as Jan-Frode have filesystems that is not growing so much and waste a large amount of disk just to keep it green is never going to happend. Below is an example of one of our filesystems. /dev/smdb04 942T 900T 42T 96% /gpfs/smdb04 /Ola Fr?n: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] F?r Stephen Ulmer Skickat: den 27 oktober 2016 02:49 Till: gpfsug main discussion list ?mne: Re: [gpfsug-discuss] filesystem thresholds in gui alerting It seems prudent to note here that a file system that reports 100% full can have quite a bit of free space. Take for example a 10TB file system that is no longer written to ? no one wants to use 100GB just to make the alert color green. This gets silly very quickly at even ?medium? sizes... What would be really cool would be to decide how much reserved space you want, look at the velocity[1] with which the filesystem is filling, and then alert when the time until left until full is less than an operations shift (or something useful). Maybe that is just silly. Hey, Jan-Frode, can you mount the no-more-data file systems read only? Maybe not alerting on the full-ness of read only filesystems is easier and more sane? I mean, they can?t fill up any more. Liberty, -- Stephen On Oct 26, 2016, at 11:14 AM, Stefan Schmidt > wrote: Hi Jan-Frode Myklebust, The current GUI-internal logic has a hard coded warning level of 80% and an hard-coded error level of 90%. With 4.2.2 the system health monitoring will have thresholds set and monitored via ZIMON and the system health component will provide events.With 4.2.2 those can be changed with an mm command. In general it is not a good idea to let a filesystem get loaded 100%. It could be, if this is done with the filesystem used by the system to store some configuration data, that the system can run into an error (usually this is the filesystem we call gpfs0). I suggest you get the 4.2.2 version which will GA quite soon. I have to apologize that I'm not allowed to talk about GA dates. If you want to get this feature earlier you may contact IBM and ask for participation on the beta program which starts very soon but the beta code is usually limited to non productive systems. As said , ask your IBM contact for the 4.2.2 GA update ( hopefully GA is soon) and you can get this feature you are looking for. Mit freundlichen Gr??en / Kind regards Stefan Schmidt Scrum Master IBM Spectrum Scale GUI / Senior IT Architect/PMP?-Dept. M069 / IBM Spectrum Scale Software Development IBM Systems Group IBM Deutschland ________________________________ Phone: +49-703 4274 1966 IBM Deutschland Am Weiher 24 E-Mail: stschmid at de.ibm.com 65421 Kelsterbach Germany ________________________________ IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: Jan-Frode Myklebust > To: gpfsug main discussion list > Date: 26.10.2016 13:53 Subject: [gpfsug-discuss] filesystem thresholds in gui alerting Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Does anybody know if there are any way to define what thresholds are to be used for alterting in the GUI? F.ex. we have some filesystems that are very full, but won't be getting any more data added.. we'd like to turn off monitoring of these, are raise the threshold to allow them to be ~100% full. -jf_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From stijn.deweirdt at ugent.be Thu Oct 27 18:27:44 2016 From: stijn.deweirdt at ugent.be (Stijn De Weirdt) Date: Thu, 27 Oct 2016 19:27:44 +0200 Subject: [gpfsug-discuss] workaround gpfs 4.2.1-0 rpm issue Message-ID: <86d24598-7af8-38f0-fe80-998294c7bef9@ugent.be> hi all, gpfs.base 4.2.1-0 rpm has following postuninstall snippet it will disable the gpfs unit always (when previously enabled), whether this is a removal or an upgrade. this however prevents an update to 4.2.1-1, as it will remove the unit that is added during install (post has 'systemctl reenable /usr/lpp/mmfs/lib/systemd/gpfs.service') so after the upgrade, we are left with nodes that have no gpfs service unit (and thus no gpfs). it would have been better if the rpm symlinked teh service to the /usr/lib/systemd/... units, and enabled/disabled it. i'll probably rpmrebuild the 4.2.1-1 rpms to make a more permanent unit in a proper system location. other tips are welcome. stijn > postuninstall scriptlet (using /bin/sh): > if test -n "$DEBUG" || test -n "$DEBUGpostun"; then > set -x > fi > packageCnt=$1 > debian_release="/etc/debian_version" > > # set the system utilities if they are in different path on different systems > if [ -f "$debian_release" ] > then > AWK=/usr/bin/awk > else > AWK=/bin/awk > fi > > if /usr/bin/systemctl -q is-enabled gpfs.service 2>/dev/null > then > /usr/bin/systemctl -q disable gpfs.service > fi > From douglasof at us.ibm.com Fri Oct 28 14:37:53 2016 From: douglasof at us.ibm.com (Douglas O'flaherty) Date: Fri, 28 Oct 2016 09:37:53 -0400 Subject: [gpfsug-discuss] SC16 Registration Message-ID: Greetings: If you haven't yet registered, planned a 1:1 meeting, or interested in a specific topic ... Here you go https://www-01.ibm.com/events/wwe/grp/grp305.nsf/Registration.xsp?openform&seminar=357M7UES&locale=en_US&auth=anonymous Sign up so we can plan the cocktail hour on Sunday evening -- Thanks to NetApp for sponsoring again. doug IBM Spectrum Storage PMM -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mark.Bush at siriuscom.com Fri Oct 28 14:52:47 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Fri, 28 Oct 2016 13:52:47 +0000 Subject: [gpfsug-discuss] GUI and pmsensors/pmcollector Message-ID: <9566FC70-85D3-4D9B-B28B-58C4535B8972@siriuscom.com> I am able to do everything I need just fine when I have a single node cluster and both pmsensors and pmcollector are installed all on the same node. When I try and get pmsensors on my NSD nodes and a separate pmcollector elsewhere I never can get anything to show up in the graphs. I?ve configured the pmsensors with mmperfmon config generate ?collectors=(my node) and I still get nothing. The logs show that I get connections but the GUI never show and I?m unable to get mmperfmon query to work either as it fails saying it can?t find the collector. Anyone had this much trouble just getting this to work? Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From stschmid at de.ibm.com Fri Oct 28 14:59:02 2016 From: stschmid at de.ibm.com (Stefan Schmidt) Date: Fri, 28 Oct 2016 15:59:02 +0200 Subject: [gpfsug-discuss] GUI and pmsensors/pmcollector In-Reply-To: <9566FC70-85D3-4D9B-B28B-58C4535B8972@siriuscom.com> References: <9566FC70-85D3-4D9B-B28B-58C4535B8972@siriuscom.com> Message-ID: Hi, the PMCollector must be installed on the same node as the GUI. The collectors can be installed on any node you want to monitor. Hope that helps. Mit freundlichen Gr??en / Kind regards Stefan Schmidt Scrum Master IBM Spectrum Scale GUI / Senior IT Architect/PMP?-Dept. M069 / IBM Spectrum Scale Software Development IBM Systems Group IBM Deutschland Phone: +49-703 4274 1966 IBM Deutschland Am Weiher 24 E-Mail: stschmid at de.ibm.com 65421 Kelsterbach Germany IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Mark.Bush at siriuscom.com" To: "gpfsug-discuss at spectrumscale.org" Date: 28.10.2016 15:53 Subject: [gpfsug-discuss] GUI and pmsensors/pmcollector Sent by: gpfsug-discuss-bounces at spectrumscale.org I am able to do everything I need just fine when I have a single node cluster and both pmsensors and pmcollector are installed all on the same node. When I try and get pmsensors on my NSD nodes and a separate pmcollector elsewhere I never can get anything to show up in the graphs. I?ve configured the pmsensors with mmperfmon config generate ?collectors=(my node) and I still get nothing. The logs show that I get connections but the GUI never show and I?m unable to get mmperfmon query to work either as it fails saying it can?t find the collector. Anyone had this much trouble just getting this to work? Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mark.Bush at siriuscom.com Fri Oct 28 15:12:25 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Fri, 28 Oct 2016 14:12:25 +0000 Subject: [gpfsug-discuss] GUI and pmsensors/pmcollector In-Reply-To: References: <9566FC70-85D3-4D9B-B28B-58C4535B8972@siriuscom.com> Message-ID: <66CEE4CD-BD01-4F50-839A-529062DC38D3@siriuscom.com> Perhaps I needed more description I have a 3 node cluster 2 NDS?s (with tiebreaker) 1 gui node The NDS?s all have pmsensors installed The gui node had pmcollector installed I?ve modified the /opt/IBM/zimon/ZIMSensors.cfg file on the NSD?s to point to my gui node Systemctl start pmsensors on NSD?s Systemclt start pmcollector on gui Systemctl start gpfsgui on gui The log even shows connection from the two NSD?s but for some reason the GUI always has a red x in the performance graphs and claims it can?t connect to the collector (which is running on the same node). Not sure what I?m missing here. It all works fine when it?s all on one node. Mark From: on behalf of Stefan Schmidt Reply-To: gpfsug main discussion list Date: Friday, October 28, 2016 at 8:59 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GUI and pmsensors/pmcollector Hi, the PMCollector must be installed on the same node as the GUI. The collectors can be installed on any node you want to monitor. Hope that helps. Mit freundlichen Gr??en / Kind regards Stefan Schmidt Scrum Master IBM Spectrum Scale GUI / Senior IT Architect/PMP?-Dept. M069 / IBM Spectrum Scale Software Development IBM Systems Group IBM Deutschland ________________________________ Phone: +49-703 4274 1966 IBM Deutschland Am Weiher 24 E-Mail: stschmid at de.ibm.com 65421 Kelsterbach Germany ________________________________ IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Mark.Bush at siriuscom.com" To: "gpfsug-discuss at spectrumscale.org" Date: 28.10.2016 15:53 Subject: [gpfsug-discuss] GUI and pmsensors/pmcollector Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ I am able to do everything I need just fine when I have a single node cluster and both pmsensors and pmcollector are installed all on the same node. When I try and get pmsensors on my NSD nodes and a separate pmcollector elsewhere I never can get anything to show up in the graphs. I?ve configured the pmsensors with mmperfmon config generate ?collectors=(my node) and I still get nothing. The logs show that I get connections but the GUI never show and I?m unable to get mmperfmon query to work either as it fails saying it can?t find the collector. Anyone had this much trouble just getting this to work? Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From akers at vt.edu Fri Oct 28 15:22:49 2016 From: akers at vt.edu (Joshua Akers) Date: Fri, 28 Oct 2016 14:22:49 +0000 Subject: [gpfsug-discuss] GPFS Log Levels Message-ID: Hi all, I am trying to find more information on GPFS log levels. Here is what I have so far: [D] - Detail info [I] - Info [N] - Notice [W] - Warning [E] - Error [X] - Critical Error [A] - Deadlock related? Any corrections or additional information would be greatly appreciated. Thanks, Josh -- *Joshua D. Akers* *HPC Systems Specialist* NI&S Systems Support (MC0214) 1700 Pratt Drive Blacksburg, VA 24061 540-231-9506 -------------- next part -------------- An HTML attachment was scrubbed... URL: From billowen at us.ibm.com Fri Oct 28 15:37:27 2016 From: billowen at us.ibm.com (Bill Owen) Date: Fri, 28 Oct 2016 07:37:27 -0700 Subject: [gpfsug-discuss] unified file and object In-Reply-To: References: Message-ID: Hi Leslie, For the issues you list below: 1. AD configuration - there is a known issue with quoted names passed to mmuserauth - we are investigating how to correct this. 2. Can you provide more details on how you configured file access? The normal procedure is to use "mmobj file-access enable", and this will set up the required settings in the config file. Can you send us: - the steps used to configure file access - the resulting /etc/swift/object-server-sof.conf - log files from /var/log/swift or output of "systemctl status openstack-swift-object-sof" We can schedule a short call to help debug if needed. Here is a more detailed description of enabling file access to object data: Enable unified file and object access See: http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.1/com.ibm.spectrum.scale.v4r21.doc/bl1adm_enablefileaccess.htm 1. enable & configure: # mmobj file-access enable # mmobj config list --ccrfile spectrum-scale-object.conf --section capabilities --property file-access-enabled file-access-enabled = true Different user id mapping approaches are supporting. The simplest is id_mgmt = local_mode. This example leaves that default in place. Verify id_mgmt mode: # mmobj config list --ccrfile object-server-sof.conf --section DEFAULT --property id_mgmt id_mgmt = local_mode At this point, you can create a storage policy that will be used for file & object access containers. We'll create a new fileset for this step (in 4.2.2 we will support objectizing existing filesets). In this example, we create the fileset with 1M inodes initially allocated: # mmobj policy create fileaccess --enable-file-access -i 1000000 [I] Getting latest configuration from ccr [I] Creating fileset fs2-1m-03:obj_fileaccess [I] Creating new unique index and building the object rings [I] Updating the configuration [I] Uploading the changed configuration The new storage policy is listed: # mmobj policy list Index Name Default Deprecated Fileset Functions -------------------------------------------------------------------------------- 0 policy-0 yes Object_Fileset default 87811608170 fileaccess obj_fileaccess file-and-object-access The new fileset called obj_fileaccess is shown below: # mmlsfileset fs2-1m-03 obj_fileaccess -L Filesets in file system 'fs2-1m-03': Name Id RootInode ParentId Created InodeSpace MaxInodes AllocInodes Comment obj_fileaccess 6 134217731 0 Wed Aug 17 10:08:51 2016 4 1000192 1000192 2. Create one or more containers that are file-access enabled: Next, you can create one or more containers that use this storage policy (and have file access enabled): # source openrc # swift post fileaccess_container --header "x-storage-policy: fileaccess" # swift stat fileaccess_container Account: AUTH_acad33df8cdf402ebab6bfb87b1674af Container: fileaccess_container Objects: 0 Bytes: 0 Read ACL: Write ACL: Sync To: Sync Key: Accept-Ranges: bytes X-Storage-Policy: fileaccess X-Timestamp: 1471456287.21609 X-Trans-Id: tx0f305353a20a4ac1ab927-0057b4a428 Content-Type: text/plain; charset=utf-8 Objects can be added to the container from the object or file interface, the file path is structured like this: ///// In our example it would be: /ibm/fs2-1m-03/obj_fileaccess/s87811608170z1device1/AUTH_acad33df8cdf402ebab6bfb87b1674af/fileaccess_container/ For convenience, create a soft link: ln -s /ibm/fs2-1m-03/obj_fileaccess/s87811608170z1device1/AUTH_acad33df8cdf402ebab6bfb87b1674af/fileaccess_container/ /ibm/fs2-1m-03/unified_file_object 3. Verify access from both file & object interface: Upload a file from object interface, it will be readable from file interface, owned by swift user. For example: # date > test1.obj # swift upload fileaccess_container test1.obj test1.obj # ls -l /ibm/fs2-1m-03/unified_file_object/ total 0 -rwxr-xr-x 1 swift swift 29 Aug 17 11:03 test1.obj Similarly, create an example object (or set of objects) from file interface: # date > /ibm/fs2-1m-03/unified_file_object/test2.obj The object must be readable by swift user - either by setting file ownership or using file acls: # chown swift:swift /ibm/fs2-1m-03/unified_file_object/test2.obj # ls -l /ibm/fs2-1m-03/unified_file_object/ -rwxr-xr-x 1 swift swift 29 Aug 17 11:03 test1.obj -rw-r--r-- 1 swift swift 29 Aug 17 11:05 test2.obj New files will be "objectized" (made visible to the object interface) by a periodic process. The default period for this is 30 min, but can be changed using mmob config change command. This process can be resource intensive, so don't see to run too often if there are many containers & files. # mmobj config list --ccrfile spectrum-scale-objectizer.conf --section DEFAULT --property objectization_interval objectization_interval = 1800 After waiting the objectizer interval, the new object shows up from the object interface, and can be accessed like any other object: # swift list fileaccess_container test1.obj test2.obj # swift download fileaccess_container test2.obj test2.obj [auth 0.229s, headers 0.862s, total 0.862s, 0.000 MB/s] Verify the contents match # cat test2.obj Wed Aug 17 11:05:00 PDT 2016 # cat /ibm/fs2-1m-03/unified_file_object/test2.obj Wed Aug 17 11:05:00 PDT 2016 And update from file interface: # echo "bill was here" >> /ibm/fs2-1m-03/unified_file_object/test2.obj Redownload from object interface and confirm: # swift download fileaccess_container test2.obj test2.obj [auth 0.230s, headers 0.729s, total 0.729s, 0.000 MB/s] [ # cat test2.obj Wed Aug 17 11:05:00 PDT 2016 bill was here Use Case Example: One usecase is adding/expanding an archive file in file interface, and results can be accessed from file interface. For example, create a set of test files: # mkdir test [root at client22 ~]# for i in {1..10}; do date > test/smallobj$i ; done [root at client22 ~]# ll test total 40 -rw-r--r-- 1 root root 29 Aug 17 11:31 smallobj1 -rw-r--r-- 1 root root 29 Aug 17 11:31 smallobj10 -rw-r--r-- 1 root root 29 Aug 17 11:31 smallobj2 -rw-r--r-- 1 root root 29 Aug 17 11:31 smallobj3 -rw-r--r-- 1 root root 29 Aug 17 11:31 smallobj4 -rw-r--r-- 1 root root 29 Aug 17 11:31 smallobj5 -rw-r--r-- 1 root root 29 Aug 17 11:31 smallobj6 -rw-r--r-- 1 root root 29 Aug 17 11:31 smallobj7 -rw-r--r-- 1 root root 29 Aug 17 11:31 smallobj8 -rw-r--r-- 1 root root 29 Aug 17 11:31 smallobj9 # tar -cf databag.tar test Expand the file into file interface and set ownership to swift: # cd /ibm/fs2-1m-03/unified_file_object/ # tar -xf ~/databag.tar # chown -R swift:swift test After objectizer runs, all files are available from object interface: # swift list fileaccess_container test/smallobj1 test/smallobj10 test/smallobj2 test/smallobj3 test/smallobj4 test/smallobj5 test/smallobj6 test/smallobj7 test/smallobj8 test/smallobj9 test1.obj test2.obj Regards, Bill Owen billowen at us.ibm.com Spectrum Scale Object Storage 520-799-4829 From: leslie elliott To: gpfsug-discuss at spectrumscale.org Date: 10/25/2016 03:01 PM Subject: [gpfsug-discuss] unified file and object Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi We are in the process of trying to configure unified file and object storage in unified_mode and have a few problems We are running 4.2.1 and do not have any current issues the file access protocols setting up the authentication First issue we do have is binding the object data to our Active Directory, we seem to be hitting a road block due to the fact that the bind DN has spaces in it, if we enclose the DN in quotes it still fails, if we escape them with the appropriate RFC ?value we can get the mmuserauth to complete but the lookups from the local keystone fail for the authentication of the users The DN for the swift user and swift admin also have quotes in them, so just doing it on the command line is not enough to get the mmuserauth command to complete Second problem is OBJ:openstack-swift-object-sof ? ? ? ? ? is not running This seems to be due to the config file not having ?bind_ip and bind_port values, if these are added then the error turns to pipeline of other settings in the config file missing This particular issue occurs no matter what the auth type is set to be for object Hopefully this make some sense to someone Thanks leslie Leslie Elliott, Infrastructure Support Specialist, ?Faculty Infrastructure and Applications Support Information Technology Services, The University of Queensland _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From taylorm at us.ibm.com Fri Oct 28 15:56:30 2016 From: taylorm at us.ibm.com (Michael L Taylor) Date: Fri, 28 Oct 2016 07:56:30 -0700 Subject: [gpfsug-discuss] GUI and pmsensors/pmcollector In-Reply-To: References: Message-ID: Have a quick read of this link and see if it helps: https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.1/com.ibm.spectrum.scale.v4r21.doc/bl1ins_manualinstallofgui.htm From: gpfsug-discuss-request at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Date: 10/28/2016 07:23 AM Subject: gpfsug-discuss Digest, Vol 57, Issue 76 Sent by: gpfsug-discuss-bounces at spectrumscale.org Message: 1 Date: Fri, 28 Oct 2016 14:12:25 +0000 From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GUI and pmsensors/pmcollector Message-ID: <66CEE4CD-BD01-4F50-839A-529062DC38D3 at siriuscom.com> Content-Type: text/plain; charset="utf-8" Perhaps I needed more description I have a 3 node cluster 2 NDS?s (with tiebreaker) 1 gui node The NDS?s all have pmsensors installed The gui node had pmcollector installed I?ve modified the /opt/IBM/zimon/ZIMSensors.cfg file on the NSD?s to point to my gui node Systemctl start pmsensors on NSD?s Systemclt start pmcollector on gui Systemctl start gpfsgui on gui The log even shows connection from the two NSD?s but for some reason the GUI always has a red x in the performance graphs and claims it can?t connect to the collector (which is running on the same node). Not sure what I?m missing here. It all works fine when it?s all on one node. Mark From: on behalf of Stefan Schmidt Reply-To: gpfsug main discussion list Date: Friday, October 28, 2016 at 8:59 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GUI and pmsensors/pmcollector Hi, the PMCollector must be installed on the same node as the GUI. The collectors can be installed on any node you want to monitor. Hope that helps. Mit freundlichen Gr??en / Kind regards Stefan Schmidt Scrum Master IBM Spectrum Scale GUI / Senior IT Architect/PMP?-Dept. M069 / IBM Spectrum Scale Software Development IBM Systems Group IBM Deutschland ________________________________ Phone: +49-703 4274 1966 IBM Deutschland Am Weiher 24 E-Mail: stschmid at de.ibm.com 65421 Kelsterbach Germany ________________________________ IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Mark.Bush at siriuscom.com" To: "gpfsug-discuss at spectrumscale.org" Date: 28.10.2016 15:53 Subject: [gpfsug-discuss] GUI and pmsensors/pmcollector Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ I am able to do everything I need just fine when I have a single node cluster and both pmsensors and pmcollector are installed all on the same node. When I try and get pmsensors on my NSD nodes and a separate pmcollector elsewhere I never can get anything to show up in the graphs. I?ve configured the pmsensors with mmperfmon config generate ?collectors= (my node) and I still get nothing. The logs show that I get connections but the GUI never show and I?m unable to get mmperfmon query to work either as it fails saying it can?t find the collector. Anyone had this much trouble just getting this to work? Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161028/b4552796/attachment-0001.html > ------------------------------ Message: 2 Date: Fri, 28 Oct 2016 14:22:49 +0000 From: Joshua Akers To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] GPFS Log Levels Message-ID: Content-Type: text/plain; charset="utf-8" Hi all, I am trying to find more information on GPFS log levels. Here is what I have so far: [D] - Detail info [I] - Info [N] - Notice [W] - Warning [E] - Error [X] - Critical Error [A] - Deadlock related? Any corrections or additional information would be greatly appreciated. Thanks, Josh -- *Joshua D. Akers* *HPC Systems Specialist* NI&S Systems Support (MC0214) 1700 Pratt Drive Blacksburg, VA 24061 540-231-9506 -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161028/7ebac661/attachment.html > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 76 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: From rohwedder at de.ibm.com Fri Oct 28 16:08:17 2016 From: rohwedder at de.ibm.com (Markus Rohwedder) Date: Fri, 28 Oct 2016 17:08:17 +0200 Subject: [gpfsug-discuss] GUI and pmsensors/pmcollector In-Reply-To: References: Message-ID: Hi, Some more questions and things to look for: 1. Are there several collectors running, potentially at different code levels? All collectors should run at the same code level. Simplest case, there is only one collector. Sensors and collectors can have different code levels. 2. Is the sensor config updated manually or per mmperfmon config update? Manual update might be overwritten if the system thinks that the config should be distr?buted automatically. See Knowledge center and mmperfmon CLI command for info. 3. How does the sensor config look like (output of mmperfmon config show)? 4. Are only NSD charts showing that there is no data, or all charts? Please note; In a SAN environment (Every node sees every NSD), there is no NSD server side data reported. GPFS is smart and knows to use the local device and circumvents the NSD code stack. However, we dont get notified on traffic to the disks. 5. Which code level? Thanks, Mit freundlichen Gr??en / Kind regards Dr. Markus Rohwedder Spectrum Scale GUI Development Phone: +49 7034 6430190 IBM Deutschland E-Mail: rohwedder at de.ibm.com Am Weiher 24 65451 Kelsterbach Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Martina K?deritz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: gpfsug-discuss-request at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Date: 28.10.2016 16:23 Subject: gpfsug-discuss Digest, Vol 57, Issue 76 Sent by: gpfsug-discuss-bounces at spectrumscale.org Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: GUI and pmsensors/pmcollector (Mark.Bush at siriuscom.com) 2. GPFS Log Levels (Joshua Akers) ---------------------------------------------------------------------- Message: 1 Date: Fri, 28 Oct 2016 14:12:25 +0000 From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GUI and pmsensors/pmcollector Message-ID: <66CEE4CD-BD01-4F50-839A-529062DC38D3 at siriuscom.com> Content-Type: text/plain; charset="utf-8" Perhaps I needed more description I have a 3 node cluster 2 NDS?s (with tiebreaker) 1 gui node The NDS?s all have pmsensors installed The gui node had pmcollector installed I?ve modified the /opt/IBM/zimon/ZIMSensors.cfg file on the NSD?s to point to my gui node Systemctl start pmsensors on NSD?s Systemclt start pmcollector on gui Systemctl start gpfsgui on gui The log even shows connection from the two NSD?s but for some reason the GUI always has a red x in the performance graphs and claims it can?t connect to the collector (which is running on the same node). Not sure what I?m missing here. It all works fine when it?s all on one node. Mark From: on behalf of Stefan Schmidt Reply-To: gpfsug main discussion list Date: Friday, October 28, 2016 at 8:59 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GUI and pmsensors/pmcollector Hi, the PMCollector must be installed on the same node as the GUI. The collectors can be installed on any node you want to monitor. Hope that helps. Mit freundlichen Gr??en / Kind regards Stefan Schmidt Scrum Master IBM Spectrum Scale GUI / Senior IT Architect/PMP?-Dept. M069 / IBM Spectrum Scale Software Development IBM Systems Group IBM Deutschland ________________________________ Phone: +49-703 4274 1966 IBM Deutschland Am Weiher 24 E-Mail: stschmid at de.ibm.com 65421 Kelsterbach Germany ________________________________ IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Mark.Bush at siriuscom.com" To: "gpfsug-discuss at spectrumscale.org" Date: 28.10.2016 15:53 Subject: [gpfsug-discuss] GUI and pmsensors/pmcollector Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ I am able to do everything I need just fine when I have a single node cluster and both pmsensors and pmcollector are installed all on the same node. When I try and get pmsensors on my NSD nodes and a separate pmcollector elsewhere I never can get anything to show up in the graphs. I?ve configured the pmsensors with mmperfmon config generate ?collectors= (my node) and I still get nothing. The logs show that I get connections but the GUI never show and I?m unable to get mmperfmon query to work either as it fails saying it can?t find the collector. Anyone had this much trouble just getting this to work? Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161028/b4552796/attachment-0001.html > ------------------------------ Message: 2 Date: Fri, 28 Oct 2016 14:22:49 +0000 From: Joshua Akers To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] GPFS Log Levels Message-ID: Content-Type: text/plain; charset="utf-8" Hi all, I am trying to find more information on GPFS log levels. Here is what I have so far: [D] - Detail info [I] - Info [N] - Notice [W] - Warning [E] - Error [X] - Critical Error [A] - Deadlock related? Any corrections or additional information would be greatly appreciated. Thanks, Josh -- *Joshua D. Akers* *HPC Systems Specialist* NI&S Systems Support (MC0214) 1700 Pratt Drive Blacksburg, VA 24061 540-231-9506 -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161028/7ebac661/attachment.html > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 76 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecblank.gif Type: image/gif Size: 45 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 06388797.gif Type: image/gif Size: 1851 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From mweil at wustl.edu Fri Oct 28 21:49:37 2016 From: mweil at wustl.edu (Matt Weil) Date: Fri, 28 Oct 2016 15:49:37 -0500 Subject: [gpfsug-discuss] Leave protocol detail info Message-ID: anybody no what this means? Wed Oct 26 20:08:29.619 2016: [D] Leave protocol detail info: LA: 75 LFLG: 24409951 LFLG delta: 75 ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. From mweil at wustl.edu Fri Oct 28 22:02:59 2016 From: mweil at wustl.edu (Matt Weil) Date: Fri, 28 Oct 2016 16:02:59 -0500 Subject: [gpfsug-discuss] Leave protocol detail info In-Reply-To: References: Message-ID: Also is there any document that explains what happens to the cluster when a client node stops responding and get evicted. On 10/28/16 3:49 PM, Matt Weil wrote: > anybody no what this means? > > Wed Oct 26 20:08:29.619 2016: [D] Leave protocol detail info: LA: 75 > LFLG: 24409951 LFLG delta: 75 > ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. From leslie.james.elliott at gmail.com Sat Oct 29 11:53:19 2016 From: leslie.james.elliott at gmail.com (leslie elliott) Date: Sat, 29 Oct 2016 20:53:19 +1000 Subject: [gpfsug-discuss] unified file and object In-Reply-To: References: Message-ID: Bill to be clear the file access I mentioned was in relation to SMB and NFS using mmuserauth rather than the unification with the object store since it is required as well but I did try to do this for object as well using the Administration and Programming Reference from page 142, was using unified_mode rather than local_mode mmobj config change --ccrfile spectrum-scale-object.conf --section capabilities --property file-access-enabled --value true the mmuserauth failed as you are aware, we have created test accounts without spaces in the DN and were successful with this step, so eagerly await a fix to be able to use the correct accounts mmobj config change --ccrfile object-server-sof.conf --section DEFAULT --property id_mgmt --value unified_mode mmobj config change --ccrfile object-server-sof.conf --section DEFAULT --property ad_domain --value DOMAIN we have successfully tested object stores on this cluster with simple auth the output you asked for is as follows [root at pren-gs7k-vm4 ~]# cat /etc/swift/object-server-sof.conf [DEFAULT] devices = /gpfs/pren01/ObjectFileset/o log_level = ERROR [root at pren-gs7k-vm4 ~]# systemctl -l status openstack-swift-object-sof ? openstack-swift-object-sof.service - OpenStack Object Storage (swift) - Object Server Loaded: loaded (/usr/lib/systemd/system/openstack-swift-object-sof.service; disabled; vendor preset: disabled) Active: failed (Result: exit-code) since Sat 2016-10-29 10:30:22 UTC; 27s ago Process: 8086 ExecStart=/usr/bin/swift-object-server-sof /etc/swift/object-server-sof.conf (code=exited, status=1/FAILURE) Main PID: 8086 (code=exited, status=1/FAILURE) Oct 29 10:30:22 pren-gs7k-vm4 systemd[1]: Started OpenStack Object Storage (swift) - Object Server. Oct 29 10:30:22 pren-gs7k-vm4 systemd[1]: Starting OpenStack Object Storage (swift) - Object Server... Oct 29 10:30:22 pren-gs7k-vm4 swift-object-server-sof[8086]: Error trying to load config from /etc/swift/object-server-sof.conf: No section 'object-server' (prefixed by 'app' or 'application' or 'composite' or 'composit' or 'pipeline' or 'filter-app') found in config /etc/swift/object-server-sof.conf Oct 29 10:30:22 pren-gs7k-vm4 systemd[1]: openstack-swift-object-sof.service: main process exited, code=exited, status=1/FAILURE Oct 29 10:30:22 pren-gs7k-vm4 systemd[1]: Unit openstack-swift-object-sof.service entered failed state. Oct 29 10:30:22 pren-gs7k-vm4 systemd[1]: openstack-swift-object-sof.service failed. I am happy to help you or for you to help to debug this problem via a short call thanks leslie On 29 October 2016 at 00:37, Bill Owen wrote: > > 2. Can you provide more details on how you configured file access? The > normal procedure is to use "mmobj file-access enable", and this will set up > the required settings in the config file. Can you send us: > - the steps used to configure file access > - the resulting /etc/swift/object-server-sof.conf > - log files from /var/log/swift or output of "systemctl status > openstack-swift-object-sof" > > We can schedule a short call to help debug if needed. > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Greg.Lehmann at csiro.au Mon Oct 31 07:03:35 2016 From: Greg.Lehmann at csiro.au (Greg.Lehmann at csiro.au) Date: Mon, 31 Oct 2016 07:03:35 +0000 Subject: [gpfsug-discuss] Two new whitepapers published In-Reply-To: References: Message-ID: <33bda80b2ba24bcb996819dbc30f1d75@exch1-cdc.nexus.csiro.au> Thanks Yuri. These were great. I'm not trying to be impertinent, but I have one suggestion - If you can find the time, add some diagrams to help readers visualise the various data structures and layouts. I am thinking along the lines of what was in "The Magic Garden Explained" and "The Design and Implementation of the 4.3BSD UNIX Operating System" where they talked about their filesystems. Cheers, Greg From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Yuri L Volobuev Sent: Friday, 21 October 2016 11:59 AM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Two new whitepapers published Esteemed GPFS and Spectrum Scale users, For your reading enjoyment, two new whitepapers have been posted to the Spectrum Scale Wiki: Spectrum Scale: Replication in GPFS Spectrum Scale: GPFS Metadata The URL for the parent page is https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/White%20Papers%20%26%20Media The two .pdf documents are accessible through the Attachment section at the bottom of the page. Unfortunately, dW "spam prevention engine" does a very good job preventing me from "spamming" the page to actually add links. Best regards, Yuri -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Mon Oct 31 09:53:38 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 31 Oct 2016 09:53:38 +0000 Subject: [gpfsug-discuss] Recent Whitepapers from Yuri Volobuev Message-ID: For those of you who may not know, Yuri Volobuev has left IBM to pursue new challenges. Myself along with many others received so much help and keen insight from Yuri on all things GPFS. He will be missed. Bob Oesterlin Sr Principal Storage Engineer, Nuance -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.knister at gmail.com Mon Oct 31 16:19:11 2016 From: aaron.knister at gmail.com (Aaron Knister) Date: Mon, 31 Oct 2016 12:19:11 -0400 Subject: [gpfsug-discuss] Recent Whitepapers from Yuri Volobuev In-Reply-To: References: Message-ID: <9C6B34F6-2849-4B77-9E8B-3806D634C281@gmail.com> Wow! That's a real shame. His ability to articulate his immense knowledge of GPFS was truly impressive. Sent from my iPhone > On Oct 31, 2016, at 5:53 AM, Oesterlin, Robert wrote: > > For those of you who may not know, Yuri Volobuev has left IBM to pursue new challenges. Myself along with many others received so much help and keen insight from Yuri on all things GPFS. > > He will be missed. > > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.wonderley at vt.edu Mon Oct 31 16:52:43 2016 From: eric.wonderley at vt.edu (J. Eric Wonderley) Date: Mon, 31 Oct 2016 12:52:43 -0400 Subject: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size Message-ID: I wanted to do something like this... [root at cl001 ~]# cat /opt/gpfs/home.ply /*Failsafe migration of old small files back to spinning media pool(fc_8T) */ RULE 'theshold' MIGRATE FROM POOL 'system' THRESHOLD(90,70) WEIGHT(ACCESS_TIME) TO POOL 'fc_8T' /*Write files larger than 16MB to pool called "fc_8T" */ RULE 'bigfiles' SET POOL 'fc_8T' WHERE FILE_SIZE>16777216 /*Move anything else to system pool */ RULE 'default' SET POOL 'system' Apparently there is no happiness using FILE_SIZE in a placement policy: [root at cl001 ~]# mmchpolicy home /opt/gpfs/home.ply Error while validating policy `home.ply': rc=22: PCSQLERR: 'FILE_SIZE' is an unsupported or unknown attribute or variable name in this context. PCSQLCTX: at line 4 of 6: RULE 'bigfiles' SET POOL 'fc_8T' WHERE {{{FILE_SIZE}}}>16777216 runRemoteCommand_v2: cl002.cl.arc.internal: tschpolicy /dev/home /var/mmfs/tmp/tspolicyFile.mmchpolicy.113372 -t home.ply failed. mmchpolicy: Command failed. Examine previous error messages to determine cause. Can anyone suggest a way to accomplish this using policy? -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Mon Oct 31 17:09:55 2016 From: bbanister at jumptrading.com (Bryan Banister) Date: Mon, 31 Oct 2016 17:09:55 +0000 Subject: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size In-Reply-To: References: Message-ID: <21BC488F0AEA2245B2C3E83FC0B33DBB063A1D4A@CHI-EXCHANGEW1.w2k.jumptrading.com> The File Placement Policy that you are trying to set cannot use the size of the file to determine the placement of the file in a GPFS Storage Pool. This is because GPFS has no idea what the file size will be when the file is open()?d for writing. Hope that helps! -Bryan PS. I really wish that we could use a path for specifying data placement in a GPFS Pool, and not just the file name, owner, etc. I?ll submit a RFE for this. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of J. Eric Wonderley Sent: Monday, October 31, 2016 11:53 AM To: gpfsug main discussion list Subject: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size I wanted to do something like this... [root at cl001 ~]# cat /opt/gpfs/home.ply /*Failsafe migration of old small files back to spinning media pool(fc_8T) */ RULE 'theshold' MIGRATE FROM POOL 'system' THRESHOLD(90,70) WEIGHT(ACCESS_TIME) TO POOL 'fc_8T' /*Write files larger than 16MB to pool called "fc_8T" */ RULE 'bigfiles' SET POOL 'fc_8T' WHERE FILE_SIZE>16777216 /*Move anything else to system pool */ RULE 'default' SET POOL 'system' Apparently there is no happiness using FILE_SIZE in a placement policy: [root at cl001 ~]# mmchpolicy home /opt/gpfs/home.ply Error while validating policy `home.ply': rc=22: PCSQLERR: 'FILE_SIZE' is an unsupported or unknown attribute or variable name in this context. PCSQLCTX: at line 4 of 6: RULE 'bigfiles' SET POOL 'fc_8T' WHERE {{{FILE_SIZE}}}>16777216 runRemoteCommand_v2: cl002.cl.arc.internal: tschpolicy /dev/home /var/mmfs/tmp/tspolicyFile.mmchpolicy.113372 -t home.ply failed. mmchpolicy: Command failed. Examine previous error messages to determine cause. Can anyone suggest a way to accomplish this using policy? ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jez.tucker at gpfsug.org Mon Oct 31 17:20:30 2016 From: jez.tucker at gpfsug.org (Jez Tucker) Date: Mon, 31 Oct 2016 17:20:30 +0000 Subject: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size In-Reply-To: <21BC488F0AEA2245B2C3E83FC0B33DBB063A1D4A@CHI-EXCHANGEW1.w2k.jumptrading.com> References: <21BC488F0AEA2245B2C3E83FC0B33DBB063A1D4A@CHI-EXCHANGEW1.w2k.jumptrading.com> Message-ID: Hey Bryan There was a previous RFE for path placement from the UG, but Yuri told me this was not techically possible as an inode has no knowledge about the parent dentry. (IIRC). You can see this in effect in the C API. It is possible to work this out at kernel level, but it's so costly that it becomes non-viable at scale / performance. IBMers please chip in and expand if you will. Jez On 31/10/16 17:09, Bryan Banister wrote: > > The File Placement Policy that you are trying to set cannot use the > size of the file to determine the placement of the file in a GPFS > Storage Pool. This is because GPFS has no idea what the file size > will be when the file is open()?d for writing. > > Hope that helps! > > -Bryan > > PS. I really wish that we could use a path for specifying data > placement in a GPFS Pool, and not just the file name, owner, etc. > I?ll submit a RFE for this. > > *From:*gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of *J. > Eric Wonderley > *Sent:* Monday, October 31, 2016 11:53 AM > *To:* gpfsug main discussion list > *Subject:* [gpfsug-discuss] wanted...gpfs policy that places larger > files onto a pool based on size > > I wanted to do something like this... > > > [root at cl001 ~]# cat /opt/gpfs/home.ply > /*Failsafe migration of old small files back to spinning media > pool(fc_8T) */ > RULE 'theshold' MIGRATE FROM POOL 'system' THRESHOLD(90,70) > WEIGHT(ACCESS_TIME) TO POOL 'fc_8T' > /*Write files larger than 16MB to pool called "fc_8T" */ > RULE 'bigfiles' SET POOL 'fc_8T' WHERE FILE_SIZE>16777216 > /*Move anything else to system pool */ > RULE 'default' SET POOL 'system' > > Apparently there is no happiness using FILE_SIZE in a placement policy: > [root at cl001 ~]# mmchpolicy home /opt/gpfs/home.ply > Error while validating policy `home.ply': rc=22: > PCSQLERR: 'FILE_SIZE' is an unsupported or unknown attribute or > variable name in this context. > PCSQLCTX: at line 4 of 6: RULE 'bigfiles' SET POOL 'fc_8T' WHERE > {{{FILE_SIZE}}}>16777216 > runRemoteCommand_v2: cl002.cl.arc.internal: tschpolicy /dev/home > /var/mmfs/tmp/tspolicyFile.mmchpolicy.113372 -t home.ply failed. > mmchpolicy: Command failed. Examine previous error messages to > determine cause. > > Can anyone suggest a way to accomplish this using policy? > > > ------------------------------------------------------------------------ > > Note: This email is for the confidential use of the named addressee(s) > only and may contain proprietary, confidential or privileged > information. If you are not the intended recipient, you are hereby > notified that any review, dissemination or copying of this email is > strictly prohibited, and to please notify the sender immediately and > destroy this email and any attachments. Email transmission cannot be > guaranteed to be secure or error-free. The Company, therefore, does > not make any guarantees as to the completeness or accuracy of this > email or any attachments. This email is for informational purposes > only and does not constitute a recommendation, offer, request or > solicitation of any kind to buy, sell, subscribe, redeem or perform > any type of transaction of a financial product. > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From mimarsh2 at vt.edu Mon Oct 31 17:23:40 2016 From: mimarsh2 at vt.edu (Brian Marshall) Date: Mon, 31 Oct 2016 13:23:40 -0400 Subject: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size In-Reply-To: References: <21BC488F0AEA2245B2C3E83FC0B33DBB063A1D4A@CHI-EXCHANGEW1.w2k.jumptrading.com> Message-ID: When creating a "fast tier" storage pool in a filesystem is the normal case to create a placement policy that places all files in the fast tier and migrates out old and large files? Brian Marshall On Mon, Oct 31, 2016 at 1:20 PM, Jez Tucker wrote: > Hey Bryan > > There was a previous RFE for path placement from the UG, but Yuri told > me this was not techically possible as an inode has no knowledge about the > parent dentry. (IIRC). You can see this in effect in the C API. It is > possible to work this out at kernel level, but it's so costly that it > becomes non-viable at scale / performance. > > IBMers please chip in and expand if you will. > > Jez > > > On 31/10/16 17:09, Bryan Banister wrote: > > The File Placement Policy that you are trying to set cannot use the size > of the file to determine the placement of the file in a GPFS Storage Pool. > This is because GPFS has no idea what the file size will be when the file > is open()?d for writing. > > > > Hope that helps! > > -Bryan > > > > PS. I really wish that we could use a path for specifying data placement > in a GPFS Pool, and not just the file name, owner, etc. I?ll submit a RFE > for this. > > > > *From:* gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss- > bounces at spectrumscale.org ] *On > Behalf Of *J. Eric Wonderley > *Sent:* Monday, October 31, 2016 11:53 AM > *To:* gpfsug main discussion list > *Subject:* [gpfsug-discuss] wanted...gpfs policy that places larger files > onto a pool based on size > > > > I wanted to do something like this... > > > [root at cl001 ~]# cat /opt/gpfs/home.ply > /*Failsafe migration of old small files back to spinning media pool(fc_8T) > */ > RULE 'theshold' MIGRATE FROM POOL 'system' THRESHOLD(90,70) > WEIGHT(ACCESS_TIME) TO POOL 'fc_8T' > /*Write files larger than 16MB to pool called "fc_8T" */ > RULE 'bigfiles' SET POOL 'fc_8T' WHERE FILE_SIZE>16777216 > /*Move anything else to system pool */ > RULE 'default' SET POOL 'system' > > Apparently there is no happiness using FILE_SIZE in a placement policy: > [root at cl001 ~]# mmchpolicy home /opt/gpfs/home.ply > Error while validating policy `home.ply': rc=22: > PCSQLERR: 'FILE_SIZE' is an unsupported or unknown attribute or variable > name in this context. > PCSQLCTX: at line 4 of 6: RULE 'bigfiles' SET POOL 'fc_8T' WHERE > {{{FILE_SIZE}}}>16777216 > runRemoteCommand_v2: cl002.cl.arc.internal: tschpolicy /dev/home > /var/mmfs/tmp/tspolicyFile.mmchpolicy.113372 -t home.ply failed. > mmchpolicy: Command failed. Examine previous error messages to determine > cause. > > Can anyone suggest a way to accomplish this using policy? > > ------------------------------ > > Note: This email is for the confidential use of the named addressee(s) > only and may contain proprietary, confidential or privileged information. > If you are not the intended recipient, you are hereby notified that any > review, dissemination or copying of this email is strictly prohibited, and > to please notify the sender immediately and destroy this email and any > attachments. Email transmission cannot be guaranteed to be secure or > error-free. The Company, therefore, does not make any guarantees as to the > completeness or accuracy of this email or any attachments. This email is > for informational purposes only and does not constitute a recommendation, > offer, request or solicitation of any kind to buy, sell, subscribe, redeem > or perform any type of transaction of a financial product. > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ewahl at osc.edu Mon Oct 31 17:26:28 2016 From: ewahl at osc.edu (Edward Wahl) Date: Mon, 31 Oct 2016 13:26:28 -0400 Subject: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size In-Reply-To: <21BC488F0AEA2245B2C3E83FC0B33DBB063A1D4A@CHI-EXCHANGEW1.w2k.jumptrading.com> References: <21BC488F0AEA2245B2C3E83FC0B33DBB063A1D4A@CHI-EXCHANGEW1.w2k.jumptrading.com> Message-ID: <20161031132628.5c952a40@osc.edu> On Mon, 31 Oct 2016 17:09:55 +0000 Bryan Banister wrote: > -Bryan > ? > PS. I really wish that we could use a path for specifying data > placement in a GPFS Pool, and not just the file name, owner, etc. > I?ll submit a RFE for this. So... use a fileset with a linked directory and a fileset placement policy to a pool? Might be a bit more rigid for what you really want, and it would be messy, but it would work just fine. -- Ed Wahl Ohio Supercomputer Center 614-292-9302 From bbanister at jumptrading.com Mon Oct 31 17:29:05 2016 From: bbanister at jumptrading.com (Bryan Banister) Date: Mon, 31 Oct 2016 17:29:05 +0000 Subject: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size In-Reply-To: <20161031132628.5c952a40@osc.edu> References: <21BC488F0AEA2245B2C3E83FC0B33DBB063A1D4A@CHI-EXCHANGEW1.w2k.jumptrading.com> <20161031132628.5c952a40@osc.edu> Message-ID: <21BC488F0AEA2245B2C3E83FC0B33DBB063A1F04@CHI-EXCHANGEW1.w2k.jumptrading.com> Thanks for that suggestion, which is something we already do, but as you say is not as flexible. I think Jez is correct about the overhead being too high to support directory path for data placement, -B -----Original Message----- From: Edward Wahl [mailto:ewahl at osc.edu] Sent: Monday, October 31, 2016 12:26 PM To: Bryan Banister Cc: gpfsug main discussion list Subject: Re: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size On Mon, 31 Oct 2016 17:09:55 +0000 Bryan Banister wrote: > -Bryan > > PS. I really wish that we could use a path for specifying data > placement in a GPFS Pool, and not just the file name, owner, etc. > I?ll submit a RFE for this. So... use a fileset with a linked directory and a fileset placement policy to a pool? Might be a bit more rigid for what you really want, and it would be messy, but it would work just fine. -- Ed Wahl Ohio Supercomputer Center 614-292-9302 ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. From chrisjscott at gmail.com Mon Oct 31 17:29:45 2016 From: chrisjscott at gmail.com (Chris Scott) Date: Mon, 31 Oct 2016 17:29:45 +0000 Subject: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size In-Reply-To: References: <21BC488F0AEA2245B2C3E83FC0B33DBB063A1D4A@CHI-EXCHANGEW1.w2k.jumptrading.com> Message-ID: Hi Brian This is exactly what I do with a SSD tier on top of 10K and 7.2K tiers. HAWC is another recent option that might address Eric's requirement but needs further consideration of the read requirements you want from the small files. Cheers Chris On 31 October 2016 at 17:23, Brian Marshall wrote: > When creating a "fast tier" storage pool in a filesystem is the normal > case to create a placement policy that places all files in the fast tier > and migrates out old and large files? > > > Brian Marshall > > On Mon, Oct 31, 2016 at 1:20 PM, Jez Tucker wrote: > >> Hey Bryan >> >> There was a previous RFE for path placement from the UG, but Yuri told >> me this was not techically possible as an inode has no knowledge about the >> parent dentry. (IIRC). You can see this in effect in the C API. It is >> possible to work this out at kernel level, but it's so costly that it >> becomes non-viable at scale / performance. >> >> IBMers please chip in and expand if you will. >> >> Jez >> >> >> On 31/10/16 17:09, Bryan Banister wrote: >> >> The File Placement Policy that you are trying to set cannot use the size >> of the file to determine the placement of the file in a GPFS Storage Pool. >> This is because GPFS has no idea what the file size will be when the file >> is open()?d for writing. >> >> >> >> Hope that helps! >> >> -Bryan >> >> >> >> PS. I really wish that we could use a path for specifying data placement >> in a GPFS Pool, and not just the file name, owner, etc. I?ll submit a RFE >> for this. >> >> >> >> *From:* gpfsug-discuss-bounces at spectrumscale.org [ >> mailto:gpfsug-discuss-bounces at spectrumscale.org >> ] *On Behalf Of *J. Eric >> Wonderley >> *Sent:* Monday, October 31, 2016 11:53 AM >> *To:* gpfsug main discussion list >> *Subject:* [gpfsug-discuss] wanted...gpfs policy that places larger >> files onto a pool based on size >> >> >> >> I wanted to do something like this... >> >> >> [root at cl001 ~]# cat /opt/gpfs/home.ply >> /*Failsafe migration of old small files back to spinning media >> pool(fc_8T) */ >> RULE 'theshold' MIGRATE FROM POOL 'system' THRESHOLD(90,70) >> WEIGHT(ACCESS_TIME) TO POOL 'fc_8T' >> /*Write files larger than 16MB to pool called "fc_8T" */ >> RULE 'bigfiles' SET POOL 'fc_8T' WHERE FILE_SIZE>16777216 >> /*Move anything else to system pool */ >> RULE 'default' SET POOL 'system' >> >> Apparently there is no happiness using FILE_SIZE in a placement policy: >> [root at cl001 ~]# mmchpolicy home /opt/gpfs/home.ply >> Error while validating policy `home.ply': rc=22: >> PCSQLERR: 'FILE_SIZE' is an unsupported or unknown attribute or variable >> name in this context. >> PCSQLCTX: at line 4 of 6: RULE 'bigfiles' SET POOL 'fc_8T' WHERE >> {{{FILE_SIZE}}}>16777216 >> runRemoteCommand_v2: cl002.cl.arc.internal: tschpolicy /dev/home >> /var/mmfs/tmp/tspolicyFile.mmchpolicy.113372 -t home.ply failed. >> mmchpolicy: Command failed. Examine previous error messages to determine >> cause. >> >> Can anyone suggest a way to accomplish this using policy? >> >> ------------------------------ >> >> Note: This email is for the confidential use of the named addressee(s) >> only and may contain proprietary, confidential or privileged information. >> If you are not the intended recipient, you are hereby notified that any >> review, dissemination or copying of this email is strictly prohibited, and >> to please notify the sender immediately and destroy this email and any >> attachments. Email transmission cannot be guaranteed to be secure or >> error-free. The Company, therefore, does not make any guarantees as to the >> completeness or accuracy of this email or any attachments. This email is >> for informational purposes only and does not constitute a recommendation, >> offer, request or solicitation of any kind to buy, sell, subscribe, redeem >> or perform any type of transaction of a financial product. >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chrisjscott at gmail.com Mon Oct 31 17:33:57 2016 From: chrisjscott at gmail.com (Chris Scott) Date: Mon, 31 Oct 2016 17:33:57 +0000 Subject: [gpfsug-discuss] Recent Whitepapers from Yuri Volobuev In-Reply-To: <9C6B34F6-2849-4B77-9E8B-3806D634C281@gmail.com> References: <9C6B34F6-2849-4B77-9E8B-3806D634C281@gmail.com> Message-ID: Hear, hear. My appreciation to Yuri for his great contribution to the user community of important details. On 31 October 2016 at 16:19, Aaron Knister wrote: > Wow! That's a real shame. His ability to articulate his immense knowledge > of GPFS was truly impressive. > > Sent from my iPhone > > On Oct 31, 2016, at 5:53 AM, Oesterlin, Robert < > Robert.Oesterlin at nuance.com> wrote: > > For those of you who may not know, Yuri Volobuev has left IBM to pursue > new challenges. Myself along with many others received so much help and > keen insight from Yuri on all things GPFS. > > > > He will be missed. > > > > > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.wonderley at vt.edu Mon Oct 31 17:45:48 2016 From: eric.wonderley at vt.edu (J. Eric Wonderley) Date: Mon, 31 Oct 2016 13:45:48 -0400 Subject: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size In-Reply-To: References: <21BC488F0AEA2245B2C3E83FC0B33DBB063A1D4A@CHI-EXCHANGEW1.w2k.jumptrading.com> Message-ID: Placement policy only applies to writes and I had thought that gpfs did enough writing to memory "pagepool" to figure out the size before committing the write to pool. I also admit I don't know all of the innards of gpfs. Pehaps being a copy on write type filesystem prevents this for occurring. On Mon, Oct 31, 2016 at 1:29 PM, Chris Scott wrote: > Hi Brian > > This is exactly what I do with a SSD tier on top of 10K and 7.2K tiers. > > HAWC is another recent option that might address Eric's requirement but > needs further consideration of the read requirements you want from the > small files. > > Cheers > Chris > > On 31 October 2016 at 17:23, Brian Marshall wrote: > >> When creating a "fast tier" storage pool in a filesystem is the normal >> case to create a placement policy that places all files in the fast tier >> and migrates out old and large files? >> >> >> Brian Marshall >> >> On Mon, Oct 31, 2016 at 1:20 PM, Jez Tucker >> wrote: >> >>> Hey Bryan >>> >>> There was a previous RFE for path placement from the UG, but Yuri told >>> me this was not techically possible as an inode has no knowledge about the >>> parent dentry. (IIRC). You can see this in effect in the C API. It is >>> possible to work this out at kernel level, but it's so costly that it >>> becomes non-viable at scale / performance. >>> >>> IBMers please chip in and expand if you will. >>> >>> Jez >>> >>> >>> On 31/10/16 17:09, Bryan Banister wrote: >>> >>> The File Placement Policy that you are trying to set cannot use the size >>> of the file to determine the placement of the file in a GPFS Storage Pool. >>> This is because GPFS has no idea what the file size will be when the file >>> is open()?d for writing. >>> >>> >>> >>> Hope that helps! >>> >>> -Bryan >>> >>> >>> >>> PS. I really wish that we could use a path for specifying data placement >>> in a GPFS Pool, and not just the file name, owner, etc. I?ll submit a RFE >>> for this. >>> >>> >>> >>> *From:* gpfsug-discuss-bounces at spectrumscale.org [ >>> mailto:gpfsug-discuss-bounces at spectrumscale.org >>> ] *On Behalf Of *J. Eric >>> Wonderley >>> *Sent:* Monday, October 31, 2016 11:53 AM >>> *To:* gpfsug main discussion list >>> *Subject:* [gpfsug-discuss] wanted...gpfs policy that places larger >>> files onto a pool based on size >>> >>> >>> >>> I wanted to do something like this... >>> >>> >>> [root at cl001 ~]# cat /opt/gpfs/home.ply >>> /*Failsafe migration of old small files back to spinning media >>> pool(fc_8T) */ >>> RULE 'theshold' MIGRATE FROM POOL 'system' THRESHOLD(90,70) >>> WEIGHT(ACCESS_TIME) TO POOL 'fc_8T' >>> /*Write files larger than 16MB to pool called "fc_8T" */ >>> RULE 'bigfiles' SET POOL 'fc_8T' WHERE FILE_SIZE>16777216 >>> /*Move anything else to system pool */ >>> RULE 'default' SET POOL 'system' >>> >>> Apparently there is no happiness using FILE_SIZE in a placement policy: >>> [root at cl001 ~]# mmchpolicy home /opt/gpfs/home.ply >>> Error while validating policy `home.ply': rc=22: >>> PCSQLERR: 'FILE_SIZE' is an unsupported or unknown attribute or variable >>> name in this context. >>> PCSQLCTX: at line 4 of 6: RULE 'bigfiles' SET POOL 'fc_8T' WHERE >>> {{{FILE_SIZE}}}>16777216 >>> runRemoteCommand_v2: cl002.cl.arc.internal: tschpolicy /dev/home >>> /var/mmfs/tmp/tspolicyFile.mmchpolicy.113372 -t home.ply failed. >>> mmchpolicy: Command failed. Examine previous error messages to determine >>> cause. >>> >>> Can anyone suggest a way to accomplish this using policy? >>> >>> ------------------------------ >>> >>> Note: This email is for the confidential use of the named addressee(s) >>> only and may contain proprietary, confidential or privileged information. >>> If you are not the intended recipient, you are hereby notified that any >>> review, dissemination or copying of this email is strictly prohibited, and >>> to please notify the sender immediately and destroy this email and any >>> attachments. Email transmission cannot be guaranteed to be secure or >>> error-free. The Company, therefore, does not make any guarantees as to the >>> completeness or accuracy of this email or any attachments. This email is >>> for informational purposes only and does not constitute a recommendation, >>> offer, request or solicitation of any kind to buy, sell, subscribe, redeem >>> or perform any type of transaction of a financial product. >>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sfadden at us.ibm.com Mon Oct 3 13:39:00 2016 From: sfadden at us.ibm.com (Scott Fadden) Date: Mon, 3 Oct 2016 05:39:00 -0700 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? In-Reply-To: <0D55AB1D-DB9D-45CF-AB27-157CDA1172D9@nuance.com> References: <0D55AB1D-DB9D-45CF-AB27-157CDA1172D9@nuance.com> Message-ID: About 3.5 KiB if the inode is 4k and you are not using encryption (uses EA space) or large custom extended attributes, Scott Fadden Spectrum Scale - Technical Marketing Phone: (503) 880-5833 sfadden at us.ibm.com http://www.ibm.com/systems/storage/spectrum/scale From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 09/28/2016 11:14 AM Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? Sent by: gpfsug-discuss-bounces at spectrumscale.org What the largest file that will fit inside a 1K, 2K, or 4K inode? Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Mon Oct 3 13:54:47 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Mon, 3 Oct 2016 12:54:47 +0000 Subject: [gpfsug-discuss] License question Message-ID: I have a question about licensing. If I have a system which is NFS mounting (from a CES node), that is then serving data via HTTP, does this system need to have a Server license? Reading the FAQ (in the context of VMs): https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html "The IBM Spectrum Scale Client may not be used for virtual servers to share IBM Spectrum Scale data directly through any application, service protocol or method, such as NFS, CIFS, FTP, HTTP, or OpenStack Swift" My reading of this is that if the VM is mounting via NFS, then as it is not a Spectrum Scale client (rather an NFS client), then it doesn't require a Scale license. Secondly. If the VM is mounting from the hypervisor via NFS (Manilla), is the VM then allowed to serve data out by e.g. Web server, or does the hypervisor require a server license to do this. Assuming my hypervisor is licensed for Spectrum Scale of course. I think my interpretation is that if a VM is using NFS mounting from either a CES node or using Manilla to the hypervisor, then this is all OK to "re-export" data as the VM isn't talking GPFS protocol. Is this correct? Simon From daniel.kidger at uk.ibm.com Mon Oct 3 14:23:35 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Mon, 3 Oct 2016 13:23:35 +0000 Subject: [gpfsug-discuss] License question In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Mon Oct 3 15:53:51 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 3 Oct 2016 10:53:51 -0400 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? 3876! In-Reply-To: References: <0D55AB1D-DB9D-45CF-AB27-157CDA1172D9@nuance.com> Message-ID: Pretty easy to determine experimentally, with a few trials.... And the answer is 3876 for an simple file with just a selinux EA which is apparently created by default on my test system. So apparently there are actually only 220 bytes of metadata in this case (4096-3876=220). As they say YMMV ;-) [root at bog-wifi cmvc]# mmlsfs yy -i flag value description ------------------- ------------------------ ----------------------------------- -i 4096 Inode size in bytes [root at bog-wifi isize]# stat i387[67] File: ?i3876? Size: 3876 Blocks: 0 IO Block: 262144 regular file File: ?i3877? Size: 3877 Blocks: 16 IO Block: 262144 regular file [root at bog-wifi isize]# ls -ild i3876 19406 -rw-r--r--. 1 root root 3876 Oct 3 10:39 i3876 tsdbfs yy inode 19406 Inode 19406 [19406] snap 0 (index 14 in block 303): Inode address: 1:511600 size 4096 nAddrs 323 indirectionLevel=INODE status=USERFILE objectVersion=2 generation=0x7C5599C4 nlink=1 owner uid=0 gid=0 mode=0200100644: -rw-r--r-- ... Data [3876]: 0000000000000000: 136D7B1F 1D29CA0C B59BC917 D9AE0220 *.m{..)..........* 0000000000000010: F18512CF 998AC02E DC6926FA 3FDC3EAF *.........i&.?.>.* ... 0000000000000F20: EFF02209 00000000 05077365 6C696E75 *..".......selinu* trailer: 0x20005993FD8 diskAddrs 323 xattrLen 48 (free 3856/3904) res 0,0,0 overflow pri 0 vers 0 nsub 0 repl 1/2 DA Null [0] id 0x5 prefLen 8 keyLen 7 key security.selinux len 37 val '756E636F6E66696E65645F753A6F626A6563745F723A756E6C6162656C65645F743 A733000' -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 21994 bytes Desc: not available URL: From makaplan at us.ibm.com Mon Oct 3 16:46:09 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 3 Oct 2016 11:46:09 -0400 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? 3968!! In-Reply-To: References: <0D55AB1D-DB9D-45CF-AB27-157CDA1172D9@nuance.com> Message-ID: On a non-SELINUX system the answer is 3968 of data in a 4K inode, just 128 bytes of metadata. Caution: it's possible in some future release, this could change ... I don't know of any plans, I'm just saying ... Inode 16346892 [16346892] snap 0 (index 12 in block 255420): Inode address: 6:123049056 size 4096 nAddrs 330 indirectionLevel=INODE status=USERFILE objectVersion=1 generation=0xC0156CB nlink=1 owner uid=0 gid=0 mode=0200100644: -rw-r--r-- blocksize code=5 (32 subblocks) lastBlockSubblocks=0 checksum=0xAD8E0B4B is Valid fileSize=3968 nFullBlocks=0 currentMetadataReplicas=1 maxMetadataReplicas=2 currentDataReplicas=1 maxDataReplicas=2 ... Data [3968]: 0000000000000000: BCA91252 2B64BEDC A7D7BA9D D5BE8C30 *...R+d.........0* ... 0000000000000F70: DA925E2F 16A68C01 03CA5E37 08D72B7F *..^/......^7..+.* trailer: is NULL -------------- next part -------------- An HTML attachment was scrubbed... URL: From A.K.Ghumra at bham.ac.uk Mon Oct 3 16:52:52 2016 From: A.K.Ghumra at bham.ac.uk (Aslam Ghumra (IT Services, Facilities Management)) Date: Mon, 3 Oct 2016 15:52:52 +0000 Subject: [gpfsug-discuss] WebDAV protocol access support Message-ID: Does anyone know whether WebDAV protocol access support will ( if ever ) be implemented within GPFS ? Regards, Aslam Ghumra Research Data Management ____________________________ IT Services Elms Road Data Centre Building G5 Edgbaston Birmingham B15 2TT T: 0121 414 5877 F; 0121 414 3952 Skype : JanitorX Twitter : @aslamghumra in : https://uk.linkedin.com/in/aslam-ghumra-13907993 http://intranet.bham.ac.uk/bear -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Mon Oct 3 16:56:52 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 3 Oct 2016 15:56:52 +0000 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? Message-ID: <1BDA1763-A6FE-4A52-B2ED-22324C9AA16F@nuance.com> What's going be taken away if you use Encryption or Transparent Cloud Tiering? Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: on behalf of Marc A Kaplan Reply-To: gpfsug main discussion list Date: Monday, October 3, 2016 at 10:46 AM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Biggest file that will fit inside an inode? 3968!! On a non-SELINUX system the answer is 3968 of data in a 4K inode, just 128 bytes of metadata. Caution: it's possible in some future release, this could change ... I don't know of any plans, I'm just saying ... Inode 16346892 [16346892] snap 0 (index 12 in block 255420): Inode address: 6:123049056 size 4096 nAddrs 330 indirectionLevel=INODE status=USERFILE objectVersion=1 generation=0xC0156CB nlink=1 owner uid=0 gid=0 mode=0200100644: -rw-r--r-- blocksize code=5 (32 subblocks) lastBlockSubblocks=0 checksum=0xAD8E0B4B is Valid fileSize=3968 nFullBlocks=0 currentMetadataReplicas=1 maxMetadataReplicas=2 currentDataReplicas=1 maxDataReplicas=2 ... Data [3968]: 0000000000000000: BCA91252 2B64BEDC A7D7BA9D D5BE8C30 *...R+d.........0* ... 0000000000000F70: DA925E2F 16A68C01 03CA5E37 08D72B7F *..^/......^7..+.* trailer: is NULL -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Mon Oct 3 16:59:54 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Mon, 3 Oct 2016 15:59:54 +0000 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? In-Reply-To: <1BDA1763-A6FE-4A52-B2ED-22324C9AA16F@nuance.com> References: <1BDA1763-A6FE-4A52-B2ED-22324C9AA16F@nuance.com> Message-ID: Would you tier an in-inode file to the cloud? I mean, I wouldn't tier an in-inode file out to tape? Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Oesterlin, Robert [Robert.Oesterlin at nuance.com] Sent: 03 October 2016 16:56 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? What's going be taken away if you use Encryption or Transparent Cloud Tiering? Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: on behalf of Marc A Kaplan Reply-To: gpfsug main discussion list Date: Monday, October 3, 2016 at 10:46 AM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Biggest file that will fit inside an inode? 3968!! On a non-SELINUX system the answer is 3968 of data in a 4K inode, just 128 bytes of metadata. Caution: it's possible in some future release, this could change ... I don't know of any plans, I'm just saying ... Inode 16346892 [16346892] snap 0 (index 12 in block 255420): Inode address: 6:123049056 size 4096 nAddrs 330 indirectionLevel=INODE status=USERFILE objectVersion=1 generation=0xC0156CB nlink=1 owner uid=0 gid=0 mode=0200100644: -rw-r--r-- blocksize code=5 (32 subblocks) lastBlockSubblocks=0 checksum=0xAD8E0B4B is Valid fileSize=3968 nFullBlocks=0 currentMetadataReplicas=1 maxMetadataReplicas=2 currentDataReplicas=1 maxDataReplicas=2 ... Data [3968]: 0000000000000000: BCA91252 2B64BEDC A7D7BA9D D5BE8C30 *...R+d.........0* ... 0000000000000F70: DA925E2F 16A68C01 03CA5E37 08D72B7F *..^/......^7..+.* trailer: is NULL From Robert.Oesterlin at nuance.com Mon Oct 3 17:06:41 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 3 Oct 2016 16:06:41 +0000 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? In-Reply-To: References: <1BDA1763-A6FE-4A52-B2ED-22324C9AA16F@nuance.com> Message-ID: <37F799FA-7200-4011-81F0-6FF0DF3A4006@nuance.com> No, but (I think) TCT uses some of the EA space in the inode to indicate that the data is in the cloud tier. And yes, it does copy the data blocks in the inode to the clod tier if you migrate it. Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: on behalf of "Simon Thompson (Research Computing - IT Services)" Reply-To: gpfsug main discussion list Date: Monday, October 3, 2016 at 10:59 AM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Biggest file that will fit inside an inode? Would you tier an in-inode file to the cloud? I mean, I wouldn't tier an in-inode file out to tape? Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Oesterlin, Robert [Robert.Oesterlin at nuance.com] Sent: 03 October 2016 16:56 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? What's going be taken away if you use Encryption or Transparent Cloud Tiering? Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: > on behalf of Marc A Kaplan > Reply-To: gpfsug main discussion list > Date: Monday, October 3, 2016 at 10:46 AM To: gpfsug main discussion list > Subject: [EXTERNAL] Re: [gpfsug-discuss] Biggest file that will fit inside an inode? 3968!! On a non-SELINUX system the answer is 3968 of data in a 4K inode, just 128 bytes of metadata. Caution: it's possible in some future release, this could change ... I don't know of any plans, I'm just saying ... Inode 16346892 [16346892] snap 0 (index 12 in block 255420): Inode address: 6:123049056 size 4096 nAddrs 330 indirectionLevel=INODE status=USERFILE objectVersion=1 generation=0xC0156CB nlink=1 owner uid=0 gid=0 mode=0200100644: -rw-r--r-- blocksize code=5 (32 subblocks) lastBlockSubblocks=0 checksum=0xAD8E0B4B is Valid fileSize=3968 nFullBlocks=0 currentMetadataReplicas=1 maxMetadataReplicas=2 currentDataReplicas=1 maxDataReplicas=2 ... Data [3968]: 0000000000000000: BCA91252 2B64BEDC A7D7BA9D D5BE8C30 *...R+d.........0* ... 0000000000000F70: DA925E2F 16A68C01 03CA5E37 08D72B7F *..^/......^7..+.* trailer: is NULL _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=1C_5OQ7M5ibcRAY6xnDfRpKRJiw3qUawXpXihmGoQcs&s=skn6tFln6MeuKACIoxnUL_kuxHIC7AkskTni8IjzLj0&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From luke.raimbach at googlemail.com Mon Oct 3 17:07:39 2016 From: luke.raimbach at googlemail.com (Luke Raimbach) Date: Mon, 03 Oct 2016 16:07:39 +0000 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? In-Reply-To: References: <1BDA1763-A6FE-4A52-B2ED-22324C9AA16F@nuance.com> Message-ID: Surely it wouldn't go? Maybe the data would get copied out rather than stubbed... DMAPI can't be stupid enough to stub data out of an inode? Can it? Interesting question. Maybe I'll test that one. On Mon, 3 Oct 2016 at 17:00 Simon Thompson (Research Computing - IT Services) wrote: > > Would you tier an in-inode file to the cloud? > > I mean, I wouldn't tier an in-inode file out to tape? > > Simon > ________________________________________ > From: gpfsug-discuss-bounces at spectrumscale.org [ > gpfsug-discuss-bounces at spectrumscale.org] on behalf of Oesterlin, Robert [ > Robert.Oesterlin at nuance.com] > Sent: 03 October 2016 16:56 > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? > > What's going be taken away if you use Encryption or Transparent Cloud > Tiering? > > > Bob Oesterlin > Sr Storage Engineer, Nuance HPC Grid > > > From: on behalf of Marc A > Kaplan > Reply-To: gpfsug main discussion list > Date: Monday, October 3, 2016 at 10:46 AM > To: gpfsug main discussion list > Subject: [EXTERNAL] Re: [gpfsug-discuss] Biggest file that will fit inside > an inode? 3968!! > > On a non-SELINUX system the answer is 3968 of data in a 4K inode, just 128 > bytes of metadata. > > Caution: it's possible in some future release, this could change ... I > don't know of any plans, I'm just saying ... > > Inode 16346892 [16346892] snap 0 (index 12 in block 255420): > Inode address: 6:123049056 size 4096 nAddrs 330 > indirectionLevel=INODE status=USERFILE > objectVersion=1 generation=0xC0156CB nlink=1 > owner uid=0 gid=0 mode=0200100644: -rw-r--r-- > blocksize code=5 (32 subblocks) > lastBlockSubblocks=0 > checksum=0xAD8E0B4B is Valid > fileSize=3968 nFullBlocks=0 > currentMetadataReplicas=1 maxMetadataReplicas=2 > currentDataReplicas=1 maxDataReplicas=2 > ... > Data [3968]: > 0000000000000000: BCA91252 2B64BEDC A7D7BA9D D5BE8C30 *...R+d.........0* > ... > 0000000000000F70: DA925E2F 16A68C01 03CA5E37 08D72B7F *..^/......^7..+.* > trailer: is NULL > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Mon Oct 3 17:10:34 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Mon, 3 Oct 2016 16:10:34 +0000 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? In-Reply-To: References: <1BDA1763-A6FE-4A52-B2ED-22324C9AA16F@nuance.com> , Message-ID: TCT doesn't use dmapi though I thought? ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Luke Raimbach [luke.raimbach at googlemail.com] Sent: 03 October 2016 17:07 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? Surely it wouldn't go? Maybe the data would get copied out rather than stubbed... DMAPI can't be stupid enough to stub data out of an inode? Can it? Interesting question. Maybe I'll test that one. On Mon, 3 Oct 2016 at 17:00 Simon Thompson (Research Computing - IT Services) > wrote: Would you tier an in-inode file to the cloud? I mean, I wouldn't tier an in-inode file out to tape? Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Oesterlin, Robert [Robert.Oesterlin at nuance.com] Sent: 03 October 2016 16:56 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? What's going be taken away if you use Encryption or Transparent Cloud Tiering? Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: > on behalf of Marc A Kaplan > Reply-To: gpfsug main discussion list > Date: Monday, October 3, 2016 at 10:46 AM To: gpfsug main discussion list > Subject: [EXTERNAL] Re: [gpfsug-discuss] Biggest file that will fit inside an inode? 3968!! On a non-SELINUX system the answer is 3968 of data in a 4K inode, just 128 bytes of metadata. Caution: it's possible in some future release, this could change ... I don't know of any plans, I'm just saying ... Inode 16346892 [16346892] snap 0 (index 12 in block 255420): Inode address: 6:123049056 size 4096 nAddrs 330 indirectionLevel=INODE status=USERFILE objectVersion=1 generation=0xC0156CB nlink=1 owner uid=0 gid=0 mode=0200100644: -rw-r--r-- blocksize code=5 (32 subblocks) lastBlockSubblocks=0 checksum=0xAD8E0B4B is Valid fileSize=3968 nFullBlocks=0 currentMetadataReplicas=1 maxMetadataReplicas=2 currentDataReplicas=1 maxDataReplicas=2 ... Data [3968]: 0000000000000000: BCA91252 2B64BEDC A7D7BA9D D5BE8C30 *...R+d.........0* ... 0000000000000F70: DA925E2F 16A68C01 03CA5E37 08D72B7F *..^/......^7..+.* trailer: is NULL _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From luke.raimbach at googlemail.com Mon Oct 3 17:16:05 2016 From: luke.raimbach at googlemail.com (Luke Raimbach) Date: Mon, 03 Oct 2016 16:16:05 +0000 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? In-Reply-To: References: <1BDA1763-A6FE-4A52-B2ED-22324C9AA16F@nuance.com> Message-ID: It doesn't, but the end result is the same... data shipped off 'somewhere else' with a stub file? I have in my mind that DMAPI support was around before data-in-inode (or at least 4K inodes) was introduced, so it had to be made a bit cleverer to cope, but I may be misremembering that. On Mon, 3 Oct 2016 at 17:10 Simon Thompson (Research Computing - IT Services) wrote: > > TCT doesn't use dmapi though I thought? > ________________________________________ > From: gpfsug-discuss-bounces at spectrumscale.org [ > gpfsug-discuss-bounces at spectrumscale.org] on behalf of Luke Raimbach [ > luke.raimbach at googlemail.com] > Sent: 03 October 2016 17:07 > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? > > Surely it wouldn't go? Maybe the data would get copied out rather than > stubbed... DMAPI can't be stupid enough to stub data out of an inode? Can > it? Interesting question. > > Maybe I'll test that one. > > On Mon, 3 Oct 2016 at 17:00 Simon Thompson (Research Computing - IT > Services) > wrote: > > Would you tier an in-inode file to the cloud? > > I mean, I wouldn't tier an in-inode file out to tape? > > Simon > ________________________________________ > From: gpfsug-discuss-bounces at spectrumscale.org gpfsug-discuss-bounces at spectrumscale.org> [ > gpfsug-discuss-bounces at spectrumscale.org gpfsug-discuss-bounces at spectrumscale.org>] on behalf of Oesterlin, Robert > [Robert.Oesterlin at nuance.com] > Sent: 03 October 2016 16:56 > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? > > What's going be taken away if you use Encryption or Transparent Cloud > Tiering? > > > Bob Oesterlin > Sr Storage Engineer, Nuance HPC Grid > > > From: gpfsug-discuss-bounces at spectrumscale.org>> on behalf of Marc A Kaplan < > makaplan at us.ibm.com> > Reply-To: gpfsug main discussion list > > Date: Monday, October 3, 2016 at 10:46 AM > To: gpfsug main discussion list gpfsug-discuss at spectrumscale.org>> > Subject: [EXTERNAL] Re: [gpfsug-discuss] Biggest file that will fit inside > an inode? 3968!! > > On a non-SELINUX system the answer is 3968 of data in a 4K inode, just 128 > bytes of metadata. > > Caution: it's possible in some future release, this could change ... I > don't know of any plans, I'm just saying ... > > Inode 16346892 [16346892] snap 0 (index 12 in block 255420): > Inode address: 6:123049056 size 4096 nAddrs 330 > indirectionLevel=INODE status=USERFILE > objectVersion=1 generation=0xC0156CB nlink=1 > owner uid=0 gid=0 mode=0200100644: -rw-r--r-- > blocksize code=5 (32 subblocks) > lastBlockSubblocks=0 > checksum=0xAD8E0B4B is Valid > fileSize=3968 nFullBlocks=0 > currentMetadataReplicas=1 maxMetadataReplicas=2 > currentDataReplicas=1 maxDataReplicas=2 > ... > Data [3968]: > 0000000000000000: BCA91252 2B64BEDC A7D7BA9D D5BE8C30 *...R+d.........0* > ... > 0000000000000F70: DA925E2F 16A68C01 03CA5E37 08D72B7F *..^/......^7..+.* > trailer: is NULL > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Mon Oct 3 17:26:08 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 3 Oct 2016 12:26:08 -0400 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? What it says! In-Reply-To: References: <1BDA1763-A6FE-4A52-B2ED-22324C9AA16F@nuance.com>, Message-ID: I just noticed that tsdbfs shows you how much data fits in the inode, even if you don't fill it... So a few commands will give you the answer: dd, sync, ls, tsdbfs, inode Encryption uses EAs, so you lose some space in inode to that. I don't have an encrypted FS handy for testing, if you have one, go ahead and use tsdbfs to look at an inode and see whether or not encryption pushes the data out of the inode, I wouldn't be surprised either way. [root at n2 isizes]# dd if=/dev/urandom bs=1 count=100 of=i100 100+0 records in 100+0 records out 100 bytes (100 B) copied, 0.00226643 s, 44.1 kB/s [root at n2 isizes]# sync [root at n2 isizes]# ls -ild i100 16346894 -rw-r--r-- 1 root root 100 Oct 3 09:14 i100 [root at n2 isizes]# tsdbfs mak Enter command or null to read next sector. Type ? for help. inode 16346894 Inode 16346894 [16346894] snap 0 (index 14 in block 255420): Inode address: 6:123049072 size 4096 nAddrs 330 indirectionLevel=INODE status=USERFILE objectVersion=1 generation=0x9D45DC1 nlink=1 owner uid=0 gid=0 mode=0200100644: -rw-r--r-- blocksize code=5 (32 subblocks) lastBlockSubblocks=0 checksum=0x2496E008 is Valid fileSize=100 nFullBlocks=0 ... Data [3968]: 0000000000000000: 17699B6F 5F9ACD28 70C05242 9268F44E *.i.o_..(p.RB.h.N* 0000000000000010: 8FFCDCC1 C2ACE0EC 69C1FE2A 986B752C *........i..*.ku,* 0000000000000020: 8E0434E6 1904B7FC 8B0C5709 2243343C *..4.......W."C4<* 0000000000000030: BD317374 FB5C322D 8E257B69 FF573283 *.1st.\2-.%{i.W2.* 0000000000000040: 515B333B EDFAE930 9B6F8712 0814BF65 *Q[3;...0.o.....e* 0000000000000050: DBDCC25E 87EB2C16 77AE672D 45FB6BE3 *...^..,.w.g-E.k.* 0000000000000060: 65D52D17 00000000 00000000 00000000 *e.-.............* 0000000000000070: 00000000 00000000 00000000 00000000 *................* ... trailer: is NULL From: "Simon Thompson (Research Computing - IT Services)" To: gpfsug main discussion list Date: 10/03/2016 12:10 PM Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? Sent by: gpfsug-discuss-bounces at spectrumscale.org TCT doesn't use dmapi though I thought? ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Luke Raimbach [luke.raimbach at googlemail.com] Sent: 03 October 2016 17:07 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? Surely it wouldn't go? Maybe the data would get copied out rather than stubbed... DMAPI can't be stupid enough to stub data out of an inode? Can it? Interesting question. Maybe I'll test that one. On Mon, 3 Oct 2016 at 17:00 Simon Thompson (Research Computing - IT Services) > wrote: Would you tier an in-inode file to the cloud? I mean, I wouldn't tier an in-inode file out to tape? Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org< mailto:gpfsug-discuss-bounces at spectrumscale.org> [gpfsug-discuss-bounces at spectrumscale.org< mailto:gpfsug-discuss-bounces at spectrumscale.org>] on behalf of Oesterlin, Robert [Robert.Oesterlin at nuance.com] Sent: 03 October 2016 16:56 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? What's going be taken away if you use Encryption or Transparent Cloud Tiering? Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: > on behalf of Marc A Kaplan > Reply-To: gpfsug main discussion list > Date: Monday, October 3, 2016 at 10:46 AM To: gpfsug main discussion list > Subject: [EXTERNAL] Re: [gpfsug-discuss] Biggest file that will fit inside an inode? 3968!! On a non-SELINUX system the answer is 3968 of data in a 4K inode, just 128 bytes of metadata. Caution: it's possible in some future release, this could change ... I don't know of any plans, I'm just saying ... Inode 16346892 [16346892] snap 0 (index 12 in block 255420): Inode address: 6:123049056 size 4096 nAddrs 330 indirectionLevel=INODE status=USERFILE objectVersion=1 generation=0xC0156CB nlink=1 owner uid=0 gid=0 mode=0200100644: -rw-r--r-- blocksize code=5 (32 subblocks) lastBlockSubblocks=0 checksum=0xAD8E0B4B is Valid fileSize=3968 nFullBlocks=0 currentMetadataReplicas=1 maxMetadataReplicas=2 currentDataReplicas=1 maxDataReplicas=2 ... Data [3968]: 0000000000000000: BCA91252 2B64BEDC A7D7BA9D D5BE8C30 *...R+d.........0* ... 0000000000000F70: DA925E2F 16A68C01 03CA5E37 08D72B7F *..^/......^7..+.* trailer: is NULL _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan at buzzard.me.uk Mon Oct 3 17:34:33 2016 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Mon, 03 Oct 2016 17:34:33 +0100 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? In-Reply-To: References: <1BDA1763-A6FE-4A52-B2ED-22324C9AA16F@nuance.com> Message-ID: <1475512473.3748.14.camel@buzzard.phy.strath.ac.uk> On Mon, 2016-10-03 at 16:16 +0000, Luke Raimbach wrote: > It doesn't, but the end result is the same... data shipped off > 'somewhere else' with a stub file? > > I have in my mind that DMAPI support was around before data-in-inode > (or at least 4K inodes) was introduced, so it had to be made a bit > cleverer to cope, but I may be misremembering that. > It did but you can always specify that files below a certain size don't get migrated, be it tape cloud or elsewhere. Never made sense migrating small files to tape anyway, and it certainly makes no sense to migrate a file in an inode anywhere. Perhaps that is the source of a request for enhancement? Stop files existing in inodes being migrated. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From luis.bolinches at fi.ibm.com Mon Oct 3 18:33:04 2016 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Mon, 3 Oct 2016 17:33:04 +0000 Subject: [gpfsug-discuss] WebDAV protocol access support In-Reply-To: Message-ID: Hi You might want to take a look to https://owncloud.com/ibm/ -- Cheers > On 3 Oct 2016, at 18.53, Aslam Ghumra (IT Services, Facilities Management) wrote: > > Does anyone know whether WebDAV protocol access support will ( if ever ) be implemented within GPFS ? > > Regards, > > Aslam Ghumra > Research Data Management > ____________________________ > IT Services > Elms Road Data Centre > Building G5 > Edgbaston > Birmingham B15 2TT > T: 0121 414 5877 > F; 0121 414 3952 > Skype : JanitorX > Twitter : @aslamghumra > in : https://uk.linkedin.com/in/aslam-ghumra-13907993 > http://intranet.bham.ac.uk/bear > Ellei edell?? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.kidger at uk.ibm.com Tue Oct 4 11:43:45 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Tue, 4 Oct 2016 10:43:45 +0000 Subject: [gpfsug-discuss] File_heat for GPFS File Systems In-Reply-To: References: , <8c7fd395-1efc-7197-4a98-763ba784cafd@stanford.edu> Message-ID: An HTML attachment was scrubbed... URL: From alandhae at gmx.de Tue Oct 4 12:25:52 2016 From: alandhae at gmx.de (=?ISO-8859-15?Q?Andreas_Landh=E4u=DFer?=) Date: Tue, 4 Oct 2016 13:25:52 +0200 (CEST) Subject: [gpfsug-discuss] File_heat for GPFS File Systems In-Reply-To: References: , <8c7fd395-1efc-7197-4a98-763ba784cafd@stanford.edu> Message-ID: On Tue, 4 Oct 2016, Daniel Kidger wrote: Daniel, ahh this might also help in my HPC Environment. We are already using AFM, just on HDD, for distributing HPC Data for usage on remote scientific sites. Using the AFM for a temp space seems to be a good idea, will the individual file heat be increased by using ILM policy and AFM? Using a large GPFS Filesystem for scientific results, it seems to be very unlikely all users starting using their cold files at the same point of time, so filesystem space should be sufficient. Thanks for the hint Andreas T-Systems SfR GmbH > If you need functionality like you describe, then as well as using tiering > with the ILM Policy Engine, also consider using AFM to cache onto say SSDs.? > That way you would have dynamic caching: as soon as a file gets accessed, it > goes into the cache and further I/O will be do the SSD copy so faster until > such time as the file ages and newer files replace it in the AFM cache. The > cost of this is perhaps a little more complexity and also a 'hot' file > consumes space on both SSD and disk. > ? > Daniel > /spectrum_storage-banne > ?????????????????????????????? v > ? > Spectrum Scale Logo > ? > ? > Dr Daniel Kidger > IBM?Technical Sales Specialist > Software Defined Solution Sales > > +44-(0)7818 522 266 > daniel.kidger at uk.ibm.com > ? > ? > ? > ----- Original message ----- > From: Eric Horst > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug main discussion list > > Cc: > Subject: Re: [gpfsug-discuss] File_heat for GPFS File Systems > Date: Tue, Sep 27, 2016 9:56 PM > ? >> > >> if a file gets hot again, there is no rule for putting the > file back > >> into a faster storage device? > > > > > > The file will get moved when you run the policy again. ?You > can run the > > policy as often as you like. > > I think its worth stating clearly that if a file is in the > Thrifty > slow pool and a user opens and reads/writes the file there is > nothing > that moves this file to a different tier. A policy run is the > only > action that relocates files. So if you apply the policy daily > and over > the course of the day users access many cold files, the > performance > accessing those cold files may not be ideal until the next day > when > they are repacked by heat. A file is not automatically moved to > the > fast tier on access read or write. I mention this because this > aspect > of tiering was not immediately clear from the docs when I was a > neophyte GPFS admin and I had to learn by observation. It is > easy for > one to make an assumption that it is a more dynamic tiering > system > than it is. > > -Eric > > -- > Eric Horst > University of Washington > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > ? > > ? > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > -- Andreas Landh?u?er +49 151 12133027 (mobile) alandhae at gmx.de From alandhae at gmx.de Tue Oct 4 13:54:16 2016 From: alandhae at gmx.de (=?ISO-8859-15?Q?Andreas_Landh=E4u=DFer?=) Date: Tue, 4 Oct 2016 14:54:16 +0200 (CEST) Subject: [gpfsug-discuss] Fileheat reporting Message-ID: Customer needs a report of filenames being Hot, warm and cold. As far as I'm understanding fileheat can be any value larger than zero. A value of zero or almost zero equals to COLD How am I classifying warm and hot? I'm needing a standardized value between 0 and 1 for fileheat. Is it possible creating such a number when when activating fileheat and creating reports and finally using ILM policies according to this classification? Best regards Andreas -- Andreas Landh?u?er +49 151 12133027 (mobile) alandhae at gmx.de From eric.wonderley at vt.edu Tue Oct 4 14:39:02 2016 From: eric.wonderley at vt.edu (J. Eric Wonderley) Date: Tue, 4 Oct 2016 09:39:02 -0400 Subject: [gpfsug-discuss] migrate policy vs restripe Message-ID: We have the need to move data from one set of spindles to another. Are there any performance or availability considerations when choosing to do either a migration policy or a restripe to make this move? I did discover that a restripe only works within the same pool...even though you setup two pools in different failure groups. -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Tue Oct 4 15:32:55 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 4 Oct 2016 10:32:55 -0400 Subject: [gpfsug-discuss] migrate policy vs restripe In-Reply-To: References: Message-ID: Doubtless there are subtleties and special cases, but my should-work-well advice is: A) Use mmapplypolicy ... -I defer -P policy-to-set-pool-assignments-and-replication-factor ... where the policy rules file contains rules like RULE 'adjust' MIGRATE [FROM POOL 'x'] TO POOL 'y' REPLICATE({1|2}) WHERE decide to which file this rule shall apply The '-I defer' option says just mark the inodes, do not actually move the data B) Use mmrestripefs ... -b ... -N nodeclass The move data to correct pool assignments and rebalance the entire file system, also fix ill-replicated and ill-placed files. (keep reading...) C) "restripe only works within the same pool" -- I believe there is a misunderstanding. For any given file, all the data blocks are supposed to be in the one pool indicated by pool assignment which is recorded in the file's inode. If the pool assignment changes but not all of the blocks are migrated to the new pool, then the file is considered "ill-placed". I believe mmrestripefs endeavors to move blocks so as to "fix" ill-placed files. Failure groups are a different concept that is associated with replication. The replicas of a given data or meta data block should be in different failure groups. If not, the file is considered "ill-replicated" and mmrestripefs endeavors to fix that also. From: "J. Eric Wonderley" To: gpfsug main discussion list Date: 10/04/2016 09:39 AM Subject: [gpfsug-discuss] migrate policy vs restripe Sent by: gpfsug-discuss-bounces at spectrumscale.org We have the need to move data from one set of spindles to another. Are there any performance or availability considerations when choosing to do either a migration policy or a restripe to make this move? I did discover that a restripe only works within the same pool...even though you setup two pools in different failure groups. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From aspalazz at us.ibm.com Tue Oct 4 15:44:33 2016 From: aspalazz at us.ibm.com (Aaron S Palazzolo) Date: Tue, 4 Oct 2016 14:44:33 +0000 Subject: [gpfsug-discuss] Toolkit Message-ID: An HTML attachment was scrubbed... URL: From stef.coene at docum.org Tue Oct 4 15:53:48 2016 From: stef.coene at docum.org (Stef Coene) Date: Tue, 4 Oct 2016 16:53:48 +0200 Subject: [gpfsug-discuss] Toolkit In-Reply-To: References: Message-ID: <6192009e-afea-36f1-d79a-9ec78194fe6d@docum.org> On 10/04/2016 04:44 PM, Aaron S Palazzolo wrote: > *Hi Stef,* > > Thanks for your Install Toolkit feature request: > -------------------- > /Is it possible to recreate the clusterdefinition.txt based on the > current configuration?/ > -------------------- > > A couple questions if you don't mind: > *1) *Are you using the Install Toolkit for base gpfs installation or > solely protocol deployment? I used the spectrum toolkit for basic setup. Then I used some mm commands to add disks and change other stuff. But the toolkit is unaware of theses changes. > *2) *If using it for base gpfs installation, do you create both NSDs and > file systems with it or do you do this manually? Like said, I used the toolkit for the first few NSDs and later used mm commands for other NSDs. I have no problems using the mm commands, but I don't know what will happen if I want to use the toolkit to create proto nodes. Stef From makaplan at us.ibm.com Tue Oct 4 15:56:46 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 4 Oct 2016 10:56:46 -0400 Subject: [gpfsug-discuss] Fileheat reporting In-Reply-To: References: Message-ID: FILE_HEAT value is a non-negative floating point value. The value can grow quite large -- in theory it can grow to the IEEE 64 bit maximum floating point value and maybe even to "infinity". Each time a file is read completely from disk it's FILE_HEAT value increases by 1. If a fraction, f, of the file is read then FILE_HEAT += f. (f<=1) On the other hand, as time passes the FILE_HEAT "decays" at the rate given by the (mmchconfig) parameters fileHeatPeriodMinutes and fileHeatLossPercent. In fact I recently corresponding with a customer (who is no stranger to this forum and may wish to comment!) who found that a bunch of hot little script files had accumulated heat values greater than 100,000,000 ! We were both surprised. Apparently these files are read and re-read many times every day by many nodes and so their heat builds up! So what would you call "Hot", "Warm", "Cold"? It depends... To get a good idea, I suggest that a "survey" of FILE_HEAT values be done by using an mmapplypolicy command with a rule like. rule 'y' list 'fhn0' weight(FILE_HEAT) SHOW(HEX(XATTR('gpfs.FileHeat')) || ' A=' || varchar(ACCESS_TIME) || ' K=' || varchar(KB_ALLOCATED) || ' H=' || varchar(FILE_HEAT)) where FILE_HEAT != 0.0 Using `mmapplypolicy FS -P policy-rules-file -I defer -f /tmp/whatever [other options such as... -N nodeclass -g /shared-temp ... ] This will produce a file list sorted by FILE_HEAT, which you can then peruse or otherwise process.... Perhaps a histogram or other display of (number of files) vs (heat values) would be interesting... From: Andreas Landh?u?er To: gpfsug-discuss at spectrumscale.org Date: 10/04/2016 08:54 AM Subject: [gpfsug-discuss] Fileheat reporting Sent by: gpfsug-discuss-bounces at spectrumscale.org Customer needs a report of filenames being Hot, warm and cold. As far as I'm understanding fileheat can be any value larger than zero. A value of zero or almost zero equals to COLD How am I classifying warm and hot? I'm needing a standardized value between 0 and 1 for fileheat. Is it possible creating such a number when when activating fileheat and creating reports and finally using ILM policies according to this classification? Best regards Andreas -- Andreas Landh?u?er +49 151 12133027 (mobile) alandhae at gmx.de_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From kronberg at nsc.liu.se Tue Oct 4 16:43:51 2016 From: kronberg at nsc.liu.se (Mats Kronberg) Date: Tue, 4 Oct 2016 17:43:51 +0200 Subject: [gpfsug-discuss] GSS 3.0 (GPFS 4.2.1) upgrade experiences? Message-ID: Hi, We are in the planning phase for upgrading our GSS system from GSS 2.0.7 (GPFS 4.1.0-8) to something more recent. Lenovo recommends going directly to the recently released GSS 3.0 (GPFS 4.2.1-0, released late September 2016). I'm inclined to agree with them, but the conservative, let-others-betatest-stuff part of me is suggesting something a little more mature like GSS 2.5.9 (GPFS 4.1.1-2, released late-2015). Has anyone got any success or horror stories to share regarding either of these releases? :) -- Mats Kronberg, Systems Expert NSC, National Supercomputer Centre -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Tue Oct 4 16:48:31 2016 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Tue, 4 Oct 2016 15:48:31 +0000 Subject: [gpfsug-discuss] GSS 3.0 (GPFS 4.2.1) upgrade experiences? In-Reply-To: References: Message-ID: We?re not huge, and not running GSS, but we are running CES for 3000+ concurrent users. We went from 3.5 to 4.2.1 via 4.1.1 and our experience was outstanding. We had some enforced downtime during the upgrad and a few hiccups with CES specifically, but overall very happy. That may help you in some shape or form ? From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Mats Kronberg Sent: 04 October 2016 16:44 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] GSS 3.0 (GPFS 4.2.1) upgrade experiences? Hi, We are in the planning phase for upgrading our GSS system from GSS 2.0.7 (GPFS 4.1.0-8) to something more recent. Lenovo recommends going directly to the recently released GSS 3.0 (GPFS 4.2.1-0, released late September 2016). I'm inclined to agree with them, but the conservative, let-others-betatest-stuff part of me is suggesting something a little more mature like GSS 2.5.9 (GPFS 4.1.1-2, released late-2015). Has anyone got any success or horror stories to share regarding either of these releases? :) -- Mats Kronberg, Systems Expert NSC, National Supercomputer Centre -------------- next part -------------- An HTML attachment was scrubbed... URL: From mimarsh2 at vt.edu Tue Oct 4 18:04:59 2016 From: mimarsh2 at vt.edu (Brian Marshall) Date: Tue, 4 Oct 2016 13:04:59 -0400 Subject: [gpfsug-discuss] testing commands Message-ID: All, Is there a way to "test" GPFS commands and see what the output or result would be before running them? For example, I'd like to verify a command string before actually running it on a production system. Does IBM offer "test" licenses for setting up a small debug/devel environment? I'd be interested to hear everyone's best practices on verifying commands/actions. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Tue Oct 4 18:30:03 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 4 Oct 2016 13:30:03 -0400 Subject: [gpfsug-discuss] testing commands In-Reply-To: References: Message-ID: Simple functional testing can be done on a small system with as little as one Linux node and one disk. Personally, I do a lot of testing on a several years old laptop running Linux. My test gpfs file system doesn't even use a real disk partition! Just some data files I initialized with dd if=/dev/zero ... (This is not officially supported, blah, blah, blah...) [root at bog-wifi cmvc]# mmlsnsd -X Disk name NSD volume ID Device Devtype Node name Remarks --------------------------------------------------------------------------------------------------- y1 C0A8015D57586A5E /vds/y1 file bog-wifi.lan.makaplan.us server node y2 C0A8015D57586A5F /vds/y2 file bog-wifi.lan.makaplan.us server node y3 C0A8015D575EF0A3 /vds/y3 file bog-wifi.lan.makaplan.us server node [root at bog-wifi cmvc]# ls -ld /vds/y? -rw-r--r--. 1 root root 536870912 Oct 4 12:16 /vds/y1 -rw-r--r--. 1 root root 536870912 Oct 3 10:39 /vds/y2 -rw-r--r--. 1 root root 801000000 Sep 29 18:54 /vds/y3 [root at bog-wifi cmvc]# df -T /vds Filesystem Type 1K-blocks Used Available Use% Mounted on /dev/mapper/rhelw_bog--wifi-root ext4 183862728 42118120 132381768 25% / -------------- next part -------------- An HTML attachment was scrubbed... URL: From mimarsh2 at vt.edu Tue Oct 4 21:13:05 2016 From: mimarsh2 at vt.edu (Brian Marshall) Date: Tue, 4 Oct 2016 16:13:05 -0400 Subject: [gpfsug-discuss] testing commands In-Reply-To: References: Message-ID: Do I require additional NSD Server licenses to do this? i.e. What triggers the need to purchase a license for a NSD Server? Brian On Tue, Oct 4, 2016 at 1:30 PM, Marc A Kaplan wrote: > Simple functional testing can be done on a small system with as little as > one Linux node and one disk. > > Personally, I do a lot of testing on a several years old laptop running > Linux. My test gpfs file system doesn't even use a real disk partition! > Just some data files > I initialized with dd if=/dev/zero ... > > (This is not officially supported, blah, blah, blah...) > > [root at bog-wifi cmvc]# mmlsnsd -X > > Disk name NSD volume ID Device Devtype Node name > Remarks > ------------------------------------------------------------ > --------------------------------------- > y1 C0A8015D57586A5E /vds/y1 file > bog-wifi.lan.makaplan.us server node > y2 C0A8015D57586A5F /vds/y2 file > bog-wifi.lan.makaplan.us server node > y3 C0A8015D575EF0A3 /vds/y3 file > bog-wifi.lan.makaplan.us server node > > [root at bog-wifi cmvc]# ls -ld /vds/y? > -rw-r--r--. 1 root root 536870912 Oct 4 12:16 /vds/y1 > -rw-r--r--. 1 root root 536870912 Oct 3 10:39 /vds/y2 > -rw-r--r--. 1 root root 801000000 Sep 29 18:54 /vds/y3 > > [root at bog-wifi cmvc]# df -T /vds > Filesystem Type 1K-blocks Used Available Use% > Mounted on > /dev/mapper/rhelw_bog--wifi-root ext4 183862728 42118120 132381768 25% / > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From swinglet at us.ibm.com Tue Oct 4 21:24:43 2016 From: swinglet at us.ibm.com (Teresa S Swingler) Date: Tue, 4 Oct 2016 20:24:43 +0000 Subject: [gpfsug-discuss] Survey - IBM Spectrum Scale Everyday Management Tasks Message-ID: An HTML attachment was scrubbed... URL: From jtucker at pixitmedia.com Tue Oct 4 21:37:42 2016 From: jtucker at pixitmedia.com (Jez Tucker) Date: Tue, 4 Oct 2016 21:37:42 +0100 Subject: [gpfsug-discuss] testing commands In-Reply-To: References: Message-ID: Hi So we were chatting a while ago about dry-run requirements, but more with respect to mmapplypolicy. For relevance to those on the list who run our Python API, you'll find dry-run functionality here: http://arcapix.com/gpfsapi/execute.html#dry-run Cheers, Jez On 04/10/16 18:04, Brian Marshall wrote: > All, > > Is there a way to "test" GPFS commands and see what the output or > result would be before running them? For example, I'd like to verify > a command string before actually running it on a production system. > > Does IBM offer "test" licenses for setting up a small debug/devel > environment? > > I'd be interested to hear everyone's best practices on verifying > commands/actions. > > > Thanks, > Brian > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Jez Tucker Head of Research & Product Development Pixit Media | ArcaStream www.pixitmedia.com www.arcastream.com -- This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From swinglet at us.ibm.com Tue Oct 4 21:37:39 2016 From: swinglet at us.ibm.com (Teresa S Swingler) Date: Tue, 4 Oct 2016 13:37:39 -0700 Subject: [gpfsug-discuss] Survey - Spectrum Scale Everyday Management Message-ID: Hi, We in IBM Spectrum Scale design and development are interested in learning about the frequency and usability of your everyday management tasks. Your input will help us with our upcoming priorities. Please take 5 minutes to complete this brief survey: https://www.surveygizmo.com/s3/3090036/IBM-Spectrum-Scale-Everyday-Management Thank you, Teresa Swingler Teresa S. Swingler IBM Systems Senior Designer | Lead UX Architect Spectrum Scale and Spectrum Control 2929 N. Central Ave. Phoenix, AZ 85012 Phone: (602) 217-2763 Tieline: 667-2763 E-mail: swinglet at us.ibm.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Tue Oct 4 22:49:01 2016 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 4 Oct 2016 17:49:01 -0400 Subject: [gpfsug-discuss] testing commands In-Reply-To: References: Message-ID: <797ae533-edee-102a-3f3e-f53687cfcf83@nasa.gov> Not to get off-topic here, but WHOA! I had no idea somebody had created a python API for GPFS. How does one get access to said API? On 10/4/16 4:37 PM, Jez Tucker wrote: > Hi > > So we were chatting a while ago about dry-run requirements, but more > with respect to mmapplypolicy. > For relevance to those on the list who run our Python API, you'll find > dry-run functionality here: > > http://arcapix.com/gpfsapi/execute.html#dry-run > > Cheers, > > Jez > > > > On 04/10/16 18:04, Brian Marshall wrote: >> All, >> >> Is there a way to "test" GPFS commands and see what the output or >> result would be before running them? For example, I'd like to verify >> a command string before actually running it on a production system. >> >> Does IBM offer "test" licenses for setting up a small debug/devel >> environment? >> >> I'd be interested to hear everyone's best practices on verifying >> commands/actions. >> >> >> Thanks, >> Brian >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- > Jez Tucker > Head of Research & Product Development > Pixit Media | ArcaStream > www.pixitmedia.com > www.arcastream.com > > > This email is confidential in that it is intended for the exclusive > attention of the addressee(s) indicated. If you are not the intended > recipient, this email should not be read or disclosed to any other > person. Please notify the sender immediately and delete this email from > your computer system. Any opinions expressed are not necessarily those > of the company from which this email was sent and, whilst to the best of > our knowledge no viruses or defects exist, no responsibility can be > accepted for any loss or damage arising from its receipt or subsequent > use of this email. > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From makaplan at us.ibm.com Tue Oct 4 23:15:32 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 4 Oct 2016 18:15:32 -0400 Subject: [gpfsug-discuss] testing commands - licensing In-Reply-To: References: Message-ID: http://www.ibm.com/developerworks/servicemanagement/sdi/spectrum-scale/resources.html ... Please contact your IBM sales representative for more information. ... http://www.ibm.com/developerworks/servicemanagement/tc/gpfs/evaluate.html https://www-01.ibm.com/marketing/iwm/iwm/web/preLogin.do?source=swerpzsw-scale42-3 The 90 day trial license includes these terms... If the Program is designated as "Non-Production", the Program can only be deployed as part of the Licensee's internal development and test environment for internal non-production activities, including but not limited to testing, performance tuning, fault diagnosis, internal benchmarking, staging, quality assurance activity and/or developing internally used additions or extensions to the Program using published application programming interfaces. Licensee is not authorized to use any part of the Program for any other purposes without acquiring the appropriate production entitlements. ... This is posting does not constitute or include the full terms of a license ... It just provides some pointers and lets you know that IBM does offer "Trial" licenses for Spectrum Scale. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aspalazz at us.ibm.com Wed Oct 5 02:22:35 2016 From: aspalazz at us.ibm.com (Aaron S Palazzolo) Date: Wed, 5 Oct 2016 01:22:35 +0000 Subject: [gpfsug-discuss] Toolkit Message-ID: An HTML attachment was scrubbed... URL: From stef.coene at docum.org Wed Oct 5 07:30:21 2016 From: stef.coene at docum.org (Stef Coene) Date: Wed, 5 Oct 2016 08:30:21 +0200 Subject: [gpfsug-discuss] Toolkit In-Reply-To: References: Message-ID: On 10/05/2016 03:22 AM, Aaron S Palazzolo wrote: > *Hi Stef,* > > Good info and thanks for the scenario specifics. I will include your > input in our upcoming design sessions and we can let you know as soon as > any functional improvements are on their way. I just realized that this also a good way to document a GPFS cluster ;) Stef From daniel.kidger at uk.ibm.com Wed Oct 5 14:27:39 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Wed, 5 Oct 2016 13:27:39 +0000 Subject: [gpfsug-discuss] testing commands In-Reply-To: References: , Message-ID: An HTML attachment was scrubbed... URL: From Richard.E.Powell at boeing.com Thu Oct 6 22:32:31 2016 From: Richard.E.Powell at boeing.com (Powell, Richard E) Date: Thu, 6 Oct 2016 21:32:31 +0000 Subject: [gpfsug-discuss] Will there be a UG meeting at SC16? Message-ID: Has there been any word on a Spectrum Scale User Group meeting in conjunction with SC16 in November? -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Thu Oct 6 22:36:18 2016 From: bbanister at jumptrading.com (Bryan Banister) Date: Thu, 6 Oct 2016 21:36:18 +0000 Subject: [gpfsug-discuss] Will there be a UG meeting at SC16? In-Reply-To: References: Message-ID: <21BC488F0AEA2245B2C3E83FC0B33DBB0634F863@CHI-EXCHANGEW1.w2k.jumptrading.com> Word on the street is that the SS UG meeting will be on Sunday, Nov 13th, in the afternoon, -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Powell, Richard E Sent: Thursday, October 06, 2016 4:33 PM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Will there be a UG meeting at SC16? Has there been any word on a Spectrum Scale User Group meeting in conjunction with SC16 in November? ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From swinglet at us.ibm.com Thu Oct 6 22:38:13 2016 From: swinglet at us.ibm.com (Teresa S Swingler) Date: Thu, 6 Oct 2016 14:38:13 -0700 Subject: [gpfsug-discuss] Thank you and Last Call - Everyday Management 5 minute Survey Message-ID: Thank you to all of you who have responded to our short survey of everyday management tasks. We value your feedback as it helps to better inform our upcoming priorities. For those of you who have not had a chance to respond yet, please take 5 minutes to fill out this quick survey: https://www.surveygizmo.com/s3/3090036/IBM-Spectrum-Scale-Everyday-Management The survey will be closed on Sunday. We appreciate your input and thank you for your time. Teresa Teresa S. Swingler IBM Systems Senior Designer | Lead UX Architect Spectrum Scale and Spectrum Control 2929 N. Central Ave. Phoenix, AZ 85012 Phone: (602) 217-2763 Tieline: 667-2763 E-mail: swinglet at us.ibm.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Thu Oct 6 22:40:33 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Thu, 6 Oct 2016 21:40:33 +0000 Subject: [gpfsug-discuss] Will there be a UG meeting at SC16? In-Reply-To: <21BC488F0AEA2245B2C3E83FC0B33DBB0634F863@CHI-EXCHANGEW1.w2k.jumptrading.com> References: , <21BC488F0AEA2245B2C3E83FC0B33DBB0634F863@CHI-EXCHANGEW1.w2k.jumptrading.com> Message-ID: That's correct Sunday (13th?). Agenda is just being finished up at the moment. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Bryan Banister [bbanister at jumptrading.com] Sent: 06 October 2016 22:36 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Will there be a UG meeting at SC16? Word on the street is that the SS UG meeting will be on Sunday, Nov 13th, in the afternoon, -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Powell, Richard E Sent: Thursday, October 06, 2016 4:33 PM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Will there be a UG meeting at SC16? Has there been any word on a Spectrum Scale User Group meeting in conjunction with SC16 in November? ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. From kraemerf at de.ibm.com Fri Oct 7 13:08:19 2016 From: kraemerf at de.ibm.com (Frank Kraemer) Date: Fri, 7 Oct 2016 14:08:19 +0200 Subject: [gpfsug-discuss] Spectrum Scale for Linux on z Systems now supports native Spectrum Scale encryption Message-ID: >Peter Muench1/Germany/IBM wrote on 07.10.2016 12:11:52: Since September 20th, Spectrum Scale for Linux on z Systems 4.2.1.1 code is available. (Download is via fixcentral) Spectrum Scale for Linux on z Systems now supports native GPFS encryption: >From the latest FAQ (see Q2.28: What are the current requirements / limitations for IBM Spectrum Scale for Linux on z Systems?) If you have further questions, please let me know. Thank you. Peter Muench Spectrum Scale for Linux on z Systems Development mailto:muenchp at de.ibm.com IBM Deutschland Research & Development GmbH -------------- next part -------------- An HTML attachment was scrubbed... URL: From volobuev at us.ibm.com Fri Oct 7 17:20:01 2016 From: volobuev at us.ibm.com (Yuri L Volobuev) Date: Fri, 7 Oct 2016 09:20:01 -0700 Subject: [gpfsug-discuss] Biggest file that will fit inside an inode? In-Reply-To: References: <1BDA1763-A6FE-4A52-B2ED-22324C9AA16F@nuance.com> Message-ID: Exactly how much data can fit into an inode depends on whether any extended attributes (EAs) are present, which makes this complicated, as EAs may be set explicitly by the user or implicitly by the system. In any inode, there's a fixed footprint of the inode header, 128 bytes on recent GPFS streams. So if there's no possibility to fit more than (inodeSize - 128) bytes into an inode. If some EAs are present, this becomes more complicated. GPFS can store EAs either in the inode, or in a special "EA overflow block". Certain EAs used by GPFS internally (for encryption, AFM, DMAPI, clones) must reside in the inode, due to the need to have those set at certain junctures when it's tricky to accommodate EA overflow allocation with proper atomicity guarantees. Other subsystems, e.g. SELinux, may implicitly use EAs as well. So a given inode may have less space for data than described above. This is another big part of the reason why 4K is the current default inode size. And of course this is not an external spec set in stone, but rather the way the current implementation works. A future version of GPFS may have a bigger inode header, for example. So it would be imprudent to bank on file/directory data fitting into an inode with a great deal of precision. A better way to view this is as a kind of a performance optimization. DMAPI won't "punch out" data that resides in inode. Such a file has zero blocks allocated, so there's nothing for an HSM app to migrate. TCT can copy in-inode data to cloud for co-resident mode. And yes, TCT doesn't rely on the same DMAPI API as HSM, per se, but naturally the underlying code shares some of the infrastructure. yuri From: Luke Raimbach To: gpfsug main discussion list , Date: 10/03/2016 09:16 AM Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? Sent by: gpfsug-discuss-bounces at spectrumscale.org It doesn't, but the end result is the same... data shipped off 'somewhere else' with a stub file? I have in my mind that DMAPI support was around before data-in-inode (or at least 4K inodes) was introduced, so it had to be made a bit cleverer to cope, but I may be misremembering that. On Mon, 3 Oct 2016 at 17:10 Simon Thompson (Research Computing - IT Services) wrote: TCT doesn't use dmapi though I thought? ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [ gpfsug-discuss-bounces at spectrumscale.org] on behalf of Luke Raimbach [ luke.raimbach at googlemail.com] Sent: 03 October 2016 17:07 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? Surely it wouldn't go? Maybe the data would get copied out rather than stubbed... DMAPI can't be stupid enough to stub data out of an inode? Can it? Interesting question. Maybe I'll test that one. On Mon, 3 Oct 2016 at 17:00 Simon Thompson (Research Computing - IT Services) > wrote: Would you tier an in-inode file to the cloud? I mean, I wouldn't tier an in-inode file out to tape? Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [ gpfsug-discuss-bounces at spectrumscale.org] on behalf of Oesterlin, Robert [Robert.Oesterlin at nuance.com] Sent: 03 October 2016 16:56 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Biggest file that will fit inside an inode? What's going be taken away if you use Encryption or Transparent Cloud Tiering? Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: > on behalf of Marc A Kaplan < makaplan at us.ibm.com> Reply-To: gpfsug main discussion list > Date: Monday, October 3, 2016 at 10:46 AM To: gpfsug main discussion list > Subject: [EXTERNAL] Re: [gpfsug-discuss] Biggest file that will fit inside an inode? 3968!! On a non-SELINUX system the answer is 3968 of data in a 4K inode, just 128 bytes of metadata. Caution: it's possible in some future release, this could change ... I don't know of any plans, I'm just saying ... Inode 16346892 [16346892] snap 0 (index 12 in block 255420): ? Inode address: 6:123049056 size 4096 nAddrs 330 ? indirectionLevel=INODE status=USERFILE ? objectVersion=1 generation=0xC0156CB nlink=1 ? owner uid=0 gid=0 mode=0200100644: -rw-r--r-- ? blocksize code=5 (32 subblocks) ? lastBlockSubblocks=0 ? checksum=0xAD8E0B4B is Valid ? fileSize=3968 nFullBlocks=0 ? currentMetadataReplicas=1 maxMetadataReplicas=2 ? currentDataReplicas=1 maxDataReplicas=2 ? ... ? Data [3968]: 0000000000000000: BCA91252 2B64BEDC A7D7BA9D D5BE8C30? *...R+d.........0* ... 0000000000000F70: DA925E2F 16A68C01 03CA5E37 08D72B7F? *..^/......^7..+.* ? trailer: is NULL _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From Richard.E.Powell at boeing.com Sat Oct 8 00:55:28 2016 From: Richard.E.Powell at boeing.com (Powell, Richard E) Date: Fri, 7 Oct 2016 23:55:28 +0000 Subject: [gpfsug-discuss] Will there be a UG meeting at SC16? Message-ID: Thanks for the info! Can you tell us when and where, yet? We're trying to make our travel plans early, (for a change:)) -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Sat Oct 8 22:57:56 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Sat, 8 Oct 2016 21:57:56 +0000 Subject: [gpfsug-discuss] Will there be a UG meeting at SC16? Message-ID: <82021AF7-4908-4D7B-B3B9-557954A692C8@nuance.com> Sunday November 13th, from about 1PM - 5:30 PM. Not sure of the location, but it will be at one of the hotels near the convention center in downtown SLC. More details soon! Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid GPFS-UG Co-Principal From: on behalf of "Powell, Richard E" Reply-To: gpfsug main discussion list Date: Friday, October 7, 2016 at 6:55 PM To: "gpfsug-discuss at spectrumscale.org" Subject: [EXTERNAL] Re: [gpfsug-discuss] Will there be a UG meeting at SC16? Thanks for the info! Can you tell us when and where, yet? We?re trying to make our travel plans early, (for a change?) -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.childs at qmul.ac.uk Mon Oct 10 13:32:26 2016 From: p.childs at qmul.ac.uk (Peter Childs) Date: Mon, 10 Oct 2016 12:32:26 +0000 Subject: [gpfsug-discuss] GPFS Upgrade 3.5 -> 4.1 Message-ID: We are finishing upgrading our GPFS cluster of around 250 (client) nodes from GPFS 3.5.0.31 to Spectrum Scale 4.1.1.8, and have just about upgraded all the computers. We are looking at running the "mmchfs -V LATEST" step and where wondering how much io this takes and if it was likely to interrupt service? We are looking at upgrading to 4.2 but plan to do that via Multi-cluster and AFM as we are integrating new hardware and wish to increase the block and inode size at the same time. Peter Childs Research Storage Expert ITS Research Infrastructure Queen Mary, University of London From robert at strubi.ox.ac.uk Mon Oct 10 14:21:30 2016 From: robert at strubi.ox.ac.uk (Robert Esnouf) Date: Mon, 10 Oct 2016 14:21:30 +0100 (BST) Subject: [gpfsug-discuss] NEW DEPARTMENT/NEW POST: Head of Research Computing at the Oxford Big Data Institute Message-ID: <201610101321.061282@mail.strubi.ox.ac.uk> Dear All, PLEASE FEEL FREE TO PASS THIS EMAIL ON TO OTHER RELEVANT MAILING LISTS AND ANYONE WHO MIGHT BE INTERESTED The University of Oxford Big Data Institute is looking for a highly skilled research computing manager to establish a significant computing capability in Oxford over the next few years. The job advertisement can be found here: http://www.jobs.ac.uk/job/AUT085/head-of-bdi-research-computing/ The post is not just traditional HPC as it involves creating a large infrastructure for distributed, collaborative processing of sensitive data. If anyone wants an informal chat about how this role could develop then please drop me an email and I'll try to help. Regards, Robert -- Dr. Robert Esnouf, University Research Lecturer, Head of Research Computing Core, NDM Research Computing Strategy Officer Room 10/028, Wellcome Trust Centre for Human Genetics, Old Road Campus, Roosevelt Drive, Oxford OX3 7BN, UK Email: robert at strubi.ox.ac.uk / robert at well.ox.ac.uk Tel: (+44) - 1865 - 287783 From janfrode at tanso.net Mon Oct 10 15:35:02 2016 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Mon, 10 Oct 2016 14:35:02 +0000 Subject: [gpfsug-discuss] GPFS Upgrade 3.5 -> 4.1 In-Reply-To: References: Message-ID: I've also always been worried about that one, but never experienced it taking any time, I/O or interruption. I've the interpreted it to just start using new features, but not really changing anything with the existing metadata. Things needing on disk changes are probably put in mmmigratefs I have not heard about anything needing mmmigratefs since GPFS v3.3 (fs version 11.03) added fast extended attributes. Would be great to hear otherwize, or confirmations. -jf man. 10. okt. 2016 kl. 14.32 skrev Peter Childs : > We are finishing upgrading our GPFS cluster of around 250 (client) nodes > from GPFS 3.5.0.31 to Spectrum Scale 4.1.1.8, and have just about upgraded > all the computers. > > We are looking at running the "mmchfs -V LATEST" step and where wondering > how much io this takes and if it was likely to interrupt service? > > We are looking at upgrading to 4.2 but plan to do that via Multi-cluster > and AFM as we are integrating new hardware and wish to increase the block > and inode size at the same time. > > Peter Childs > Research Storage Expert > ITS Research Infrastructure > Queen Mary, University of London > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From volobuev at us.ibm.com Mon Oct 10 16:39:44 2016 From: volobuev at us.ibm.com (Yuri L Volobuev) Date: Mon, 10 Oct 2016 08:39:44 -0700 Subject: [gpfsug-discuss] GPFS Upgrade 3.5 -> 4.1 In-Reply-To: References: Message-ID: Correct. mmchfs -V only done quick operations (that can be easily undone if something goes wrong). Essentially the big task here is to increase on-disk file system descriptor version number, to allow using those features that require a higher version. Bigger "conversion"-style tasks belong in mmmigratefs. The only way to increase the inode size and the data block size is to format a new file system. This cannot be done on an existing file system. yuri From: Jan-Frode Myklebust To: gpfsug main discussion list , Date: 10/10/2016 07:35 AM Subject: Re: [gpfsug-discuss] GPFS Upgrade 3.5 -> 4.1 Sent by: gpfsug-discuss-bounces at spectrumscale.org I've also always been worried about that one, but never experienced it taking any time, I/O or interruption. I've the interpreted it to just start using new features, but not really changing anything with the existing metadata. Things needing on disk changes are probably put in mmmigratefs I have not heard about anything needing mmmigratefs since GPFS v3.3 (fs version 11.03) added fast extended attributes. Would be great to hear otherwize, or confirmations. -jf man. 10. okt. 2016 kl. 14.32 skrev Peter Childs : We are finishing upgrading our GPFS cluster of around 250 (client) nodes from GPFS 3.5.0.31 to Spectrum Scale 4.1.1.8, and have just about upgraded all the computers. We are looking at running the "mmchfs -V LATEST" step and where wondering how much io this takes and if it was likely to interrupt service? We are looking at upgrading to 4.2 but plan to do that via Multi-cluster and AFM as we are integrating new hardware and wish to increase the block and inode size at the same time. Peter Childs Research Storage Expert ITS Research Infrastructure Queen Mary, University of London _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From p.childs at qmul.ac.uk Mon Oct 10 19:11:31 2016 From: p.childs at qmul.ac.uk (Peter Childs) Date: Mon, 10 Oct 2016 18:11:31 +0000 Subject: [gpfsug-discuss] GPFS Upgrade 3.5 -> 4.1 In-Reply-To: References: , Message-ID: <4i99d985lghdmb5sptsfjp2n.1476123091191@email.android.com> So in short we're saying, "mmchfs -V LATEST" increments a version number and allows new features to become possible, it does not start using them straight away. Hence Directories will shrink in 4.1 but you need to run "mmchattr --compact" on all the old ones before anything actually changes. (new ones are fine) increasing the version number makes this possible but it does not actually do it, as doing it would mean walking every directory and updating stuff. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ---- Yuri L Volobuev wrote ---- Correct. mmchfs -V only done quick operations (that can be easily undone if something goes wrong). Essentially the big task here is to increase on-disk file system descriptor version number, to allow using those features that require a higher version. Bigger "conversion"-style tasks belong in mmmigratefs. The only way to increase the inode size and the data block size is to format a new file system. This cannot be done on an existing file system. yuri [Inactive hide details for Jan-Frode Myklebust ---10/10/2016 07:35:48 AM---I've also always been worried about that one, but nev]Jan-Frode Myklebust ---10/10/2016 07:35:48 AM---I've also always been worried about that one, but never experienced it taking any time, I/O or inter From: Jan-Frode Myklebust To: gpfsug main discussion list , Date: 10/10/2016 07:35 AM Subject: Re: [gpfsug-discuss] GPFS Upgrade 3.5 -> 4.1 Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ I've also always been worried about that one, but never experienced it taking any time, I/O or interruption. I've the interpreted it to just start using new features, but not really changing anything with the existing metadata. Things needing on disk changes are probably put in mmmigratefs I have not heard about anything needing mmmigratefs since GPFS v3.3 (fs version 11.03) added fast extended attributes. Would be great to hear otherwize, or confirmations. -jf man. 10. okt. 2016 kl. 14.32 skrev Peter Childs >: We are finishing upgrading our GPFS cluster of around 250 (client) nodes from GPFS 3.5.0.31 to Spectrum Scale 4.1.1.8, and have just about upgraded all the computers. We are looking at running the "mmchfs -V LATEST" step and where wondering how much io this takes and if it was likely to interrupt service? We are looking at upgrading to 4.2 but plan to do that via Multi-cluster and AFM as we are integrating new hardware and wish to increase the block and inode size at the same time. Peter Childs Research Storage Expert ITS Research Infrastructure Queen Mary, University of London _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: graycol.gif URL: From Mark.Bush at siriuscom.com Mon Oct 10 20:56:22 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Mon, 10 Oct 2016 19:56:22 +0000 Subject: [gpfsug-discuss] Hardware refresh Message-ID: <0663AFB2-7F70-4167-A2D1-F96B5394EBD2@siriuscom.com> Have a very old cluster built on IBM X3650?s and DS3500. Need to refresh hardware. Any lessons learned in this process? Is it easiest to just build new cluster and then use AFM? Add to existing cluster then decommission nodes? What is the recommended process for this? Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Mon Oct 10 21:04:36 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Mon, 10 Oct 2016 20:04:36 +0000 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: <0663AFB2-7F70-4167-A2D1-F96B5394EBD2@siriuscom.com> References: <0663AFB2-7F70-4167-A2D1-F96B5394EBD2@siriuscom.com> Message-ID: Hi Mark, The last time we did something like this was 2010 (we?re doing rolling refreshes now), so there are probably lots of better ways to do this than what we did, but we: 1) set up the new hardware 2) created new filesystems (so that we could make adjustments we wanted to make that can only be made at FS creation time) 3) used rsync to make a 1st pass copy of everything 4) coordinated a time with users / groups to do a 2nd rsync when they weren?t active 5) used symbolic links during the transition (i.e. rm -rvf /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) 6) once everybody was migrated, updated the symlinks (i.e. /home became a symlink to /gpfs2/home) HTHAL? Kevin On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com wrote: Have a very old cluster built on IBM X3650?s and DS3500. Need to refresh hardware. Any lessons learned in this process? Is it easiest to just build new cluster and then use AFM? Add to existing cluster then decommission nodes? What is the recommended process for this? Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From usa-principal at gpfsug.org Mon Oct 10 23:59:46 2016 From: usa-principal at gpfsug.org (GPFS UG USA Principal) Date: Mon, 10 Oct 2016 18:59:46 -0400 Subject: [gpfsug-discuss] SC16 Draft Agenda [Save the date: Sunday, November 13th] Message-ID: Hello all, There have been some questions about the Spectrum Scale Users Group event for SC16 and we thought it best to publish our draft agenda and continue to fill it in as details get settled. That way you can make your travel plans and schedules. The date and rough times are set, we may adjust the time range a bit depending upon the number of user talks that people would like to give. So note, yes! we are asking for user talks. Please consider doing this. We always receive positive feedback about talks given by users that describe real world experiences. If any of the user talk slots below are of interest to you, please let us know as soon as you can. We may turn at least one user talk into a panel if we can get enough participants. As always, your feedback is welcome. This is a users group after all. So, here?s what we have planned so far: Date: Sunday November 13th Venue: TBD, but near the convention center in downtown SLC Time: ~12:30p - ~17:30p Agenda: 12:30 - 12:50 Welcome / Overview 12:50 - 13:50 The latest in IBM Spectrum Scale and ESS 13:50 - 14:20 User Talk: Big Data in HPC 14:20 - 14:50 User Talk: Tiering to Object Storage 14:50 - 15:20 - break - 15:20 - 15:50 User Talk: TBD ? volunteers! please! 15:50 - 16:35 Spectrum Scale Enhancements for CORAL 16:35 - 17:20 News from IBM Research 17:20 - 17:30 Closing Best, Kristy & Bob Kristy Kallback-Rose Manager, Research Storage Indiana University -------------- next part -------------- An HTML attachment was scrubbed... URL: From luis.bolinches at fi.ibm.com Tue Oct 11 00:40:54 2016 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Mon, 10 Oct 2016 23:40:54 +0000 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: Message-ID: Hi Creating a new FS sounds like a best way to go. NSDv2 being a very good reason to do so. AFM for migrations is quite good, latest versions allows to use NSD protocol for mounts as well. Olaf did a great job explaining this scenario on the redbook chapter 6 http://www.redbooks.ibm.com/abstracts/sg248254.html?Open -- Cheers > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L wrote: > > Hi Mark, > > The last time we did something like this was 2010 (we?re doing rolling refreshes now), so there are probably lots of better ways to do this than what we did, but we: > > 1) set up the new hardware > 2) created new filesystems (so that we could make adjustments we wanted to make that can only be made at FS creation time) > 3) used rsync to make a 1st pass copy of everything > 4) coordinated a time with users / groups to do a 2nd rsync when they weren?t active > 5) used symbolic links during the transition (i.e. rm -rvf /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) > 6) once everybody was migrated, updated the symlinks (i.e. /home became a symlink to /gpfs2/home) > > HTHAL? > > Kevin > >> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com wrote: >> >> Have a very old cluster built on IBM X3650?s and DS3500. Need to refresh hardware. Any lessons learned in this process? Is it easiest to just build new cluster and then use AFM? Add to existing cluster then decommission nodes? What is the recommended process for this? >> >> >> Mark >> This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. >> >> Sirius Computer Solutions >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and Education > Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 > > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Tue Oct 11 00:45:04 2016 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Mon, 10 Oct 2016 19:45:04 -0400 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: References: Message-ID: Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron On 10/10/16 7:40 PM, Luis Bolinches wrote: > Hi > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > reason to do so. > > AFM for migrations is quite good, latest versions allows to use NSD > protocol for mounts as well. Olaf did a great job explaining this > scenario on the redbook chapter 6 > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > -- > Cheers > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > wrote: > >> Hi Mark, >> >> The last time we did something like this was 2010 (we?re doing rolling >> refreshes now), so there are probably lots of better ways to do this >> than what we did, but we: >> >> 1) set up the new hardware >> 2) created new filesystems (so that we could make adjustments we >> wanted to make that can only be made at FS creation time) >> 3) used rsync to make a 1st pass copy of everything >> 4) coordinated a time with users / groups to do a 2nd rsync when they >> weren?t active >> 5) used symbolic links during the transition (i.e. rm -rvf >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >> 6) once everybody was migrated, updated the symlinks (i.e. /home >> became a symlink to /gpfs2/home) >> >> HTHAL? >> >> Kevin >> >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>> wrote: >>> >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>> refresh hardware. Any lessons learned in this process? Is it >>> easiest to just build new cluster and then use AFM? Add to existing >>> cluster then decommission nodes? What is the recommended process for >>> this? >>> >>> >>> Mark >>> >>> This message (including any attachments) is intended only for the use >>> of the individual or entity to which it is addressed and may contain >>> information that is non-public, proprietary, privileged, >>> confidential, and exempt from disclosure under applicable law. If you >>> are not the intended recipient, you are hereby notified that any use, >>> dissemination, distribution, or copying of this communication is >>> strictly prohibited. This message may be viewed by parties at Sirius >>> Computer Solutions other than those named in the message header. This >>> message does not contain an official representation of Sirius >>> Computer Solutions. If you have received this communication in error, >>> notify Sirius Computer Solutions immediately and (i) destroy this >>> message if a facsimile or (ii) delete this message immediately if >>> this is an electronic communication. Thank you. >>> >>> Sirius Computer Solutions >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and >> Education >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From volobuev at us.ibm.com Tue Oct 11 18:22:17 2016 From: volobuev at us.ibm.com (Yuri L Volobuev) Date: Tue, 11 Oct 2016 10:22:17 -0700 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: References: Message-ID: This depends on the committed cluster version level (minReleaseLevel) and file system format. Since NFSv2 is an on-disk format change, older code wouldn't be able to understand what it is, and thus if there's a possibility of a downlevel node looking at the NSD, the NFSv1 format is going to be used. The code does NSDv1<->NSDv2 conversions under the covers as needed when adding an empty NSD to a file system. I'd strongly recommend getting a fresh start by formatting a new file system. Many things have changed over the course of the last few years. In particular, having a 4K-aligned file system can be a pretty big deal, depending on what hardware one is going to deploy in the future, and this is something that can't be bolted onto an existing file system. Having 4K inodes is very handy for many reasons. New directory format and NSD format changes are attractive, too. And disks generally tend to get larger with time, and at some point you may want to add a disk to an existing storage pool that's larger than the existing allocation map format allows. Obviously, it's more hassle to migrate data to a new file system, as opposed to extending an existing one. In a perfect world, GPFS would offer a conversion tool that seamlessly and robustly converts old file systems, making them as good as new, but in the real world such a tool doesn't exist. Getting a clean slate by formatting a new file system every few years is a good long-term investment of time, although it comes front-loaded with extra work. yuri From: Aaron Knister To: , Date: 10/10/2016 04:45 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron On 10/10/16 7:40 PM, Luis Bolinches wrote: > Hi > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > reason to do so. > > AFM for migrations is quite good, latest versions allows to use NSD > protocol for mounts as well. Olaf did a great job explaining this > scenario on the redbook chapter 6 > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > -- > Cheers > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > wrote: > >> Hi Mark, >> >> The last time we did something like this was 2010 (we?re doing rolling >> refreshes now), so there are probably lots of better ways to do this >> than what we did, but we: >> >> 1) set up the new hardware >> 2) created new filesystems (so that we could make adjustments we >> wanted to make that can only be made at FS creation time) >> 3) used rsync to make a 1st pass copy of everything >> 4) coordinated a time with users / groups to do a 2nd rsync when they >> weren?t active >> 5) used symbolic links during the transition (i.e. rm -rvf >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >> 6) once everybody was migrated, updated the symlinks (i.e. /home >> became a symlink to /gpfs2/home) >> >> HTHAL? >> >> Kevin >> >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>> wrote: >>> >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>> refresh hardware. Any lessons learned in this process? Is it >>> easiest to just build new cluster and then use AFM? Add to existing >>> cluster then decommission nodes? What is the recommended process for >>> this? >>> >>> >>> Mark >>> >>> This message (including any attachments) is intended only for the use >>> of the individual or entity to which it is addressed and may contain >>> information that is non-public, proprietary, privileged, >>> confidential, and exempt from disclosure under applicable law. If you >>> are not the intended recipient, you are hereby notified that any use, >>> dissemination, distribution, or copying of this communication is >>> strictly prohibited. This message may be viewed by parties at Sirius >>> Computer Solutions other than those named in the message header. This >>> message does not contain an official representation of Sirius >>> Computer Solutions. If you have received this communication in error, >>> notify Sirius Computer Solutions immediately and (i) destroy this >>> message if a facsimile or (ii) delete this message immediately if >>> this is an electronic communication. Thank you. >>> >>> Sirius Computer Solutions >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and >> Education >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From Mark.Bush at siriuscom.com Tue Oct 11 20:34:27 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Tue, 11 Oct 2016 19:34:27 +0000 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: References: Message-ID: <9E337B56-A274-41B6-84C2-7084AAB771F4@siriuscom.com> Ok. I think I am hearing that a new cluster with a new FS and copying data from old to new cluster is the best way forward. Thanks everyone for your input. From: on behalf of Yuri L Volobuev Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 12:22 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh This depends on the committed cluster version level (minReleaseLevel) and file system format. Since NFSv2 is an on-disk format change, older code wouldn't be able to understand what it is, and thus if there's a possibility of a downlevel node looking at the NSD, the NFSv1 format is going to be used. The code does NSDv1<->NSDv2 conversions under the covers as needed when adding an empty NSD to a file system. I'd strongly recommend getting a fresh start by formatting a new file system. Many things have changed over the course of the last few years. In particular, having a 4K-aligned file system can be a pretty big deal, depending on what hardware one is going to deploy in the future, and this is something that can't be bolted onto an existing file system. Having 4K inodes is very handy for many reasons. New directory format and NSD format changes are attractive, too. And disks generally tend to get larger with time, and at some point you may want to add a disk to an existing storage pool that's larger than the existing allocation map format allows. Obviously, it's more hassle to migrate data to a new file system, as opposed to extending an existing one. In a perfect world, GPFS would offer a conversion tool that seamlessly and robustly converts old file systems, making them as good as new, but in the real world such a tool doesn't exist. Getting a clean slate by formatting a new file system every few years is a good long-term investment of time, although it comes front-loaded with extra work. yuri [nactive hide details for Aaron Knister ---10/10/2016 04:45:31 PM---Can on]Aaron Knister ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron From: Aaron Knister To: , Date: 10/10/2016 04:45 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron On 10/10/16 7:40 PM, Luis Bolinches wrote: > Hi > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > reason to do so. > > AFM for migrations is quite good, latest versions allows to use NSD > protocol for mounts as well. Olaf did a great job explaining this > scenario on the redbook chapter 6 > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > -- > Cheers > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > wrote: > >> Hi Mark, >> >> The last time we did something like this was 2010 (we?re doing rolling >> refreshes now), so there are probably lots of better ways to do this >> than what we did, but we: >> >> 1) set up the new hardware >> 2) created new filesystems (so that we could make adjustments we >> wanted to make that can only be made at FS creation time) >> 3) used rsync to make a 1st pass copy of everything >> 4) coordinated a time with users / groups to do a 2nd rsync when they >> weren?t active >> 5) used symbolic links during the transition (i.e. rm -rvf >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >> 6) once everybody was migrated, updated the symlinks (i.e. /home >> became a symlink to /gpfs2/home) >> >> HTHAL? >> >> Kevin >> >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>> wrote: >>> >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>> refresh hardware. Any lessons learned in this process? Is it >>> easiest to just build new cluster and then use AFM? Add to existing >>> cluster then decommission nodes? What is the recommended process for >>> this? >>> >>> >>> Mark >>> >>> This message (including any attachments) is intended only for the use >>> of the individual or entity to which it is addressed and may contain >>> information that is non-public, proprietary, privileged, >>> confidential, and exempt from disclosure under applicable law. If you >>> are not the intended recipient, you are hereby notified that any use, >>> dissemination, distribution, or copying of this communication is >>> strictly prohibited. This message may be viewed by parties at Sirius >>> Computer Solutions other than those named in the message header. This >>> message does not contain an official representation of Sirius >>> Computer Solutions. If you have received this communication in error, >>> notify Sirius Computer Solutions immediately and (i) destroy this >>> message if a facsimile or (ii) delete this message immediately if >>> this is an electronic communication. Thank you. >>> >>> Sirius Computer Solutions >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and >> Education >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 106 bytes Desc: image001.gif URL: From makaplan at us.ibm.com Tue Oct 11 20:58:53 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 11 Oct 2016 15:58:53 -0400 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: <9E337B56-A274-41B6-84C2-7084AAB771F4@siriuscom.com> References: <9E337B56-A274-41B6-84C2-7084AAB771F4@siriuscom.com> Message-ID: New FS? Yes there are some good reasons. New cluster? I did not see a compelling argument either way. From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Date: 10/11/2016 03:34 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org Ok. I think I am hearing that a new cluster with a new FS and copying data from old to new cluster is the best way forward. Thanks everyone for your input. From: on behalf of Yuri L Volobuev Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 12:22 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh This depends on the committed cluster version level (minReleaseLevel) and file system format. Since NFSv2 is an on-disk format change, older code wouldn't be able to understand what it is, and thus if there's a possibility of a downlevel node looking at the NSD, the NFSv1 format is going to be used. The code does NSDv1<->NSDv2 conversions under the covers as needed when adding an empty NSD to a file system. I'd strongly recommend getting a fresh start by formatting a new file system. Many things have changed over the course of the last few years. In particular, having a 4K-aligned file system can be a pretty big deal, depending on what hardware one is going to deploy in the future, and this is something that can't be bolted onto an existing file system. Having 4K inodes is very handy for many reasons. New directory format and NSD format changes are attractive, too. And disks generally tend to get larger with time, and at some point you may want to add a disk to an existing storage pool that's larger than the existing allocation map format allows. Obviously, it's more hassle to migrate data to a new file system, as opposed to extending an existing one. In a perfect world, GPFS would offer a conversion tool that seamlessly and robustly converts old file systems, making them as good as new, but in the real world such a tool doesn't exist. Getting a clean slate by formatting a new file system every few years is a good long-term investment of time, although it comes front-loaded with extra work. yuri Aaron Knister ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron From: Aaron Knister To: , Date: 10/10/2016 04:45 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron On 10/10/16 7:40 PM, Luis Bolinches wrote: > Hi > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > reason to do so. > > AFM for migrations is quite good, latest versions allows to use NSD > protocol for mounts as well. Olaf did a great job explaining this > scenario on the redbook chapter 6 > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > -- > Cheers > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > wrote: > >> Hi Mark, >> >> The last time we did something like this was 2010 (we?re doing rolling >> refreshes now), so there are probably lots of better ways to do this >> than what we did, but we: >> >> 1) set up the new hardware >> 2) created new filesystems (so that we could make adjustments we >> wanted to make that can only be made at FS creation time) >> 3) used rsync to make a 1st pass copy of everything >> 4) coordinated a time with users / groups to do a 2nd rsync when they >> weren?t active >> 5) used symbolic links during the transition (i.e. rm -rvf >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >> 6) once everybody was migrated, updated the symlinks (i.e. /home >> became a symlink to /gpfs2/home) >> >> HTHAL? >> >> Kevin >> >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>> wrote: >>> >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>> refresh hardware. Any lessons learned in this process? Is it >>> easiest to just build new cluster and then use AFM? Add to existing >>> cluster then decommission nodes? What is the recommended process for >>> this? >>> >>> >>> Mark >>> >>> This message (including any attachments) is intended only for the use >>> of the individual or entity to which it is addressed and may contain >>> information that is non-public, proprietary, privileged, >>> confidential, and exempt from disclosure under applicable law. If you >>> are not the intended recipient, you are hereby notified that any use, >>> dissemination, distribution, or copying of this communication is >>> strictly prohibited. This message may be viewed by parties at Sirius >>> Computer Solutions other than those named in the message header. This >>> message does not contain an official representation of Sirius >>> Computer Solutions. If you have received this communication in error, >>> notify Sirius Computer Solutions immediately and (i) destroy this >>> message if a facsimile or (ii) delete this message immediately if >>> this is an electronic communication. Thank you. >>> >>> Sirius Computer Solutions >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and >> Education >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 106 bytes Desc: not available URL: From p.childs at qmul.ac.uk Tue Oct 11 22:25:50 2016 From: p.childs at qmul.ac.uk (Peter Childs) Date: Tue, 11 Oct 2016 21:25:50 +0000 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: References: <9E337B56-A274-41B6-84C2-7084AAB771F4@siriuscom.com>, Message-ID: <7lb12taeugiu5fh1vd718dp5.1476220357109@email.android.com> My reading is, If you are running a small cluster with tie-breaker disks and your wanting to change the manager servers, or you want to switch to using the new config management method in v4 then new cluster, and use multicluster to upgrade. Otherwise just use a new Filesystem within the old cluster. But I'm interested to hear otherwise, as I'm about to embark on this myself. I note you can switch an old cluster but need to shutdown to do so. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ---- Marc A Kaplan wrote ---- New FS? Yes there are some good reasons. New cluster? I did not see a compelling argument either way. From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Date: 10/11/2016 03:34 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Ok. I think I am hearing that a new cluster with a new FS and copying data from old to new cluster is the best way forward. Thanks everyone for your input. From: on behalf of Yuri L Volobuev Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 12:22 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh This depends on the committed cluster version level (minReleaseLevel) and file system format. Since NFSv2 is an on-disk format change, older code wouldn't be able to understand what it is, and thus if there's a possibility of a downlevel node looking at the NSD, the NFSv1 format is going to be used. The code does NSDv1<->NSDv2 conversions under the covers as needed when adding an empty NSD to a file system. I'd strongly recommend getting a fresh start by formatting a new file system. Many things have changed over the course of the last few years. In particular, having a 4K-aligned file system can be a pretty big deal, depending on what hardware one is going to deploy in the future, and this is something that can't be bolted onto an existing file system. Having 4K inodes is very handy for many reasons. New directory format and NSD format changes are attractive, too. And disks generally tend to get larger with time, and at some point you may want to add a disk to an existing storage pool that's larger than the existing allocation map format allows. Obviously, it's more hassle to migrate data to a new file system, as opposed to extending an existing one. In a perfect world, GPFS would offer a conversion tool that seamlessly and robustly converts old file systems, making them as good as new, but in the real world such a tool doesn't exist. Getting a clean slate by formatting a new file system every few years is a good long-term investment of time, although it comes front-loaded with extra work. yuri [nactive hide details for Aaron Knister ---10/10/2016 04:45:31 PM---Can on]Aaron Knister ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron From: Aaron Knister To: , Date: 10/10/2016 04:45 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron On 10/10/16 7:40 PM, Luis Bolinches wrote: > Hi > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > reason to do so. > > AFM for migrations is quite good, latest versions allows to use NSD > protocol for mounts as well. Olaf did a great job explaining this > scenario on the redbook chapter 6 > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > -- > Cheers > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > wrote: > >> Hi Mark, >> >> The last time we did something like this was 2010 (we?re doing rolling >> refreshes now), so there are probably lots of better ways to do this >> than what we did, but we: >> >> 1) set up the new hardware >> 2) created new filesystems (so that we could make adjustments we >> wanted to make that can only be made at FS creation time) >> 3) used rsync to make a 1st pass copy of everything >> 4) coordinated a time with users / groups to do a 2nd rsync when they >> weren?t active >> 5) used symbolic links during the transition (i.e. rm -rvf >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >> 6) once everybody was migrated, updated the symlinks (i.e. /home >> became a symlink to /gpfs2/home) >> >> HTHAL? >> >> Kevin >> >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>> wrote: >>> >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>> refresh hardware. Any lessons learned in this process? Is it >>> easiest to just build new cluster and then use AFM? Add to existing >>> cluster then decommission nodes? What is the recommended process for >>> this? >>> >>> >>> Mark >>> >>> This message (including any attachments) is intended only for the use >>> of the individual or entity to which it is addressed and may contain >>> information that is non-public, proprietary, privileged, >>> confidential, and exempt from disclosure under applicable law. If you >>> are not the intended recipient, you are hereby notified that any use, >>> dissemination, distribution, or copying of this communication is >>> strictly prohibited. This message may be viewed by parties at Sirius >>> Computer Solutions other than those named in the message header. This >>> message does not contain an official representation of Sirius >>> Computer Solutions. If you have received this communication in error, >>> notify Sirius Computer Solutions immediately and (i) destroy this >>> message if a facsimile or (ii) delete this message immediately if >>> this is an electronic communication. Thank you. >>> >>> Sirius Computer Solutions >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and >> Education >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00001.gif Type: image/gif Size: 106 bytes Desc: ATT00001.gif URL: From aaron.s.knister at nasa.gov Tue Oct 11 23:41:36 2016 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 11 Oct 2016 18:41:36 -0400 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: References: Message-ID: Thanks Yuri. I'm asking for my own purposes but I think it's still relevant here: we're still at GPFS 3.5 and will be adding dataOnly NSDs with 4K sectors in the near future. We're planning to update to 4.1 before we format these NSDs, though. If I understand you correctly we can't bring these 4K NSDv2 NSDs into a filesystem with 512b-based NSDv1 NSDs? That's a pretty big deal :( Reformatting every few years with 10's of petabytes of data is not realistic for us (it would take years to move the data around). It also goes against my personal preachings about GPFS's storage virtualization capabilities: the ability to perform upgrades/make underlying storage infrastructure changes with behind-the-scenes data migration, eliminating much of the manual hassle of storage administrators doing rsync dances. I guess it's RFE time? It also seems as though AFM could help with automating the migration, although many of our filesystems do not have filesets on them so we would have to re-think how we lay out our filesystems. This is also curious to me with IBM pitching GPFS as a filesystem for cloud services (the cloud *never* goes down, right?). Granted I believe this pitch started after the NSDv2 format was defined, but if somebody is building a large cloud with GPFS as the underlying filesystem for an object or an image store one might think the idea of having to re-format the filesystem to gain access to critical new features is inconsistent with this pitch. It would be hugely impactful. Just my $.02. As you can tell, I'm frustrated there's no online conversion tool :) Not that there couldn't be... you all are brilliant developers. -Aaron On 10/11/16 1:22 PM, Yuri L Volobuev wrote: > This depends on the committed cluster version level (minReleaseLevel) > and file system format. Since NFSv2 is an on-disk format change, older > code wouldn't be able to understand what it is, and thus if there's a > possibility of a downlevel node looking at the NSD, the NFSv1 format is > going to be used. The code does NSDv1<->NSDv2 conversions under the > covers as needed when adding an empty NSD to a file system. > > I'd strongly recommend getting a fresh start by formatting a new file > system. Many things have changed over the course of the last few years. > In particular, having a 4K-aligned file system can be a pretty big deal, > depending on what hardware one is going to deploy in the future, and > this is something that can't be bolted onto an existing file system. > Having 4K inodes is very handy for many reasons. New directory format > and NSD format changes are attractive, too. And disks generally tend to > get larger with time, and at some point you may want to add a disk to an > existing storage pool that's larger than the existing allocation map > format allows. Obviously, it's more hassle to migrate data to a new file > system, as opposed to extending an existing one. In a perfect world, > GPFS would offer a conversion tool that seamlessly and robustly converts > old file systems, making them as good as new, but in the real world such > a tool doesn't exist. Getting a clean slate by formatting a new file > system every few years is a good long-term investment of time, although > it comes front-loaded with extra work. > > yuri > > Inactive hide details for Aaron Knister ---10/10/2016 04:45:31 PM---Can > one format NSDv2 NSDs and put them in a filesystem withAaron Knister > ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a > filesystem with NSDv1 NSD's? -Aaron > > From: Aaron Knister > To: , > Date: 10/10/2016 04:45 PM > Subject: Re: [gpfsug-discuss] Hardware refresh > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > ------------------------------------------------------------------------ > > > > Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? > > -Aaron > > On 10/10/16 7:40 PM, Luis Bolinches wrote: >> Hi >> >> Creating a new FS sounds like a best way to go. NSDv2 being a very good >> reason to do so. >> >> AFM for migrations is quite good, latest versions allows to use NSD >> protocol for mounts as well. Olaf did a great job explaining this >> scenario on the redbook chapter 6 >> >> http://www.redbooks.ibm.com/abstracts/sg248254.html?Open >> >> -- >> Cheers >> >> On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L >> > > wrote: >> >>> Hi Mark, >>> >>> The last time we did something like this was 2010 (we?re doing rolling >>> refreshes now), so there are probably lots of better ways to do this >>> than what we did, but we: >>> >>> 1) set up the new hardware >>> 2) created new filesystems (so that we could make adjustments we >>> wanted to make that can only be made at FS creation time) >>> 3) used rsync to make a 1st pass copy of everything >>> 4) coordinated a time with users / groups to do a 2nd rsync when they >>> weren?t active >>> 5) used symbolic links during the transition (i.e. rm -rvf >>> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >>> 6) once everybody was migrated, updated the symlinks (i.e. /home >>> became a symlink to /gpfs2/home) >>> >>> HTHAL? >>> >>> Kevin >>> >>>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>>> wrote: >>>> >>>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>>> refresh hardware. Any lessons learned in this process? Is it >>>> easiest to just build new cluster and then use AFM? Add to existing >>>> cluster then decommission nodes? What is the recommended process for >>>> this? >>>> >>>> >>>> Mark >>>> >>>> This message (including any attachments) is intended only for the use >>>> of the individual or entity to which it is addressed and may contain >>>> information that is non-public, proprietary, privileged, >>>> confidential, and exempt from disclosure under applicable law. If you >>>> are not the intended recipient, you are hereby notified that any use, >>>> dissemination, distribution, or copying of this communication is >>>> strictly prohibited. This message may be viewed by parties at Sirius >>>> Computer Solutions other than those named in the message header. This >>>> message does not contain an official representation of Sirius >>>> Computer Solutions. If you have received this communication in error, >>>> notify Sirius Computer Solutions immediately and (i) destroy this >>>> message if a facsimile or (ii) delete this message immediately if >>>> this is an electronic communication. Thank you. >>>> >>>> Sirius Computer Solutions >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> ? >>> Kevin Buterbaugh - Senior System Administrator >>> Vanderbilt University - Advanced Computing Center for Research and >>> Education >>> Kevin.Buterbaugh at vanderbilt.edu >>> - (615)875-9633 >>> >>> >>> >> >> Ellei edell? ole toisin mainittu: / Unless stated otherwise above: >> Oy IBM Finland Ab >> PL 265, 00101 Helsinki, Finland >> Business ID, Y-tunnus: 0195876-3 >> Registered in Finland >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From aaron.s.knister at nasa.gov Wed Oct 12 00:57:38 2016 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 11 Oct 2016 19:57:38 -0400 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: References: Message-ID: <63bd99ad-f684-d1bb-371f-e2c21849ff29@nasa.gov> I think I was a little quick to the trigger. I re-read your last mail after doing some testing and understand it differently. I was wrong about my interpretation-- you can add 4K NSDv2 formatted NSDs to a filesystem previously created with NSDv1 NSDs assuming, as you say, the minReleaseLevel and filesystem version are high enough. That negates about half of my last e-mail. The fs still doesn't show as 4K aligned: loressd01:~ # /usr/lpp/mmfs/bin/mmlsfs tnb4k --is4KAligned flag value description ------------------- ------------------------ ----------------------------------- --is4KAligned No is4KAligned? but *shrug* most of the I/O to these disks should be 1MB anyway. If somebody is pounding the FS with smaller than 4K I/O they're gonna get a talkin' to. -Aaron On 10/11/16 6:41 PM, Aaron Knister wrote: > Thanks Yuri. > > I'm asking for my own purposes but I think it's still relevant here: > we're still at GPFS 3.5 and will be adding dataOnly NSDs with 4K sectors > in the near future. We're planning to update to 4.1 before we format > these NSDs, though. If I understand you correctly we can't bring these > 4K NSDv2 NSDs into a filesystem with 512b-based NSDv1 NSDs? That's a > pretty big deal :( > > Reformatting every few years with 10's of petabytes of data is not > realistic for us (it would take years to move the data around). It also > goes against my personal preachings about GPFS's storage virtualization > capabilities: the ability to perform upgrades/make underlying storage > infrastructure changes with behind-the-scenes data migration, > eliminating much of the manual hassle of storage administrators doing > rsync dances. I guess it's RFE time? It also seems as though AFM could > help with automating the migration, although many of our filesystems do > not have filesets on them so we would have to re-think how we lay out > our filesystems. > > This is also curious to me with IBM pitching GPFS as a filesystem for > cloud services (the cloud *never* goes down, right?). Granted I believe > this pitch started after the NSDv2 format was defined, but if somebody > is building a large cloud with GPFS as the underlying filesystem for an > object or an image store one might think the idea of having to re-format > the filesystem to gain access to critical new features is inconsistent > with this pitch. It would be hugely impactful. Just my $.02. > > As you can tell, I'm frustrated there's no online conversion tool :) Not > that there couldn't be... you all are brilliant developers. > > -Aaron > > On 10/11/16 1:22 PM, Yuri L Volobuev wrote: >> This depends on the committed cluster version level (minReleaseLevel) >> and file system format. Since NFSv2 is an on-disk format change, older >> code wouldn't be able to understand what it is, and thus if there's a >> possibility of a downlevel node looking at the NSD, the NFSv1 format is >> going to be used. The code does NSDv1<->NSDv2 conversions under the >> covers as needed when adding an empty NSD to a file system. >> >> I'd strongly recommend getting a fresh start by formatting a new file >> system. Many things have changed over the course of the last few years. >> In particular, having a 4K-aligned file system can be a pretty big deal, >> depending on what hardware one is going to deploy in the future, and >> this is something that can't be bolted onto an existing file system. >> Having 4K inodes is very handy for many reasons. New directory format >> and NSD format changes are attractive, too. And disks generally tend to >> get larger with time, and at some point you may want to add a disk to an >> existing storage pool that's larger than the existing allocation map >> format allows. Obviously, it's more hassle to migrate data to a new file >> system, as opposed to extending an existing one. In a perfect world, >> GPFS would offer a conversion tool that seamlessly and robustly converts >> old file systems, making them as good as new, but in the real world such >> a tool doesn't exist. Getting a clean slate by formatting a new file >> system every few years is a good long-term investment of time, although >> it comes front-loaded with extra work. >> >> yuri >> >> Inactive hide details for Aaron Knister ---10/10/2016 04:45:31 PM---Can >> one format NSDv2 NSDs and put them in a filesystem withAaron Knister >> ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a >> filesystem with NSDv1 NSD's? -Aaron >> >> From: Aaron Knister >> To: , >> Date: 10/10/2016 04:45 PM >> Subject: Re: [gpfsug-discuss] Hardware refresh >> Sent by: gpfsug-discuss-bounces at spectrumscale.org >> >> ------------------------------------------------------------------------ >> >> >> >> Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? >> >> -Aaron >> >> On 10/10/16 7:40 PM, Luis Bolinches wrote: >>> Hi >>> >>> Creating a new FS sounds like a best way to go. NSDv2 being a very good >>> reason to do so. >>> >>> AFM for migrations is quite good, latest versions allows to use NSD >>> protocol for mounts as well. Olaf did a great job explaining this >>> scenario on the redbook chapter 6 >>> >>> http://www.redbooks.ibm.com/abstracts/sg248254.html?Open >>> >>> -- >>> Cheers >>> >>> On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L >>> >> > wrote: >>> >>>> Hi Mark, >>>> >>>> The last time we did something like this was 2010 (we?re doing rolling >>>> refreshes now), so there are probably lots of better ways to do this >>>> than what we did, but we: >>>> >>>> 1) set up the new hardware >>>> 2) created new filesystems (so that we could make adjustments we >>>> wanted to make that can only be made at FS creation time) >>>> 3) used rsync to make a 1st pass copy of everything >>>> 4) coordinated a time with users / groups to do a 2nd rsync when they >>>> weren?t active >>>> 5) used symbolic links during the transition (i.e. rm -rvf >>>> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >>>> 6) once everybody was migrated, updated the symlinks (i.e. /home >>>> became a symlink to /gpfs2/home) >>>> >>>> HTHAL? >>>> >>>> Kevin >>>> >>>>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>>>> wrote: >>>>> >>>>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>>>> refresh hardware. Any lessons learned in this process? Is it >>>>> easiest to just build new cluster and then use AFM? Add to existing >>>>> cluster then decommission nodes? What is the recommended process for >>>>> this? >>>>> >>>>> >>>>> Mark >>>>> >>>>> This message (including any attachments) is intended only for the use >>>>> of the individual or entity to which it is addressed and may contain >>>>> information that is non-public, proprietary, privileged, >>>>> confidential, and exempt from disclosure under applicable law. If you >>>>> are not the intended recipient, you are hereby notified that any use, >>>>> dissemination, distribution, or copying of this communication is >>>>> strictly prohibited. This message may be viewed by parties at Sirius >>>>> Computer Solutions other than those named in the message header. This >>>>> message does not contain an official representation of Sirius >>>>> Computer Solutions. If you have received this communication in error, >>>>> notify Sirius Computer Solutions immediately and (i) destroy this >>>>> message if a facsimile or (ii) delete this message immediately if >>>>> this is an electronic communication. Thank you. >>>>> >>>>> Sirius Computer Solutions >>>>> _______________________________________________ >>>>> gpfsug-discuss mailing list >>>>> gpfsug-discuss at spectrumscale.org >>>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>>> >>>> ? >>>> Kevin Buterbaugh - Senior System Administrator >>>> Vanderbilt University - Advanced Computing Center for Research and >>>> Education >>>> Kevin.Buterbaugh at vanderbilt.edu >>>> - (615)875-9633 >>>> >>>> >>>> >>> >>> Ellei edell? ole toisin mainittu: / Unless stated otherwise above: >>> Oy IBM Finland Ab >>> PL 265, 00101 Helsinki, Finland >>> Business ID, Y-tunnus: 0195876-3 >>> Registered in Finland >>> >>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Mark.Bush at siriuscom.com Wed Oct 12 01:40:39 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Wed, 12 Oct 2016 00:40:39 +0000 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: References: <9E337B56-A274-41B6-84C2-7084AAB771F4@siriuscom.com> Message-ID: Only compelling reason for new cluster would be old hardware is EOL or no longer want to pay maintenance on it. From: on behalf of Marc A Kaplan Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 2:58 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh New FS? Yes there are some good reasons. New cluster? I did not see a compelling argument either way. From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Date: 10/11/2016 03:34 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Ok. I think I am hearing that a new cluster with a new FS and copying data from old to new cluster is the best way forward. Thanks everyone for your input. From: on behalf of Yuri L Volobuev Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 12:22 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh This depends on the committed cluster version level (minReleaseLevel) and file system format. Since NFSv2 is an on-disk format change, older code wouldn't be able to understand what it is, and thus if there's a possibility of a downlevel node looking at the NSD, the NFSv1 format is going to be used. The code does NSDv1<->NSDv2 conversions under the covers as needed when adding an empty NSD to a file system. I'd strongly recommend getting a fresh start by formatting a new file system. Many things have changed over the course of the last few years. In particular, having a 4K-aligned file system can be a pretty big deal, depending on what hardware one is going to deploy in the future, and this is something that can't be bolted onto an existing file system. Having 4K inodes is very handy for many reasons. New directory format and NSD format changes are attractive, too. And disks generally tend to get larger with time, and at some point you may want to add a disk to an existing storage pool that's larger than the existing allocation map format allows. Obviously, it's more hassle to migrate data to a new file system, as opposed to extending an existing one. In a perfect world, GPFS would offer a conversion tool that seamlessly and robustly converts old file systems, making them as good as new, but in the real world such a tool doesn't exist. Getting a clean slate by formatting a new file system every few years is a good long-term investment of time, although it comes front-loaded with extra work. yuri [active hide details for Aaron Knister ---10/10/2016 04:45:31 PM---Can on]Aaron Knister ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron From: Aaron Knister To: , Date: 10/10/2016 04:45 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron On 10/10/16 7:40 PM, Luis Bolinches wrote: > Hi > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > reason to do so. > > AFM for migrations is quite good, latest versions allows to use NSD > protocol for mounts as well. Olaf did a great job explaining this > scenario on the redbook chapter 6 > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > -- > Cheers > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > wrote: > >> Hi Mark, >> >> The last time we did something like this was 2010 (we?re doing rolling >> refreshes now), so there are probably lots of better ways to do this >> than what we did, but we: >> >> 1) set up the new hardware >> 2) created new filesystems (so that we could make adjustments we >> wanted to make that can only be made at FS creation time) >> 3) used rsync to make a 1st pass copy of everything >> 4) coordinated a time with users / groups to do a 2nd rsync when they >> weren?t active >> 5) used symbolic links during the transition (i.e. rm -rvf >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >> 6) once everybody was migrated, updated the symlinks (i.e. /home >> became a symlink to /gpfs2/home) >> >> HTHAL? >> >> Kevin >> >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>> wrote: >>> >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>> refresh hardware. Any lessons learned in this process? Is it >>> easiest to just build new cluster and then use AFM? Add to existing >>> cluster then decommission nodes? What is the recommended process for >>> this? >>> >>> >>> Mark >>> >>> This message (including any attachments) is intended only for the use >>> of the individual or entity to which it is addressed and may contain >>> information that is non-public, proprietary, privileged, >>> confidential, and exempt from disclosure under applicable law. If you >>> are not the intended recipient, you are hereby notified that any use, >>> dissemination, distribution, or copying of this communication is >>> strictly prohibited. This message may be viewed by parties at Sirius >>> Computer Solutions other than those named in the message header. This >>> message does not contain an official representation of Sirius >>> Computer Solutions. If you have received this communication in error, >>> notify Sirius Computer Solutions immediately and (i) destroy this >>> message if a facsimile or (ii) delete this message immediately if >>> this is an electronic communication. Thank you. >>> >>> Sirius Computer Solutions >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and >> Education >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 107 bytes Desc: image001.gif URL: From aaron.s.knister at nasa.gov Wed Oct 12 01:50:38 2016 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 11 Oct 2016 20:50:38 -0400 Subject: [gpfsug-discuss] 4K sector NSD support (was: Hardware refresh) In-Reply-To: <63bd99ad-f684-d1bb-371f-e2c21849ff29@nasa.gov> References: <63bd99ad-f684-d1bb-371f-e2c21849ff29@nasa.gov> Message-ID: Yuri, (Sorry for being somewhat spammy) I now understand the limitation after some more testing (I'm a hands-on learner, can you tell?). Given the right code/cluster/fs version levels I can add 4K dataOnly NSDv2 NSDs to a filesystem created with NSDv1 NSDs. What I can't do is seemingly add any metadataOnly or dataAndMetadata 4K luns to an fs that is not 4K aligned which I assume would be any fs originally created with NSDv1 LUNs. It seems possible to move all data away from NSDv1 LUNS in a filesystem behind-the-scenes using GPFS migration tools, and move the data to NSDv2 LUNs. In this case I believe what's missing is a tool to convert just the metadata structures to be 4K aligned since the data would already on 4k-based NSDv2 LUNS, is that the case? I'm trying to figure out what exactly I'm asking for in an RFE. -Aaron On 10/11/16 7:57 PM, Aaron Knister wrote: > I think I was a little quick to the trigger. I re-read your last mail > after doing some testing and understand it differently. I was wrong > about my interpretation-- you can add 4K NSDv2 formatted NSDs to a > filesystem previously created with NSDv1 NSDs assuming, as you say, the. > minReleaseLevel and filesystem version are high enough. That negates > about half of my last e-mail. The fs still doesn't show as 4K aligned: > > loressd01:~ # /usr/lpp/mmfs/bin/mmlsfs tnb4k --is4KAligned > flag value description > ------------------- ------------------------ > ----------------------------------- > --is4KAligned No is4KAligned? > > but *shrug* most of the I/O to these disks should be 1MB anyway. If > somebody is pounding the FS with smaller than 4K I/O they're gonna get a > talkin' to. > > -Aaron > > On 10/11/16 6:41 PM, Aaron Knister wrote: >> Thanks Yuri. >> >> I'm asking for my own purposes but I think it's still relevant here: >> we're still at GPFS 3.5 and will be adding dataOnly NSDs with 4K sectors >> in the near future. We're planning to update to 4.1 before we format >> these NSDs, though. If I understand you correctly we can't bring these >> 4K NSDv2 NSDs into a filesystem with 512b-based NSDv1 NSDs? That's a >> pretty big deal :( >> >> Reformatting every few years with 10's of petabytes of data is not >> realistic for us (it would take years to move the data around). It also >> goes against my personal preachings about GPFS's storage virtualization >> capabilities: the ability to perform upgrades/make underlying storage >> infrastructure changes with behind-the-scenes data migration, >> eliminating much of the manual hassle of storage administrators doing >> rsync dances. I guess it's RFE time? It also seems as though AFM could >> help with automating the migration, although many of our filesystems do >> not have filesets on them so we would have to re-think how we lay out >> our filesystems. >> >> This is also curious to me with IBM pitching GPFS as a filesystem for >> cloud services (the cloud *never* goes down, right?). Granted I believe >> this pitch started after the NSDv2 format was defined, but if somebody >> is building a large cloud with GPFS as the underlying filesystem for an >> object or an image store one might think the idea of having to re-format >> the filesystem to gain access to critical new features is inconsistent >> with this pitch. It would be hugely impactful. Just my $.02. >> >> As you can tell, I'm frustrated there's no online conversion tool :) Not >> that there couldn't be... you all are brilliant developers. >> >> -Aaron >> >> On 10/11/16 1:22 PM, Yuri L Volobuev wrote: >>> This depends on the committed cluster version level (minReleaseLevel) >>> and file system format. Since NFSv2 is an on-disk format change, older >>> code wouldn't be able to understand what it is, and thus if there's a >>> possibility of a downlevel node looking at the NSD, the NFSv1 format is >>> going to be used. The code does NSDv1<->NSDv2 conversions under the >>> covers as needed when adding an empty NSD to a file system. >>> >>> I'd strongly recommend getting a fresh start by formatting a new file >>> system. Many things have changed over the course of the last few years. >>> In particular, having a 4K-aligned file system can be a pretty big deal, >>> depending on what hardware one is going to deploy in the future, and >>> this is something that can't be bolted onto an existing file system. >>> Having 4K inodes is very handy for many reasons. New directory format >>> and NSD format changes are attractive, too. And disks generally tend to >>> get larger with time, and at some point you may want to add a disk to an >>> existing storage pool that's larger than the existing allocation map >>> format allows. Obviously, it's more hassle to migrate data to a new file >>> system, as opposed to extending an existing one. In a perfect world, >>> GPFS would offer a conversion tool that seamlessly and robustly converts >>> old file systems, making them as good as new, but in the real world such >>> a tool doesn't exist. Getting a clean slate by formatting a new file >>> system every few years is a good long-term investment of time, although >>> it comes front-loaded with extra work. >>> >>> yuri >>> >>> Inactive hide details for Aaron Knister ---10/10/2016 04:45:31 PM---Can >>> one format NSDv2 NSDs and put them in a filesystem withAaron Knister >>> ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a >>> filesystem with NSDv1 NSD's? -Aaron >>> >>> From: Aaron Knister >>> To: , >>> Date: 10/10/2016 04:45 PM >>> Subject: Re: [gpfsug-discuss] Hardware refresh >>> Sent by: gpfsug-discuss-bounces at spectrumscale.org >>> >>> ------------------------------------------------------------------------ >>> >>> >>> >>> Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? >>> >>> -Aaron >>> >>> On 10/10/16 7:40 PM, Luis Bolinches wrote: >>>> Hi >>>> >>>> Creating a new FS sounds like a best way to go. NSDv2 being a very good >>>> reason to do so. >>>> >>>> AFM for migrations is quite good, latest versions allows to use NSD >>>> protocol for mounts as well. Olaf did a great job explaining this >>>> scenario on the redbook chapter 6 >>>> >>>> http://www.redbooks.ibm.com/abstracts/sg248254.html?Open >>>> >>>> -- >>>> Cheers >>>> >>>> On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L >>>> >>> > wrote: >>>> >>>>> Hi Mark, >>>>> >>>>> The last time we did something like this was 2010 (we?re doing rolling >>>>> refreshes now), so there are probably lots of better ways to do this >>>>> than what we did, but we: >>>>> >>>>> 1) set up the new hardware >>>>> 2) created new filesystems (so that we could make adjustments we >>>>> wanted to make that can only be made at FS creation time) >>>>> 3) used rsync to make a 1st pass copy of everything >>>>> 4) coordinated a time with users / groups to do a 2nd rsync when they >>>>> weren?t active >>>>> 5) used symbolic links during the transition (i.e. rm -rvf >>>>> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >>>>> 6) once everybody was migrated, updated the symlinks (i.e. /home >>>>> became a symlink to /gpfs2/home) >>>>> >>>>> HTHAL? >>>>> >>>>> Kevin >>>>> >>>>>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>>>>> wrote: >>>>>> >>>>>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>>>>> refresh hardware. Any lessons learned in this process? Is it >>>>>> easiest to just build new cluster and then use AFM? Add to existing >>>>>> cluster then decommission nodes? What is the recommended process for >>>>>> this? >>>>>> >>>>>> >>>>>> Mark >>>>>> >>>>>> This message (including any attachments) is intended only for the use >>>>>> of the individual or entity to which it is addressed and may contain >>>>>> information that is non-public, proprietary, privileged, >>>>>> confidential, and exempt from disclosure under applicable law. If you >>>>>> are not the intended recipient, you are hereby notified that any use, >>>>>> dissemination, distribution, or copying of this communication is >>>>>> strictly prohibited. This message may be viewed by parties at Sirius >>>>>> Computer Solutions other than those named in the message header. This >>>>>> message does not contain an official representation of Sirius >>>>>> Computer Solutions. If you have received this communication in error, >>>>>> notify Sirius Computer Solutions immediately and (i) destroy this >>>>>> message if a facsimile or (ii) delete this message immediately if >>>>>> this is an electronic communication. Thank you. >>>>>> >>>>>> Sirius Computer Solutions >>>>>> _______________________________________________ >>>>>> gpfsug-discuss mailing list >>>>>> gpfsug-discuss at spectrumscale.org >>>>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>>>> >>>>> ? >>>>> Kevin Buterbaugh - Senior System Administrator >>>>> Vanderbilt University - Advanced Computing Center for Research and >>>>> Education >>>>> Kevin.Buterbaugh at vanderbilt.edu >>>>> - (615)875-9633 >>>>> >>>>> >>>>> >>>> >>>> Ellei edell? ole toisin mainittu: / Unless stated otherwise above: >>>> Oy IBM Finland Ab >>>> PL 265, 00101 Helsinki, Finland >>>> Business ID, Y-tunnus: 0195876-3 >>>> Registered in Finland >>>> >>>> >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> >>> >>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From ulmer at ulmer.org Wed Oct 12 02:29:36 2016 From: ulmer at ulmer.org (Stephen Ulmer) Date: Tue, 11 Oct 2016 21:29:36 -0400 Subject: [gpfsug-discuss] Hardware refresh In-Reply-To: References: <9E337B56-A274-41B6-84C2-7084AAB771F4@siriuscom.com> Message-ID: <279BAB6D-21E6-4EC1-98CD-CA409A24F8F6@ulmer.org> I think that the OP was asking why not expand the existing cluster with the new hardware, and just make a new FS? I?ve not tried to make a cluster talk AFM to itself yet. If that?s impossible, then there?s one good reason to make a new cluster (to use AFM for migration). Liberty, -- Stephen > On Oct 11, 2016, at 8:40 PM, Mark.Bush at siriuscom.com wrote: > > Only compelling reason for new cluster would be old hardware is EOL or no longer want to pay maintenance on it. > > From: > on behalf of Marc A Kaplan > > Reply-To: gpfsug main discussion list > > Date: Tuesday, October 11, 2016 at 2:58 PM > To: gpfsug main discussion list > > Subject: Re: [gpfsug-discuss] Hardware refresh > > New FS? Yes there are some good reasons. > New cluster? I did not see a compelling argument either way. > > > > From: "Mark.Bush at siriuscom.com " > > To: gpfsug main discussion list > > Date: 10/11/2016 03:34 PM > Subject: Re: [gpfsug-discuss] Hardware refresh > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > Ok. I think I am hearing that a new cluster with a new FS and copying data from old to new cluster is the best way forward. Thanks everyone for your input. > > From: > on behalf of Yuri L Volobuev > > Reply-To: gpfsug main discussion list > > Date: Tuesday, October 11, 2016 at 12:22 PM > To: gpfsug main discussion list > > Subject: Re: [gpfsug-discuss] Hardware refresh > > This depends on the committed cluster version level (minReleaseLevel) and file system format. Since NFSv2 is an on-disk format change, older code wouldn't be able to understand what it is, and thus if there's a possibility of a downlevel node looking at the NSD, the NFSv1 format is going to be used. The code does NSDv1<->NSDv2 conversions under the covers as needed when adding an empty NSD to a file system. > > I'd strongly recommend getting a fresh start by formatting a new file system. Many things have changed over the course of the last few years. In particular, having a 4K-aligned file system can be a pretty big deal, depending on what hardware one is going to deploy in the future, and this is something that can't be bolted onto an existing file system. Having 4K inodes is very handy for many reasons. New directory format and NSD format changes are attractive, too. And disks generally tend to get larger with time, and at some point you may want to add a disk to an existing storage pool that's larger than the existing allocation map format allows. Obviously, it's more hassle to migrate data to a new file system, as opposed to extending an existing one. In a perfect world, GPFS would offer a conversion tool that seamlessly and robustly converts old file systems, making them as good as new, but in the real world such a tool doesn't exist. Getting a clean slate by formatting a new file system every few years is a good long-term investment of time, although it comes front-loaded with extra work. > > yuri > > Aaron Knister ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron > > From: Aaron Knister > > To: >, > Date: 10/10/2016 04:45 PM > Subject: Re: [gpfsug-discuss] Hardware refresh > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > > Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? > > -Aaron > > On 10/10/16 7:40 PM, Luis Bolinches wrote: > > Hi > > > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > > reason to do so. > > > > AFM for migrations is quite good, latest versions allows to use NSD > > protocol for mounts as well. Olaf did a great job explaining this > > scenario on the redbook chapter 6 > > > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > > > -- > > Cheers > > > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > > > >> wrote: > > > >> Hi Mark, > >> > >> The last time we did something like this was 2010 (we?re doing rolling > >> refreshes now), so there are probably lots of better ways to do this > >> than what we did, but we: > >> > >> 1) set up the new hardware > >> 2) created new filesystems (so that we could make adjustments we > >> wanted to make that can only be made at FS creation time) > >> 3) used rsync to make a 1st pass copy of everything > >> 4) coordinated a time with users / groups to do a 2nd rsync when they > >> weren?t active > >> 5) used symbolic links during the transition (i.e. rm -rvf > >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) > >> 6) once everybody was migrated, updated the symlinks (i.e. /home > >> became a symlink to /gpfs2/home) > >> > >> HTHAL? > >> > >> Kevin > >> > >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com > >>> > wrote: > >>> > >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to > >>> refresh hardware. Any lessons learned in this process? Is it > >>> easiest to just build new cluster and then use AFM? Add to existing > >>> cluster then decommission nodes? What is the recommended process for > >>> this? > >>> > >>> > >>> Mark > >>> > >>> This message (including any attachments) is intended only for the use > >>> of the individual or entity to which it is addressed and may contain > >>> information that is non-public, proprietary, privileged, > >>> confidential, and exempt from disclosure under applicable law. If you > >>> are not the intended recipient, you are hereby notified that any use, > >>> dissemination, distribution, or copying of this communication is > >>> strictly prohibited. This message may be viewed by parties at Sirius > >>> Computer Solutions other than those named in the message header. This > >>> message does not contain an official representation of Sirius > >>> Computer Solutions. If you have received this communication in error, > >>> notify Sirius Computer Solutions immediately and (i) destroy this > >>> message if a facsimile or (ii) delete this message immediately if > >>> this is an electronic communication. Thank you. > >>> > >>> Sirius Computer Solutions > > >>> _______________________________________________ > >>> gpfsug-discuss mailing list > >>> gpfsug-discuss at spectrumscale.org > > >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > >> ? > >> Kevin Buterbaugh - Senior System Administrator > >> Vanderbilt University - Advanced Computing Center for Research and > >> Education > >> Kevin.Buterbaugh at vanderbilt.edu > >> > - (615)875-9633 > >> > >> > >> > > > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > > Oy IBM Finland Ab > > PL 265, 00101 Helsinki, Finland > > Business ID, Y-tunnus: 0195876-3 > > Registered in Finland > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. > > Sirius Computer Solutions _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Wed Oct 12 02:33:32 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Wed, 12 Oct 2016 01:33:32 +0000 Subject: [gpfsug-discuss] 4K sector NSD support (was: Hardware refresh) In-Reply-To: Message-ID: If you File System was created with i=512 you wont benefit from 4k Disk technologies ... some backend emulate it by Controller Software but most likely you'll get in trouble, when trying to add 4k Disk into your filesystem ... Gesendet von IBM Verse Aaron Knister --- Re: [gpfsug-discuss] 4K sector NSD support (was: Hardware refresh) --- Von:"Aaron Knister" An:gpfsug-discuss at spectrumscale.orgDatum:Di. 11.10.2016 17:59Betreff:Re: [gpfsug-discuss] 4K sector NSD support (was: Hardware refresh) Yuri,(Sorry for being somewhat spammy) I now understand the limitation after some more testing (I'm a hands-on learner, can you tell?). Given the right code/cluster/fs version levels I can add 4K dataOnly NSDv2 NSDs to a filesystem created with NSDv1 NSDs. What I can't do is seemingly add any metadataOnly or dataAndMetadata 4K luns to an fs that is not 4K aligned which I assume would be any fs originally created with NSDv1 LUNs. It seems possible to move all data away from NSDv1 LUNS in a filesystem behind-the-scenes using GPFS migration tools, and move the data to NSDv2 LUNs. In this case I believe what's missing is a tool to convert just the metadata structures to be 4K aligned since the data would already on 4k-based NSDv2 LUNS, is that the case? I'm trying to figure out what exactly I'm asking for in an RFE.-AaronOn 10/11/16 7:57 PM, Aaron Knister wrote:> I think I was a little quick to the trigger. I re-read your last mail> after doing some testing and understand it differently. I was wrong> about my interpretation-- you can add 4K NSDv2 formatted NSDs to a> filesystem previously created with NSDv1 NSDs assuming, as you say, the.> minReleaseLevel and filesystem version are high enough. That negates> about half of my last e-mail. The fs still doesn't show as 4K aligned:>> loressd01:~ # /usr/lpp/mmfs/bin/mmlsfs tnb4k --is4KAligned> flag value description> ------------------- ------------------------> -----------------------------------> --is4KAligned No is4KAligned?>> but *shrug* most of the I/O to these disks should be 1MB anyway. If> somebody is pounding the FS with smaller than 4K I/O they're gonna get a> talkin' to.>> -Aaron>> On 10/11/16 6:41 PM, Aaron Knister wrote:>> Thanks Yuri.>>>> I'm asking for my own purposes but I think it's still relevant here:>> we're still at GPFS 3.5 and will be adding dataOnly NSDs with 4K sectors>> in the near future. We're planning to update to 4.1 before we format>> these NSDs, though. If I understand you correctly we can't bring these>> 4K NSDv2 NSDs into a filesystem with 512b-based NSDv1 NSDs? That's a>> pretty big deal :(>>>> Reformatting every few years with 10's of petabytes of data is not>> realistic for us (it would take years to move the data around). It also>> goes against my personal preachings about GPFS's storage virtualization>> capabilities: the ability to perform upgrades/make underlying storage>> infrastructure changes with behind-the-scenes data migration,>> eliminating much of the manual hassle of storage administrators doing>> rsync dances. I guess it's RFE time? It also seems as though AFM could>> help with automating the migration, although many of our filesystems do>> not have filesets on them so we would have to re-think how we lay out>> our filesystems.>>>> This is also curious to me with IBM pitching GPFS as a filesystem for>> cloud services (the cloud *never* goes down, right?). Granted I believe>> this pitch started after the NSDv2 format was defined, but if somebody>> is building a large cloud with GPFS as the underlying filesystem for an>> object or an image store one might think the idea of having to re-format>> the filesystem to gain access to critical new features is inconsistent>> with this pitch. It would be hugely impactful. Just my $.02.>>>> As you can tell, I'm frustrated there's no online conversion tool :) Not>> that there couldn't be... you all are brilliant developers.>>>> -Aaron>>>> On 10/11/16 1:22 PM, Yuri L Volobuev wrote:>>> This depends on the committed cluster version level (minReleaseLevel)>>> and file system format. Since NFSv2 is an on-disk format change, older>>> code wouldn't be able to understand what it is, and thus if there's a>>> possibility of a downlevel node looking at the NSD, the NFSv1 format is>>> going to be used. The code does NSDv1<->NSDv2 conversions under the>>> covers as needed when adding an empty NSD to a file system.>>>>>> I'd strongly recommend getting a fresh start by formatting a new file>>> system. Many things have changed over the course of the last few years.>>> In particular, having a 4K-aligned file system can be a pretty big deal,>>> depending on what hardware one is going to deploy in the future, and>>> this is something that can't be bolted onto an existing file system.>>> Having 4K inodes is very handy for many reasons. New directory format>>> and NSD format changes are attractive, too. And disks generally tend to>>> get larger with time, and at some point you may want to add a disk to an>>> existing storage pool that's larger than the existing allocation map>>> format allows. Obviously, it's more hassle to migrate data to a new file>>> system, as opposed to extending an existing one. In a perfect world,>>> GPFS would offer a conversion tool that seamlessly and robustly converts>>> old file systems, making them as good as new, but in the real world such>>> a tool doesn't exist. Getting a clean slate by formatting a new file>>> system every few years is a good long-term investment of time, although>>> it comes front-loaded with extra work.>>>>>> yuri>>>>>> Inactive hide details for Aaron Knister ---10/10/2016 04:45:31 PM---Can>>> one format NSDv2 NSDs and put them in a filesystem withAaron Knister>>> ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a>>> filesystem with NSDv1 NSD's? -Aaron>>>>>> From: Aaron Knister >>> To: ,>>> Date: 10/10/2016 04:45 PM>>> Subject: Re: [gpfsug-discuss] Hardware refresh>>> Sent by: gpfsug-discuss-bounces at spectrumscale.org>>>>>> ------------------------------------------------------------------------>>>>>>>>>>>> Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's?>>>>>> -Aaron>>>>>> On 10/10/16 7:40 PM, Luis Bolinches wrote:>>>> Hi>>>>>>>> Creating a new FS sounds like a best way to go. NSDv2 being a very good>>>> reason to do so.>>>>>>>> AFM for migrations is quite good, latest versions allows to use NSD>>>> protocol for mounts as well. Olaf did a great job explaining this>>>> scenario on the redbook chapter 6>>>>>>>> http://www.redbooks.ibm.com/abstracts/sg248254.html?Open>>>>>>>> -->>>> Cheers>>>>>>>> On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L>>>> >>> > wrote:>>>>>>>>> Hi Mark,>>>>>>>>>> The last time we did something like this was 2010 (we?re doing rolling>>>>> refreshes now), so there are probably lots of better ways to do this>>>>> than what we did, but we:>>>>>>>>>> 1) set up the new hardware>>>>> 2) created new filesystems (so that we could make adjustments we>>>>> wanted to make that can only be made at FS creation time)>>>>> 3) used rsync to make a 1st pass copy of everything>>>>> 4) coordinated a time with users / groups to do a 2nd rsync when they>>>>> weren?t active>>>>> 5) used symbolic links during the transition (i.e. rm -rvf>>>>> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser)>>>>> 6) once everybody was migrated, updated the symlinks (i.e. /home>>>>> became a symlink to /gpfs2/home)>>>>>>>>>> HTHAL?>>>>>>>>>> Kevin>>>>>>>>>>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com>>>>>> wrote:>>>>>>>>>>>> Have a very old cluster built on IBM X3650?s and DS3500. Need to>>>>>> refresh hardware. Any lessons learned in this process? Is it>>>>>> easiest to just build new cluster and then use AFM? Add to existing>>>>>> cluster then decommission nodes? What is the recommended process for>>>>>> this?>>>>>>>>>>>>>>>>>> Mark>>>>>>>>>>>> This message (including any attachments) is intended only for the use>>>>>> of the individual or entity to which it is addressed and may contain>>>>>> information that is non-public, proprietary, privileged,>>>>>> confidential, and exempt from disclosure under applicable law. If you>>>>>> are not the intended recipient, you are hereby notified that any use,>>>>>> dissemination, distribution, or copying of this communication is>>>>>> strictly prohibited. This message may be viewed by parties at Sirius>>>>>> Computer Solutions other than those named in the message header. This>>>>>> message does not contain an official representation of Sirius>>>>>> Computer Solutions. If you have received this communication in error,>>>>>> notify Sirius Computer Solutions immediately and (i) destroy this>>>>>> message if a facsimile or (ii) delete this message immediately if>>>>>> this is an electronic communication. Thank you.>>>>>>>>>>>> Sirius Computer Solutions >>>>>> _______________________________________________>>>>>> gpfsug-discuss mailing list>>>>>> gpfsug-discuss at spectrumscale.org >>>>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss>>>>>>>>>> ?>>>>> Kevin Buterbaugh - Senior System Administrator>>>>> Vanderbilt University - Advanced Computing Center for Research and>>>>> Education>>>>> Kevin.Buterbaugh at vanderbilt.edu>>>>> - (615)875-9633>>>>>>>>>>>>>>>>>>>>>>> Ellei edell? ole toisin mainittu: / Unless stated otherwise above:>>>> Oy IBM Finland Ab>>>> PL 265, 00101 Helsinki, Finland>>>> Business ID, Y-tunnus: 0195876-3>>>> Registered in Finland>>>>>>>>>>>>>>>> _______________________________________________>>>> gpfsug-discuss mailing list>>>> gpfsug-discuss at spectrumscale.org>>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss>>>>>>> _______________________________________________>>> gpfsug-discuss mailing list>>> gpfsug-discuss at spectrumscale.org>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss>>>>>>>>>>>>>>>>>> _______________________________________________>>> gpfsug-discuss mailing list>>> gpfsug-discuss at spectrumscale.org>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss>>>>> _______________________________________________>> gpfsug-discuss mailing list>> gpfsug-discuss at spectrumscale.org>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss> _______________________________________________> gpfsug-discuss mailing list> gpfsug-discuss at spectrumscale.org> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_______________________________________________gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From volobuev at us.ibm.com Wed Oct 12 06:44:40 2016 From: volobuev at us.ibm.com (Yuri L Volobuev) Date: Tue, 11 Oct 2016 22:44:40 -0700 Subject: [gpfsug-discuss] 4K sector NSD support (was: Hardware refresh) In-Reply-To: References: <63bd99ad-f684-d1bb-371f-e2c21849ff29@nasa.gov> Message-ID: Yes, it is possible to add a 4KN dataOnly NSD to a non-4K-aligned file system, as you figured out. This is something we didn't plan on doing originally, but then had to implement based on the feedback from the field. There's clearly a need for this. However, this statement is exactly it -- dataOnly NSDs. The only way to put metadata on a 4KN disk is to use a 4K-aligned file system. There are several kinds of metadata present in non-4K-aligned file system that generate non-4K IOs (512 byte inodes being the biggest problem), and there's no way to work around this short of using the new format, and there's no way to perform a conversion to the new format in-place. You're welcome to submit an RFE, of course, but I'd recommend being pragmatic about the chances of such an RFE being implemented. As you can imagine, the main reason why an all-encompassing file system conversion tool doesn't exist is not GPFS developers having no idea that such a tool is wanted. There are several considerations that conspire to make this an unlikely candidate to ever be implemented: 1) The task is hard and has no finish line. In most GPFS releases, something changes, necessitating an added piece of work for the hypothetical conversion tool, and the matrix of from-to format version combinations gets to be big quite quickly. 2) A file system conversion is something that is needed very infrequently, but when this code does run, it absolutely has to run and run perfectly, else the result would be a half-converted file system, i.e. a royal mess. This is a tester's nightmare. 3) The failure scenarios are all unpalatable. What should the conversion tool do if it runs out of space replacing smaller metadata structures with bigger ones? Undoing a partially finished conversion is even harder than doing it in the first place. 4) Doing an on-disk conversion on-line is simply very difficult. Consider the task of converting an inode file to use a different inode size. The file can be huge (billions of records), and it would take a fair chunk of time to rewrite it, but the file is changing while it's being converted (can't simply lock the whole thing down for so long), simultaneously on multiple nodes. Orchestrating the processing of updates in the presence of two inode files, with proper atomicity guarantees (to guard against a node failure) is a task of considerable complexity. None of this means the task is impossible, of course. It is, however, a very big chunk of very complex work, all towards a tool that on an average cluster may run somewhere between zero and one times, not something that benefits day-to-day operations. Where the complexity of the task allows for a reasonably affordable implementation, e.g. conversion from an old-style EA file to the FASTEA format, a conversion tool has been implemented (mmmigratefs). However, doing this for every single changed aspect of the file system format is simply too expensive to justify, given other tasks in front of us. On the other hand, a well-implemented migration mechanism solves the file system reformatting scenario (which covers all aspects of file system format changes) as well as a number of other scenarios. This is a cleaner, more general solution. Migration doesn't have to mean an outage. A simple rsync-based migration requires downtime for a cutover, while an AFM-based migration doesn't necessarily require one. I'm not saying that GPFS has a particularly strong migration story at the moment, but this is a much more productive direction for applying resources than a mythical all-encompassing conversion tool. yuri From: Aaron Knister To: , Date: 10/11/2016 05:59 PM Subject: Re: [gpfsug-discuss] 4K sector NSD support (was: Hardware refresh) Sent by: gpfsug-discuss-bounces at spectrumscale.org Yuri, (Sorry for being somewhat spammy) I now understand the limitation after some more testing (I'm a hands-on learner, can you tell?). Given the right code/cluster/fs version levels I can add 4K dataOnly NSDv2 NSDs to a filesystem created with NSDv1 NSDs. What I can't do is seemingly add any metadataOnly or dataAndMetadata 4K luns to an fs that is not 4K aligned which I assume would be any fs originally created with NSDv1 LUNs. It seems possible to move all data away from NSDv1 LUNS in a filesystem behind-the-scenes using GPFS migration tools, and move the data to NSDv2 LUNs. In this case I believe what's missing is a tool to convert just the metadata structures to be 4K aligned since the data would already on 4k-based NSDv2 LUNS, is that the case? I'm trying to figure out what exactly I'm asking for in an RFE. -Aaron On 10/11/16 7:57 PM, Aaron Knister wrote: > I think I was a little quick to the trigger. I re-read your last mail > after doing some testing and understand it differently. I was wrong > about my interpretation-- you can add 4K NSDv2 formatted NSDs to a > filesystem previously created with NSDv1 NSDs assuming, as you say, the. > minReleaseLevel and filesystem version are high enough. That negates > about half of my last e-mail. The fs still doesn't show as 4K aligned: > > loressd01:~ # /usr/lpp/mmfs/bin/mmlsfs tnb4k --is4KAligned > flag value description > ------------------- ------------------------ > ----------------------------------- > --is4KAligned No is4KAligned? > > but *shrug* most of the I/O to these disks should be 1MB anyway. If > somebody is pounding the FS with smaller than 4K I/O they're gonna get a > talkin' to. > > -Aaron > > On 10/11/16 6:41 PM, Aaron Knister wrote: >> Thanks Yuri. >> >> I'm asking for my own purposes but I think it's still relevant here: >> we're still at GPFS 3.5 and will be adding dataOnly NSDs with 4K sectors >> in the near future. We're planning to update to 4.1 before we format >> these NSDs, though. If I understand you correctly we can't bring these >> 4K NSDv2 NSDs into a filesystem with 512b-based NSDv1 NSDs? That's a >> pretty big deal :( >> >> Reformatting every few years with 10's of petabytes of data is not >> realistic for us (it would take years to move the data around). It also >> goes against my personal preachings about GPFS's storage virtualization >> capabilities: the ability to perform upgrades/make underlying storage >> infrastructure changes with behind-the-scenes data migration, >> eliminating much of the manual hassle of storage administrators doing >> rsync dances. I guess it's RFE time? It also seems as though AFM could >> help with automating the migration, although many of our filesystems do >> not have filesets on them so we would have to re-think how we lay out >> our filesystems. >> >> This is also curious to me with IBM pitching GPFS as a filesystem for >> cloud services (the cloud *never* goes down, right?). Granted I believe >> this pitch started after the NSDv2 format was defined, but if somebody >> is building a large cloud with GPFS as the underlying filesystem for an >> object or an image store one might think the idea of having to re-format >> the filesystem to gain access to critical new features is inconsistent >> with this pitch. It would be hugely impactful. Just my $.02. >> >> As you can tell, I'm frustrated there's no online conversion tool :) Not >> that there couldn't be... you all are brilliant developers. >> >> -Aaron >> >> On 10/11/16 1:22 PM, Yuri L Volobuev wrote: >>> This depends on the committed cluster version level (minReleaseLevel) >>> and file system format. Since NFSv2 is an on-disk format change, older >>> code wouldn't be able to understand what it is, and thus if there's a >>> possibility of a downlevel node looking at the NSD, the NFSv1 format is >>> going to be used. The code does NSDv1<->NSDv2 conversions under the >>> covers as needed when adding an empty NSD to a file system. >>> >>> I'd strongly recommend getting a fresh start by formatting a new file >>> system. Many things have changed over the course of the last few years. >>> In particular, having a 4K-aligned file system can be a pretty big deal, >>> depending on what hardware one is going to deploy in the future, and >>> this is something that can't be bolted onto an existing file system. >>> Having 4K inodes is very handy for many reasons. New directory format >>> and NSD format changes are attractive, too. And disks generally tend to >>> get larger with time, and at some point you may want to add a disk to an >>> existing storage pool that's larger than the existing allocation map >>> format allows. Obviously, it's more hassle to migrate data to a new file >>> system, as opposed to extending an existing one. In a perfect world, >>> GPFS would offer a conversion tool that seamlessly and robustly converts >>> old file systems, making them as good as new, but in the real world such >>> a tool doesn't exist. Getting a clean slate by formatting a new file >>> system every few years is a good long-term investment of time, although >>> it comes front-loaded with extra work. >>> >>> yuri >>> >>> Inactive hide details for Aaron Knister ---10/10/2016 04:45:31 PM---Can >>> one format NSDv2 NSDs and put them in a filesystem withAaron Knister >>> ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a >>> filesystem with NSDv1 NSD's? -Aaron >>> >>> From: Aaron Knister >>> To: , >>> Date: 10/10/2016 04:45 PM >>> Subject: Re: [gpfsug-discuss] Hardware refresh >>> Sent by: gpfsug-discuss-bounces at spectrumscale.org >>> >>> ------------------------------------------------------------------------ >>> >>> >>> >>> Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? >>> >>> -Aaron >>> >>> On 10/10/16 7:40 PM, Luis Bolinches wrote: >>>> Hi >>>> >>>> Creating a new FS sounds like a best way to go. NSDv2 being a very good >>>> reason to do so. >>>> >>>> AFM for migrations is quite good, latest versions allows to use NSD >>>> protocol for mounts as well. Olaf did a great job explaining this >>>> scenario on the redbook chapter 6 >>>> >>>> http://www.redbooks.ibm.com/abstracts/sg248254.html?Open >>>> >>>> -- >>>> Cheers >>>> >>>> On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L >>>> >>> > wrote: >>>> >>>>> Hi Mark, >>>>> >>>>> The last time we did something like this was 2010 (we?re doing rolling >>>>> refreshes now), so there are probably lots of better ways to do this >>>>> than what we did, but we: >>>>> >>>>> 1) set up the new hardware >>>>> 2) created new filesystems (so that we could make adjustments we >>>>> wanted to make that can only be made at FS creation time) >>>>> 3) used rsync to make a 1st pass copy of everything >>>>> 4) coordinated a time with users / groups to do a 2nd rsync when they >>>>> weren?t active >>>>> 5) used symbolic links during the transition (i.e. rm -rvf >>>>> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >>>>> 6) once everybody was migrated, updated the symlinks (i.e. /home >>>>> became a symlink to /gpfs2/home) >>>>> >>>>> HTHAL? >>>>> >>>>> Kevin >>>>> >>>>>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>>>>> wrote: >>>>>> >>>>>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>>>>> refresh hardware. Any lessons learned in this process? Is it >>>>>> easiest to just build new cluster and then use AFM? Add to existing >>>>>> cluster then decommission nodes? What is the recommended process for >>>>>> this? >>>>>> >>>>>> >>>>>> Mark >>>>>> >>>>>> This message (including any attachments) is intended only for the use >>>>>> of the individual or entity to which it is addressed and may contain >>>>>> information that is non-public, proprietary, privileged, >>>>>> confidential, and exempt from disclosure under applicable law. If you >>>>>> are not the intended recipient, you are hereby notified that any use, >>>>>> dissemination, distribution, or copying of this communication is >>>>>> strictly prohibited. This message may be viewed by parties at Sirius >>>>>> Computer Solutions other than those named in the message header. This >>>>>> message does not contain an official representation of Sirius >>>>>> Computer Solutions. If you have received this communication in error, >>>>>> notify Sirius Computer Solutions immediately and (i) destroy this >>>>>> message if a facsimile or (ii) delete this message immediately if >>>>>> this is an electronic communication. Thank you. >>>>>> >>>>>> Sirius Computer Solutions >>>>>> _______________________________________________ >>>>>> gpfsug-discuss mailing list >>>>>> gpfsug-discuss at spectrumscale.org >>>>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>>>> >>>>> ? >>>>> Kevin Buterbaugh - Senior System Administrator >>>>> Vanderbilt University - Advanced Computing Center for Research and >>>>> Education >>>>> Kevin.Buterbaugh at vanderbilt.edu >>>>> - (615)875-9633 >>>>> >>>>> >>>>> >>>> >>>> Ellei edell? ole toisin mainittu: / Unless stated otherwise above: >>>> Oy IBM Finland Ab >>>> PL 265, 00101 Helsinki, Finland >>>> Business ID, Y-tunnus: 0195876-3 >>>> Registered in Finland >>>> >>>> >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> >>> >>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From makaplan at us.ibm.com Wed Oct 12 18:54:52 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Wed, 12 Oct 2016 13:54:52 -0400 Subject: [gpfsug-discuss] Hardware refresh - In-Reply-To: <279BAB6D-21E6-4EC1-98CD-CA409A24F8F6@ulmer.org> References: <9E337B56-A274-41B6-84C2-7084AAB771F4@siriuscom.com> <279BAB6D-21E6-4EC1-98CD-CA409A24F8F6@ulmer.org> Message-ID: Yes, you can AFM within a single cluster, in fact with just a single node. I just set this up on my toy system: [root at bog-wifi cmvc]# mmlsfileset yy afmlu --afm Filesets in file system 'yy': Name Status Path afmTarget afmlu Linked /yy/afmlu gpfs:///xx [root at bog-wifi cmvc]# mount ... yy on /yy type gpfs (rw,relatime,seclabel) xx on /xx type gpfs (rw,relatime,seclabel) [root at bog-wifi cmvc]# mmafmctl yy getstate Fileset Name Fileset Target Cache State Gateway Node Queue Length Queue numExec ------------ -------------- ------------- ------------ ------------ ------------- afmlu gpfs:///xx Active bog-wifi 0 7 So, you may add nodes, add disks to an existing cluster, upgrade your software, define a new FS, migrate data from an old FS to a new FS then delete nodes and disks that are no longer needed... From: Stephen Ulmer To: gpfsug main discussion list Date: 10/11/2016 09:30 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org I think that the OP was asking why not expand the existing cluster with the new hardware, and just make a new FS? I?ve not tried to make a cluster talk AFM to itself yet. If that?s impossible, then there?s one good reason to make a new cluster (to use AFM for migration). Liberty, -- Stephen On Oct 11, 2016, at 8:40 PM, Mark.Bush at siriuscom.com wrote: Only compelling reason for new cluster would be old hardware is EOL or no longer want to pay maintenance on it. From: on behalf of Marc A Kaplan Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 2:58 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh New FS? Yes there are some good reasons. New cluster? I did not see a compelling argument either way. From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Date: 10/11/2016 03:34 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org Ok. I think I am hearing that a new cluster with a new FS and copying data from old to new cluster is the best way forward. Thanks everyone for your input. From: on behalf of Yuri L Volobuev Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 12:22 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh This depends on the committed cluster version level (minReleaseLevel) and file system format. Since NFSv2 is an on-disk format change, older code wouldn't be able to understand what it is, and thus if there's a possibility of a downlevel node looking at the NSD, the NFSv1 format is going to be used. The code does NSDv1<->NSDv2 conversions under the covers as needed when adding an empty NSD to a file system. I'd strongly recommend getting a fresh start by formatting a new file system. Many things have changed over the course of the last few years. In particular, having a 4K-aligned file system can be a pretty big deal, depending on what hardware one is going to deploy in the future, and this is something that can't be bolted onto an existing file system. Having 4K inodes is very handy for many reasons. New directory format and NSD format changes are attractive, too. And disks generally tend to get larger with time, and at some point you may want to add a disk to an existing storage pool that's larger than the existing allocation map format allows. Obviously, it's more hassle to migrate data to a new file system, as opposed to extending an existing one. In a perfect world, GPFS would offer a conversion tool that seamlessly and robustly converts old file systems, making them as good as new, but in the real world such a tool doesn't exist. Getting a clean slate by formatting a new file system every few years is a good long-term investment of time, although it comes front-loaded with extra work. yuri Aaron Knister ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron From: Aaron Knister To: , Date: 10/10/2016 04:45 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron On 10/10/16 7:40 PM, Luis Bolinches wrote: > Hi > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > reason to do so. > > AFM for migrations is quite good, latest versions allows to use NSD > protocol for mounts as well. Olaf did a great job explaining this > scenario on the redbook chapter 6 > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > -- > Cheers > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > wrote: > >> Hi Mark, >> >> The last time we did something like this was 2010 (we?re doing rolling >> refreshes now), so there are probably lots of better ways to do this >> than what we did, but we: >> >> 1) set up the new hardware >> 2) created new filesystems (so that we could make adjustments we >> wanted to make that can only be made at FS creation time) >> 3) used rsync to make a 1st pass copy of everything >> 4) coordinated a time with users / groups to do a 2nd rsync when they >> weren?t active >> 5) used symbolic links during the transition (i.e. rm -rvf >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >> 6) once everybody was migrated, updated the symlinks (i.e. /home >> became a symlink to /gpfs2/home) >> >> HTHAL? >> >> Kevin >> >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>> wrote: >>> >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>> refresh hardware. Any lessons learned in this process? Is it >>> easiest to just build new cluster and then use AFM? Add to existing >>> cluster then decommission nodes? What is the recommended process for >>> this? >>> >>> >>> Mark >>> >>> This message (including any attachments) is intended only for the use >>> of the individual or entity to which it is addressed and may contain >>> information that is non-public, proprietary, privileged, >>> confidential, and exempt from disclosure under applicable law. If you >>> are not the intended recipient, you are hereby notified that any use, >>> dissemination, distribution, or copying of this communication is >>> strictly prohibited. This message may be viewed by parties at Sirius >>> Computer Solutions other than those named in the message header. This >>> message does not contain an official representation of Sirius >>> Computer Solutions. If you have received this communication in error, >>> notify Sirius Computer Solutions immediately and (i) destroy this >>> message if a facsimile or (ii) delete this message immediately if >>> this is an electronic communication. Thank you. >>> >>> Sirius Computer Solutions >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and >> Education >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mark.Bush at siriuscom.com Wed Oct 12 21:19:19 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Wed, 12 Oct 2016 20:19:19 +0000 Subject: [gpfsug-discuss] Spectrum Scale RAID usable space Message-ID: <5B378555-7956-40A1-8840-8090B43CAC7F@siriuscom.com> Anyone know how to calculate this? I realize this is an ESS thing now but I wonder if someone on here knows how to extrapolate usable space. Thanks Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From abeattie at au1.ibm.com Wed Oct 12 22:58:04 2016 From: abeattie at au1.ibm.com (Andrew Beattie) Date: Wed, 12 Oct 2016 21:58:04 +0000 Subject: [gpfsug-discuss] Spectrum Scale RAID usable space In-Reply-To: <5B378555-7956-40A1-8840-8090B43CAC7F@siriuscom.com> Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.14763088875930.png Type: image/png Size: 177376 bytes Desc: not available URL: From Mark.Bush at siriuscom.com Wed Oct 12 23:25:48 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Wed, 12 Oct 2016 22:25:48 +0000 Subject: [gpfsug-discuss] Spectrum Scale RAID usable space In-Reply-To: References: <5B378555-7956-40A1-8840-8090B43CAC7F@siriuscom.com> Message-ID: <7407A861-BC0F-40EA-97F9-24FAEA00C838@siriuscom.com> GL2 with 6TB drives. From: on behalf of Andrew Beattie Reply-To: gpfsug main discussion list Date: Wednesday, October 12, 2016 at 4:58 PM To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] Spectrum Scale RAID usable space Hi Mark, If you let me know which model ESS and what drive size I can give you the possible capacities. there are effectively 4 options 8+2P 8+3P 3-way replication 4-way replication for example: GL6 - 8TB drives Possible capacity options: 8+2P - 2,036TB 8+3P - 1,851TB 3 Way - 848TB 4 Way - 636TB [cid:image002.png at 01D224AD.AB668310] Business Partners: http://www.ibm.com/partnerworld/wps/servlet/ContentHandler/SSPQ048068H83479I86 There is a ESS capacity calculator in the same set of BP links with Capacity Magic and Disk magic Andrew Beattie Software Defined Storage - IT Specialist Phone: 614-2133-7927 E-mail: abeattie at au1.ibm.com ----- Original message ----- From: "Mark.Bush at siriuscom.com" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list Cc: Subject: [gpfsug-discuss] Spectrum Scale RAID usable space Date: Thu, Oct 13, 2016 7:19 AM Anyone know how to calculate this? I realize this is an ESS thing now but I wonder if someone on here knows how to extrapolate usable space. Thanks Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.png Type: image/png Size: 170988 bytes Desc: image002.png URL: From ulmer at ulmer.org Thu Oct 13 01:48:23 2016 From: ulmer at ulmer.org (Stephen Ulmer) Date: Wed, 12 Oct 2016 20:48:23 -0400 Subject: [gpfsug-discuss] Hardware refresh - In-Reply-To: References: <9E337B56-A274-41B6-84C2-7084AAB771F4@siriuscom.com> <279BAB6D-21E6-4EC1-98CD-CA409A24F8F6@ulmer.org> Message-ID: <1F6BD2DF-10F4-456B-9061-67D6D6B90B3D@ulmer.org> Nice! -- Stephen Ulmer Sent from a mobile device; please excuse auto-correct silliness. > On Oct 12, 2016, at 1:54 PM, Marc A Kaplan wrote: > > Yes, you can AFM within a single cluster, in fact with just a single node. I just set this up on my toy system: > > [root at bog-wifi cmvc]# mmlsfileset yy afmlu --afm > Filesets in file system 'yy': > Name Status Path afmTarget > afmlu Linked /yy/afmlu gpfs:///xx > > [root at bog-wifi cmvc]# mount > ... > yy on /yy type gpfs (rw,relatime,seclabel) > xx on /xx type gpfs (rw,relatime,seclabel) > > [root at bog-wifi cmvc]# mmafmctl yy getstate > Fileset Name Fileset Target Cache State Gateway Node Queue Length Queue numExec > ------------ -------------- ------------- ------------ ------------ ------------- > afmlu gpfs:///xx Active bog-wifi 0 7 > > So, you may add nodes, add disks to an existing cluster, upgrade your software, define a new FS, > migrate data from an old FS to a new FS > then delete nodes and disks that are no longer needed... > > > > From: Stephen Ulmer > To: gpfsug main discussion list > Date: 10/11/2016 09:30 PM > Subject: Re: [gpfsug-discuss] Hardware refresh > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > I think that the OP was asking why not expand the existing cluster with the new hardware, and just make a new FS? > > I?ve not tried to make a cluster talk AFM to itself yet. If that?s impossible, then there?s one good reason to make a new cluster (to use AFM for migration). > > Liberty, > > -- > Stephen > > > > On Oct 11, 2016, at 8:40 PM, Mark.Bush at siriuscom.comwrote: > > Only compelling reason for new cluster would be old hardware is EOL or no longer want to pay maintenance on it. > > From: on behalf of Marc A Kaplan > Reply-To: gpfsug main discussion list > Date: Tuesday, October 11, 2016 at 2:58 PM > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Hardware refresh > > New FS? Yes there are some good reasons. > New cluster? I did not see a compelling argument either way. > > > > From: "Mark.Bush at siriuscom.com" > To: gpfsug main discussion list > Date: 10/11/2016 03:34 PM > Subject: Re: [gpfsug-discuss] Hardware refresh > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > > Ok. I think I am hearing that a new cluster with a new FS and copying data from old to new cluster is the best way forward. Thanks everyone for your input. > > From: on behalf of Yuri L Volobuev > Reply-To: gpfsug main discussion list > Date: Tuesday, October 11, 2016 at 12:22 PM > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Hardware refresh > > This depends on the committed cluster version level (minReleaseLevel) and file system format. Since NFSv2 is an on-disk format change, older code wouldn't be able to understand what it is, and thus if there's a possibility of a downlevel node looking at the NSD, the NFSv1 format is going to be used. The code does NSDv1<->NSDv2 conversions under the covers as needed when adding an empty NSD to a file system. > > I'd strongly recommend getting a fresh start by formatting a new file system. Many things have changed over the course of the last few years. In particular, having a 4K-aligned file system can be a pretty big deal, depending on what hardware one is going to deploy in the future, and this is something that can't be bolted onto an existing file system. Having 4K inodes is very handy for many reasons. New directory format and NSD format changes are attractive, too. And disks generally tend to get larger with time, and at some point you may want to add a disk to an existing storage pool that's larger than the existing allocation map format allows. Obviously, it's more hassle to migrate data to a new file system, as opposed to extending an existing one. In a perfect world, GPFS would offer a conversion tool that seamlessly and robustly converts old file systems, making them as good as new, but in the real world such a tool doesn't exist. Getting a clean slate by formatting a new file system every few years is a good long-term investment of time, although it comes front-loaded with extra work. > > yuri > > Aaron Knister ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron > > From: Aaron Knister > To: , > Date: 10/10/2016 04:45 PM > Subject: Re: [gpfsug-discuss] Hardware refresh > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > > > > Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? > > -Aaron > > On 10/10/16 7:40 PM, Luis Bolinches wrote: > > Hi > > > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > > reason to do so. > > > > AFM for migrations is quite good, latest versions allows to use NSD > > protocol for mounts as well. Olaf did a great job explaining this > > scenario on the redbook chapter 6 > > > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > > > -- > > Cheers > > > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > > > wrote: > > > >> Hi Mark, > >> > >> The last time we did something like this was 2010 (we?re doing rolling > >> refreshes now), so there are probably lots of better ways to do this > >> than what we did, but we: > >> > >> 1) set up the new hardware > >> 2) created new filesystems (so that we could make adjustments we > >> wanted to make that can only be made at FS creation time) > >> 3) used rsync to make a 1st pass copy of everything > >> 4) coordinated a time with users / groups to do a 2nd rsync when they > >> weren?t active > >> 5) used symbolic links during the transition (i.e. rm -rvf > >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) > >> 6) once everybody was migrated, updated the symlinks (i.e. /home > >> became a symlink to /gpfs2/home) > >> > >> HTHAL? > >> > >> Kevin > >> > >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com > >>> wrote: > >>> > >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to > >>> refresh hardware. Any lessons learned in this process? Is it > >>> easiest to just build new cluster and then use AFM? Add to existing > >>> cluster then decommission nodes? What is the recommended process for > >>> this? > >>> > >>> > >>> Mark > >>> > >>> This message (including any attachments) is intended only for the use > >>> of the individual or entity to which it is addressed and may contain > >>> information that is non-public, proprietary, privileged, > >>> confidential, and exempt from disclosure under applicable law. If you > >>> are not the intended recipient, you are hereby notified that any use, > >>> dissemination, distribution, or copying of this communication is > >>> strictly prohibited. This message may be viewed by parties at Sirius > >>> Computer Solutions other than those named in the message header. This > >>> message does not contain an official representation of Sirius > >>> Computer Solutions. If you have received this communication in error, > >>> notify Sirius Computer Solutions immediately and (i) destroy this > >>> message if a facsimile or (ii) delete this message immediately if > >>> this is an electronic communication. Thank you. > >>> > >>> Sirius Computer Solutions > >>> _______________________________________________ > >>> gpfsug-discuss mailing list > >>> gpfsug-discuss at spectrumscale.org > >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > >> ? > >> Kevin Buterbaugh - Senior System Administrator > >> Vanderbilt University - Advanced Computing Center for Research and > >> Education > >> Kevin.Buterbaugh at vanderbilt.edu > >> - (615)875-9633 > >> > >> > >> > > > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > > Oy IBM Finland Ab > > PL 265, 00101 Helsinki, Finland > > Business ID, Y-tunnus: 0195876-3 > > Registered in Finland > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. > > Sirius Computer Solutions_______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From shankbal at in.ibm.com Thu Oct 13 11:43:23 2016 From: shankbal at in.ibm.com (Shankar Balasubramanian) Date: Thu, 13 Oct 2016 16:13:23 +0530 Subject: [gpfsug-discuss] Hardware refresh - In-Reply-To: References: <9E337B56-A274-41B6-84C2-7084AAB771F4@siriuscom.com><279BAB6D-21E6-4EC1-98CD-CA409A24F8F6@ulmer.org> Message-ID: Please note - though one of the supported use case of AFM is such migration scenarios, infact AFM does particularly well in faithfully migrating the data when the source and destination cluster are GPFS, the scalability of this solution for multi million/multi terabyte file systems has its own set of challenges. These have to carefully understood and checked if AFM will fit the bill. Best Regards, Shankar Balasubramanian STSM, AFM & Async DR Development IBM Systems Bangalore - Embassy Golf Links India From: "Marc A Kaplan" To: gpfsug main discussion list Date: 10/12/2016 11:25 PM Subject: Re: [gpfsug-discuss] Hardware refresh - Sent by: gpfsug-discuss-bounces at spectrumscale.org Yes, you can AFM within a single cluster, in fact with just a single node. I just set this up on my toy system: [root at bog-wifi cmvc]# mmlsfileset yy afmlu --afm Filesets in file system 'yy': Name Status Path afmTarget afmlu Linked /yy/afmlu gpfs:///xx [root at bog-wifi cmvc]# mount ... yy on /yy type gpfs (rw,relatime,seclabel) xx on /xx type gpfs (rw,relatime,seclabel) [root at bog-wifi cmvc]# mmafmctl yy getstate Fileset Name Fileset Target Cache State Gateway Node Queue Length Queue numExec ------------ -------------- ------------- ------------ ------------ ------------- afmlu gpfs:///xx Active bog-wifi 0 7 So, you may add nodes, add disks to an existing cluster, upgrade your software, define a new FS, migrate data from an old FS to a new FS then delete nodes and disks that are no longer needed... From: Stephen Ulmer To: gpfsug main discussion list Date: 10/11/2016 09:30 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org I think that the OP was asking why not expand the existing cluster with the new hardware, and just make a new FS? I?ve not tried to make a cluster talk AFM to itself yet. If that?s impossible, then there?s one good reason to make a new cluster (to use AFM for migration). Liberty, -- Stephen On Oct 11, 2016, at 8:40 PM, Mark.Bush at siriuscom.comwrote: Only compelling reason for new cluster would be old hardware is EOL or no longer want to pay maintenance on it. From: on behalf of Marc A Kaplan Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 2:58 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh New FS? Yes there are some good reasons. New cluster? I did not see a compelling argument either way. From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Date: 10/11/2016 03:34 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org Ok. I think I am hearing that a new cluster with a new FS and copying data from old to new cluster is the best way forward. Thanks everyone for your input. From: on behalf of Yuri L Volobuev Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 12:22 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh This depends on the committed cluster version level (minReleaseLevel) and file system format. Since NFSv2 is an on-disk format change, older code wouldn't be able to understand what it is, and thus if there's a possibility of a downlevel node looking at the NSD, the NFSv1 format is going to be used. The code does NSDv1<->NSDv2 conversions under the covers as needed when adding an empty NSD to a file system. I'd strongly recommend getting a fresh start by formatting a new file system. Many things have changed over the course of the last few years. In particular, having a 4K-aligned file system can be a pretty big deal, depending on what hardware one is going to deploy in the future, and this is something that can't be bolted onto an existing file system. Having 4K inodes is very handy for many reasons. New directory format and NSD format changes are attractive, too. And disks generally tend to get larger with time, and at some point you may want to add a disk to an existing storage pool that's larger than the existing allocation map format allows. Obviously, it's more hassle to migrate data to a new file system, as opposed to extending an existing one. In a perfect world, GPFS would offer a conversion tool that seamlessly and robustly converts old file systems, making them as good as new, but in the real world such a tool doesn't exist. Getting a clean slate by formatting a new file system every few years is a good long-term investment of time, although it comes front-loaded with extra work. yuri Aaron Knister ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron From: Aaron Knister To: , Date: 10/10/2016 04:45 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron On 10/10/16 7:40 PM, Luis Bolinches wrote: > Hi > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > reason to do so. > > AFM for migrations is quite good, latest versions allows to use NSD > protocol for mounts as well. Olaf did a great job explaining this > scenario on the redbook chapter 6 > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > -- > Cheers > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > wrote: > >> Hi Mark, >> >> The last time we did something like this was 2010 (we?re doing rolling >> refreshes now), so there are probably lots of better ways to do this >> than what we did, but we: >> >> 1) set up the new hardware >> 2) created new filesystems (so that we could make adjustments we >> wanted to make that can only be made at FS creation time) >> 3) used rsync to make a 1st pass copy of everything >> 4) coordinated a time with users / groups to do a 2nd rsync when they >> weren?t active >> 5) used symbolic links during the transition (i.e. rm -rvf >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >> 6) once everybody was migrated, updated the symlinks (i.e. /home >> became a symlink to /gpfs2/home) >> >> HTHAL? >> >> Kevin >> >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>> wrote: >>> >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>> refresh hardware. Any lessons learned in this process? Is it >>> easiest to just build new cluster and then use AFM? Add to existing >>> cluster then decommission nodes? What is the recommended process for >>> this? >>> >>> >>> Mark >>> >>> This message (including any attachments) is intended only for the use >>> of the individual or entity to which it is addressed and may contain >>> information that is non-public, proprietary, privileged, >>> confidential, and exempt from disclosure under applicable law. If you >>> are not the intended recipient, you are hereby notified that any use, >>> dissemination, distribution, or copying of this communication is >>> strictly prohibited. This message may be viewed by parties at Sirius >>> Computer Solutions other than those named in the message header. This >>> message does not contain an official representation of Sirius >>> Computer Solutions. If you have received this communication in error, >>> notify Sirius Computer Solutions immediately and (i) destroy this >>> message if a facsimile or (ii) delete this message immediately if >>> this is an electronic communication. Thank you. >>> >>> Sirius Computer Solutions >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and >> Education >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From mimarsh2 at vt.edu Thu Oct 13 14:30:03 2016 From: mimarsh2 at vt.edu (Brian Marshall) Date: Thu, 13 Oct 2016 09:30:03 -0400 Subject: [gpfsug-discuss] SC16 Draft Agenda [Save the date: Sunday, November 13th] In-Reply-To: References: Message-ID: Virginia Tech ARC purchased a SANDisk IF150 (IBM DeepFlash 150) this Summer and are just getting it into production (maybe next week). I have some dev benchmarking numbers and may have production/user feedback by SC16. I'm not sure I can fill an entire 30 minutes with worthwhile information, but I can give at least 15 minutes on: DeepFlash: a computational scientist's first impressions Best, Brian Marshall On Mon, Oct 10, 2016 at 6:59 PM, GPFS UG USA Principal < usa-principal at gpfsug.org> wrote: > Hello all, > > There have been some questions about the Spectrum Scale Users Group event > for SC16 and we thought it best to publish our draft agenda and continue to > fill it in as details get settled. That way you can make your travel plans > and schedules. > > The date and rough times are set, we may adjust the time range a bit > depending upon the number of user talks that people would like to give. So > note, yes! *we are asking for user talks. Please consider doing this. *We > always receive positive feedback about talks given by users that describe > real world experiences. If *any* of the user talk slots below are of > interest to you, please let us know as soon as you can. We may turn at > least one user talk into a panel if we can get enough participants. > > As always, your feedback is welcome. This is a *users* group after all. > > So, here?s what we have planned so far: > > *Date: Sunday November 13th* > *Venue: TBD, but near the convention center in downtown SLC* > *Time: ~12:30p - ~17:30p* > > *Agenda:* > > > > > > > > > *12:30 - 12:50 Welcome / Overview12:50 - 13:50 The latest in IBM Spectrum > Scale and ESS13:50 - 14:20 User Talk: Big Data in HPC14:20 - 14:50 User > Talk: Tiering to Object Storage14:50 - 15:20 - break -15:20 - 15:50 > User Talk: TBD ? volunteers! please! 15:50 - 16:35 Spectrum Scale > Enhancements for CORAL16:35 - 17:20 News from IBM Research17:20 - 17:30 > Closing* > > Best, > > Kristy & Bob > > Kristy Kallback-Rose > Manager, Research Storage > Indiana University > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Thu Oct 13 14:55:14 2016 From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]) Date: Thu, 13 Oct 2016 13:55:14 +0000 Subject: [gpfsug-discuss] SC16 Draft Agenda [Save the date: Sunday, November 13th] Message-ID: <5F910253243E6A47B81A9A2EB424BBA101D90737@NDMSMBX404.ndc.nasa.gov> I'd be quite interested in hearing about that Brian. I have to double check this with a couple folks here first but is there interest in hear about some of the stuff NASA has been up to with GPFS? I'd be tickled pink to share details on our GPFS activities: - FPO/Hadoop efforts - a site report (we're not *huge* but we're a modest Asize - 3600 nodes and almost 50PB at this point) - demoing the monitoring dashboard we've created with influxdb and grafana. - talk about the tools we've developed to deal with mass uid number changes on GPFS filesystems -Aaron From: Brian Marshall Sent: 10/13/16, 9:30 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] SC16 Draft Agenda [Save the date: Sunday, November 13th] Virginia Tech ARC purchased a SANDisk IF150 (IBM DeepFlash 150) this Summer and are just getting it into production (maybe next week). I have some dev benchmarking numbers and may have production/user feedback by SC16. I'm not sure I can fill an entire 30 minutes with worthwhile information, but I can give at least 15 minutes on: DeepFlash: a computational scientist's first impressions Best, Brian Marshall On Mon, Oct 10, 2016 at 6:59 PM, GPFS UG USA Principal > wrote: Hello all, There have been some questions about the Spectrum Scale Users Group event for SC16 and we thought it best to publish our draft agenda and continue to fill it in as details get settled. That way you can make your travel plans and schedules. The date and rough times are set, we may adjust the time range a bit depending upon the number of user talks that people would like to give. So note, yes! we are asking for user talks. Please consider doing this. We always receive positive feedback about talks given by users that describe real world experiences. If any of the user talk slots below are of interest to you, please let us know as soon as you can. We may turn at least one user talk into a panel if we can get enough participants. As always, your feedback is welcome. This is a users group after all. So, here?s what we have planned so far: Date: Sunday November 13th Venue: TBD, but near the convention center in downtown SLC Time: ~12:30p - ~17:30p Agenda: 12:30 - 12:50 Welcome / Overview 12:50 - 13:50 The latest in IBM Spectrum Scale and ESS 13:50 - 14:20 User Talk: Big Data in HPC 14:20 - 14:50 User Talk: Tiering to Object Storage 14:50 - 15:20 - break - 15:20 - 15:50 User Talk: TBD ? volunteers! please! 15:50 - 16:35 Spectrum Scale Enhancements for CORAL 16:35 - 17:20 News from IBM Research 17:20 - 17:30 Closing Best, Kristy & Bob Kristy Kallback-Rose Manager, Research Storage Indiana University _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew at ellexus.com Thu Oct 13 16:00:54 2016 From: matthew at ellexus.com (Matthew Harris) Date: Thu, 13 Oct 2016 16:00:54 +0100 Subject: [gpfsug-discuss] SC16 Draft Agenda [Save the date: Sunday, November 13th] In-Reply-To: References: Message-ID: Who is delivering the 12:50 talk please? Regards, Matthew Harris Account Manager & Business Development - Ellexus Ltd *www.ellexus.com * *Ellexus Ltd is a limited company registered in England & Wales* *Company registration no. 07166034* *Registered address: 198 High Street, Tonbridge, Kent TN9 1BE, UK* *Operating address: St John's Innovation Centre, Cowley Road, Cambridge CB4 0WS, UK* On Mon, Oct 10, 2016 at 11:59 PM, GPFS UG USA Principal < usa-principal at gpfsug.org> wrote: > Hello all, > > There have been some questions about the Spectrum Scale Users Group event > for SC16 and we thought it best to publish our draft agenda and continue to > fill it in as details get settled. That way you can make your travel plans > and schedules. > > The date and rough times are set, we may adjust the time range a bit > depending upon the number of user talks that people would like to give. So > note, yes! *we are asking for user talks. Please consider doing this. *We > always receive positive feedback about talks given by users that describe > real world experiences. If *any* of the user talk slots below are of > interest to you, please let us know as soon as you can. We may turn at > least one user talk into a panel if we can get enough participants. > > As always, your feedback is welcome. This is a *users* group after all. > > So, here?s what we have planned so far: > > *Date: Sunday November 13th* > *Venue: TBD, but near the convention center in downtown SLC* > *Time: ~12:30p - ~17:30p* > > *Agenda:* > > > > > > > > > *12:30 - 12:50 Welcome / Overview12:50 - 13:50 The latest in IBM Spectrum > Scale and ESS13:50 - 14:20 User Talk: Big Data in HPC14:20 - 14:50 User > Talk: Tiering to Object Storage14:50 - 15:20 - break -15:20 - 15:50 > User Talk: TBD ? volunteers! please! 15:50 - 16:35 Spectrum Scale > Enhancements for CORAL16:35 - 17:20 News from IBM Research17:20 - 17:30 > Closing* > > Best, > > Kristy & Bob > > Kristy Kallback-Rose > Manager, Research Storage > Indiana University > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Thu Oct 13 16:27:14 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Thu, 13 Oct 2016 11:27:14 -0400 Subject: [gpfsug-discuss] Hardware refresh - In-Reply-To: References: <9E337B56-A274-41B6-84C2-7084AAB771F4@siriuscom.com><279BAB6D-21E6-4EC1-98CD-CA409A24F8F6@ulmer.org> Message-ID: IMO, it is simplest to have both the old and new file systems both mounted on the same node(s). Then you can use AFM and/or any other utilities to migrate/copy your files from the old file system to the new. From: "Shankar Balasubramanian" To: gpfsug main discussion list Date: 10/13/2016 06:46 AM Subject: Re: [gpfsug-discuss] Hardware refresh - Sent by: gpfsug-discuss-bounces at spectrumscale.org Please note - though one of the supported use case of AFM is such migration scenarios, infact AFM does particularly well in faithfully migrating the data when the source and destination cluster are GPFS, the scalability of this solution for multi million/multi terabyte file systems has its own set of challenges. These have to carefully understood and checked if AFM will fit the bill. Best Regards, Shankar Balasubramanian STSM, AFM & Async DR Development IBM Systems Bangalore - Embassy Golf Links India "Marc A Kaplan" ---10/12/2016 11:25:20 PM---Yes, you can AFM within a single cluster, in fact with just a single node. I just set this up on m From: "Marc A Kaplan" To: gpfsug main discussion list Date: 10/12/2016 11:25 PM Subject: Re: [gpfsug-discuss] Hardware refresh - Sent by: gpfsug-discuss-bounces at spectrumscale.org Yes, you can AFM within a single cluster, in fact with just a single node. I just set this up on my toy system: [root at bog-wifi cmvc]# mmlsfileset yy afmlu --afm Filesets in file system 'yy': Name Status Path afmTarget afmlu Linked /yy/afmlu gpfs:///xx [root at bog-wifi cmvc]# mount ... yy on /yy type gpfs (rw,relatime,seclabel) xx on /xx type gpfs (rw,relatime,seclabel) [root at bog-wifi cmvc]# mmafmctl yy getstate Fileset Name Fileset Target Cache State Gateway Node Queue Length Queue numExec ------------ -------------- ------------- ------------ ------------ ------------- afmlu gpfs:///xx Active bog-wifi 0 7 So, you may add nodes, add disks to an existing cluster, upgrade your software, define a new FS, migrate data from an old FS to a new FS then delete nodes and disks that are no longer needed... From: Stephen Ulmer To: gpfsug main discussion list Date: 10/11/2016 09:30 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org I think that the OP was asking why not expand the existing cluster with the new hardware, and just make a new FS? I?ve not tried to make a cluster talk AFM to itself yet. If that?s impossible, then there?s one good reason to make a new cluster (to use AFM for migration). Liberty, -- Stephen On Oct 11, 2016, at 8:40 PM, Mark.Bush at siriuscom.comwrote: Only compelling reason for new cluster would be old hardware is EOL or no longer want to pay maintenance on it. From: on behalf of Marc A Kaplan Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 2:58 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh New FS? Yes there are some good reasons. New cluster? I did not see a compelling argument either way. From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Date: 10/11/2016 03:34 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org Ok. I think I am hearing that a new cluster with a new FS and copying data from old to new cluster is the best way forward. Thanks everyone for your input. From: on behalf of Yuri L Volobuev Reply-To: gpfsug main discussion list Date: Tuesday, October 11, 2016 at 12:22 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Hardware refresh This depends on the committed cluster version level (minReleaseLevel) and file system format. Since NFSv2 is an on-disk format change, older code wouldn't be able to understand what it is, and thus if there's a possibility of a downlevel node looking at the NSD, the NFSv1 format is going to be used. The code does NSDv1<->NSDv2 conversions under the covers as needed when adding an empty NSD to a file system. I'd strongly recommend getting a fresh start by formatting a new file system. Many things have changed over the course of the last few years. In particular, having a 4K-aligned file system can be a pretty big deal, depending on what hardware one is going to deploy in the future, and this is something that can't be bolted onto an existing file system. Having 4K inodes is very handy for many reasons. New directory format and NSD format changes are attractive, too. And disks generally tend to get larger with time, and at some point you may want to add a disk to an existing storage pool that's larger than the existing allocation map format allows. Obviously, it's more hassle to migrate data to a new file system, as opposed to extending an existing one. In a perfect world, GPFS would offer a conversion tool that seamlessly and robustly converts old file systems, making them as good as new, but in the real world such a tool doesn't exist. Getting a clean slate by formatting a new file system every few years is a good long-term investment of time, although it comes front-loaded with extra work. yuri Aaron Knister ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron From: Aaron Knister To: , Date: 10/10/2016 04:45 PM Subject: Re: [gpfsug-discuss] Hardware refresh Sent by: gpfsug-discuss-bounces at spectrumscale.org Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? -Aaron On 10/10/16 7:40 PM, Luis Bolinches wrote: > Hi > > Creating a new FS sounds like a best way to go. NSDv2 being a very good > reason to do so. > > AFM for migrations is quite good, latest versions allows to use NSD > protocol for mounts as well. Olaf did a great job explaining this > scenario on the redbook chapter 6 > > http://www.redbooks.ibm.com/abstracts/sg248254.html?Open > > -- > Cheers > > On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L > > wrote: > >> Hi Mark, >> >> The last time we did something like this was 2010 (we?re doing rolling >> refreshes now), so there are probably lots of better ways to do this >> than what we did, but we: >> >> 1) set up the new hardware >> 2) created new filesystems (so that we could make adjustments we >> wanted to make that can only be made at FS creation time) >> 3) used rsync to make a 1st pass copy of everything >> 4) coordinated a time with users / groups to do a 2nd rsync when they >> weren?t active >> 5) used symbolic links during the transition (i.e. rm -rvf >> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >> 6) once everybody was migrated, updated the symlinks (i.e. /home >> became a symlink to /gpfs2/home) >> >> HTHAL? >> >> Kevin >> >>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>> wrote: >>> >>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>> refresh hardware. Any lessons learned in this process? Is it >>> easiest to just build new cluster and then use AFM? Add to existing >>> cluster then decommission nodes? What is the recommended process for >>> this? >>> >>> >>> Mark >>> >>> This message (including any attachments) is intended only for the use >>> of the individual or entity to which it is addressed and may contain >>> information that is non-public, proprietary, privileged, >>> confidential, and exempt from disclosure under applicable law. If you >>> are not the intended recipient, you are hereby notified that any use, >>> dissemination, distribution, or copying of this communication is >>> strictly prohibited. This message may be viewed by parties at Sirius >>> Computer Solutions other than those named in the message header. This >>> message does not contain an official representation of Sirius >>> Computer Solutions. If you have received this communication in error, >>> notify Sirius Computer Solutions immediately and (i) destroy this >>> message if a facsimile or (ii) delete this message immediately if >>> this is an electronic communication. Thank you. >>> >>> Sirius Computer Solutions >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and >> Education >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 105 bytes Desc: not available URL: From aaron.s.knister at nasa.gov Fri Oct 14 03:01:32 2016 From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]) Date: Fri, 14 Oct 2016 02:01:32 +0000 Subject: [gpfsug-discuss] 4K sector NSD support (was: Hardware refresh) In-Reply-To: References: <63bd99ad-f684-d1bb-371f-e2c21849ff29@nasa.gov> , Message-ID: <5F910253243E6A47B81A9A2EB424BBA101D91022@NDMSMBX404.ndc.nasa.gov> Hi Yuri, Thank you for your excellent reply. The immediate issue ahead of us is getting the 4K LUNs formatted as NSDs into the filesystem and since we can do that life is good for a while. The next hurdle for me won't be for another 3 years when I'll likely need to bring into existing filesystems metadata NSDs that have 4k sectors. I wonder if in that time frame a migration tool could be developed to convert existing filesystems to 4K metadata? I do recognize what you're saying, though, about return on investment of effort from a developing a tool to migrate the on-disk format vs migrating the entire filesystem. What were you thinking as far as ways to strengthen the migration story? In an ideal world I'd like to be able to pull the trigger on a migration and have it have no visible impact to end-users other than an understandable performance impact. If a mechanism existed to move data to a new filesystem (a hypothetical mmreplacefs comes to mind) in-place, that would be quite wonderful. At a high level, perhaps one would create a new filesystem that wouldn't be directly mounted but would be an "internal mount". An admin would then issue an mmreplacefs $OLD_FS $NEW_FS command. All activity would be quiesced on the old filesystem and in the background an AFM-like process would be initiated. Files not on the new fs would be migrated in the background. Once the migration process is complete the old fs would perhaps be renamed and the new fs would effectively take its place. This raises more questions in my mind than it does provide answers, such as how quotas would be handled during the migration, how filesets would get migrated/created, how ILM/DMAPI would be affected during the migration. I'll give it some more thought, but understanding a little more about what you were thinking would help me craft the RFE. -Aaron ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Yuri L Volobuev [volobuev at us.ibm.com] Sent: Wednesday, October 12, 2016 1:44 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] 4K sector NSD support (was: Hardware refresh) Yes, it is possible to add a 4KN dataOnly NSD to a non-4K-aligned file system, as you figured out. This is something we didn't plan on doing originally, but then had to implement based on the feedback from the field. There's clearly a need for this. However, this statement is exactly it -- dataOnly NSDs. The only way to put metadata on a 4KN disk is to use a 4K-aligned file system. There are several kinds of metadata present in non-4K-aligned file system that generate non-4K IOs (512 byte inodes being the biggest problem), and there's no way to work around this short of using the new format, and there's no way to perform a conversion to the new format in-place. You're welcome to submit an RFE, of course, but I'd recommend being pragmatic about the chances of such an RFE being implemented. As you can imagine, the main reason why an all-encompassing file system conversion tool doesn't exist is not GPFS developers having no idea that such a tool is wanted. There are several considerations that conspire to make this an unlikely candidate to ever be implemented: 1) The task is hard and has no finish line. In most GPFS releases, something changes, necessitating an added piece of work for the hypothetical conversion tool, and the matrix of from-to format version combinations gets to be big quite quickly. 2) A file system conversion is something that is needed very infrequently, but when this code does run, it absolutely has to run and run perfectly, else the result would be a half-converted file system, i.e. a royal mess. This is a tester's nightmare. 3) The failure scenarios are all unpalatable. What should the conversion tool do if it runs out of space replacing smaller metadata structures with bigger ones? Undoing a partially finished conversion is even harder than doing it in the first place. 4) Doing an on-disk conversion on-line is simply very difficult. Consider the task of converting an inode file to use a different inode size. The file can be huge (billions of records), and it would take a fair chunk of time to rewrite it, but the file is changing while it's being converted (can't simply lock the whole thing down for so long), simultaneously on multiple nodes. Orchestrating the processing of updates in the presence of two inode files, with proper atomicity guarantees (to guard against a node failure) is a task of considerable complexity. None of this means the task is impossible, of course. It is, however, a very big chunk of very complex work, all towards a tool that on an average cluster may run somewhere between zero and one times, not something that benefits day-to-day operations. Where the complexity of the task allows for a reasonably affordable implementation, e.g. conversion from an old-style EA file to the FASTEA format, a conversion tool has been implemented (mmmigratefs). However, doing this for every single changed aspect of the file system format is simply too expensive to justify, given other tasks in front of us. On the other hand, a well-implemented migration mechanism solves the file system reformatting scenario (which covers all aspects of file system format changes) as well as a number of other scenarios. This is a cleaner, more general solution. Migration doesn't have to mean an outage. A simple rsync-based migration requires downtime for a cutover, while an AFM-based migration doesn't necessarily require one. I'm not saying that GPFS has a particularly strong migration story at the moment, but this is a much more productive direction for applying resources than a mythical all-encompassing conversion tool. yuri [Inactive hide details for Aaron Knister ---10/11/2016 05:59:25 PM---Yuri, (Sorry for being somewhat spammy) I now understand th]Aaron Knister ---10/11/2016 05:59:25 PM---Yuri, (Sorry for being somewhat spammy) I now understand the limitation after From: Aaron Knister To: , Date: 10/11/2016 05:59 PM Subject: Re: [gpfsug-discuss] 4K sector NSD support (was: Hardware refresh) Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Yuri, (Sorry for being somewhat spammy) I now understand the limitation after some more testing (I'm a hands-on learner, can you tell?). Given the right code/cluster/fs version levels I can add 4K dataOnly NSDv2 NSDs to a filesystem created with NSDv1 NSDs. What I can't do is seemingly add any metadataOnly or dataAndMetadata 4K luns to an fs that is not 4K aligned which I assume would be any fs originally created with NSDv1 LUNs. It seems possible to move all data away from NSDv1 LUNS in a filesystem behind-the-scenes using GPFS migration tools, and move the data to NSDv2 LUNs. In this case I believe what's missing is a tool to convert just the metadata structures to be 4K aligned since the data would already on 4k-based NSDv2 LUNS, is that the case? I'm trying to figure out what exactly I'm asking for in an RFE. -Aaron On 10/11/16 7:57 PM, Aaron Knister wrote: > I think I was a little quick to the trigger. I re-read your last mail > after doing some testing and understand it differently. I was wrong > about my interpretation-- you can add 4K NSDv2 formatted NSDs to a > filesystem previously created with NSDv1 NSDs assuming, as you say, the. > minReleaseLevel and filesystem version are high enough. That negates > about half of my last e-mail. The fs still doesn't show as 4K aligned: > > loressd01:~ # /usr/lpp/mmfs/bin/mmlsfs tnb4k --is4KAligned > flag value description > ------------------- ------------------------ > ----------------------------------- > --is4KAligned No is4KAligned? > > but *shrug* most of the I/O to these disks should be 1MB anyway. If > somebody is pounding the FS with smaller than 4K I/O they're gonna get a > talkin' to. > > -Aaron > > On 10/11/16 6:41 PM, Aaron Knister wrote: >> Thanks Yuri. >> >> I'm asking for my own purposes but I think it's still relevant here: >> we're still at GPFS 3.5 and will be adding dataOnly NSDs with 4K sectors >> in the near future. We're planning to update to 4.1 before we format >> these NSDs, though. If I understand you correctly we can't bring these >> 4K NSDv2 NSDs into a filesystem with 512b-based NSDv1 NSDs? That's a >> pretty big deal :( >> >> Reformatting every few years with 10's of petabytes of data is not >> realistic for us (it would take years to move the data around). It also >> goes against my personal preachings about GPFS's storage virtualization >> capabilities: the ability to perform upgrades/make underlying storage >> infrastructure changes with behind-the-scenes data migration, >> eliminating much of the manual hassle of storage administrators doing >> rsync dances. I guess it's RFE time? It also seems as though AFM could >> help with automating the migration, although many of our filesystems do >> not have filesets on them so we would have to re-think how we lay out >> our filesystems. >> >> This is also curious to me with IBM pitching GPFS as a filesystem for >> cloud services (the cloud *never* goes down, right?). Granted I believe >> this pitch started after the NSDv2 format was defined, but if somebody >> is building a large cloud with GPFS as the underlying filesystem for an >> object or an image store one might think the idea of having to re-format >> the filesystem to gain access to critical new features is inconsistent >> with this pitch. It would be hugely impactful. Just my $.02. >> >> As you can tell, I'm frustrated there's no online conversion tool :) Not >> that there couldn't be... you all are brilliant developers. >> >> -Aaron >> >> On 10/11/16 1:22 PM, Yuri L Volobuev wrote: >>> This depends on the committed cluster version level (minReleaseLevel) >>> and file system format. Since NFSv2 is an on-disk format change, older >>> code wouldn't be able to understand what it is, and thus if there's a >>> possibility of a downlevel node looking at the NSD, the NFSv1 format is >>> going to be used. The code does NSDv1<->NSDv2 conversions under the >>> covers as needed when adding an empty NSD to a file system. >>> >>> I'd strongly recommend getting a fresh start by formatting a new file >>> system. Many things have changed over the course of the last few years. >>> In particular, having a 4K-aligned file system can be a pretty big deal, >>> depending on what hardware one is going to deploy in the future, and >>> this is something that can't be bolted onto an existing file system. >>> Having 4K inodes is very handy for many reasons. New directory format >>> and NSD format changes are attractive, too. And disks generally tend to >>> get larger with time, and at some point you may want to add a disk to an >>> existing storage pool that's larger than the existing allocation map >>> format allows. Obviously, it's more hassle to migrate data to a new file >>> system, as opposed to extending an existing one. In a perfect world, >>> GPFS would offer a conversion tool that seamlessly and robustly converts >>> old file systems, making them as good as new, but in the real world such >>> a tool doesn't exist. Getting a clean slate by formatting a new file >>> system every few years is a good long-term investment of time, although >>> it comes front-loaded with extra work. >>> >>> yuri >>> >>> Inactive hide details for Aaron Knister ---10/10/2016 04:45:31 PM---Can >>> one format NSDv2 NSDs and put them in a filesystem withAaron Knister >>> ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a >>> filesystem with NSDv1 NSD's? -Aaron >>> >>> From: Aaron Knister >>> To: , >>> Date: 10/10/2016 04:45 PM >>> Subject: Re: [gpfsug-discuss] Hardware refresh >>> Sent by: gpfsug-discuss-bounces at spectrumscale.org >>> >>> ------------------------------------------------------------------------ >>> >>> >>> >>> Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's? >>> >>> -Aaron >>> >>> On 10/10/16 7:40 PM, Luis Bolinches wrote: >>>> Hi >>>> >>>> Creating a new FS sounds like a best way to go. NSDv2 being a very good >>>> reason to do so. >>>> >>>> AFM for migrations is quite good, latest versions allows to use NSD >>>> protocol for mounts as well. Olaf did a great job explaining this >>>> scenario on the redbook chapter 6 >>>> >>>> http://www.redbooks.ibm.com/abstracts/sg248254.html?Open >>>> >>>> -- >>>> Cheers >>>> >>>> On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L >>>> >>> > wrote: >>>> >>>>> Hi Mark, >>>>> >>>>> The last time we did something like this was 2010 (we?re doing rolling >>>>> refreshes now), so there are probably lots of better ways to do this >>>>> than what we did, but we: >>>>> >>>>> 1) set up the new hardware >>>>> 2) created new filesystems (so that we could make adjustments we >>>>> wanted to make that can only be made at FS creation time) >>>>> 3) used rsync to make a 1st pass copy of everything >>>>> 4) coordinated a time with users / groups to do a 2nd rsync when they >>>>> weren?t active >>>>> 5) used symbolic links during the transition (i.e. rm -rvf >>>>> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser) >>>>> 6) once everybody was migrated, updated the symlinks (i.e. /home >>>>> became a symlink to /gpfs2/home) >>>>> >>>>> HTHAL? >>>>> >>>>> Kevin >>>>> >>>>>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com >>>>>> wrote: >>>>>> >>>>>> Have a very old cluster built on IBM X3650?s and DS3500. Need to >>>>>> refresh hardware. Any lessons learned in this process? Is it >>>>>> easiest to just build new cluster and then use AFM? Add to existing >>>>>> cluster then decommission nodes? What is the recommended process for >>>>>> this? >>>>>> >>>>>> >>>>>> Mark >>>>>> >>>>>> This message (including any attachments) is intended only for the use >>>>>> of the individual or entity to which it is addressed and may contain >>>>>> information that is non-public, proprietary, privileged, >>>>>> confidential, and exempt from disclosure under applicable law. If you >>>>>> are not the intended recipient, you are hereby notified that any use, >>>>>> dissemination, distribution, or copying of this communication is >>>>>> strictly prohibited. This message may be viewed by parties at Sirius >>>>>> Computer Solutions other than those named in the message header. This >>>>>> message does not contain an official representation of Sirius >>>>>> Computer Solutions. If you have received this communication in error, >>>>>> notify Sirius Computer Solutions immediately and (i) destroy this >>>>>> message if a facsimile or (ii) delete this message immediately if >>>>>> this is an electronic communication. Thank you. >>>>>> >>>>>> Sirius Computer Solutions >>>>>> _______________________________________________ >>>>>> gpfsug-discuss mailing list >>>>>> gpfsug-discuss at spectrumscale.org >>>>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>>>> >>>>> ? >>>>> Kevin Buterbaugh - Senior System Administrator >>>>> Vanderbilt University - Advanced Computing Center for Research and >>>>> Education >>>>> Kevin.Buterbaugh at vanderbilt.edu >>>>> - (615)875-9633 >>>>> >>>>> >>>>> >>>> >>>> Ellei edell? ole toisin mainittu: / Unless stated otherwise above: >>>> Oy IBM Finland Ab >>>> PL 265, 00101 Helsinki, Finland >>>> Business ID, Y-tunnus: 0195876-3 >>>> Registered in Finland >>>> >>>> >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> >>> >>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: graycol.gif URL: From daniel.kidger at uk.ibm.com Fri Oct 14 11:09:18 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Fri, 14 Oct 2016 10:09:18 +0000 Subject: [gpfsug-discuss] testing commands In-Reply-To: <797ae533-edee-102a-3f3e-f53687cfcf83@nasa.gov> Message-ID: An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Fri Oct 14 14:30:55 2016 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Fri, 14 Oct 2016 13:30:55 +0000 Subject: [gpfsug-discuss] LROC benefits Message-ID: Hi all, Is there any way to gauge the performance benefits that could be realised from installing some LROCs in our CES cluster nodes? Or just suck it and see as it were? Cheers Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Sat Oct 15 15:23:01 2016 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Sat, 15 Oct 2016 10:23:01 -0400 Subject: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter Message-ID: <1c7562cf-d61a-adda-66b4-5fd00d75ae75@nasa.gov> I've got a node that's got some curious waiters on it (see below). Could someone explain what the "SGExceptionLogBufferFullThread" waiter means? Thanks! -Aaron === mmdiag: waiters === 0x7FFFF040D600 waiting 0.038822715 seconds, SGExceptionLogBufferFullThread: on ThCond 0x7FFFDBB07628 (0x7FFFDBB07628) (parallelWaitCond), reason 'wait for parallel write' for NSD I/O completion on node 10.1.53.5 0x7FFFE83F3D60 waiting 0.039629116 seconds, CleanBufferThread: on ThCond 0x17B1488 (0x17B1488) (MsgRecordCondvar), reason 'RPC wait' for NSD I/O completion on node 10.1.53.7 0x7FFFE8373A90 waiting 0.038921480 seconds, CleanBufferThread: on ThCond 0x7FFFCD2B4E30 (0x7FFFCD2B4E30) (LogFileBufferDescriptorCondvar), reason 'force wait on force active buffer write' 0x42CD9B0 waiting 0.028227004 seconds, CleanBufferThread: on ThCond 0x7FFFCD2B4E30 (0x7FFFCD2B4E30) (LogFileBufferDescriptorCondvar), reason 'force wait for buffer write to complete' 0x7FFFE0F0EAD0 waiting 0.027864343 seconds, CleanBufferThread: on ThCond 0x7FFFDC0EEA88 (0x7FFFDC0EEA88) (MsgRecordCondvar), reason 'RPC wait' for NSD I/O completion on node 10.1.53.7 0x1575560 waiting 0.028045975 seconds, RemoveHandlerThread: on ThCond 0x18020CE4E08 (0xFFFFC90020CE4E08) (LkObjCondvar), reason 'waiting for LX lock' 0x1570560 waiting 0.038724949 seconds, CreateHandlerThread: on ThCond 0x18020CE50A0 (0xFFFFC90020CE50A0) (LkObjCondvar), reason 'waiting for LX lock' 0x1563D60 waiting 0.073919918 seconds, RemoveHandlerThread: on ThCond 0x180235F6440 (0xFFFFC900235F6440) (LkObjCondvar), reason 'waiting for LX lock' 0x1561560 waiting 0.054854513 seconds, RemoveHandlerThread: on ThCond 0x1802292D200 (0xFFFFC9002292D200) (LkObjCondvar), reason 'waiting for LX lock' From oehmes at gmail.com Sat Oct 15 15:50:13 2016 From: oehmes at gmail.com (Sven Oehme) Date: Sat, 15 Oct 2016 14:50:13 +0000 Subject: [gpfsug-discuss] LROC benefits In-Reply-To: References: Message-ID: all depends on workload. we will publish a paper comparing mixed workload with and without LROC pretty soon. most numbers i have seen show anywhere between 30% - 1000% (very rare case) improvements, so its for sure worth a test. sven On Fri, Oct 14, 2016 at 6:31 AM Sobey, Richard A wrote: > Hi all, > > > > Is there any way to gauge the performance benefits that could be realised > from installing some LROCs in our CES cluster nodes? Or just suck it and > see as it were? > > > > Cheers > > Richard > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Sat Oct 15 16:23:57 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Sat, 15 Oct 2016 08:23:57 -0700 Subject: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter In-Reply-To: <1c7562cf-d61a-adda-66b4-5fd00d75ae75@nasa.gov> References: <1c7562cf-d61a-adda-66b4-5fd00d75ae75@nasa.gov> Message-ID: An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Sat Oct 15 16:27:42 2016 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Sat, 15 Oct 2016 11:27:42 -0400 Subject: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter In-Reply-To: References: <1c7562cf-d61a-adda-66b4-5fd00d75ae75@nasa.gov> Message-ID: <96434600-7c7a-d835-e7b5-9d6a80d9add5@nasa.gov> It absolutely does, thanks Olaf! The tasks running on these nodes are running on 63 other nodes and generating ~60K iop/s of metadata writes and I *think* about the same in reads. Do you think that could be contributing to the higher waiter times? I'm not sure quite what the job is up to. It's seemingly doing very little data movement, the cpu %used is very low but the load is rather high. -Aaron On 10/15/16 11:23 AM, Olaf Weiser wrote: > from your file system configuration .. mmfs -L you'll find the > size of the LOG > since release 4.x ..you can change it, but you need to re-mount the FS > on every client , to make the change effective ... > > when a clients initiate writes/changes to GPFS it needs to update its > changes to the log - if this narrows a certain filling degree, GPFS > triggers so called logWrapThreads to write content to disk and so free > space > > with your given numbers ... double digit [ms] waiter times .. you fs > get's probably slowed down.. and there's something suspect with the > storage, because LOG-IOs are rather small and should not take that long > > to give you an example from a healthy environment... the IO times are so > small, that you usually don't see waiters for this.. > > I/O start time RW Buf type disk:sectorNum nSec time ms tag1 > tag2 Disk UID typ NSD node context thread > --------------- -- ----------- ----------------- ----- ------- > --------- --------- ------------------ --- --------------- --------- > ---------- > 06:23:32.358851 W logData 2:524306424 8 0.439 > 0 0 C0A70D08:57CF40D1 cli 192.167.20.17 LogData > SGExceptionLogBufferFullThread > 06:23:33.576367 W logData 1:524257280 8 0.646 > 0 0 C0A70D08:57CF40D0 cli 192.167.20.16 LogData > SGExceptionLogBufferFullThread > 06:23:32.358851 W logData 2:524306424 8 0.439 > 0 0 C0A70D08:57CF40D1 cli 192.167.20.17 LogData > SGExceptionLogBufferFullThread > 06:23:33.576367 W logData 1:524257280 8 0.646 > 0 0 C0A70D08:57CF40D0 cli 192.167.20.16 LogData > SGExceptionLogBufferFullThread > 06:23:32.212426 W iallocSeg 1:524490048 64 0.733 > 2 245 C0A70D08:57CF40D0 cli 192.167.20.16 Logwrap > LogWrapHelperThread > 06:23:32.212412 W logWrap 2:524552192 8 0.755 > 0 179200 C0A70D08:57CF40D1 cli 192.167.20.17 Logwrap > LogWrapHelperThread > 06:23:32.212432 W logWrap 2:525162760 8 0.737 > 0 125473 C0A70D08:57CF40D1 cli 192.167.20.17 Logwrap > LogWrapHelperThread > 06:23:32.212416 W iallocSeg 2:524488384 64 0.763 > 2 347 C0A70D08:57CF40D1 cli 192.167.20.17 Logwrap > LogWrapHelperThread > 06:23:32.212414 W logWrap 2:525266944 8 2.160 > 0 177664 C0A70D08:57CF40D1 cli 192.167.20.17 Logwrap > LogWrapHelperThread > > > hope this helps .. > > > Mit freundlichen Gr??en / Kind regards > > > Olaf Weiser > > EMEA Storage Competence Center Mainz, German / IBM Systems, Storage > Platform, > ------------------------------------------------------------------------------------------------------------------------------------------- > IBM Deutschland > IBM Allee 1 > 71139 Ehningen > Phone: +49-170-579-44-66 > E-Mail: olaf.weiser at de.ibm.com > ------------------------------------------------------------------------------------------------------------------------------------------- > IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter > Gesch?ftsf?hrung: Martina Koederitz (Vorsitzende), Susanne Peter, > Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Markus Koerner > Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht > Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 > > > > From: Aaron Knister > To: gpfsug main discussion list > Date: 10/15/2016 07:23 AM > Subject: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------------------------------------------------ > > > > I've got a node that's got some curious waiters on it (see below). Could > someone explain what the "SGExceptionLogBufferFullThread" waiter means? > > Thanks! > > -Aaron > > === mmdiag: waiters === > 0x7FFFF040D600 waiting 0.038822715 seconds, > SGExceptionLogBufferFullThread: on ThCond 0x7FFFDBB07628 > (0x7FFFDBB07628) (parallelWaitCond), reason 'wait for parallel write' > for NSD I/O completion on node 10.1.53.5 > 0x7FFFE83F3D60 waiting 0.039629116 seconds, CleanBufferThread: on ThCond > 0x17B1488 (0x17B1488) (MsgRecordCondvar), reason 'RPC wait' for NSD I/O > completion on node 10.1.53.7 > 0x7FFFE8373A90 waiting 0.038921480 seconds, CleanBufferThread: on ThCond > 0x7FFFCD2B4E30 (0x7FFFCD2B4E30) (LogFileBufferDescriptorCondvar), reason > 'force wait on force active buffer write' > 0x42CD9B0 waiting 0.028227004 seconds, CleanBufferThread: on ThCond > 0x7FFFCD2B4E30 (0x7FFFCD2B4E30) (LogFileBufferDescriptorCondvar), reason > 'force wait for buffer write to complete' > 0x7FFFE0F0EAD0 waiting 0.027864343 seconds, CleanBufferThread: on ThCond > 0x7FFFDC0EEA88 (0x7FFFDC0EEA88) (MsgRecordCondvar), reason 'RPC wait' > for NSD I/O completion on node 10.1.53.7 > 0x1575560 waiting 0.028045975 seconds, RemoveHandlerThread: on ThCond > 0x18020CE4E08 (0xFFFFC90020CE4E08) (LkObjCondvar), reason 'waiting for > LX lock' > 0x1570560 waiting 0.038724949 seconds, CreateHandlerThread: on ThCond > 0x18020CE50A0 (0xFFFFC90020CE50A0) (LkObjCondvar), reason 'waiting for > LX lock' > 0x1563D60 waiting 0.073919918 seconds, RemoveHandlerThread: on ThCond > 0x180235F6440 (0xFFFFC900235F6440) (LkObjCondvar), reason 'waiting for > LX lock' > 0x1561560 waiting 0.054854513 seconds, RemoveHandlerThread: on ThCond > 0x1802292D200 (0xFFFFC9002292D200) (LkObjCondvar), reason 'waiting for > LX lock' > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From olaf.weiser at de.ibm.com Sat Oct 15 17:46:19 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Sat, 15 Oct 2016 09:46:19 -0700 Subject: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter In-Reply-To: <96434600-7c7a-d835-e7b5-9d6a80d9add5@nasa.gov> References: <1c7562cf-d61a-adda-66b4-5fd00d75ae75@nasa.gov> <96434600-7c7a-d835-e7b5-9d6a80d9add5@nasa.gov> Message-ID: An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Sat Oct 15 18:07:42 2016 From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]) Date: Sat, 15 Oct 2016 17:07:42 +0000 Subject: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter References: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter Message-ID: <5F910253243E6A47B81A9A2EB424BBA101D9277B@NDMSMBX404.ndc.nasa.gov> Understood. Thank you for your help. By the way, I was able to figure out by poking mmpmon gfis that the job is performing 20k a second each of inode creations, updates and deletions across 64 nodes. There's my 60k iops on the backend. While I'm impressed and not surprised GPFS can keep up with this...that's a pretty hefty workload. From: Olaf Weiser Sent: 10/15/16, 12:47 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter well - hard to say.. 60K IO may or may not be a problem... it depends on your storage backends.. check the response times to the physical disk on the NSD server... concerning the output you provided ... check particularly 10.1.53.5 and 10.1.53.7 .... if they are in the same (bad/ poor) range .. then your storage back end is in trouble or maybe just too heavily utilized ... if the response times to physical disks on the NSD server are ok... .. than maybe the network from client <--> NSD server is somehow in trouble .. From: Aaron Knister To: Date: 10/15/2016 08:28 AM Subject: Re: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ It absolutely does, thanks Olaf! The tasks running on these nodes are running on 63 other nodes and generating ~60K iop/s of metadata writes and I *think* about the same in reads. Do you think that could be contributing to the higher waiter times? I'm not sure quite what the job is up to. It's seemingly doing very little data movement, the cpu %used is very low but the load is rather high. -Aaron On 10/15/16 11:23 AM, Olaf Weiser wrote: > from your file system configuration .. mmfs -L you'll find the > size of the LOG > since release 4.x ..you can change it, but you need to re-mount the FS > on every client , to make the change effective ... > > when a clients initiate writes/changes to GPFS it needs to update its > changes to the log - if this narrows a certain filling degree, GPFS > triggers so called logWrapThreads to write content to disk and so free > space > > with your given numbers ... double digit [ms] waiter times .. you fs > get's probably slowed down.. and there's something suspect with the > storage, because LOG-IOs are rather small and should not take that long > > to give you an example from a healthy environment... the IO times are so > small, that you usually don't see waiters for this.. > > I/O start time RW Buf type disk:sectorNum nSec time ms tag1 > tag2 Disk UID typ NSD node context thread > --------------- -- ----------- ----------------- ----- ------- > --------- --------- ------------------ --- --------------- --------- > ---------- > 06:23:32.358851 W logData 2:524306424 8 0.439 > 0 0 C0A70D08:57CF40D1 cli 192.167.20.17 LogData > SGExceptionLogBufferFullThread > 06:23:33.576367 W logData 1:524257280 8 0.646 > 0 0 C0A70D08:57CF40D0 cli 192.167.20.16 LogData > SGExceptionLogBufferFullThread > 06:23:32.358851 W logData 2:524306424 8 0.439 > 0 0 C0A70D08:57CF40D1 cli 192.167.20.17 LogData > SGExceptionLogBufferFullThread > 06:23:33.576367 W logData 1:524257280 8 0.646 > 0 0 C0A70D08:57CF40D0 cli 192.167.20.16 LogData > SGExceptionLogBufferFullThread > 06:23:32.212426 W iallocSeg 1:524490048 64 0.733 > 2 245 C0A70D08:57CF40D0 cli 192.167.20.16 Logwrap > LogWrapHelperThread > 06:23:32.212412 W logWrap 2:524552192 8 0.755 > 0 179200 C0A70D08:57CF40D1 cli 192.167.20.17 Logwrap > LogWrapHelperThread > 06:23:32.212432 W logWrap 2:525162760 8 0.737 > 0 125473 C0A70D08:57CF40D1 cli 192.167.20.17 Logwrap > LogWrapHelperThread > 06:23:32.212416 W iallocSeg 2:524488384 64 0.763 > 2 347 C0A70D08:57CF40D1 cli 192.167.20.17 Logwrap > LogWrapHelperThread > 06:23:32.212414 W logWrap 2:525266944 8 2.160 > 0 177664 C0A70D08:57CF40D1 cli 192.167.20.17 Logwrap > LogWrapHelperThread > > > hope this helps .. > > > Mit freundlichen Gr??en / Kind regards > > > Olaf Weiser > > EMEA Storage Competence Center Mainz, German / IBM Systems, Storage > Platform, > ------------------------------------------------------------------------------------------------------------------------------------------- > IBM Deutschland > IBM Allee 1 > 71139 Ehningen > Phone: +49-170-579-44-66 > E-Mail: olaf.weiser at de.ibm.com > ------------------------------------------------------------------------------------------------------------------------------------------- > IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter > Gesch?ftsf?hrung: Martina Koederitz (Vorsitzende), Susanne Peter, > Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Markus Koerner > Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht > Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 > > > > From: Aaron Knister > To: gpfsug main discussion list > Date: 10/15/2016 07:23 AM > Subject: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------------------------------------------------ > > > > I've got a node that's got some curious waiters on it (see below). Could > someone explain what the "SGExceptionLogBufferFullThread" waiter means? > > Thanks! > > -Aaron > > === mmdiag: waiters === > 0x7FFFF040D600 waiting 0.038822715 seconds, > SGExceptionLogBufferFullThread: on ThCond 0x7FFFDBB07628 > (0x7FFFDBB07628) (parallelWaitCond), reason 'wait for parallel write' > for NSD I/O completion on node 10.1.53.5 > 0x7FFFE83F3D60 waiting 0.039629116 seconds, CleanBufferThread: on ThCond > 0x17B1488 (0x17B1488) (MsgRecordCondvar), reason 'RPC wait' for NSD I/O > completion on node 10.1.53.7 > 0x7FFFE8373A90 waiting 0.038921480 seconds, CleanBufferThread: on ThCond > 0x7FFFCD2B4E30 (0x7FFFCD2B4E30) (LogFileBufferDescriptorCondvar), reason > 'force wait on force active buffer write' > 0x42CD9B0 waiting 0.028227004 seconds, CleanBufferThread: on ThCond > 0x7FFFCD2B4E30 (0x7FFFCD2B4E30) (LogFileBufferDescriptorCondvar), reason > 'force wait for buffer write to complete' > 0x7FFFE0F0EAD0 waiting 0.027864343 seconds, CleanBufferThread: on ThCond > 0x7FFFDC0EEA88 (0x7FFFDC0EEA88) (MsgRecordCondvar), reason 'RPC wait' > for NSD I/O completion on node 10.1.53.7 > 0x1575560 waiting 0.028045975 seconds, RemoveHandlerThread: on ThCond > 0x18020CE4E08 (0xFFFFC90020CE4E08) (LkObjCondvar), reason 'waiting for > LX lock' > 0x1570560 waiting 0.038724949 seconds, CreateHandlerThread: on ThCond > 0x18020CE50A0 (0xFFFFC90020CE50A0) (LkObjCondvar), reason 'waiting for > LX lock' > 0x1563D60 waiting 0.073919918 seconds, RemoveHandlerThread: on ThCond > 0x180235F6440 (0xFFFFC900235F6440) (LkObjCondvar), reason 'waiting for > LX lock' > 0x1561560 waiting 0.054854513 seconds, RemoveHandlerThread: on ThCond > 0x1802292D200 (0xFFFFC9002292D200) (LkObjCondvar), reason 'waiting for > LX lock' > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Sat Oct 15 18:43:34 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Sat, 15 Oct 2016 10:43:34 -0700 Subject: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter In-Reply-To: <5F910253243E6A47B81A9A2EB424BBA101D9277B@NDMSMBX404.ndc.nasa.gov> References: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter <5F910253243E6A47B81A9A2EB424BBA101D9277B@NDMSMBX404.ndc.nasa.gov> Message-ID: An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Mon Oct 17 01:06:09 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 17 Oct 2016 00:06:09 +0000 Subject: [gpfsug-discuss] CES and NFS Tuning suggestions Message-ID: Looking for some pointers or suggestions on what I should look at changing in Linux and/or GPFS "mmchconfg" settings to help boost NFS performance. Out of the box it seems "poor". Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid 507-269-0413 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ckrafft at de.ibm.com Mon Oct 17 14:59:39 2016 From: ckrafft at de.ibm.com (Christoph Krafft) Date: Mon, 17 Oct 2016 15:59:39 +0200 Subject: [gpfsug-discuss] Looking for experiences with Huawei Oceanstore and GPFS / Spectrum Scale Message-ID: Hi folks, has anyone made experiences with Huawei Oceanstore and GPFS - and would be willing to share some details with me? Any helpful hints are deeply appreciated - THANK you in advance! Mit freundlichen Gr??en / Sincerely Christoph Krafft Client Technical Specialist - Power Systems, IBM Systems Certified IT Specialist @ The Open Group Phone: +49 (0) 7034 643 2171 IBM Deutschland GmbH Mobile: +49 (0) 160 97 81 86 12 Am Weiher 24 Email: ckrafft at de.ibm.com 65451 Kelsterbach Germany IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter Gesch?ftsf?hrung: Martina Koederitz (Vorsitzende), Nicole Reimer, Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Markus Koerner Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecblank.gif Type: image/gif Size: 45 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0B092252.gif Type: image/gif Size: 1851 bytes Desc: not available URL: From Robert.Oesterlin at nuance.com Mon Oct 17 16:00:20 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 17 Oct 2016 15:00:20 +0000 Subject: [gpfsug-discuss] CES: IP address won't assign: "handleNetworkProblem with lock held" Message-ID: <0669A3C9-4F5E-4834-A1C4-D8D124B0A4F3@nuance.com> Can anyone help me pinpoint the issue here? These message repeat and the IP addresses never get assigned. [root at tct-gw01 ~]# tail /var/mmfs/gen/mmfslog Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: Found unassigned address 10.30.22.178 Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: Found unassigned address 10.30.22.179 Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: Found unassigned address 10.30.22.177 Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: Found unassigned address 10.30.22.176 Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: handleNetworkProblem with lock held: assignIP 10.30.22.178_0-_+,10.30.22.179_0-_+,10.30.22.177_0-_+,10.30.22.176_0-_+ 1 Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: Assigning addresses: 10.30.22.178_0-_+,10.30.22.179_0-_+,10.30.22.177_0-_+,10.30.22.176_0-_+ Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: moveCesIPs: 10.30.22.178_0-_+,10.30.22.179_0-_+,10.30.22.177_0-_+,10.30.22.176_0-_+ [root at tct-gw01 ~]# mmces state show -a NODE AUTH AUTH_OBJ NETWORK NFS OBJ SMB CES tct-gw01.infra.us.grid.nuance.com DISABLED DISABLED HEALTHY HEALTHY DISABLED DISABLED HEALTHY tct-gw02.infra.us.grid.nuance.com DISABLED DISABLED HEALTHY HEALTHY DISABLED DISABLED HEALTHY tct-gw03.infra.us.grid.nuance.com DISABLED DISABLED HEALTHY HEALTHY DISABLED DISABLED HEALTHY tct-gw04.infra.us.grid.nuance.com DISABLED DISABLED HEALTHY HEALTHY DISABLED DISABLED HEALTHY [root at tct-gw01 ~]# mmces address list Address Node Group Attribute ------------------------------------------------------------------------- 10.30.22.178 unassigned none none 10.30.22.179 unassigned none none 10.30.22.177 unassigned none none 10.30.22.176 unassigned none none Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid 507-269-0413 -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenneth.waegeman at ugent.be Mon Oct 17 16:16:05 2016 From: kenneth.waegeman at ugent.be (Kenneth Waegeman) Date: Mon, 17 Oct 2016 17:16:05 +0200 Subject: [gpfsug-discuss] Disk can't be recovered due to uncorrectable read error in vdisk (GSS) Message-ID: Hi, Currently our file system is down due to down/unrecovered disks. We try to start the disks again with mmchdisk, but when we do this, we see this error in our mmfs.log: Mon Oct 17 15:28:18.122 2016: [E] smallRead VIO failed due to uncorrectable read error in vdisk nsd11_MetaData_8M_3p_2 vtrack 33630 vsector 34437120 number of vsectors 1024 segments 0 to 16 startBufOffset 0 endB ufOffset 1023 vtrkOffset 0 vioLen 524288 This is a 3-way replicated vdisk, and not one of the recovering disks, but this disk is in 'up' state.. Does someone has a clue what we could to recover our fs? Thanks! Kenneth From bbanister at jumptrading.com Mon Oct 17 17:00:22 2016 From: bbanister at jumptrading.com (Bryan Banister) Date: Mon, 17 Oct 2016 16:00:22 +0000 Subject: [gpfsug-discuss] CES and NFS Tuning suggestions In-Reply-To: References: Message-ID: <21BC488F0AEA2245B2C3E83FC0B33DBB06364770@CHI-EXCHANGEW1.w2k.jumptrading.com> One major issue is the maxFilesToCache and maybe the maxStatCache (though I hear that Linux negates the use of this parameter now? I don?t quite remember). Ganesha apparently likes to hold open a large number of files and this means that it will quickly fill up the maxFilesToCache. When this happens the [gpfsSwapdKproc] process will start to eat up CPU time. This is the daemon that tries to find a file to evict from the cache when a new file is opened. This overhead will also hurt performance. IBM in a PMR we opened suggested setting this to something like 5 Million for the protocol nodes. I think we started with 1.5 Million. You have to be mindful of memory requirements on the token servers to handle the total sum of all maxFilesToCache settings from all nodes that mount the file system. Of course the other, standard NFS tuning parameters for number of threads and NFS client mount options still should be adjusted too. Hope that helps, -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Oesterlin, Robert Sent: Sunday, October 16, 2016 7:06 PM To: gpfsug main discussion list Subject: [gpfsug-discuss] CES and NFS Tuning suggestions Looking for some pointers or suggestions on what I should look at changing in Linux and/or GPFS "mmchconfg" settings to help boost NFS performance. Out of the box it seems "poor". Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid 507-269-0413 ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralphbsz at us.ibm.com Mon Oct 17 17:39:18 2016 From: ralphbsz at us.ibm.com (Ralph A Becker-szendy) Date: Mon, 17 Oct 2016 16:39:18 +0000 Subject: [gpfsug-discuss] Disk can't be recovered due to uncorrectable read error in vdisk (GSS) Message-ID: An HTML attachment was scrubbed... URL: From stijn.deweirdt at ugent.be Mon Oct 17 18:35:37 2016 From: stijn.deweirdt at ugent.be (Stijn De Weirdt) Date: Mon, 17 Oct 2016 19:35:37 +0200 Subject: [gpfsug-discuss] Disk can't be recovered due to uncorrectable read error in vdisk (GSS) In-Reply-To: References: Message-ID: <8f93f6bf-6b42-c3bf-11b4-39dc53b4451f@ugent.be> hi ralph, >>Currently our file system is down due to down/unrecovered disks. We >> try to start the disks again with mmchdisk, but when we do this, we >> see this error in our mmfs.log: >> ... >> This is a 3-way replicated vdisk, and not one of the recovering disks,but >> this disk is in 'up' state.. > First, please open a PMR through your normal support organization, and make it > clear in the PMR that the problem is GPFS and GNR (a.k.a. ESS). Like that, it > will be assigned to the correct support group. Support will request that you > upload a snap. PMR is created, but we are in particular puzzled by the IO read error. and getting some details about what is gone here (with details as you provided) is somethign we usually do not get from support ;) > There seems to be a combination of two problems here: > One, a NSD (which is also a GNR vdisk) is down, which is usually caused by an IO > error on the vdisk, or by both servers for the recovery group that contains the > vdisk being down simultaneously. Usually, that is easily fixed by running > mmchdisk with a start option, but you tried that and it didn't work. This > problem is at the NSD layer (meaning in the GPFS client that accesses the GNR > vdisk), not in the GNR layer. it's actually more than disk down, all on same recoverygroup. the vdisk with the read error is on the other recoverygroup (only 2, this is/was a GSS24) > Second, another vdisk has an internal error, caused by read error from the > physical disks (which is what "uncorrectable read error" means). what does physical mean here? we have no IO error anywhere (OS/kernel/scsi, also the error counters on the mmlspdisk output do not increase). Now, give that > you say that this vdisk is 3-way replicated, that probably means that there are > multiple problems. This error is purely in the GNR layer, and the error message > you quote "smallRead VIO..." comes from the GNR layer. Now, an error from one > vdisk can't prevent mmchdisk on a different vdisk from working, so these two > problems seem unrelated. well, they can: the disks with all metadata replica on the one recoverygroup are down. starting those forces the read of the ones on the other group, and this runs into the IOread error, and everything stops (well, that's how we understand it ;) > Furthermore, I'm going to bet that the two problems (which at first seem > unrelated) must in reality have a common root cause; it would be too weird a > coincidence to get two problems that are unrelated at the same time. To debug > this requires looking at way more information than a single line from the > mmfs.log file, which is why the support organization needs a complete PMR > opened, and then have the usual snap (with logs, dumps, ...) uploaded, so it can > see what the cause of the problem is. yep, trace and snap uploaded. > Good luck! thanks (and thanks again for some insights, much appreciated !) > Ralph Becker-Szendy > IBM Almaden Research Center - Computer Science -Storage Systems > ralphbsz at us.ibm.com > 408-927-2752 > 650 Harry Road, K56-B3, San Jose, CA 95120 > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From olaf.weiser at de.ibm.com Mon Oct 17 18:40:13 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Mon, 17 Oct 2016 10:40:13 -0700 Subject: [gpfsug-discuss] CES and NFS Tuning suggestions In-Reply-To: <21BC488F0AEA2245B2C3E83FC0B33DBB06364770@CHI-EXCHANGEW1.w2k.jumptrading.com> References: <21BC488F0AEA2245B2C3E83FC0B33DBB06364770@CHI-EXCHANGEW1.w2k.jumptrading.com> Message-ID: An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Mon Oct 17 19:27:01 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Mon, 17 Oct 2016 11:27:01 -0700 Subject: [gpfsug-discuss] CES: IP address won't assign: "handleNetworkProblem with lock held" In-Reply-To: <0669A3C9-4F5E-4834-A1C4-D8D124B0A4F3@nuance.com> References: <0669A3C9-4F5E-4834-A1C4-D8D124B0A4F3@nuance.com> Message-ID: An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Mon Oct 17 19:42:58 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 17 Oct 2016 18:42:58 +0000 Subject: [gpfsug-discuss] CES: IP address won't assign: "handleNetworkProblem with lock held" Message-ID: <99C9D37E-3387-48A6-88BB-DFE711C7CC8F@nuance.com> Yes - so interesting - it looks like the nodes have the addresses assigned but CES doesn?t know that. [root at tct-gw01 ~]# mmlscluster --ces GPFS cluster information ======================== GPFS cluster name: nrg1-tct.nrg1.us.grid.nuance.com GPFS cluster id: 17869514639699411874 Cluster Export Services global parameters ----------------------------------------- Shared root directory: /gpfs/fs1/ces Enabled Services: NFS Log level: 0 Address distribution policy: even-coverage Node Daemon node name IP address CES IP address list ----------------------------------------------------------------------- 4 tct-gw01.infra.us.grid.nuance.com 10.30.22.160 None 5 tct-gw02.infra.us.grid.nuance.com 10.30.22.161 None 6 tct-gw03.infra.us.grid.nuance.com 10.30.22.162 None 7 tct-gw04.infra.us.grid.nuance.com 10.30.22.163 None [root at tct-gw01 ~]# mmdsh -N cesnodes "ip a | grep 10.30.22." | sort -k 1 tct-gw01.infra.us.grid.nuance.com: inet 10.30.22.160/24 brd 10.30.22.255 scope global bond0 tct-gw01.infra.us.grid.nuance.com: inet 10.30.22.176/24 brd 10.30.22.255 scope global secondary bond0:0 tct-gw02.infra.us.grid.nuance.com: inet 10.30.22.161/24 brd 10.30.22.255 scope global bond0 tct-gw02.infra.us.grid.nuance.com: inet 10.30.22.177/24 brd 10.30.22.255 scope global secondary bond0:0 tct-gw02.infra.us.grid.nuance.com: inet 10.30.22.178/24 brd 10.30.22.255 scope global secondary bond0:1 tct-gw03.infra.us.grid.nuance.com: inet 10.30.22.162/24 brd 10.30.22.255 scope global bond0 tct-gw03.infra.us.grid.nuance.com: inet 10.30.22.178/24 brd 10.30.22.255 scope global secondary bond0:0 tct-gw04.infra.us.grid.nuance.com: inet 10.30.22.163/24 brd 10.30.22.255 scope global bond0 tct-gw04.infra.us.grid.nuance.com: inet 10.30.22.179/24 brd 10.30.22.255 scope global secondary bond0:0 Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: on behalf of Olaf Weiser Reply-To: gpfsug main discussion list Date: Monday, October 17, 2016 at 1:27 PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] CES: IP address won't assign: "handleNetworkProblem with lock held" ip a | grep 10.30.22. -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Mon Oct 17 19:53:01 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Mon, 17 Oct 2016 18:53:01 +0000 Subject: [gpfsug-discuss] CES: IP address won't assign: "handleNetworkProblem with lock held" In-Reply-To: <99C9D37E-3387-48A6-88BB-DFE711C7CC8F@nuance.com> References: <99C9D37E-3387-48A6-88BB-DFE711C7CC8F@nuance.com> Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 37817 bytes Desc: not available URL: From S.J.Thompson at bham.ac.uk Mon Oct 17 19:57:15 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Mon, 17 Oct 2016 18:57:15 +0000 Subject: [gpfsug-discuss] CES: IP address won't assign: "handleNetworkProblem with lock held" In-Reply-To: References: <0669A3C9-4F5E-4834-A1C4-D8D124B0A4F3@nuance.com>, Message-ID: Does it strictly follow this? We were doing some testing with tap interfaces into vxlan networks and found that if we simulated taking down the vxlan interface (which appears in ifconfig as a physical int really), then it moved the ces ip onto the box's primary Nic which was a different subnet. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Olaf Weiser [olaf.weiser at de.ibm.com] Sent: 17 October 2016 19:27 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] CES: IP address won't assign: "handleNetworkProblem with lock held" simple question -sorry for that - your Nodes.. do they have an IP address in the same subnet as your IP address listed here ? and if, is this network up n running so that GPFS can find/detect it ? what tells mmlscluster --ces ? from each node - assuming class C /24 network , do a ip a | grep 10.30.22. cheers From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 10/17/2016 08:00 AM Subject: [gpfsug-discuss] CES: IP address won't assign: "handleNetworkProblem with lock held" Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Can anyone help me pinpoint the issue here? These message repeat and the IP addresses never get assigned. [root at tct-gw01 ~]# tail /var/mmfs/gen/mmfslog Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: Found unassigned address 10.30.22.178 Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: Found unassigned address 10.30.22.179 Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: Found unassigned address 10.30.22.177 Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: Found unassigned address 10.30.22.176 Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: handleNetworkProblem with lock held: assignIP 10.30.22.178_0-_+,10.30.22.179_0-_+,10.30.22.177_0-_+,10.30.22.176_0-_+ 1 Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: Assigning addresses: 10.30.22.178_0-_+,10.30.22.179_0-_+,10.30.22.177_0-_+,10.30.22.176_0-_+ Mon Oct 17 10:57:55 EDT 2016: mmcesnetworkmonitor: moveCesIPs: 10.30.22.178_0-_+,10.30.22.179_0-_+,10.30.22.177_0-_+,10.30.22.176_0-_+ [root at tct-gw01 ~]# mmces state show -a NODE AUTH AUTH_OBJ NETWORK NFS OBJ SMB CES tct-gw01.infra.us.grid.nuance.com DISABLED DISABLED HEALTHY HEALTHY DISABLED DISABLED HEALTHY tct-gw02.infra.us.grid.nuance.com DISABLED DISABLED HEALTHY HEALTHY DISABLED DISABLED HEALTHY tct-gw03.infra.us.grid.nuance.com DISABLED DISABLED HEALTHY HEALTHY DISABLED DISABLED HEALTHY tct-gw04.infra.us.grid.nuance.com DISABLED DISABLED HEALTHY HEALTHY DISABLED DISABLED HEALTHY [root at tct-gw01 ~]# mmces address list Address Node Group Attribute ------------------------------------------------------------------------- 10.30.22.178 unassigned none none 10.30.22.179 unassigned none none 10.30.22.177 unassigned none none 10.30.22.176 unassigned none none Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid 507-269-0413 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Robert.Oesterlin at nuance.com Mon Oct 17 20:01:48 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 17 Oct 2016 19:01:48 +0000 Subject: [gpfsug-discuss] [EXTERNAL] Re: CES: IP address won't assign: "handleNetworkProblem with lock held" In-Reply-To: References: <99C9D37E-3387-48A6-88BB-DFE711C7CC8F@nuance.com> Message-ID: <0E4224B9-F266-4BB2-9B6A-F580976C2207@nuance.com> No - the :0 and :1 address are floating addresses *assigned by CES* - it created those interfaces. The issue seems to be that these are assigned and CES doesn't know it. Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: on behalf of Olaf Weiser Reply-To: gpfsug main discussion list Date: Monday, October 17, 2016 at 1:53 PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] CES: IP address won't assign: "handleNetworkProblem with lock held" ah .. I see.. seems, that you already has IP aliases around .. GPFS don't like it... eg. your node tct-gw01.infra.us.grid.nuance.com: inet 10.30.22.160/24 has already an alias - 10.30.22.176 ... if I understand you answers correctly... from the doc'... [...] you need to provide a static IP adress, that is not already[...] as an alias [...] -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Mon Oct 17 21:13:28 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Mon, 17 Oct 2016 13:13:28 -0700 Subject: [gpfsug-discuss] CES: IP address won't assign: "handleNetworkProblem with lock held" In-Reply-To: References: <0669A3C9-4F5E-4834-A1C4-D8D124B0A4F3@nuance.com>, Message-ID: An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Mon Oct 17 21:13:28 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Mon, 17 Oct 2016 13:13:28 -0700 Subject: [gpfsug-discuss] [EXTERNAL] Re: CES: IP address won't assign: "handleNetworkProblem with lock held" In-Reply-To: <0E4224B9-F266-4BB2-9B6A-F580976C2207@nuance.com> References: <99C9D37E-3387-48A6-88BB-DFE711C7CC8F@nuance.com> <0E4224B9-F266-4BB2-9B6A-F580976C2207@nuance.com> Message-ID: An HTML attachment was scrubbed... URL: From jake.carroll at uq.edu.au Tue Oct 18 05:21:12 2016 From: jake.carroll at uq.edu.au (Jake Carroll) Date: Tue, 18 Oct 2016 04:21:12 +0000 Subject: [gpfsug-discuss] Any spaces left for a user-presentation at the UG, SC16? Message-ID: <1EECF1D7-DD5E-47BD-A99F-57A4E0294442@uq.edu.au> Hi, I have something interesting that I think the user group might find novel, or at least, hopefully interesting, at the UG meetup at SC16. It would entail an unusual use-case for AFM and some of the unusual things we are doing with it. All good if no spots left ? but I?d be happy to present something if the group thinks it would be beneficial. Thanks! -jc From Robert.Oesterlin at nuance.com Tue Oct 18 12:37:17 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Tue, 18 Oct 2016 11:37:17 +0000 Subject: [gpfsug-discuss] Any spaces left for a user-presentation at the UG, SC16? Message-ID: <7A87F87D-8345-4254-B39A-DE38B61D63AB@nuance.com> Hi Jake At this point, we're pretty well filled up. And yes, you should be seeing a final agenda "soon" - still shuffling things around a bit, trying to get as much great content as we can into one afternoon Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid GPFS UG Co-Principal From: on behalf of Jake Carroll Reply-To: gpfsug main discussion list Date: Monday, October 17, 2016 at 11:21 PM To: "gpfsug-discuss at spectrumscale.org" Subject: [EXTERNAL] [gpfsug-discuss] Any spaces left for a user-presentation at the UG, SC16? Hi, I have something interesting that I think the user group might find novel, or at least, hopefully interesting, at the UG meetup at SC16. It would entail an unusual use-case for AFM and some of the unusual things we are doing with it. All good if no spots left ? but I?d be happy to present something if the group thinks it would be beneficial. Thanks! -jc _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwIGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=SI4sTXt3Q32Lv6WKkeOdNHvhsvEAdzOYPmJMU0cqe38&s=UiXMs0ix00_xIWkODVa9o_km3OQ5mSVKXkE_w6lC8ls&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From kallbac at iu.edu Tue Oct 18 14:26:20 2016 From: kallbac at iu.edu (Kallback-Rose, Kristy A) Date: Tue, 18 Oct 2016 13:26:20 +0000 Subject: [gpfsug-discuss] Any spaces left for a user-presentation at the UG, SC16? In-Reply-To: <7A87F87D-8345-4254-B39A-DE38B61D63AB@nuance.com> References: <7A87F87D-8345-4254-B39A-DE38B61D63AB@nuance.com> Message-ID: JC, can you email a little more on the AFM topic to me and Bob? For future reference if nothing else. Thanks, Kristy On Oct 18, 2016, at 7:37 AM, Oesterlin, Robert > wrote: Hi Jake At this point, we're pretty well filled up. And yes, you should be seeing a final agenda "soon" - still shuffling things around a bit, trying to get as much great content as we can into one afternoon Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid GPFS UG Co-Principal From: > on behalf of Jake Carroll > Reply-To: gpfsug main discussion list > Date: Monday, October 17, 2016 at 11:21 PM To: "gpfsug-discuss at spectrumscale.org" > Subject: [EXTERNAL] [gpfsug-discuss] Any spaces left for a user-presentation at the UG, SC16? Hi, I have something interesting that I think the user group might find novel, or at least, hopefully interesting, at the UG meetup at SC16. It would entail an unusual use-case for AFM and some of the unusual things we are doing with it. All good if no spots left ? but I?d be happy to present something if the group thinks it would be beneficial. Thanks! -jc _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwIGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=SI4sTXt3Q32Lv6WKkeOdNHvhsvEAdzOYPmJMU0cqe38&s=UiXMs0ix00_xIWkODVa9o_km3OQ5mSVKXkE_w6lC8ls&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.childs at qmul.ac.uk Wed Oct 19 15:12:41 2016 From: p.childs at qmul.ac.uk (Peter Childs) Date: Wed, 19 Oct 2016 14:12:41 +0000 Subject: [gpfsug-discuss] Using AFM to migrate files. In-Reply-To: References: <7j173edatp91hqocoto2bau8.1476818437765@email.android.com>, Message-ID: We are planning to use AFM to migrate our old GPFS file store to a new GPFS file store. This will give us the advantages of Spectrum Scale (GPFS) 4.2, such as larger block and inode size. I would like to attempt to gain some insight on my plans before I start. The old file store was running GPFS 3.5 with 512 byte inodes and 1MB block size. We have now upgraded it to 4.1 and are working towards 4.2 with 300TB of files. (385TB max space) this is so we can use both the old and new storage via multi-cluster. We are moving to a new GPFS cluster so we can use the new protocol nodes eventually and also put the new storage machines as cluster managers, as this should be faster and future proof The new hardware has 1PB of space running GPFS 4.2 We have multiple filesets, and would like to maintain our namespace as far as possible. My plan was to. 1. Create a read-only (RO) AFM cache on the new storage (ro) 2a. Move old fileset and replace with SymLink to new. 2b. Convert RO AFM to Local Update (LU) AFM pointing to new parking area of old files. 2c. move user access to new location in cache. 3. Flush everything into cache and disconnect. I've read the docs including the ones on migration but it's not clear if it's safe to move the home of a cache and update the target. It looks like it should be possible and my tests say it works. An alternative plan is to use a Independent Writer (IW) AFM Cache to move the home directories which are pointed to by LDAP. Hence we can move users one at a time and only have to drain the HPC cluster at the end to disconnect the cache. I assume that migrating users over an Independent Writer is safe so long as the users don't use both sides of the cache at once (ie home and target) I'm also interested in any recipe people have on GPFS policies to preseed and flush the cache. We plan to do all the migration using AFM over GPFS we're not currently using NFS and have no plans to start. I believe using GPFS is the faster method to preform the migration. Any suggestions and experience of doing similar migration jobs would be helpful. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London From mimarsh2 at vt.edu Wed Oct 19 18:46:02 2016 From: mimarsh2 at vt.edu (Brian Marshall) Date: Wed, 19 Oct 2016 13:46:02 -0400 Subject: [gpfsug-discuss] subnets Message-ID: All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Wed Oct 19 19:10:38 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Wed, 19 Oct 2016 18:10:38 +0000 Subject: [gpfsug-discuss] subnets In-Reply-To: References: Message-ID: mmdiag --network Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Brian Marshall [mimarsh2 at vt.edu] Sent: 19 October 2016 18:46 To: gpfsug main discussion list Subject: [gpfsug-discuss] subnets All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall From UWEFALKE at de.ibm.com Wed Oct 19 19:15:52 2016 From: UWEFALKE at de.ibm.com (Uwe Falke) Date: Wed, 19 Oct 2016 20:15:52 +0200 Subject: [gpfsug-discuss] subnets In-Reply-To: References: Message-ID: Hi Brian, you might use mmfsadm saferdump tscomm to check on which route peer cluster members are reached. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Frank Hammer, Thorsten Moehring Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: Brian Marshall To: gpfsug main discussion list Date: 10/19/2016 07:46 PM Subject: [gpfsug-discuss] subnets Sent by: gpfsug-discuss-bounces at spectrumscale.org All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Kevin.Buterbaugh at Vanderbilt.Edu Wed Oct 19 21:11:57 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Wed, 19 Oct 2016 20:11:57 +0000 Subject: [gpfsug-discuss] Will there be any more GPFS 4.2.0-x releases? Message-ID: <142FECE0-E157-42D9-BC10-4C48E78FA065@vanderbilt.edu> Hi All, We?re currently running GPFS 4.2.0-4 with an efix installed and now we need a 2nd efix. I?m not a big fan of adding efix to efix and would prefer to go to a new PTF that contains both efixes. So ? is there going to be a GPFS 4.2.0-5 (it?s been a longer than normal interval since PTF 4 came out) or do we need to go to GPFS 4.2.1-x? If the latter, any major changes to watch out for? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bpappas at dstonline.com Wed Oct 19 21:44:39 2016 From: bpappas at dstonline.com (Bill Pappas) Date: Wed, 19 Oct 2016 20:44:39 +0000 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) In-Reply-To: References: Message-ID: I have some ideas to suggest given some of my experiences. First, I have some questions: How many files are you migrating? Will you be creating multiple file sets on the target system based off of business or project needs? Like, file set a is for "department a" and file set b is for "large scale project a" Thanks. Bill Pappas 901-619-0585 bpappas at dstonline.com [1466780990050_DSTlogo.png] [http://www.prweb.com/releases/2016/06/prweb13504050.htm] http://www.prweb.com/releases/2016/06/prweb13504050.htm ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org Sent: Wednesday, October 19, 2016 3:12 PM To: gpfsug-discuss at spectrumscale.org Subject: gpfsug-discuss Digest, Vol 57, Issue 49 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Using AFM to migrate files. (Peter Childs) 2. subnets (Brian Marshall) 3. Re: subnets (Simon Thompson (Research Computing - IT Services)) 4. Re: subnets (Uwe Falke) 5. Will there be any more GPFS 4.2.0-x releases? (Buterbaugh, Kevin L) ---------------------------------------------------------------------- Message: 1 Date: Wed, 19 Oct 2016 14:12:41 +0000 From: Peter Childs To: gpfsug main discussion list Subject: [gpfsug-discuss] Using AFM to migrate files. Message-ID: Content-Type: text/plain; charset="iso-8859-1" We are planning to use AFM to migrate our old GPFS file store to a new GPFS file store. This will give us the advantages of Spectrum Scale (GPFS) 4.2, such as larger block and inode size. I would like to attempt to gain some insight on my plans before I start. The old file store was running GPFS 3.5 with 512 byte inodes and 1MB block size. We have now upgraded it to 4.1 and are working towards 4.2 with 300TB of files. (385TB max space) this is so we can use both the old and new storage via multi-cluster. We are moving to a new GPFS cluster so we can use the new protocol nodes eventually and also put the new storage machines as cluster managers, as this should be faster and future proof The new hardware has 1PB of space running GPFS 4.2 We have multiple filesets, and would like to maintain our namespace as far as possible. My plan was to. 1. Create a read-only (RO) AFM cache on the new storage (ro) 2a. Move old fileset and replace with SymLink to new. 2b. Convert RO AFM to Local Update (LU) AFM pointing to new parking area of old files. 2c. move user access to new location in cache. 3. Flush everything into cache and disconnect. I've read the docs including the ones on migration but it's not clear if it's safe to move the home of a cache and update the target. It looks like it should be possible and my tests say it works. An alternative plan is to use a Independent Writer (IW) AFM Cache to move the home directories which are pointed to by LDAP. Hence we can move users one at a time and only have to drain the HPC cluster at the end to disconnect the cache. I assume that migrating users over an Independent Writer is safe so long as the users don't use both sides of the cache at once (ie home and target) I'm also interested in any recipe people have on GPFS policies to preseed and flush the cache. We plan to do all the migration using AFM over GPFS we're not currently using NFS and have no plans to start. I believe using GPFS is the faster method to preform the migration. Any suggestions and experience of doing similar migration jobs would be helpful. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ------------------------------ Message: 2 Date: Wed, 19 Oct 2016 13:46:02 -0400 From: Brian Marshall To: gpfsug main discussion list Subject: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="utf-8" All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 3 Date: Wed, 19 Oct 2016 18:10:38 +0000 From: "Simon Thompson (Research Computing - IT Services)" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="us-ascii" mmdiag --network Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Brian Marshall [mimarsh2 at vt.edu] Sent: 19 October 2016 18:46 To: gpfsug main discussion list Subject: [gpfsug-discuss] subnets All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall ------------------------------ Message: 4 Date: Wed, 19 Oct 2016 20:15:52 +0200 From: "Uwe Falke" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="ISO-8859-1" Hi Brian, you might use mmfsadm saferdump tscomm to check on which route peer cluster members are reached. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Frank Hammer, Thorsten Moehring Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: Brian Marshall To: gpfsug main discussion list Date: 10/19/2016 07:46 PM Subject: [gpfsug-discuss] subnets Sent by: gpfsug-discuss-bounces at spectrumscale.org All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ------------------------------ Message: 5 Date: Wed, 19 Oct 2016 20:11:57 +0000 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Subject: [gpfsug-discuss] Will there be any more GPFS 4.2.0-x releases? Message-ID: <142FECE0-E157-42D9-BC10-4C48E78FA065 at vanderbilt.edu> Content-Type: text/plain; charset="utf-8" Hi All, We?re currently running GPFS 4.2.0-4 with an efix installed and now we need a 2nd efix. I?m not a big fan of adding efix to efix and would prefer to go to a new PTF that contains both efixes. So ? is there going to be a GPFS 4.2.0-5 (it?s been a longer than normal interval since PTF 4 came out) or do we need to go to GPFS 4.2.1-x? If the latter, any major changes to watch out for? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 49 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-1466780990050_DSTlogo.png.png Type: image/png Size: 6282 bytes Desc: OutlookEmoji-1466780990050_DSTlogo.png.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg Type: image/jpeg Size: 14887 bytes Desc: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg URL: From knop at us.ibm.com Wed Oct 19 22:35:43 2016 From: knop at us.ibm.com (Felipe Knop) Date: Wed, 19 Oct 2016 17:35:43 -0400 Subject: [gpfsug-discuss] Will there be any more GPFS 4.2.0-x releases? In-Reply-To: <142FECE0-E157-42D9-BC10-4C48E78FA065@vanderbilt.edu> References: <142FECE0-E157-42D9-BC10-4C48E78FA065@vanderbilt.edu> Message-ID: Kevin, There will no longer be further PTFs on the 4.2.0 stream. Fixes are now provided on 4.2.1. See https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.1/com.ibm.spectrum.scale.v4r21.doc/bl1xx_soc.htm for the functional enhancements for 4.2.1. See https://www.ibm.com/developerworks/community/forums/html/topic?id=14de7136-e7da-4f93-9f50-5981af1b3f54&ps=50 for announcements on 4.2.1, including the changelog for each PTF. Felipe ---- Felipe Knop knop at us.ibm.com GPFS Development and Security IBM Systems IBM Building 008 2455 South Rd, Poughkeepsie, NY 12601 (845) 433-9314 T/L 293-9314 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 10/19/2016 04:12 PM Subject: [gpfsug-discuss] Will there be any more GPFS 4.2.0-x releases? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, We?re currently running GPFS 4.2.0-4 with an efix installed and now we need a 2nd efix. I?m not a big fan of adding efix to efix and would prefer to go to a new PTF that contains both efixes. So ? is there going to be a GPFS 4.2.0-5 (it?s been a longer than normal interval since PTF 4 came out) or do we need to go to GPFS 4.2.1-x? If the latter, any major changes to watch out for? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From douglasof at us.ibm.com Thu Oct 20 01:07:17 2016 From: douglasof at us.ibm.com (Douglas O'flaherty) Date: Wed, 19 Oct 2016 20:07:17 -0400 Subject: [gpfsug-discuss] SC16 & U/G Registration Message-ID: All: Posting... or re-posting, the URLs for registration to the IBM Spectrum Scale User Group and other IBM briefings IBM summary page: http://www-03.ibm.com/systems/technicalcomputing/supercomputing/index.html Link to IBM event and registrations https://www-01.ibm.com/events/wwe/grp/grp305.nsf/Agenda.xsp?openform&seminar=357M7UES&locale=en_US&S_TACT=000001CP&S_OFF_CD=10001235 We are ramping up for a fantastic event at SC16. Keep an ear out for our next announcements in November, which we will cover in both the user group and more deeply in the IBM seminars and briefings. For those not attending SC16, we will have a briefing webinar on the latest advances in IBM Spectrum Scale and ESS on 11/14. Registration link to follow. doug IBM Spectrum Storage PMM -------------- next part -------------- An HTML attachment was scrubbed... URL: From YARD at il.ibm.com Thu Oct 20 07:15:54 2016 From: YARD at il.ibm.com (Yaron Daniel) Date: Thu, 20 Oct 2016 09:15:54 +0300 Subject: [gpfsug-discuss] Using AFM to migrate files. In-Reply-To: References: <7j173edatp91hqocoto2bau8.1476818437765@email.android.com>, Message-ID: Hi Does you use NFSv4 acls in your old cluster ? Regards Yaron Daniel 94 Em Ha'Moshavot Rd Server, Storage and Data Services - Team Leader Petach Tiqva, 49527 Global Technology Services Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com IBM Israel From: Peter Childs To: gpfsug main discussion list Date: 10/19/2016 05:34 PM Subject: [gpfsug-discuss] Using AFM to migrate files. Sent by: gpfsug-discuss-bounces at spectrumscale.org We are planning to use AFM to migrate our old GPFS file store to a new GPFS file store. This will give us the advantages of Spectrum Scale (GPFS) 4.2, such as larger block and inode size. I would like to attempt to gain some insight on my plans before I start. The old file store was running GPFS 3.5 with 512 byte inodes and 1MB block size. We have now upgraded it to 4.1 and are working towards 4.2 with 300TB of files. (385TB max space) this is so we can use both the old and new storage via multi-cluster. We are moving to a new GPFS cluster so we can use the new protocol nodes eventually and also put the new storage machines as cluster managers, as this should be faster and future proof The new hardware has 1PB of space running GPFS 4.2 We have multiple filesets, and would like to maintain our namespace as far as possible. My plan was to. 1. Create a read-only (RO) AFM cache on the new storage (ro) 2a. Move old fileset and replace with SymLink to new. 2b. Convert RO AFM to Local Update (LU) AFM pointing to new parking area of old files. 2c. move user access to new location in cache. 3. Flush everything into cache and disconnect. I've read the docs including the ones on migration but it's not clear if it's safe to move the home of a cache and update the target. It looks like it should be possible and my tests say it works. An alternative plan is to use a Independent Writer (IW) AFM Cache to move the home directories which are pointed to by LDAP. Hence we can move users one at a time and only have to drain the HPC cluster at the end to disconnect the cache. I assume that migrating users over an Independent Writer is safe so long as the users don't use both sides of the cache at once (ie home and target) I'm also interested in any recipe people have on GPFS policies to preseed and flush the cache. We plan to do all the migration using AFM over GPFS we're not currently using NFS and have no plans to start. I believe using GPFS is the faster method to preform the migration. Any suggestions and experience of doing similar migration jobs would be helpful. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1851 bytes Desc: not available URL: From p.childs at qmul.ac.uk Thu Oct 20 11:12:36 2016 From: p.childs at qmul.ac.uk (Peter Childs) Date: Thu, 20 Oct 2016 10:12:36 +0000 Subject: [gpfsug-discuss] Using AFM to migrate files. In-Reply-To: References: <7j173edatp91hqocoto2bau8.1476818437765@email.android.com>, , Message-ID: Yes but not a great deal, Peter Childs Research Storage Expert ITS Research Infrastructure Queen Mary, University of London ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Yaron Daniel Sent: Thursday, October 20, 2016 7:15:54 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Using AFM to migrate files. Hi Does you use NFSv4 acls in your old cluster ? Regards ________________________________ Yaron Daniel 94 Em Ha'Moshavot Rd [cid:_1_09E5055809E4FFC4002269E8C2258052] Server, Storage and Data Services- Team Leader Petach Tiqva, 49527 Global Technology Services Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com IBM Israel From: Peter Childs To: gpfsug main discussion list Date: 10/19/2016 05:34 PM Subject: [gpfsug-discuss] Using AFM to migrate files. Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ We are planning to use AFM to migrate our old GPFS file store to a new GPFS file store. This will give us the advantages of Spectrum Scale (GPFS) 4.2, such as larger block and inode size. I would like to attempt to gain some insight on my plans before I start. The old file store was running GPFS 3.5 with 512 byte inodes and 1MB block size. We have now upgraded it to 4.1 and are working towards 4.2 with 300TB of files. (385TB max space) this is so we can use both the old and new storage via multi-cluster. We are moving to a new GPFS cluster so we can use the new protocol nodes eventually and also put the new storage machines as cluster managers, as this should be faster and future proof The new hardware has 1PB of space running GPFS 4.2 We have multiple filesets, and would like to maintain our namespace as far as possible. My plan was to. 1. Create a read-only (RO) AFM cache on the new storage (ro) 2a. Move old fileset and replace with SymLink to new. 2b. Convert RO AFM to Local Update (LU) AFM pointing to new parking area of old files. 2c. move user access to new location in cache. 3. Flush everything into cache and disconnect. I've read the docs including the ones on migration but it's not clear if it's safe to move the home of a cache and update the target. It looks like it should be possible and my tests say it works. An alternative plan is to use a Independent Writer (IW) AFM Cache to move the home directories which are pointed to by LDAP. Hence we can move users one at a time and only have to drain the HPC cluster at the end to disconnect the cache. I assume that migrating users over an Independent Writer is safe so long as the users don't use both sides of the cache at once (ie home and target) I'm also interested in any recipe people have on GPFS policies to preseed and flush the cache. We plan to do all the migration using AFM over GPFS we're not currently using NFS and have no plans to start. I believe using GPFS is the faster method to preform the migration. Any suggestions and experience of doing similar migration jobs would be helpful. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00001.gif Type: image/gif Size: 1851 bytes Desc: ATT00001.gif URL: From p.childs at qmul.ac.uk Thu Oct 20 20:07:44 2016 From: p.childs at qmul.ac.uk (Peter Childs) Date: Thu, 20 Oct 2016 19:07:44 +0000 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) In-Reply-To: References: , Message-ID: <5qv6d7inj2j1pa94kqamk2uf.1476989646711@email.android.com> Yes, most of the filesets are based on research groups, projects or departments, with the exception of scratch and home, hence the idea to use a different method for these filesets. There are approximately 230 million files, the largest of the filesets has 52TB and 63 million files. 300TB in total. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ---- Bill Pappas wrote ---- I have some ideas to suggest given some of my experiences. First, I have some questions: How many files are you migrating? Will you be creating multiple file sets on the target system based off of business or project needs? Like, file set a is for "department a" and file set b is for "large scale project a" Thanks. Bill Pappas 901-619-0585 bpappas at dstonline.com [1466780990050_DSTlogo.png] [http://www.prweb.com/releases/2016/06/prweb13504050.htm] http://www.prweb.com/releases/2016/06/prweb13504050.htm ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org Sent: Wednesday, October 19, 2016 3:12 PM To: gpfsug-discuss at spectrumscale.org Subject: gpfsug-discuss Digest, Vol 57, Issue 49 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Using AFM to migrate files. (Peter Childs) 2. subnets (Brian Marshall) 3. Re: subnets (Simon Thompson (Research Computing - IT Services)) 4. Re: subnets (Uwe Falke) 5. Will there be any more GPFS 4.2.0-x releases? (Buterbaugh, Kevin L) ---------------------------------------------------------------------- Message: 1 Date: Wed, 19 Oct 2016 14:12:41 +0000 From: Peter Childs To: gpfsug main discussion list Subject: [gpfsug-discuss] Using AFM to migrate files. Message-ID: Content-Type: text/plain; charset="iso-8859-1" We are planning to use AFM to migrate our old GPFS file store to a new GPFS file store. This will give us the advantages of Spectrum Scale (GPFS) 4.2, such as larger block and inode size. I would like to attempt to gain some insight on my plans before I start. The old file store was running GPFS 3.5 with 512 byte inodes and 1MB block size. We have now upgraded it to 4.1 and are working towards 4.2 with 300TB of files. (385TB max space) this is so we can use both the old and new storage via multi-cluster. We are moving to a new GPFS cluster so we can use the new protocol nodes eventually and also put the new storage machines as cluster managers, as this should be faster and future proof The new hardware has 1PB of space running GPFS 4.2 We have multiple filesets, and would like to maintain our namespace as far as possible. My plan was to. 1. Create a read-only (RO) AFM cache on the new storage (ro) 2a. Move old fileset and replace with SymLink to new. 2b. Convert RO AFM to Local Update (LU) AFM pointing to new parking area of old files. 2c. move user access to new location in cache. 3. Flush everything into cache and disconnect. I've read the docs including the ones on migration but it's not clear if it's safe to move the home of a cache and update the target. It looks like it should be possible and my tests say it works. An alternative plan is to use a Independent Writer (IW) AFM Cache to move the home directories which are pointed to by LDAP. Hence we can move users one at a time and only have to drain the HPC cluster at the end to disconnect the cache. I assume that migrating users over an Independent Writer is safe so long as the users don't use both sides of the cache at once (ie home and target) I'm also interested in any recipe people have on GPFS policies to preseed and flush the cache. We plan to do all the migration using AFM over GPFS we're not currently using NFS and have no plans to start. I believe using GPFS is the faster method to preform the migration. Any suggestions and experience of doing similar migration jobs would be helpful. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ------------------------------ Message: 2 Date: Wed, 19 Oct 2016 13:46:02 -0400 From: Brian Marshall To: gpfsug main discussion list Subject: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="utf-8" All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 3 Date: Wed, 19 Oct 2016 18:10:38 +0000 From: "Simon Thompson (Research Computing - IT Services)" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="us-ascii" mmdiag --network Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Brian Marshall [mimarsh2 at vt.edu] Sent: 19 October 2016 18:46 To: gpfsug main discussion list Subject: [gpfsug-discuss] subnets All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall ------------------------------ Message: 4 Date: Wed, 19 Oct 2016 20:15:52 +0200 From: "Uwe Falke" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="ISO-8859-1" Hi Brian, you might use mmfsadm saferdump tscomm to check on which route peer cluster members are reached. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Frank Hammer, Thorsten Moehring Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: Brian Marshall To: gpfsug main discussion list Date: 10/19/2016 07:46 PM Subject: [gpfsug-discuss] subnets Sent by: gpfsug-discuss-bounces at spectrumscale.org All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ------------------------------ Message: 5 Date: Wed, 19 Oct 2016 20:11:57 +0000 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Subject: [gpfsug-discuss] Will there be any more GPFS 4.2.0-x releases? Message-ID: <142FECE0-E157-42D9-BC10-4C48E78FA065 at vanderbilt.edu> Content-Type: text/plain; charset="utf-8" Hi All, We?re currently running GPFS 4.2.0-4 with an efix installed and now we need a 2nd efix. I?m not a big fan of adding efix to efix and would prefer to go to a new PTF that contains both efixes. So ? is there going to be a GPFS 4.2.0-5 (it?s been a longer than normal interval since PTF 4 came out) or do we need to go to GPFS 4.2.1-x? If the latter, any major changes to watch out for? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 49 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-1466780990050_DSTlogo.png.png Type: image/png Size: 6282 bytes Desc: OutlookEmoji-1466780990050_DSTlogo.png.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-httpwww.prweb.comreleases201606prweb13504050.htm.jpg Type: image/jpeg Size: 14887 bytes Desc: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg URL: From volobuev at us.ibm.com Fri Oct 21 02:58:43 2016 From: volobuev at us.ibm.com (Yuri L Volobuev) Date: Thu, 20 Oct 2016 18:58:43 -0700 Subject: [gpfsug-discuss] Two new whitepapers published Message-ID: Esteemed GPFS and Spectrum Scale users, For your reading enjoyment, two new whitepapers have been posted to the Spectrum Scale Wiki: Spectrum?Scale:?Replication?in?GPFS Spectrum?Scale:?GPFS?Metadata The URL for the parent page is https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20 (GPFS)/page/White%20Papers%20%26%20Media The two .pdf documents are accessible through the Attachment section at the bottom of the page. ?Unfortunately, dW "spam prevention engine" does a very good job preventing me from "spamming" the page to actually add links. Best regards, Yuri -------------- next part -------------- An HTML attachment was scrubbed... URL: From alandhae at gmx.de Fri Oct 21 07:43:40 2016 From: alandhae at gmx.de (=?ISO-8859-15?Q?Andreas_Landh=E4u=DFer?=) Date: Fri, 21 Oct 2016 08:43:40 +0200 (CEST) Subject: [gpfsug-discuss] Two new whitepapers published In-Reply-To: References: Message-ID: Hi Yuri, Arrg, can't find them, page has been last updated on Aug, 18 by JohnTOlson, maybe its internal and not open for the public? Ciao Andreas On Fri, 21 Oct 2016, Yuri L Volobuev wrote: > > > Esteemed GPFS and Spectrum Scale users, > > For your reading enjoyment, two new whitepapers have been posted to the > Spectrum Scale Wiki: > > Spectrum?Scale:?Replication?in?GPFS > Spectrum?Scale:?GPFS?Metadata > > The URL for the parent page is > https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20 > (GPFS)/page/White%20Papers%20%26%20Media > > The two .pdf documents are accessible through the Attachment section at the > bottom of the page. ?Unfortunately, dW "spam prevention engine" does a very > good job preventing me from "spamming" the page to actually add links. > > Best regards, > > Yuri > -- Andreas Landh?u?er +49 151 12133027 (mobile) alandhae at gmx.de From dave at milk-vfx.com Fri Oct 21 07:51:51 2016 From: dave at milk-vfx.com (Dave Goodbourn) Date: Fri, 21 Oct 2016 07:51:51 +0100 Subject: [gpfsug-discuss] Two new whitepapers published In-Reply-To: References: Message-ID: I can find them. They're in the attachments tab in the bottom set of tabs. And a very good read. Thanks Yuri! Dave. > On 21 Oct 2016, at 07:43, Andreas Landh?u?er wrote: > > > Hi Yuri, > > Arrg, can't find them, page has been last updated on Aug, 18 by JohnTOlson, maybe its internal and not open for the public? > > Ciao > > Andreas > >> On Fri, 21 Oct 2016, Yuri L Volobuev wrote: >> >> >> >> Esteemed GPFS and Spectrum Scale users, >> >> For your reading enjoyment, two new whitepapers have been posted to the >> Spectrum Scale Wiki: >> >> Spectrum Scale: Replication in GPFS >> Spectrum Scale: GPFS Metadata >> >> The URL for the parent page is >> https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20 >> (GPFS)/page/White%20Papers%20%26%20Media >> >> The two .pdf documents are accessible through the Attachment section at the >> bottom of the page. Unfortunately, dW "spam prevention engine" does a very >> good job preventing me from "spamming" the page to actually add links. >> >> Best regards, >> >> Yuri >> > > -- > Andreas Landh?u?er +49 151 12133027 (mobile) > alandhae at gmx.de > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From laurence at qsplace.co.uk Fri Oct 21 08:10:56 2016 From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow) Date: Fri, 21 Oct 2016 08:10:56 +0100 Subject: [gpfsug-discuss] Two new whitepapers published In-Reply-To: References: Message-ID: <36EC1962-F71F-4492-A175-4BA3E9D97A8D@qsplace.co.uk> Right down the bottom of the page under attachments. -- Lauz On 21 October 2016 07:43:40 BST, "Andreas Landh?u?er" wrote: > >Hi Yuri, > >Arrg, can't find them, page has been last updated on Aug, 18 by >JohnTOlson, maybe its internal and not open for the public? > >Ciao > > Andreas > >On Fri, 21 Oct 2016, Yuri L Volobuev wrote: > >> >> >> Esteemed GPFS and Spectrum Scale users, >> >> For your reading enjoyment, two new whitepapers have been posted to >the >> Spectrum Scale Wiki: >> >> Spectrum?Scale:?Replication?in?GPFS >> Spectrum?Scale:?GPFS?Metadata >> >> The URL for the parent page is >> >https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20 >> (GPFS)/page/White%20Papers%20%26%20Media >> >> The two .pdf documents are accessible through the Attachment section >at the >> bottom of the page. ?Unfortunately, dW "spam prevention engine" does >a very >> good job preventing me from "spamming" the page to actually add >links. >> >> Best regards, >> >> Yuri >> > >-- >Andreas Landh?u?er +49 151 12133027 (mobile) >alandhae at gmx.de > >------------------------------------------------------------------------ > >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alandhae at gmx.de Fri Oct 21 08:31:43 2016 From: alandhae at gmx.de (=?ISO-8859-15?Q?Andreas_Landh=E4u=DFer?=) Date: Fri, 21 Oct 2016 09:31:43 +0200 (CEST) Subject: [gpfsug-discuss] Two new whitepapers published In-Reply-To: <36EC1962-F71F-4492-A175-4BA3E9D97A8D@qsplace.co.uk> References: <36EC1962-F71F-4492-A175-4BA3E9D97A8D@qsplace.co.uk> Message-ID: On Fri, 21 Oct 2016, Laurence Horrocks-Barlow wrote: Found it! thanks Yuri, for the two very interesting papers about the Metadata and replication. We always used the rule of thumb 5% for metadata, since we are having separate devices for metadata, and tiny fast disks aren't available anymore, we are getting a bunch of larger fast disks. We never experienced a problem with the (more or less) amount of metadata ... Andreas > Right down the bottom of the page under attachments. > > -- Lauz > > On 21 October 2016 07:43:40 BST, "Andreas Landh?u?er" wrote: >> >> Hi Yuri, >> >> Arrg, can't find them, page has been last updated on Aug, 18 by >> JohnTOlson, maybe its internal and not open for the public? >> >> Ciao >> >> Andreas >> >> On Fri, 21 Oct 2016, Yuri L Volobuev wrote: >> >>> >>> >>> Esteemed GPFS and Spectrum Scale users, >>> >>> For your reading enjoyment, two new whitepapers have been posted to >> the >>> Spectrum Scale Wiki: >>> >>> Spectrum?Scale:?Replication?in?GPFS >>> Spectrum?Scale:?GPFS?Metadata >>> >>> The URL for the parent page is >>> >> https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20 >>> (GPFS)/page/White%20Papers%20%26%20Media >>> >>> The two .pdf documents are accessible through the Attachment section >> at the >>> bottom of the page. ?Unfortunately, dW "spam prevention engine" does >> a very >>> good job preventing me from "spamming" the page to actually add >> links. >>> >>> Best regards, >>> >>> Yuri >>> >> >> -- >> Andreas Landh?u?er +49 151 12133027 (mobile) >> alandhae at gmx.de >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- Andreas Landh?u?er +49 151 12133027 (mobile) alandhae at gmx.de From daniel.kidger at uk.ibm.com Fri Oct 21 14:48:10 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Fri, 21 Oct 2016 13:48:10 +0000 Subject: [gpfsug-discuss] Two new whitepapers published In-Reply-To: <36EC1962-F71F-4492-A175-4BA3E9D97A8D@qsplace.co.uk> Message-ID: Great papers ! @Yuri: do you want to add entries at the top of that page to: a) help people find those papers, and b) make it more obvious that this page is not obsolete and no longer maintained. Daniel IBM Spectrum Storage Software +44 (0)7818 522266 Sent from my iPad using IBM Verse On 21 Oct 2016, 08:11:23, laurence at qsplace.co.uk wrote: From: laurence at qsplace.co.uk To: alandhae at gmx.de, gpfsug-discuss at spectrumscale.org Cc: Date: 21 Oct 2016 08:11:23 Subject: Re: [gpfsug-discuss] Two new whitepapers published Right down the bottom of the page under attachments. -- Lauz On 21 October 2016 07:43:40 BST, "Andreas Landh?u?er" wrote: Hi Yuri,Arrg, can't find them, page has been last updated on Aug, 18 by JohnTOlson, maybe its internal and not open for the public?Ciao AndreasOn Fri, 21 Oct 2016, Yuri L Volobuev wrote: Esteemed GPFS and Spectrum Scale users, For your reading enjoyment, two new whitepapers have been posted to the Spectrum Scale Wiki: Spectrum Scale: Replication in GPFS Spectrum Scale: GPFS Metadata The URL for the parent page is https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20 (GPFS)/page/White%20Papers%20%26%20Media The two .pdf documents are accessible through the Attachment section at the bottom of the page. Unfortunately, dW "spam prevention engine" does a very good job preventing me from "spamming" the page to actually add links. Best regards, Yuri -- Sent from my Android device with K-9 Mail. Please excuse my brevity.Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From volobuev at us.ibm.com Fri Oct 21 16:45:42 2016 From: volobuev at us.ibm.com (Yuri L Volobuev) Date: Fri, 21 Oct 2016 08:45:42 -0700 Subject: [gpfsug-discuss] Two new whitepapers published In-Reply-To: References: <36EC1962-F71F-4492-A175-4BA3E9D97A8D@qsplace.co.uk> Message-ID: I tried to do just that... the dW "spam prevention engine" wouldn't let me update the page. It may take some time to do this right. yuri From: "Daniel Kidger" To: "gpfsug main discussion list" , Date: 10/21/2016 06:48 AM Subject: Re: [gpfsug-discuss] Two new whitepapers published Sent by: gpfsug-discuss-bounces at spectrumscale.org Great papers ! @Yuri: do you want to add entries at the top of that page to: a) help people find those papers, and b) make it more obvious that this page is not obsolete and no longer maintained. Daniel IBM Spectrum Storage Software +44 (0)7818 522266 Sent from my iPad using IBM Verse On 21 Oct 2016, 08:11:23, laurence at qsplace.co.uk wrote: From: laurence at qsplace.co.uk To: alandhae at gmx.de, gpfsug-discuss at spectrumscale.org Cc: Date: 21 Oct 2016 08:11:23 Subject: Re: [gpfsug-discuss] Two new whitepapers published Right down the bottom of the page under attachments. -- Lauz On 21 October 2016 07:43:40 BST, "Andreas Landh?u?er" wrote: Hi Yuri, Arrg, can't find them, page has been last updated on Aug, 18 by JohnTOlson, maybe its internal and not open for the public? Ciao Andreas On Fri, 21 Oct 2016, Yuri L Volobuev wrote: Esteemed GPFS and Spectrum Scale users, For your reading enjoyment, two new whitepapers have been posted to the Spectrum Scale Wiki: Spectrum Scale: Replication in GPFS Spectrum Scale: GPFS Metadata The URL for the parent page is https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20 (GPFS)/page/White%20Papers%20%26%20Media The two .pdf documents are accessible through the Attachment section at the bottom of the page. Unfortunately, dW "spam prevention engine" does a very good job preventing me from "spamming" the page to actually add links. Best regards, Yuri -- Sent from my Android device with K-9 Mail. Please excuse my brevity. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From bpappas at dstonline.com Fri Oct 21 19:46:09 2016 From: bpappas at dstonline.com (Bill Pappas) Date: Fri, 21 Oct 2016 18:46:09 +0000 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) In-Reply-To: References: Message-ID: >>the largest of the filesets has 52TB and 63 million files Are you using NFS as the transport path between the home and cache? If you are using NFS, how are you producing the list of files to migrate? mmafmctl with the prefetch option? If so, I would measure the time it takes for that command (with that option) to produce the list of files it intends to prefetch. From my experience, this is very important as a) it can take a long time if you have >10 million of files and b) I've seen this operation crash when the list grew large. Does anyone else on this thread have any experiences? I would love to hear positive experiences as well. I tried so hard and for so long to make AFM work with one customer, but we gave up as it was not reliable and stable for large scale (many files) migrations. If you are using GPFS as the conduit between the home and cache (i.e. no NFS), I would still ask the same question, more with respect to stability for large file lists during the initial prefetch stages. As far as I could tell, from GPFS 3.5 to 4.2, the phases of prefetch where the home and cache are compared (i.e. let's make a list of what is ot be migrated over) before the data transfer begins only runs on the GW node managing that cache. It does not leverage multiple gw nodes and multiple home nodes to speed up this 'list and find' stage of prefetch. I hope some AFM developers can clarify or correct my findings. This was a huge impediment for large file migrations where it is difficult (organizationally, not technically) to split a folder structure into multiple file sets. The lack of stability under these large scans was the real failing for us. Bill Pappas 901-619-0585 bpappas at dstonline.com [1466780990050_DSTlogo.png] [http://www.prweb.com/releases/2016/06/prweb13504050.htm] http://www.prweb.com/releases/2016/06/prweb13504050.htm ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org Sent: Thursday, October 20, 2016 2:07 PM To: gpfsug-discuss at spectrumscale.org Subject: gpfsug-discuss Digest, Vol 57, Issue 53 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Using AFM to migrate files. (Peter Childs) (Peter Childs) ---------------------------------------------------------------------- Message: 1 Date: Thu, 20 Oct 2016 19:07:44 +0000 From: Peter Childs To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) Message-ID: <5qv6d7inj2j1pa94kqamk2uf.1476989646711 at email.android.com> Content-Type: text/plain; charset="iso-8859-1" Yes, most of the filesets are based on research groups, projects or departments, with the exception of scratch and home, hence the idea to use a different method for these filesets. There are approximately 230 million files, the largest of the filesets has 52TB and 63 million files. 300TB in total. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ---- Bill Pappas wrote ---- I have some ideas to suggest given some of my experiences. First, I have some questions: How many files are you migrating? Will you be creating multiple file sets on the target system based off of business or project needs? Like, file set a is for "department a" and file set b is for "large scale project a" Thanks. Bill Pappas 901-619-0585 bpappas at dstonline.com [1466780990050_DSTlogo.png] [http://www.prweb.com/releases/2016/06/prweb13504050.htm] http://www.prweb.com/releases/2016/06/prweb13504050.htm ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org Sent: Wednesday, October 19, 2016 3:12 PM To: gpfsug-discuss at spectrumscale.org Subject: gpfsug-discuss Digest, Vol 57, Issue 49 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Using AFM to migrate files. (Peter Childs) 2. subnets (Brian Marshall) 3. Re: subnets (Simon Thompson (Research Computing - IT Services)) 4. Re: subnets (Uwe Falke) 5. Will there be any more GPFS 4.2.0-x releases? (Buterbaugh, Kevin L) ---------------------------------------------------------------------- Message: 1 Date: Wed, 19 Oct 2016 14:12:41 +0000 From: Peter Childs To: gpfsug main discussion list Subject: [gpfsug-discuss] Using AFM to migrate files. Message-ID: Content-Type: text/plain; charset="iso-8859-1" We are planning to use AFM to migrate our old GPFS file store to a new GPFS file store. This will give us the advantages of Spectrum Scale (GPFS) 4.2, such as larger block and inode size. I would like to attempt to gain some insight on my plans before I start. The old file store was running GPFS 3.5 with 512 byte inodes and 1MB block size. We have now upgraded it to 4.1 and are working towards 4.2 with 300TB of files. (385TB max space) this is so we can use both the old and new storage via multi-cluster. We are moving to a new GPFS cluster so we can use the new protocol nodes eventually and also put the new storage machines as cluster managers, as this should be faster and future proof The new hardware has 1PB of space running GPFS 4.2 We have multiple filesets, and would like to maintain our namespace as far as possible. My plan was to. 1. Create a read-only (RO) AFM cache on the new storage (ro) 2a. Move old fileset and replace with SymLink to new. 2b. Convert RO AFM to Local Update (LU) AFM pointing to new parking area of old files. 2c. move user access to new location in cache. 3. Flush everything into cache and disconnect. I've read the docs including the ones on migration but it's not clear if it's safe to move the home of a cache and update the target. It looks like it should be possible and my tests say it works. An alternative plan is to use a Independent Writer (IW) AFM Cache to move the home directories which are pointed to by LDAP. Hence we can move users one at a time and only have to drain the HPC cluster at the end to disconnect the cache. I assume that migrating users over an Independent Writer is safe so long as the users don't use both sides of the cache at once (ie home and target) I'm also interested in any recipe people have on GPFS policies to preseed and flush the cache. We plan to do all the migration using AFM over GPFS we're not currently using NFS and have no plans to start. I believe using GPFS is the faster method to preform the migration. Any suggestions and experience of doing similar migration jobs would be helpful. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ------------------------------ Message: 2 Date: Wed, 19 Oct 2016 13:46:02 -0400 From: Brian Marshall To: gpfsug main discussion list Subject: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="utf-8" All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 3 Date: Wed, 19 Oct 2016 18:10:38 +0000 From: "Simon Thompson (Research Computing - IT Services)" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="us-ascii" mmdiag --network Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Brian Marshall [mimarsh2 at vt.edu] Sent: 19 October 2016 18:46 To: gpfsug main discussion list Subject: [gpfsug-discuss] subnets All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall ------------------------------ Message: 4 Date: Wed, 19 Oct 2016 20:15:52 +0200 From: "Uwe Falke" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="ISO-8859-1" Hi Brian, you might use mmfsadm saferdump tscomm to check on which route peer cluster members are reached. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Frank Hammer, Thorsten Moehring Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: Brian Marshall To: gpfsug main discussion list Date: 10/19/2016 07:46 PM Subject: [gpfsug-discuss] subnets Sent by: gpfsug-discuss-bounces at spectrumscale.org All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ------------------------------ Message: 5 Date: Wed, 19 Oct 2016 20:11:57 +0000 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Subject: [gpfsug-discuss] Will there be any more GPFS 4.2.0-x releases? Message-ID: <142FECE0-E157-42D9-BC10-4C48E78FA065 at vanderbilt.edu> Content-Type: text/plain; charset="utf-8" Hi All, We?re currently running GPFS 4.2.0-4 with an efix installed and now we need a 2nd efix. I?m not a big fan of adding efix to efix and would prefer to go to a new PTF that contains both efixes. So ? is there going to be a GPFS 4.2.0-5 (it?s been a longer than normal interval since PTF 4 came out) or do we need to go to GPFS 4.2.1-x? If the latter, any major changes to watch out for? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 49 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-1466780990050_DSTlogo.png.png Type: image/png Size: 6282 bytes Desc: OutlookEmoji-1466780990050_DSTlogo.png.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-httpwww.prweb.comreleases201606prweb13504050.htm.jpg Type: image/jpeg Size: 14887 bytes Desc: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 53 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-1466780990050_DSTlogo.png.png Type: image/png Size: 6282 bytes Desc: OutlookEmoji-1466780990050_DSTlogo.png.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg Type: image/jpeg Size: 14887 bytes Desc: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg URL: From p.childs at qmul.ac.uk Fri Oct 21 21:35:15 2016 From: p.childs at qmul.ac.uk (Peter Childs) Date: Fri, 21 Oct 2016 20:35:15 +0000 Subject: [gpfsug-discuss] Two new whitepapers published In-Reply-To: References: <36EC1962-F71F-4492-A175-4BA3E9D97A8D@qsplace.co.uk>, Message-ID: <7fifaf55s6bq10jlpkvl5he9.1477081370569@email.android.com> Reading through them and we'll worth read. How important is setting the correct cluster size at file system creation time? Ie with "mmchfs -n 256" ie how much band of error can you get away with? We have a cluster of 240 nodes that was setup with the default 32 setting, it's now about to grow to ~300 nodes. Is this likly to causing as an issue, which is difficult to fix. The manual says it can be adjusted but this white paper suggests not. Fortunately were also migrating our storage to new hardware, so have a good opportunity to get the setting right, this time around. Has anyone got any stats on the benefits of getting it "right" vs getting it wrong. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ---- Andreas Landh?u?er wrote ---- On Fri, 21 Oct 2016, Laurence Horrocks-Barlow wrote: Found it! thanks Yuri, for the two very interesting papers about the Metadata and replication. We always used the rule of thumb 5% for metadata, since we are having separate devices for metadata, and tiny fast disks aren't available anymore, we are getting a bunch of larger fast disks. We never experienced a problem with the (more or less) amount of metadata ... Andreas > Right down the bottom of the page under attachments. > > -- Lauz > > On 21 October 2016 07:43:40 BST, "Andreas Landh?u?er" wrote: >> >> Hi Yuri, >> >> Arrg, can't find them, page has been last updated on Aug, 18 by >> JohnTOlson, maybe its internal and not open for the public? >> >> Ciao >> >> Andreas >> >> On Fri, 21 Oct 2016, Yuri L Volobuev wrote: >> >>> >>> >>> Esteemed GPFS and Spectrum Scale users, >>> >>> For your reading enjoyment, two new whitepapers have been posted to >> the >>> Spectrum Scale Wiki: >>> >>> Spectrum Scale: Replication in GPFS >>> Spectrum Scale: GPFS Metadata >>> >>> The URL for the parent page is >>> >> https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20 >>> (GPFS)/page/White%20Papers%20%26%20Media >>> >>> The two .pdf documents are accessible through the Attachment section >> at the >>> bottom of the page. Unfortunately, dW "spam prevention engine" does >> a very >>> good job preventing me from "spamming" the page to actually add >> links. >>> >>> Best regards, >>> >>> Yuri >>> >> >> -- >> Andreas Landh?u?er +49 151 12133027 (mobile) >> alandhae at gmx.de >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- Andreas Landh?u?er +49 151 12133027 (mobile) alandhae at gmx.de -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.childs at qmul.ac.uk Fri Oct 21 21:44:18 2016 From: p.childs at qmul.ac.uk (Peter Childs) Date: Fri, 21 Oct 2016 20:44:18 +0000 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) In-Reply-To: References: , Message-ID: <9msuk69qtso8etpj6c9mq2pk.1477082656580@email.android.com> ---- Bill Pappas wrote ---- > >>the largest of the filesets has 52TB and 63 million files > > > Are you using NFS as the transport path between the home and cache? No plans to was planning to use gpfs multi-cluster, as transport. > If you are using NFS, how are you producing the list of files to migrate? mmafmctl with the prefetch option? If so, I would measure the time it takes for that command (with that option) to produce the list of files it intends to prefetch. From my experience, this is very important as a) it can take a long time if you have >10 million of files and b) I've seen this operation crash when the list grew large. Does anyone else on this thread have any experiences? I would love to hear positive experiences as well. I tried so hard and for so long to make AFM work with one customer, but we gave up as it was not reliable and stable for large scale (many files) migrations. > If you are using GPFS as the conduit between the home and cache (i.e. no NFS), I would still ask the same question, more with respect to stability for large file lists during the initial prefetch stages. I was planning to use a gpfs policy to create the list, but I guess a find should work, I'm guessing your saying don't migrate the files in bulk by using a find onto cache. It would be nice to see some examples recipes to prefetch files into afm. > > > As far as I could tell, from GPFS 3.5 to 4.2, the phases of prefetch where the home and cache are compared (i.e. let's make a list of what is ot be migrated over) before the data transfer begins only runs on the GW node managing that cache. It does not leverage multiple gw nodes and multiple home nodes to speed up this 'list and find' stage of prefetch. I hope some AFM developers can clarify or correct my findings. This was a huge impediment for large file migrations where it is difficult (organizationally, not technically) to split a folder structure into multiple file sets. The lack of stability under these large scans was the real failing for us. Interesting. > > > Bill Pappas > > 901-619-0585 > > bpappas at dstonline.com > > Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London > > > > > > http://www.prweb.com/releases/2016/06/prweb13504050.htm > > > > From: gpfsug-discuss-bounces at spectrumscale.org > on behalf of gpfsug-discuss-request at spectrumscale.org > > Sent: Thursday, October 20, 2016 2:07 PM > To: gpfsug-discuss at spectrumscale.org > Subject: gpfsug-discuss Digest, Vol 57, Issue 53 > > Send gpfsug-discuss mailing list submissions to > gpfsug-discuss at spectrumscale.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > or, via email, send a message with subject or body 'help' to > gpfsug-discuss-request at spectrumscale.org > > You can reach the person managing the list at > gpfsug-discuss-owner at spectrumscale.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of gpfsug-discuss digest..." > > > Today's Topics: > > 1. Re: Using AFM to migrate files. (Peter Childs) (Peter Childs) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 20 Oct 2016 19:07:44 +0000 > From: Peter Childs > > To: gpfsug main discussion list > > Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter > Childs) > Message-ID: <5qv6d7inj2j1pa94kqamk2uf.1476989646711 at email.android.com> > Content-Type: text/plain; charset="iso-8859-1" > > Yes, most of the filesets are based on research groups, projects or departments, with the exception of scratch and home, hence the idea to use a different method for these filesets. > > There are approximately 230 million files, the largest of the filesets has 52TB and 63 million files. 300TB in total. > > Peter Childs > Research Storage > ITS Research and Teaching Support > Queen Mary, University of London > > > ---- Bill Pappas wrote ---- > > > I have some ideas to suggest given some of my experiences. First, I have some questions: > > > How many files are you migrating? > > Will you be creating multiple file sets on the target system based off of business or project needs? Like, file set a is for "department a" and file set b is for "large scale project a" > > > Thanks. > > > Bill Pappas > > 901-619-0585 > > bpappas at dstonline.com > > > [1466780990050_DSTlogo.png] > > > [http://www.prweb.com/releases/2016/06/prweb13504050.htm] > > http://www.prweb.com/releases/2016/06/prweb13504050.htm > > > ________________________________ > From: gpfsug-discuss-bounces at spectrumscale.org > on behalf of gpfsug-discuss-request at spectrumscale.org > > Sent: Wednesday, October 19, 2016 3:12 PM > To: gpfsug-discuss at spectrumscale.org > Subject: gpfsug-discuss Digest, Vol 57, Issue 49 > > Send gpfsug-discuss mailing list submissions to > gpfsug-discuss at spectrumscale.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > or, via email, send a message with subject or body 'help' to > gpfsug-discuss-request at spectrumscale.org > > You can reach the person managing the list at > gpfsug-discuss-owner at spectrumscale.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of gpfsug-discuss digest..." > > > Today's Topics: > > 1. Using AFM to migrate files. (Peter Childs) > 2. subnets (Brian Marshall) > 3. Re: subnets (Simon Thompson (Research Computing - IT Services)) > 4. Re: subnets (Uwe Falke) > 5. Will there be any more GPFS 4.2.0-x releases? > (Buterbaugh, Kevin L) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 19 Oct 2016 14:12:41 +0000 > From: Peter Childs > > To: gpfsug main discussion list > > Subject: [gpfsug-discuss] Using AFM to migrate files. > Message-ID: > > > > Content-Type: text/plain; charset="iso-8859-1" > > > We are planning to use AFM to migrate our old GPFS file store to a new GPFS file store. This will give us the advantages of Spectrum Scale (GPFS) 4.2, such as larger block and inode size. I would like to attempt to gain some insight on my plans before I start. > > The old file store was running GPFS 3.5 with 512 byte inodes and 1MB block size. We have now upgraded it to 4.1 and are working towards 4.2 with 300TB of files. (385TB max space) this is so we can use both the old and new storage via multi-cluster. > > We are moving to a new GPFS cluster so we can use the new protocol nodes eventually and also put the new storage machines as cluster managers, as this should be faster and future proof > > The new hardware has 1PB of space running GPFS 4.2 > > We have multiple filesets, and would like to maintain our namespace as far as possible. > > My plan was to. > > 1. Create a read-only (RO) AFM cache on the new storage (ro) > 2a. Move old fileset and replace with SymLink to new. > 2b. Convert RO AFM to Local Update (LU) AFM pointing to new parking area of old files. > 2c. move user access to new location in cache. > 3. Flush everything into cache and disconnect. > > I've read the docs including the ones on migration but it's not clear if it's safe to move the home of a cache and update the target. It looks like it should be possible and my tests say it works. > > An alternative plan is to use a Independent Writer (IW) AFM Cache to move the home directories which are pointed to by LDAP. Hence we can move users one at a time and only have to drain the HPC cluster at the end to disconnect the cache. I assume that migrating users over an Independent Writer is safe so long as the users don't use both sides of the cache at once (ie home and target) > > I'm also interested in any recipe people have on GPFS policies to preseed and flush the cache. > > We plan to do all the migration using AFM over GPFS we're not currently using NFS and have no plans to start. I believe using GPFS is the faster method to preform the migration. > > Any suggestions and experience of doing similar migration jobs would be helpful. > > Peter Childs > Research Storage > ITS Research and Teaching Support > Queen Mary, University of London > > > > ------------------------------ > > Message: 2 > Date: Wed, 19 Oct 2016 13:46:02 -0400 > From: Brian Marshall > > To: gpfsug main discussion list > > Subject: [gpfsug-discuss] subnets > Message-ID: > > > Content-Type: text/plain; charset="utf-8" > > All, > > We are setting up communication between 2 clusters using ethernet and > IPoFabric. > > The Daemon interface is running on ethernet, so all admin traffic will use > it. > > We are still getting the subnets setting correct. > > Question: > > Does GPFS have a way to query how it is connecting to a given cluster/node? > i.e. once we have subnets setup how can we tell GPFS is actually using > them. Currently we just do a large transfer and check tcpdump for any > packets flowing on the high-speed/data/non-admin subnet. > > > Thank you, > Brian Marshall > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: ; > > ------------------------------ > > Message: 3 > Date: Wed, 19 Oct 2016 18:10:38 +0000 > From: "Simon Thompson (Research Computing - IT Services)" > > > To: gpfsug main discussion list > > Subject: Re: [gpfsug-discuss] subnets > Message-ID: > > > Content-Type: text/plain; charset="us-ascii" > > > mmdiag --network > > Simon > ________________________________________ > From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Brian Marshall [mimarsh2 at vt.edu] > Sent: 19 October 2016 18:46 > To: gpfsug main discussion list > Subject: [gpfsug-discuss] subnets > > All, > > We are setting up communication between 2 clusters using ethernet and IPoFabric. > > The Daemon interface is running on ethernet, so all admin traffic will use it. > > We are still getting the subnets setting correct. > > Question: > > Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. > > > Thank you, > Brian Marshall > > > ------------------------------ > > Message: 4 > Date: Wed, 19 Oct 2016 20:15:52 +0200 > From: "Uwe Falke" > > To: gpfsug main discussion list > > Subject: Re: [gpfsug-discuss] subnets > Message-ID: > > > > Content-Type: text/plain; charset="ISO-8859-1" > > Hi Brian, > you might use > > mmfsadm saferdump tscomm > > to check on which route peer cluster members are reached. > > > Mit freundlichen Gr??en / Kind regards > > > Dr. Uwe Falke > > IT Specialist > High Performance Computing Services / Integrated Technology Services / > Data Center Services > ------------------------------------------------------------------------------------------------------------------------------------------- > IBM Deutschland > Rathausstr. 7 > 09111 Chemnitz > Phone: +49 371 6978 2165 > Mobile: +49 175 575 2877 > E-Mail: uwefalke at de.ibm.com > ------------------------------------------------------------------------------------------------------------------------------------------- > IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: > Frank Hammer, Thorsten Moehring > Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, > HRB 17122 > > > > > From: Brian Marshall > > To: gpfsug main discussion list > > Date: 10/19/2016 07:46 PM > Subject: [gpfsug-discuss] subnets > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > All, > > We are setting up communication between 2 clusters using ethernet and > IPoFabric. > > The Daemon interface is running on ethernet, so all admin traffic will use > it. > > We are still getting the subnets setting correct. > > Question: > > Does GPFS have a way to query how it is connecting to a given > cluster/node? i.e. once we have subnets setup how can we tell GPFS is > actually using them. Currently we just do a large transfer and check > tcpdump for any packets flowing on the high-speed/data/non-admin subnet. > > > Thank you, > Brian Marshall_______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > ------------------------------ > > Message: 5 > Date: Wed, 19 Oct 2016 20:11:57 +0000 > From: "Buterbaugh, Kevin L" > > To: gpfsug main discussion list > > Subject: [gpfsug-discuss] Will there be any more GPFS 4.2.0-x > releases? > Message-ID: <142FECE0-E157-42D9-BC10-4C48E78FA065 at vanderbilt.edu> > Content-Type: text/plain; charset="utf-8" > > Hi All, > > We?re currently running GPFS 4.2.0-4 with an efix installed and now we need a 2nd efix. I?m not a big fan of adding efix to efix and would prefer to go to a new PTF that contains both efixes. > > So ? is there going to be a GPFS 4.2.0-5 (it?s been a longer than normal interval since PTF 4 came out) or do we need to go to GPFS 4.2.1-x? If the latter, any major changes to watch out for? Thanks? > > Kevin > > ? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and Education > Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633 > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: ; > > ------------------------------ > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > End of gpfsug-discuss Digest, Vol 57, Issue 49 > ********************************************** > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: ; > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: OutlookEmoji-1466780990050_DSTlogo.png.png > Type: image/png > Size: 6282 bytes > Desc: OutlookEmoji-1466780990050_DSTlogo.png.png > URL: ; > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: OutlookEmoji-httpwww.prweb.comreleases201606prweb13504050.htm.jpg > Type: image/jpeg > Size: 14887 bytes > Desc: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg > URL: ; > > ------------------------------ > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > End of gpfsug-discuss Digest, Vol 57, Issue 53 > ********************************************** >>the largest of the filesets has 52TB and 63 million files Are you using NFS as the transport path between the home and cache? If you are using NFS, how are you producing the list of files to migrate? mmafmctl with the prefetch option? If so, I would measure the time it takes for that command (with that option) to produce the list of files it intends to prefetch. From my experience, this is very important as a) it can take a long time if you have >10 million of files and b) I've seen this operation crash when the list grew large. Does anyone else on this thread have any experiences? I would love to hear positive experiences as well. I tried so hard and for so long to make AFM work with one customer, but we gave up as it was not reliable and stable for large scale (many files) migrations. If you are using GPFS as the conduit between the home and cache (i.e. no NFS), I would still ask the same question, more with respect to stability for large file lists during the initial prefetch stages. As far as I could tell, from GPFS 3.5 to 4.2, the phases of prefetch where the home and cache are compared (i.e. let's make a list of what is ot be migrated over) before the data transfer begins only runs on the GW node managing that cache. It does not leverage multiple gw nodes and multiple home nodes to speed up this 'list and find' stage of prefetch. I hope some AFM developers can clarify or correct my findings. This was a huge impediment for large file migrations where it is difficult (organizationally, not technically) to split a folder structure into multiple file sets. The lack of stability under these large scans was the real failing for us. Bill Pappas 901-619-0585 bpappas at dstonline.com [1466780990050_DSTlogo.png] [http://www.prweb.com/releases/2016/06/prweb13504050.htm] http://www.prweb.com/releases/2016/06/prweb13504050.htm ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org Sent: Thursday, October 20, 2016 2:07 PM To: gpfsug-discuss at spectrumscale.org Subject: gpfsug-discuss Digest, Vol 57, Issue 53 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Using AFM to migrate files. (Peter Childs) (Peter Childs) ---------------------------------------------------------------------- Message: 1 Date: Thu, 20 Oct 2016 19:07:44 +0000 From: Peter Childs To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) Message-ID: <5qv6d7inj2j1pa94kqamk2uf.1476989646711 at email.android.com> Content-Type: text/plain; charset="iso-8859-1" Yes, most of the filesets are based on research groups, projects or departments, with the exception of scratch and home, hence the idea to use a different method for these filesets. There are approximately 230 million files, the largest of the filesets has 52TB and 63 million files. 300TB in total. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ---- Bill Pappas wrote ---- I have some ideas to suggest given some of my experiences. First, I have some questions: How many files are you migrating? Will you be creating multiple file sets on the target system based off of business or project needs? Like, file set a is for "department a" and file set b is for "large scale project a" Thanks. Bill Pappas 901-619-0585 bpappas at dstonline.com [1466780990050_DSTlogo.png] [http://www.prweb.com/releases/2016/06/prweb13504050.htm] http://www.prweb.com/releases/2016/06/prweb13504050.htm ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org Sent: Wednesday, October 19, 2016 3:12 PM To: gpfsug-discuss at spectrumscale.org Subject: gpfsug-discuss Digest, Vol 57, Issue 49 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Using AFM to migrate files. (Peter Childs) 2. subnets (Brian Marshall) 3. Re: subnets (Simon Thompson (Research Computing - IT Services)) 4. Re: subnets (Uwe Falke) 5. Will there be any more GPFS 4.2.0-x releases? (Buterbaugh, Kevin L) ---------------------------------------------------------------------- Message: 1 Date: Wed, 19 Oct 2016 14:12:41 +0000 From: Peter Childs To: gpfsug main discussion list Subject: [gpfsug-discuss] Using AFM to migrate files. Message-ID: Content-Type: text/plain; charset="iso-8859-1" We are planning to use AFM to migrate our old GPFS file store to a new GPFS file store. This will give us the advantages of Spectrum Scale (GPFS) 4.2, such as larger block and inode size. I would like to attempt to gain some insight on my plans before I start. The old file store was running GPFS 3.5 with 512 byte inodes and 1MB block size. We have now upgraded it to 4.1 and are working towards 4.2 with 300TB of files. (385TB max space) this is so we can use both the old and new storage via multi-cluster. We are moving to a new GPFS cluster so we can use the new protocol nodes eventually and also put the new storage machines as cluster managers, as this should be faster and future proof The new hardware has 1PB of space running GPFS 4.2 We have multiple filesets, and would like to maintain our namespace as far as possible. My plan was to. 1. Create a read-only (RO) AFM cache on the new storage (ro) 2a. Move old fileset and replace with SymLink to new. 2b. Convert RO AFM to Local Update (LU) AFM pointing to new parking area of old files. 2c. move user access to new location in cache. 3. Flush everything into cache and disconnect. I've read the docs including the ones on migration but it's not clear if it's safe to move the home of a cache and update the target. It looks like it should be possible and my tests say it works. An alternative plan is to use a Independent Writer (IW) AFM Cache to move the home directories which are pointed to by LDAP. Hence we can move users one at a time and only have to drain the HPC cluster at the end to disconnect the cache. I assume that migrating users over an Independent Writer is safe so long as the users don't use both sides of the cache at once (ie home and target) I'm also interested in any recipe people have on GPFS policies to preseed and flush the cache. We plan to do all the migration using AFM over GPFS we're not currently using NFS and have no plans to start. I believe using GPFS is the faster method to preform the migration. Any suggestions and experience of doing similar migration jobs would be helpful. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ------------------------------ Message: 2 Date: Wed, 19 Oct 2016 13:46:02 -0400 From: Brian Marshall To: gpfsug main discussion list Subject: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="utf-8" All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 3 Date: Wed, 19 Oct 2016 18:10:38 +0000 From: "Simon Thompson (Research Computing - IT Services)" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="us-ascii" mmdiag --network Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Brian Marshall [mimarsh2 at vt.edu] Sent: 19 October 2016 18:46 To: gpfsug main discussion list Subject: [gpfsug-discuss] subnets All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall ------------------------------ Message: 4 Date: Wed, 19 Oct 2016 20:15:52 +0200 From: "Uwe Falke" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] subnets Message-ID: Content-Type: text/plain; charset="ISO-8859-1" Hi Brian, you might use mmfsadm saferdump tscomm to check on which route peer cluster members are reached. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Frank Hammer, Thorsten Moehring Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: Brian Marshall To: gpfsug main discussion list Date: 10/19/2016 07:46 PM Subject: [gpfsug-discuss] subnets Sent by: gpfsug-discuss-bounces at spectrumscale.org All, We are setting up communication between 2 clusters using ethernet and IPoFabric. The Daemon interface is running on ethernet, so all admin traffic will use it. We are still getting the subnets setting correct. Question: Does GPFS have a way to query how it is connecting to a given cluster/node? i.e. once we have subnets setup how can we tell GPFS is actually using them. Currently we just do a large transfer and check tcpdump for any packets flowing on the high-speed/data/non-admin subnet. Thank you, Brian Marshall_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ------------------------------ Message: 5 Date: Wed, 19 Oct 2016 20:11:57 +0000 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Subject: [gpfsug-discuss] Will there be any more GPFS 4.2.0-x releases? Message-ID: <142FECE0-E157-42D9-BC10-4C48E78FA065 at vanderbilt.edu> Content-Type: text/plain; charset="utf-8" Hi All, We?re currently running GPFS 4.2.0-4 with an efix installed and now we need a 2nd efix. I?m not a big fan of adding efix to efix and would prefer to go to a new PTF that contains both efixes. So ? is there going to be a GPFS 4.2.0-5 (it?s been a longer than normal interval since PTF 4 came out) or do we need to go to GPFS 4.2.1-x? If the latter, any major changes to watch out for? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 49 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-1466780990050_DSTlogo.png.png Type: image/png Size: 6282 bytes Desc: OutlookEmoji-1466780990050_DSTlogo.png.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-httpwww.prweb.comreleases201606prweb13504050.htm.jpg Type: image/jpeg Size: 14887 bytes Desc: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 53 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-1466780990050_DSTlogo.png.png Type: image/png Size: 6282 bytes Desc: OutlookEmoji-1466780990050_DSTlogo.png.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-httpwww.prweb.comreleases201606prweb13504050.htm.jpg Type: image/jpeg Size: 14887 bytes Desc: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg URL: From tortay at cc.in2p3.fr Sat Oct 22 11:45:23 2016 From: tortay at cc.in2p3.fr (Loic Tortay) Date: Sat, 22 Oct 2016 12:45:23 +0200 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) In-Reply-To: References: Message-ID: <580B4343.5020700@cc.in2p3.fr> On 21/10/2016 20:46, Bill Pappas wrote: [...] > > If you are using GPFS as the conduit between the home and cache > (i.e. no NFS), I would still ask the same question, more with respect to > stability for large file lists during the initial prefetch stages. > Hello, I'm in the final stage of what the AFM documentation calls an "incremental migration", for a filesystem with 100 million files. (GPFS 4.1.1, single cluster migration, "hardware/filesystem refresh" use case) I initially tried to use the NFS transport but found it too unreliable (and, in my opinion, very poorly documented). As I was about to give up on AFM, I tried using the GPFS transport (after seeing a trivially simple example on slides by someone from ANL) and things just started to work (almost) as I expected. For the files lists, I use data produced for our monitoring system that relies on snapshots fast scans (we do daily statistics on all objects in our GPFS filesystems). Our data gathering tool encodes object names in the RFC3986 (URL encoding) format which is what I found "mmafmctl prefetch" expects for "special" filenames. I understand that the policy engine does this too which, I guess, is what the documentation means by "generate a file list using policy" (sic), yet "mmafmctl prefetch" does not seem to accept/like files produced by a simple "LIST" policy (and the documentation lacks an example). As you did, I found that trying to prefetch large lists of files does not work reliably. I remember reading on that list someone (from IBM Germany, I think) recommanding to limit the number of a files in a single prefetch to 2 millions. This appears to be the sweet spot for my needs, as I can split the files list in 2 millions parts (the largest fileset in the "home" filesystem has 26 million files) and at the same time manage the issues I mention below. To keep up with the updates on the "home" filesystem (modified files), I rely on the "gpfs_winflags" GPFS extended attribute (the GPFS_WINATTR_OFFLINE bit is on for modified files, see "mmlsattr -L /cachefs/objectname" output). By chance, this attribute is included in the files produced for our statistics. This allows us to avoid doing a prefetch of all the files "continuously", since the file scan indeed appears to use only the (single) gateway node for the fileset being prefetched. In my specific configuration/environment, there are still several issues: . There is a significant memory and "slab" leak on the gateway nodes which can easily lead to a completely unreachable gateway node. These leaks appear directly related to the number of files prefetched. Stoping GPFS on the gateway node only releases some of the memory but none of the "slab", which requires a node reboot. . There is also a need to increase the "fs.file-max" sysctl on the gateway nodes to a value larger than the default (I use 10M), to avoid the kernel running out of file descriptors, since this leads to node being unreachable too. . Sometimes, an AFM association will go to the "Unmounted" state (for no apparent reason). The only reliable way to bring it back to "Active" state I found is to : unmount the "cache" filesystem from all nodes mouting it, unmounting/remounting the "home" filesystem on the gateway nodes, then remounting the "cache" filesystem where it is needed (gateway nodes, etc.) and finally using "ls -la" in the "cache" filesystem to bring the AFM associations back into the Active state. As I'm doing an "incremental migration", the "home" fileystem is still actively used and unmounting it on all nodes (as suggested by the documentation) is not an option. . I include directories and symlinks in the file lists for the prefetch. This ensures symlinks targets are there without needing a "ls" or "stat" in the "cache" filesystem ("incremental migration"). . The only reliable way I found to have "mmafmctl prefetch" accept files lists is to use the "--home-list-file" & "--home-fs-path" options. In my experience, in my environment, using AFM for migrations requires a significant amount of work and hand-holding to get a useful result. Since this migration is actually only the first step in a extensive multiple filesystems migration/merge plan, I'm pondering wether I'll use AFM for the rest of the operations. Sorry, this mail is too long, Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2931 bytes Desc: S/MIME Cryptographic Signature URL: From makaplan at us.ibm.com Sat Oct 22 14:05:34 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Sat, 22 Oct 2016 09:05:34 -0400 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames In-Reply-To: <580B4343.5020700@cc.in2p3.fr> References: <580B4343.5020700@cc.in2p3.fr> Message-ID: RULE .... EXTERNAL LIST ... ESCAPE '%' will direct mmapplypolicy to encode pathnames following RFC3986. Use ESCAPE '%/@' or similar for a more "relaxed" encoding. See the "Information lifecycle management" chapter of the official pubs for more details. Read the section that begins ... ESCAPE '%SpecialCharacters' Specifies that path names and the SHOW('string') expressions within the associated file lists are encoded with a scheme based on RFC3986 URI percent encoding. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tortay at cc.in2p3.fr Sat Oct 22 14:15:25 2016 From: tortay at cc.in2p3.fr (Loic Tortay) Date: Sat, 22 Oct 2016 15:15:25 +0200 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames In-Reply-To: References: <580B4343.5020700@cc.in2p3.fr> Message-ID: <580B666D.3090200@cc.in2p3.fr> On 22/10/2016 15:05, Marc A Kaplan wrote: > RULE .... EXTERNAL LIST ... ESCAPE '%' > will direct mmapplypolicy to encode pathnames following RFC3986. > Use ESCAPE '%/@' or similar for a more "relaxed" encoding. > > See the "Information lifecycle management" chapter of the official pubs > for more details. > Read the section that begins ... > > ESCAPE '%SpecialCharacters' > Specifies that path names and the SHOW('string') expressions within the > associated file lists are > encoded with a scheme based on RFC3986 URI percent encoding. > Hello, I've read (and used many times) that part of the fine documentation. My issue is with the documentation of what "mmafcmtl prefetch" expects. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2931 bytes Desc: S/MIME Cryptographic Signature URL: From makaplan at us.ibm.com Sat Oct 22 20:24:22 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Sat, 22 Oct 2016 15:24:22 -0400 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames In-Reply-To: <580B666D.3090200@cc.in2p3.fr> References: <580B4343.5020700@cc.in2p3.fr> <580B666D.3090200@cc.in2p3.fr> Message-ID: Hmmm.... Peeking into the script, it looks like some of AFM uses the simple \\ and \n file list escaping convention (the mmapplypolicy default with no ESCAPE specified) ... And other parts use ESCAPE '%'. Is this inconsistency hidden or does the CLI user have to deal with it? From: Loic Tortay To: gpfsug main discussion list Date: 10/22/2016 09:15 AM Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Sent by: gpfsug-discuss-bounces at spectrumscale.org On 22/10/2016 15:05, Marc A Kaplan wrote: > RULE .... EXTERNAL LIST ... ESCAPE '%' > will direct mmapplypolicy to encode pathnames following RFC3986. > Use ESCAPE '%/@' or similar for a more "relaxed" encoding. > > See the "Information lifecycle management" chapter of the official pubs > for more details. > Read the section that begins ... > > ESCAPE '%SpecialCharacters' > Specifies that path names and the SHOW('string') expressions within the > associated file lists are > encoded with a scheme based on RFC3986 URI percent encoding. > Hello, I've read (and used many times) that part of the fine documentation. My issue is with the documentation of what "mmafcmtl prefetch" expects. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | [attachment "smime.p7s" deleted by Marc A Kaplan/Watson/IBM] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtolson at us.ibm.com Mon Oct 24 01:41:07 2016 From: jtolson at us.ibm.com (John T Olson) Date: Sun, 23 Oct 2016 17:41:07 -0700 Subject: [gpfsug-discuss] New whitepaper published In-Reply-To: References: Message-ID: A new white paper has been published which shows integration of the Varonis UNIX agent in Spectrum Scale for audit logging. Here is a link to the paper: https://www.ibm.com/developerworks/community/wikis/form/api/wiki/fa32927c-e904-49cc-a4cc-870bcc8e307c/page/f0cc9b82-a133-41b4-83fe-3f560e95b35a/attachment/0ab62645-e0ab-4377-81e7-abd11879bb75/media/Spectrum_Scale_Varonis_Audit_Logging.pdf Thanks, John John T. Olson, Ph.D., MI.C., K.EY. Master Inventor, Software Defined Storage 957/9032-1 Tucson, AZ, 85744 (520) 799-5185, tie 321-5185 (FAX: 520-799-4237) Email: jtolson at us.ibm.com "Do or do not. There is no try." - Yoda Olson's Razor: Any situation that we, as humans, can encounter in life can be modeled by either an episode of The Simpsons or Seinfeld. -------------- next part -------------- An HTML attachment was scrubbed... URL: From laurence at qsplace.co.uk Mon Oct 24 08:10:12 2016 From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow) Date: Mon, 24 Oct 2016 08:10:12 +0100 Subject: [gpfsug-discuss] Two new whitepapers published In-Reply-To: <7fifaf55s6bq10jlpkvl5he9.1477081370569@email.android.com> References: <36EC1962-F71F-4492-A175-4BA3E9D97A8D@qsplace.co.uk>, <7fifaf55s6bq10jlpkvl5he9.1477081370569@email.android.com> Message-ID: <0386E88D-DC69-42DA-ACF7-BF0B0F45CD4E@qsplace.co.uk> Hi Peter, I've always been under the impression that this is a ball park figure that changes some of the on disk data structures to help parallel access to the filesystem I try to estimate the number of clients and then add ~10% (depending on the cluster size ofc). I have tested the default 32 vs 200 on a previous cluster however didn't find a difference in performance when testing up to 50 clients concurrently with IOR random and sequential. Maybe the difference is more subtle that just throughput? What I've always found strange is that the value can be changed on the filesystem however I don't believe this change effects an already created filesystem. -- Lauz On 21 October 2016 21:35:15 BST, Peter Childs wrote: > >Reading through them and we'll worth read. > >How important is setting the correct cluster size at file system >creation time? Ie with "mmchfs -n 256" ie how much band of error can >you get away with? > >We have a cluster of 240 nodes that was setup with the default 32 >setting, it's now about to grow to ~300 nodes. Is this likly to causing >as an issue, which is difficult to fix. The manual says it can be >adjusted but this white paper suggests not. > >Fortunately were also migrating our storage to new hardware, so have a >good opportunity to get the setting right, this time around. > >Has anyone got any stats on the benefits of getting it "right" vs >getting it wrong. > > > > >Peter Childs >Research Storage >ITS Research and Teaching Support >Queen Mary, University of London > > >---- Andreas Landh?u?er wrote ---- > >On Fri, 21 Oct 2016, Laurence Horrocks-Barlow >wrote: > >Found it! > >thanks Yuri, for the two very interesting papers about the Metadata and >replication. > >We always used the rule of thumb 5% for metadata, since we are having >separate devices for metadata, and tiny fast disks aren't available >anymore, we are getting a bunch of larger fast disks. We never >experienced >a problem with the (more or less) amount of metadata ... > > Andreas > > >> Right down the bottom of the page under attachments. >> >> -- Lauz >> >> On 21 October 2016 07:43:40 BST, "Andreas Landh?u?er" > wrote: >>> >>> Hi Yuri, >>> >>> Arrg, can't find them, page has been last updated on Aug, 18 by >>> JohnTOlson, maybe its internal and not open for the public? >>> >>> Ciao >>> >>> Andreas >>> >>> On Fri, 21 Oct 2016, Yuri L Volobuev wrote: >>> >>>> >>>> >>>> Esteemed GPFS and Spectrum Scale users, >>>> >>>> For your reading enjoyment, two new whitepapers have been posted to >>> the >>>> Spectrum Scale Wiki: >>>> >>>> Spectrum Scale: Replication in GPFS >>>> Spectrum Scale: GPFS Metadata >>>> >>>> The URL for the parent page is >>>> >>> >https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20 >>>> (GPFS)/page/White%20Papers%20%26%20Media >>>> >>>> The two .pdf documents are accessible through the Attachment >section >>> at the >>>> bottom of the page. Unfortunately, dW "spam prevention engine" >does >>> a very >>>> good job preventing me from "spamming" the page to actually add >>> links. >>>> >>>> Best regards, >>>> >>>> Yuri >>>> >>> >>> -- >>> Andreas Landh?u?er +49 151 12133027 >(mobile) >>> alandhae at gmx.de >>> >>> >------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> > >-- >Andreas Landh?u?er +49 151 12133027 >(mobile) >alandhae at gmx.de > > >------------------------------------------------------------------------ > >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vpuvvada at in.ibm.com Mon Oct 24 10:44:19 2016 From: vpuvvada at in.ibm.com (Venkateswara R Puvvada) Date: Mon, 24 Oct 2016 15:14:19 +0530 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames In-Reply-To: References: <580B4343.5020700@cc.in2p3.fr><580B666D.3090200@cc.in2p3.fr> Message-ID: Loic, mmafmctl prefecth expects encoded list file, and it is not documented correctly. Issues like memory leak, file descriptor leak, and fileset going into Unmounted state were fixed in later releases (4.2.1/4.2.2). All your points are correct with respect to AFM migration. There is manual intervention required. Also prefetch does not give list of files which were failed during data read. Users need to run policy to find all uncached files today. ~Venkat (vpuvvada at in.ibm.com) From: "Marc A Kaplan" To: gpfsug main discussion list Date: 10/23/2016 12:54 AM Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Sent by: gpfsug-discuss-bounces at spectrumscale.org Hmmm.... Peeking into the script, it looks like some of AFM uses the simple \\ and \n file list escaping convention (the mmapplypolicy default with no ESCAPE specified) ... And other parts use ESCAPE '%'. Is this inconsistency hidden or does the CLI user have to deal with it? From: Loic Tortay To: gpfsug main discussion list Date: 10/22/2016 09:15 AM Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Sent by: gpfsug-discuss-bounces at spectrumscale.org On 22/10/2016 15:05, Marc A Kaplan wrote: > RULE .... EXTERNAL LIST ... ESCAPE '%' > will direct mmapplypolicy to encode pathnames following RFC3986. > Use ESCAPE '%/@' or similar for a more "relaxed" encoding. > > See the "Information lifecycle management" chapter of the official pubs > for more details. > Read the section that begins ... > > ESCAPE '%SpecialCharacters' > Specifies that path names and the SHOW('string') expressions within the > associated file lists are > encoded with a scheme based on RFC3986 URI percent encoding. > Hello, I've read (and used many times) that part of the fine documentation. My issue is with the documentation of what "mmafcmtl prefetch" expects. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | [attachment "smime.p7s" deleted by Marc A Kaplan/Watson/IBM] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From erich at uw.edu Mon Oct 24 17:16:56 2016 From: erich at uw.edu (Eric Horst) Date: Mon, 24 Oct 2016 09:16:56 -0700 Subject: [gpfsug-discuss] Using AFM to migrate within the same cluster Message-ID: The recent conversation about AFM has been interesting. I've read the documentation several times and this is my question. Can AFM be used to migrate between two filesystems in the same cluster? There are examples of moving between clusters with NFS or native protocol but I've got a simple situation of needing to transparently move 100M files between two existing filesystems. Thanks, -Eric -- Eric Horst University of Washington From tortay at cc.in2p3.fr Mon Oct 24 17:50:51 2016 From: tortay at cc.in2p3.fr (Loic Tortay) Date: Mon, 24 Oct 2016 18:50:51 +0200 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames In-Reply-To: References: <580B4343.5020700@cc.in2p3.fr> <580B666D.3090200@cc.in2p3.fr> Message-ID: On 10/24/2016 11:44 AM, Venkateswara R Puvvada wrote: > > mmafmctl prefecth expects encoded list file, and it is not documented > correctly. Issues like memory leak, file descriptor leak, and fileset > going into Unmounted state were fixed in later releases (4.2.1/4.2.2). All > your points are correct with respect to AFM migration. There is manual > intervention required. Also prefetch does not give list of files which > were failed during data read. Users need to run policy to find all > uncached files today. > Hello, For the record, I have completed today my AFM migration of a filesystem with 100 million files. Users are now accessing the new filesystem. After disabling user access and a last "prefetch", the AFM filesets were converted to independent filesets. Less than 600 files were then found to be different between the "home" and the "cache" filesystems with a metadata comparison (I just copied the files from the old filesystem to the new one). I have compared the MD5 of a few thousand randomly selected files and found no differences between the "home" and the "cache" filesystems. I expect the users to let us know if they find something different (they have been instructed to do so). We'll keep the "home" filesystem around for some time, just in case there is a problem. Maybe something else that should be mentionned in the documentation is what to do with the ".ptrash" directories after the AFM filesets have been converted. I removed them since they contained files that had clearly been deleted by the users. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | From sfadden at us.ibm.com Mon Oct 24 17:57:33 2016 From: sfadden at us.ibm.com (Scott Fadden) Date: Mon, 24 Oct 2016 09:57:33 -0700 Subject: [gpfsug-discuss] Using AFM to migrate within the same cluster In-Reply-To: References: Message-ID: Yes you can use AFM to move data within a cluster. If you are using the NSD protocol the target needs to be a separate file system, if you are using NFS it needs to be an NFS export. Scott Fadden Spectrum Scale - Technical Marketing Phone: (503) 880-5833 sfadden at us.ibm.com http://www.ibm.com/systems/storage/spectrum/scale From: Eric Horst To: gpfsug main discussion list Date: 10/24/2016 09:17 AM Subject: [gpfsug-discuss] Using AFM to migrate within the same cluster Sent by: gpfsug-discuss-bounces at spectrumscale.org The recent conversation about AFM has been interesting. I've read the documentation several times and this is my question. Can AFM be used to migrate between two filesystems in the same cluster? There are examples of moving between clusters with NFS or native protocol but I've got a simple situation of needing to transparently move 100M files between two existing filesystems. Thanks, -Eric -- Eric Horst University of Washington _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From YARD at il.ibm.com Mon Oct 24 18:05:00 2016 From: YARD at il.ibm.com (Yaron Daniel) Date: Mon, 24 Oct 2016 20:05:00 +0300 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames In-Reply-To: References: <580B4343.5020700@cc.in2p3.fr><580B666D.3090200@cc.in2p3.fr> Message-ID: Hi Maybe worth also to check if there are any orphan files in the NEW fs ? Regards Yaron Daniel 94 Em Ha'Moshavot Rd Server, Storage and Data Services - Team Leader Petach Tiqva, 49527 Global Technology Services Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com IBM Israel From: Loic Tortay To: gpfsug main discussion list Date: 10/24/2016 07:50 PM Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Sent by: gpfsug-discuss-bounces at spectrumscale.org On 10/24/2016 11:44 AM, Venkateswara R Puvvada wrote: > > mmafmctl prefecth expects encoded list file, and it is not documented > correctly. Issues like memory leak, file descriptor leak, and fileset > going into Unmounted state were fixed in later releases (4.2.1/4.2.2). All > your points are correct with respect to AFM migration. There is manual > intervention required. Also prefetch does not give list of files which > were failed during data read. Users need to run policy to find all > uncached files today. > Hello, For the record, I have completed today my AFM migration of a filesystem with 100 million files. Users are now accessing the new filesystem. After disabling user access and a last "prefetch", the AFM filesets were converted to independent filesets. Less than 600 files were then found to be different between the "home" and the "cache" filesystems with a metadata comparison (I just copied the files from the old filesystem to the new one). I have compared the MD5 of a few thousand randomly selected files and found no differences between the "home" and the "cache" filesystems. I expect the users to let us know if they find something different (they have been instructed to do so). We'll keep the "home" filesystem around for some time, just in case there is a problem. Maybe something else that should be mentionned in the documentation is what to do with the ".ptrash" directories after the AFM filesets have been converted. I removed them since they contained files that had clearly been deleted by the users. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1851 bytes Desc: not available URL: From bpappas at dstonline.com Mon Oct 24 19:03:07 2016 From: bpappas at dstonline.com (Bill Pappas) Date: Mon, 24 Oct 2016 18:03:07 +0000 Subject: [gpfsug-discuss] Using AFM to migrate files. (Bill Pappas to Loric Totay In-Reply-To: References: Message-ID: >>For the record, I have completed today my AFM migration of a filesystem with 100 million files. Users are now accessing the new filesystem. Loric-> Hi. I was wondering, what version of GPFS where you running on the home and cache clusters? I take it you broke up the prefetch list into smaller (for example <2 million) file lists? If not, how? How much capacity did you migfrate over and how long did this process take? Thanks. Bill Pappas 901-619-0585 bpappas at dstonline.com [1466780990050_DSTlogo.png] [http://www.prweb.com/releases/2016/06/prweb13504050.htm] http://www.prweb.com/releases/2016/06/prweb13504050.htm ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org Sent: Monday, October 24, 2016 12:05 PM To: gpfsug-discuss at spectrumscale.org Subject: gpfsug-discuss Digest, Vol 57, Issue 60 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Using AFM to migrate within the same cluster (Eric Horst) 2. Re: Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames (Loic Tortay) 3. Re: Using AFM to migrate within the same cluster (Scott Fadden) 4. Re: Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames (Yaron Daniel) ---------------------------------------------------------------------- Message: 1 Date: Mon, 24 Oct 2016 09:16:56 -0700 From: Eric Horst To: gpfsug main discussion list Subject: [gpfsug-discuss] Using AFM to migrate within the same cluster Message-ID: Content-Type: text/plain; charset=UTF-8 The recent conversation about AFM has been interesting. I've read the documentation several times and this is my question. Can AFM be used to migrate between two filesystems in the same cluster? There are examples of moving between clusters with NFS or native protocol but I've got a simple situation of needing to transparently move 100M files between two existing filesystems. Thanks, -Eric -- Eric Horst University of Washington ------------------------------ Message: 2 Date: Mon, 24 Oct 2016 18:50:51 +0200 From: Loic Tortay To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Message-ID: Content-Type: text/plain; charset=utf-8 On 10/24/2016 11:44 AM, Venkateswara R Puvvada wrote: > > mmafmctl prefecth expects encoded list file, and it is not documented > correctly. Issues like memory leak, file descriptor leak, and fileset > going into Unmounted state were fixed in later releases (4.2.1/4.2.2). All > your points are correct with respect to AFM migration. There is manual > intervention required. Also prefetch does not give list of files which > were failed during data read. Users need to run policy to find all > uncached files today. > Hello, For the record, I have completed today my AFM migration of a filesystem with 100 million files. Users are now accessing the new filesystem. After disabling user access and a last "prefetch", the AFM filesets were converted to independent filesets. Less than 600 files were then found to be different between the "home" and the "cache" filesystems with a metadata comparison (I just copied the files from the old filesystem to the new one). I have compared the MD5 of a few thousand randomly selected files and found no differences between the "home" and the "cache" filesystems. I expect the users to let us know if they find something different (they have been instructed to do so). We'll keep the "home" filesystem around for some time, just in case there is a problem. Maybe something else that should be mentionned in the documentation is what to do with the ".ptrash" directories after the AFM filesets have been converted. I removed them since they contained files that had clearly been deleted by the users. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | ------------------------------ Message: 3 Date: Mon, 24 Oct 2016 09:57:33 -0700 From: "Scott Fadden" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Using AFM to migrate within the same cluster Message-ID: Content-Type: text/plain; charset="us-ascii" Yes you can use AFM to move data within a cluster. If you are using the NSD protocol the target needs to be a separate file system, if you are using NFS it needs to be an NFS export. Scott Fadden Spectrum Scale - Technical Marketing Phone: (503) 880-5833 sfadden at us.ibm.com http://www.ibm.com/systems/storage/spectrum/scale From: Eric Horst To: gpfsug main discussion list Date: 10/24/2016 09:17 AM Subject: [gpfsug-discuss] Using AFM to migrate within the same cluster Sent by: gpfsug-discuss-bounces at spectrumscale.org The recent conversation about AFM has been interesting. I've read the documentation several times and this is my question. Can AFM be used to migrate between two filesystems in the same cluster? There are examples of moving between clusters with NFS or native protocol but I've got a simple situation of needing to transparently move 100M files between two existing filesystems. Thanks, -Eric -- Eric Horst University of Washington _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 4 Date: Mon, 24 Oct 2016 20:05:00 +0300 From: "Yaron Daniel" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Message-ID: Content-Type: text/plain; charset="iso-8859-1" Hi Maybe worth also to check if there are any orphan files in the NEW fs ? Regards Yaron Daniel 94 Em Ha'Moshavot Rd Server, Storage and Data Services - Team Leader Petach Tiqva, 49527 Global Technology Services Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com IBM Israel From: Loic Tortay To: gpfsug main discussion list Date: 10/24/2016 07:50 PM Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Sent by: gpfsug-discuss-bounces at spectrumscale.org On 10/24/2016 11:44 AM, Venkateswara R Puvvada wrote: > > mmafmctl prefecth expects encoded list file, and it is not documented > correctly. Issues like memory leak, file descriptor leak, and fileset > going into Unmounted state were fixed in later releases (4.2.1/4.2.2). All > your points are correct with respect to AFM migration. There is manual > intervention required. Also prefetch does not give list of files which > were failed during data read. Users need to run policy to find all > uncached files today. > Hello, For the record, I have completed today my AFM migration of a filesystem with 100 million files. Users are now accessing the new filesystem. After disabling user access and a last "prefetch", the AFM filesets were converted to independent filesets. Less than 600 files were then found to be different between the "home" and the "cache" filesystems with a metadata comparison (I just copied the files from the old filesystem to the new one). I have compared the MD5 of a few thousand randomly selected files and found no differences between the "home" and the "cache" filesystems. I expect the users to let us know if they find something different (they have been instructed to do so). We'll keep the "home" filesystem around for some time, just in case there is a problem. Maybe something else that should be mentionned in the documentation is what to do with the ".ptrash" directories after the AFM filesets have been converted. I removed them since they contained files that had clearly been deleted by the users. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1851 bytes Desc: not available URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 60 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-1466780990050_DSTlogo.png.png Type: image/png Size: 6282 bytes Desc: OutlookEmoji-1466780990050_DSTlogo.png.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg Type: image/jpeg Size: 14887 bytes Desc: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg URL: From vpuvvada at in.ibm.com Tue Oct 25 05:53:10 2016 From: vpuvvada at in.ibm.com (Venkateswara R Puvvada) Date: Tue, 25 Oct 2016 10:23:10 +0530 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames In-Reply-To: References: <580B4343.5020700@cc.in2p3.fr><580B666D.3090200@cc.in2p3.fr> Message-ID: Loic, It is good to hear that migration is completed. Did you find any prefetch errors in mmfs.log for the files which are different between cache and home ? Not all errors are logged, there is a plan to generate the list of read failed file names after prefetch completion. Informaion about files moving to .ptrash is documented @ http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.1/com.ibm.spectrum.scale.v4r21.doc/bl1ins_internaldirectoriesAFM.htm ~Venkat (vpuvvada at in.ibm.com) From: Loic Tortay To: gpfsug main discussion list Date: 10/24/2016 10:20 PM Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Sent by: gpfsug-discuss-bounces at spectrumscale.org On 10/24/2016 11:44 AM, Venkateswara R Puvvada wrote: > > mmafmctl prefecth expects encoded list file, and it is not documented > correctly. Issues like memory leak, file descriptor leak, and fileset > going into Unmounted state were fixed in later releases (4.2.1/4.2.2). All > your points are correct with respect to AFM migration. There is manual > intervention required. Also prefetch does not give list of files which > were failed during data read. Users need to run policy to find all > uncached files today. > Hello, For the record, I have completed today my AFM migration of a filesystem with 100 million files. Users are now accessing the new filesystem. After disabling user access and a last "prefetch", the AFM filesets were converted to independent filesets. Less than 600 files were then found to be different between the "home" and the "cache" filesystems with a metadata comparison (I just copied the files from the old filesystem to the new one). I have compared the MD5 of a few thousand randomly selected files and found no differences between the "home" and the "cache" filesystems. I expect the users to let us know if they find something different (they have been instructed to do so). We'll keep the "home" filesystem around for some time, just in case there is a problem. Maybe something else that should be mentionned in the documentation is what to do with the ".ptrash" directories after the AFM filesets have been converted. I removed them since they contained files that had clearly been deleted by the users. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From radhika.p at in.ibm.com Tue Oct 25 08:24:56 2016 From: radhika.p at in.ibm.com (Radhika A Parameswaran) Date: Tue, 25 Oct 2016 12:54:56 +0530 Subject: [gpfsug-discuss] Using AFM to migrate files In-Reply-To: References: Message-ID: Bill, The issues/limitation with preparing a listfile with >2M files in the file list has been fixed in 4.2.1. It has also been internally checked with 45M file list. Peter, The 4.2.1 man pages and prefetch section has added some examples of policy and listfile options for prefetch. Please take a look at them and let us know if those help. Loic, We will add about removing .ptrash to the migration usecase documentation. Can you please share some details about your dataset and performance for migrating the 100M (time for listfile processing and actual transfer) ? Thanks and Regards Radhika From: gpfsug-discuss-request at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Date: 10/24/2016 11:33 PM Subject: gpfsug-discuss Digest, Vol 57, Issue 61 Sent by: gpfsug-discuss-bounces at spectrumscale.org Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Using AFM to migrate files. (Bill Pappas to Loric Totay (Bill Pappas) ---------------------------------------------------------------------- Message: 1 Date: Mon, 24 Oct 2016 18:03:07 +0000 From: Bill Pappas To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Bill Pappas to Loric Totay Message-ID: Content-Type: text/plain; charset="iso-8859-1" >>For the record, I have completed today my AFM migration of a filesystem with 100 million files. Users are now accessing the new filesystem. Loric-> Hi. I was wondering, what version of GPFS where you running on the home and cache clusters? I take it you broke up the prefetch list into smaller (for example <2 million) file lists? If not, how? How much capacity did you migfrate over and how long did this process take? Thanks. Bill Pappas 901-619-0585 bpappas at dstonline.com [1466780990050_DSTlogo.png] [http://www.prweb.com/releases/2016/06/prweb13504050.htm] http://www.prweb.com/releases/2016/06/prweb13504050.htm ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org Sent: Monday, October 24, 2016 12:05 PM To: gpfsug-discuss at spectrumscale.org Subject: gpfsug-discuss Digest, Vol 57, Issue 60 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Using AFM to migrate within the same cluster (Eric Horst) 2. Re: Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames (Loic Tortay) 3. Re: Using AFM to migrate within the same cluster (Scott Fadden) 4. Re: Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames (Yaron Daniel) ---------------------------------------------------------------------- Message: 1 Date: Mon, 24 Oct 2016 09:16:56 -0700 From: Eric Horst To: gpfsug main discussion list Subject: [gpfsug-discuss] Using AFM to migrate within the same cluster Message-ID: Content-Type: text/plain; charset=UTF-8 The recent conversation about AFM has been interesting. I've read the documentation several times and this is my question. Can AFM be used to migrate between two filesystems in the same cluster? There are examples of moving between clusters with NFS or native protocol but I've got a simple situation of needing to transparently move 100M files between two existing filesystems. Thanks, -Eric -- Eric Horst University of Washington ------------------------------ Message: 2 Date: Mon, 24 Oct 2016 18:50:51 +0200 From: Loic Tortay To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Message-ID: Content-Type: text/plain; charset=utf-8 On 10/24/2016 11:44 AM, Venkateswara R Puvvada wrote: > > mmafmctl prefecth expects encoded list file, and it is not documented > correctly. Issues like memory leak, file descriptor leak, and fileset > going into Unmounted state were fixed in later releases (4.2.1/4.2.2). All > your points are correct with respect to AFM migration. There is manual > intervention required. Also prefetch does not give list of files which > were failed during data read. Users need to run policy to find all > uncached files today. > Hello, For the record, I have completed today my AFM migration of a filesystem with 100 million files. Users are now accessing the new filesystem. After disabling user access and a last "prefetch", the AFM filesets were converted to independent filesets. Less than 600 files were then found to be different between the "home" and the "cache" filesystems with a metadata comparison (I just copied the files from the old filesystem to the new one). I have compared the MD5 of a few thousand randomly selected files and found no differences between the "home" and the "cache" filesystems. I expect the users to let us know if they find something different (they have been instructed to do so). We'll keep the "home" filesystem around for some time, just in case there is a problem. Maybe something else that should be mentionned in the documentation is what to do with the ".ptrash" directories after the AFM filesets have been converted. I removed them since they contained files that had clearly been deleted by the users. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | ------------------------------ Message: 3 Date: Mon, 24 Oct 2016 09:57:33 -0700 From: "Scott Fadden" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Using AFM to migrate within the same cluster Message-ID: Content-Type: text/plain; charset="us-ascii" Yes you can use AFM to move data within a cluster. If you are using the NSD protocol the target needs to be a separate file system, if you are using NFS it needs to be an NFS export. Scott Fadden Spectrum Scale - Technical Marketing Phone: (503) 880-5833 sfadden at us.ibm.com http://www.ibm.com/systems/storage/spectrum/scale From: Eric Horst To: gpfsug main discussion list Date: 10/24/2016 09:17 AM Subject: [gpfsug-discuss] Using AFM to migrate within the same cluster Sent by: gpfsug-discuss-bounces at spectrumscale.org The recent conversation about AFM has been interesting. I've read the documentation several times and this is my question. Can AFM be used to migrate between two filesystems in the same cluster? There are examples of moving between clusters with NFS or native protocol but I've got a simple situation of needing to transparently move 100M files between two existing filesystems. Thanks, -Eric -- Eric Horst University of Washington _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161024/866c55cf/attachment-0001.html > ------------------------------ Message: 4 Date: Mon, 24 Oct 2016 20:05:00 +0300 From: "Yaron Daniel" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Message-ID: Content-Type: text/plain; charset="iso-8859-1" Hi Maybe worth also to check if there are any orphan files in the NEW fs ? Regards Yaron Daniel 94 Em Ha'Moshavot Rd Server, Storage and Data Services - Team Leader Petach Tiqva, 49527 Global Technology Services Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com IBM Israel From: Loic Tortay To: gpfsug main discussion list Date: 10/24/2016 07:50 PM Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames Sent by: gpfsug-discuss-bounces at spectrumscale.org On 10/24/2016 11:44 AM, Venkateswara R Puvvada wrote: > > mmafmctl prefecth expects encoded list file, and it is not documented > correctly. Issues like memory leak, file descriptor leak, and fileset > going into Unmounted state were fixed in later releases (4.2.1/4.2.2). All > your points are correct with respect to AFM migration. There is manual > intervention required. Also prefetch does not give list of files which > were failed during data read. Users need to run policy to find all > uncached files today. > Hello, For the record, I have completed today my AFM migration of a filesystem with 100 million files. Users are now accessing the new filesystem. After disabling user access and a last "prefetch", the AFM filesets were converted to independent filesets. Less than 600 files were then found to be different between the "home" and the "cache" filesystems with a metadata comparison (I just copied the files from the old filesystem to the new one). I have compared the MD5 of a few thousand randomly selected files and found no differences between the "home" and the "cache" filesystems. I expect the users to let us know if they find something different (they have been instructed to do so). We'll keep the "home" filesystem around for some time, just in case there is a problem. Maybe something else that should be mentionned in the documentation is what to do with the ".ptrash" directories after the AFM filesets have been converted. I removed them since they contained files that had clearly been deleted by the users. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161024/449a0cf1/attachment.html > -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1851 bytes Desc: not available URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161024/449a0cf1/attachment.gif > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 60 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161024/fff3f370/attachment.html > -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-1466780990050_DSTlogo.png.png Type: image/png Size: 6282 bytes Desc: OutlookEmoji-1466780990050_DSTlogo.png.png URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161024/fff3f370/attachment.png > -------------- next part -------------- A non-text attachment was scrubbed... Name: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg Type: image/jpeg Size: 14887 bytes Desc: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161024/fff3f370/attachment.jpg > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 61 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: From tortay at cc.in2p3.fr Tue Oct 25 13:01:01 2016 From: tortay at cc.in2p3.fr (Loic Tortay) Date: Tue, 25 Oct 2016 14:01:01 +0200 Subject: [gpfsug-discuss] Using AFM to migrate files. (Bill Pappas to Loric Totay In-Reply-To: References: Message-ID: <05eb9917-0b3a-1bde-fc04-9acaed06fe29@cc.in2p3.fr> On 10/24/2016 08:03 PM, Bill Pappas wrote: > > Loric-> Hi. I was wondering, what version of GPFS where you running > on the home and cache clusters? I take it you broke up the prefetch list > into smaller (for example <2 million) file lists? If not, how? How much > capacity did you migfrate over and how long did this process take? Thanks. > Hello, We use GPFS 4.1.1-8. The migration was within a single cluster. The files lists were split in 2M files parts. The amount of data migrated is rather small (~70 TB), but there are ~100M files, including many extremely small files: the median (not average) file size is 2705 bytes. The migration itself took about 3 weeks: I initially did the first "mmafmctl prefetches" one fileset at a time (& in slices of 2M files), to manage the various issues mentionned previously and minimize the impact on the "home" filesystem since it was still being used by the end users. Following "prefetches" were done a few filesets at a time (filesets chosen to spread the load on the gateway nodes). We chose to do an "incremental migration" instead of a "progressive migration", since we were unable to get a good understanding of the performance impact of the AFM migration during our tests on a toy cluster running on VMs. Lo?c (Loic w/ a diaresis on the i, not Loric :-) -- | Lo?c Tortay - IN2P3 Computing Centre | From matt.thorpe at bodleian.ox.ac.uk Tue Oct 25 13:05:36 2016 From: matt.thorpe at bodleian.ox.ac.uk (Matt Thorpe) Date: Tue, 25 Oct 2016 12:05:36 +0000 Subject: [gpfsug-discuss] Forcing which node gets expelled? Message-ID: <63906C4AF5FFD0429565A743F6AD5BEDDAF29F@MBX04.ad.oak.ox.ac.uk> Hi, We are in the process of diagnosing a networking issue that is causing 2 of our 6 node GPFS cluster to expel each other (it appears they experience a temporary network connection outage and lose contact with each other). At present it's not consistent which gets expelled by the cluster manager, and I wondered if there was any way to always force a specific node to be expelled in this situation? Thanks and best regards, Matt -------- Matt Thorpe | BDLSS Systems Administrator Bodleian Libraries Osney One Building, Osney Mead, Oxford, OX2 0EW matt.thorpe at bodleian.ox.ac.uk | 01865 (2)80027 From tortay at cc.in2p3.fr Tue Oct 25 13:17:56 2016 From: tortay at cc.in2p3.fr (Loic Tortay) Date: Tue, 25 Oct 2016 14:17:56 +0200 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames In-Reply-To: References: <580B4343.5020700@cc.in2p3.fr> <580B666D.3090200@cc.in2p3.fr> Message-ID: On 10/24/2016 07:05 PM, Yaron Daniel wrote: > > Maybe worth also to check if there are any orphan files in the NEW fs ? > Hello, I ran a online "dry run" (-n) mmfsck which found a few lost blocks, but I'm not sure orphan files are detected with an online mmfsck. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | From Robert.Oesterlin at nuance.com Tue Oct 25 13:22:30 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Tue, 25 Oct 2016 12:22:30 +0000 Subject: [gpfsug-discuss] Forcing which node gets expelled? In-Reply-To: <63906C4AF5FFD0429565A743F6AD5BEDDAF29F@MBX04.ad.oak.ox.ac.uk> References: <63906C4AF5FFD0429565A743F6AD5BEDDAF29F@MBX04.ad.oak.ox.ac.uk> Message-ID: <4403E9DB-88B4-4281-91B5-E171D1284899@nuance.com> If you look at /usr/lpp/mmfs/samples/expelnode.sample you can use this as a base and install this in /var/mmfs/etc on the cluster manager. You can control which of the two nodes get expelled. We use it here to send an alert when a node is expelled. There is also "mmexpelnode" which you can force a particular node to be expelled. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Matt Thorpe Reply-To: gpfsug main discussion list Date: Tuesday, October 25, 2016 at 7:05 AM To: gpfsug main discussion list Subject: [EXTERNAL] [gpfsug-discuss] Forcing which node gets expelled? Hi, We are in the process of diagnosing a networking issue that is causing 2 of our 6 node GPFS cluster to expel each other (it appears they experience a temporary network connection outage and lose contact with each other). At present it's not consistent which gets expelled by the cluster manager, and I wondered if there was any way to always force a specific node to be expelled in this situation? Thanks and best regards, Matt -------- Matt Thorpe | BDLSS Systems Administrator Bodleian Libraries Osney One Building, Osney Mead, Oxford, OX2 0EW matt.thorpe at bodleian.ox.ac.uk | 01865 (2)80027 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Q_f6z64tvENxA9ac7rqWFj2jNd5IrpWEXcynJzHFjz4&s=jqso6xVB-V_zgLba-xjlWwiw3fNRan9NspsVq4PY4nA&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From tortay at cc.in2p3.fr Tue Oct 25 13:31:59 2016 From: tortay at cc.in2p3.fr (Loic Tortay) Date: Tue, 25 Oct 2016 14:31:59 +0200 Subject: [gpfsug-discuss] Using AFM to migrate files. (Peter Childs) (Peter Childs) - URL encoding for pathnames In-Reply-To: References: <580B4343.5020700@cc.in2p3.fr> <580B666D.3090200@cc.in2p3.fr> Message-ID: On 25/10/2016 06:53, Venkateswara R Puvvada wrote: > Loic, > > It is good to hear that migration is completed. Did you find any prefetch > errors in mmfs.log for the files which are different between cache and > home ? Not all errors are logged, there is a plan to generate the list of > read failed file names after prefetch completion. > Hello, Indeed, I had missed those messages: $GATEWAY1: Mon Oct 24 10:44:05.725 2016: [E] AFM: Read file system $FS fileset juno file IDs [469826991.470473596.-1.-1,N] name CMakeFiles local error 21 $GATEWAY1: Mon Oct 24 10:50:05.655 2016: [E] AFM: Read file system $FS fileset mnm file IDs [637555582.-1.-1.-1,N] name Mn_G4_SansTube local error 21 >From what I can see, for the previous runs of "mmafmctl prefetch", there are much less error messages logged than the values of the "asyncReadFailed" counters displayed by "mmafmctl prefetch -j $FILESET -Y". Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | From UWEFALKE at de.ibm.com Tue Oct 25 13:32:11 2016 From: UWEFALKE at de.ibm.com (Uwe Falke) Date: Tue, 25 Oct 2016 14:32:11 +0200 Subject: [gpfsug-discuss] Forcing which node gets expelled? In-Reply-To: <63906C4AF5FFD0429565A743F6AD5BEDDAF29F@MBX04.ad.oak.ox.ac.uk> References: <63906C4AF5FFD0429565A743F6AD5BEDDAF29F@MBX04.ad.oak.ox.ac.uk> Message-ID: Usually, the cluster mgr, receiving a complaint from a node about another node being gone, checks its own connection to that other node. If that is positive it expells the requester, if not it follows the request and expells the other node. AFAIK, there are some more subtle algorithms in place if managers or quorum nodes are affected. Maybe that can be used to protect certain nodes from getting expelled by assigning some role in the cluster to them. I do however not know these exactly. That means: it is not easily controllable which one gets expelled. It is better to concentrate on fixing your connectivity issues, as GPFS will not feel comfortable in such a unreliable environment anyway. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Frank Hammer, Thorsten Moehring Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: Matt Thorpe To: gpfsug main discussion list Date: 10/25/2016 02:05 PM Subject: [gpfsug-discuss] Forcing which node gets expelled? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi, We are in the process of diagnosing a networking issue that is causing 2 of our 6 node GPFS cluster to expel each other (it appears they experience a temporary network connection outage and lose contact with each other). At present it's not consistent which gets expelled by the cluster manager, and I wondered if there was any way to always force a specific node to be expelled in this situation? Thanks and best regards, Matt -------- Matt Thorpe | BDLSS Systems Administrator Bodleian Libraries Osney One Building, Osney Mead, Oxford, OX2 0EW matt.thorpe at bodleian.ox.ac.uk | 01865 (2)80027 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From tortay at cc.in2p3.fr Tue Oct 25 14:01:20 2016 From: tortay at cc.in2p3.fr (Loic Tortay) Date: Tue, 25 Oct 2016 15:01:20 +0200 Subject: [gpfsug-discuss] Using AFM to migrate files In-Reply-To: References: Message-ID: On 10/25/2016 09:24 AM, Radhika A Parameswaran wrote: > > Loic, > We will add about removing .ptrash to the migration usecase documentation. > Can you please share some details about your dataset and performance for > migrating the 100M (time for listfile processing and actual transfer) ? > Hello, According to my logs, the files lists processing by "mmafmctl prefetch" with 2M files took between ~30 minutes and ~70 minutes. >From what I can see, the longer processing times were for filesets with many directories (less files per directory). For the actual data transfers, it varied widely from a few minutes (small files stored in the inode on a Flash-based storage unit) to a few hours (1 or 2 TB of files not large enough to trigger the parallel migration, while many batch jobs were accessing the "home" filesystem). Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | From knop at us.ibm.com Tue Oct 25 14:02:39 2016 From: knop at us.ibm.com (Felipe Knop) Date: Tue, 25 Oct 2016 09:02:39 -0400 Subject: [gpfsug-discuss] Forcing which node gets expelled? In-Reply-To: References: <63906C4AF5FFD0429565A743F6AD5BEDDAF29F@MBX04.ad.oak.ox.ac.uk> Message-ID: All, As Bob Oesterlin indicated, it is possible to define an expel script (see /usr/lpp/mmfs/samples/expelnode.sample) to control which of the two nodes to get expelled. The script can also be used to issue alerts, etc. The current policy used (before the script is invoked) when deciding which node to expel is the following: 1. quorum nodes over non-quorum nodes 2. local nodes over remote nodes 3. manager-capable nodes over non-manager-capable nodes 4. nodes managing more FSs over nodes managing fewer FSs 5. NSD server over non-NSD server Otherwise, expel whoever joined the cluster more recently. The statement below from Dr. Uwe Falke is also correct: addressing the network connectivity is the better long-term approach, but the callback script could be used to control which node to expel. Felipe ---- Felipe Knop knop at us.ibm.com GPFS Development and Security IBM Systems IBM Building 008 2455 South Rd, Poughkeepsie, NY 12601 (845) 433-9314 T/L 293-9314 From: "Uwe Falke" To: gpfsug main discussion list Date: 10/25/2016 08:32 AM Subject: Re: [gpfsug-discuss] Forcing which node gets expelled? Sent by: gpfsug-discuss-bounces at spectrumscale.org Usually, the cluster mgr, receiving a complaint from a node about another node being gone, checks its own connection to that other node. If that is positive it expells the requester, if not it follows the request and expells the other node. AFAIK, there are some more subtle algorithms in place if managers or quorum nodes are affected. Maybe that can be used to protect certain nodes from getting expelled by assigning some role in the cluster to them. I do however not know these exactly. That means: it is not easily controllable which one gets expelled. It is better to concentrate on fixing your connectivity issues, as GPFS will not feel comfortable in such a unreliable environment anyway. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Frank Hammer, Thorsten Moehring Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: Matt Thorpe To: gpfsug main discussion list Date: 10/25/2016 02:05 PM Subject: [gpfsug-discuss] Forcing which node gets expelled? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi, We are in the process of diagnosing a networking issue that is causing 2 of our 6 node GPFS cluster to expel each other (it appears they experience a temporary network connection outage and lose contact with each other). At present it's not consistent which gets expelled by the cluster manager, and I wondered if there was any way to always force a specific node to be expelled in this situation? Thanks and best regards, Matt -------- Matt Thorpe | BDLSS Systems Administrator Bodleian Libraries Osney One Building, Osney Mead, Oxford, OX2 0EW matt.thorpe at bodleian.ox.ac.uk | 01865 (2)80027 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From matt.thorpe at bodleian.ox.ac.uk Tue Oct 25 14:06:12 2016 From: matt.thorpe at bodleian.ox.ac.uk (Matt Thorpe) Date: Tue, 25 Oct 2016 13:06:12 +0000 Subject: [gpfsug-discuss] Forcing which node gets expelled? In-Reply-To: <4403E9DB-88B4-4281-91B5-E171D1284899@nuance.com> References: <63906C4AF5FFD0429565A743F6AD5BEDDAF29F@MBX04.ad.oak.ox.ac.uk> <4403E9DB-88B4-4281-91B5-E171D1284899@nuance.com> Message-ID: <63906C4AF5FFD0429565A743F6AD5BEDDAF314@MBX04.ad.oak.ox.ac.uk> Hi Bob, That is exactly what I was after, thanks very much! Should buy us a little time so we can resolve our networking issue. Thanks again Matt. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Oesterlin, Robert Sent: 25 October 2016 13:23 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Forcing which node gets expelled? If you look at /usr/lpp/mmfs/samples/expelnode.sample you can use this as a base and install this in /var/mmfs/etc on the cluster manager. You can control which of the two nodes get expelled. We use it here to send an alert when a node is expelled. There is also "mmexpelnode" which you can force a particular node to be expelled. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: > on behalf of Matt Thorpe > Reply-To: gpfsug main discussion list > Date: Tuesday, October 25, 2016 at 7:05 AM To: gpfsug main discussion list > Subject: [EXTERNAL] [gpfsug-discuss] Forcing which node gets expelled? Hi, We are in the process of diagnosing a networking issue that is causing 2 of our 6 node GPFS cluster to expel each other (it appears they experience a temporary network connection outage and lose contact with each other). At present it's not consistent which gets expelled by the cluster manager, and I wondered if there was any way to always force a specific node to be expelled in this situation? Thanks and best regards, Matt -------- Matt Thorpe | BDLSS Systems Administrator Bodleian Libraries Osney One Building, Osney Mead, Oxford, OX2 0EW matt.thorpe at bodleian.ox.ac.uk | 01865 (2)80027 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Q_f6z64tvENxA9ac7rqWFj2jNd5IrpWEXcynJzHFjz4&s=jqso6xVB-V_zgLba-xjlWwiw3fNRan9NspsVq4PY4nA&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Tue Oct 25 16:33:16 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Tue, 25 Oct 2016 15:33:16 +0000 Subject: [gpfsug-discuss] October meet the devs report Message-ID: The October meet the devs workshop on cloud was last week. Thanks to Dean Hildebrand and John Lewars for flying in from the US to support us, Ulf Troppens and Dan Kidger also from IBM. And finally thanks to OCF for buying the pizza (one day we'll manage to get the pizza co to deliver on time!). The event report is now up on the group website at: http://www.spectrumscale.org/meet-the-devs-cloud-workshop-birmingham-uk/ Simon From Mark.Bush at siriuscom.com Tue Oct 25 19:46:26 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Tue, 25 Oct 2016 18:46:26 +0000 Subject: [gpfsug-discuss] Virtualized Spectrum Scale Message-ID: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> Anyone running SpectrumScale on Virtual Machines (intel)? I?m curious how you manage disks? Do you use RDM?s? Does this even make sense to do? If you have a 2-3 node cluster how do you share the disks across? Do you have VM?s with their own VMDK?s (if not RDM) in each node or is there some way to share access to the same VMDK?s? What are the advantages doing this other than existing HW use? Seems to me for a lab environment or very small nonperformance focused implementation this may be a viable option. Thanks Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From luis.bolinches at fi.ibm.com Tue Oct 25 19:58:21 2016 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Tue, 25 Oct 2016 18:58:21 +0000 Subject: [gpfsug-discuss] Virtualized Spectrum Scale In-Reply-To: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> Message-ID: Hi You must use RDM. Otherwise is not supported. SCSI commands is the reason. Furthermore on some versions I managed to crash the ESXi as well. -- Cheers > On 25 Oct 2016, at 19.46, "Mark.Bush at siriuscom.com" wrote: > > Anyone running SpectrumScale on Virtual Machines (intel)? I?m curious how you manage disks? Do you use RDM?s? Does this even make sense to do? If you have a 2-3 node cluster how do you share the disks across? Do you have VM?s with their own VMDK?s (if not RDM) in each node or is there some way to share access to the same VMDK?s? What are the advantages doing this other than existing HW use? Seems to me for a lab environment or very small nonperformance focused implementation this may be a viable option. > > Thanks > > Mark > This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. > > Sirius Computer Solutions Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: From kevindjo at us.ibm.com Tue Oct 25 20:05:12 2016 From: kevindjo at us.ibm.com (Kevin D Johnson) Date: Tue, 25 Oct 2016 19:05:12 +0000 Subject: [gpfsug-discuss] Virtualized Spectrum Scale In-Reply-To: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> References: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> Message-ID: An HTML attachment was scrubbed... URL: From Mark.Bush at siriuscom.com Tue Oct 25 20:34:29 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Tue, 25 Oct 2016 19:34:29 +0000 Subject: [gpfsug-discuss] Virtualized Spectrum Scale In-Reply-To: References: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> Message-ID: <5482A551-873C-4F9F-89BF-0CF0FB926D92@siriuscom.com> This is interesting since the SS FAQ leads me to believe that I have some options here. Does GPFS nodes with no direct disk access still mean RDMs? [cid:image001.png at 01D22ECC.E20EC2F0] From: on behalf of Luis Bolinches Reply-To: gpfsug main discussion list Date: Tuesday, October 25, 2016 at 1:58 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Virtualized Spectrum Scale Hi You must use RDM. Otherwise is not supported. SCSI commands is the reason. Furthermore on some versions I managed to crash the ESXi as well. -- Cheers On 25 Oct 2016, at 19.46, "Mark.Bush at siriuscom.com" > wrote: Anyone running SpectrumScale on Virtual Machines (intel)? I?m curious how you manage disks? Do you use RDM?s? Does this even make sense to do? If you have a 2-3 node cluster how do you share the disks across? Do you have VM?s with their own VMDK?s (if not RDM) in each node or is there some way to share access to the same VMDK?s? What are the advantages doing this other than existing HW use? Seems to me for a lab environment or very small nonperformance focused implementation this may be a viable option. Thanks Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 169073 bytes Desc: image001.png URL: From gsanjay at us.ibm.com Tue Oct 25 20:48:58 2016 From: gsanjay at us.ibm.com (Sanjay Gandhi) Date: Tue, 25 Oct 2016 12:48:58 -0700 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 57, Issue 66 In-Reply-To: References: Message-ID: VMDK is not supported due to http://kb.vmware.com/kb/2032940 It can cause GPFS deadlock due to hung IO. Thanks, Sanjay Gandhi GPFS FVT IBM, Beaverton Phone/FAX : 503-578-4141 T/L 775-4141 gsanjay at us.ibm.com From: gpfsug-discuss-request at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Date: 10/25/2016 12:35 PM Subject: gpfsug-discuss Digest, Vol 57, Issue 66 Sent by: gpfsug-discuss-bounces at spectrumscale.org Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Virtualized Spectrum Scale (Mark.Bush at siriuscom.com) ---------------------------------------------------------------------- Message: 1 Date: Tue, 25 Oct 2016 19:34:29 +0000 From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Virtualized Spectrum Scale Message-ID: <5482A551-873C-4F9F-89BF-0CF0FB926D92 at siriuscom.com> Content-Type: text/plain; charset="utf-8" This is interesting since the SS FAQ leads me to believe that I have some options here. Does GPFS nodes with no direct disk access still mean RDMs? [cid:image001.png at 01D22ECC.E20EC2F0] From: on behalf of Luis Bolinches Reply-To: gpfsug main discussion list Date: Tuesday, October 25, 2016 at 1:58 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Virtualized Spectrum Scale Hi You must use RDM. Otherwise is not supported. SCSI commands is the reason. Furthermore on some versions I managed to crash the ESXi as well. -- Cheers On 25 Oct 2016, at 19.46, "Mark.Bush at siriuscom.com< mailto:Mark.Bush at siriuscom.com>" > wrote: Anyone running SpectrumScale on Virtual Machines (intel)? I?m curious how you manage disks? Do you use RDM?s? Does this even make sense to do? If you have a 2-3 node cluster how do you share the disks across? Do you have VM?s with their own VMDK?s (if not RDM) in each node or is there some way to share access to the same VMDK?s? What are the advantages doing this other than existing HW use? Seems to me for a lab environment or very small nonperformance focused implementation this may be a viable option. Thanks Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161025/fc9a8bf7/attachment.html > -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 169073 bytes Desc: image001.png URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161025/fc9a8bf7/attachment.png > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 66 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From Mark.Bush at siriuscom.com Tue Oct 25 20:50:51 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Tue, 25 Oct 2016 19:50:51 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 57, Issue 66 In-Reply-To: References: Message-ID: <0A2BC512-5EB7-4531-91DD-3C862A89C2E2@siriuscom.com> Ok. I re-read the FAQ and I think the option in the Table is a bit misleading. I think it means it?s supported without direct disk access and that means GPFS client not an NSD. Sound right? From: on behalf of Sanjay Gandhi Reply-To: gpfsug main discussion list Date: Tuesday, October 25, 2016 at 2:48 PM To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 57, Issue 66 VMDK is not supported due to http://kb.vmware.com/kb/2032940 It can cause GPFS deadlock due to hung IO. Thanks, Sanjay Gandhi GPFS FVT IBM, Beaverton Phone/FAX : 503-578-4141 T/L 775-4141 gsanjay at us.ibm.com [nactive hide details for gpfsug-discuss-request---10/25/2016 12:35:04 PM-]gpfsug-discuss-request---10/25/2016 12:35:04 PM---Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org From: gpfsug-discuss-request at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Date: 10/25/2016 12:35 PM Subject: gpfsug-discuss Digest, Vol 57, Issue 66 Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Virtualized Spectrum Scale (Mark.Bush at siriuscom.com) ---------------------------------------------------------------------- Message: 1 Date: Tue, 25 Oct 2016 19:34:29 +0000 From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Virtualized Spectrum Scale Message-ID: <5482A551-873C-4F9F-89BF-0CF0FB926D92 at siriuscom.com> Content-Type: text/plain; charset="utf-8" This is interesting since the SS FAQ leads me to believe that I have some options here. Does GPFS nodes with no direct disk access still mean RDMs? [cid:image001.png at 01D22ECC.E20EC2F0] From: on behalf of Luis Bolinches Reply-To: gpfsug main discussion list Date: Tuesday, October 25, 2016 at 1:58 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Virtualized Spectrum Scale Hi You must use RDM. Otherwise is not supported. SCSI commands is the reason. Furthermore on some versions I managed to crash the ESXi as well. -- Cheers On 25 Oct 2016, at 19.46, "Mark.Bush at siriuscom.com" > wrote: Anyone running SpectrumScale on Virtual Machines (intel)? I?m curious how you manage disks? Do you use RDM?s? Does this even make sense to do? If you have a 2-3 node cluster how do you share the disks across? Do you have VM?s with their own VMDK?s (if not RDM) in each node or is there some way to share access to the same VMDK?s? What are the advantages doing this other than existing HW use? Seems to me for a lab environment or very small nonperformance focused implementation this may be a viable option. Thanks Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 169073 bytes Desc: image001.png URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 66 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 106 bytes Desc: image001.gif URL: From laurence at qsplace.co.uk Tue Oct 25 20:53:55 2016 From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow) Date: Tue, 25 Oct 2016 20:53:55 +0100 Subject: [gpfsug-discuss] Virtualized Spectrum Scale In-Reply-To: References: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> Message-ID: Kevin, This is how I run test systems, I let iovirt devices to be attached to multiple kvm systems. It works well. -- Lauz On 25 October 2016 20:05:12 BST, Kevin D Johnson wrote: >Mark, > > > >You can run Spectrum Scale with virtual machines. As long as the >virtual disks present to the file system as devices, you should be good >to go (for example, "cat /proc/partitions" should show your virtual >disks as devices). I typically use scsi raw devices with virtual >machines and that seems to work well. KVM allows you to share the >disks as well between hosts and that's important to emulate more >production level uses. It is good for a lab or to try something >quickly without provisioning actual machines. We have sometimes used >SS with virtual machines in production but we typically recommend bare >metal if/when possible. > > > >Kevin D. Johnson, MBA, MAFM >Spectrum Computing, Senior Managing Consultant > >IBM Certified Deployment Professional - Spectrum Scale V4.1.1 >IBM Certified Deployment Professional - Cloud Object Storage V3.8 > >IBM Certified Solution Advisor - Spectrum Computing V1 > > > >720.349.6199 - kevindjo at us.ibm.com > > > > > > > >----- Original message ----- >From: "Mark.Bush at siriuscom.com" >Sent by: gpfsug-discuss-bounces at spectrumscale.org >To: "gpfsug-discuss at spectrumscale.org" > >Cc: >Subject: [gpfsug-discuss] Virtualized Spectrum Scale >Date: Tue, Oct 25, 2016 2:47 PM > > >Anyone running SpectrumScale on Virtual Machines (intel)? I?m curious >how you manage disks? Do you use RDM?s? Does this even make sense to >do? If you have a 2-3 node cluster how do you share the disks across? >Do you have VM?s with their own VMDK?s (if not RDM) in each node or is >there some way to share access to the same VMDK?s? What are the >advantages doing this other than existing HW use? Seems to me for a >lab environment or very small nonperformance focused implementation >this may be a viable option. > > > >Thanks > > > >Mark > >This message (including any attachments) is intended only for the use >of the individual or entity to which it is addressed and may contain >information that is non-public, proprietary, privileged, confidential, >and exempt from disclosure under applicable law. If you are not the >intended recipient, you are hereby notified that any use, >dissemination, distribution, or copying of this communication is >strictly prohibited. This message may be viewed by parties at Sirius >Computer Solutions other than those named in the message header. This >message does not contain an official representation of Sirius Computer >Solutions. If you have received this communication in error, notify >Sirius Computer Solutions immediately and (i) destroy this message if a >facsimile or (ii) delete this message immediately if this is an >electronic communication. Thank you. > >Sirius Computer Solutions > > > >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > >------------------------------------------------------------------------ > >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aspalazz at us.ibm.com Tue Oct 25 20:59:31 2016 From: aspalazz at us.ibm.com (Aaron S Palazzolo) Date: Tue, 25 Oct 2016 19:59:31 +0000 Subject: [gpfsug-discuss] Virtualized Spectrum Scale In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From kevindjo at us.ibm.com Tue Oct 25 21:02:42 2016 From: kevindjo at us.ibm.com (Kevin D Johnson) Date: Tue, 25 Oct 2016 20:02:42 +0000 Subject: [gpfsug-discuss] Virtualized Spectrum Scale In-Reply-To: References: , <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> Message-ID: An HTML attachment was scrubbed... URL: From Mark.Bush at siriuscom.com Tue Oct 25 21:36:51 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Tue, 25 Oct 2016 20:36:51 +0000 Subject: [gpfsug-discuss] Virtualized Spectrum Scale In-Reply-To: References: Message-ID: <82AD0334-B515-48FD-8C30-FC8F309AD00F@siriuscom.com> Excellent info Aaron. Thanks for this. I may reach out via PM to explore this more with you. From: on behalf of Aaron S Palazzolo Reply-To: gpfsug main discussion list Date: Tuesday, October 25, 2016 at 2:59 PM To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] Virtualized Spectrum Scale Hi Mark, Great questions on the VM side. Within our Spectrum Scale Test labs at IBM, we run within both VMware and KVM. It sounds as though your questions are more towards the VMware side so I'll give a few pointers. For more detailed info, feel free to grab me on the side or continue the discussion within the user group so that others can learn from this as well. #1) As Luis suggests, using vmdks will not give you full support of the scsi protocol. Specifically persistent reserve. EIO errors are also not passed correctly in some path down situations. - For certain test/dev environments, in which data protection and performance are not high priorities, then this may be fine. Some of our test environments do run like this. - Note that vmdks can be shared among virtual Spectrum Scale nodes using the multi-writer flag in VMware. To do this, you'll first need to create your vmdks using 'Thick Provision Eager Zeroed'. You'll then need to configure the multi-writer flag for scsi-sharing such as this, for each vmdk: scsi1:0.sharing = "multi-writer" scsi1:1.sharing = "multi-writer" scsi1:2.sharing = "multi-writer" You'll find this in the Advanced configuration parameters. Finally, you'll need to set all of these vmdks to a separate scsi adapter which is configured for either virtual or physical sharing. Downsides are lack of support, some degradation of performance, lack of failover support, and inability to use VMware snapshots of VMs with scsi sharing/multi-writer enabled. Upsides are ease of setup both with virtually and physically and ability to store all VMs on a single datastore that can itself be snapshotted if the underlying physical storage supports this. In our testlabs, we create 4node -> 50node virtual Spectrum Scale clusters, each node is a VM, some of these VMs have extra vmdks for NSDs and some do not. All VMs belonging to a cluster reside on a single datastore which ends up being a single XIV Volume. We then can snapshot this XIV volume and in essence, snapshot the entire Spectrum Scale cluster back and forth in time. I'm sure this is overly complicated for what you want to do, but it may get you thinking of use cases. #2) RDM will give you both performance and the piece of mind that you're using a fully supported config. Officially, I will always recommend RDM due to this. The downside to RDM is complexity in setup unless you have a fairly static config or unless you can automate the zoning. #3) No matter what type of VM infrastructure you use, make sure you investigate the memory/cpu requirements. - Check our FAQ, specifically 6.1 for tuning info regarding vm.min_free_kbytes: https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#gpfsclustersfaqAugust2016-gen4__lintunq - We have run some of our test clusters with as little as 4GB of memory but I would definitely recommend quite a bit more memory in each VM for production use. If you use additional functions other than core file system, pay attention to the memory requirements of these. #4) KVM is a viable alternative if needed..... Regards, Aaron Palazzolo IBM Spectrum Scale Deployment, Infrastructure, Virtualization 9042 S Rita Road, Tucson AZ 85744 Phone: 520-799-5161, T/L: 321-5161 E-mail: aspalazz at us.ibm.com ----- Original message ----- From: gpfsug-discuss-request at spectrumscale.org Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Cc: Subject: gpfsug-discuss Digest, Vol 57, Issue 65 Date: Tue, Oct 25, 2016 12:05 PM Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. October meet the devs report (Simon Thompson (Research Computing - IT Services)) 2. Virtualized Spectrum Scale (Mark.Bush at siriuscom.com) 3. Re: Virtualized Spectrum Scale (Luis Bolinches) 4. Re: Virtualized Spectrum Scale (Kevin D Johnson) ---------------------------------------------------------------------- Message: 1 Date: Tue, 25 Oct 2016 15:33:16 +0000 From: "Simon Thompson (Research Computing - IT Services)" To: "gpfsug-discuss at spectrumscale.org" Subject: [gpfsug-discuss] October meet the devs report Message-ID: Content-Type: text/plain; charset="us-ascii" The October meet the devs workshop on cloud was last week. Thanks to Dean Hildebrand and John Lewars for flying in from the US to support us, Ulf Troppens and Dan Kidger also from IBM. And finally thanks to OCF for buying the pizza (one day we'll manage to get the pizza co to deliver on time!). The event report is now up on the group website at: http://www.spectrumscale.org/meet-the-devs-cloud-workshop-birmingham-uk/ Simon ------------------------------ Message: 2 Date: Tue, 25 Oct 2016 18:46:26 +0000 From: "Mark.Bush at siriuscom.com" To: "gpfsug-discuss at spectrumscale.org" Subject: [gpfsug-discuss] Virtualized Spectrum Scale Message-ID: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD at siriuscom.com> Content-Type: text/plain; charset="utf-8" Anyone running SpectrumScale on Virtual Machines (intel)? I?m curious how you manage disks? Do you use RDM?s? Does this even make sense to do? If you have a 2-3 node cluster how do you share the disks across? Do you have VM?s with their own VMDK?s (if not RDM) in each node or is there some way to share access to the same VMDK?s? What are the advantages doing this other than existing HW use? Seems to me for a lab environment or very small nonperformance focused implementation this may be a viable option. Thanks Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 3 Date: Tue, 25 Oct 2016 18:58:21 +0000 From: "Luis Bolinches" To: "gpfsug main discussion list" Subject: Re: [gpfsug-discuss] Virtualized Spectrum Scale Message-ID: Content-Type: text/plain; charset="utf-8" Hi You must use RDM. Otherwise is not supported. SCSI commands is the reason. Furthermore on some versions I managed to crash the ESXi as well. -- Cheers > On 25 Oct 2016, at 19.46, "Mark.Bush at siriuscom.com" wrote: > > Anyone running SpectrumScale on Virtual Machines (intel)? I?m curious how you manage disks? Do you use RDM?s? Does this even make sense to do? If you have a 2-3 node cluster how do you share the disks across? Do you have VM?s with their own VMDK?s (if not RDM) in each node or is there some way to share access to the same VMDK?s? What are the advantages doing this other than existing HW use? Seems to me for a lab environment or very small nonperformance focused implementation this may be a viable option. > > Thanks > > Mark > This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. > > Sirius Computer Solutions Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 4 Date: Tue, 25 Oct 2016 19:05:12 +0000 From: "Kevin D Johnson" To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Virtualized Spectrum Scale Message-ID: Content-Type: text/plain; charset="us-ascii" An HTML attachment was scrubbed... URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 65 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: From leslie.james.elliott at gmail.com Tue Oct 25 23:01:25 2016 From: leslie.james.elliott at gmail.com (leslie elliott) Date: Wed, 26 Oct 2016 08:01:25 +1000 Subject: [gpfsug-discuss] unified file and object Message-ID: Hi We are in the process of trying to configure unified file and object storage in unified_mode and have a few problems We are running 4.2.1 and do not have any current issues the file access protocols setting up the authentication First issue we do have is binding the object data to our Active Directory, we seem to be hitting a road block due to the fact that the bind DN has spaces in it, if we enclose the DN in quotes it still fails, if we escape them with the appropriate RFC value we can get the mmuserauth to complete but the lookups from the local keystone fail for the authentication of the users The DN for the swift user and swift admin also have quotes in them, so just doing it on the command line is not enough to get the mmuserauth command to complete Second problem is OBJ:openstack-swift-object-sof is not running This seems to be due to the config file not having bind_ip and bind_port values, if these are added then the error turns to pipeline of other settings in the config file missing This particular issue occurs no matter what the auth type is set to be for object Hopefully this make some sense to someone Thanks leslie Leslie Elliott, Infrastructure Support Specialist, Faculty Infrastructure and Applications Support Information Technology Services, The University of Queensland -------------- next part -------------- An HTML attachment was scrubbed... URL: From Greg.Lehmann at csiro.au Wed Oct 26 01:51:16 2016 From: Greg.Lehmann at csiro.au (Greg.Lehmann at csiro.au) Date: Wed, 26 Oct 2016 00:51:16 +0000 Subject: [gpfsug-discuss] Forcing which node gets expelled? In-Reply-To: <63906C4AF5FFD0429565A743F6AD5BEDDAF314@MBX04.ad.oak.ox.ac.uk> References: <63906C4AF5FFD0429565A743F6AD5BEDDAF29F@MBX04.ad.oak.ox.ac.uk> <4403E9DB-88B4-4281-91B5-E171D1284899@nuance.com> <63906C4AF5FFD0429565A743F6AD5BEDDAF314@MBX04.ad.oak.ox.ac.uk> Message-ID: We hit something like this due to a bug in gskit. We all thought it was networking at first and it took me a fair bit of time to check all that. We have 7 nsd servers and around 400 clients running 4.2.0.4. We are just trying a workaround now that looks promising. The bug will be fixed at some point. Cheers, Greg From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Thorpe Sent: Tuesday, 25 October 2016 11:06 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Forcing which node gets expelled? Hi Bob, That is exactly what I was after, thanks very much! Should buy us a little time so we can resolve our networking issue. Thanks again Matt. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Oesterlin, Robert Sent: 25 October 2016 13:23 To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Forcing which node gets expelled? If you look at /usr/lpp/mmfs/samples/expelnode.sample you can use this as a base and install this in /var/mmfs/etc on the cluster manager. You can control which of the two nodes get expelled. We use it here to send an alert when a node is expelled. There is also "mmexpelnode" which you can force a particular node to be expelled. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: > on behalf of Matt Thorpe > Reply-To: gpfsug main discussion list > Date: Tuesday, October 25, 2016 at 7:05 AM To: gpfsug main discussion list > Subject: [EXTERNAL] [gpfsug-discuss] Forcing which node gets expelled? Hi, We are in the process of diagnosing a networking issue that is causing 2 of our 6 node GPFS cluster to expel each other (it appears they experience a temporary network connection outage and lose contact with each other). At present it's not consistent which gets expelled by the cluster manager, and I wondered if there was any way to always force a specific node to be expelled in this situation? Thanks and best regards, Matt -------- Matt Thorpe | BDLSS Systems Administrator Bodleian Libraries Osney One Building, Osney Mead, Oxford, OX2 0EW matt.thorpe at bodleian.ox.ac.uk | 01865 (2)80027 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Q_f6z64tvENxA9ac7rqWFj2jNd5IrpWEXcynJzHFjz4&s=jqso6xVB-V_zgLba-xjlWwiw3fNRan9NspsVq4PY4nA&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From Greg.Lehmann at csiro.au Wed Oct 26 02:04:35 2016 From: Greg.Lehmann at csiro.au (Greg.Lehmann at csiro.au) Date: Wed, 26 Oct 2016 01:04:35 +0000 Subject: [gpfsug-discuss] Virtualized Spectrum Scale In-Reply-To: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> References: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> Message-ID: <72fda8bf40634ad38e72689756a444cf@exch1-cdc.nexus.csiro.au> We use KVM running on a Debian host, with CentOS guests. Storage is zoned from our DDN Infiniband array to the host and then passed through to the guests. We would like to zone it directly to the guests SRIOV IB HCA, but srp seems to be a bit of dead code tree. We had to do a bit of work to get it working with Debian and haven?t as yet spent the time on getting it going with CentOS. We also run Spectrum Archive on the guest with tape drives and libraries zoned to the guest?s PCIe HBAs which are passed through from the host. We are working towards going production with this setup. Xen was a bit a failure for us so we switched to KVM. Cheers, Greg From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Mark.Bush at siriuscom.com Sent: Wednesday, 26 October 2016 4:46 AM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Virtualized Spectrum Scale Anyone running SpectrumScale on Virtual Machines (intel)? I?m curious how you manage disks? Do you use RDM?s? Does this even make sense to do? If you have a 2-3 node cluster how do you share the disks across? Do you have VM?s with their own VMDK?s (if not RDM) in each node or is there some way to share access to the same VMDK?s? What are the advantages doing this other than existing HW use? Seems to me for a lab environment or very small nonperformance focused implementation this may be a viable option. Thanks Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Wed Oct 26 03:48:38 2016 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 25 Oct 2016 22:48:38 -0400 Subject: [gpfsug-discuss] Virtualized Spectrum Scale In-Reply-To: <72fda8bf40634ad38e72689756a444cf@exch1-cdc.nexus.csiro.au> References: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> <72fda8bf40634ad38e72689756a444cf@exch1-cdc.nexus.csiro.au> Message-ID: <2e759489-a38f-58d2-4e20-b9c090c21b4a@nasa.gov> Hi Greg, I'm rather curious about your srp difficulties (not to get too off topic). Is it an issue with srp by itself or the interaction of srp with the SRIOV IB HCA? We've used SRP quite a bit both virtualized and not and have seen good results both in terms of stability and performance. -Aaron On 10/25/16 9:04 PM, Greg.Lehmann at csiro.au wrote: > We use KVM running on a Debian host, with CentOS guests. Storage is > zoned from our DDN Infiniband array to the host and then passed through > to the guests. We would like to zone it directly to the guests SRIOV IB > HCA, but srp seems to be a bit of dead code tree. We had to do a bit of > work to get it working with Debian and haven?t as yet spent the time on > getting it going with CentOS. > > > > We also run Spectrum Archive on the guest with tape drives and libraries > zoned to the guest?s PCIe HBAs which are passed through from the host. > We are working towards going production with this setup. > > > > Xen was a bit a failure for us so we switched to KVM. > > > > Cheers, > > > > Greg > > > > *From:*gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of > *Mark.Bush at siriuscom.com > *Sent:* Wednesday, 26 October 2016 4:46 AM > *To:* gpfsug-discuss at spectrumscale.org > *Subject:* [gpfsug-discuss] Virtualized Spectrum Scale > > > > Anyone running SpectrumScale on Virtual Machines (intel)? I?m curious > how you manage disks? Do you use RDM?s? Does this even make sense to > do? If you have a 2-3 node cluster how do you share the disks across? > Do you have VM?s with their own VMDK?s (if not RDM) in each node or is > there some way to share access to the same VMDK?s? What are the > advantages doing this other than existing HW use? Seems to me for a lab > environment or very small nonperformance focused implementation this may > be a viable option. > > > > Thanks > > > > Mark > > This message (including any attachments) is intended only for the use of > the individual or entity to which it is addressed and may contain > information that is non-public, proprietary, privileged, confidential, > and exempt from disclosure under applicable law. If you are not the > intended recipient, you are hereby notified that any use, dissemination, > distribution, or copying of this communication is strictly prohibited. > This message may be viewed by parties at Sirius Computer Solutions other > than those named in the message header. This message does not contain an > official representation of Sirius Computer Solutions. If you have > received this communication in error, notify Sirius Computer Solutions > immediately and (i) destroy this message if a facsimile or (ii) delete > this message immediately if this is an electronic communication. Thank you. > > *Sirius Computer Solutions * > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From Greg.Lehmann at csiro.au Wed Oct 26 04:07:19 2016 From: Greg.Lehmann at csiro.au (Greg.Lehmann at csiro.au) Date: Wed, 26 Oct 2016 03:07:19 +0000 Subject: [gpfsug-discuss] Virtualized Spectrum Scale In-Reply-To: <2e759489-a38f-58d2-4e20-b9c090c21b4a@nasa.gov> References: <1C767B09-3E62-4A4D-814C-FCFC7321A8FD@siriuscom.com> <72fda8bf40634ad38e72689756a444cf@exch1-cdc.nexus.csiro.au> <2e759489-a38f-58d2-4e20-b9c090c21b4a@nasa.gov> Message-ID: The srp work was done a few years ago now. We use the same srp code for both physical and virtual, so I am guessing it has nothing to do with the SRIOV side of things. Somebody else did the work, so I will try and get an answer for you. I agree performance and stability is good with physical and virtual. Cheers, Greg -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aaron Knister Sent: Wednesday, 26 October 2016 12:49 PM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Virtualized Spectrum Scale Hi Greg, I'm rather curious about your srp difficulties (not to get too off topic). Is it an issue with srp by itself or the interaction of srp with the SRIOV IB HCA? We've used SRP quite a bit both virtualized and not and have seen good results both in terms of stability and performance. -Aaron On 10/25/16 9:04 PM, Greg.Lehmann at csiro.au wrote: > We use KVM running on a Debian host, with CentOS guests. Storage is > zoned from our DDN Infiniband array to the host and then passed > through to the guests. We would like to zone it directly to the guests > SRIOV IB HCA, but srp seems to be a bit of dead code tree. We had to > do a bit of work to get it working with Debian and haven't as yet > spent the time on getting it going with CentOS. > > > > We also run Spectrum Archive on the guest with tape drives and > libraries zoned to the guest's PCIe HBAs which are passed through from the host. > We are working towards going production with this setup. > > > > Xen was a bit a failure for us so we switched to KVM. > > > > Cheers, > > > > Greg > > > > *From:*gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of > *Mark.Bush at siriuscom.com > *Sent:* Wednesday, 26 October 2016 4:46 AM > *To:* gpfsug-discuss at spectrumscale.org > *Subject:* [gpfsug-discuss] Virtualized Spectrum Scale > > > > Anyone running SpectrumScale on Virtual Machines (intel)? I'm curious > how you manage disks? Do you use RDM's? Does this even make sense to > do? If you have a 2-3 node cluster how do you share the disks across? > Do you have VM's with their own VMDK's (if not RDM) in each node or is > there some way to share access to the same VMDK's? What are the > advantages doing this other than existing HW use? Seems to me for a > lab environment or very small nonperformance focused implementation > this may be a viable option. > > > > Thanks > > > > Mark > > This message (including any attachments) is intended only for the use > of the individual or entity to which it is addressed and may contain > information that is non-public, proprietary, privileged, confidential, > and exempt from disclosure under applicable law. If you are not the > intended recipient, you are hereby notified that any use, > dissemination, distribution, or copying of this communication is strictly prohibited. > This message may be viewed by parties at Sirius Computer Solutions > other than those named in the message header. This message does not > contain an official representation of Sirius Computer Solutions. If > you have received this communication in error, notify Sirius Computer > Solutions immediately and (i) destroy this message if a facsimile or > (ii) delete this message immediately if this is an electronic communication. Thank you. > > *Sirius Computer Solutions * > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From janfrode at tanso.net Wed Oct 26 12:53:21 2016 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Wed, 26 Oct 2016 13:53:21 +0200 Subject: [gpfsug-discuss] filesystem thresholds in gui alerting Message-ID: Does anybody know if there are any way to define what thresholds are to be used for alterting in the GUI? F.ex. we have some filesystems that are very full, but won't be getting any more data added.. we'd like to turn off monitoring of these, are raise the threshold to allow them to be ~100% full. -jf -------------- next part -------------- An HTML attachment was scrubbed... URL: From stschmid at de.ibm.com Wed Oct 26 16:14:27 2016 From: stschmid at de.ibm.com (Stefan Schmidt) Date: Wed, 26 Oct 2016 17:14:27 +0200 Subject: [gpfsug-discuss] filesystem thresholds in gui alerting In-Reply-To: References: Message-ID: Hi Jan-Frode Myklebust, The current GUI-internal logic has a hard coded warning level of 80% and an hard-coded error level of 90%. With 4.2.2 the system health monitoring will have thresholds set and monitored via ZIMON and the system health component will provide events. With 4.2.2 those can be changed with an mm command. In general it is not a good idea to let a filesystem get loaded 100%. It could be, if this is done with the filesystem used by the system to store some configuration data, that the system can run into an error (usually this is the filesystem we call gpfs0). I suggest you get the 4.2.2 version which will GA quite soon. I have to apologize that I'm not allowed to talk about GA dates. If you want to get this feature earlier you may contact IBM and ask for participation on the beta program which starts very soon but the beta code is usually limited to non productive systems. As said , ask your IBM contact for the 4.2.2 GA update ( hopefully GA is soon) and you can get this feature you are looking for. Mit freundlichen Gr??en / Kind regards Stefan Schmidt Scrum Master IBM Spectrum Scale GUI / Senior IT Architect/PMP?-Dept. M069 / IBM Spectrum Scale Software Development IBM Systems Group IBM Deutschland Phone: +49-703 4274 1966 IBM Deutschland Am Weiher 24 E-Mail: stschmid at de.ibm.com 65421 Kelsterbach Germany IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: Jan-Frode Myklebust To: gpfsug main discussion list Date: 26.10.2016 13:53 Subject: [gpfsug-discuss] filesystem thresholds in gui alerting Sent by: gpfsug-discuss-bounces at spectrumscale.org Does anybody know if there are any way to define what thresholds are to be used for alterting in the GUI? F.ex. we have some filesystems that are very full, but won't be getting any more data added.. we'd like to turn off monitoring of these, are raise the threshold to allow them to be ~100% full. -jf_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulmer at ulmer.org Thu Oct 27 01:48:50 2016 From: ulmer at ulmer.org (Stephen Ulmer) Date: Wed, 26 Oct 2016 20:48:50 -0400 Subject: [gpfsug-discuss] filesystem thresholds in gui alerting In-Reply-To: References: Message-ID: <76513F28-0646-4282-A595-C69A9F32A945@ulmer.org> It seems prudent to note here that a file system that reports 100% full can have quite a bit of free space. Take for example a 10TB file system that is no longer written to ? no one wants to use 100GB just to make the alert color green. This gets silly very quickly at even ?medium? sizes... What would be really cool would be to decide how much reserved space you want, look at the velocity[1] with which the filesystem is filling, and then alert when the time until left until full is less than an operations shift (or something useful). Maybe that is just silly. Hey, Jan-Frode, can you mount the no-more-data file systems read only? Maybe not alerting on the full-ness of read only filesystems is easier and more sane? I mean, they can?t fill up any more. Liberty, -- Stephen > On Oct 26, 2016, at 11:14 AM, Stefan Schmidt wrote: > > Hi Jan-Frode Myklebust, > > The current GUI-internal logic has a hard coded warning level of 80% and an hard-coded error level of 90%. > > With 4.2.2 the system health monitoring will have thresholds set and monitored via ZIMON and the system health > component will provide events.With 4.2.2 those can be changed with an mm command. > > In general it is not a good idea to let a filesystem get loaded 100%. It could be, if this is done with the filesystem used by the system to store some configuration data, that the system can run into an error (usually this is the filesystem we call gpfs0). > > I suggest you get the 4.2.2 version which will GA quite soon. I have to apologize that I'm not allowed to talk about GA dates. If you want to get this feature earlier you may contact IBM and ask for participation on the beta program which starts very soon but the beta code is usually limited to non productive systems. > > As said , ask your IBM contact for the 4.2.2 GA update ( hopefully GA is soon) and you can get this feature you are looking for. > Mit freundlichen Gr??en / Kind regards > > > Stefan Schmidt > > > Scrum Master IBM Spectrum Scale GUI / Senior IT Architect/PMP?-Dept. M069 / IBM Spectrum Scale Software Development > > IBM Systems Group > > IBM Deutschland > > Phone: +49-703 4274 1966 IBM Deutschland > Am Weiher 24 > E-Mail: stschmid at de.ibm.com 65421 Kelsterbach > Germany > IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Martina Koederitz > Gesch?ftsf?hrung: Dirk Wittkopp > Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 > > > > > > > > From: Jan-Frode Myklebust > To: gpfsug main discussion list > Date: 26.10.2016 13:53 > Subject: [gpfsug-discuss] filesystem thresholds in gui alerting > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > > > Does anybody know if there are any way to define what thresholds are to be used for alterting in the GUI? F.ex. we have some filesystems that are very full, but won't be getting any more data added.. we'd like to turn off monitoring of these, are raise the threshold to allow them to be ~100% full. > > > > -jf_______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From usa-principal at gpfsug.org Thu Oct 27 02:06:06 2016 From: usa-principal at gpfsug.org (Spectrum Scale Users Group - USA Principal Kristy Kallback-Rose) Date: Wed, 26 Oct 2016 21:06:06 -0400 Subject: [gpfsug-discuss] [UPDATE] SC16 Draft Agenda [Save the date: Sunday, November 13th] In-Reply-To: References: Message-ID: <10BF7862-5EB0-412E-9B1D-9531D89B87C2@gpfsug.org> All, Here is where we?re at with the agenda. Still finalizing a few things, but wanted to keep you all in the loop. Registration Web Site [Register for the SC16 UG event here]: https://www-01.ibm.com/events/wwe/grp/grp305.nsf/Agenda.xsp?openform&seminar=357M7UES&locale=en_US&S_TACT=000001CP&S_OFF_CD=10001235 IBM summary page: http://www-03.ibm.com/systems/technicalcomputing/supercomputing/index.html Location: Salt Lake City Downtown Marriott [Room Number TBD] 75 SW Temple Street, Salt Lake City UT 84101 Draft Agenda: 12:30 12:40 Welcome 12:40 12:50 Overview 12:50 13:35 The latest in IBM Spectrum Scale and ESS 13:55 14:20 NASA Goddard: Big Data in HPC 14:20 14:45 Nuance: Tiering to Object Storage 14:45 15:15 - break - 15:15 15:30 Sponsor Talk: TOPIC TBD (NetApp) 13:35 13:55 Virginia Tech ARC: SANDisk IF150 (IBM DeepFlash 150) 15:30 15:50 NASA Goddard: Monitoring with Grafana and InfluxDB 15:50 16:35 Spectrum Scale Enhancements for CORAL 16:35 17:20 News from IBM Research 17:20 17:30 Closing [Reception to follow closing, details TBD] Questions? Comments? Let us know. Hope to see many of you next month. Final agenda will be made available before the event. Cheers, Kristy & Bob Kristy Kallback-Rose Manager, Research Storage Indiana University > On Oct 10, 2016, at 6:59 PM, GPFS UG USA Principal > wrote: > > Hello all, > > There have been some questions about the Spectrum Scale Users Group event for SC16 and we thought it best to publish our draft agenda and continue to fill it in as details get settled. That way you can make your travel plans and schedules. > > The date and rough times are set, we may adjust the time range a bit depending upon the number of user talks that people would like to give. So note, yes! we are asking for user talks. Please consider doing this. We always receive positive feedback about talks given by users that describe real world experiences. If any of the user talk slots below are of interest to you, please let us know as soon as you can. We may turn at least one user talk into a panel if we can get enough participants. > > As always, your feedback is welcome. This is a users group after all. > > So, here?s what we have planned so far: > > Date: Sunday November 13th > Venue: TBD, but near the convention center in downtown SLC > Time: ~12:30p - ~17:30p > > Agenda: > 12:30 - 12:50 Welcome / Overview > 12:50 - 13:50 The latest in IBM Spectrum Scale and ESS > 13:50 - 14:20 User Talk: Big Data in HPC > 14:20 - 14:50 User Talk: Tiering to Object Storage > 14:50 - 15:20 - break - > 15:20 - 15:50 User Talk: TBD ? volunteers! please! > 15:50 - 16:35 Spectrum Scale Enhancements for CORAL > 16:35 - 17:20 News from IBM Research > 17:20 - 17:30 Closing > > Best, > > Kristy & Bob > > Kristy Kallback-Rose > Manager, Research Storage > Indiana University > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Ola.Pontusson at kb.se Thu Oct 27 07:49:24 2016 From: Ola.Pontusson at kb.se (Ola Pontusson) Date: Thu, 27 Oct 2016 06:49:24 +0000 Subject: [gpfsug-discuss] filesystem thresholds in gui alerting In-Reply-To: <76513F28-0646-4282-A595-C69A9F32A945@ulmer.org> References: <76513F28-0646-4282-A595-C69A9F32A945@ulmer.org> Message-ID: Hi I am really looking forward to the 4.2.2 where I hopefully can adjust the thresholds of alarm levels. We just as Jan-Frode have filesystems that is not growing so much and waste a large amount of disk just to keep it green is never going to happend. Below is an example of one of our filesystems. /dev/smdb04 942T 900T 42T 96% /gpfs/smdb04 /Ola Fr?n: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] F?r Stephen Ulmer Skickat: den 27 oktober 2016 02:49 Till: gpfsug main discussion list ?mne: Re: [gpfsug-discuss] filesystem thresholds in gui alerting It seems prudent to note here that a file system that reports 100% full can have quite a bit of free space. Take for example a 10TB file system that is no longer written to ? no one wants to use 100GB just to make the alert color green. This gets silly very quickly at even ?medium? sizes... What would be really cool would be to decide how much reserved space you want, look at the velocity[1] with which the filesystem is filling, and then alert when the time until left until full is less than an operations shift (or something useful). Maybe that is just silly. Hey, Jan-Frode, can you mount the no-more-data file systems read only? Maybe not alerting on the full-ness of read only filesystems is easier and more sane? I mean, they can?t fill up any more. Liberty, -- Stephen On Oct 26, 2016, at 11:14 AM, Stefan Schmidt > wrote: Hi Jan-Frode Myklebust, The current GUI-internal logic has a hard coded warning level of 80% and an hard-coded error level of 90%. With 4.2.2 the system health monitoring will have thresholds set and monitored via ZIMON and the system health component will provide events.With 4.2.2 those can be changed with an mm command. In general it is not a good idea to let a filesystem get loaded 100%. It could be, if this is done with the filesystem used by the system to store some configuration data, that the system can run into an error (usually this is the filesystem we call gpfs0). I suggest you get the 4.2.2 version which will GA quite soon. I have to apologize that I'm not allowed to talk about GA dates. If you want to get this feature earlier you may contact IBM and ask for participation on the beta program which starts very soon but the beta code is usually limited to non productive systems. As said , ask your IBM contact for the 4.2.2 GA update ( hopefully GA is soon) and you can get this feature you are looking for. Mit freundlichen Gr??en / Kind regards Stefan Schmidt Scrum Master IBM Spectrum Scale GUI / Senior IT Architect/PMP?-Dept. M069 / IBM Spectrum Scale Software Development IBM Systems Group IBM Deutschland ________________________________ Phone: +49-703 4274 1966 IBM Deutschland Am Weiher 24 E-Mail: stschmid at de.ibm.com 65421 Kelsterbach Germany ________________________________ IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: Jan-Frode Myklebust > To: gpfsug main discussion list > Date: 26.10.2016 13:53 Subject: [gpfsug-discuss] filesystem thresholds in gui alerting Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Does anybody know if there are any way to define what thresholds are to be used for alterting in the GUI? F.ex. we have some filesystems that are very full, but won't be getting any more data added.. we'd like to turn off monitoring of these, are raise the threshold to allow them to be ~100% full. -jf_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From stijn.deweirdt at ugent.be Thu Oct 27 18:27:44 2016 From: stijn.deweirdt at ugent.be (Stijn De Weirdt) Date: Thu, 27 Oct 2016 19:27:44 +0200 Subject: [gpfsug-discuss] workaround gpfs 4.2.1-0 rpm issue Message-ID: <86d24598-7af8-38f0-fe80-998294c7bef9@ugent.be> hi all, gpfs.base 4.2.1-0 rpm has following postuninstall snippet it will disable the gpfs unit always (when previously enabled), whether this is a removal or an upgrade. this however prevents an update to 4.2.1-1, as it will remove the unit that is added during install (post has 'systemctl reenable /usr/lpp/mmfs/lib/systemd/gpfs.service') so after the upgrade, we are left with nodes that have no gpfs service unit (and thus no gpfs). it would have been better if the rpm symlinked teh service to the /usr/lib/systemd/... units, and enabled/disabled it. i'll probably rpmrebuild the 4.2.1-1 rpms to make a more permanent unit in a proper system location. other tips are welcome. stijn > postuninstall scriptlet (using /bin/sh): > if test -n "$DEBUG" || test -n "$DEBUGpostun"; then > set -x > fi > packageCnt=$1 > debian_release="/etc/debian_version" > > # set the system utilities if they are in different path on different systems > if [ -f "$debian_release" ] > then > AWK=/usr/bin/awk > else > AWK=/bin/awk > fi > > if /usr/bin/systemctl -q is-enabled gpfs.service 2>/dev/null > then > /usr/bin/systemctl -q disable gpfs.service > fi > From douglasof at us.ibm.com Fri Oct 28 14:37:53 2016 From: douglasof at us.ibm.com (Douglas O'flaherty) Date: Fri, 28 Oct 2016 09:37:53 -0400 Subject: [gpfsug-discuss] SC16 Registration Message-ID: Greetings: If you haven't yet registered, planned a 1:1 meeting, or interested in a specific topic ... Here you go https://www-01.ibm.com/events/wwe/grp/grp305.nsf/Registration.xsp?openform&seminar=357M7UES&locale=en_US&auth=anonymous Sign up so we can plan the cocktail hour on Sunday evening -- Thanks to NetApp for sponsoring again. doug IBM Spectrum Storage PMM -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mark.Bush at siriuscom.com Fri Oct 28 14:52:47 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Fri, 28 Oct 2016 13:52:47 +0000 Subject: [gpfsug-discuss] GUI and pmsensors/pmcollector Message-ID: <9566FC70-85D3-4D9B-B28B-58C4535B8972@siriuscom.com> I am able to do everything I need just fine when I have a single node cluster and both pmsensors and pmcollector are installed all on the same node. When I try and get pmsensors on my NSD nodes and a separate pmcollector elsewhere I never can get anything to show up in the graphs. I?ve configured the pmsensors with mmperfmon config generate ?collectors=(my node) and I still get nothing. The logs show that I get connections but the GUI never show and I?m unable to get mmperfmon query to work either as it fails saying it can?t find the collector. Anyone had this much trouble just getting this to work? Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From stschmid at de.ibm.com Fri Oct 28 14:59:02 2016 From: stschmid at de.ibm.com (Stefan Schmidt) Date: Fri, 28 Oct 2016 15:59:02 +0200 Subject: [gpfsug-discuss] GUI and pmsensors/pmcollector In-Reply-To: <9566FC70-85D3-4D9B-B28B-58C4535B8972@siriuscom.com> References: <9566FC70-85D3-4D9B-B28B-58C4535B8972@siriuscom.com> Message-ID: Hi, the PMCollector must be installed on the same node as the GUI. The collectors can be installed on any node you want to monitor. Hope that helps. Mit freundlichen Gr??en / Kind regards Stefan Schmidt Scrum Master IBM Spectrum Scale GUI / Senior IT Architect/PMP?-Dept. M069 / IBM Spectrum Scale Software Development IBM Systems Group IBM Deutschland Phone: +49-703 4274 1966 IBM Deutschland Am Weiher 24 E-Mail: stschmid at de.ibm.com 65421 Kelsterbach Germany IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Mark.Bush at siriuscom.com" To: "gpfsug-discuss at spectrumscale.org" Date: 28.10.2016 15:53 Subject: [gpfsug-discuss] GUI and pmsensors/pmcollector Sent by: gpfsug-discuss-bounces at spectrumscale.org I am able to do everything I need just fine when I have a single node cluster and both pmsensors and pmcollector are installed all on the same node. When I try and get pmsensors on my NSD nodes and a separate pmcollector elsewhere I never can get anything to show up in the graphs. I?ve configured the pmsensors with mmperfmon config generate ?collectors=(my node) and I still get nothing. The logs show that I get connections but the GUI never show and I?m unable to get mmperfmon query to work either as it fails saying it can?t find the collector. Anyone had this much trouble just getting this to work? Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mark.Bush at siriuscom.com Fri Oct 28 15:12:25 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Fri, 28 Oct 2016 14:12:25 +0000 Subject: [gpfsug-discuss] GUI and pmsensors/pmcollector In-Reply-To: References: <9566FC70-85D3-4D9B-B28B-58C4535B8972@siriuscom.com> Message-ID: <66CEE4CD-BD01-4F50-839A-529062DC38D3@siriuscom.com> Perhaps I needed more description I have a 3 node cluster 2 NDS?s (with tiebreaker) 1 gui node The NDS?s all have pmsensors installed The gui node had pmcollector installed I?ve modified the /opt/IBM/zimon/ZIMSensors.cfg file on the NSD?s to point to my gui node Systemctl start pmsensors on NSD?s Systemclt start pmcollector on gui Systemctl start gpfsgui on gui The log even shows connection from the two NSD?s but for some reason the GUI always has a red x in the performance graphs and claims it can?t connect to the collector (which is running on the same node). Not sure what I?m missing here. It all works fine when it?s all on one node. Mark From: on behalf of Stefan Schmidt Reply-To: gpfsug main discussion list Date: Friday, October 28, 2016 at 8:59 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GUI and pmsensors/pmcollector Hi, the PMCollector must be installed on the same node as the GUI. The collectors can be installed on any node you want to monitor. Hope that helps. Mit freundlichen Gr??en / Kind regards Stefan Schmidt Scrum Master IBM Spectrum Scale GUI / Senior IT Architect/PMP?-Dept. M069 / IBM Spectrum Scale Software Development IBM Systems Group IBM Deutschland ________________________________ Phone: +49-703 4274 1966 IBM Deutschland Am Weiher 24 E-Mail: stschmid at de.ibm.com 65421 Kelsterbach Germany ________________________________ IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Mark.Bush at siriuscom.com" To: "gpfsug-discuss at spectrumscale.org" Date: 28.10.2016 15:53 Subject: [gpfsug-discuss] GUI and pmsensors/pmcollector Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ I am able to do everything I need just fine when I have a single node cluster and both pmsensors and pmcollector are installed all on the same node. When I try and get pmsensors on my NSD nodes and a separate pmcollector elsewhere I never can get anything to show up in the graphs. I?ve configured the pmsensors with mmperfmon config generate ?collectors=(my node) and I still get nothing. The logs show that I get connections but the GUI never show and I?m unable to get mmperfmon query to work either as it fails saying it can?t find the collector. Anyone had this much trouble just getting this to work? Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From akers at vt.edu Fri Oct 28 15:22:49 2016 From: akers at vt.edu (Joshua Akers) Date: Fri, 28 Oct 2016 14:22:49 +0000 Subject: [gpfsug-discuss] GPFS Log Levels Message-ID: Hi all, I am trying to find more information on GPFS log levels. Here is what I have so far: [D] - Detail info [I] - Info [N] - Notice [W] - Warning [E] - Error [X] - Critical Error [A] - Deadlock related? Any corrections or additional information would be greatly appreciated. Thanks, Josh -- *Joshua D. Akers* *HPC Systems Specialist* NI&S Systems Support (MC0214) 1700 Pratt Drive Blacksburg, VA 24061 540-231-9506 -------------- next part -------------- An HTML attachment was scrubbed... URL: From billowen at us.ibm.com Fri Oct 28 15:37:27 2016 From: billowen at us.ibm.com (Bill Owen) Date: Fri, 28 Oct 2016 07:37:27 -0700 Subject: [gpfsug-discuss] unified file and object In-Reply-To: References: Message-ID: Hi Leslie, For the issues you list below: 1. AD configuration - there is a known issue with quoted names passed to mmuserauth - we are investigating how to correct this. 2. Can you provide more details on how you configured file access? The normal procedure is to use "mmobj file-access enable", and this will set up the required settings in the config file. Can you send us: - the steps used to configure file access - the resulting /etc/swift/object-server-sof.conf - log files from /var/log/swift or output of "systemctl status openstack-swift-object-sof" We can schedule a short call to help debug if needed. Here is a more detailed description of enabling file access to object data: Enable unified file and object access See: http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.1/com.ibm.spectrum.scale.v4r21.doc/bl1adm_enablefileaccess.htm 1. enable & configure: # mmobj file-access enable # mmobj config list --ccrfile spectrum-scale-object.conf --section capabilities --property file-access-enabled file-access-enabled = true Different user id mapping approaches are supporting. The simplest is id_mgmt = local_mode. This example leaves that default in place. Verify id_mgmt mode: # mmobj config list --ccrfile object-server-sof.conf --section DEFAULT --property id_mgmt id_mgmt = local_mode At this point, you can create a storage policy that will be used for file & object access containers. We'll create a new fileset for this step (in 4.2.2 we will support objectizing existing filesets). In this example, we create the fileset with 1M inodes initially allocated: # mmobj policy create fileaccess --enable-file-access -i 1000000 [I] Getting latest configuration from ccr [I] Creating fileset fs2-1m-03:obj_fileaccess [I] Creating new unique index and building the object rings [I] Updating the configuration [I] Uploading the changed configuration The new storage policy is listed: # mmobj policy list Index Name Default Deprecated Fileset Functions -------------------------------------------------------------------------------- 0 policy-0 yes Object_Fileset default 87811608170 fileaccess obj_fileaccess file-and-object-access The new fileset called obj_fileaccess is shown below: # mmlsfileset fs2-1m-03 obj_fileaccess -L Filesets in file system 'fs2-1m-03': Name Id RootInode ParentId Created InodeSpace MaxInodes AllocInodes Comment obj_fileaccess 6 134217731 0 Wed Aug 17 10:08:51 2016 4 1000192 1000192 2. Create one or more containers that are file-access enabled: Next, you can create one or more containers that use this storage policy (and have file access enabled): # source openrc # swift post fileaccess_container --header "x-storage-policy: fileaccess" # swift stat fileaccess_container Account: AUTH_acad33df8cdf402ebab6bfb87b1674af Container: fileaccess_container Objects: 0 Bytes: 0 Read ACL: Write ACL: Sync To: Sync Key: Accept-Ranges: bytes X-Storage-Policy: fileaccess X-Timestamp: 1471456287.21609 X-Trans-Id: tx0f305353a20a4ac1ab927-0057b4a428 Content-Type: text/plain; charset=utf-8 Objects can be added to the container from the object or file interface, the file path is structured like this: ///// In our example it would be: /ibm/fs2-1m-03/obj_fileaccess/s87811608170z1device1/AUTH_acad33df8cdf402ebab6bfb87b1674af/fileaccess_container/ For convenience, create a soft link: ln -s /ibm/fs2-1m-03/obj_fileaccess/s87811608170z1device1/AUTH_acad33df8cdf402ebab6bfb87b1674af/fileaccess_container/ /ibm/fs2-1m-03/unified_file_object 3. Verify access from both file & object interface: Upload a file from object interface, it will be readable from file interface, owned by swift user. For example: # date > test1.obj # swift upload fileaccess_container test1.obj test1.obj # ls -l /ibm/fs2-1m-03/unified_file_object/ total 0 -rwxr-xr-x 1 swift swift 29 Aug 17 11:03 test1.obj Similarly, create an example object (or set of objects) from file interface: # date > /ibm/fs2-1m-03/unified_file_object/test2.obj The object must be readable by swift user - either by setting file ownership or using file acls: # chown swift:swift /ibm/fs2-1m-03/unified_file_object/test2.obj # ls -l /ibm/fs2-1m-03/unified_file_object/ -rwxr-xr-x 1 swift swift 29 Aug 17 11:03 test1.obj -rw-r--r-- 1 swift swift 29 Aug 17 11:05 test2.obj New files will be "objectized" (made visible to the object interface) by a periodic process. The default period for this is 30 min, but can be changed using mmob config change command. This process can be resource intensive, so don't see to run too often if there are many containers & files. # mmobj config list --ccrfile spectrum-scale-objectizer.conf --section DEFAULT --property objectization_interval objectization_interval = 1800 After waiting the objectizer interval, the new object shows up from the object interface, and can be accessed like any other object: # swift list fileaccess_container test1.obj test2.obj # swift download fileaccess_container test2.obj test2.obj [auth 0.229s, headers 0.862s, total 0.862s, 0.000 MB/s] Verify the contents match # cat test2.obj Wed Aug 17 11:05:00 PDT 2016 # cat /ibm/fs2-1m-03/unified_file_object/test2.obj Wed Aug 17 11:05:00 PDT 2016 And update from file interface: # echo "bill was here" >> /ibm/fs2-1m-03/unified_file_object/test2.obj Redownload from object interface and confirm: # swift download fileaccess_container test2.obj test2.obj [auth 0.230s, headers 0.729s, total 0.729s, 0.000 MB/s] [ # cat test2.obj Wed Aug 17 11:05:00 PDT 2016 bill was here Use Case Example: One usecase is adding/expanding an archive file in file interface, and results can be accessed from file interface. For example, create a set of test files: # mkdir test [root at client22 ~]# for i in {1..10}; do date > test/smallobj$i ; done [root at client22 ~]# ll test total 40 -rw-r--r-- 1 root root 29 Aug 17 11:31 smallobj1 -rw-r--r-- 1 root root 29 Aug 17 11:31 smallobj10 -rw-r--r-- 1 root root 29 Aug 17 11:31 smallobj2 -rw-r--r-- 1 root root 29 Aug 17 11:31 smallobj3 -rw-r--r-- 1 root root 29 Aug 17 11:31 smallobj4 -rw-r--r-- 1 root root 29 Aug 17 11:31 smallobj5 -rw-r--r-- 1 root root 29 Aug 17 11:31 smallobj6 -rw-r--r-- 1 root root 29 Aug 17 11:31 smallobj7 -rw-r--r-- 1 root root 29 Aug 17 11:31 smallobj8 -rw-r--r-- 1 root root 29 Aug 17 11:31 smallobj9 # tar -cf databag.tar test Expand the file into file interface and set ownership to swift: # cd /ibm/fs2-1m-03/unified_file_object/ # tar -xf ~/databag.tar # chown -R swift:swift test After objectizer runs, all files are available from object interface: # swift list fileaccess_container test/smallobj1 test/smallobj10 test/smallobj2 test/smallobj3 test/smallobj4 test/smallobj5 test/smallobj6 test/smallobj7 test/smallobj8 test/smallobj9 test1.obj test2.obj Regards, Bill Owen billowen at us.ibm.com Spectrum Scale Object Storage 520-799-4829 From: leslie elliott To: gpfsug-discuss at spectrumscale.org Date: 10/25/2016 03:01 PM Subject: [gpfsug-discuss] unified file and object Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi We are in the process of trying to configure unified file and object storage in unified_mode and have a few problems We are running 4.2.1 and do not have any current issues the file access protocols setting up the authentication First issue we do have is binding the object data to our Active Directory, we seem to be hitting a road block due to the fact that the bind DN has spaces in it, if we enclose the DN in quotes it still fails, if we escape them with the appropriate RFC ?value we can get the mmuserauth to complete but the lookups from the local keystone fail for the authentication of the users The DN for the swift user and swift admin also have quotes in them, so just doing it on the command line is not enough to get the mmuserauth command to complete Second problem is OBJ:openstack-swift-object-sof ? ? ? ? ? is not running This seems to be due to the config file not having ?bind_ip and bind_port values, if these are added then the error turns to pipeline of other settings in the config file missing This particular issue occurs no matter what the auth type is set to be for object Hopefully this make some sense to someone Thanks leslie Leslie Elliott, Infrastructure Support Specialist, ?Faculty Infrastructure and Applications Support Information Technology Services, The University of Queensland _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From taylorm at us.ibm.com Fri Oct 28 15:56:30 2016 From: taylorm at us.ibm.com (Michael L Taylor) Date: Fri, 28 Oct 2016 07:56:30 -0700 Subject: [gpfsug-discuss] GUI and pmsensors/pmcollector In-Reply-To: References: Message-ID: Have a quick read of this link and see if it helps: https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.1/com.ibm.spectrum.scale.v4r21.doc/bl1ins_manualinstallofgui.htm From: gpfsug-discuss-request at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Date: 10/28/2016 07:23 AM Subject: gpfsug-discuss Digest, Vol 57, Issue 76 Sent by: gpfsug-discuss-bounces at spectrumscale.org Message: 1 Date: Fri, 28 Oct 2016 14:12:25 +0000 From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GUI and pmsensors/pmcollector Message-ID: <66CEE4CD-BD01-4F50-839A-529062DC38D3 at siriuscom.com> Content-Type: text/plain; charset="utf-8" Perhaps I needed more description I have a 3 node cluster 2 NDS?s (with tiebreaker) 1 gui node The NDS?s all have pmsensors installed The gui node had pmcollector installed I?ve modified the /opt/IBM/zimon/ZIMSensors.cfg file on the NSD?s to point to my gui node Systemctl start pmsensors on NSD?s Systemclt start pmcollector on gui Systemctl start gpfsgui on gui The log even shows connection from the two NSD?s but for some reason the GUI always has a red x in the performance graphs and claims it can?t connect to the collector (which is running on the same node). Not sure what I?m missing here. It all works fine when it?s all on one node. Mark From: on behalf of Stefan Schmidt Reply-To: gpfsug main discussion list Date: Friday, October 28, 2016 at 8:59 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GUI and pmsensors/pmcollector Hi, the PMCollector must be installed on the same node as the GUI. The collectors can be installed on any node you want to monitor. Hope that helps. Mit freundlichen Gr??en / Kind regards Stefan Schmidt Scrum Master IBM Spectrum Scale GUI / Senior IT Architect/PMP?-Dept. M069 / IBM Spectrum Scale Software Development IBM Systems Group IBM Deutschland ________________________________ Phone: +49-703 4274 1966 IBM Deutschland Am Weiher 24 E-Mail: stschmid at de.ibm.com 65421 Kelsterbach Germany ________________________________ IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Mark.Bush at siriuscom.com" To: "gpfsug-discuss at spectrumscale.org" Date: 28.10.2016 15:53 Subject: [gpfsug-discuss] GUI and pmsensors/pmcollector Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ I am able to do everything I need just fine when I have a single node cluster and both pmsensors and pmcollector are installed all on the same node. When I try and get pmsensors on my NSD nodes and a separate pmcollector elsewhere I never can get anything to show up in the graphs. I?ve configured the pmsensors with mmperfmon config generate ?collectors= (my node) and I still get nothing. The logs show that I get connections but the GUI never show and I?m unable to get mmperfmon query to work either as it fails saying it can?t find the collector. Anyone had this much trouble just getting this to work? Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161028/b4552796/attachment-0001.html > ------------------------------ Message: 2 Date: Fri, 28 Oct 2016 14:22:49 +0000 From: Joshua Akers To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] GPFS Log Levels Message-ID: Content-Type: text/plain; charset="utf-8" Hi all, I am trying to find more information on GPFS log levels. Here is what I have so far: [D] - Detail info [I] - Info [N] - Notice [W] - Warning [E] - Error [X] - Critical Error [A] - Deadlock related? Any corrections or additional information would be greatly appreciated. Thanks, Josh -- *Joshua D. Akers* *HPC Systems Specialist* NI&S Systems Support (MC0214) 1700 Pratt Drive Blacksburg, VA 24061 540-231-9506 -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161028/7ebac661/attachment.html > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 76 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: From rohwedder at de.ibm.com Fri Oct 28 16:08:17 2016 From: rohwedder at de.ibm.com (Markus Rohwedder) Date: Fri, 28 Oct 2016 17:08:17 +0200 Subject: [gpfsug-discuss] GUI and pmsensors/pmcollector In-Reply-To: References: Message-ID: Hi, Some more questions and things to look for: 1. Are there several collectors running, potentially at different code levels? All collectors should run at the same code level. Simplest case, there is only one collector. Sensors and collectors can have different code levels. 2. Is the sensor config updated manually or per mmperfmon config update? Manual update might be overwritten if the system thinks that the config should be distr?buted automatically. See Knowledge center and mmperfmon CLI command for info. 3. How does the sensor config look like (output of mmperfmon config show)? 4. Are only NSD charts showing that there is no data, or all charts? Please note; In a SAN environment (Every node sees every NSD), there is no NSD server side data reported. GPFS is smart and knows to use the local device and circumvents the NSD code stack. However, we dont get notified on traffic to the disks. 5. Which code level? Thanks, Mit freundlichen Gr??en / Kind regards Dr. Markus Rohwedder Spectrum Scale GUI Development Phone: +49 7034 6430190 IBM Deutschland E-Mail: rohwedder at de.ibm.com Am Weiher 24 65451 Kelsterbach Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Martina K?deritz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: gpfsug-discuss-request at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Date: 28.10.2016 16:23 Subject: gpfsug-discuss Digest, Vol 57, Issue 76 Sent by: gpfsug-discuss-bounces at spectrumscale.org Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: GUI and pmsensors/pmcollector (Mark.Bush at siriuscom.com) 2. GPFS Log Levels (Joshua Akers) ---------------------------------------------------------------------- Message: 1 Date: Fri, 28 Oct 2016 14:12:25 +0000 From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GUI and pmsensors/pmcollector Message-ID: <66CEE4CD-BD01-4F50-839A-529062DC38D3 at siriuscom.com> Content-Type: text/plain; charset="utf-8" Perhaps I needed more description I have a 3 node cluster 2 NDS?s (with tiebreaker) 1 gui node The NDS?s all have pmsensors installed The gui node had pmcollector installed I?ve modified the /opt/IBM/zimon/ZIMSensors.cfg file on the NSD?s to point to my gui node Systemctl start pmsensors on NSD?s Systemclt start pmcollector on gui Systemctl start gpfsgui on gui The log even shows connection from the two NSD?s but for some reason the GUI always has a red x in the performance graphs and claims it can?t connect to the collector (which is running on the same node). Not sure what I?m missing here. It all works fine when it?s all on one node. Mark From: on behalf of Stefan Schmidt Reply-To: gpfsug main discussion list Date: Friday, October 28, 2016 at 8:59 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GUI and pmsensors/pmcollector Hi, the PMCollector must be installed on the same node as the GUI. The collectors can be installed on any node you want to monitor. Hope that helps. Mit freundlichen Gr??en / Kind regards Stefan Schmidt Scrum Master IBM Spectrum Scale GUI / Senior IT Architect/PMP?-Dept. M069 / IBM Spectrum Scale Software Development IBM Systems Group IBM Deutschland ________________________________ Phone: +49-703 4274 1966 IBM Deutschland Am Weiher 24 E-Mail: stschmid at de.ibm.com 65421 Kelsterbach Germany ________________________________ IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Mark.Bush at siriuscom.com" To: "gpfsug-discuss at spectrumscale.org" Date: 28.10.2016 15:53 Subject: [gpfsug-discuss] GUI and pmsensors/pmcollector Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ I am able to do everything I need just fine when I have a single node cluster and both pmsensors and pmcollector are installed all on the same node. When I try and get pmsensors on my NSD nodes and a separate pmcollector elsewhere I never can get anything to show up in the graphs. I?ve configured the pmsensors with mmperfmon config generate ?collectors= (my node) and I still get nothing. The logs show that I get connections but the GUI never show and I?m unable to get mmperfmon query to work either as it fails saying it can?t find the collector. Anyone had this much trouble just getting this to work? Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161028/b4552796/attachment-0001.html > ------------------------------ Message: 2 Date: Fri, 28 Oct 2016 14:22:49 +0000 From: Joshua Akers To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] GPFS Log Levels Message-ID: Content-Type: text/plain; charset="utf-8" Hi all, I am trying to find more information on GPFS log levels. Here is what I have so far: [D] - Detail info [I] - Info [N] - Notice [W] - Warning [E] - Error [X] - Critical Error [A] - Deadlock related? Any corrections or additional information would be greatly appreciated. Thanks, Josh -- *Joshua D. Akers* *HPC Systems Specialist* NI&S Systems Support (MC0214) 1700 Pratt Drive Blacksburg, VA 24061 540-231-9506 -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161028/7ebac661/attachment.html > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 57, Issue 76 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecblank.gif Type: image/gif Size: 45 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 06388797.gif Type: image/gif Size: 1851 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From mweil at wustl.edu Fri Oct 28 21:49:37 2016 From: mweil at wustl.edu (Matt Weil) Date: Fri, 28 Oct 2016 15:49:37 -0500 Subject: [gpfsug-discuss] Leave protocol detail info Message-ID: anybody no what this means? Wed Oct 26 20:08:29.619 2016: [D] Leave protocol detail info: LA: 75 LFLG: 24409951 LFLG delta: 75 ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. From mweil at wustl.edu Fri Oct 28 22:02:59 2016 From: mweil at wustl.edu (Matt Weil) Date: Fri, 28 Oct 2016 16:02:59 -0500 Subject: [gpfsug-discuss] Leave protocol detail info In-Reply-To: References: Message-ID: Also is there any document that explains what happens to the cluster when a client node stops responding and get evicted. On 10/28/16 3:49 PM, Matt Weil wrote: > anybody no what this means? > > Wed Oct 26 20:08:29.619 2016: [D] Leave protocol detail info: LA: 75 > LFLG: 24409951 LFLG delta: 75 > ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. From leslie.james.elliott at gmail.com Sat Oct 29 11:53:19 2016 From: leslie.james.elliott at gmail.com (leslie elliott) Date: Sat, 29 Oct 2016 20:53:19 +1000 Subject: [gpfsug-discuss] unified file and object In-Reply-To: References: Message-ID: Bill to be clear the file access I mentioned was in relation to SMB and NFS using mmuserauth rather than the unification with the object store since it is required as well but I did try to do this for object as well using the Administration and Programming Reference from page 142, was using unified_mode rather than local_mode mmobj config change --ccrfile spectrum-scale-object.conf --section capabilities --property file-access-enabled --value true the mmuserauth failed as you are aware, we have created test accounts without spaces in the DN and were successful with this step, so eagerly await a fix to be able to use the correct accounts mmobj config change --ccrfile object-server-sof.conf --section DEFAULT --property id_mgmt --value unified_mode mmobj config change --ccrfile object-server-sof.conf --section DEFAULT --property ad_domain --value DOMAIN we have successfully tested object stores on this cluster with simple auth the output you asked for is as follows [root at pren-gs7k-vm4 ~]# cat /etc/swift/object-server-sof.conf [DEFAULT] devices = /gpfs/pren01/ObjectFileset/o log_level = ERROR [root at pren-gs7k-vm4 ~]# systemctl -l status openstack-swift-object-sof ? openstack-swift-object-sof.service - OpenStack Object Storage (swift) - Object Server Loaded: loaded (/usr/lib/systemd/system/openstack-swift-object-sof.service; disabled; vendor preset: disabled) Active: failed (Result: exit-code) since Sat 2016-10-29 10:30:22 UTC; 27s ago Process: 8086 ExecStart=/usr/bin/swift-object-server-sof /etc/swift/object-server-sof.conf (code=exited, status=1/FAILURE) Main PID: 8086 (code=exited, status=1/FAILURE) Oct 29 10:30:22 pren-gs7k-vm4 systemd[1]: Started OpenStack Object Storage (swift) - Object Server. Oct 29 10:30:22 pren-gs7k-vm4 systemd[1]: Starting OpenStack Object Storage (swift) - Object Server... Oct 29 10:30:22 pren-gs7k-vm4 swift-object-server-sof[8086]: Error trying to load config from /etc/swift/object-server-sof.conf: No section 'object-server' (prefixed by 'app' or 'application' or 'composite' or 'composit' or 'pipeline' or 'filter-app') found in config /etc/swift/object-server-sof.conf Oct 29 10:30:22 pren-gs7k-vm4 systemd[1]: openstack-swift-object-sof.service: main process exited, code=exited, status=1/FAILURE Oct 29 10:30:22 pren-gs7k-vm4 systemd[1]: Unit openstack-swift-object-sof.service entered failed state. Oct 29 10:30:22 pren-gs7k-vm4 systemd[1]: openstack-swift-object-sof.service failed. I am happy to help you or for you to help to debug this problem via a short call thanks leslie On 29 October 2016 at 00:37, Bill Owen wrote: > > 2. Can you provide more details on how you configured file access? The > normal procedure is to use "mmobj file-access enable", and this will set up > the required settings in the config file. Can you send us: > - the steps used to configure file access > - the resulting /etc/swift/object-server-sof.conf > - log files from /var/log/swift or output of "systemctl status > openstack-swift-object-sof" > > We can schedule a short call to help debug if needed. > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Greg.Lehmann at csiro.au Mon Oct 31 07:03:35 2016 From: Greg.Lehmann at csiro.au (Greg.Lehmann at csiro.au) Date: Mon, 31 Oct 2016 07:03:35 +0000 Subject: [gpfsug-discuss] Two new whitepapers published In-Reply-To: References: Message-ID: <33bda80b2ba24bcb996819dbc30f1d75@exch1-cdc.nexus.csiro.au> Thanks Yuri. These were great. I'm not trying to be impertinent, but I have one suggestion - If you can find the time, add some diagrams to help readers visualise the various data structures and layouts. I am thinking along the lines of what was in "The Magic Garden Explained" and "The Design and Implementation of the 4.3BSD UNIX Operating System" where they talked about their filesystems. Cheers, Greg From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Yuri L Volobuev Sent: Friday, 21 October 2016 11:59 AM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Two new whitepapers published Esteemed GPFS and Spectrum Scale users, For your reading enjoyment, two new whitepapers have been posted to the Spectrum Scale Wiki: Spectrum Scale: Replication in GPFS Spectrum Scale: GPFS Metadata The URL for the parent page is https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/White%20Papers%20%26%20Media The two .pdf documents are accessible through the Attachment section at the bottom of the page. Unfortunately, dW "spam prevention engine" does a very good job preventing me from "spamming" the page to actually add links. Best regards, Yuri -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Mon Oct 31 09:53:38 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 31 Oct 2016 09:53:38 +0000 Subject: [gpfsug-discuss] Recent Whitepapers from Yuri Volobuev Message-ID: For those of you who may not know, Yuri Volobuev has left IBM to pursue new challenges. Myself along with many others received so much help and keen insight from Yuri on all things GPFS. He will be missed. Bob Oesterlin Sr Principal Storage Engineer, Nuance -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.knister at gmail.com Mon Oct 31 16:19:11 2016 From: aaron.knister at gmail.com (Aaron Knister) Date: Mon, 31 Oct 2016 12:19:11 -0400 Subject: [gpfsug-discuss] Recent Whitepapers from Yuri Volobuev In-Reply-To: References: Message-ID: <9C6B34F6-2849-4B77-9E8B-3806D634C281@gmail.com> Wow! That's a real shame. His ability to articulate his immense knowledge of GPFS was truly impressive. Sent from my iPhone > On Oct 31, 2016, at 5:53 AM, Oesterlin, Robert wrote: > > For those of you who may not know, Yuri Volobuev has left IBM to pursue new challenges. Myself along with many others received so much help and keen insight from Yuri on all things GPFS. > > He will be missed. > > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.wonderley at vt.edu Mon Oct 31 16:52:43 2016 From: eric.wonderley at vt.edu (J. Eric Wonderley) Date: Mon, 31 Oct 2016 12:52:43 -0400 Subject: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size Message-ID: I wanted to do something like this... [root at cl001 ~]# cat /opt/gpfs/home.ply /*Failsafe migration of old small files back to spinning media pool(fc_8T) */ RULE 'theshold' MIGRATE FROM POOL 'system' THRESHOLD(90,70) WEIGHT(ACCESS_TIME) TO POOL 'fc_8T' /*Write files larger than 16MB to pool called "fc_8T" */ RULE 'bigfiles' SET POOL 'fc_8T' WHERE FILE_SIZE>16777216 /*Move anything else to system pool */ RULE 'default' SET POOL 'system' Apparently there is no happiness using FILE_SIZE in a placement policy: [root at cl001 ~]# mmchpolicy home /opt/gpfs/home.ply Error while validating policy `home.ply': rc=22: PCSQLERR: 'FILE_SIZE' is an unsupported or unknown attribute or variable name in this context. PCSQLCTX: at line 4 of 6: RULE 'bigfiles' SET POOL 'fc_8T' WHERE {{{FILE_SIZE}}}>16777216 runRemoteCommand_v2: cl002.cl.arc.internal: tschpolicy /dev/home /var/mmfs/tmp/tspolicyFile.mmchpolicy.113372 -t home.ply failed. mmchpolicy: Command failed. Examine previous error messages to determine cause. Can anyone suggest a way to accomplish this using policy? -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Mon Oct 31 17:09:55 2016 From: bbanister at jumptrading.com (Bryan Banister) Date: Mon, 31 Oct 2016 17:09:55 +0000 Subject: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size In-Reply-To: References: Message-ID: <21BC488F0AEA2245B2C3E83FC0B33DBB063A1D4A@CHI-EXCHANGEW1.w2k.jumptrading.com> The File Placement Policy that you are trying to set cannot use the size of the file to determine the placement of the file in a GPFS Storage Pool. This is because GPFS has no idea what the file size will be when the file is open()?d for writing. Hope that helps! -Bryan PS. I really wish that we could use a path for specifying data placement in a GPFS Pool, and not just the file name, owner, etc. I?ll submit a RFE for this. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of J. Eric Wonderley Sent: Monday, October 31, 2016 11:53 AM To: gpfsug main discussion list Subject: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size I wanted to do something like this... [root at cl001 ~]# cat /opt/gpfs/home.ply /*Failsafe migration of old small files back to spinning media pool(fc_8T) */ RULE 'theshold' MIGRATE FROM POOL 'system' THRESHOLD(90,70) WEIGHT(ACCESS_TIME) TO POOL 'fc_8T' /*Write files larger than 16MB to pool called "fc_8T" */ RULE 'bigfiles' SET POOL 'fc_8T' WHERE FILE_SIZE>16777216 /*Move anything else to system pool */ RULE 'default' SET POOL 'system' Apparently there is no happiness using FILE_SIZE in a placement policy: [root at cl001 ~]# mmchpolicy home /opt/gpfs/home.ply Error while validating policy `home.ply': rc=22: PCSQLERR: 'FILE_SIZE' is an unsupported or unknown attribute or variable name in this context. PCSQLCTX: at line 4 of 6: RULE 'bigfiles' SET POOL 'fc_8T' WHERE {{{FILE_SIZE}}}>16777216 runRemoteCommand_v2: cl002.cl.arc.internal: tschpolicy /dev/home /var/mmfs/tmp/tspolicyFile.mmchpolicy.113372 -t home.ply failed. mmchpolicy: Command failed. Examine previous error messages to determine cause. Can anyone suggest a way to accomplish this using policy? ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jez.tucker at gpfsug.org Mon Oct 31 17:20:30 2016 From: jez.tucker at gpfsug.org (Jez Tucker) Date: Mon, 31 Oct 2016 17:20:30 +0000 Subject: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size In-Reply-To: <21BC488F0AEA2245B2C3E83FC0B33DBB063A1D4A@CHI-EXCHANGEW1.w2k.jumptrading.com> References: <21BC488F0AEA2245B2C3E83FC0B33DBB063A1D4A@CHI-EXCHANGEW1.w2k.jumptrading.com> Message-ID: Hey Bryan There was a previous RFE for path placement from the UG, but Yuri told me this was not techically possible as an inode has no knowledge about the parent dentry. (IIRC). You can see this in effect in the C API. It is possible to work this out at kernel level, but it's so costly that it becomes non-viable at scale / performance. IBMers please chip in and expand if you will. Jez On 31/10/16 17:09, Bryan Banister wrote: > > The File Placement Policy that you are trying to set cannot use the > size of the file to determine the placement of the file in a GPFS > Storage Pool. This is because GPFS has no idea what the file size > will be when the file is open()?d for writing. > > Hope that helps! > > -Bryan > > PS. I really wish that we could use a path for specifying data > placement in a GPFS Pool, and not just the file name, owner, etc. > I?ll submit a RFE for this. > > *From:*gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of *J. > Eric Wonderley > *Sent:* Monday, October 31, 2016 11:53 AM > *To:* gpfsug main discussion list > *Subject:* [gpfsug-discuss] wanted...gpfs policy that places larger > files onto a pool based on size > > I wanted to do something like this... > > > [root at cl001 ~]# cat /opt/gpfs/home.ply > /*Failsafe migration of old small files back to spinning media > pool(fc_8T) */ > RULE 'theshold' MIGRATE FROM POOL 'system' THRESHOLD(90,70) > WEIGHT(ACCESS_TIME) TO POOL 'fc_8T' > /*Write files larger than 16MB to pool called "fc_8T" */ > RULE 'bigfiles' SET POOL 'fc_8T' WHERE FILE_SIZE>16777216 > /*Move anything else to system pool */ > RULE 'default' SET POOL 'system' > > Apparently there is no happiness using FILE_SIZE in a placement policy: > [root at cl001 ~]# mmchpolicy home /opt/gpfs/home.ply > Error while validating policy `home.ply': rc=22: > PCSQLERR: 'FILE_SIZE' is an unsupported or unknown attribute or > variable name in this context. > PCSQLCTX: at line 4 of 6: RULE 'bigfiles' SET POOL 'fc_8T' WHERE > {{{FILE_SIZE}}}>16777216 > runRemoteCommand_v2: cl002.cl.arc.internal: tschpolicy /dev/home > /var/mmfs/tmp/tspolicyFile.mmchpolicy.113372 -t home.ply failed. > mmchpolicy: Command failed. Examine previous error messages to > determine cause. > > Can anyone suggest a way to accomplish this using policy? > > > ------------------------------------------------------------------------ > > Note: This email is for the confidential use of the named addressee(s) > only and may contain proprietary, confidential or privileged > information. If you are not the intended recipient, you are hereby > notified that any review, dissemination or copying of this email is > strictly prohibited, and to please notify the sender immediately and > destroy this email and any attachments. Email transmission cannot be > guaranteed to be secure or error-free. The Company, therefore, does > not make any guarantees as to the completeness or accuracy of this > email or any attachments. This email is for informational purposes > only and does not constitute a recommendation, offer, request or > solicitation of any kind to buy, sell, subscribe, redeem or perform > any type of transaction of a financial product. > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From mimarsh2 at vt.edu Mon Oct 31 17:23:40 2016 From: mimarsh2 at vt.edu (Brian Marshall) Date: Mon, 31 Oct 2016 13:23:40 -0400 Subject: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size In-Reply-To: References: <21BC488F0AEA2245B2C3E83FC0B33DBB063A1D4A@CHI-EXCHANGEW1.w2k.jumptrading.com> Message-ID: When creating a "fast tier" storage pool in a filesystem is the normal case to create a placement policy that places all files in the fast tier and migrates out old and large files? Brian Marshall On Mon, Oct 31, 2016 at 1:20 PM, Jez Tucker wrote: > Hey Bryan > > There was a previous RFE for path placement from the UG, but Yuri told > me this was not techically possible as an inode has no knowledge about the > parent dentry. (IIRC). You can see this in effect in the C API. It is > possible to work this out at kernel level, but it's so costly that it > becomes non-viable at scale / performance. > > IBMers please chip in and expand if you will. > > Jez > > > On 31/10/16 17:09, Bryan Banister wrote: > > The File Placement Policy that you are trying to set cannot use the size > of the file to determine the placement of the file in a GPFS Storage Pool. > This is because GPFS has no idea what the file size will be when the file > is open()?d for writing. > > > > Hope that helps! > > -Bryan > > > > PS. I really wish that we could use a path for specifying data placement > in a GPFS Pool, and not just the file name, owner, etc. I?ll submit a RFE > for this. > > > > *From:* gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss- > bounces at spectrumscale.org ] *On > Behalf Of *J. Eric Wonderley > *Sent:* Monday, October 31, 2016 11:53 AM > *To:* gpfsug main discussion list > *Subject:* [gpfsug-discuss] wanted...gpfs policy that places larger files > onto a pool based on size > > > > I wanted to do something like this... > > > [root at cl001 ~]# cat /opt/gpfs/home.ply > /*Failsafe migration of old small files back to spinning media pool(fc_8T) > */ > RULE 'theshold' MIGRATE FROM POOL 'system' THRESHOLD(90,70) > WEIGHT(ACCESS_TIME) TO POOL 'fc_8T' > /*Write files larger than 16MB to pool called "fc_8T" */ > RULE 'bigfiles' SET POOL 'fc_8T' WHERE FILE_SIZE>16777216 > /*Move anything else to system pool */ > RULE 'default' SET POOL 'system' > > Apparently there is no happiness using FILE_SIZE in a placement policy: > [root at cl001 ~]# mmchpolicy home /opt/gpfs/home.ply > Error while validating policy `home.ply': rc=22: > PCSQLERR: 'FILE_SIZE' is an unsupported or unknown attribute or variable > name in this context. > PCSQLCTX: at line 4 of 6: RULE 'bigfiles' SET POOL 'fc_8T' WHERE > {{{FILE_SIZE}}}>16777216 > runRemoteCommand_v2: cl002.cl.arc.internal: tschpolicy /dev/home > /var/mmfs/tmp/tspolicyFile.mmchpolicy.113372 -t home.ply failed. > mmchpolicy: Command failed. Examine previous error messages to determine > cause. > > Can anyone suggest a way to accomplish this using policy? > > ------------------------------ > > Note: This email is for the confidential use of the named addressee(s) > only and may contain proprietary, confidential or privileged information. > If you are not the intended recipient, you are hereby notified that any > review, dissemination or copying of this email is strictly prohibited, and > to please notify the sender immediately and destroy this email and any > attachments. Email transmission cannot be guaranteed to be secure or > error-free. The Company, therefore, does not make any guarantees as to the > completeness or accuracy of this email or any attachments. This email is > for informational purposes only and does not constitute a recommendation, > offer, request or solicitation of any kind to buy, sell, subscribe, redeem > or perform any type of transaction of a financial product. > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ewahl at osc.edu Mon Oct 31 17:26:28 2016 From: ewahl at osc.edu (Edward Wahl) Date: Mon, 31 Oct 2016 13:26:28 -0400 Subject: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size In-Reply-To: <21BC488F0AEA2245B2C3E83FC0B33DBB063A1D4A@CHI-EXCHANGEW1.w2k.jumptrading.com> References: <21BC488F0AEA2245B2C3E83FC0B33DBB063A1D4A@CHI-EXCHANGEW1.w2k.jumptrading.com> Message-ID: <20161031132628.5c952a40@osc.edu> On Mon, 31 Oct 2016 17:09:55 +0000 Bryan Banister wrote: > -Bryan > ? > PS. I really wish that we could use a path for specifying data > placement in a GPFS Pool, and not just the file name, owner, etc. > I?ll submit a RFE for this. So... use a fileset with a linked directory and a fileset placement policy to a pool? Might be a bit more rigid for what you really want, and it would be messy, but it would work just fine. -- Ed Wahl Ohio Supercomputer Center 614-292-9302 From bbanister at jumptrading.com Mon Oct 31 17:29:05 2016 From: bbanister at jumptrading.com (Bryan Banister) Date: Mon, 31 Oct 2016 17:29:05 +0000 Subject: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size In-Reply-To: <20161031132628.5c952a40@osc.edu> References: <21BC488F0AEA2245B2C3E83FC0B33DBB063A1D4A@CHI-EXCHANGEW1.w2k.jumptrading.com> <20161031132628.5c952a40@osc.edu> Message-ID: <21BC488F0AEA2245B2C3E83FC0B33DBB063A1F04@CHI-EXCHANGEW1.w2k.jumptrading.com> Thanks for that suggestion, which is something we already do, but as you say is not as flexible. I think Jez is correct about the overhead being too high to support directory path for data placement, -B -----Original Message----- From: Edward Wahl [mailto:ewahl at osc.edu] Sent: Monday, October 31, 2016 12:26 PM To: Bryan Banister Cc: gpfsug main discussion list Subject: Re: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size On Mon, 31 Oct 2016 17:09:55 +0000 Bryan Banister wrote: > -Bryan > > PS. I really wish that we could use a path for specifying data > placement in a GPFS Pool, and not just the file name, owner, etc. > I?ll submit a RFE for this. So... use a fileset with a linked directory and a fileset placement policy to a pool? Might be a bit more rigid for what you really want, and it would be messy, but it would work just fine. -- Ed Wahl Ohio Supercomputer Center 614-292-9302 ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. From chrisjscott at gmail.com Mon Oct 31 17:29:45 2016 From: chrisjscott at gmail.com (Chris Scott) Date: Mon, 31 Oct 2016 17:29:45 +0000 Subject: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size In-Reply-To: References: <21BC488F0AEA2245B2C3E83FC0B33DBB063A1D4A@CHI-EXCHANGEW1.w2k.jumptrading.com> Message-ID: Hi Brian This is exactly what I do with a SSD tier on top of 10K and 7.2K tiers. HAWC is another recent option that might address Eric's requirement but needs further consideration of the read requirements you want from the small files. Cheers Chris On 31 October 2016 at 17:23, Brian Marshall wrote: > When creating a "fast tier" storage pool in a filesystem is the normal > case to create a placement policy that places all files in the fast tier > and migrates out old and large files? > > > Brian Marshall > > On Mon, Oct 31, 2016 at 1:20 PM, Jez Tucker wrote: > >> Hey Bryan >> >> There was a previous RFE for path placement from the UG, but Yuri told >> me this was not techically possible as an inode has no knowledge about the >> parent dentry. (IIRC). You can see this in effect in the C API. It is >> possible to work this out at kernel level, but it's so costly that it >> becomes non-viable at scale / performance. >> >> IBMers please chip in and expand if you will. >> >> Jez >> >> >> On 31/10/16 17:09, Bryan Banister wrote: >> >> The File Placement Policy that you are trying to set cannot use the size >> of the file to determine the placement of the file in a GPFS Storage Pool. >> This is because GPFS has no idea what the file size will be when the file >> is open()?d for writing. >> >> >> >> Hope that helps! >> >> -Bryan >> >> >> >> PS. I really wish that we could use a path for specifying data placement >> in a GPFS Pool, and not just the file name, owner, etc. I?ll submit a RFE >> for this. >> >> >> >> *From:* gpfsug-discuss-bounces at spectrumscale.org [ >> mailto:gpfsug-discuss-bounces at spectrumscale.org >> ] *On Behalf Of *J. Eric >> Wonderley >> *Sent:* Monday, October 31, 2016 11:53 AM >> *To:* gpfsug main discussion list >> *Subject:* [gpfsug-discuss] wanted...gpfs policy that places larger >> files onto a pool based on size >> >> >> >> I wanted to do something like this... >> >> >> [root at cl001 ~]# cat /opt/gpfs/home.ply >> /*Failsafe migration of old small files back to spinning media >> pool(fc_8T) */ >> RULE 'theshold' MIGRATE FROM POOL 'system' THRESHOLD(90,70) >> WEIGHT(ACCESS_TIME) TO POOL 'fc_8T' >> /*Write files larger than 16MB to pool called "fc_8T" */ >> RULE 'bigfiles' SET POOL 'fc_8T' WHERE FILE_SIZE>16777216 >> /*Move anything else to system pool */ >> RULE 'default' SET POOL 'system' >> >> Apparently there is no happiness using FILE_SIZE in a placement policy: >> [root at cl001 ~]# mmchpolicy home /opt/gpfs/home.ply >> Error while validating policy `home.ply': rc=22: >> PCSQLERR: 'FILE_SIZE' is an unsupported or unknown attribute or variable >> name in this context. >> PCSQLCTX: at line 4 of 6: RULE 'bigfiles' SET POOL 'fc_8T' WHERE >> {{{FILE_SIZE}}}>16777216 >> runRemoteCommand_v2: cl002.cl.arc.internal: tschpolicy /dev/home >> /var/mmfs/tmp/tspolicyFile.mmchpolicy.113372 -t home.ply failed. >> mmchpolicy: Command failed. Examine previous error messages to determine >> cause. >> >> Can anyone suggest a way to accomplish this using policy? >> >> ------------------------------ >> >> Note: This email is for the confidential use of the named addressee(s) >> only and may contain proprietary, confidential or privileged information. >> If you are not the intended recipient, you are hereby notified that any >> review, dissemination or copying of this email is strictly prohibited, and >> to please notify the sender immediately and destroy this email and any >> attachments. Email transmission cannot be guaranteed to be secure or >> error-free. The Company, therefore, does not make any guarantees as to the >> completeness or accuracy of this email or any attachments. This email is >> for informational purposes only and does not constitute a recommendation, >> offer, request or solicitation of any kind to buy, sell, subscribe, redeem >> or perform any type of transaction of a financial product. >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chrisjscott at gmail.com Mon Oct 31 17:33:57 2016 From: chrisjscott at gmail.com (Chris Scott) Date: Mon, 31 Oct 2016 17:33:57 +0000 Subject: [gpfsug-discuss] Recent Whitepapers from Yuri Volobuev In-Reply-To: <9C6B34F6-2849-4B77-9E8B-3806D634C281@gmail.com> References: <9C6B34F6-2849-4B77-9E8B-3806D634C281@gmail.com> Message-ID: Hear, hear. My appreciation to Yuri for his great contribution to the user community of important details. On 31 October 2016 at 16:19, Aaron Knister wrote: > Wow! That's a real shame. His ability to articulate his immense knowledge > of GPFS was truly impressive. > > Sent from my iPhone > > On Oct 31, 2016, at 5:53 AM, Oesterlin, Robert < > Robert.Oesterlin at nuance.com> wrote: > > For those of you who may not know, Yuri Volobuev has left IBM to pursue > new challenges. Myself along with many others received so much help and > keen insight from Yuri on all things GPFS. > > > > He will be missed. > > > > > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.wonderley at vt.edu Mon Oct 31 17:45:48 2016 From: eric.wonderley at vt.edu (J. Eric Wonderley) Date: Mon, 31 Oct 2016 13:45:48 -0400 Subject: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size In-Reply-To: References: <21BC488F0AEA2245B2C3E83FC0B33DBB063A1D4A@CHI-EXCHANGEW1.w2k.jumptrading.com> Message-ID: Placement policy only applies to writes and I had thought that gpfs did enough writing to memory "pagepool" to figure out the size before committing the write to pool. I also admit I don't know all of the innards of gpfs. Pehaps being a copy on write type filesystem prevents this for occurring. On Mon, Oct 31, 2016 at 1:29 PM, Chris Scott wrote: > Hi Brian > > This is exactly what I do with a SSD tier on top of 10K and 7.2K tiers. > > HAWC is another recent option that might address Eric's requirement but > needs further consideration of the read requirements you want from the > small files. > > Cheers > Chris > > On 31 October 2016 at 17:23, Brian Marshall wrote: > >> When creating a "fast tier" storage pool in a filesystem is the normal >> case to create a placement policy that places all files in the fast tier >> and migrates out old and large files? >> >> >> Brian Marshall >> >> On Mon, Oct 31, 2016 at 1:20 PM, Jez Tucker >> wrote: >> >>> Hey Bryan >>> >>> There was a previous RFE for path placement from the UG, but Yuri told >>> me this was not techically possible as an inode has no knowledge about the >>> parent dentry. (IIRC). You can see this in effect in the C API. It is >>> possible to work this out at kernel level, but it's so costly that it >>> becomes non-viable at scale / performance. >>> >>> IBMers please chip in and expand if you will. >>> >>> Jez >>> >>> >>> On 31/10/16 17:09, Bryan Banister wrote: >>> >>> The File Placement Policy that you are trying to set cannot use the size >>> of the file to determine the placement of the file in a GPFS Storage Pool. >>> This is because GPFS has no idea what the file size will be when the file >>> is open()?d for writing. >>> >>> >>> >>> Hope that helps! >>> >>> -Bryan >>> >>> >>> >>> PS. I really wish that we could use a path for specifying data placement >>> in a GPFS Pool, and not just the file name, owner, etc. I?ll submit a RFE >>> for this. >>> >>> >>> >>> *From:* gpfsug-discuss-bounces at spectrumscale.org [ >>> mailto:gpfsug-discuss-bounces at spectrumscale.org >>> ] *On Behalf Of *J. Eric >>> Wonderley >>> *Sent:* Monday, October 31, 2016 11:53 AM >>> *To:* gpfsug main discussion list >>> *Subject:* [gpfsug-discuss] wanted...gpfs policy that places larger >>> files onto a pool based on size >>> >>> >>> >>> I wanted to do something like this... >>> >>> >>> [root at cl001 ~]# cat /opt/gpfs/home.ply >>> /*Failsafe migration of old small files back to spinning media >>> pool(fc_8T) */ >>> RULE 'theshold' MIGRATE FROM POOL 'system' THRESHOLD(90,70) >>> WEIGHT(ACCESS_TIME) TO POOL 'fc_8T' >>> /*Write files larger than 16MB to pool called "fc_8T" */ >>> RULE 'bigfiles' SET POOL 'fc_8T' WHERE FILE_SIZE>16777216 >>> /*Move anything else to system pool */ >>> RULE 'default' SET POOL 'system' >>> >>> Apparently there is no happiness using FILE_SIZE in a placement policy: >>> [root at cl001 ~]# mmchpolicy home /opt/gpfs/home.ply >>> Error while validating policy `home.ply': rc=22: >>> PCSQLERR: 'FILE_SIZE' is an unsupported or unknown attribute or variable >>> name in this context. >>> PCSQLCTX: at line 4 of 6: RULE 'bigfiles' SET POOL 'fc_8T' WHERE >>> {{{FILE_SIZE}}}>16777216 >>> runRemoteCommand_v2: cl002.cl.arc.internal: tschpolicy /dev/home >>> /var/mmfs/tmp/tspolicyFile.mmchpolicy.113372 -t home.ply failed. >>> mmchpolicy: Command failed. Examine previous error messages to determine >>> cause. >>> >>> Can anyone suggest a way to accomplish this using policy? >>> >>> ------------------------------ >>> >>> Note: This email is for the confidential use of the named addressee(s) >>> only and may contain proprietary, confidential or privileged information. >>> If you are not the intended recipient, you are hereby notified that any >>> review, dissemination or copying of this email is strictly prohibited, and >>> to please notify the sender immediately and destroy this email and any >>> attachments. Email transmission cannot be guaranteed to be secure or >>> error-free. The Company, therefore, does not make any guarantees as to the >>> completeness or accuracy of this email or any attachments. This email is >>> for informational purposes only and does not constitute a recommendation, >>> offer, request or solicitation of any kind to buy, sell, subscribe, redeem >>> or perform any type of transaction of a financial product. >>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: