From a.g.richmond at leeds.ac.uk Wed Mar 1 11:19:17 2017 From: a.g.richmond at leeds.ac.uk (Aidan Richmond) Date: Wed, 1 Mar 2017 11:19:17 +0000 Subject: [gpfsug-discuss] Issues getting SMB shares working. Message-ID: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> Hello I've recently taken over managing a GPFS installation providing storage to a research facility at Leeds University. The system was setup by a colleague who recently left and before he left he configured NFSV4 access which services an important subset, but not all the users who require access to the storage. SMB access was part of the deployment plan, but he didn't have time to get that working. I've been attempting to get an SMB share up and running myself and have enabled the SMB service with "mmces service enable SMB", have added a share with "mmsmb export add share /path/to/folder" and "mmsmb export list" seems to show everything is okay. However attempts to access the share through the floating protocol addresses fail and I have no idea how to diagnose the issue. Any advice/help would be welcome. -- Aidan Richmond Apple/Unix Support Officer, IT Garstang 10.137 Faculty of Biological Sciences University of Leeds Clarendon Way LS2 9JT Tel:0113 3434252 a.g.richmond at leeds.ac.uk From r.sobey at imperial.ac.uk Wed Mar 1 11:33:13 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Wed, 1 Mar 2017 11:33:13 +0000 Subject: [gpfsug-discuss] Issues getting SMB shares working. In-Reply-To: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> References: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> Message-ID: Have you configured authentication? Mmuserauth service list --data-access-method file -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aidan Richmond Sent: 01 March 2017 11:19 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Issues getting SMB shares working. Hello I've recently taken over managing a GPFS installation providing storage to a research facility at Leeds University. The system was setup by a colleague who recently left and before he left he configured NFSV4 access which services an important subset, but not all the users who require access to the storage. SMB access was part of the deployment plan, but he didn't have time to get that working. I've been attempting to get an SMB share up and running myself and have enabled the SMB service with "mmces service enable SMB", have added a share with "mmsmb export add share /path/to/folder" and "mmsmb export list" seems to show everything is okay. However attempts to access the share through the floating protocol addresses fail and I have no idea how to diagnose the issue. Any advice/help would be welcome. -- Aidan Richmond Apple/Unix Support Officer, IT Garstang 10.137 Faculty of Biological Sciences University of Leeds Clarendon Way LS2 9JT Tel:0113 3434252 a.g.richmond at leeds.ac.uk _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From a.g.richmond at leeds.ac.uk Wed Mar 1 11:36:04 2017 From: a.g.richmond at leeds.ac.uk (Aidan Richmond) Date: Wed, 1 Mar 2017 11:36:04 +0000 Subject: [gpfsug-discuss] Issues getting SMB shares working. In-Reply-To: References: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> Message-ID: <901c6472-1c9c-3551-555e-22bc849fe2b1@leeds.ac.uk> Hello It returns the following: FILE access configuration : USERDEFINED PARAMETERS VALUES ------------------------------------------------- On 01/03/17 11:33, Sobey, Richard A wrote: > Mmuserauth service list --data-access-method file -- Aidan Richmond Apple/Unix Support Officer, IT Garstang 10.137 Faculty of Biological Sciences University of Leeds Clarendon Way LS2 9JT Tel:0113 3434252 a.g.richmond at leeds.ac.uk From r.sobey at imperial.ac.uk Wed Mar 1 11:42:06 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Wed, 1 Mar 2017 11:42:06 +0000 Subject: [gpfsug-discuss] Issues getting SMB shares working. In-Reply-To: <901c6472-1c9c-3551-555e-22bc849fe2b1@leeds.ac.uk> References: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> <901c6472-1c9c-3551-555e-22bc849fe2b1@leeds.ac.uk> Message-ID: That's probably the first thing you need to sort out then. Check out the mmuserauth service create command. There was an example on this list on Monday so depending when you subscribed you may not have seen it. FYI the command cited was: mmuserauth service create --type ad --data-access-method file --servers 192.168.88.3 --user-name administrator --netbios-name scale --idmap-role master --password ********* --idmap-range-size 1000000 --idmap-range 10000000-299999999 --enable-nfs-kerberos --unixmap-domains 'sirius(10000-20000)' Change the parameters to cater to your environment and needs of course. Richard -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aidan Richmond Sent: 01 March 2017 11:36 To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Issues getting SMB shares working. Hello It returns the following: FILE access configuration : USERDEFINED PARAMETERS VALUES ------------------------------------------------- On 01/03/17 11:33, Sobey, Richard A wrote: > Mmuserauth service list --data-access-method file -- Aidan Richmond Apple/Unix Support Officer, IT Garstang 10.137 Faculty of Biological Sciences University of Leeds Clarendon Way LS2 9JT Tel:0113 3434252 a.g.richmond at leeds.ac.uk _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From a.g.richmond at leeds.ac.uk Wed Mar 1 12:07:28 2017 From: a.g.richmond at leeds.ac.uk (Aidan Richmond) Date: Wed, 1 Mar 2017 12:07:28 +0000 Subject: [gpfsug-discuss] Issues getting SMB shares working. In-Reply-To: References: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> <901c6472-1c9c-3551-555e-22bc849fe2b1@leeds.ac.uk> Message-ID: <2ac0911f-65f7-d500-1d01-7ba4b5929a79@leeds.ac.uk> Hello I'm a little hesitant to mess with the authentication as I we are wanting consistent UIDs across our systems and I know my predecessor struggled to get them consistent. Our AD environment stores the UID and GID settings in msSFU30uid and msSFU30gid. I'm also concerned that the system is already in use with nfsv4 access and don't want to break existing access unless I have to. On 01/03/17 11:42, Sobey, Richard A wrote: > That's probably the first thing you need to sort out then. > > Check out the mmuserauth service create command. > > There was an example on this list on Monday so depending when you subscribed you may not have seen it. FYI the command cited was: > > mmuserauth service create --type ad --data-access-method file --servers 192.168.88.3 --user-name administrator --netbios-name scale --idmap-role master --password ********* --idmap-range-size 1000000 --idmap-range 10000000-299999999 --enable-nfs-kerberos --unixmap-domains 'sirius(10000-20000)' > > Change the parameters to cater to your environment and needs of course. > > Richard > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aidan Richmond > Sent: 01 March 2017 11:36 > To: gpfsug-discuss at spectrumscale.org > Subject: Re: [gpfsug-discuss] Issues getting SMB shares working. > > Hello > > It returns the following: > > FILE access configuration : USERDEFINED > PARAMETERS VALUES > ------------------------------------------------- > > On 01/03/17 11:33, Sobey, Richard A wrote: >> Mmuserauth service list --data-access-method file > > -- Aidan Richmond Apple/Unix Support Officer, IT Garstang 10.137 Faculty of Biological Sciences University of Leeds Clarendon Way LS2 9JT Tel:0113 3434252 a.g.richmond at leeds.ac.uk From janfrode at tanso.net Wed Mar 1 14:12:04 2017 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Wed, 1 Mar 2017 15:12:04 +0100 Subject: [gpfsug-discuss] Issues getting SMB shares working. In-Reply-To: <2ac0911f-65f7-d500-1d01-7ba4b5929a79@leeds.ac.uk> References: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> <901c6472-1c9c-3551-555e-22bc849fe2b1@leeds.ac.uk> <2ac0911f-65f7-d500-1d01-7ba4b5929a79@leeds.ac.uk> Message-ID: Lets figure out how your NFS is authenticating then. The userdefined authentication you have, could mean that your linux host is configured to authenticated towards AD --- or it could be that you're using simple sys-authentication for NFS. Could you please post the output of: mmnfs export list mmnfs config list -jf On Wed, Mar 1, 2017 at 1:07 PM, Aidan Richmond wrote: > Hello > > I'm a little hesitant to mess with the authentication as I we are wanting > consistent UIDs across our systems and I know my predecessor struggled to > get them consistent. Our AD environment stores the UID and GID settings in > msSFU30uid and msSFU30gid. > > I'm also concerned that the system is already in use with nfsv4 access and > don't want to break existing access unless I have to. > > > > On 01/03/17 11:42, Sobey, Richard A wrote: > >> That's probably the first thing you need to sort out then. >> >> Check out the mmuserauth service create command. >> >> There was an example on this list on Monday so depending when you >> subscribed you may not have seen it. FYI the command cited was: >> >> mmuserauth service create --type ad --data-access-method file --servers >> 192.168.88.3 --user-name administrator --netbios-name scale --idmap-role >> master --password ********* --idmap-range-size 1000000 --idmap-range >> 10000000-299999999 --enable-nfs-kerberos --unixmap-domains >> 'sirius(10000-20000)' >> >> Change the parameters to cater to your environment and needs of course. >> >> Richard >> >> -----Original Message----- >> From: gpfsug-discuss-bounces at spectrumscale.org [mailto: >> gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aidan Richmond >> Sent: 01 March 2017 11:36 >> To: gpfsug-discuss at spectrumscale.org >> Subject: Re: [gpfsug-discuss] Issues getting SMB shares working. >> >> Hello >> >> It returns the following: >> >> FILE access configuration : USERDEFINED >> PARAMETERS VALUES >> ------------------------------------------------- >> >> On 01/03/17 11:33, Sobey, Richard A wrote: >> >>> Mmuserauth service list --data-access-method file >>> >> >> >> > > -- > Aidan Richmond > Apple/Unix Support Officer, IT > Garstang 10.137 > Faculty of Biological Sciences > University of Leeds > Clarendon Way > LS2 9JT > > Tel:0113 3434252 > a.g.richmond at leeds.ac.uk > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.g.richmond at leeds.ac.uk Wed Mar 1 15:22:36 2017 From: a.g.richmond at leeds.ac.uk (Aidan Richmond) Date: Wed, 1 Mar 2017 15:22:36 +0000 Subject: [gpfsug-discuss] Issues getting SMB shares working. In-Reply-To: References: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> <901c6472-1c9c-3551-555e-22bc849fe2b1@leeds.ac.uk> <2ac0911f-65f7-d500-1d01-7ba4b5929a79@leeds.ac.uk> Message-ID: mmnfs export list Path Delegations Clients ------------------------------------- /absl/SCRATCH none * /absl/SCRATCH none fbscpcu097 mmnfs config list NFS Ganesha Configuration: ========================== NFS_PROTOCOLS: 3,4 NFS_PORT: 2049 MNT_PORT: 0 NLM_PORT: 0 RQUOTA_PORT: 0 SHORT_FILE_HANDLE: FALSE LEASE_LIFETIME: 60 DOMAINNAME: LEEDS.AC.UK DELEGATIONS: Disabled ========================== STATD Configuration ========================== STATD_PORT: 0 ========================== CacheInode Configuration ========================== ENTRIES_HWMARK: 1500000 ========================== Export Defaults ========================== ACCESS_TYPE: NONE PROTOCOLS: 3,4 TRANSPORTS: TCP ANONYMOUS_UID: -2 ANONYMOUS_GID: -2 SECTYPE: SYS PRIVILEGEDPORT: FALSE MANAGE_GIDS: FALSE SQUASH: ROOT_SQUASH NFS_COMMIT: FALSE ========================== Log Configuration ========================== LOG_LEVEL: EVENT ========================== Idmapd Configuration ========================== DOMAIN: DS.LEEDS.AC.UK ========================== On 01/03/17 14:12, Jan-Frode Myklebust wrote: > Lets figure out how your NFS is authenticating then. The userdefined > authentication you have, could mean that your linux host is configured to > authenticated towards AD --- or it could be that you're using simple > sys-authentication for NFS. > > Could you please post the output of: > > mmnfs export list > mmnfs config list > > > -jf > > > On Wed, Mar 1, 2017 at 1:07 PM, Aidan Richmond > wrote: > >> Hello >> >> I'm a little hesitant to mess with the authentication as I we are wanting >> consistent UIDs across our systems and I know my predecessor struggled to >> get them consistent. Our AD environment stores the UID and GID settings in >> msSFU30uid and msSFU30gid. >> >> I'm also concerned that the system is already in use with nfsv4 access and >> don't want to break existing access unless I have to. >> >> >> >> On 01/03/17 11:42, Sobey, Richard A wrote: >> >>> That's probably the first thing you need to sort out then. >>> >>> Check out the mmuserauth service create command. >>> >>> There was an example on this list on Monday so depending when you >>> subscribed you may not have seen it. FYI the command cited was: >>> >>> mmuserauth service create --type ad --data-access-method file --servers >>> 192.168.88.3 --user-name administrator --netbios-name scale --idmap-role >>> master --password ********* --idmap-range-size 1000000 --idmap-range >>> 10000000-299999999 --enable-nfs-kerberos --unixmap-domains >>> 'sirius(10000-20000)' >>> >>> Change the parameters to cater to your environment and needs of course. >>> >>> Richard >>> >>> -----Original Message----- >>> From: gpfsug-discuss-bounces at spectrumscale.org [mailto: >>> gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aidan Richmond >>> Sent: 01 March 2017 11:36 >>> To: gpfsug-discuss at spectrumscale.org >>> Subject: Re: [gpfsug-discuss] Issues getting SMB shares working. >>> >>> Hello >>> >>> It returns the following: >>> >>> FILE access configuration : USERDEFINED >>> PARAMETERS VALUES >>> ------------------------------------------------- >>> >>> On 01/03/17 11:33, Sobey, Richard A wrote: >>> >>>> Mmuserauth service list --data-access-method file >>>> >>> >>> >>> >> >> -- >> Aidan Richmond >> Apple/Unix Support Officer, IT >> Garstang 10.137 >> Faculty of Biological Sciences >> University of Leeds >> Clarendon Way >> LS2 9JT >> >> Tel:0113 3434252 >> a.g.richmond at leeds.ac.uk >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aidan Richmond Apple/Unix Support Officer, IT Garstang 10.137 Faculty of Biological Sciences University of Leeds Clarendon Way LS2 9JT Tel:0113 3434252 a.g.richmond at leeds.ac.uk From Mark.Bush at siriuscom.com Wed Mar 1 19:28:24 2017 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Wed, 1 Mar 2017 19:28:24 +0000 Subject: [gpfsug-discuss] Issues getting SMB shares working. In-Reply-To: References: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> <901c6472-1c9c-3551-555e-22bc849fe2b1@leeds.ac.uk> <2ac0911f-65f7-d500-1d01-7ba4b5929a79@leeds.ac.uk> Message-ID: I?m pretty noob still here but I think what may be going on in your situation is that since you are just now enabling SMB service you need to figure out how you want users to authenticate. If you want to stick with the way things are currently authenticating on your system that is to say, USERDEFINED that really just means it?s delegated to the underlying system for authentication. So, you can change your authentication strategy to use Active Directory (like was mentioned by Richard), or use the local SMB authentication which means you?ll have to add users manually with the /usr/lpp/mmfs/bin/smbpasswd command. This works if you just want a few users to get access to SMB shares but it becomes troublesome when you get loads of users. You can use LDAP or AD to make things more manageable. Mark On 3/1/17, 9:22 AM, "Aidan Richmond" wrote: mmnfs export list Path Delegations Clients ------------------------------------- /absl/SCRATCH none * /absl/SCRATCH none fbscpcu097 mmnfs config list NFS Ganesha Configuration: ========================== NFS_PROTOCOLS: 3,4 NFS_PORT: 2049 MNT_PORT: 0 NLM_PORT: 0 RQUOTA_PORT: 0 SHORT_FILE_HANDLE: FALSE LEASE_LIFETIME: 60 DOMAINNAME: LEEDS.AC.UK DELEGATIONS: Disabled ========================== STATD Configuration ========================== STATD_PORT: 0 ========================== CacheInode Configuration ========================== ENTRIES_HWMARK: 1500000 ========================== Export Defaults ========================== ACCESS_TYPE: NONE PROTOCOLS: 3,4 TRANSPORTS: TCP ANONYMOUS_UID: -2 ANONYMOUS_GID: -2 SECTYPE: SYS PRIVILEGEDPORT: FALSE MANAGE_GIDS: FALSE SQUASH: ROOT_SQUASH NFS_COMMIT: FALSE ========================== Log Configuration ========================== LOG_LEVEL: EVENT ========================== Idmapd Configuration ========================== DOMAIN: DS.LEEDS.AC.UK ========================== On 01/03/17 14:12, Jan-Frode Myklebust wrote: > Lets figure out how your NFS is authenticating then. The userdefined > authentication you have, could mean that your linux host is configured to > authenticated towards AD --- or it could be that you're using simple > sys-authentication for NFS. > > Could you please post the output of: > > mmnfs export list > mmnfs config list > > > -jf > > > On Wed, Mar 1, 2017 at 1:07 PM, Aidan Richmond > wrote: > >> Hello >> >> I'm a little hesitant to mess with the authentication as I we are wanting >> consistent UIDs across our systems and I know my predecessor struggled to >> get them consistent. Our AD environment stores the UID and GID settings in >> msSFU30uid and msSFU30gid. >> >> I'm also concerned that the system is already in use with nfsv4 access and >> don't want to break existing access unless I have to. >> >> >> >> On 01/03/17 11:42, Sobey, Richard A wrote: >> >>> That's probably the first thing you need to sort out then. >>> >>> Check out the mmuserauth service create command. >>> >>> There was an example on this list on Monday so depending when you >>> subscribed you may not have seen it. FYI the command cited was: >>> >>> mmuserauth service create --type ad --data-access-method file --servers >>> 192.168.88.3 --user-name administrator --netbios-name scale --idmap-role >>> master --password ********* --idmap-range-size 1000000 --idmap-range >>> 10000000-299999999 --enable-nfs-kerberos --unixmap-domains >>> 'sirius(10000-20000)' >>> >>> Change the parameters to cater to your environment and needs of course. >>> >>> Richard >>> >>> -----Original Message----- >>> From: gpfsug-discuss-bounces at spectrumscale.org [mailto: >>> gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aidan Richmond >>> Sent: 01 March 2017 11:36 >>> To: gpfsug-discuss at spectrumscale.org >>> Subject: Re: [gpfsug-discuss] Issues getting SMB shares working. >>> >>> Hello >>> >>> It returns the following: >>> >>> FILE access configuration : USERDEFINED >>> PARAMETERS VALUES >>> ------------------------------------------------- >>> >>> On 01/03/17 11:33, Sobey, Richard A wrote: >>> >>>> Mmuserauth service list --data-access-method file >>>> >>> >>> >>> >> >> -- >> Aidan Richmond >> Apple/Unix Support Officer, IT >> Garstang 10.137 >> Faculty of Biological Sciences >> University of Leeds >> Clarendon Way >> LS2 9JT >> >> Tel:0113 3434252 >> a.g.richmond at leeds.ac.uk >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aidan Richmond Apple/Unix Support Officer, IT Garstang 10.137 Faculty of Biological Sciences University of Leeds Clarendon Way LS2 9JT Tel:0113 3434252 a.g.richmond at leeds.ac.uk _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions From janfrode at tanso.net Wed Mar 1 20:21:03 2017 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Wed, 01 Mar 2017 20:21:03 +0000 Subject: [gpfsug-discuss] Issues getting SMB shares working. In-Reply-To: References: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> <901c6472-1c9c-3551-555e-22bc849fe2b1@leeds.ac.uk> <2ac0911f-65f7-d500-1d01-7ba4b5929a79@leeds.ac.uk> Message-ID: This looks to me like a quite plain SYS authorized NFS, maybe also verify that the individual NFS shares has sec=sys with "mmnfs export list -n /absl/SCRATCH". If you check "man mmuserauth" there are quite clear examples for how to connect it to AD populated with unix id and gid. I don't think this will affect your NFS service, since there doesn't seem to be any kerberos involved. But, please beware that mmuserauth will overwrite any customized sssd.conf, krb5.conf, winbind, etc.. As it configures the authentication for the whole host, not just samba/nfs-services. -jf ons. 1. mar. 2017 kl. 16.22 skrev Aidan Richmond : > mmnfs export list > Path Delegations Clients > ------------------------------------- > /absl/SCRATCH none * > /absl/SCRATCH none fbscpcu097 > > mmnfs config list > > NFS Ganesha Configuration: > ========================== > NFS_PROTOCOLS: 3,4 > NFS_PORT: 2049 > MNT_PORT: 0 > NLM_PORT: 0 > RQUOTA_PORT: 0 > SHORT_FILE_HANDLE: FALSE > LEASE_LIFETIME: 60 > DOMAINNAME: LEEDS.AC.UK > DELEGATIONS: Disabled > ========================== > > STATD Configuration > ========================== > STATD_PORT: 0 > ========================== > > CacheInode Configuration > ========================== > ENTRIES_HWMARK: 1500000 > ========================== > > Export Defaults > ========================== > ACCESS_TYPE: NONE > PROTOCOLS: 3,4 > TRANSPORTS: TCP > ANONYMOUS_UID: -2 > ANONYMOUS_GID: -2 > SECTYPE: SYS > PRIVILEGEDPORT: FALSE > MANAGE_GIDS: FALSE > SQUASH: ROOT_SQUASH > NFS_COMMIT: FALSE > ========================== > > Log Configuration > ========================== > LOG_LEVEL: EVENT > ========================== > > Idmapd Configuration > ========================== > DOMAIN: DS.LEEDS.AC.UK > ========================== > > On 01/03/17 14:12, Jan-Frode Myklebust wrote: > > Lets figure out how your NFS is authenticating then. The userdefined > > authentication you have, could mean that your linux host is configured to > > authenticated towards AD --- or it could be that you're using simple > > sys-authentication for NFS. > > > > Could you please post the output of: > > > > mmnfs export list > > mmnfs config list > > > > > > -jf > > > > > > On Wed, Mar 1, 2017 at 1:07 PM, Aidan Richmond > > > wrote: > > > >> Hello > >> > >> I'm a little hesitant to mess with the authentication as I we are > wanting > >> consistent UIDs across our systems and I know my predecessor struggled > to > >> get them consistent. Our AD environment stores the UID and GID settings > in > >> msSFU30uid and msSFU30gid. > >> > >> I'm also concerned that the system is already in use with nfsv4 access > and > >> don't want to break existing access unless I have to. > >> > >> > >> > >> On 01/03/17 11:42, Sobey, Richard A wrote: > >> > >>> That's probably the first thing you need to sort out then. > >>> > >>> Check out the mmuserauth service create command. > >>> > >>> There was an example on this list on Monday so depending when you > >>> subscribed you may not have seen it. FYI the command cited was: > >>> > >>> mmuserauth service create --type ad --data-access-method file --servers > >>> 192.168.88.3 --user-name administrator --netbios-name scale > --idmap-role > >>> master --password ********* --idmap-range-size 1000000 --idmap-range > >>> 10000000-299999999 --enable-nfs-kerberos --unixmap-domains > >>> 'sirius(10000-20000)' > >>> > >>> Change the parameters to cater to your environment and needs of course. > >>> > >>> Richard > >>> > >>> -----Original Message----- > >>> From: gpfsug-discuss-bounces at spectrumscale.org [mailto: > >>> gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aidan Richmond > >>> Sent: 01 March 2017 11:36 > >>> To: gpfsug-discuss at spectrumscale.org > >>> Subject: Re: [gpfsug-discuss] Issues getting SMB shares working. > >>> > >>> Hello > >>> > >>> It returns the following: > >>> > >>> FILE access configuration : USERDEFINED > >>> PARAMETERS VALUES > >>> ------------------------------------------------- > >>> > >>> On 01/03/17 11:33, Sobey, Richard A wrote: > >>> > >>>> Mmuserauth service list --data-access-method file > >>>> > >>> > >>> > >>> > >> > >> -- > >> Aidan Richmond > >> Apple/Unix Support Officer, IT > >> Garstang 10.137 > >> Faculty of Biological Sciences > >> University of Leeds > >> Clarendon Way > >> LS2 9JT > >> > >> Tel:0113 3434252 > >> a.g.richmond at leeds.ac.uk > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > -- > Aidan Richmond > Apple/Unix Support Officer, IT > Garstang 10.137 > Faculty of Biological Sciences > University of Leeds > Clarendon Way > LS2 9JT > > Tel:0113 3434252 > a.g.richmond at leeds.ac.uk > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Thu Mar 2 14:17:45 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Thu, 2 Mar 2017 14:17:45 +0000 Subject: [gpfsug-discuss] GPFSUG-USA Meeting at NERSC April 4-5th - Registration Reminder and Agenda Details Message-ID: Here?s a reminder to register for the upcoming user group meeting! Registration deadline is March 24th, only 3 weeks away. You won?t want to miss the 2 hour session on Tuesday on problem determination by Yuri Volobuev. Any long time GPFS admin has no doubt interacted with Yuri. He?s moved on to new challenges from GPFS, but has a wealth of problem determination tips that he wants to pass along. We?re happy to have him back in the GPFS community. We also have a few slots (20 mins each) for customer and partner talks ? please contact me if you want to do a short presentation. Register here: https://www.nersc.gov/research-and-development/data-analytics/spectrum-user-group-meeting/ Tuesday, April 4th 09:00-10:00 Registration/Coffee/Networking 10:00-10:30 Welcome - Bob Oesterlin, Kristy Kallback, Doris Conti 10:30-11:30 Spectrum Scale & ESS Update - Scott Fadden / Puneet Chaudhary 11:30-12:30 Problem Avoidance - Best Practices in the Field - IBM Support 12:30-13:30 Lunch and Networking 13:30-15:30 Problem Determination - Deep Dive - Yuri Volobuev 15:30-16:00 Break/Networking 16:00-17:00 Problem Determination - What's New? - Matthias Dietz 17:00-18:00 Bring your favorite problem or performance issue. We will fix it now! Wednesday April 5th 08:30-09:00 Coffee and Networking 09:00-10:00 Transparent Cloud Tiering - Introduction and Use Cases (Meet the Devs) - Nikhil Khandelwal or Rob Basham Life Sciences and Next Generation Sequencing with Spectrum Scale (Solutions) 10:00-11:00 AFM - Introduction and Use Cases (Meet the Devs) - Scott Fadden Spectrum Protect on Spectrum Scale (Solutions) - Jason Basler 11:00-12:00 Meet the Devs - IBM Development Hadoop Integration and Support for HortonWorks HDP (Solutions) - Ted Hoover 12:00-13:00 Lunch and Networking 13:00-13:20 Customer/Partner Talk 13:20-13:40 Customer/Partner Talk 13:40-14:00 Customer/Partner Talk 14:00-14:30 Break 14:30-15:00 Challenges in Tracing - Vasily Tarasov 15:00-15:30 Application Integration with Containers - Dean Hildebrand 15:30-16:00 Wrap up - Bob Oesterlin, Kristy Kallback, Doris Conti Bob Oesterlin Sr Principal Storage Engineer, Nuance Spectrum Scale UG-USA Co-Principal -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.g.richmond at leeds.ac.uk Thu Mar 2 15:41:40 2017 From: a.g.richmond at leeds.ac.uk (Aidan Richmond) Date: Thu, 2 Mar 2017 15:41:40 +0000 Subject: [gpfsug-discuss] Issues getting SMB shares working. In-Reply-To: References: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> <901c6472-1c9c-3551-555e-22bc849fe2b1@leeds.ac.uk> <2ac0911f-65f7-d500-1d01-7ba4b5929a79@leeds.ac.uk> Message-ID: <97b56c55-58d4-3400-f8cf-5f6745583fc9@leeds.ac.uk> mmnfs export list -n /absl/SCRATCH Path Delegations Clients Access_Type Protocols Transports Squash Anonymous_uid Anonymous_gid SecType PrivilegedPort DefaultDelegations Manage_Gids NFS_Commit ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- /absl/SCRATCH none * RW 4 TCP ROOT_SQUASH -2 -2 KRB5I FALSE none FALSE FALSE /absl/SCRATCH none fbscpcu097 RW 3 TCP ROOT_SQUASH -2 -2 sys FALSE none FALSE FALSE Ignore the fbscpcu097 entry, I think my departed colleague was just using that for testing, the NFS clients all access it though nfsv4 which looks to be using kerberos from this. On 01/03/17 20:21, Jan-Frode Myklebust wrote: > This looks to me like a quite plain SYS authorized NFS, maybe also verify > that the individual NFS shares has sec=sys with "mmnfs export list -n > /absl/SCRATCH". > > > If you check "man mmuserauth" there are quite clear examples for how to > connect it to AD populated with unix id and gid. I don't think this will > affect your NFS service, since there doesn't seem to be any kerberos > involved. > > But, please beware that mmuserauth will overwrite any customized sssd.conf, > krb5.conf, winbind, etc.. As it configures the authentication for the whole > host, not just samba/nfs-services. > > > -jf > > ons. 1. mar. 2017 kl. 16.22 skrev Aidan Richmond : > >> mmnfs export list >> Path Delegations Clients >> ------------------------------------- >> /absl/SCRATCH none * >> /absl/SCRATCH none fbscpcu097 >> >> mmnfs config list >> >> NFS Ganesha Configuration: >> ========================== >> NFS_PROTOCOLS: 3,4 >> NFS_PORT: 2049 >> MNT_PORT: 0 >> NLM_PORT: 0 >> RQUOTA_PORT: 0 >> SHORT_FILE_HANDLE: FALSE >> LEASE_LIFETIME: 60 >> DOMAINNAME: LEEDS.AC.UK >> DELEGATIONS: Disabled >> ========================== >> >> STATD Configuration >> ========================== >> STATD_PORT: 0 >> ========================== >> >> CacheInode Configuration >> ========================== >> ENTRIES_HWMARK: 1500000 >> ========================== >> >> Export Defaults >> ========================== >> ACCESS_TYPE: NONE >> PROTOCOLS: 3,4 >> TRANSPORTS: TCP >> ANONYMOUS_UID: -2 >> ANONYMOUS_GID: -2 >> SECTYPE: SYS >> PRIVILEGEDPORT: FALSE >> MANAGE_GIDS: FALSE >> SQUASH: ROOT_SQUASH >> NFS_COMMIT: FALSE >> ========================== >> >> Log Configuration >> ========================== >> LOG_LEVEL: EVENT >> ========================== >> >> Idmapd Configuration >> ========================== >> DOMAIN: DS.LEEDS.AC.UK >> ========================== >> -- Aidan Richmond Apple/Unix Support Officer, IT Garstang 10.137 Faculty of Biological Sciences University of Leeds Clarendon Way LS2 9JT Tel:0113 3434252 a.g.richmond at leeds.ac.uk From p.childs at qmul.ac.uk Thu Mar 2 16:34:09 2017 From: p.childs at qmul.ac.uk (Peter Childs) Date: Thu, 2 Mar 2017 16:34:09 +0000 Subject: [gpfsug-discuss] mmbackup logging issue In-Reply-To: References: Message-ID: We had that issue. we had to export MMBACKUP_PROGRESS_CONTENT=5 export MMBACKUP_PROGRESS_INTERVAL=300 before we run it to get it back. Lets just say IBM changed the behaviour, We ended up opening a PRM to get that answer ;) We also set -L 1 you can change how often the messages are displayed by changing MMBACKUP_PROGRESS_INTERVAL flexable but the default is different;) Peter Childs ITS Research Infrastructure Queen Mary, University of London ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Ashish Thandavan Sent: Tuesday, February 28, 2017 4:10:44 PM To: gpfsug main discussion list Subject: [gpfsug-discuss] mmbackup logging issue Dear all, We have a small GPFS cluster and a separate server running TSM and one of the three NSD servers backs up our GPFS filesystem to the TSM server using mmbackup. After a recent upgrade from v3.5 to 4.1.1, we've noticed that mmbackup no longer logs stuff like it used to : ... Thu Jan 19 05:45:41 2017 mmbackup:Backing up files: 0 backed up, 870532 expired, 2 failed. Thu Jan 19 06:15:41 2017 mmbackup:Backing up files: 0 backed up, 870532 expired, 3 failed. Thu Jan 19 06:45:41 2017 mmbackup:Backing up files: 0 backed up, 870532 expired, 3 failed. ... instead of ... Sat Dec 3 12:01:00 2016 mmbackup:Backing up files: 105030 backed up, 635456 expired, 30 failed. Sat Dec 3 12:31:00 2016 mmbackup:Backing up files: 205934 backed up, 635456 expired, 57 failed. Sat Dec 3 13:01:00 2016 mmbackup:Backing up files: 321702 backed up, 635456 expired, 169 failed. ... like it used to pre-upgrade. I am therefore unable to see how far long it has got, and indeed if it completed successfully, as this is what it logs at the end of a job : ... Tue Jan 17 18:07:31 2017 mmbackup:Completed policy backup run with 0 policy errors, 10012 files failed, 0 severe errors, returning rc=9. Tue Jan 17 18:07:31 2017 mmbackup:Policy for backup returned 9 Highest TSM error 12 mmbackup: TSM Summary Information: Total number of objects inspected: 20617273 Total number of objects backed up: 0 Total number of objects updated: 0 Total number of objects rebound: 0 Total number of objects deleted: 0 Total number of objects expired: 1 Total number of objects failed: 10012 Total number of objects encrypted: 0 Total number of bytes inspected: 3821624716861 Total number of bytes transferred: 3712040943672 Tue Jan 17 18:07:31 2017 mmbackup:Audit files /cs/mmbackup.audit.gpfs* contain 0 failed paths but there were 10012 failures. Cannot reconcile shadow database. Unable to compensate for all TSM errors in new shadow database. Preserving previous shadow database. Run next mmbackup with -q to synchronize shadow database. exit 12 If it helps, the mmbackup job is kicked off with the following options : /usr/lpp/mmfs/bin/mmbackup gpfs -n 8 -t full -B 20000 -L 1 --tsm-servers gpfs_weekly_stanza -N glossop1a | /usr/bin/tee /var/log/mmbackup/gpfs_weekly/backup_log.`date +%Y%m%d_%H_%M` (The excerpts above are from the backup_log. file.) Our NSD servers are running GPFS 4.1.1-11, TSM is at 7.1.1.100 and the File system version is 12.06 (3.4.0.3). Has anyone else seen this behaviour with mmbackup and if so, found a fix? Thanks, Regards, Ash -- ------------------------- Ashish Thandavan UNIX Support Computing Officer Department of Computer Science University of Oxford Wolfson Building Parks Road Oxford OX1 3QD Phone: 01865 610733 Email: ashish.thandavan at cs.ox.ac.uk _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From ashish.thandavan at cs.ox.ac.uk Thu Mar 2 16:50:23 2017 From: ashish.thandavan at cs.ox.ac.uk (Ashish Thandavan) Date: Thu, 2 Mar 2017 16:50:23 +0000 Subject: [gpfsug-discuss] mmbackup logging issue In-Reply-To: References: Message-ID: <6cd84459-ecc3-1ef9-da22-dea843cff154@cs.ox.ac.uk> Dear Peter, On 02/03/17 16:34, Peter Childs wrote: > We had that issue. > > we had to > > export MMBACKUP_PROGRESS_CONTENT=5 > export MMBACKUP_PROGRESS_INTERVAL=300 > > before we run it to get it back. > > Lets just say IBM changed the behaviour, We ended up opening a PRM to get that answer ;) We also set -L 1 > > you can change how often the messages are displayed by changing MMBACKUP_PROGRESS_INTERVAL flexable but the default is different;) > I'll set those variables before kicking off the next mmbackup and hope that fixes it. Thank you!! Regards, Ash -- ------------------------- Ashish Thandavan UNIX Support Computing Officer Department of Computer Science University of Oxford Wolfson Building Parks Road Oxford OX1 3QD Phone: 01865 610733 Email: ashish.thandavan at cs.ox.ac.uk From janfrode at tanso.net Thu Mar 2 20:42:23 2017 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Thu, 02 Mar 2017 20:42:23 +0000 Subject: [gpfsug-discuss] Issues getting SMB shares working. In-Reply-To: <97b56c55-58d4-3400-f8cf-5f6745583fc9@leeds.ac.uk> References: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> <901c6472-1c9c-3551-555e-22bc849fe2b1@leeds.ac.uk> <2ac0911f-65f7-d500-1d01-7ba4b5929a79@leeds.ac.uk> <97b56c55-58d4-3400-f8cf-5f6745583fc9@leeds.ac.uk> Message-ID: Ouch, yes.. Then the switch to mmuserauth is more difficult. I would recommend setting up a lab cluster (only need a single node), and use mmuserauth to connect it to AD and see that you get both kerberized NFS and SMB working by default there, before doing the same on your production cluster. -jf tor. 2. mar. 2017 kl. 16.42 skrev Aidan Richmond : > mmnfs export list -n /absl/SCRATCH > Path Delegations Clients Access_Type Protocols Transports > Squash Anonymous_uid Anonymous_gid SecType PrivilegedPort > DefaultDelegations Manage_Gids NFS_Commit > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > /absl/SCRATCH none * RW 4 TCP > ROOT_SQUASH -2 -2 KRB5I FALSE none > FALSE FALSE > /absl/SCRATCH none fbscpcu097 RW 3 TCP > ROOT_SQUASH -2 -2 sys FALSE none > FALSE FALSE > > > Ignore the fbscpcu097 entry, I think my departed colleague was just > using that for testing, the NFS clients all access it though nfsv4 which > looks to be using kerberos from this. > > On 01/03/17 20:21, Jan-Frode Myklebust wrote: > > This looks to me like a quite plain SYS authorized NFS, maybe also verify > > that the individual NFS shares has sec=sys with "mmnfs export list -n > > /absl/SCRATCH". > > > > > > If you check "man mmuserauth" there are quite clear examples for how to > > connect it to AD populated with unix id and gid. I don't think this will > > affect your NFS service, since there doesn't seem to be any kerberos > > involved. > > > > But, please beware that mmuserauth will overwrite any customized > sssd.conf, > > krb5.conf, winbind, etc.. As it configures the authentication for the > whole > > host, not just samba/nfs-services. > > > > > > -jf > > > > ons. 1. mar. 2017 kl. 16.22 skrev Aidan Richmond < > a.g.richmond at leeds.ac.uk>: > > > >> mmnfs export list > >> Path Delegations Clients > >> ------------------------------------- > >> /absl/SCRATCH none * > >> /absl/SCRATCH none fbscpcu097 > >> > >> mmnfs config list > >> > >> NFS Ganesha Configuration: > >> ========================== > >> NFS_PROTOCOLS: 3,4 > >> NFS_PORT: 2049 > >> MNT_PORT: 0 > >> NLM_PORT: 0 > >> RQUOTA_PORT: 0 > >> SHORT_FILE_HANDLE: FALSE > >> LEASE_LIFETIME: 60 > >> DOMAINNAME: LEEDS.AC.UK > >> DELEGATIONS: Disabled > >> ========================== > >> > >> STATD Configuration > >> ========================== > >> STATD_PORT: 0 > >> ========================== > >> > >> CacheInode Configuration > >> ========================== > >> ENTRIES_HWMARK: 1500000 > >> ========================== > >> > >> Export Defaults > >> ========================== > >> ACCESS_TYPE: NONE > >> PROTOCOLS: 3,4 > >> TRANSPORTS: TCP > >> ANONYMOUS_UID: -2 > >> ANONYMOUS_GID: -2 > >> SECTYPE: SYS > >> PRIVILEGEDPORT: FALSE > >> MANAGE_GIDS: FALSE > >> SQUASH: ROOT_SQUASH > >> NFS_COMMIT: FALSE > >> ========================== > >> > >> Log Configuration > >> ========================== > >> LOG_LEVEL: EVENT > >> ========================== > >> > >> Idmapd Configuration > >> ========================== > >> DOMAIN: DS.LEEDS.AC.UK > >> ========================== > >> > > -- > Aidan Richmond > Apple/Unix Support Officer, IT > Garstang 10.137 > Faculty of Biological Sciences > University of Leeds > Clarendon Way > LS2 9JT > > Tel:0113 3434252 > a.g.richmond at leeds.ac.uk > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Fri Mar 3 09:20:24 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Fri, 3 Mar 2017 09:20:24 +0000 Subject: [gpfsug-discuss] mmbackup logging issue In-Reply-To: <6cd84459-ecc3-1ef9-da22-dea843cff154@cs.ox.ac.uk> References: <6cd84459-ecc3-1ef9-da22-dea843cff154@cs.ox.ac.uk> Message-ID: HI all We have the same problem (less of a problem, more lack of visibilitiy). Can I just add those lines to the top of our mmbackup.sh script? -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Ashish Thandavan Sent: 02 March 2017 16:50 To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] mmbackup logging issue Dear Peter, On 02/03/17 16:34, Peter Childs wrote: > We had that issue. > > we had to > > export MMBACKUP_PROGRESS_CONTENT=5 > export MMBACKUP_PROGRESS_INTERVAL=300 > > before we run it to get it back. > > Lets just say IBM changed the behaviour, We ended up opening a PRM to > get that answer ;) We also set -L 1 > > you can change how often the messages are displayed by changing > MMBACKUP_PROGRESS_INTERVAL flexable but the default is different;) > I'll set those variables before kicking off the next mmbackup and hope that fixes it. Thank you!! Regards, Ash -- ------------------------- Ashish Thandavan UNIX Support Computing Officer Department of Computer Science University of Oxford Wolfson Building Parks Road Oxford OX1 3QD Phone: 01865 610733 Email: ashish.thandavan at cs.ox.ac.uk _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From p.childs at qmul.ac.uk Fri Mar 3 10:44:03 2017 From: p.childs at qmul.ac.uk (Peter Childs) Date: Fri, 3 Mar 2017 10:44:03 +0000 Subject: [gpfsug-discuss] mmbackup logging issue In-Reply-To: References: <6cd84459-ecc3-1ef9-da22-dea843cff154@cs.ox.ac.uk>, Message-ID: That's basically what we did, They are only environment variables, so if you not using bash to call mmbackup you will need to change the lines accordingly. What they do is in the manual the issue is the default changed between versions..... Peter Childs ITS Research Infrastructure Queen Mary, University of London ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Sobey, Richard A Sent: Friday, March 3, 2017 9:20:24 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] mmbackup logging issue HI all We have the same problem (less of a problem, more lack of visibilitiy). Can I just add those lines to the top of our mmbackup.sh script? -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Ashish Thandavan Sent: 02 March 2017 16:50 To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] mmbackup logging issue Dear Peter, On 02/03/17 16:34, Peter Childs wrote: > We had that issue. > > we had to > > export MMBACKUP_PROGRESS_CONTENT=5 > export MMBACKUP_PROGRESS_INTERVAL=300 > > before we run it to get it back. > > Lets just say IBM changed the behaviour, We ended up opening a PRM to > get that answer ;) We also set -L 1 > > you can change how often the messages are displayed by changing > MMBACKUP_PROGRESS_INTERVAL flexable but the default is different;) > I'll set those variables before kicking off the next mmbackup and hope that fixes it. Thank you!! Regards, Ash -- ------------------------- Ashish Thandavan UNIX Support Computing Officer Department of Computer Science University of Oxford Wolfson Building Parks Road Oxford OX1 3QD Phone: 01865 610733 Email: ashish.thandavan at cs.ox.ac.uk _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From jtucker at pixitmedia.com Fri Mar 3 12:59:48 2017 From: jtucker at pixitmedia.com (Jez Tucker) Date: Fri, 3 Mar 2017 12:59:48 +0000 Subject: [gpfsug-discuss] mmbackup logging issue In-Reply-To: References: Message-ID: <7ca99092-63e6-c1a1-d9a7-e1907bcef7c8@pixitmedia.com> Hi Ash, This is the primary reason for using snapshots for mmbackup ( -S snapname ) and making sure that only mmbackup sends data to backup rather than an oob dsmc: Tue Jan 17 18:07:31 2017 mmbackup:Audit files /cs/mmbackup.audit.gpfs* contain 0 failed paths but there were 10012 failures. Cannot reconcile shadow database. Unable to compensate for all TSM errors in new shadow database. Preserving previous shadow database. Run next mmbackup with -q to synchronize shadow database. exit 12 <------------ ick! Other obvious causes are very, very, odd filenames / paths. Jez On 28/02/17 16:10, Ashish Thandavan wrote: > Dear all, > > We have a small GPFS cluster and a separate server running TSM and one > of the three NSD servers backs up our GPFS filesystem to the TSM > server using mmbackup. After a recent upgrade from v3.5 to 4.1.1, > we've noticed that mmbackup no longer logs stuff like it used to : > > ... > Thu Jan 19 05:45:41 2017 mmbackup:Backing up files: 0 backed up, > 870532 expired, 2 failed. > Thu Jan 19 06:15:41 2017 mmbackup:Backing up files: 0 backed up, > 870532 expired, 3 failed. > Thu Jan 19 06:45:41 2017 mmbackup:Backing up files: 0 backed up, > 870532 expired, 3 failed. > ... > > > instead of > > ... > Sat Dec 3 12:01:00 2016 mmbackup:Backing up files: 105030 backed up, > 635456 expired, 30 failed. > Sat Dec 3 12:31:00 2016 mmbackup:Backing up files: 205934 backed up, > 635456 expired, 57 failed. > Sat Dec 3 13:01:00 2016 mmbackup:Backing up files: 321702 backed up, > 635456 expired, 169 failed. > ... > > like it used to pre-upgrade. > > I am therefore unable to see how far long it has got, and indeed if it > completed successfully, as this is what it logs at the end of a job : > > ... > Tue Jan 17 18:07:31 2017 mmbackup:Completed policy backup run with 0 > policy errors, 10012 files failed, 0 severe errors, returning rc=9. > Tue Jan 17 18:07:31 2017 mmbackup:Policy for backup returned 9 Highest > TSM error 12 > mmbackup: TSM Summary Information: > Total number of objects inspected: 20617273 > Total number of objects backed up: 0 > Total number of objects updated: 0 > Total number of objects rebound: 0 > Total number of objects deleted: 0 > Total number of objects expired: 1 > Total number of objects failed: 10012 > Total number of objects encrypted: 0 > Total number of bytes inspected: 3821624716861 > Total number of bytes transferred: 3712040943672 > Tue Jan 17 18:07:31 2017 mmbackup:Audit files /cs/mmbackup.audit.gpfs* > contain 0 failed paths but there were 10012 failures. > Cannot reconcile shadow database. > Unable to compensate for all TSM errors in new shadow database. > Preserving previous shadow database. > Run next mmbackup with -q to synchronize shadow database. exit 12 > > If it helps, the mmbackup job is kicked off with the following options : > /usr/lpp/mmfs/bin/mmbackup gpfs -n 8 -t full -B 20000 -L 1 > --tsm-servers gpfs_weekly_stanza -N glossop1a | /usr/bin/tee > /var/log/mmbackup/gpfs_weekly/backup_log.`date +%Y%m%d_%H_%M` > > (The excerpts above are from the backup_log. file.) > > Our NSD servers are running GPFS 4.1.1-11, TSM is at 7.1.1.100 and the > File system version is 12.06 (3.4.0.3). Has anyone else seen this > behaviour with mmbackup and if so, found a fix? > > Thanks, > > Regards, > Ash > -- *Jez Tucker* Head of Research and Development, Pixit Media 07764193820 | jtucker at pixitmedia.com www.pixitmedia.com | Tw:@pixitmedia.com -- This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ashish.thandavan at cs.ox.ac.uk Fri Mar 3 13:05:56 2017 From: ashish.thandavan at cs.ox.ac.uk (Ashish Thandavan) Date: Fri, 3 Mar 2017 13:05:56 +0000 Subject: [gpfsug-discuss] mmbackup logging issue In-Reply-To: <7ca99092-63e6-c1a1-d9a7-e1907bcef7c8@pixitmedia.com> References: <7ca99092-63e6-c1a1-d9a7-e1907bcef7c8@pixitmedia.com> Message-ID: <5882c991-4cb0-4658-e25b-b2a1bbc26a48@cs.ox.ac.uk> Hi Jez, > > This is the primary reason for using snapshots for mmbackup ( -S > snapname ) and making sure that only mmbackup sends data to backup > rather than an oob dsmc: > > Tue Jan 17 18:07:31 2017 mmbackup:Audit files /cs/mmbackup.audit.gpfs* > contain 0 failed paths but there were 10012 failures. > Cannot reconcile shadow database. > Unable to compensate for all TSM errors in new shadow database. > Preserving previous shadow database. > Run next mmbackup with -q to synchronize shadow database. exit 12 > <------------ ick! > I was going to send a separate email about this, but as you mention it -- I was wondering if the TSM server requires the shadow files to also be backed up? And that reconciliation of the shadow database will fail if they are not? (Unless of course, you mean above that the shadow database failure is purely on account of the 10012 failures...) Re the use of snapshots for mmbackup, are you saying dsmc does not get involved in sending the files to the TSM server? I haven't used snapshots for mmbackup, but will certainly try! Regards, Ash -- ------------------------- Ashish Thandavan UNIX Support Computing Officer Department of Computer Science University of Oxford Wolfson Building Parks Road Oxford OX1 3QD Phone: 01865 610733 Email: ashish.thandavan at cs.ox.ac.uk From jez at rib-it.org Fri Mar 3 13:19:24 2017 From: jez at rib-it.org (Jez Tucker) Date: Fri, 3 Mar 2017 13:19:24 +0000 Subject: [gpfsug-discuss] mmbackup logging issue In-Reply-To: <5882c991-4cb0-4658-e25b-b2a1bbc26a48@cs.ox.ac.uk> References: <7ca99092-63e6-c1a1-d9a7-e1907bcef7c8@pixitmedia.com> <5882c991-4cb0-4658-e25b-b2a1bbc26a48@cs.ox.ac.uk> Message-ID: <754ae5a7-82e0-17a1-2ab6-b1538bcf4e73@rib-it.org> Comments inline... On 03/03/17 13:05, Ashish Thandavan wrote: > Hi Jez, > >> >> This is the primary reason for using snapshots for mmbackup ( -S >> snapname ) and making sure that only mmbackup sends data to backup >> rather than an oob dsmc: >> >> Tue Jan 17 18:07:31 2017 mmbackup:Audit files >> /cs/mmbackup.audit.gpfs* contain 0 failed paths but there were 10012 >> failures. >> Cannot reconcile shadow database. >> Unable to compensate for all TSM errors in new shadow database. >> Preserving previous shadow database. >> Run next mmbackup with -q to synchronize shadow database. exit 12 >> <------------ ick! >> > > I was going to send a separate email about this, but as you mention it > -- I was wondering if the TSM server requires the shadow files to also > be backed up? And that reconciliation of the shadow database will fail > if they are not? No, that's not a requirement. > (Unless of course, you mean above that the shadow database failure is > purely on account of the 10012 failures...) More than likely. Check the dsmerror.log and the mmbackup logs. There are also other possibilities which could exhibit the similar cases. > > Re the use of snapshots for mmbackup, are you saying dsmc does not get > involved in sending the files to the TSM server? I haven't used > snapshots for mmbackup, but will certainly try! Snapshots provide a consistent dataset to backup. I.E. no errors from 'trying to backup a file which has been deleted since I first knew about it in the scan phase'. But as per previous related thread 'Tracking deleted files', YMMV depending on your workload and environment. > > Regards, > Ash > From chair at spectrumscale.org Fri Mar 3 15:23:00 2017 From: chair at spectrumscale.org (GPFS UG Chair (Simon Thompson)) Date: Fri, 03 Mar 2017 15:23:00 +0000 Subject: [gpfsug-discuss] Save the Date 9th/10th May - UK/European/Global (?) Spectrum Scale user group meeting Message-ID: Hi All, Save the date for the next UK Spectrum Scale user group meeting on 9th/10th May 2017. As last year, this will be a 2 day meeting and having out-grown IBM South Bank client centre who have hosted us over the past few years, we're heading off to Manchester to a bigger venue. Further details on registration and agenda will be available in the next few weeks. Thanks to IBM for supporting the event funding the venue, we are working up our sponsorship package for other companies for an evening function. This year we are able to take up to 200 delegates and we're expecting that it will be expanded out to other European customers looking for an English language event, and we're flattered that Doug listed us a the Global event :-D Thanks Simon (Group Chair) From ewahl at osc.edu Fri Mar 3 18:50:55 2017 From: ewahl at osc.edu (Edward Wahl) Date: Fri, 3 Mar 2017 13:50:55 -0500 Subject: [gpfsug-discuss] mmbackup logging issue In-Reply-To: <7ca99092-63e6-c1a1-d9a7-e1907bcef7c8@pixitmedia.com> References: <7ca99092-63e6-c1a1-d9a7-e1907bcef7c8@pixitmedia.com> Message-ID: <20170303135055.7f046983@osc.edu> Comments inline On Fri, 3 Mar 2017 12:59:48 +0000 Jez Tucker wrote: > Hi Ash, > > This is the primary reason for using snapshots for mmbackup ( -S > snapname ) and making sure that only mmbackup sends data to backup > rather than an oob dsmc: > > Tue Jan 17 18:07:31 2017 mmbackup:Audit files /cs/mmbackup.audit.gpfs* > contain 0 failed paths but there were 10012 failures. > Cannot reconcile shadow database. > Unable to compensate for all TSM errors in new shadow database. > Preserving previous shadow database. > Run next mmbackup with -q to synchronize shadow database. exit 12 > <------------ ick! > > Other obvious causes are very, very, odd filenames / paths. Yes, GPFS allows much more lenient filenames than TSM does. You can attempt to cut this back a bit with 'WILDCARDSARELITERAL yes' and 'QUOTESARELITERAL yes' in your dsm.opt (and I recommend SKIPACLUPDATECHECK yes and UPDATECTIME yes ) But this still won't catch everything. We still tend to find foreign character sets, control sequences that look like people trying to exit emacs, etc. in file names. Valid for the Filesystem, not so much for TSM. Ed > > Jez > > > On 28/02/17 16:10, Ashish Thandavan wrote: > > Dear all, > > > > We have a small GPFS cluster and a separate server running TSM and one > > of the three NSD servers backs up our GPFS filesystem to the TSM > > server using mmbackup. After a recent upgrade from v3.5 to 4.1.1, > > we've noticed that mmbackup no longer logs stuff like it used to : > > > > ... > > Thu Jan 19 05:45:41 2017 mmbackup:Backing up files: 0 backed up, > > 870532 expired, 2 failed. > > Thu Jan 19 06:15:41 2017 mmbackup:Backing up files: 0 backed up, > > 870532 expired, 3 failed. > > Thu Jan 19 06:45:41 2017 mmbackup:Backing up files: 0 backed up, > > 870532 expired, 3 failed. > > ... > > > > > > instead of > > > > ... > > Sat Dec 3 12:01:00 2016 mmbackup:Backing up files: 105030 backed up, > > 635456 expired, 30 failed. > > Sat Dec 3 12:31:00 2016 mmbackup:Backing up files: 205934 backed up, > > 635456 expired, 57 failed. > > Sat Dec 3 13:01:00 2016 mmbackup:Backing up files: 321702 backed up, > > 635456 expired, 169 failed. > > ... > > > > like it used to pre-upgrade. > > > > I am therefore unable to see how far long it has got, and indeed if it > > completed successfully, as this is what it logs at the end of a job : > > > > ... > > Tue Jan 17 18:07:31 2017 mmbackup:Completed policy backup run with 0 > > policy errors, 10012 files failed, 0 severe errors, returning rc=9. > > Tue Jan 17 18:07:31 2017 mmbackup:Policy for backup returned 9 Highest > > TSM error 12 > > mmbackup: TSM Summary Information: > > Total number of objects inspected: 20617273 > > Total number of objects backed up: 0 > > Total number of objects updated: 0 > > Total number of objects rebound: 0 > > Total number of objects deleted: 0 > > Total number of objects expired: 1 > > Total number of objects failed: 10012 > > Total number of objects encrypted: 0 > > Total number of bytes inspected: 3821624716861 > > Total number of bytes transferred: 3712040943672 > > Tue Jan 17 18:07:31 2017 mmbackup:Audit files /cs/mmbackup.audit.gpfs* > > contain 0 failed paths but there were 10012 failures. > > Cannot reconcile shadow database. > > Unable to compensate for all TSM errors in new shadow database. > > Preserving previous shadow database. > > Run next mmbackup with -q to synchronize shadow database. exit 12 > > > > If it helps, the mmbackup job is kicked off with the following options : > > /usr/lpp/mmfs/bin/mmbackup gpfs -n 8 -t full -B 20000 -L 1 > > --tsm-servers gpfs_weekly_stanza -N glossop1a | /usr/bin/tee > > /var/log/mmbackup/gpfs_weekly/backup_log.`date +%Y%m%d_%H_%M` > > > > (The excerpts above are from the backup_log. file.) > > > > Our NSD servers are running GPFS 4.1.1-11, TSM is at 7.1.1.100 and the > > File system version is 12.06 (3.4.0.3). Has anyone else seen this > > behaviour with mmbackup and if so, found a fix? > > > > Thanks, > > > > Regards, > > Ash > > > > -- Ed Wahl Ohio Supercomputer Center 614-292-9302 From joseph_grace at us.ibm.com Fri Mar 3 19:35:48 2017 From: joseph_grace at us.ibm.com (Joseph Grace) Date: Fri, 3 Mar 2017 14:35:48 -0500 Subject: [gpfsug-discuss] shutdown Message-ID: An HTML attachment was scrubbed... URL: From janfrode at tanso.net Fri Mar 3 20:54:23 2017 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Fri, 03 Mar 2017 20:54:23 +0000 Subject: [gpfsug-discuss] shutdown In-Reply-To: References: Message-ID: Unmount filesystems cleanly "mmumount all -a", stop gpfs "mmshutdown -N gss_ppc64" and poweroff "xdsh gss_ppc64 poweroff". -jf fre. 3. mar. 2017 kl. 20.55 skrev Joseph Grace : > Please excuse the newb question but we have a planned power outage coming > and I can't locate a procedure to shutdown an ESSp GL2 system? > Thanks, > > Joe Grace > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jake.carroll at uq.edu.au Sat Mar 4 20:34:51 2017 From: jake.carroll at uq.edu.au (Jake Carroll) Date: Sat, 4 Mar 2017 20:34:51 +0000 Subject: [gpfsug-discuss] Quota issues, eviction, AFM won't stop throwing data to a full location - probably a rookie AFM mistake? Message-ID: <168ADEDD-5712-42E8-9003-CAE63B2F1192@uq.edu.au> Hi all, I think I need some help with GPFS quotas and hard limits vs soft limits + eviction in AFM scenarios. We?ve got a couple of issues: One: ------- We?ve come across a scenario where if a user hits the hard quota while ingesting into cache in an AFM ?home to cache? relationship whilst an eviction loop is being triggered, things seem to go wrong ? and the filesystem runs off into locking up territory. The report I have on the last incident is that a file-set got stuck at 100% (capacity utilisation), the eviction loop either failed or blocked and the IO requests blocked and/or failed (this one I'm a little fuzzy on). Maybe it isn?t a bug and our guess is that someone on here will probably tell us that the likely ?fix? is to right-size our high and low water marks appropriately. We considered a potential bug mechanism or race condition if the eviction loop uses space in the file-set in the .afm directory ? but I then thought better of it and though ?Nah, surely IBM would have thought of that!?. Two: ------- We witness a scenario where AFM doesn't back off if it gets a filesystem full error code when trying to make the cache clean in migrating data to ?home?. If this takes a couple of seconds to raise the error each attempt, gpfs/mmfsd will deplete NFS daemons causing a DoS against the NFS server that is powering the cache/home relationship for the AFM transport. We had a mental model that AFM cache wouldn?t or shouldn?t overload hard and soft quota as the high and low watermarks for cache eviction policies. I guess in our heads, we?d like caches to also enforce and respect quotas based on requests received from home. There are probably lots of reasons this doesn?t make sense programmatically, or to the rest of scale ? but it would (seem to us) that it would clean up this problem or at least some of it. Happy to chat through his further and explain it more if anyone is interested. If there are any AFM users out there, we?d love to hear about how you deal with quotas, hwm/lwm and eviction over-flow scenarios, if they exist in your environment. Thank you as always, list. -jc -------------- next part -------------- An HTML attachment was scrubbed... URL: From mweil at wustl.edu Tue Mar 7 20:21:42 2017 From: mweil at wustl.edu (Matt Weil) Date: Tue, 7 Mar 2017 14:21:42 -0600 Subject: [gpfsug-discuss] numaMemoryInterleave=yes Message-ID: <85f1c29c-c71d-2d43-bda7-65a7278c1c40@wustl.edu> Hello all Is this necessary any more? numastat -p mmfsd seems to spread it out without it. Thanks Matt ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. From Robert.Oesterlin at nuance.com Tue Mar 7 20:32:32 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Tue, 7 Mar 2017 20:32:32 +0000 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? Message-ID: I?m considering enabling trace on all nodes all the time, doing something like this: mmtracectl --set --trace=def --trace-recycle=global --tracedev-write-mode=overwrite --tracedev-overwrite-buffer-size=256M mmtracectl --start My questions are: - What is the performance penalty of leaving this on 100% of the time on a node? - Does anyone have any suggestions on automation on stopping trace when a particular event occurs? - What other issues, if any? Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Tue Mar 7 20:53:44 2017 From: bbanister at jumptrading.com (Bryan Banister) Date: Tue, 7 Mar 2017 20:53:44 +0000 Subject: [gpfsug-discuss] numaMemoryInterleave=yes In-Reply-To: <85f1c29c-c71d-2d43-bda7-65a7278c1c40@wustl.edu> References: <85f1c29c-c71d-2d43-bda7-65a7278c1c40@wustl.edu> Message-ID: I believe the last I saw from Yuri was that this should always be enabled... but don't recall if there are other things this tunes outside of the numactl stuff, -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Tuesday, March 07, 2017 2:22 PM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] numaMemoryInterleave=yes Hello all Is this necessary any more? numastat -p mmfsd seems to spread it out without it. Thanks Matt ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. From bbanister at jumptrading.com Tue Mar 7 20:58:15 2017 From: bbanister at jumptrading.com (Bryan Banister) Date: Tue, 7 Mar 2017 20:58:15 +0000 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? In-Reply-To: References: Message-ID: The performance impact can be quite significant depending on what you are tracing. We even having monitoring that looks for long running traces and the recommended action is to ?kill with impunity!!? I believe IBM recommends never running clusters with continuous tracing. -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Oesterlin, Robert Sent: Tuesday, March 07, 2017 2:33 PM To: gpfsug main discussion list Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? I?m considering enabling trace on all nodes all the time, doing something like this: mmtracectl --set --trace=def --trace-recycle=global --tracedev-write-mode=overwrite --tracedev-overwrite-buffer-size=256M mmtracectl --start My questions are: - What is the performance penalty of leaving this on 100% of the time on a node? - Does anyone have any suggestions on automation on stopping trace when a particular event occurs? - What other issues, if any? Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Tue Mar 7 21:11:33 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Tue, 7 Mar 2017 21:11:33 +0000 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? Message-ID: I?ve been told that V70o0 unified nodes (GPFS under the covers) run with tracing enabled all the time.. but I agree with you Brian on the potential impacts. But when you must catch a trace for a problem that occurs once every few weeks ? how else would I do it? Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 From: on behalf of Bryan Banister Reply-To: gpfsug main discussion list Date: Tuesday, March 7, 2017 at 2:58 PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? I believe IBM recommends never running clusters with continuous tracing. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Tue Mar 7 21:17:35 2017 From: bbanister at jumptrading.com (Bryan Banister) Date: Tue, 7 Mar 2017 21:17:35 +0000 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? In-Reply-To: References: Message-ID: <098c950ba76e454d861d784f7f66cba1@jumptrading.com> Just depends on how your problem is detected? is it in a log? Is it found by running a command (.e.g mm*)? Is it discovered in `ps` output? Is your scheduler failing jobs? We have ongoing monitoring of most of these types of problem detection points and an automated process to capture a trace could be added to trigger upon detection depending on how the problem is detected? -B From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Oesterlin, Robert Sent: Tuesday, March 07, 2017 3:12 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? I?ve been told that V70o0 unified nodes (GPFS under the covers) run with tracing enabled all the time.. but I agree with you Brian on the potential impacts. But when you must catch a trace for a problem that occurs once every few weeks ? how else would I do it? Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 From: > on behalf of Bryan Banister > Reply-To: gpfsug main discussion list > Date: Tuesday, March 7, 2017 at 2:58 PM To: gpfsug main discussion list > Subject: [EXTERNAL] Re: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? I believe IBM recommends never running clusters with continuous tracing. ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Tue Mar 7 21:25:05 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Tue, 7 Mar 2017 21:25:05 +0000 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? Message-ID: mmfsd crash - IBM says, ?we need a trace to debug the issue?. Sigh Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Bryan Banister Reply-To: gpfsug main discussion list Date: Tuesday, March 7, 2017 at 3:17 PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? Just depends on how your problem is detected? is it in a log? Is it found by running a command (.e.g mm*)? Is it discovered in `ps` output? Is your scheduler failing jobs? -------------- next part -------------- An HTML attachment was scrubbed... URL: From valdis.kletnieks at vt.edu Tue Mar 7 21:36:44 2017 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Tue, 07 Mar 2017 16:36:44 -0500 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? In-Reply-To: <098c950ba76e454d861d784f7f66cba1@jumptrading.com> References: <098c950ba76e454d861d784f7f66cba1@jumptrading.com> Message-ID: <19135.1488922604@turing-police.cc.vt.edu> On Tue, 07 Mar 2017 21:17:35 +0000, Bryan Banister said: > Just depends on how your problem is detected??? is it in a log? Is it found by > running a command (.e.g mm*)? Is it discovered in `ps` output? Is your > scheduler failing jobs? I think the problem here is that if you have a sudden cataclysmic event, you want to have been in flight-recorder mode and be able to look at the last 5 or 10 seconds of trace *before* you became aware that your filesystem just went walkies. Sure, you can start tracing when the filesystem dies - but at that point you just get a whole mess of failed I/O requests in the trace, and no hint of where things went south... -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 484 bytes Desc: not available URL: From aaron.s.knister at nasa.gov Tue Mar 7 21:51:23 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 7 Mar 2017 16:51:23 -0500 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? In-Reply-To: References: Message-ID: <6bc4cf00-7904-0ab3-5f93-963beaca1dbc@nasa.gov> Hi Bob, I have the impression the biggest impact is to metadata-type operations rather than throughput but don't quote me on that because I have very little data to back it up. In the process of testing upgrading from GPFS 3.5 to 4.1 we ran fio on 1000 some nodes against an FS in our test environment which sustained about 60-80k iops on the filesystem's metadata LUNs. At one point I couldn't understand why I was struggling to get about 13k iops and realized tracing was turned on on some subset of nsd servers (which are also manager nodes). After turning it off the throughput immediately shot back up to where I was expecting it to be. Also during testing we were tracking down a bug for which I needed to run tracing *everywhere* and then turn it off when one of the manager nodes saw a particular error. I used a script IBM had sent me a while back to help with this that I made some tweaks to. I've attached it in case its helpful. In a nutshell the process looks like: - start tracing everywhere (/usr/lpp/mmfs/bin/mmdsh -Nall /usr/lpp/mmfs/bin/mmtrace start). Doing it this way avoids the need to change the sdrfs file which depending on your cluster size may or may not have some benefits. - run a command to watch for the event in question that when triggered runs /usr/lpp/mmfs/bin/mmdsh -Nall /usr/lpp/mmfs/bin/mmtrace stop If the condition could present itself on multiple nodes within quick succession (as was the case for me) you could wrap the mmdsh for stopping tracing in an flock, using an arbitrary node that stores the lock locally: ssh $stopHost flock -xn /tmp/mmfsTraceStopLock -c "'/usr/lpp/mmfs/bin/mmdsh -N all /usr/lpp/mmfs/bin/mmtrace stop'" Wrapping it in an flock avoids multiple trace format format attempts. -Aaron On 3/7/17 3:32 PM, Oesterlin, Robert wrote: > I?m considering enabling trace on all nodes all the time, doing > something like this: > > > > mmtracectl --set --trace=def --trace-recycle=global > --tracedev-write-mode=overwrite --tracedev-overwrite-buffer-size=256M > mmtracectl --start > > > > My questions are: > > > > - What is the performance penalty of leaving this on 100% of the time on > a node? > > - Does anyone have any suggestions on automation on stopping trace when > a particular event occurs? > > - What other issues, if any? > > > > > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > 507-269-0413 > > > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 -------------- next part -------------- #!/usr/bin/ksh stopHost=loremds20 mmtrace=/usr/lpp/mmfs/bin/mmtrace mmtracectl=/usr/lpp/mmfs/bin/mmtracectl # No automatic start of mmtrace. # Second to sleep between checking. secondsToSleep=2 # Flag to know when tripped or stopped tripped=0 # mmfs log file to monitor logToGrep=/var/log/messages # Path to mmfs bin directory MMFSbin=/usr/lpp/mmfs/bin # Trip file. Will exist if trap is sprung trapHasSprung=/tmp/mmfsTrapHasSprung rm $trapHasSprung 2>/dev/null # Start tracing on this node #${mmtrace} start # Initial count of expelled message in mmfs log baseCount=$(grep "unmounted by the system with return code 301 reason code" $logToGrep | wc -l) # do this loop while the trip file does not exist while [[ ! -f $trapHasSprung ]] do sleep $secondsToSleep # Get current count of expelled to check against the initial. currentCount=$(grep "unmounted by the system with return code 301 reason code" $logToGrep | wc -l) if [[ $currentCount > $baseCount ]] then tripped=1 /usr/lpp/mmfs/bin/mmdsh -N managernodes,quorumnodes touch $trapHasSprung # cluster manager? #stopHost=$(/usr/lpp/mmfs/bin/tslsmgr | grep '^Cluster manager' | awk '{ print $NF }' | sed -e 's/[()]//g') ssh $stopHost flock -xn /tmp/mmfsTraceStopLock -c "'/usr/lpp/mmfs/bin/mmdsh -N all -f128 /usr/lpp/mmfs/bin/mmtrace stop noformat'" fi done From r.sobey at imperial.ac.uk Wed Mar 8 06:53:17 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Wed, 8 Mar 2017 06:53:17 +0000 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? In-Reply-To: <19135.1488922604@turing-police.cc.vt.edu> References: <098c950ba76e454d861d784f7f66cba1@jumptrading.com> <19135.1488922604@turing-police.cc.vt.edu> Message-ID: We're in the same boat...gpfs snap hangs when the cluster / node is unresponsive but they don't know how to give us a root cause without one. Very frustrating. -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of valdis.kletnieks at vt.edu Sent: 07 March 2017 21:37 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? On Tue, 07 Mar 2017 21:17:35 +0000, Bryan Banister said: > Just depends on how your problem is detected??? is it in a log? Is it > found by running a command (.e.g mm*)? Is it discovered in `ps` > output? Is your scheduler failing jobs? I think the problem here is that if you have a sudden cataclysmic event, you want to have been in flight-recorder mode and be able to look at the last 5 or 10 seconds of trace *before* you became aware that your filesystem just went walkies. Sure, you can start tracing when the filesystem dies - but at that point you just get a whole mess of failed I/O requests in the trace, and no hint of where things went south... From vpuvvada at in.ibm.com Wed Mar 8 10:28:35 2017 From: vpuvvada at in.ibm.com (Venkateswara R Puvvada) Date: Wed, 8 Mar 2017 15:58:35 +0530 Subject: [gpfsug-discuss] Quota issues, eviction, AFM won't stop throwing data to a full location - probably a rookie AFM mistake? In-Reply-To: <168ADEDD-5712-42E8-9003-CAE63B2F1192@uq.edu.au> References: <168ADEDD-5712-42E8-9003-CAE63B2F1192@uq.edu.au> Message-ID: 1. What is the version of GPFS ? Eviction should not be blocking the applications. Was partial file caching enabled ? Eviction cannot evict partially cached files in recent releases. Eviction does not use space inside .afm directory, and its logs are stored under /var/mmfs/tmp by default. 2. I did not understand this requirement. a. When IO to home fails with quota exceeded or no space error , the messages are requeued at gateway node and will be retried later (usually 15 minutes). Cache cannot read home quotas today, and in most of the cases this is not valid. b. When soft quota is exceeded on AFM fileset, auto eviction clears data blocks on files based on LRU policy to bring quota below soft limit. These evicted files are uncached and there is no real migration of data to home during eviction. Eviction should get triggered before fileset usage nearing hard quota and applications getting errors. ~Venkat (vpuvvada at in.ibm.com) From: Jake Carroll To: "gpfsug-discuss at spectrumscale.org" Date: 03/05/2017 02:05 AM Subject: [gpfsug-discuss] Quota issues, eviction, AFM won't stop throwing data to a full location - probably a rookie AFM mistake? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi all, I think I need some help with GPFS quotas and hard limits vs soft limits + eviction in AFM scenarios. We?ve got a couple of issues: One: ------- We?ve come across a scenario where if a user hits the hard quota while ingesting into cache in an AFM ?home to cache? relationship whilst an eviction loop is being triggered, things seem to go wrong ? and the filesystem runs off into locking up territory. The report I have on the last incident is that a file-set got stuck at 100% (capacity utilisation), the eviction loop either failed or blocked and the IO requests blocked and/or failed (this one I'm a little fuzzy on). Maybe it isn?t a bug and our guess is that someone on here will probably tell us that the likely ?fix? is to right-size our high and low water marks appropriately. We considered a potential bug mechanism or race condition if the eviction loop uses space in the file-set in the .afm directory ? but I then thought better of it and though ?Nah, surely IBM would have thought of that!?. Two: ------- We witness a scenario where AFM doesn't back off if it gets a filesystem full error code when trying to make the cache clean in migrating data to ?home?. If this takes a couple of seconds to raise the error each attempt, gpfs/mmfsd will deplete NFS daemons causing a DoS against the NFS server that is powering the cache/home relationship for the AFM transport. We had a mental model that AFM cache wouldn?t or shouldn?t overload hard and soft quota as the high and low watermarks for cache eviction policies. I guess in our heads, we?d like caches to also enforce and respect quotas based on requests received from home. There are probably lots of reasons this doesn?t make sense programmatically, or to the rest of scale ? but it would (seem to us) that it would clean up this problem or at least some of it. Happy to chat through his further and explain it more if anyone is interested. If there are any AFM users out there, we?d love to hear about how you deal with quotas, hwm/lwm and eviction over-flow scenarios, if they exist in your environment. Thank you as always, list. -jc _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From duersch at us.ibm.com Wed Mar 8 13:32:37 2017 From: duersch at us.ibm.com (Steve Duersch) Date: Wed, 8 Mar 2017 08:32:37 -0500 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? In-Reply-To: References: Message-ID: You are correct. There is some level of tracing on the v7000U. Also, if the gpfs daemon hits an assert or signal and tracing is running then traces are recycled automatically. There is no need to have another script monitoring for such events. You probably want to have mmtracectl --trace-recycle=global so that an assert on one node grabs traces on all nodes (ie fs manager). As for performance, this one is more difficult and I don't have details. I can ping a colleague to see if he has numbers. Obviously there will be a performance penalty for running tracing. Steve Duersch Spectrum Scale IBM Poughkeepsie, New York > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 7 Mar 2017 21:11:33 +0000 > From: "Oesterlin, Robert" > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Potential problems - leaving trace > enabled in over-write mode? > Message-ID: > Content-Type: text/plain; charset="utf-8" > > I?ve been told that V70o0 unified nodes (GPFS under the covers) run > with tracing enabled all the time.. but I agree with you Brian on > the potential impacts. But when you must catch a trace for a problem > that occurs once every few weeks ? how else would I do it? > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > 507-269-0413 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oehmes at gmail.com Wed Mar 8 13:37:15 2017 From: oehmes at gmail.com (Sven Oehme) Date: Wed, 08 Mar 2017 13:37:15 +0000 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? In-Reply-To: References: Message-ID: given that i love data points, let me put some behind this thread :-) starting in version 3.4 we enhanced the trace code of scale significant. this went on release to release all the way up to 4.2.1. since 4.2.1 we made further improvements, but much smaller changes, more optimization , e.g. reducing of trace levels verbosity, etc . with 4.2.2 we switched from blocking traces to in memory traces as the default trace infrastructure, this infrastructure was designed to be turned on all the time with minimal impact on performance. i just took a 4.2.3 build and did load it on one of my faster NVMe nodes and did a 100% random read test with 64 threads against very fast storage. while this was running i started trace with mmtracectl --start ; sleep 10 ; mmtracectl --off , attached is a screenshot of this run : [image: pasted1] the trace started at 14:19:15 and stopped at 14:20:00 (that includes 10 command processing, format of traces, etc) as one can see on the chart the dip in performance is very small, measured looking at raw data its 8%. the system runs at ~920k 4k random read iops /sec and the workload running on the node pushes the system almost to 100% cpu time even without trace running, reason i tested under this condition is because also on a HPC compute oriented workload you will run without spare cpu cycles, so the 8% is under extreme conditions. said all this, the workload i am running is not causing a lot of metadata i/o as i am running within a small number of files, its all read, so no write, therefore this isn't representative to a lot of real world workloads, but covers one very extreme case , high iops requirements under heavy cpu contention. therefore i repeated the test under a heavy metadata workload , SpecSFS SWBUILD. when i run the test with and without traces turned on i see a ~15% loss on max operations/sec/node. again this is another extreme case as there are only few workloads which have a metadata workload component as heavy as SWBUILD. but given that between the 2 extreme cases we are talking about 8 - 15% overhead its is very likely that all real world workloads suffer less than 15% when traces are turned on. only a test in your environment will really show . i would not recommend to try this with any version prior 4.2.1 as the impact will be significant higher than what i measured. hope this helps make a informed decision. Sven On Tue, Mar 7, 2017 at 9:32 PM Oesterlin, Robert < Robert.Oesterlin at nuance.com> wrote: > I?m considering enabling trace on all nodes all the time, doing something > like this: > > > > mmtracectl --set --trace=def --trace-recycle=global > --tracedev-write-mode=overwrite --tracedev-overwrite-buffer-size=256M > mmtracectl --start > > > > My questions are: > > > > - What is the performance penalty of leaving this on 100% of the time on a > node? > > - Does anyone have any suggestions on automation on stopping trace when a > particular event occurs? > > - What other issues, if any? > > > > > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > 507-269-0413 <(507)%20269-0413> > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pasted1 Type: image/png Size: 216901 bytes Desc: not available URL: From Robert.Oesterlin at nuance.com Wed Mar 8 13:56:32 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Wed, 8 Mar 2017 13:56:32 +0000 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? Message-ID: <90CE7D81-D1F0-4E29-B9F2-02D467911E6C@nuance.com> As always, Sven comes in to back this up with real data :) To net this out, Sven ? I should be able enable trace on my NSD servers running 4.2.2 without much impact, correct? Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Sven Oehme Reply-To: gpfsug main discussion list Date: Wednesday, March 8, 2017 at 7:37 AM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? starting in version 3.4 we enhanced the trace code of scale significant. this went on release to release all the way up to 4.2.1. since 4.2.1 we made further improvements, but much smaller changes, more optimization , e.g. reducing of trace levels verbosity, etc . with 4.2.2 we switched from blocking traces to in memory traces as the default trace infrastructure, this infrastructure was designed to be turned on all the time with minimal impact on performance. -------------- next part -------------- An HTML attachment was scrubbed... URL: From oehmes at gmail.com Wed Mar 8 14:15:55 2017 From: oehmes at gmail.com (Sven Oehme) Date: Wed, 08 Mar 2017 14:15:55 +0000 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? In-Reply-To: <90CE7D81-D1F0-4E29-B9F2-02D467911E6C@nuance.com> References: <90CE7D81-D1F0-4E29-B9F2-02D467911E6C@nuance.com> Message-ID: yes , but i would do this in stages given how large your system is. pick one set of nodes (lets say) 100 out of 200 that do similar things and turn tracing on there. this will give you data you can compare between the 2 set of nodes. let it run for a week and if the data with your real workload doesn't show any significant degradation (which is what i expect) turn it on everywhere. the one thing i am not 100% sure about is size of trace buffer as well as the global cut config. what this means is if you apply the settings as mentioned in this first post, if one node asserts in your cluster you will cut a trace on all nodes that will write a 256M buffer into your dump file location. if you have a node thats in an assert loop (asserts, restarts , asserts) this can cause significant load on all nodes. therefore i would probably start without cutting a global trace and reduce the trace size to 64M. i (and i am sure other dev folks) would be very interested in the outcome as we have this debate on a yearly basis if we shouldn't just turn tracing on by default, in the past performance was the biggest hurdle, this is solved now (my claim) . next big questions is how well does that work on larger scale systems with production workloads. as more feedback we will get in this area as better we can make informed decision how and if it could be turned on all the time and work harder on handling cases like i mentioned above to mitigate the risks . sven On Wed, Mar 8, 2017 at 2:56 PM Oesterlin, Robert < Robert.Oesterlin at nuance.com> wrote: > As always, Sven comes in to back this up with real data :) > > > > To net this out, Sven ? I should be able enable trace on my NSD servers > running 4.2.2 without much impact, correct? > > > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > > > > > > *From: * on behalf of Sven > Oehme > *Reply-To: *gpfsug main discussion list > *Date: *Wednesday, March 8, 2017 at 7:37 AM > > > *To: *gpfsug main discussion list > > *Subject: *[EXTERNAL] Re: [gpfsug-discuss] Potential problems - leaving > trace enabled in over-write mode? > > > > starting in version 3.4 we enhanced the trace code of scale significant. > this went on release to release all the way up to 4.2.1. since 4.2.1 we > made further improvements, but much smaller changes, more optimization , > e.g. reducing of trace levels verbosity, etc . > > with 4.2.2 we switched from blocking traces to in memory traces as the > default trace infrastructure, this infrastructure was designed to be turned > on all the time with minimal impact on performance. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Wed Mar 8 15:25:16 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Wed, 8 Mar 2017 15:25:16 +0000 Subject: [gpfsug-discuss] Registration Reminder: GPFSUG-USA Meeting at NERSC April 4-5th Message-ID: Here?s a reminder to register for the upcoming user group meeting! Registration deadline is March 24th! We Still have slots (20 mins each) for customer and partner talks ? please contact me if you want to do a short presentation. Register here: https://www.nersc.gov/research-and-development/data-analytics/spectrum-user-group-meeting/ Bob Oesterlin Sr Principal Storage Engineer, Nuance -------------- next part -------------- An HTML attachment was scrubbed... URL: From jenocram at gmail.com Wed Mar 8 20:15:44 2017 From: jenocram at gmail.com (Jeno Cram) Date: Wed, 8 Mar 2017 15:15:44 -0500 Subject: [gpfsug-discuss] CES, NFSv4 and Kerberos with AD authentication Message-ID: I'm running into an issue and I can't find the answer anywhere in the Spectrum Scale documentation. I'm trying to keberize nfsv4 with AD authentication for file access. I have the instructions for the server (--enable-nfs-kerberos) and also ran "mmnfs configuration change From jenocram at gmail.com Wed Mar 8 20:18:07 2017 From: jenocram at gmail.com (Jeno Cram) Date: Wed, 8 Mar 2017 15:18:07 -0500 Subject: [gpfsug-discuss] CES, NFSv4 and Kerberos with AD authentication In-Reply-To: References: Message-ID: My apologies.. I hit send too soon. I'm running into an issue and I can't find the answer anywhere in the Spectrum Scale documentation. I'm trying to keberize nfsv4 with AD authentication for file access. I have the instructions for the server (--enable-nfs-kerberos) and also ran "mmnfs configuration change idmapd_domain=example.com". Is there anything else that needs to be done on the server side and what needs to be configured on the client to allow this? On Wed, Mar 8, 2017 at 3:15 PM, Jeno Cram wrote: > I'm running into an issue and I can't find the answer anywhere in the > Spectrum Scale documentation. > > I'm trying to keberize nfsv4 with AD authentication for file access. > I have the instructions for the server (--enable-nfs-kerberos) and > also ran "mmnfs configuration change From Robert.Oesterlin at nuance.com Wed Mar 8 21:54:33 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Wed, 8 Mar 2017 21:54:33 +0000 Subject: [gpfsug-discuss] Problem installin CES - 4.2.2-2 Message-ID: <19C1E64C-76E3-4C59-914E-199EA5D37CCF@nuance.com> OK, I?ll admit I?m not protocols install expert. Using the ?spectrumscale? installer command, it?s failing to install the object packages due to some internal dependencies from the IBM supplied repo. Who can help me? Excerpt from the install log: 2017-03-08 16:47:34,319 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com * log[IBM SPECTRUM SCALE: Installing Object packages (SS50).] action write 2017-03-08 16:47:34,319 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,554 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com * yum_package[spectrum-scale-object] action install 2017-03-08 16:47:59,555 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,555 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com ================================================================================ 2017-03-08 16:47:59,556 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error executing action `install` on resource 'yum_package[spectrum-scale-object]' 2017-03-08 16:47:59,556 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com ================================================================================ 2017-03-08 16:47:59,557 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,557 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Chef::Exceptions::Exec 2017-03-08 16:47:59,558 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com ---------------------- 2017-03-08 16:47:59,558 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com yum -d0 -e0 -y install spectrum-scale-object-4.2.2-2 returned 1: 2017-03-08 16:47:59,558 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com STDOUT: Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,560 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,560 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,560 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,563 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,563 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,563 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com You could try using --skip-broken to work around the problem 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com You could try running: rpm -Va --nofiles --nodigest 2017-03-08 16:47:59,565 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,565 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com STDERR: Error: Package: 1:python-novaclient-2.30.3-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,565 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error: Package: 1:python-glanceclient-1.1.2-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error: Package: python-openstackclient-1.7.3-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,569 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,569 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,569 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error: Package: python-neutronclient-3.1.2-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,571 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,571 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,571 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 Bob Oesterlin Sr Principal Storage Engineer, Nuance -------------- next part -------------- An HTML attachment was scrubbed... URL: From stockf at us.ibm.com Wed Mar 8 22:06:18 2017 From: stockf at us.ibm.com (Frederick Stock) Date: Wed, 8 Mar 2017 17:06:18 -0500 Subject: [gpfsug-discuss] Problem installin CES - 4.2.2-2 In-Reply-To: <19C1E64C-76E3-4C59-914E-199EA5D37CCF@nuance.com> References: <19C1E64C-76E3-4C59-914E-199EA5D37CCF@nuance.com> Message-ID: What version of python do you have installed? I think you need at least version 2.7. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 03/08/2017 04:55 PM Subject: [gpfsug-discuss] Problem installin CES - 4.2.2-2 Sent by: gpfsug-discuss-bounces at spectrumscale.org OK, I?ll admit I?m not protocols install expert. Using the ?spectrumscale? installer command, it?s failing to install the object packages due to some internal dependencies from the IBM supplied repo. Who can help me? Excerpt from the install log: 2017-03-08 16:47:34,319 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com * log[IBM SPECTRUM SCALE: Installing Object packages (SS50).] action write 2017-03-08 16:47:34,319 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,554 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com * yum_package[spectrum-scale-object] action install 2017-03-08 16:47:59,555 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,555 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com ================================================================================ 2017-03-08 16:47:59,556 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error executing action `install` on resource 'yum_package[spectrum-scale-object]' 2017-03-08 16:47:59,556 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com ================================================================================ 2017-03-08 16:47:59,557 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,557 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Chef::Exceptions::Exec 2017-03-08 16:47:59,558 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com ---------------------- 2017-03-08 16:47:59,558 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com yum -d0 -e0 -y install spectrum-scale-object-4.2.2-2 returned 1: 2017-03-08 16:47:59,558 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com STDOUT: Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,560 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,560 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,560 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,563 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,563 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,563 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com You could try using --skip-broken to work around the problem 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com You could try running: rpm -Va --nofiles --nodigest 2017-03-08 16:47:59,565 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,565 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com STDERR: Error: Package: 1:python-novaclient-2.30.3-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,565 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error: Package: 1:python-glanceclient-1.1.2-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error: Package: python-openstackclient-1.7.3-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,569 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,569 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,569 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error: Package: python-neutronclient-3.1.2-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,571 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,571 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,571 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 Bob Oesterlin Sr Principal Storage Engineer, Nuance _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mark.Bush at siriuscom.com Wed Mar 8 23:25:07 2017 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Wed, 8 Mar 2017 23:25:07 +0000 Subject: [gpfsug-discuss] Problem installin CES - 4.2.2-2 Message-ID: <4FA8EF66-4E49-4C65-AA6E-5166AF8F45C7@siriuscom.com> What version of RHEL/CENTOS? From: "Oesterlin, Robert" Reply-To: gpfsug main discussion list Date: Wednesday, March 8, 2017 at 3:54 PM To: gpfsug main discussion list Subject: [gpfsug-discuss] Problem installin CES - 4.2.2-2 OK, I?ll admit I?m not protocols install expert. Using the ?spectrumscale? installer command, it?s failing to install the object packages due to some internal dependencies from the IBM supplied repo. Who can help me? Excerpt from the install log: 2017-03-08 16:47:34,319 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com * log[IBM SPECTRUM SCALE: Installing Object packages (SS50).] action write 2017-03-08 16:47:34,319 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,554 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com * yum_package[spectrum-scale-object] action install 2017-03-08 16:47:59,555 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,555 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com ================================================================================ 2017-03-08 16:47:59,556 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error executing action `install` on resource 'yum_package[spectrum-scale-object]' 2017-03-08 16:47:59,556 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com ================================================================================ 2017-03-08 16:47:59,557 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,557 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Chef::Exceptions::Exec 2017-03-08 16:47:59,558 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com ---------------------- 2017-03-08 16:47:59,558 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com yum -d0 -e0 -y install spectrum-scale-object-4.2.2-2 returned 1: 2017-03-08 16:47:59,558 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com STDOUT: Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,560 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,560 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,560 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,563 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,563 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,563 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com You could try using --skip-broken to work around the problem 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com You could try running: rpm -Va --nofiles --nodigest 2017-03-08 16:47:59,565 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,565 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com STDERR: Error: Package: 1:python-novaclient-2.30.3-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,565 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error: Package: 1:python-glanceclient-1.1.2-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error: Package: python-openstackclient-1.7.3-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,569 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,569 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,569 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error: Package: python-neutronclient-3.1.2-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,571 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,571 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,571 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 Bob Oesterlin Sr Principal Storage Engineer, Nuance This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Thu Mar 9 00:25:41 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Thu, 9 Mar 2017 00:25:41 +0000 Subject: [gpfsug-discuss] Problem installin CES - 4.2.2-2 In-Reply-To: <4FA8EF66-4E49-4C65-AA6E-5166AF8F45C7@siriuscom.com> References: <4FA8EF66-4E49-4C65-AA6E-5166AF8F45C7@siriuscom.com> Message-ID: 7.2 No other conflicting ?openstack|keystone|swift|glance|nova|neutron? package installed that I can see. Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 From: on behalf of "Mark.Bush at siriuscom.com" Reply-To: gpfsug main discussion list Date: Wednesday, March 8, 2017 at 5:25 PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Problem installin CES - 4.2.2-2 What version of RHEL/CENTOS? -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mark.Bush at siriuscom.com Thu Mar 9 14:12:40 2017 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Thu, 9 Mar 2017 14:12:40 +0000 Subject: [gpfsug-discuss] Problem installin CES - 4.2.2-2 In-Reply-To: References: <4FA8EF66-4E49-4C65-AA6E-5166AF8F45C7@siriuscom.com> Message-ID: I don?t normally use the chef based installer but I have had issues getting OBJ installed on 7.x before and it was all based on the fact that the packages were moved to subdirs /usr/local/mmfs/4.2.2.0/object_rpms/rhel7 where before they were just in the root of object_rpms. Not sure that applies here since I would think the installer was updated to look there instead of the root. It sure seems from your log output that it?s not happy trying to install packages. Worse case you can install manually. Here?s how I do that in my automated script cat </etc/yum.repos.d/IBM-SpecScaleOBJ.repo [spectrum_scale] name=spectrum_scale baseurl=file:///vagrant/4.2.2.0/object_rpms/rhel7/ enabled=1 EOL yum update -y 2>&1 yum install -y /vagrant/4.2.2.0/zimon_rpms/rhel7/gpfs.gss.pm{sensors,collector}*.rpm > /dev/null 2>&1 yum install -y /vagrant/4.2.2.0/gpfs_rpms/gpfs.{base,ext,gskit,gpl,msg,docs,adv,crypto,java,gui}*.rpm > /dev/null 2>&1 yum install -y /vagrant/4.2.2.0/smb_rpms/rhel7/gpfs.*.rpm > /dev/null 2>&1 yum install -y /vagrant/4.2.2.0/ganesha_rpms/rhel7/nfs-*.rpm > /dev/null 2>&1 yum install -y /vagrant/4.2.2.0/object_rpms/rhel7/*.rpm >/dev/null 2>&1 yum install -y spectrum-scale-object 2>&1 From: "Oesterlin, Robert" Reply-To: gpfsug main discussion list Date: Wednesday, March 8, 2017 at 6:25 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Problem installin CES - 4.2.2-2 7.2 No other conflicting ?openstack|keystone|swift|glance|nova|neutron? package installed that I can see. Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 From: on behalf of "Mark.Bush at siriuscom.com" Reply-To: gpfsug main discussion list Date: Wednesday, March 8, 2017 at 5:25 PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Problem installin CES - 4.2.2-2 What version of RHEL/CENTOS? This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Thu Mar 9 14:53:33 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Thu, 9 Mar 2017 14:53:33 +0000 Subject: [gpfsug-discuss] Problem installin CES - 4.2.2-2 Message-ID: Bingo! That worked flawlessly. You are a lifesaver, thanks! Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of "Mark.Bush at siriuscom.com" Reply-To: gpfsug main discussion list Date: Thursday, March 9, 2017 at 8:12 AM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Problem installin CES - 4.2.2-2 I don?t normally use the chef based installer but I have had issues getting OBJ installed on 7.x before and it was all based on the fact that the packages were moved to subdirs /usr/local/mmfs/4.2.2.0/object_rpms/rhel7 where before they were just in the root of object_rpms. Not sure that applies here since I would think the installer was updated to look there instead of the root. It sure seems from your log output that it?s not happy trying to install packages. Worse case you can install manually. Here?s how I do that in my automated script cat </etc/yum.repos.d/IBM-SpecScaleOBJ.repo [spectrum_scale] name=spectrum_scale baseurl=file:///vagrant/4.2.2.0/object_rpms/rhel7/ enabled=1 EOL yum update -y 2>&1 yum install -y /vagrant/4.2.2.0/zimon_rpms/rhel7/gpfs.gss.pm{sensors,collector}*.rpm > /dev/null 2>&1 yum install -y /vagrant/4.2.2.0/gpfs_rpms/gpfs.{base,ext,gskit,gpl,msg,docs,adv,crypto,java,gui}*.rpm > /dev/null 2>&1 yum install -y /vagrant/4.2.2.0/smb_rpms/rhel7/gpfs.*.rpm > /dev/null 2>&1 yum install -y /vagrant/4.2.2.0/ganesha_rpms/rhel7/nfs-*.rpm > /dev/null 2>&1 yum install -y /vagrant/4.2.2.0/object_rpms/rhel7/*.rpm >/dev/null 2>&1 yum install -y spectrum-scale-object 2>&1 From: "Oesterlin, Robert" Reply-To: gpfsug main discussion list Date: Wednesday, March 8, 2017 at 6:25 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Problem installin CES - 4.2.2-2 7.2 No other conflicting ?openstack|keystone|swift|glance|nova|neutron? package installed that I can see. Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 From: on behalf of "Mark.Bush at siriuscom.com" Reply-To: gpfsug main discussion list Date: Wednesday, March 8, 2017 at 5:25 PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Problem installin CES - 4.2.2-2 What version of RHEL/CENTOS? This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Thu Mar 9 16:11:56 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Thu, 9 Mar 2017 16:11:56 +0000 Subject: [gpfsug-discuss] Running gpfs.snap outside of problems Message-ID: Hi all, Is there any best practice as to when to run a gpfs.snap, and the impact it will have on a healthy cluster? Some of my colleagues are keen to gather a snap for example every 2 hours to assist in problem determination. I can elaborate on the "why" part of this if needed but I'm not fully convinced it would help. Said I'd look into it though so I'm asking the question. Cheers Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Thu Mar 9 16:29:52 2017 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Thu, 09 Mar 2017 16:29:52 +0000 Subject: [gpfsug-discuss] Running gpfs.snap outside of problems In-Reply-To: References: Message-ID: There's a manual for it now.. and it points out "The tool impacts performance.." Also it has caused mmfsd crashes for me earlier, so I've learned to be weary of running it.. The manual also says it's collecting using mmfsadm, and the mmfsadm manual warns that it might cause GPFS to fail ("in certain rare cases"). I wouldn't dare running it every few hours. -jf tor. 9. mar. 2017 kl. 17.12 skrev Sobey, Richard A : > Hi all, > > > > Is there any best practice as to when to run a gpfs.snap, and the impact > it will have on a healthy cluster? Some of my colleagues are keen to gather > a snap for example every 2 hours to assist in problem determination. > > > > I can elaborate on the ?why? part of this if needed but I?m not fully > convinced it would help. Said I?d look into it though so I?m asking the > question. > > > > Cheers > > Richard > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mxcheng at uchicago.edu Thu Mar 9 19:23:38 2017 From: mxcheng at uchicago.edu (Mengxing Cheng) Date: Thu, 9 Mar 2017 19:23:38 +0000 Subject: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem Message-ID: Dear all, My name is Mengxing Cheng and I am a HPC system administrator at the University of Chicago. We have a GSS26 running gss2.5.10.3-3b and gpfs-4.2.0.3. Recently, we run mmfsck on a relatively small filesystem with 14TB block and 73863102 inodes but it was unusually slow so as to not be able to finish in 48 hours. In contrast, mmfsck run on a filesystem with the same size and inodes but sitting on a traditional IBM DS3512 storage took only 2 hours to complete. In particular, the mmfsck run in parallel using 3 nodes within the GSS storage cluster, we notice that one gss storage server scans inodes much slower than the other gss storage server and the quorum service node. Has anyone experience the same mmfsck performance issue? Could anyone make recommendation to troubleshoot and improve mmfsck performance? Thank you! Mengxing --- Mengxing Cheng, Ph.D. HPC System Administrator Research Computing Center The University of Chicago 5607 S. Drexel Ave. Chicago, IL 60637 email: mxcheng at uchicago.edu phone: (773) 702-4070 -------------- next part -------------- An HTML attachment was scrubbed... URL: From kevindjo at us.ibm.com Thu Mar 9 19:47:56 2017 From: kevindjo at us.ibm.com (Kevin D Johnson) Date: Thu, 9 Mar 2017 19:47:56 +0000 Subject: [gpfsug-discuss] Running gpfs.snap outside of problems In-Reply-To: References: , Message-ID: An HTML attachment was scrubbed... URL: From esperle at us.ibm.com Thu Mar 9 21:13:08 2017 From: esperle at us.ibm.com (Eric Sperley) Date: Thu, 9 Mar 2017 13:13:08 -0800 Subject: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem In-Reply-To: References: Message-ID: Mengxing, It is nice meeting you. I have seen a situation where the amount of RAM on a node can affect mmfsck times. Do all the nodes have the same amount of RAM, or does the slow running node have less RAM? Best Regards, Eric Eric Sperley SDI Architect "Carpe Diem" IBM Systems esperle at us.ibm.com +15033088721 From: Mengxing Cheng To: "gpfsug-discuss at spectrumscale.org" Date: 03/09/2017 11:24 AM Subject: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem Sent by: gpfsug-discuss-bounces at spectrumscale.org Dear all, My name is Mengxing Cheng and I am a HPC system administrator at the University of Chicago. We have a GSS26 running gss2.5.10.3-3b and gpfs-4.2.0.3. Recently, we run mmfsck on a relatively small filesystem with 14TB block and 73863102 inodes but it was unusually slow so as to not be able to finish in 48 hours. In contrast, mmfsck run on a filesystem with the same size and inodes but sitting on a traditional IBM DS3512 storage took only 2 hours to complete. In particular, the mmfsck run in parallel using 3 nodes within the GSS storage cluster, we notice that one gss storage server scans inodes much slower than the other gss storage server and the quorum service node. Has anyone experience the same mmfsck performance issue? Could anyone make recommendation to troubleshoot and improve mmfsck performance? Thank you! Mengxing --- Mengxing Cheng, Ph.D. HPC System Administrator Research Computing Center The University of Chicago 5607 S. Drexel Ave. Chicago, IL 60637 email: mxcheng at uchicago.edu phone: (773) 702-4070 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 1C803747.gif Type: image/gif Size: 481 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecblank.gif Type: image/gif Size: 45 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 1C166252.gif Type: image/gif Size: 2322 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From mxcheng at uchicago.edu Thu Mar 9 21:24:21 2017 From: mxcheng at uchicago.edu (Mengxing Cheng) Date: Thu, 9 Mar 2017 21:24:21 +0000 Subject: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem In-Reply-To: References: , Message-ID: Eric, thank you very much for replying. Here is the memory configuration and current usage. Note that mmfsck is not running now. The two gss servers have the same 256GB memory and the service node has 128GB. 1. service node: total used free shared buffers cached Mem: 125 58 66 0 0 4 -/+ buffers/cache: 53 71 Swap: 7 0 7 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12990 root 0 -20 71.0g 43g 885m S 7.6 34.4 9306:00 mmfsd 2. gss nodes: ==================================== total used free shared buff/cache available Mem: 251 210 37 0 4 36 Swap: 3 0 3 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 36770 root 0 -20 0.216t 0.192t 2.667g S 48.9 78.1 75684:09 /usr/lpp/mmfs/bin/mmfsd The gss nodes' memory usage is so high because their pagepool is set to 192GB while the service node has 16GB pagepool. Mengxing --- Mengxing Cheng, Ph.D. HPC System Administrator Research Computing Center The University of Chicago 5607 S. Drexel Ave. Chicago, IL 60637 email: mxcheng at uchicago.edu phone: (773) 702-4070 ________________________________ From: Eric Sperley [esperle at us.ibm.com] Sent: Thursday, March 09, 2017 3:13 PM To: Mengxing Cheng Cc: gpfsug main discussion list Subject: Re: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem Mengxing, It is nice meeting you. I have seen a situation where the amount of RAM on a node can affect mmfsck times. Do all the nodes have the same amount of RAM, or does the slow running node have less RAM? Best Regards, Eric [cid:3__=8FBB0A4DDFE7DC3F8f9e8a93df938690918c8FB@] Eric Sperley SDI Architect "Carpe Diem" IBM Systems esperle at us.ibm.com +15033088721 [Inactive hide details for Mengxing Cheng ---03/09/2017 11:24:02 AM---Dear all, My name is Mengxing Cheng and I am a HPC system]Mengxing Cheng ---03/09/2017 11:24:02 AM---Dear all, My name is Mengxing Cheng and I am a HPC system administrator at the University of Chicago From: Mengxing Cheng To: "gpfsug-discuss at spectrumscale.org" Date: 03/09/2017 11:24 AM Subject: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Dear all, My name is Mengxing Cheng and I am a HPC system administrator at the University of Chicago. We have a GSS26 running gss2.5.10.3-3b and gpfs-4.2.0.3. Recently, we run mmfsck on a relatively small filesystem with 14TB block and 73863102 inodes but it was unusually slow so as to not be able to finish in 48 hours. In contrast, mmfsck run on a filesystem with the same size and inodes but sitting on a traditional IBM DS3512 storage took only 2 hours to complete. In particular, the mmfsck run in parallel using 3 nodes within the GSS storage cluster, we notice that one gss storage server scans inodes much slower than the other gss storage server and the quorum service node. Has anyone experience the same mmfsck performance issue? Could anyone make recommendation to troubleshoot and improve mmfsck performance? Thank you! Mengxing --- Mengxing Cheng, Ph.D. HPC System Administrator Research Computing Center The University of Chicago 5607 S. Drexel Ave. Chicago, IL 60637 email: mxcheng at uchicago.edu phone: (773) 702-4070 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 1C803747.gif Type: image/gif Size: 481 bytes Desc: 1C803747.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecblank.gif Type: image/gif Size: 45 bytes Desc: ecblank.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 1C166252.gif Type: image/gif Size: 2322 bytes Desc: 1C166252.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: graycol.gif URL: From r.sobey at imperial.ac.uk Fri Mar 10 09:07:08 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Fri, 10 Mar 2017 09:07:08 +0000 Subject: [gpfsug-discuss] Running gpfs.snap outside of problems In-Reply-To: References: , Message-ID: Thanks Kevin and J-F, I appreciate your thoughts. I?ll go with ?no? then as my answer ? Cheers Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Kevin D Johnson Sent: 09 March 2017 19:48 To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Running gpfs.snap outside of problems I've never heard of running a gpfs.snap every few hours, but it is technically possible. Probably not advisable except in the most limited of circumstances. I wouldn't do it unless Support or another technical resource you trust says to do so. You can limit the snap to run on a few nodes versus a whole cluster and that way limit the effect it has overall. You can also specify a directory outside GPFS to write the snap file. Kevin D. Johnson, MBA, MAFM Spectrum Computing, Senior Managing Consultant IBM Certified Deployment Professional - Spectrum Scale V4.1.1 IBM Certified Deployment Professional - Cloud Object Storage V3.8 IBM Certified Solution Advisor - Spectrum Computing V1 720.349.6199 - kevindjo at us.ibm.com ----- Original message ----- From: Jan-Frode Myklebust Sent by: gpfsug-discuss-bounces at spectrumscale.org To: "gpfsug-discuss at spectrumscale.org" Cc: Subject: Re: [gpfsug-discuss] Running gpfs.snap outside of problems Date: Thu, Mar 9, 2017 11:30 AM There's a manual for it now.. and it points out "The tool impacts performance.." Also it has caused mmfsd crashes for me earlier, so I've learned to be weary of running it.. The manual also says it's collecting using mmfsadm, and the mmfsadm manual warns that it might cause GPFS to fail ("in certain rare cases"). I wouldn't dare running it every few hours. -jf tor. 9. mar. 2017 kl. 17.12 skrev Sobey, Richard A >: Hi all, Is there any best practice as to when to run a gpfs.snap, and the impact it will have on a healthy cluster? Some of my colleagues are keen to gather a snap for example every 2 hours to assist in problem determination. I can elaborate on the ?why? part of this if needed but I?m not fully convinced it would help. Said I?d look into it though so I?m asking the question. Cheers Richard _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From chair at spectrumscale.org Fri Mar 10 10:22:24 2017 From: chair at spectrumscale.org (GPFS UG Chair (Simon Thompson)) Date: Fri, 10 Mar 2017 10:22:24 +0000 Subject: [gpfsug-discuss] UK Spectrum Scale User Group Sponsorship packages Message-ID: Hi All, As we are developing the programme for the UK Spectrum Scale UG on 9th/10th May, we are again looking for commercial sponsors to support the evening event we hope to run. I have already contacted a number of contacts who kindly supported last year's event, however if you are interested in being a sponsor for the event, please get in touch with me directly and I can provide you with more information. Simon From Achim.Rehor at de.ibm.com Fri Mar 10 16:56:30 2017 From: Achim.Rehor at de.ibm.com (Achim Rehor) Date: Fri, 10 Mar 2017 17:56:30 +0100 Subject: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem In-Reply-To: References: , Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 7182 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 481 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 45 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 2322 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 105 bytes Desc: not available URL: From mxcheng at uchicago.edu Fri Mar 10 16:59:43 2017 From: mxcheng at uchicago.edu (Mengxing Cheng) Date: Fri, 10 Mar 2017 16:59:43 +0000 Subject: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem In-Reply-To: References: , , Message-ID: Achim, thank you very much! Mengxing --- Mengxing Cheng, Ph.D. HPC System Administrator Research Computing Center The University of Chicago 5607 S. Drexel Ave. Chicago, IL 60637 email: mxcheng at uchicago.edu phone: (773) 702-4070 ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Achim Rehor [Achim.Rehor at de.ibm.com] Sent: Friday, March 10, 2017 10:56 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem Hi, mmfsck is highly dependent on pagepool, so my best guess would be, the the EMS node with only 16GB of pagepool is sort of a bottleneck for the slow progress. You might want to try to temporarily raise the pagepool on the service node to .. lets say 64GB, or restrict the mmfsck to run only on the storage nodes. note: the fs mgr node will always be part of the mmfsck nodes. so if it is located on the service node, mmchmgr to a storage node first Mit freundlichen Gr??en / Kind regards Achim Rehor ________________________________ Software Technical Support Specialist AIX/ Emea HPC Support [cid:_1_D975D64CD975D0CC005D0FE8C12580DF] IBM Certified Advanced Technical Expert - Power Systems with AIX TSCC Software Service, Dept. 7922 Global Technology Services ________________________________ Phone: +49-7034-274-7862 IBM Deutschland E-Mail: Achim.Rehor at de.ibm.com Am Weiher 24 65451 Kelsterbach Germany ________________________________ IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter Gesch?ftsf?hrung: Martina Koederitz (Vorsitzende), Reinhard Reschke, Dieter Scholz, Gregor Pillen, Ivo Koerner, Christian Noll Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 WEEE-Reg.-Nr. DE 99369940 From: Mengxing Cheng To: Eric Sperley Cc: gpfsug main discussion list Date: 03/09/2017 10:24 PM Subject: Re: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Eric, thank you very much for replying. Here is the memory configuration and current usage. Note that mmfsck is not running now. The two gss servers have the same 256GB memory and the service node has 128GB. 1. service node: total used free shared buffers cached Mem: 125 58 66 0 0 4 -/+ buffers/cache: 53 71 Swap: 7 0 7 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12990 root 0 -20 71.0g 43g 885m S 7.6 34.4 9306:00 mmfsd 2. gss nodes: ==================================== total used free shared buff/cache available Mem: 251 210 37 0 4 36 Swap: 3 0 3 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 36770 root 0 -20 0.216t 0.192t 2.667g S 48.9 78.1 75684:09 /usr/lpp/mmfs/bin/mmfsd The gss nodes' memory usage is so high because their pagepool is set to 192GB while the service node has 16GB pagepool. Mengxing --- Mengxing Cheng, Ph.D. HPC System Administrator Research Computing Center The University of Chicago 5607 S. Drexel Ave. Chicago, IL 60637 email: mxcheng at uchicago.edu phone: (773) 702-4070 ________________________________ From: Eric Sperley [esperle at us.ibm.com] Sent: Thursday, March 09, 2017 3:13 PM To: Mengxing Cheng Cc: gpfsug main discussion list Subject: Re: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem Mengxing, It is nice meeting you. I have seen a situation where the amount of RAM on a node can affect mmfsck times. Do all the nodes have the same amount of RAM, or does the slow running node have less RAM? Best Regards, Eric [cid:_4_0B8EBB580B8EB73C005D0FE8C12580DF] Eric Sperley SDI Architect "Carpe Diem" IBM Systems esperle at us.ibm.com +15033088721 [Inactive hide details for Mengxing Cheng ---03/09/2017 11:24:02 AM---Dear all, My name is Mengxing Cheng and I am a HPC system]Mengxing Cheng ---03/09/2017 11:24:02 AM---Dear all, My name is Mengxing Cheng and I am a HPC system administrator at the University of Chicago From: Mengxing Cheng To: "gpfsug-discuss at spectrumscale.org" Date: 03/09/2017 11:24 AM Subject: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Dear all, My name is Mengxing Cheng and I am a HPC system administrator at the University of Chicago. We have a GSS26 running gss2.5.10.3-3b and gpfs-4.2.0.3. Recently, we run mmfsck on a relatively small filesystem with 14TB block and 73863102 inodes but it was unusually slow so as to not be able to finish in 48 hours. In contrast, mmfsck run on a filesystem with the same size and inodes but sitting on a traditional IBM DS3512 storage took only 2 hours to complete. In particular, the mmfsck run in parallel using 3 nodes within the GSS storage cluster, we notice that one gss storage server scans inodes much slower than the other gss storage server and the quorum service node. Has anyone experience the same mmfsck performance issue? Could anyone make recommendation to troubleshoot and improve mmfsck performance? Thank you! Mengxing --- Mengxing Cheng, Ph.D. HPC System Administrator Research Computing Center The University of Chicago 5607 S. Drexel Ave. Chicago, IL 60637 email: mxcheng at uchicago.edu phone: (773) 702-4070 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00007.gif Type: image/gif Size: 7182 bytes Desc: ATT00007.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00008.png Type: image/png Size: 481 bytes Desc: ATT00008.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00009.gif Type: image/gif Size: 45 bytes Desc: ATT00009.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00010.png Type: image/png Size: 2322 bytes Desc: ATT00010.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00011.gif Type: image/gif Size: 105 bytes Desc: ATT00011.gif URL: From Kevin.Buterbaugh at Vanderbilt.Edu Fri Mar 10 20:43:37 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 10 Mar 2017 20:43:37 +0000 Subject: [gpfsug-discuss] mmcrfs issue Message-ID: <43922F46-548D-4D34-89D4-74D35C6EB447@vanderbilt.edu> Hi All, We are testing out some flash storage. I created a couple of NSDs successfully (?): root at nec:~/gpfs# mmlsnsd -F File system Disk name NSD servers --------------------------------------------------------------------------- (free disk) nsd1 nec (free disk) nsd2 nec root at nec:~/gpfs# So I tried to create a filesystem: root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 GPFS: 6027-441 Unable to open disk 'nsd2' on node nec. GPFS: 6027-441 Unable to open disk 'nsd1' on node nec. No such device No such device GPFS: 6027-538 Error accessing disks. mmcrfs: 6027-1200 tscrfs failed. Cannot create gpfs0 mmcrfs: 6027-1639 Command failed. Examine previous error messages to determine cause. root at nec:~/gpfs# Does this output from readdescraw look normal? root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd16 NSD descriptor in sector 64 of /dev/zd16 NSDid: 0A0023D258C1C02C format version: 1403 Label: Paxos sector: -1931478434 number of sectors: 8192 isPdisk: 0 Comment: NSD descriptor for Thu Mar 9 14:50:52 2017 No Disk descriptor in sector 96 of /dev/zd16 No FS descriptor in sector 2048 of /dev/zd16 root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd32 NSD descriptor in sector 64 of /dev/zd32 NSDid: 0A0023D258C1C02B format version: 1403 Label: Paxos sector: -1880562609 number of sectors: 8192 isPdisk: 0 Comment: NSD descriptor for Thu Mar 9 14:50:51 2017 No Disk descriptor in sector 96 of /dev/zd32 No FS descriptor in sector 2048 of /dev/zd32 root at nec:~/gpfs# Thanks in advance, all? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From valdis.kletnieks at vt.edu Fri Mar 10 20:54:42 2017 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Fri, 10 Mar 2017 15:54:42 -0500 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: <43922F46-548D-4D34-89D4-74D35C6EB447@vanderbilt.edu> References: <43922F46-548D-4D34-89D4-74D35C6EB447@vanderbilt.edu> Message-ID: <16136.1489179282@turing-police.cc.vt.edu> On Fri, 10 Mar 2017 20:43:37 +0000, "Buterbaugh, Kevin L" said: > So I tried to create a filesystem: > > root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 What was in flash.stanza? -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 484 bytes Desc: not available URL: From Kevin.Buterbaugh at Vanderbilt.Edu Fri Mar 10 20:59:11 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 10 Mar 2017 20:59:11 +0000 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: <16136.1489179282@turing-police.cc.vt.edu> References: <43922F46-548D-4D34-89D4-74D35C6EB447@vanderbilt.edu> <16136.1489179282@turing-police.cc.vt.edu> Message-ID: <9FEBE666-48CA-438B-B0AE-9B954491A9FF@vanderbilt.edu> %nsd: nsd=nsd1 usage=metadataOnly failureGroup=1 pool=system servers=nec device=/dev/zd32 %nsd: nsd=nsd2 usage=dataOnly failureGroup=1 pool=gpfsdata servers=nec device=/dev/zd16 %pool: pool=system blockSize=1M usage=metadataOnly layoutMap=scatter allowWriteAffinity=no %pool: pool=gpfsdata blockSize=1M usage=dataOnly layoutMap=scatter allowWriteAffinity=no > On Mar 10, 2017, at 2:54 PM, valdis.kletnieks at vt.edu wrote: > > On Fri, 10 Mar 2017 20:43:37 +0000, "Buterbaugh, Kevin L" said: > >> So I tried to create a filesystem: >> >> root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 > > What was in flash.stanza? > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Paul.Sanchez at deshaw.com Fri Mar 10 21:36:10 2017 From: Paul.Sanchez at deshaw.com (Sanchez, Paul) Date: Fri, 10 Mar 2017 21:36:10 +0000 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: <43922F46-548D-4D34-89D4-74D35C6EB447@vanderbilt.edu> References: <43922F46-548D-4D34-89D4-74D35C6EB447@vanderbilt.edu> Message-ID: <09e39c995afc410c822c6e66665cb98b@mbxtoa1.winmail.deshaw.com> See: https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm GPFS has a limited set of device search specs it uses to find connected NSDs. When using exotic devices, you need to whitelist the devices yourself using the user exit script at /var/mmfs/etc/nsddevices. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Buterbaugh, Kevin L Sent: Friday, March 10, 2017 3:44 PM To: gpfsug main discussion list Subject: [gpfsug-discuss] mmcrfs issue Hi All, We are testing out some flash storage. I created a couple of NSDs successfully (?): root at nec:~/gpfs# mmlsnsd -F File system Disk name NSD servers --------------------------------------------------------------------------- (free disk) nsd1 nec (free disk) nsd2 nec root at nec:~/gpfs# So I tried to create a filesystem: root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 GPFS: 6027-441 Unable to open disk 'nsd2' on node nec. GPFS: 6027-441 Unable to open disk 'nsd1' on node nec. No such device No such device GPFS: 6027-538 Error accessing disks. mmcrfs: 6027-1200 tscrfs failed. Cannot create gpfs0 mmcrfs: 6027-1639 Command failed. Examine previous error messages to determine cause. root at nec:~/gpfs# Does this output from readdescraw look normal? root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd16 NSD descriptor in sector 64 of /dev/zd16 NSDid: 0A0023D258C1C02C format version: 1403 Label: Paxos sector: -1931478434 number of sectors: 8192 isPdisk: 0 Comment: NSD descriptor for Thu Mar 9 14:50:52 2017 No Disk descriptor in sector 96 of /dev/zd16 No FS descriptor in sector 2048 of /dev/zd16 root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd32 NSD descriptor in sector 64 of /dev/zd32 NSDid: 0A0023D258C1C02B format version: 1403 Label: Paxos sector: -1880562609 number of sectors: 8192 isPdisk: 0 Comment: NSD descriptor for Thu Mar 9 14:50:51 2017 No Disk descriptor in sector 96 of /dev/zd32 No FS descriptor in sector 2048 of /dev/zd32 root at nec:~/gpfs# Thanks in advance, all? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Fri Mar 10 21:44:05 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 10 Mar 2017 16:44:05 -0500 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: <09e39c995afc410c822c6e66665cb98b@mbxtoa1.winmail.deshaw.com> References: <43922F46-548D-4D34-89D4-74D35C6EB447@vanderbilt.edu> <09e39c995afc410c822c6e66665cb98b@mbxtoa1.winmail.deshaw.com> Message-ID: <4b1fbfca-95c4-a15a-ef70-fcf3e20d936d@nasa.gov> Those look like zvol's. Out of curiosity have you set sync=always on the filesystem root or zvols themselves? It's my understanding that without that you risk data loss since GPFS won't ever cause a sync to be issued to the zvol for zfs to flush acknowledged but uncommitted writes. -Aaron On 3/10/17 4:36 PM, Sanchez, Paul wrote: > See: > https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm > > > > GPFS has a limited set of device search specs it uses to find connected > NSDs. When using exotic devices, you need to whitelist the devices > yourself using the user exit script at /var/mmfs/etc/nsddevices. > > > > *From:*gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of > *Buterbaugh, Kevin L > *Sent:* Friday, March 10, 2017 3:44 PM > *To:* gpfsug main discussion list > *Subject:* [gpfsug-discuss] mmcrfs issue > > > > Hi All, > > > > We are testing out some flash storage. I created a couple of NSDs > successfully (?): > > > > root at nec:~/gpfs# mmlsnsd -F > > > > File system Disk name NSD servers > > --------------------------------------------------------------------------- > > (free disk) nsd1 nec > > (free disk) nsd2 nec > > > > root at nec:~/gpfs# > > > > So I tried to create a filesystem: > > > > root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j > scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 > > GPFS: 6027-441 Unable to open disk 'nsd2' on node nec. > > GPFS: 6027-441 Unable to open disk 'nsd1' on node nec. > > No such device > > No such device > > GPFS: 6027-538 Error accessing disks. > > mmcrfs: 6027-1200 tscrfs failed. Cannot create gpfs0 > > mmcrfs: 6027-1639 Command failed. Examine previous error messages to > determine cause. > > root at nec:~/gpfs# > > > > Does this output from readdescraw look normal? > > > > root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd16 > > NSD descriptor in sector 64 of /dev/zd16 > > NSDid: 0A0023D258C1C02C format version: 1403 Label: > > Paxos sector: -1931478434 number of sectors: 8192 isPdisk: 0 > > Comment: NSD descriptor for Thu Mar 9 14:50:52 2017 > > No Disk descriptor in sector 96 of /dev/zd16 > > No FS descriptor in sector 2048 of /dev/zd16 > > root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd32 > > NSD descriptor in sector 64 of /dev/zd32 > > NSDid: 0A0023D258C1C02B format version: 1403 Label: > > Paxos sector: -1880562609 number of sectors: 8192 isPdisk: 0 > > Comment: NSD descriptor for Thu Mar 9 14:50:51 2017 > > No Disk descriptor in sector 96 of /dev/zd32 > > No FS descriptor in sector 2048 of /dev/zd32 > > root at nec:~/gpfs# > > > > Thanks in advance, all? > > > > Kevin > > ? > > Kevin Buterbaugh - Senior System Administrator > > Vanderbilt University - Advanced Computing Center for Research and Education > > Kevin.Buterbaugh at vanderbilt.edu > - (615)875-9633 > > > > > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From daniel.kidger at uk.ibm.com Sat Mar 11 10:37:47 2017 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Sat, 11 Mar 2017 10:37:47 +0000 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: <4b1fbfca-95c4-a15a-ef70-fcf3e20d936d@nasa.gov> Message-ID: On the subject of using zvols for software Raid/ replication, can ask as a quick poll, how many people are doing this? And any feedback on stability, tuning and performance? Daniel IBM Technical Presales > On 10 Mar 2017, at 22:44, Aaron Knister wrote: > > Those look like zvol's. Out of curiosity have you set sync=always on the > filesystem root or zvols themselves? It's my understanding that without > that you risk data loss since GPFS won't ever cause a sync to be issued > to the zvol for zfs to flush acknowledged but uncommitted writes. > > -Aaron > >> On 3/10/17 4:36 PM, Sanchez, Paul wrote: >> See: >> https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm >> >> >> >> GPFS has a limited set of device search specs it uses to find connected >> NSDs. When using exotic devices, you need to whitelist the devices >> yourself using the user exit script at /var/mmfs/etc/nsddevices. >> >> >> >> *From:*gpfsug-discuss-bounces at spectrumscale.org >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of >> *Buterbaugh, Kevin L >> *Sent:* Friday, March 10, 2017 3:44 PM >> *To:* gpfsug main discussion list >> *Subject:* [gpfsug-discuss] mmcrfs issue >> >> >> >> Hi All, >> >> >> >> We are testing out some flash storage. I created a couple of NSDs >> successfully (?): >> >> >> >> root at nec:~/gpfs# mmlsnsd -F >> >> >> >> File system Disk name NSD servers >> >> --------------------------------------------------------------------------- >> >> (free disk) nsd1 nec >> >> (free disk) nsd2 nec >> >> >> >> root at nec:~/gpfs# >> >> >> >> So I tried to create a filesystem: >> >> >> >> root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j >> scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 >> >> GPFS: 6027-441 Unable to open disk 'nsd2' on node nec. >> >> GPFS: 6027-441 Unable to open disk 'nsd1' on node nec. >> >> No such device >> >> No such device >> >> GPFS: 6027-538 Error accessing disks. >> >> mmcrfs: 6027-1200 tscrfs failed. Cannot create gpfs0 >> >> mmcrfs: 6027-1639 Command failed. Examine previous error messages to >> determine cause. >> >> root at nec:~/gpfs# >> >> >> >> Does this output from readdescraw look normal? >> >> >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd16 >> >> NSD descriptor in sector 64 of /dev/zd16 >> >> NSDid: 0A0023D258C1C02C format version: 1403 Label: >> >> Paxos sector: -1931478434 number of sectors: 8192 isPdisk: 0 >> >> Comment: NSD descriptor for Thu Mar 9 14:50:52 2017 >> >> No Disk descriptor in sector 96 of /dev/zd16 >> >> No FS descriptor in sector 2048 of /dev/zd16 >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd32 >> >> NSD descriptor in sector 64 of /dev/zd32 >> >> NSDid: 0A0023D258C1C02B format version: 1403 Label: >> >> Paxos sector: -1880562609 number of sectors: 8192 isPdisk: 0 >> >> Comment: NSD descriptor for Thu Mar 9 14:50:51 2017 >> >> No Disk descriptor in sector 96 of /dev/zd32 >> >> No FS descriptor in sector 2048 of /dev/zd32 >> >> root at nec:~/gpfs# >> >> >> >> Thanks in advance, all? >> >> >> >> Kevin >> >> ? >> >> Kevin Buterbaugh - Senior System Administrator >> >> Vanderbilt University - Advanced Computing Center for Research and Education >> >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss >Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Mon Mar 13 13:50:04 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 13 Mar 2017 13:50:04 +0000 Subject: [gpfsug-discuss] Registration Reminder: GPFSUG-USA Meeting at NERSC April 4-5th In-Reply-To: References: Message-ID: Here?s a reminder to register for the upcoming US user group meeting- Registration deadline is March 24th! Register here: https://www.nersc.gov/research-and-development/data-analytics/spectrum-user-group-meeting/ Bob Oesterlin Sr Principal Storage Engineer, Nuance -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Mon Mar 13 20:44:01 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Mon, 13 Mar 2017 20:44:01 +0000 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: References: Message-ID: <357B0FF5-4B7C-4941-8496-BA70D011A898@vanderbilt.edu> Hi All, Two things: 1) Paul?s suggestion to look at the nsddevices script was the answer I needed to fix my mmcrfs issue. Thanks. 2) I am also interested in hearing if anyone is using ZFS to create the equivalent of RAID 8+2P hardware RAID 6 LUNs and presenting that to GPFS to use as disks? Thanks? Kevin On Mar 11, 2017, at 4:37 AM, Daniel Kidger > wrote: On the subject of using zvols for software Raid/ replication, can ask as a quick poll, how many people are doing this? And any feedback on stability, tuning and performance? Daniel IBM Technical Presales > On 10 Mar 2017, at 22:44, Aaron Knister > wrote: > > Those look like zvol's. Out of curiosity have you set sync=always on the > filesystem root or zvols themselves? It's my understanding that without > that you risk data loss since GPFS won't ever cause a sync to be issued > to the zvol for zfs to flush acknowledged but uncommitted writes. > > -Aaron > >> On 3/10/17 4:36 PM, Sanchez, Paul wrote: >> See: >> https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm >> >> >> >> GPFS has a limited set of device search specs it uses to find connected >> NSDs. When using exotic devices, you need to whitelist the devices >> yourself using the user exit script at /var/mmfs/etc/nsddevices. >> >> >> >> *From:*gpfsug-discuss-bounces at spectrumscale.org >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of >> *Buterbaugh, Kevin L >> *Sent:* Friday, March 10, 2017 3:44 PM >> *To:* gpfsug main discussion list >> *Subject:* [gpfsug-discuss] mmcrfs issue >> >> >> >> Hi All, >> >> >> >> We are testing out some flash storage. I created a couple of NSDs >> successfully (?): >> >> >> >> root at nec:~/gpfs# mmlsnsd -F >> >> >> >> File system Disk name NSD servers >> >> --------------------------------------------------------------------------- >> >> (free disk) nsd1 nec >> >> (free disk) nsd2 nec >> >> >> >> root at nec:~/gpfs# >> >> >> >> So I tried to create a filesystem: >> >> >> >> root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j >> scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 >> >> GPFS: 6027-441 Unable to open disk 'nsd2' on node nec. >> >> GPFS: 6027-441 Unable to open disk 'nsd1' on node nec. >> >> No such device >> >> No such device >> >> GPFS: 6027-538 Error accessing disks. >> >> mmcrfs: 6027-1200 tscrfs failed. Cannot create gpfs0 >> >> mmcrfs: 6027-1639 Command failed. Examine previous error messages to >> determine cause. >> >> root at nec:~/gpfs# >> >> >> >> Does this output from readdescraw look normal? >> >> >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd16 >> >> NSD descriptor in sector 64 of /dev/zd16 >> >> NSDid: 0A0023D258C1C02C format version: 1403 Label: >> >> Paxos sector: -1931478434 number of sectors: 8192 isPdisk: 0 >> >> Comment: NSD descriptor for Thu Mar 9 14:50:52 2017 >> >> No Disk descriptor in sector 96 of /dev/zd16 >> >> No FS descriptor in sector 2048 of /dev/zd16 >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd32 >> >> NSD descriptor in sector 64 of /dev/zd32 >> >> NSDid: 0A0023D258C1C02B format version: 1403 Label: >> >> Paxos sector: -1880562609 number of sectors: 8192 isPdisk: 0 >> >> Comment: NSD descriptor for Thu Mar 9 14:50:51 2017 >> >> No Disk descriptor in sector 96 of /dev/zd32 >> >> No FS descriptor in sector 2048 of /dev/zd32 >> >> root at nec:~/gpfs# >> >> >> >> Thanks in advance, all? >> >> >> >> Kevin >> >> ? >> >> Kevin Buterbaugh - Senior System Administrator >> >> Vanderbilt University - Advanced Computing Center for Research and Education >> >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Tue Mar 14 00:06:29 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Mon, 13 Mar 2017 20:06:29 -0400 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: <357B0FF5-4B7C-4941-8496-BA70D011A898@vanderbilt.edu> References: <357B0FF5-4B7C-4941-8496-BA70D011A898@vanderbilt.edu> Message-ID: I was doing this in testing (with fantastic performance too) until I realized the issue with ZFS's behavior with direct io on zvols (e.g. not flushing a write to stable storage after acknowledging it to GPFS). After setting the sync=always parameter to not lose data in the event of a crash or power outage the write performance became unbearably slow (under 100MB/s of writes for an 8+2 RAIDZ2 if I recall correctly). I even tried adding a battery-backed PCIe write cache (http://www.netlist.com/products/vault-memory-storage/expressvault-pcIe-ev3/default.aspx) as a log device to the zpool but the performance was still really slow. I posted to the ZFS mailing list asking about how to optimize for a large block streaming workload but I didn't many bites (http://list.zfsonlinux.org/pipermail/zfs-discuss/2016-February/024851.html). I've got an RFE open with IBM (https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84994) to see if the behavior of GPFS could be changed such that it would issue explicit cache flushes that would allow it to work with ZFS (it might even be beneficial in FPO environments too). -Aaron On 3/13/17 4:44 PM, Buterbaugh, Kevin L wrote: > Hi All, > > Two things: > > 1) Paul?s suggestion to look at the nsddevices script was the answer I > needed to fix my mmcrfs issue. Thanks. > > 2) I am also interested in hearing if anyone is using ZFS to create the > equivalent of RAID 8+2P hardware RAID 6 LUNs and presenting that to GPFS > to use as disks? > > Thanks? > > Kevin > >> On Mar 11, 2017, at 4:37 AM, Daniel Kidger > > wrote: >> >> On the subject of using zvols for software Raid/ replication, can ask >> as a quick poll, how many people are doing this? >> >> And any feedback on stability, tuning and performance? >> >> Daniel >> IBM Technical Presales >> >> > On 10 Mar 2017, at 22:44, Aaron Knister > wrote: >> > >> > Those look like zvol's. Out of curiosity have you set sync=always on the >> > filesystem root or zvols themselves? It's my understanding that without >> > that you risk data loss since GPFS won't ever cause a sync to be issued >> > to the zvol for zfs to flush acknowledged but uncommitted writes. >> > >> > -Aaron >> > >> >> On 3/10/17 4:36 PM, Sanchez, Paul wrote: >> >> See: >> >> https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm >> >> >> >> >> >> >> >> GPFS has a limited set of device search specs it uses to find connected >> >> NSDs. When using exotic devices, you need to whitelist the devices >> >> yourself using the user exit script at /var/mmfs/etc/nsddevices. >> >> >> >> >> >> >> >> *From:*gpfsug-discuss-bounces at spectrumscale.org >> >> >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of >> >> *Buterbaugh, Kevin L >> >> *Sent:* Friday, March 10, 2017 3:44 PM >> >> *To:* gpfsug main discussion list >> >> *Subject:* [gpfsug-discuss] mmcrfs issue >> >> >> >> >> >> >> >> Hi All, >> >> >> >> >> >> >> >> We are testing out some flash storage. I created a couple of NSDs >> >> successfully (?): >> >> >> >> >> >> >> >> root at nec:~/gpfs# mmlsnsd -F >> >> >> >> >> >> >> >> File system Disk name NSD servers >> >> >> >> --------------------------------------------------------------------------- >> >> >> >> (free disk) nsd1 nec >> >> >> >> (free disk) nsd2 nec >> >> >> >> >> >> >> >> root at nec:~/gpfs# >> >> >> >> >> >> >> >> So I tried to create a filesystem: >> >> >> >> >> >> >> >> root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j >> >> scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 >> >> >> >> GPFS: 6027-441 Unable to open disk 'nsd2' on node nec. >> >> >> >> GPFS: 6027-441 Unable to open disk 'nsd1' on node nec. >> >> >> >> No such device >> >> >> >> No such device >> >> >> >> GPFS: 6027-538 Error accessing disks. >> >> >> >> mmcrfs: 6027-1200 tscrfs failed. Cannot create gpfs0 >> >> >> >> mmcrfs: 6027-1639 Command failed. Examine previous error messages to >> >> determine cause. >> >> >> >> root at nec:~/gpfs# >> >> >> >> >> >> >> >> Does this output from readdescraw look normal? >> >> >> >> >> >> >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd16 >> >> >> >> NSD descriptor in sector 64 of /dev/zd16 >> >> >> >> NSDid: 0A0023D258C1C02C format version: 1403 Label: >> >> >> >> Paxos sector: -1931478434 number of sectors: 8192 isPdisk: 0 >> >> >> >> Comment: NSD descriptor for Thu Mar 9 14:50:52 2017 >> >> >> >> No Disk descriptor in sector 96 of /dev/zd16 >> >> >> >> No FS descriptor in sector 2048 of /dev/zd16 >> >> >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd32 >> >> >> >> NSD descriptor in sector 64 of /dev/zd32 >> >> >> >> NSDid: 0A0023D258C1C02B format version: 1403 Label: >> >> >> >> Paxos sector: -1880562609 number of sectors: 8192 isPdisk: 0 >> >> >> >> Comment: NSD descriptor for Thu Mar 9 14:50:51 2017 >> >> >> >> No Disk descriptor in sector 96 of /dev/zd32 >> >> >> >> No FS descriptor in sector 2048 of /dev/zd32 >> >> >> >> root at nec:~/gpfs# >> >> >> >> >> >> >> >> Thanks in advance, all? >> >> >> >> >> >> >> >> Kevin >> >> >> >> ? >> >> >> >> Kevin Buterbaugh - Senior System Administrator >> >> >> >> Vanderbilt University - Advanced Computing Center for Research and Education >> >> >> >> Kevin.Buterbaugh at vanderbilt.edu >> >> - (615)875-9633 >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> gpfsug-discuss mailing list >> >> gpfsug-discuss at spectrumscale.org >> >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> > >> > -- >> > Aaron Knister >> > NASA Center for Climate Simulation (Code 606.2) >> > Goddard Space Flight Center >> > (301) 286-2776 >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > >> Unless stated otherwise above: >> IBM United Kingdom Limited - Registered in England and Wales with >> number 741598. >> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From valdis.kletnieks at vt.edu Tue Mar 14 18:15:17 2017 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Tue, 14 Mar 2017 14:15:17 -0400 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: References: <357B0FF5-4B7C-4941-8496-BA70D011A898@vanderbilt.edu> Message-ID: <16774.1489515317@turing-police.cc.vt.edu> On Mon, 13 Mar 2017 20:06:29 -0400, Aaron Knister said: > After setting the sync=always parameter to not lose data in the event of > a crash or power outage the write performance became unbearably slow > (under 100MB/s of writes for an 8+2 RAIDZ2 if I recall correctly). I Not at all surprising. Forcing a 'sync every write' has been painful for pretty much everything - first time I saw it was on a SunOS 3.2 box doing NFS over 3 decades ago. > I've got an RFE open with IBM > (https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84994) > to see if the behavior of GPFS could be changed such that it would issue > explicit cache flushes that would allow it to work with ZFS (it might > even be beneficial in FPO environments too). Do you have a suggestion for how to do this without it turning into 'sync=always' just done at a different layer in the stack? -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 484 bytes Desc: not available URL: From aaron.s.knister at nasa.gov Tue Mar 14 18:31:55 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 14 Mar 2017 14:31:55 -0400 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: <16774.1489515317@turing-police.cc.vt.edu> References: <357B0FF5-4B7C-4941-8496-BA70D011A898@vanderbilt.edu> <16774.1489515317@turing-police.cc.vt.edu> Message-ID: <411ef927-533e-c2b6-289f-44f0d8845c4f@nasa.gov> On 3/14/17 2:15 PM, valdis.kletnieks at vt.edu wrote: > On Mon, 13 Mar 2017 20:06:29 -0400, Aaron Knister said: >> After setting the sync=always parameter to not lose data in the event of >> a crash or power outage the write performance became unbearably slow >> (under 100MB/s of writes for an 8+2 RAIDZ2 if I recall correctly). I > > Not at all surprising. Forcing a 'sync every write' has been painful for pretty > much everything - first time I saw it was on a SunOS 3.2 box doing NFS over 3 > decades ago. > You know, what's interesting is when you do direct I/O to a linux md RAID device each acknowledged I/O to the md device results in SCSI SYNCHRONIZE CACHE commands being sent to the underlying member devices. I consider that doing a sync of sorts after each write and as long as everything is properly aligned you can get pretty fantastic performance from this. No data checksums, though :( >> I've got an RFE open with IBM >> (https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84994) >> to see if the behavior of GPFS could be changed such that it would issue >> explicit cache flushes that would allow it to work with ZFS (it might >> even be beneficial in FPO environments too). > > Do you have a suggestion for how to do this without it turning into 'sync=always' > just done at a different layer in the stack? > I don't think that GPFS would need to issue a sync after *every* write. I think the key is more that GPFS is aware of the state of acknowledged but uncommitted writes so that if there's something *really* important to be written (i.e. an internal piece of metadata) GPFS can force it to be committed to avoid GPFS itself becoming inconsistent. You don't want to have to mmfsck after a power outage/crash if there were lost writes that GPFS assumed were committed. -Aaron > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: OpenPGP digital signature URL: From luke.raimbach at googlemail.com Tue Mar 14 21:59:50 2017 From: luke.raimbach at googlemail.com (Luke Raimbach) Date: Tue, 14 Mar 2017 21:59:50 +0000 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: References: <357B0FF5-4B7C-4941-8496-BA70D011A898@vanderbilt.edu> Message-ID: Can I ask what the fascination with zvols is? Using a copy-on-write file system to underpin another block based file system seems counterintuitive. Perhaps I've missed something vital, in which case I'd be delighted to have my eyes opened! On Tue, 14 Mar 2017, 00:06 Aaron Knister, wrote: > I was doing this in testing (with fantastic performance too) until I > realized the issue with ZFS's behavior with direct io on zvols (e.g. not > flushing a write to stable storage after acknowledging it to GPFS). > After setting the sync=always parameter to not lose data in the event of > a crash or power outage the write performance became unbearably slow > (under 100MB/s of writes for an 8+2 RAIDZ2 if I recall correctly). I > even tried adding a battery-backed PCIe write cache > ( > http://www.netlist.com/products/vault-memory-storage/expressvault-pcIe-ev3/default.aspx > ) > as a log device to the zpool but the performance was still really slow. > I posted to the ZFS mailing list asking about how to optimize for a > large block streaming workload but I didn't many bites > ( > http://list.zfsonlinux.org/pipermail/zfs-discuss/2016-February/024851.html > ). > > I've got an RFE open with IBM > ( > https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84994 > ) > to see if the behavior of GPFS could be changed such that it would issue > explicit cache flushes that would allow it to work with ZFS (it might > even be beneficial in FPO environments too). > > -Aaron > > On 3/13/17 4:44 PM, Buterbaugh, Kevin L wrote: > > Hi All, > > > > Two things: > > > > 1) Paul?s suggestion to look at the nsddevices script was the answer I > > needed to fix my mmcrfs issue. Thanks. > > > > 2) I am also interested in hearing if anyone is using ZFS to create the > > equivalent of RAID 8+2P hardware RAID 6 LUNs and presenting that to GPFS > > to use as disks? > > > > Thanks? > > > > Kevin > > > >> On Mar 11, 2017, at 4:37 AM, Daniel Kidger >> > wrote: > >> > >> On the subject of using zvols for software Raid/ replication, can ask > >> as a quick poll, how many people are doing this? > >> > >> And any feedback on stability, tuning and performance? > >> > >> Daniel > >> IBM Technical Presales > >> > >> > On 10 Mar 2017, at 22:44, Aaron Knister > wrote: > >> > > >> > Those look like zvol's. Out of curiosity have you set sync=always on > the > >> > filesystem root or zvols themselves? It's my understanding that > without > >> > that you risk data loss since GPFS won't ever cause a sync to be > issued > >> > to the zvol for zfs to flush acknowledged but uncommitted writes. > >> > > >> > -Aaron > >> > > >> >> On 3/10/17 4:36 PM, Sanchez, Paul wrote: > >> >> See: > >> >> > https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm > >> >> > >> >> > >> >> > >> >> GPFS has a limited set of device search specs it uses to find > connected > >> >> NSDs. When using exotic devices, you need to whitelist the devices > >> >> yourself using the user exit script at /var/mmfs/etc/nsddevices. > >> >> > >> >> > >> >> > >> >> *From:*gpfsug-discuss-bounces at spectrumscale.org > >> > >> >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of > >> >> *Buterbaugh, Kevin L > >> >> *Sent:* Friday, March 10, 2017 3:44 PM > >> >> *To:* gpfsug main discussion list > >> >> *Subject:* [gpfsug-discuss] mmcrfs issue > >> >> > >> >> > >> >> > >> >> Hi All, > >> >> > >> >> > >> >> > >> >> We are testing out some flash storage. I created a couple of NSDs > >> >> successfully (?): > >> >> > >> >> > >> >> > >> >> root at nec:~/gpfs# mmlsnsd -F > >> >> > >> >> > >> >> > >> >> File system Disk name NSD servers > >> >> > >> >> > --------------------------------------------------------------------------- > >> >> > >> >> (free disk) nsd1 nec > >> >> > >> >> (free disk) nsd2 nec > >> >> > >> >> > >> >> > >> >> root at nec:~/gpfs# > >> >> > >> >> > >> >> > >> >> So I tried to create a filesystem: > >> >> > >> >> > >> >> > >> >> root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j > >> >> scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 > >> >> > >> >> GPFS: 6027-441 Unable to open disk 'nsd2' on node nec. > >> >> > >> >> GPFS: 6027-441 Unable to open disk 'nsd1' on node nec. > >> >> > >> >> No such device > >> >> > >> >> No such device > >> >> > >> >> GPFS: 6027-538 Error accessing disks. > >> >> > >> >> mmcrfs: 6027-1200 tscrfs failed. Cannot create gpfs0 > >> >> > >> >> mmcrfs: 6027-1639 Command failed. Examine previous error messages to > >> >> determine cause. > >> >> > >> >> root at nec:~/gpfs# > >> >> > >> >> > >> >> > >> >> Does this output from readdescraw look normal? > >> >> > >> >> > >> >> > >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd16 > >> >> > >> >> NSD descriptor in sector 64 of /dev/zd16 > >> >> > >> >> NSDid: 0A0023D258C1C02C format version: 1403 Label: > >> >> > >> >> Paxos sector: -1931478434 number of sectors: 8192 isPdisk: 0 > >> >> > >> >> Comment: NSD descriptor for Thu Mar 9 14:50:52 2017 > >> >> > >> >> No Disk descriptor in sector 96 of /dev/zd16 > >> >> > >> >> No FS descriptor in sector 2048 of /dev/zd16 > >> >> > >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd32 > >> >> > >> >> NSD descriptor in sector 64 of /dev/zd32 > >> >> > >> >> NSDid: 0A0023D258C1C02B format version: 1403 Label: > >> >> > >> >> Paxos sector: -1880562609 number of sectors: 8192 isPdisk: 0 > >> >> > >> >> Comment: NSD descriptor for Thu Mar 9 14:50:51 2017 > >> >> > >> >> No Disk descriptor in sector 96 of /dev/zd32 > >> >> > >> >> No FS descriptor in sector 2048 of /dev/zd32 > >> >> > >> >> root at nec:~/gpfs# > >> >> > >> >> > >> >> > >> >> Thanks in advance, all? > >> >> > >> >> > >> >> > >> >> Kevin > >> >> > >> >> ? > >> >> > >> >> Kevin Buterbaugh - Senior System Administrator > >> >> > >> >> Vanderbilt University - Advanced Computing Center for Research and > Education > >> >> > >> >> Kevin.Buterbaugh at vanderbilt.edu Kevin.Buterbaugh at vanderbilt.edu> > >> >> - (615)875-9633 > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> _______________________________________________ > >> >> gpfsug-discuss mailing list > >> >> gpfsug-discuss at spectrumscale.org > >> >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> >> > >> > > >> > -- > >> > Aaron Knister > >> > NASA Center for Climate Simulation (Code 606.2) > >> > Goddard Space Flight Center > >> > (301) 286-2776 > >> > _______________________________________________ > >> > gpfsug-discuss mailing list > >> > gpfsug-discuss at spectrumscale.org > >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > >> Unless stated otherwise above: > >> IBM United Kingdom Limited - Registered in England and Wales with > >> number 741598. > >> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 > 3AU > >> > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Tue Mar 14 22:16:43 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 14 Mar 2017 18:16:43 -0400 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: References: <357B0FF5-4B7C-4941-8496-BA70D011A898@vanderbilt.edu> Message-ID: <2d4fa452-3137-0a44-fd18-bed4b9923279@nasa.gov> For me it's the protection against bitrot and added protection against silent data corruption and in theory the write caching offered by adding log devices that could help with small random writes (although there are other problems with ZFS + synchronous workloads that stop this from actually materializing). -Aaron On 3/14/17 5:59 PM, Luke Raimbach wrote: > Can I ask what the fascination with zvols is? Using a copy-on-write file > system to underpin another block based file system seems > counterintuitive. Perhaps I've missed something vital, in which case I'd > be delighted to have my eyes opened! > > On Tue, 14 Mar 2017, 00:06 Aaron Knister, > wrote: > > I was doing this in testing (with fantastic performance too) until I > realized the issue with ZFS's behavior with direct io on zvols (e.g. not > flushing a write to stable storage after acknowledging it to GPFS). > After setting the sync=always parameter to not lose data in the event of > a crash or power outage the write performance became unbearably slow > (under 100MB/s of writes for an 8+2 RAIDZ2 if I recall correctly). I > even tried adding a battery-backed PCIe write cache > (http://www.netlist.com/products/vault-memory-storage/expressvault-pcIe-ev3/default.aspx) > as a log device to the zpool but the performance was still really slow. > I posted to the ZFS mailing list asking about how to optimize for a > large block streaming workload but I didn't many bites > (http://list.zfsonlinux.org/pipermail/zfs-discuss/2016-February/024851.html). > > I've got an RFE open with IBM > (https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84994) > to see if the behavior of GPFS could be changed such that it would issue > explicit cache flushes that would allow it to work with ZFS (it might > even be beneficial in FPO environments too). > > -Aaron > > On 3/13/17 4:44 PM, Buterbaugh, Kevin L wrote: > > Hi All, > > > > Two things: > > > > 1) Paul?s suggestion to look at the nsddevices script was the answer I > > needed to fix my mmcrfs issue. Thanks. > > > > 2) I am also interested in hearing if anyone is using ZFS to create the > > equivalent of RAID 8+2P hardware RAID 6 LUNs and presenting that to GPFS > > to use as disks? > > > > Thanks? > > > > Kevin > > > >> On Mar 11, 2017, at 4:37 AM, Daniel Kidger > >> >> wrote: > >> > >> On the subject of using zvols for software Raid/ replication, can ask > >> as a quick poll, how many people are doing this? > >> > >> And any feedback on stability, tuning and performance? > >> > >> Daniel > >> IBM Technical Presales > >> > >> > On 10 Mar 2017, at 22:44, Aaron Knister > >> > wrote: > >> > > >> > Those look like zvol's. Out of curiosity have you set sync=always on the > >> > filesystem root or zvols themselves? It's my understanding that without > >> > that you risk data loss since GPFS won't ever cause a sync to be issued > >> > to the zvol for zfs to flush acknowledged but uncommitted writes. > >> > > >> > -Aaron > >> > > >> >> On 3/10/17 4:36 PM, Sanchez, Paul wrote: > >> >> See: > >> >> https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm > >> >> > >> >> > >> >> > >> >> GPFS has a limited set of device search specs it uses to find connected > >> >> NSDs. When using exotic devices, you need to whitelist the devices > >> >> yourself using the user exit script at /var/mmfs/etc/nsddevices. > >> >> > >> >> > >> >> > >> >> *From:*gpfsug-discuss-bounces at spectrumscale.org > > >> > > >> >> [mailto:gpfsug-discuss-bounces at spectrumscale.org > ] *On Behalf Of > >> >> *Buterbaugh, Kevin L > >> >> *Sent:* Friday, March 10, 2017 3:44 PM > >> >> *To:* gpfsug main discussion list > >> >> *Subject:* [gpfsug-discuss] mmcrfs issue > >> >> > >> >> > >> >> > >> >> Hi All, > >> >> > >> >> > >> >> > >> >> We are testing out some flash storage. I created a couple of NSDs > >> >> successfully (?): > >> >> > >> >> > >> >> > >> >> root at nec:~/gpfs# mmlsnsd -F > >> >> > >> >> > >> >> > >> >> File system Disk name NSD servers > >> >> > >> >> --------------------------------------------------------------------------- > >> >> > >> >> (free disk) nsd1 nec > >> >> > >> >> (free disk) nsd2 nec > >> >> > >> >> > >> >> > >> >> root at nec:~/gpfs# > >> >> > >> >> > >> >> > >> >> So I tried to create a filesystem: > >> >> > >> >> > >> >> > >> >> root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j > >> >> scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 > >> >> > >> >> GPFS: 6027-441 Unable to open disk 'nsd2' on node nec. > >> >> > >> >> GPFS: 6027-441 Unable to open disk 'nsd1' on node nec. > >> >> > >> >> No such device > >> >> > >> >> No such device > >> >> > >> >> GPFS: 6027-538 Error accessing disks. > >> >> > >> >> mmcrfs: 6027-1200 tscrfs failed. Cannot create gpfs0 > >> >> > >> >> mmcrfs: 6027-1639 Command failed. Examine previous error messages to > >> >> determine cause. > >> >> > >> >> root at nec:~/gpfs# > >> >> > >> >> > >> >> > >> >> Does this output from readdescraw look normal? > >> >> > >> >> > >> >> > >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd16 > >> >> > >> >> NSD descriptor in sector 64 of /dev/zd16 > >> >> > >> >> NSDid: 0A0023D258C1C02C format version: 1403 Label: > >> >> > >> >> Paxos sector: -1931478434 number of sectors: 8192 isPdisk: 0 > >> >> > >> >> Comment: NSD descriptor for Thu Mar 9 14:50:52 2017 > >> >> > >> >> No Disk descriptor in sector 96 of /dev/zd16 > >> >> > >> >> No FS descriptor in sector 2048 of /dev/zd16 > >> >> > >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd32 > >> >> > >> >> NSD descriptor in sector 64 of /dev/zd32 > >> >> > >> >> NSDid: 0A0023D258C1C02B format version: 1403 Label: > >> >> > >> >> Paxos sector: -1880562609 number of sectors: 8192 isPdisk: 0 > >> >> > >> >> Comment: NSD descriptor for Thu Mar 9 14:50:51 2017 > >> >> > >> >> No Disk descriptor in sector 96 of /dev/zd32 > >> >> > >> >> No FS descriptor in sector 2048 of /dev/zd32 > >> >> > >> >> root at nec:~/gpfs# > >> >> > >> >> > >> >> > >> >> Thanks in advance, all? > >> >> > >> >> > >> >> > >> >> Kevin > >> >> > >> >> ? > >> >> > >> >> Kevin Buterbaugh - Senior System Administrator > >> >> > >> >> Vanderbilt University - Advanced Computing Center for Research and Education > >> >> > >> >> Kevin.Buterbaugh at vanderbilt.edu > > > > >> >> > - (615)875-9633 > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> _______________________________________________ > >> >> gpfsug-discuss mailing list > >> >> gpfsug-discuss at spectrumscale.org > >> >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> >> > >> > > >> > -- > >> > Aaron Knister > >> > NASA Center for Climate Simulation (Code 606.2) > >> > Goddard Space Flight Center > >> > (301) 286-2776 > >> > _______________________________________________ > >> > gpfsug-discuss mailing list > >> > gpfsug-discuss at spectrumscale.org > >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > >> Unless stated otherwise above: > >> IBM United Kingdom Limited - Registered in England and Wales with > >> number 741598. > >> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > >> > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From xhejtman at ics.muni.cz Wed Mar 15 09:50:31 2017 From: xhejtman at ics.muni.cz (Lukas Hejtmanek) Date: Wed, 15 Mar 2017 10:50:31 +0100 Subject: [gpfsug-discuss] default inode size Message-ID: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> Hello, does anyone know why with GPFS 4.x, there is inode size 4096B by default. Is there any problem with, e.g., only 1024B inode size? -- Luk?? Hejtm?nek From ulmer at ulmer.org Wed Mar 15 12:22:22 2017 From: ulmer at ulmer.org (Stephen Ulmer) Date: Wed, 15 Mar 2017 08:22:22 -0400 Subject: [gpfsug-discuss] default inode size In-Reply-To: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> References: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> Message-ID: <0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> You need 4K nodes to store encryption keys. You can also put other useful things in there, like extended attributes and (possibly) the entire file. Are you worried about wasting space? -- Stephen > On Mar 15, 2017, at 5:50 AM, Lukas Hejtmanek > wrote: > > Hello, > > does anyone know why with GPFS 4.x, there is inode size 4096B by default. > > Is there any problem with, e.g., only 1024B inode size? > > -- > Luk?? Hejtm?nek > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From xhejtman at ics.muni.cz Wed Mar 15 12:37:00 2017 From: xhejtman at ics.muni.cz (Lukas Hejtmanek) Date: Wed, 15 Mar 2017 13:37:00 +0100 Subject: [gpfsug-discuss] default inode size In-Reply-To: <0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> References: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> <0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> Message-ID: <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> On Wed, Mar 15, 2017 at 08:22:22AM -0400, Stephen Ulmer wrote: > You need 4K nodes to store encryption keys. You can also put other useful things in there, like extended attributes and (possibly) the entire file. > > Are you worried about wasting space? well, I have 1PB of capacity for data disks and 3TB of capacity for metadata (SSD). With 512B inodes, this ration seemed to be quite ok, while with 4k inodes, I run out of free space on SSD pretty fast. So I'm thinking about re-creating file system with smaller inode size. I don't think I will ever need encryption keys. -- Luk?? Hejtm?nek From luis.bolinches at fi.ibm.com Wed Mar 15 13:09:07 2017 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Wed, 15 Mar 2017 13:09:07 +0000 Subject: [gpfsug-discuss] default inode size In-Reply-To: <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> References: <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz>, <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz><0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> Message-ID: An HTML attachment was scrubbed... URL: From dieter.gorecki at atos.net Wed Mar 15 13:12:44 2017 From: dieter.gorecki at atos.net (GORECKI, DIETER) Date: Wed, 15 Mar 2017 13:12:44 +0000 Subject: [gpfsug-discuss] default inode size In-Reply-To: <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> References: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> <0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> Message-ID: Lukas, One other thing to consider is the storage of data inside the inode itself for very small files. GPFS has the ability to use the remaining [kilo]bytes of the inode to store the data of the file whenever the file is small enough to fit in. Anyone correct me if I am wrong, but with 4k inodes, you can store up to (4096-128 header) 3968 bytes of data. (without ILM) So regarding the size of the files you intend to store into your filesystem, it might be very interesting to take advantage of the performance of your SSD's to store small files. Regards, Dieter -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Lukas Hejtmanek Sent: Wednesday, March 15, 2017 1:37 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] default inode size On Wed, Mar 15, 2017 at 08:22:22AM -0400, Stephen Ulmer wrote: > You need 4K nodes to store encryption keys. You can also put other useful things in there, like extended attributes and (possibly) the entire file. > > Are you worried about wasting space? well, I have 1PB of capacity for data disks and 3TB of capacity for metadata (SSD). With 512B inodes, this ration seemed to be quite ok, while with 4k inodes, I run out of free space on SSD pretty fast. So I'm thinking about re-creating file system with smaller inode size. I don't think I will ever need encryption keys. -- Luk?? Hejtm?nek _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From xhejtman at ics.muni.cz Wed Mar 15 13:21:40 2017 From: xhejtman at ics.muni.cz (Lukas Hejtmanek) Date: Wed, 15 Mar 2017 14:21:40 +0100 Subject: [gpfsug-discuss] default inode size In-Reply-To: References: <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> <0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> Message-ID: <20170315132140.v5rfagqy4hqzqsfh@ics.muni.cz> Hello, On Wed, Mar 15, 2017 at 01:09:07PM +0000, Luis Bolinches wrote: > My 2 cents. Before even thinking too much on that path I would check the > following > ? > - What the is physical size on those SSD if they are already 4K you won't > "save" anything size of what? > - Do you use small (>3.5Kb) files? If so I would still keep 4K inodes > - Check if 512 can hold NSDv2 format, from top of my head it cannot but > pls check. If so, do you want to use a format that anyway is going to fade > away and lose the goodies that format brings? > ? > I am sure few other pros (and cons) can be put on but I would start with > those 3 I was thinking about 1024B inodes. Is this still not enough for NSDv2 format? -- Luk?? Hejtm?nek From luis.bolinches at fi.ibm.com Wed Mar 15 13:24:11 2017 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Wed, 15 Mar 2017 13:24:11 +0000 Subject: [gpfsug-discuss] default inode size In-Reply-To: <20170315132140.v5rfagqy4hqzqsfh@ics.muni.cz> References: <20170315132140.v5rfagqy4hqzqsfh@ics.muni.cz>, <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz><20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz><0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> Message-ID: An HTML attachment was scrubbed... URL: From xhejtman at ics.muni.cz Wed Mar 15 13:26:09 2017 From: xhejtman at ics.muni.cz (Lukas Hejtmanek) Date: Wed, 15 Mar 2017 14:26:09 +0100 Subject: [gpfsug-discuss] default inode size In-Reply-To: References: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> <0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> Message-ID: <20170315132609.wjplicnvhizlwc3q@ics.muni.cz> On Wed, Mar 15, 2017 at 01:12:44PM +0000, GORECKI, DIETER wrote: > One other thing to consider is the storage of data inside the inode itself for very small files. GPFS has the ability to use the remaining [kilo]bytes of the inode to store the data of the file whenever the file is small enough to fit in. > > Anyone correct me if I am wrong, but with 4k inodes, you can store up to (4096-128 header) 3968 bytes of data. (without ILM) > > So regarding the size of the files you intend to store into your filesystem, it might be very interesting to take advantage of the performance of your SSD's to store small files. I agree it would, however, I have 30 % free capacity on SSDs only right now (with some snapshots and ~70M files). So I'm afraid I have to either change the rotational disks to hold metadata as well or do not create more files/snapshots or decrease the inode size. I cannot add more SSDs as I do not have free disk slots in HW. -- Luk?? Hejtm?nek From ulmer at ulmer.org Wed Mar 15 13:36:24 2017 From: ulmer at ulmer.org (Stephen Ulmer) Date: Wed, 15 Mar 2017 09:36:24 -0400 Subject: [gpfsug-discuss] default inode size In-Reply-To: <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> References: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> <0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> Message-ID: > On Mar 15, 2017, at 8:37 AM, Lukas Hejtmanek wrote: > > On Wed, Mar 15, 2017 at 08:22:22AM -0400, Stephen Ulmer wrote: >> You need 4K nodes to store encryption keys. You can also put other useful things in there, like extended attributes and (possibly) the entire file. >> >> Are you worried about wasting space? > > well, I have 1PB of capacity for data disks and 3TB of capacity for metadata > (SSD). > > With 512B inodes, this ration seemed to be quite ok, while with 4k inodes, > I run out of free space on SSD pretty fast. So I'm thinking about re-creating > file system with smaller inode size. > > I don't think I will ever need encryption keys. What?s the average file size? Every file that is less than *about* 3.5K will wind up in the (4k) inode itself. Also check the HW sector size of the SSDs (many are 4K now). If that?s the case, you?ll actually be much more efficient from an access point of view with 4K inodes. If you don? t have 1PB of data yet (or won?t in the next year or so), you can always expand the system pool with more SSDs in the future. It?s not scary to pick another size ? it works just fine, but you?re about to create a filesystem that will be a pain in the rear to change. It will take a long time to move/backfill even a few hundred TB of data if you change you mind later. You?ll definitely give up (possibly future) features by choosing not 4K, but you?ve got the flexibility to do so. :) Liberty, -- Stephen From xhejtman at ics.muni.cz Wed Mar 15 13:46:54 2017 From: xhejtman at ics.muni.cz (Lukas Hejtmanek) Date: Wed, 15 Mar 2017 14:46:54 +0100 Subject: [gpfsug-discuss] default inode size In-Reply-To: References: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> <0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> Message-ID: <20170315134654.335p4rsrnaxgrjrq@ics.muni.cz> On Wed, Mar 15, 2017 at 09:36:24AM -0400, Stephen Ulmer wrote: > What?s the average file size? Every file that is less than *about* 3.5K will wind up in the (4k) inode itself. average file size is about 4MB. > Also check the HW sector size of the SSDs (many are 4K now). If that?s the case, you?ll actually be much more efficient from an access point of view with 4K inodes. HW sector size reported by disk array is 512B. SSD itself uses 4k though, however, Linux thinks it is only 512B. > If you don? t have 1PB of data yet (or won?t in the next year or so), you can always expand the system pool with more SSDs in the future. I cannot. I don't have free HW slots. > It?s not scary to pick another size ? it works just fine, but you?re about to create a filesystem that will be a pain in the rear to change. It will take a long time to move/backfill even a few hundred TB of data if you change you mind later. You?ll definitely give up (possibly future) features by choosing not 4K, but you?ve got the flexibility to do so. :) But at least I will have space to store metadata :) -- Luk?? Hejtm?nek From aaron.s.knister at nasa.gov Wed Mar 15 13:59:52 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Wed, 15 Mar 2017 09:59:52 -0400 Subject: [gpfsug-discuss] default inode size In-Reply-To: <20170315134654.335p4rsrnaxgrjrq@ics.muni.cz> References: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> <0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> <20170315134654.335p4rsrnaxgrjrq@ics.muni.cz> Message-ID: <4bb5ec26-bf98-76b4-5399-0d315b502458@nasa.gov> I'm not sure because I've not tried it but if you chose an inode size less than 4K does the FS still show as being 4K aligned? If it doesn't this could prevent you from expanding the metadata capacity of the FS in the future because in a non 4k aligned fs GPFS won't let you add 4K sector size LUNS that contain metadata. -Aaron On 3/15/17 9:46 AM, Lukas Hejtmanek wrote: > On Wed, Mar 15, 2017 at 09:36:24AM -0400, Stephen Ulmer wrote: >> What?s the average file size? Every file that is less than *about* 3.5K will wind up in the (4k) inode itself. > > average file size is about 4MB. > >> Also check the HW sector size of the SSDs (many are 4K now). If that?s the case, you?ll actually be much more efficient from an access point of view with 4K inodes. > > HW sector size reported by disk array is 512B. SSD itself uses 4k though, > however, Linux thinks it is only 512B. > >> If you don? t have 1PB of data yet (or won?t in the next year or so), you can always expand the system pool with more SSDs in the future. > > I cannot. I don't have free HW slots. > >> It?s not scary to pick another size ? it works just fine, but you?re about to create a filesystem that will be a pain in the rear to change. It will take a long time to move/backfill even a few hundred TB of data if you change you mind later. You?ll definitely give up (possibly future) features by choosing not 4K, but you?ve got the flexibility to do so. :) > > But at least I will have space to store metadata :) > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From janfrode at tanso.net Wed Mar 15 14:22:36 2017 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Wed, 15 Mar 2017 15:22:36 +0100 Subject: [gpfsug-discuss] default inode size In-Reply-To: <20170315132609.wjplicnvhizlwc3q@ics.muni.cz> References: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> <0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> <20170315132609.wjplicnvhizlwc3q@ics.muni.cz> Message-ID: Another thing to consider is how many disk block pointers you have room for in the inode, and when you'll need to add additional indirect blocks. Ref: http://files.gpfsug.org/presentations/2016/south-bank/ D2_P2_A_spectrum_scale_metadata_dark_V2a.pdf If I understand that presentation correctly.. a block pointer is 12 bytes, so a 1 KB inode can hold approximately 70 block pointers -- ie. any file larger than 70 blocks will need additional indirect blocks to hold the addresses, requiring minimum one additional sub-block. With a 1 MB blocksize filesystem (data and metadata), files between 75 MB and 3 GB will need 1x inode and 1x indirect block = 1 KB + 32 KB metadata space. If your files are smaller than 75 MB, the address pointers fit in the 1K inode. If you have 4 KB inode, files smaller than 340 MB will fit all pointers in the inode. -jf On Wed, Mar 15, 2017 at 2:26 PM, Lukas Hejtmanek wrote: > On Wed, Mar 15, 2017 at 01:12:44PM +0000, GORECKI, DIETER wrote: > > One other thing to consider is the storage of data inside the inode > itself for very small files. GPFS has the ability to use the remaining > [kilo]bytes of the inode to store the data of the file whenever the file is > small enough to fit in. > > > > Anyone correct me if I am wrong, but with 4k inodes, you can store up to > (4096-128 header) 3968 bytes of data. (without ILM) > > > > So regarding the size of the files you intend to store into your > filesystem, it might be very interesting to take advantage of the > performance of your SSD's to store small files. > > I agree it would, however, I have 30 % free capacity on SSDs only right > now (with > some snapshots and ~70M files). So I'm afraid I have to either change the > rotational disks to hold metadata as well or do not create more > files/snapshots or decrease the inode size. I cannot add more SSDs as I do > not > have free disk slots in HW. > > -- > Luk?? Hejtm?nek > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Wed Mar 15 14:25:41 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Wed, 15 Mar 2017 14:25:41 +0000 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: <2d4fa452-3137-0a44-fd18-bed4b9923279@nasa.gov> References: <357B0FF5-4B7C-4941-8496-BA70D011A898@vanderbilt.edu> <2d4fa452-3137-0a44-fd18-bed4b9923279@nasa.gov> Message-ID: Hi All, Since I started this thread I guess I should chime in, too ? for us it was simply that we were testing a device that did not have hardware RAID controllers and we were wanting to implement something roughly equivalent to RAID 6 LUNs. Kevin > On Mar 14, 2017, at 5:16 PM, Aaron Knister wrote: > > For me it's the protection against bitrot and added protection against silent data corruption and in theory the write caching offered by adding log devices that could help with small random writes (although there are other problems with ZFS + synchronous workloads that stop this from actually materializing). > > -Aaron > > On 3/14/17 5:59 PM, Luke Raimbach wrote: >> Can I ask what the fascination with zvols is? Using a copy-on-write file >> system to underpin another block based file system seems >> counterintuitive. Perhaps I've missed something vital, in which case I'd >> be delighted to have my eyes opened! >> >> On Tue, 14 Mar 2017, 00:06 Aaron Knister, > > wrote: >> >> I was doing this in testing (with fantastic performance too) until I >> realized the issue with ZFS's behavior with direct io on zvols (e.g. not >> flushing a write to stable storage after acknowledging it to GPFS). >> After setting the sync=always parameter to not lose data in the event of >> a crash or power outage the write performance became unbearably slow >> (under 100MB/s of writes for an 8+2 RAIDZ2 if I recall correctly). I >> even tried adding a battery-backed PCIe write cache >> (http://www.netlist.com/products/vault-memory-storage/expressvault-pcIe-ev3/default.aspx) >> as a log device to the zpool but the performance was still really slow. >> I posted to the ZFS mailing list asking about how to optimize for a >> large block streaming workload but I didn't many bites >> (http://list.zfsonlinux.org/pipermail/zfs-discuss/2016-February/024851.html). >> >> I've got an RFE open with IBM >> (https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84994) >> to see if the behavior of GPFS could be changed such that it would issue >> explicit cache flushes that would allow it to work with ZFS (it might >> even be beneficial in FPO environments too). >> >> -Aaron >> >> On 3/13/17 4:44 PM, Buterbaugh, Kevin L wrote: >> > Hi All, >> > >> > Two things: >> > >> > 1) Paul?s suggestion to look at the nsddevices script was the answer I >> > needed to fix my mmcrfs issue. Thanks. >> > >> > 2) I am also interested in hearing if anyone is using ZFS to create the >> > equivalent of RAID 8+2P hardware RAID 6 LUNs and presenting that to GPFS >> > to use as disks? >> > >> > Thanks? >> > >> > Kevin >> > >> >> On Mar 11, 2017, at 4:37 AM, Daniel Kidger >> >> >> wrote: >> >> >> >> On the subject of using zvols for software Raid/ replication, can ask >> >> as a quick poll, how many people are doing this? >> >> >> >> And any feedback on stability, tuning and performance? >> >> >> >> Daniel >> >> IBM Technical Presales >> >> >> >> > On 10 Mar 2017, at 22:44, Aaron Knister >> >> >> wrote: >> >> > >> >> > Those look like zvol's. Out of curiosity have you set sync=always on the >> >> > filesystem root or zvols themselves? It's my understanding that without >> >> > that you risk data loss since GPFS won't ever cause a sync to be issued >> >> > to the zvol for zfs to flush acknowledged but uncommitted writes. >> >> > >> >> > -Aaron >> >> > >> >> >> On 3/10/17 4:36 PM, Sanchez, Paul wrote: >> >> >> See: >> >> >> https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm >> >> >> >> >> >> >> >> >> >> >> >> GPFS has a limited set of device search specs it uses to find connected >> >> >> NSDs. When using exotic devices, you need to whitelist the devices >> >> >> yourself using the user exit script at /var/mmfs/etc/nsddevices. >> >> >> >> >> >> >> >> >> >> >> >> *From:*gpfsug-discuss-bounces at spectrumscale.org >> >> >> > > >> >> >> [mailto:gpfsug-discuss-bounces at spectrumscale.org >> ] *On Behalf Of >> >> >> *Buterbaugh, Kevin L >> >> >> *Sent:* Friday, March 10, 2017 3:44 PM >> >> >> *To:* gpfsug main discussion list >> >> >> *Subject:* [gpfsug-discuss] mmcrfs issue >> >> >> >> >> >> >> >> >> >> >> >> Hi All, >> >> >> >> >> >> >> >> >> >> >> >> We are testing out some flash storage. I created a couple of NSDs >> >> >> successfully (?): >> >> >> >> >> >> >> >> >> >> >> >> root at nec:~/gpfs# mmlsnsd -F >> >> >> >> >> >> >> >> >> >> >> >> File system Disk name NSD servers >> >> >> >> >> >> --------------------------------------------------------------------------- >> >> >> >> >> >> (free disk) nsd1 nec >> >> >> >> >> >> (free disk) nsd2 nec >> >> >> >> >> >> >> >> >> >> >> >> root at nec:~/gpfs# >> >> >> >> >> >> >> >> >> >> >> >> So I tried to create a filesystem: >> >> >> >> >> >> >> >> >> >> >> >> root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j >> >> >> scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 >> >> >> >> >> >> GPFS: 6027-441 Unable to open disk 'nsd2' on node nec. >> >> >> >> >> >> GPFS: 6027-441 Unable to open disk 'nsd1' on node nec. >> >> >> >> >> >> No such device >> >> >> >> >> >> No such device >> >> >> >> >> >> GPFS: 6027-538 Error accessing disks. >> >> >> >> >> >> mmcrfs: 6027-1200 tscrfs failed. Cannot create gpfs0 >> >> >> >> >> >> mmcrfs: 6027-1639 Command failed. Examine previous error messages to >> >> >> determine cause. >> >> >> >> >> >> root at nec:~/gpfs# >> >> >> >> >> >> >> >> >> >> >> >> Does this output from readdescraw look normal? >> >> >> >> >> >> >> >> >> >> >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd16 >> >> >> >> >> >> NSD descriptor in sector 64 of /dev/zd16 >> >> >> >> >> >> NSDid: 0A0023D258C1C02C format version: 1403 Label: >> >> >> >> >> >> Paxos sector: -1931478434 number of sectors: 8192 isPdisk: 0 >> >> >> >> >> >> Comment: NSD descriptor for Thu Mar 9 14:50:52 2017 >> >> >> >> >> >> No Disk descriptor in sector 96 of /dev/zd16 >> >> >> >> >> >> No FS descriptor in sector 2048 of /dev/zd16 >> >> >> >> >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd32 >> >> >> >> >> >> NSD descriptor in sector 64 of /dev/zd32 >> >> >> >> >> >> NSDid: 0A0023D258C1C02B format version: 1403 Label: >> >> >> >> >> >> Paxos sector: -1880562609 number of sectors: 8192 isPdisk: 0 >> >> >> >> >> >> Comment: NSD descriptor for Thu Mar 9 14:50:51 2017 >> >> >> >> >> >> No Disk descriptor in sector 96 of /dev/zd32 >> >> >> >> >> >> No FS descriptor in sector 2048 of /dev/zd32 >> >> >> >> >> >> root at nec:~/gpfs# >> >> >> >> >> >> >> >> >> >> >> >> Thanks in advance, all? >> >> >> >> >> >> >> >> >> >> >> >> Kevin >> >> >> >> >> >> ? >> >> >> >> >> >> Kevin Buterbaugh - Senior System Administrator >> >> >> >> >> >> Vanderbilt University - Advanced Computing Center for Research and Education >> >> >> >> >> >> Kevin.Buterbaugh at vanderbilt.edu >> >> > > >> >> >> > > - (615)875-9633 >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> gpfsug-discuss mailing list >> >> >> gpfsug-discuss at spectrumscale.org >> >> >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> >> >> > >> >> > -- >> >> > Aaron Knister >> >> > NASA Center for Climate Simulation (Code 606.2) >> >> > Goddard Space Flight Center >> >> > (301) 286-2776 >> >> > _______________________________________________ >> >> > gpfsug-discuss mailing list >> >> > gpfsug-discuss at spectrumscale.org >> >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> > >> >> Unless stated otherwise above: >> >> IBM United Kingdom Limited - Registered in England and Wales with >> >> number 741598. >> >> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >> >> >> >> _______________________________________________ >> >> gpfsug-discuss mailing list >> >> gpfsug-discuss at spectrumscale.org >> >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > >> > >> > >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > >> >> -- >> Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) >> Goddard Space Flight Center >> (301) 286-2776 >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Robert.Oesterlin at nuance.com Wed Mar 15 14:27:20 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Wed, 15 Mar 2017 14:27:20 +0000 Subject: [gpfsug-discuss] Running multiple mmrestripefs in a single cluster? Message-ID: I?m looking at migrating multiple file systems from one set of NSDs to another. Assuming I put aside any potential IO bottlenecks, has anyone tried running multiple ?mmrestripefs? commands in a single cluster? Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Wed Mar 15 14:53:44 2017 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Wed, 15 Mar 2017 09:53:44 -0500 Subject: [gpfsug-discuss] default inode size - remarks In-Reply-To: <20170315132609.wjplicnvhizlwc3q@ics.muni.cz> References: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz><0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org><20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> <20170315132609.wjplicnvhizlwc3q@ics.muni.cz> Message-ID: You can set inode size to any one of 512, 1024, 2048 or 4096 when you mmcrfs. Try it on a test system. You want all the metadata of a file to fit into the inode. Also small files, and small directories, can be stored in their inodes. On a test file system you can examine how much of an inode is occupied with which data and metadata using the inode subcommand of the tsdbfs command. (Root only, and usual cautions -- use only to look, if you make a mistake patching - I told you not to do that!) inode number of any path is easily discovered with the standard ls -ild pathname command -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Wed Mar 15 20:03:35 2017 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Wed, 15 Mar 2017 21:03:35 +0100 Subject: [gpfsug-discuss] Running multiple mmrestripefs in a single cluster? In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Wed Mar 15 20:33:19 2017 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Wed, 15 Mar 2017 15:33:19 -0500 Subject: [gpfsug-discuss] Running multiple mmrestripefs in a singlecluster? In-Reply-To: References: Message-ID: You can control the load of mmrestripefs (and all maintenance commands) on your system using mmchqos ... From: "Olaf Weiser" To: gpfsug main discussion list Date: 03/15/2017 04:04 PM Subject: Re: [gpfsug-discuss] Running multiple mmrestripefs in a single cluster? Sent by: gpfsug-discuss-bounces at spectrumscale.org yes.. and please be carefully about the number of nodes , doing the job because of multiple PIT worker hammering against your data if you limit the restripe to 2 nodes (-N ......) of adjust the PITworker down to 8 or even 4 ... you can run multiple restripes.. without hurting the application workload to much ... but the final duration of your restripe then will be affected cheers From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 03/15/2017 03:27 PM Subject: [gpfsug-discuss] Running multiple mmrestripefs in a single cluster? Sent by: gpfsug-discuss-bounces at spectrumscale.org I?m looking at migrating multiple file systems from one set of NSDs to another. Assuming I put aside any potential IO bottlenecks, has anyone tried running multiple ?mmrestripefs? commands in a single cluster? Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From duersch at us.ibm.com Thu Mar 16 00:26:28 2017 From: duersch at us.ibm.com (Steve Duersch) Date: Wed, 15 Mar 2017 19:26:28 -0500 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 62, Issue 33 In-Reply-To: References: Message-ID: >>For me it's the protection against bitrot and added protection against silent data corruption GNR has this functionality. Right now that is available through ESS though. Not yet as software only. Steve Duersch Spectrum Scale 845-433-7902 IBM Poughkeepsie, New York gpfsug-discuss-bounces at spectrumscale.org wrote on 03/15/2017 10:25:59 AM: > > Message: 6 > Date: Wed, 15 Mar 2017 14:25:41 +0000 > From: "Buterbaugh, Kevin L" > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] mmcrfs issue > Message-ID: > Content-Type: text/plain; charset="utf-8" > > Hi All, > > Since I started this thread I guess I should chime in, too ? for us > it was simply that we were testing a device that did not have > hardware RAID controllers and we were wanting to implement something > roughly equivalent to RAID 6 LUNs. > > Kevin > > > On Mar 14, 2017, at 5:16 PM, Aaron Knister wrote: > > > > For me it's the protection against bitrot and added protection > against silent data corruption and in theory the write caching > offered by adding log devices that could help with small random > writes (although there are other problems with ZFS + synchronous > workloads that stop this from actually materializing). > > > > -Aaron > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.knister at gmail.com Thu Mar 16 00:52:58 2017 From: aaron.knister at gmail.com (Aaron Knister) Date: Wed, 15 Mar 2017 20:52:58 -0400 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 62, Issue 33 In-Reply-To: References: Message-ID: *drags out soapbox* Sorry in advance for the rant, this is one of my huge pet peeves :) There are some serious blockers for GNR adoption in my environment. It drives me up a wall that the only way to get end to end checksums in GPFS is with vendor hardware lock-in. I find it infuriating. Lustre can do this for free with ZFS. Historically it has also offered various other features too like eating your data so I guess it's a tradeoff ;-) I believe that either GNR should be available for any hardware that passes a validation suite or GPFS should support checksums on non-GNR NSDs either by leveraging T10-PI information or by checksumming blocks/subblocks and storing that somewhere. I opened an RFE for this and it was rejected and I was effectively told to go use GNR/ESS, but well... can't do GNR. But lets say I could run GNR on any hardware of my choosing after perhaps paying some modest licensing fee and passing a hardware validation test there's another blocker for me. Because GPFS doesn't support anything like an LNet router I'm fairly limited on the number of high speed verbs rdma fabrics I can connect GNR to. Furthermore even if I had enough PCIe slots the configuration may not be supported (e.g. a site with an OPA and an IB fabric that would like to use rdma verbs on both). There could even be a situation where a vendor of an HPC solution requires a specific OFED version for support purposes that's not the version running on the GNR nodes. If an NSD protocol router were available I could perhaps use ethernet as a common medium to work around this. I'd really like IBM to *do* something about this situation but I've not gotten any traction on it so far. -Aaron On Wed, Mar 15, 2017 at 8:26 PM, Steve Duersch wrote: > >>For me it's the protection against bitrot and added protection against > silent data corruption > GNR has this functionality. Right now that is available through ESS > though. Not yet as software only. > > Steve Duersch > Spectrum Scale > 845-433-7902 <(845)%20433-7902> > IBM Poughkeepsie, New York > > > > > gpfsug-discuss-bounces at spectrumscale.org wrote on 03/15/2017 10:25:59 AM: > > > > > > Message: 6 > > Date: Wed, 15 Mar 2017 14:25:41 +0000 > > From: "Buterbaugh, Kevin L" > > To: gpfsug main discussion list > > Subject: Re: [gpfsug-discuss] mmcrfs issue > > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > > > Hi All, > > > > Since I started this thread I guess I should chime in, too ? for us > > it was simply that we were testing a device that did not have > > hardware RAID controllers and we were wanting to implement something > > roughly equivalent to RAID 6 LUNs. > > > > Kevin > > > > > On Mar 14, 2017, at 5:16 PM, Aaron Knister > wrote: > > > > > > For me it's the protection against bitrot and added protection > > against silent data corruption and in theory the write caching > > offered by adding log devices that could help with small random > > writes (although there are other problems with ZFS + synchronous > > workloads that stop this from actually materializing). > > > > > > -Aaron > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan at buzzard.me.uk Thu Mar 16 11:19:30 2017 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Thu, 16 Mar 2017 11:19:30 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 62, Issue 33 In-Reply-To: References: Message-ID: <1489663170.17464.39.camel@buzzard.me.uk> On Wed, 2017-03-15 at 20:52 -0400, Aaron Knister wrote: > *drags out soapbox* > > > Sorry in advance for the rant, this is one of my huge pet peeves :) > > > > There are some serious blockers for GNR adoption in my environment. It > drives me up a wall that the only way to get end to end checksums in > GPFS is with vendor hardware lock-in. As I understand it that is a factually inaccurate statement. You are free to use a DIF/DIX solution which is part of the T10 SCSI specification and as such an open(ish) standard. That is if you pay a suitable amount you will get a copy of the standard and are free to implement it. It is arguably more open than ZFS that has licensing issues. Further as I understand it the DIF/DIX solution is actually better than the ZFS solution on a number of counts, that revolve around it being baked into the hardware. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From janfrode at tanso.net Thu Mar 16 13:50:48 2017 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Thu, 16 Mar 2017 14:50:48 +0100 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 62, Issue 33 In-Reply-To: References: Message-ID: Why would you need a NSD protocol router when the NSD servers can have a mix of infiniband and ethernet adapters? F.ex. 4x EDR + 2x 100GbE per io-node in an ESS should give you lots of bandwidth for your common ethernet medium. -jf On Thu, Mar 16, 2017 at 1:52 AM, Aaron Knister wrote: > *drags out soapbox* > > Sorry in advance for the rant, this is one of my huge pet peeves :) > > There are some serious blockers for GNR adoption in my environment. It > drives me up a wall that the only way to get end to end checksums in GPFS > is with vendor hardware lock-in. I find it infuriating. Lustre can do this > for free with ZFS. Historically it has also offered various other features > too like eating your data so I guess it's a tradeoff ;-) I believe that > either GNR should be available for any hardware that passes a validation > suite or GPFS should support checksums on non-GNR NSDs either by leveraging > T10-PI information or by checksumming blocks/subblocks and storing that > somewhere. I opened an RFE for this and it was rejected and I was > effectively told to go use GNR/ESS, but well... can't do GNR. > > But lets say I could run GNR on any hardware of my choosing after perhaps > paying some modest licensing fee and passing a hardware validation test > there's another blocker for me. Because GPFS doesn't support anything like > an LNet router I'm fairly limited on the number of high speed verbs rdma > fabrics I can connect GNR to. Furthermore even if I had enough PCIe slots > the configuration may not be supported (e.g. a site with an OPA and an IB > fabric that would like to use rdma verbs on both). There could even be a > situation where a vendor of an HPC solution requires a specific OFED > version for support purposes that's not the version running on the GNR > nodes. If an NSD protocol router were available I could perhaps use > ethernet as a common medium to work around this. > > I'd really like IBM to *do* something about this situation but I've not > gotten any traction on it so far. > > -Aaron > > > > On Wed, Mar 15, 2017 at 8:26 PM, Steve Duersch wrote: > >> >>For me it's the protection against bitrot and added protection against >> silent data corruption >> GNR has this functionality. Right now that is available through ESS >> though. Not yet as software only. >> >> Steve Duersch >> Spectrum Scale >> 845-433-7902 <(845)%20433-7902> >> IBM Poughkeepsie, New York >> >> >> >> >> gpfsug-discuss-bounces at spectrumscale.org wrote on 03/15/2017 10:25:59 AM: >> >> >> > >> > Message: 6 >> > Date: Wed, 15 Mar 2017 14:25:41 +0000 >> > From: "Buterbaugh, Kevin L" >> > To: gpfsug main discussion list >> > Subject: Re: [gpfsug-discuss] mmcrfs issue >> > Message-ID: >> > Content-Type: text/plain; charset="utf-8" >> > >> > Hi All, >> > >> > Since I started this thread I guess I should chime in, too ? for us >> > it was simply that we were testing a device that did not have >> > hardware RAID controllers and we were wanting to implement something >> > roughly equivalent to RAID 6 LUNs. >> > >> > Kevin >> > >> > > On Mar 14, 2017, at 5:16 PM, Aaron Knister >> wrote: >> > > >> > > For me it's the protection against bitrot and added protection >> > against silent data corruption and in theory the write caching >> > offered by adding log devices that could help with small random >> > writes (although there are other problems with ZFS + synchronous >> > workloads that stop this from actually materializing). >> > > >> > > -Aaron >> > > >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.knister at gmail.com Thu Mar 16 14:39:13 2017 From: aaron.knister at gmail.com (Aaron Knister) Date: Thu, 16 Mar 2017 10:39:13 -0400 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 62, Issue 33 In-Reply-To: <1489663170.17464.39.camel@buzzard.me.uk> References: <1489663170.17464.39.camel@buzzard.me.uk> Message-ID: In all seriousness, I'd love to be wrong about this. I'm not sure which part(s) of what I said was inaccurate-- the vendor lock in and/or that GNR is the only way to get end to end checksums. When I say end to end checksums I mean that from the moment an FS write is submitted to mmfsd a checksum is calculated that is passed through every layer (memory, network, sas, fibre channel etc.) down to individual disk drives (understanding that the RAID controller may need to derive the checksum based on whatever RAID algorithm it's using). It's my understanding that the only way to achieve this with GPFS today is with GNR which is only available via purchasing a GSS or ESS solution from IBM/Lenovo. One is of course free to by hardware from any vendor but you don't get GNR. I should really have said "if you want GNR you have to buy hardware from IBM or Lenovo" which to me is being locked in to a vendor as long as you'd like end to end checksums. If there's another way to get end-to-end checksums, could you help me understand how? Regarding DIF/DIX, it's my understanding that I can/could use T10-DIF today (with the correct hardware) without purchasing any standards which would checksum data from the HBA to the RAID controller (and in theory disks). However, in an environment with NSD servers the origin of a read/write could in theory be quite a bit further away from the HBA in terms of hops. T10-DIF would be completely separate from GPFS as I understand it. I'm not aware of any integration (T10-DIF + DIX). What I'm really looking for, I think, is T10-DIF + DIX where the application (GPFS) generates protection information that's then passed along to the layers below it. -Aaron On Thu, Mar 16, 2017 at 7:19 AM, Jonathan Buzzard wrote: > On Wed, 2017-03-15 at 20:52 -0400, Aaron Knister wrote: > > *drags out soapbox* > > > > > > Sorry in advance for the rant, this is one of my huge pet peeves :) > > > > > > > > There are some serious blockers for GNR adoption in my environment. It > > drives me up a wall that the only way to get end to end checksums in > > GPFS is with vendor hardware lock-in. > > As I understand it that is a factually inaccurate statement. > > You are free to use a DIF/DIX solution which is part of the T10 SCSI > specification and as such an open(ish) standard. That is if you pay a > suitable amount you will get a copy of the standard and are free to > implement it. It is arguably more open than ZFS that has licensing > issues. > > Further as I understand it the DIF/DIX solution is actually better than > the ZFS solution on a number of counts, that revolve around it being > baked into the hardware. > > JAB. > > -- > Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk > Fife, United Kingdom. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Thu Mar 16 14:43:47 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Thu, 16 Mar 2017 10:43:47 -0400 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 62, Issue 33 In-Reply-To: References: Message-ID: <26bc0188-b5de-a128-9122-e6ed145aba69@nasa.gov> Perhaps an environment where one has OPA and IB fabrics. Taken from here (https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html): RDMA is not supported on a node when both Mellanox HCAs and Intel Omni-Path HFIs are enabled for RDMA. The alternative being a situation where multiple IB fabrics exist that require different OFED versions from each other (and most likely from ESS) for support reasons (speaking from experience). That is to say if $VENDOR supports OFED version X on an IB fabric, and ESS/GSS ships with version Y and there's a problem on the IB fabric $VENDOR may point at the different OFED version on the ESS/GSS and say they don't support it and then one is in a bad spot. -Aaron On 3/16/17 9:50 AM, Jan-Frode Myklebust wrote: > Why would you need a NSD protocol router when the NSD servers can have a > mix of infiniband and ethernet adapters? F.ex. 4x EDR + 2x 100GbE per > io-node in an ESS should give you lots of bandwidth for your common > ethernet medium. > > > -jf > > On Thu, Mar 16, 2017 at 1:52 AM, Aaron Knister > wrote: > > *drags out soapbox* > > Sorry in advance for the rant, this is one of my huge pet peeves :) > > There are some serious blockers for GNR adoption in my environment. > It drives me up a wall that the only way to get end to end checksums > in GPFS is with vendor hardware lock-in. I find it infuriating. > Lustre can do this for free with ZFS. Historically it has also > offered various other features too like eating your data so I guess > it's a tradeoff ;-) I believe that either GNR should be available > for any hardware that passes a validation suite or GPFS should > support checksums on non-GNR NSDs either by leveraging T10-PI > information or by checksumming blocks/subblocks and storing that > somewhere. I opened an RFE for this and it was rejected and I was > effectively told to go use GNR/ESS, but well... can't do GNR. > > But lets say I could run GNR on any hardware of my choosing after > perhaps paying some modest licensing fee and passing a hardware > validation test there's another blocker for me. Because GPFS doesn't > support anything like an LNet router I'm fairly limited on the > number of high speed verbs rdma fabrics I can connect GNR to. > Furthermore even if I had enough PCIe slots the configuration may > not be supported (e.g. a site with an OPA and an IB fabric that > would like to use rdma verbs on both). There could even be a > situation where a vendor of an HPC solution requires a specific OFED > version for support purposes that's not the version running on the > GNR nodes. If an NSD protocol router were available I could perhaps > use ethernet as a common medium to work around this. > > I'd really like IBM to *do* something about this situation but I've > not gotten any traction on it so far. > > -Aaron > > > > On Wed, Mar 15, 2017 at 8:26 PM, Steve Duersch > wrote: > > >>For me it's the protection against bitrot and added protection > against silent data corruption > GNR has this functionality. Right now that is available through > ESS though. Not yet as software only. > > Steve Duersch > Spectrum Scale > 845-433-7902 > IBM Poughkeepsie, New York > > > > > gpfsug-discuss-bounces at spectrumscale.org > wrote on > 03/15/2017 10:25:59 AM: > > > > > > Message: 6 > > Date: Wed, 15 Mar 2017 14:25:41 +0000 > > From: "Buterbaugh, Kevin L" > > To: gpfsug main discussion list > > > Subject: Re: [gpfsug-discuss] mmcrfs issue > > Message-ID: > > > Content-Type: text/plain; charset="utf-8" > > > > Hi All, > > > > Since I started this thread I guess I should chime in, too ? for us > > it was simply that we were testing a device that did not have > > hardware RAID controllers and we were wanting to implement something > > roughly equivalent to RAID 6 LUNs. > > > > Kevin > > > > > On Mar 14, 2017, at 5:16 PM, Aaron Knister > wrote: > > > > > > For me it's the protection against bitrot and added protection > > against silent data corruption and in theory the write caching > > offered by adding log devices that could help with small random > > writes (although there are other problems with ZFS + synchronous > > workloads that stop this from actually materializing). > > > > > > -Aaron > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From jonathan at buzzard.me.uk Thu Mar 16 16:54:32 2017 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Thu, 16 Mar 2017 16:54:32 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 62, Issue 33 In-Reply-To: References: <1489663170.17464.39.camel@buzzard.me.uk> Message-ID: <1489683272.17464.67.camel@buzzard.me.uk> On Thu, 2017-03-16 at 10:39 -0400, Aaron Knister wrote: > In all seriousness, I'd love to be wrong about this. I'm not sure > which part(s) of what I said was inaccurate-- the vendor lock in > and/or that GNR is the only way to get end to end checksums. The end to end checksums, or at least the ability to protect against silent corruption by having the data checksummed at all stages. > > When I say end to end checksums I mean that from the moment an FS > write is submitted to mmfsd a checksum is calculated that is passed > through every layer (memory, network, sas, fibre channel etc.) down to > individual disk drives (understanding that the RAID controller may > need to derive the checksum based on whatever RAID algorithm it's > using). It's my understanding that the only way to achieve this with > GPFS today is with GNR which is only available via purchasing a GSS or > ESS solution from IBM/Lenovo. One is of course free to by hardware > from any vendor but you don't get GNR. I should really have said "if > you want GNR you have to buy hardware from IBM or Lenovo" which to me > is being locked in to a vendor as long as you'd like end to end > checksums. > > If there's another way to get end-to-end checksums, could you help me > understand how? > Bear in mind the purpose of the checksums is to ensure the data is not corrupted. Noting you don't get true end to end because the checksums are never exposed to the application even in ZFS, at least as I understand it; you just get a read failure. > Regarding DIF/DIX, it's my understanding that I can/could use T10-DIF > today (with the correct hardware) without purchasing any standards > which would checksum data from the HBA to the RAID controller (and in > theory disks). However, in an environment with NSD servers the origin > of a read/write could in theory be quite a bit further away from the > HBA in terms of hops. T10-DIF would be completely separate from GPFS > as I understand it. I'm not aware of any integration (T10-DIF + DIX). > What I'm really looking for, I think, is T10-DIF + DIX where the > application (GPFS) generates protection information that's then passed > along to the layers below it. > Well yes an end user does not need to purchase a copy of the standard. However some people take the view that as you need to spend money to get the standard it is not truly open, so I was just making that clear. Right, so bearing in mind that the purpose of the checksums is to ensure data is not corrupted if the NSD servers are using DIF/DIX, then the options for silent data corruption are very limited if you are using ECC memory and you are using standard TCP/IP to communicate between the nodes. That is the TCP/IP checksums will protect your data between the clients and the NSD nodes, and once at the NSD node the DIF/DIX will protect your data as it makes it way to the disk. In between all that the ECC memory in your machines will protect everything once in RAM, and the PCIe bus has a 32bit CRC protecting everything that moves over the PCIe bus. The only "hole" in this is as far as I am aware is that the CPU itself, because at least x86 ones (as far as I am aware) do not "checksum" themselves as they do calculations, but you have the same problem with ZFS for exactly the same reason. So the point is that what you should be interested in as an admin; ensuring your data is protected against silent corruption, is achievable without hardware vendor lock in using open standards. The advantage of DIF/DIX over ZFS is as I understand it unless you do an immediate read on written blocks with ZFS and take a large performance penalty on writes you have no idea if your data has been corrupted until you try and read it back which could be months if not years down the line. Where DIF/DIX should try and fix it immediately by a retry and if that does not work pass the failure back up the line immediately. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From jonathan at buzzard.me.uk Thu Mar 16 17:18:22 2017 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Thu, 16 Mar 2017 17:18:22 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 62, Issue 33 In-Reply-To: <26bc0188-b5de-a128-9122-e6ed145aba69@nasa.gov> References: <26bc0188-b5de-a128-9122-e6ed145aba69@nasa.gov> Message-ID: <1489684702.17464.84.camel@buzzard.me.uk> On Thu, 2017-03-16 at 10:43 -0400, Aaron Knister wrote: > Perhaps an environment where one has OPA and IB fabrics. Taken from here > (https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html): > > RDMA is not supported on a node when both Mellanox HCAs and Intel > Omni-Path HFIs are enabled for RDMA. > > The alternative being a situation where multiple IB fabrics exist that > require different OFED versions from each other (and most likely from > ESS) for support reasons (speaking from experience). That is to say if > $VENDOR supports OFED version X on an IB fabric, and ESS/GSS ships with > version Y and there's a problem on the IB fabric $VENDOR may point at > the different OFED version on the ESS/GSS and say they don't support it > and then one is in a bad spot. > Or just use Ethernet for the GPFS traffic everywhere. It's 2017, 10GbE to compute nodes is cheap enough to be the norm, and you can use 40GbE and 100GbE everywhere else as required, and note unless you are a pure SSD system (which must be vanishingly rare at the moment) the latency is all in the disks anyway. I can't imagine why you would use IB and/or Omni-Path on anything other than compute clusters, so using Ethernet for your storage has advantages too especially if you are using IB. One of the features of Omni-Path over IB is that it will prioritize MPI traffic over storage, but given current core counts in CPU's separating storage out onto Ethernet is not a bad thing anyway as it keeps the MPI bandwidth per core up. My feeling is that in a compute cluster 10Gb for storage is more than enough for a node because much more and it would only take a handful of nodes to denial of service your storage, which is not good. A 500 node compute cluster with 10GbE for storage can in theory try and hit your storage for 600GB/s which very very few storage systems will be able to keep up with. On a storage GPFS cluster then you can up to 40/100GbE and put in multiple links for redundancy which is not possible with IB/Omni-path as far as I am aware which makes it a better solution. Even in a compute cluster with Ethernet I can stick in multiple connections for my NSD nodes for redundancy which is an improvement over the IB/Omni-path options. Especially given in my experience IB links go wonky orders of magnitude more often than Ethernet ones do. Loose a link to a compute node I loose one job on the cluster. Loose a link to the NSD server and I face loosing all the jobs on the cluster... JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From Robert.Oesterlin at nuance.com Thu Mar 16 18:43:59 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Thu, 16 Mar 2017 18:43:59 +0000 Subject: [gpfsug-discuss] Registration Reminder: GPFSUG-USA Meeting at NERSC April 4-5th Message-ID: Here?s a reminder to register for the upcoming user group meeting! Registration deadline extended to March 29th! Registration is filling up ? don?t wait. We Still have slots (20 mins each) for customer and partner talks ? please contact me if you want to do a short presentation. Register here: https://www.nersc.gov/research-and-development/data-analytics/spectrum-user-group-meeting/ Bob Oesterlin Sr Principal Storage Engineer, Nuance -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Fri Mar 17 18:50:11 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Fri, 17 Mar 2017 18:50:11 +0000 Subject: [gpfsug-discuss] mmnetverify Message-ID: Hi all, Just wondering if anyone has used the mmnetverify tool at all? Having made some changes to our internal L3 routing this week, I was interested to see what it claimed. As a side-note, it picked up some DNS resolution issues, though I'm not clear on some of those why it was claiming this as doing a "dig" on the node, it resolved fine (but adding the NSD servers to the hosts files cleared the error). Its actually the bandwidth tests that I'm interested in hearing other people's experience with as the numbers that some out from it are very different (lower) than if we use iperf to test performance between two nodes. Anyone any thoughts at all on this? Thanks Simon From Paul.Sanchez at deshaw.com Fri Mar 17 19:43:20 2017 From: Paul.Sanchez at deshaw.com (Sanchez, Paul) Date: Fri, 17 Mar 2017 19:43:20 +0000 Subject: [gpfsug-discuss] mmnetverify In-Reply-To: References: Message-ID: <18729d8a86fc430fac09c1c3a22159d2@mbxtoa1.winmail.deshaw.com> Sven will tell you: "RPC isn't streaming" and that may account for the discrepancy. If the tests are doing any "fan-in" where multiple nodes are sending to single node, then it's also possible that you are exhausting switch buffer memory in a way that a 1:1 iperf wouldn't. For our internal benchmarking we've used /usr/lpp/mmfs/samples/net/nsdperf to more closely estimate the real performance. I haven't played with mmnetverify yet though. -Paul -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (Research Computing - IT Services) Sent: Friday, March 17, 2017 2:50 PM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] mmnetverify Hi all, Just wondering if anyone has used the mmnetverify tool at all? Having made some changes to our internal L3 routing this week, I was interested to see what it claimed. As a side-note, it picked up some DNS resolution issues, though I'm not clear on some of those why it was claiming this as doing a "dig" on the node, it resolved fine (but adding the NSD servers to the hosts files cleared the error). Its actually the bandwidth tests that I'm interested in hearing other people's experience with as the numbers that some out from it are very different (lower) than if we use iperf to test performance between two nodes. Anyone any thoughts at all on this? Thanks Simon _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From S.J.Thompson at bham.ac.uk Fri Mar 17 20:13:22 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Fri, 17 Mar 2017 20:13:22 +0000 Subject: [gpfsug-discuss] mmnetverify In-Reply-To: <18729d8a86fc430fac09c1c3a22159d2@mbxtoa1.winmail.deshaw.com> References: , <18729d8a86fc430fac09c1c3a22159d2@mbxtoa1.winmail.deshaw.com> Message-ID: It looks to run sequential tests to each node one at a time and isn't using NSD protocol but echo server. We found some weird looking numbers that i don't quite understand and not in the places we might expect. For example between hosts on the same switch, traffic flowing to another switch and traffic flowing to nodes in another data centre where it's several switch hops. Some nodes over there were significantly faster than switch local nodes. I think it was only added in 4.2.2 and is listed as "not yet a replacement for nsdperf". I get that is different as it's using NSD protocol, but was struggling a bit with what mmnetverify might be doing. Simon From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Sanchez, Paul [Paul.Sanchez at deshaw.com] Sent: 17 March 2017 19:43 To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] mmnetverify Sven will tell you: "RPC isn't streaming" and that may account for the discrepancy. If the tests are doing any "fan-in" where multiple nodes are sending to single node, then it's also possible that you are exhausting switch buffer memory in a way that a 1:1 iperf wouldn't. For our internal benchmarking we've used /usr/lpp/mmfs/samples/net/nsdperf to more closely estimate the real performance. I haven't played with mmnetverify yet though. -Paul -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (Research Computing - IT Services) Sent: Friday, March 17, 2017 2:50 PM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] mmnetverify Hi all, Just wondering if anyone has used the mmnetverify tool at all? Having made some changes to our internal L3 routing this week, I was interested to see what it claimed. As a side-note, it picked up some DNS resolution issues, though I'm not clear on some of those why it was claiming this as doing a "dig" on the node, it resolved fine (but adding the NSD servers to the hosts files cleared the error). Its actually the bandwidth tests that I'm interested in hearing other people's experience with as the numbers that some out from it are very different (lower) than if we use iperf to test performance between two nodes. Anyone any thoughts at all on this? Thanks Simon _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Robert.Oesterlin at nuance.com Mon Mar 20 13:07:43 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 20 Mar 2017 13:07:43 +0000 Subject: [gpfsug-discuss] SSUG-USA: Registration Reminder and Agenda Update Message-ID: Only two weeks until the user group meeting! Registration is filling up fast ? Deadline is March 29th. We Still have 2 slots (20 mins each) for customer and partner talks ? please contact me if you want to do a short presentation. Register here: https://www.nersc.gov/research-and-development/data-analytics/spectrum-user-group-meeting/ Duration Start End Title Speaker Tues April 4th 60 9:00 10:00 Registration and Coffee n/a 30 10:00 10:30 Welcome Bob Oesterlin, Kristy Kallback, Doris Conti 60 10:30 11:30 Spectrum Scale & ESS Update Ulf Troppens / Scott Fadden 60 11:30 12:30 Problem Avoidance - Best Practices in the Field Brian Yaeger 60 12:30 13:30 Lunch and Networking n/a 120 13:30 15:30 Problem Determination - Deep Dive Yuri Volobuev 30 15:30 16:00 Coffee and Networking n/a 60 16:00 17:00 Problem Determination - What's New? Matthias Dietz 60 17:00 18:00 Bring your favorite problem or performance issue. We will fix it now! All Wed April 5th 30 8:30 9:00 Coffee and Networking n/a 60 9:00 10:00 1) AFM - Introduction and Use Cases (Meet the Devs) Scott Fadden 2) Life Sciences and Next Generation Sequencing with Spectrum Scale (Solutions) Madhav Ponamgi 3) Cloud and Virtualization Discussion (BOF) John Lewars 60 10:00 11:00 1) Transparent Cloud Tiering - Introduction and Use Cases (Meet the Devs) Rob Basham 2) Spectrum Protect on Spectrum Scale (Solutions) Jason Basler 3) Open User BOF TBD 60 11:00 12:00 1) Spectrum Scale Security (Meet the Devs) Felipe Knop 2) Hadoop Integration and Support for HortonWorks HDP (Solutions) Ted Hoover 3) Open User BOF TBD 60 12:00 13:00 Lunch and Networking n/a 20 13:00 13:20 System Analytics/IO Profiling TBD Customer 20 13:20 13:40 Placeholder - Customer or Partner talk TBD Customer 20 13:40 14:00 Placeholder - Customer or Partner talk TBD Customer 20 14:00 14:20 REST API Ulf Troppens 30 14:20 14:50 Coffee and Networking n/a 30 14:50 15:20 Application Integration with Containers Dean Hildebrand 30 15:20 15:50 Challenges in Tracing Vasily Tarasov 10 15:50 16:00 Wrap-up Bob Oesterlin, Kristy Kallback, Doris Conti Bob Oesterlin Sr Principal Storage Engineer, Nuance -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.wonderley at vt.edu Mon Mar 20 14:02:56 2017 From: eric.wonderley at vt.edu (J. Eric Wonderley) Date: Mon, 20 Mar 2017 10:02:56 -0400 Subject: [gpfsug-discuss] snapshots & tiering in a busy filesystem Message-ID: I found this link and it didn't give me much hope for doing snapshots & backup in a home(busy) filesystem: http://www.spectrumscale.org/pipermail/gpfsug-discuss/2013- February/000200.html I realize this is dated and I wondered if qos etc have made is a tolerable thing to do now. Gpfs I think was barely above v3.5 in mid 2013. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Mon Mar 20 14:49:07 2017 From: bbanister at jumptrading.com (Bryan Banister) Date: Mon, 20 Mar 2017 14:49:07 +0000 Subject: [gpfsug-discuss] snapshots & tiering in a busy filesystem In-Reply-To: References: Message-ID: We run snapshots every 4 hours on a busy home file system without issues for many years now, -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of J. Eric Wonderley Sent: Monday, March 20, 2017 9:03 AM To: gpfsug main discussion list Subject: [gpfsug-discuss] snapshots & tiering in a busy filesystem I found this link and it didn't give me much hope for doing snapshots & backup in a home(busy) filesystem: http://www.spectrumscale.org/pipermail/gpfsug-discuss/2013-February/000200.html I realize this is dated and I wondered if qos etc have made is a tolerable thing to do now. Gpfs I think was barely above v3.5 in mid 2013. ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Mon Mar 20 14:53:51 2017 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 20 Mar 2017 09:53:51 -0500 Subject: [gpfsug-discuss] snapshots & tiering in a busy filesystem In-Reply-To: References: Message-ID: Yes. QOS (see mmchqos) is now the way (4.2.x) to control maintenance commands like snapshots, backup, restripe. From: "J. Eric Wonderley" To: gpfsug main discussion list Date: 03/20/2017 10:03 AM Subject: [gpfsug-discuss] snapshots & tiering in a busy filesystem Sent by: gpfsug-discuss-bounces at spectrumscale.org I found this link and it didn't give me much hope for doing snapshots & backup in a home(busy) filesystem: http://www.spectrumscale.org/pipermail/gpfsug-discuss/2013-February/000200.html I realize this is dated and I wondered if qos etc have made is a tolerable thing to do now. Gpfs I think was barely above v3.5 in mid 2013. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.g.richmond at leeds.ac.uk Tue Mar 21 10:56:41 2017 From: a.g.richmond at leeds.ac.uk (Aidan Richmond) Date: Tue, 21 Mar 2017 10:56:41 +0000 Subject: [gpfsug-discuss] Configuration spectrum scale to use SFU schema Message-ID: <8d1622e5-99dc-dad4-998b-b0d5a88c8f70@leeds.ac.uk> Hello Is it possible to configure authentication services to bind to AD using the older SFU schema for ID mappings? We have standard samba and nfs servers that use this ID mapping scheme with "idmap config DS:schema_mode = sfu" in the samba configuration files, but if its possible using the mmuserauth command it doesn't seem to be documented. Thanks. -- Aidan Richmond Apple/Unix Support Officer, IT Garstang 10.137 Faculty of Biological Sciences University of Leeds Clarendon Way LS2 9JT Tel:0113 3434252 a.g.richmond at leeds.ac.uk From chair at spectrumscale.org Tue Mar 21 14:00:50 2017 From: chair at spectrumscale.org (Spectrum Scale UG Chair (Simon Thompson)) Date: Tue, 21 Mar 2017 14:00:50 +0000 Subject: [gpfsug-discuss] Sydney April Spectrum Scale UG Meeting In-Reply-To: References: Message-ID: Hi All, If you are located in the Australia region, you may be interested in the Spectrum Scale User Group meeting being hosted in Sydney, Australia: >We have a good part of the day for a Meet the Developers team with Kedar >Karmarkar form India and his team. >Plus Mellanox, Pawsey Supercomputing Centre, and a lessons learn from DDN >on IB networking at the Australian Bureau of Meteorology (subject to >disclosure agreement). Agenda and registration are at: >https://www.eventbrite.com/e/spectrum-scale-user-group-australia-gpfsugaus >-april-2017-tickets-29136246297 There's also a slack channel for discussion at: >https://ibmau-spectrumscale.slack.com/messages/C3C10AC9H/details/ This is being organised by Chris Schlipalius from Pawsey Supercomputing Centre, please contact Chris directly for more details (see link on Eventbrite page). Simon UK Group Chair From christof.schmitt at us.ibm.com Tue Mar 21 17:24:03 2017 From: christof.schmitt at us.ibm.com (Christof Schmitt) Date: Tue, 21 Mar 2017 10:24:03 -0700 Subject: [gpfsug-discuss] Configuration spectrum scale to use SFU schema In-Reply-To: <8d1622e5-99dc-dad4-998b-b0d5a88c8f70@leeds.ac.uk> References: <8d1622e5-99dc-dad4-998b-b0d5a88c8f70@leeds.ac.uk> Message-ID: gpfsug-discuss-bounces at spectrumscale.org wrote on 03/21/2017 03:56:41 AM: > Is it possible to configure authentication services to bind to AD using > the older SFU schema for ID mappings? We have standard samba and nfs > servers that use this ID mapping scheme with "idmap config > DS:schema_mode = sfu" in the samba configuration files, but if its > possible using the mmuserauth command it doesn't seem to be documented. This falls into the category that we use Samba to provide these services, but cannot test and support every possible configuration. The feature to read the old MS style id attributes has not been tested with Spectrum Scale and is not supported. That said, it can still be set in the internal config: /usr/lpp/mmfs/bin/net conf setparm global 'idmap config DS:schema_mode' sfu Activating this change probably would require a restart of the process using this config: /usr/lpp/mmfs/bin/mmdsh -N CesNodes systemctl restart gpfs-winbind Opening a PMR on this will likely result in the answer that this is unsupported. The best way to request official support would be opening a RFE, let others vote and then product management can decide on the priority of the request. Regards, Christof Schmitt || IBM || Spectrum Scale Development || Tucson, AZ christof.schmitt at us.ibm.com || +1-520-799-2469 (T/L: 321-2469) From TROPPENS at de.ibm.com Tue Mar 21 17:51:09 2017 From: TROPPENS at de.ibm.com (Ulf Troppens) Date: Tue, 21 Mar 2017 18:51:09 +0100 Subject: [gpfsug-discuss] Charts of German User Meeting now online Message-ID: Thanks a lot to all speaker. Event report will follow later. http://www.spectrumscale.org/presentations/ -- IBM Spectrum Scale Development - Client Engagements & Solutions Delivery Consulting IT Specialist Author "Storage Networks Explained" IBM Deutschland Research & Development GmbH Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.g.richmond at leeds.ac.uk Wed Mar 22 08:01:33 2017 From: a.g.richmond at leeds.ac.uk (Aidan Richmond) Date: Wed, 22 Mar 2017 08:01:33 +0000 Subject: [gpfsug-discuss] Configuration spectrum scale to use SFU schema In-Reply-To: References: <8d1622e5-99dc-dad4-998b-b0d5a88c8f70@leeds.ac.uk> Message-ID: <75a2acf9-e598-89c2-9546-6cedd80186c1@leeds.ac.uk> On 21/03/17 17:24, Christof Schmitt wrote: > gpfsug-discuss-bounces at spectrumscale.org wrote on 03/21/2017 03:56:41 AM: > >> Is it possible to configure authentication services to bind to AD using >> the older SFU schema for ID mappings? We have standard samba and nfs >> servers that use this ID mapping scheme with "idmap config >> DS:schema_mode = sfu" in the samba configuration files, but if its >> possible using the mmuserauth command it doesn't seem to be documented. > > This falls into the category that we use Samba to provide these services, > but cannot test and support > every possible configuration. The feature to read the old MS style id > attributes has not been tested > with Spectrum Scale and is not supported. That said, it can still be set > in the internal config: > > /usr/lpp/mmfs/bin/net conf setparm global 'idmap config DS:schema_mode' > sfu > > Activating this change probably would require a restart of the process > using this config: > > /usr/lpp/mmfs/bin/mmdsh -N CesNodes systemctl restart gpfs-winbind > > Opening a PMR on this will likely result in the answer that this is > unsupported. The best way > to request official support would be opening a RFE, let others vote and > then product > management can decide on the priority of the request. > > Regards, > > Christof Schmitt || IBM || Spectrum Scale Development || Tucson, AZ > christof.schmitt at us.ibm.com || +1-520-799-2469 (T/L: 321-2469) > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > Thanks, would I need to re-run mmuserauth as per https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.1/com.ibm.spectrum.scale.v4r21.doc/bl1adm_adwithrfc2307.htm or would the schema change be needed after that when configuring the authentication services? -- Aidan Richmond Apple/Unix Support Officer, IT Garstang 10.137 Faculty of Biological Sciences University of Leeds Clarendon Way LS2 9JT Tel:0113 3434252 a.g.richmond at leeds.ac.uk From r.sobey at imperial.ac.uk Wed Mar 22 10:47:37 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Wed, 22 Mar 2017 10:47:37 +0000 Subject: [gpfsug-discuss] snapshots & tiering in a busy filesystem In-Reply-To: References: Message-ID: We?re also snapshotting 4 times a day. Filesystem isn?t tremendously busy at all but we?re creating snaps for each fileset. [root at cesnode tmp]# mmlssnapshot gpfs | wc -l 6916 From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of J. Eric Wonderley Sent: 20 March 2017 14:03 To: gpfsug main discussion list Subject: [gpfsug-discuss] snapshots & tiering in a busy filesystem I found this link and it didn't give me much hope for doing snapshots & backup in a home(busy) filesystem: http://www.spectrumscale.org/pipermail/gpfsug-discuss/2013-February/000200.html I realize this is dated and I wondered if qos etc have made is a tolerable thing to do now. Gpfs I think was barely above v3.5 in mid 2013. -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.wonderley at vt.edu Wed Mar 22 13:41:26 2017 From: eric.wonderley at vt.edu (J. Eric Wonderley) Date: Wed, 22 Mar 2017 09:41:26 -0400 Subject: [gpfsug-discuss] snapshots & tiering in a busy filesystem In-Reply-To: References: Message-ID: The filesystem I'm working with has about 100M files and 80Tb of data. What kind of metadata latency do you observe? I did a mmdiag --iohist and filtered out all of the md devices and averaged over reads and writes. I'm seeing ~.28ms on a one off dump. The pure array which we have is 10G iscsi connected and is reporting average .25ms. On Wed, Mar 22, 2017 at 6:47 AM, Sobey, Richard A wrote: > We?re also snapshotting 4 times a day. Filesystem isn?t tremendously busy > at all but we?re creating snaps for each fileset. > > > > [root at cesnode tmp]# mmlssnapshot gpfs | wc -l > > 6916 > > > > *From:* gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss- > bounces at spectrumscale.org] *On Behalf Of *J. Eric Wonderley > *Sent:* 20 March 2017 14:03 > *To:* gpfsug main discussion list > *Subject:* [gpfsug-discuss] snapshots & tiering in a busy filesystem > > > > I found this link and it didn't give me much hope for doing snapshots & > backup in a home(busy) filesystem: > > http://www.spectrumscale.org/pipermail/gpfsug-discuss/2013- > February/000200.html > > I realize this is dated and I wondered if qos etc have made is a tolerable > thing to do now. Gpfs I think was barely above v3.5 in mid 2013. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mweil at wustl.edu Wed Mar 22 16:42:35 2017 From: mweil at wustl.edu (Matt Weil) Date: Wed, 22 Mar 2017 11:42:35 -0500 Subject: [gpfsug-discuss] CES node slow to respond Message-ID: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu> All, We had an indecent yesterday where one of our CES nodes slowed to a crawl. GPFS waiters showed pre fetch threads going after inodes. iohist also showed lots of inode fetching. Then we noticed that the CES host had 5.4 million files open. The change I made was to set maxStatCache=DEFAULT because it is linux. And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. Is there something else we should change as well. Thanks Matt ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. From bbanister at jumptrading.com Wed Mar 22 17:01:26 2017 From: bbanister at jumptrading.com (Bryan Banister) Date: Wed, 22 Mar 2017 17:01:26 +0000 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu> References: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu> Message-ID: <47e18bb0062544f4b510ce0e87049fd3@jumptrading.com> We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. -Bryan -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Wednesday, March 22, 2017 11:43 AM To: gpfsug main discussion list Subject: [gpfsug-discuss] CES node slow to respond All, We had an indecent yesterday where one of our CES nodes slowed to a crawl. GPFS waiters showed pre fetch threads going after inodes. iohist also showed lots of inode fetching. Then we noticed that the CES host had 5.4 million files open. The change I made was to set maxStatCache=DEFAULT because it is linux. And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. Is there something else we should change as well. Thanks Matt ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. From r.sobey at imperial.ac.uk Wed Mar 22 17:44:56 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Wed, 22 Mar 2017 17:44:56 +0000 Subject: [gpfsug-discuss] snapshots & tiering in a busy filesystem In-Reply-To: References: Message-ID: Embarrassingly I don?t know how to do that scientifically. I can tell you latency of the storage system which at a very simplistic snapshot is averaging around 2ms. We are not using SSD for metadata but 10K SAS. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of J. Eric Wonderley Sent: 22 March 2017 13:41 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] snapshots & tiering in a busy filesystem The filesystem I'm working with has about 100M files and 80Tb of data. What kind of metadata latency do you observe? I did a mmdiag --iohist and filtered out all of the md devices and averaged over reads and writes. I'm seeing ~.28ms on a one off dump. The pure array which we have is 10G iscsi connected and is reporting average .25ms. On Wed, Mar 22, 2017 at 6:47 AM, Sobey, Richard A > wrote: We?re also snapshotting 4 times a day. Filesystem isn?t tremendously busy at all but we?re creating snaps for each fileset. [root at cesnode tmp]# mmlssnapshot gpfs | wc -l 6916 From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of J. Eric Wonderley Sent: 20 March 2017 14:03 To: gpfsug main discussion list > Subject: [gpfsug-discuss] snapshots & tiering in a busy filesystem I found this link and it didn't give me much hope for doing snapshots & backup in a home(busy) filesystem: http://www.spectrumscale.org/pipermail/gpfsug-discuss/2013-February/000200.html I realize this is dated and I wondered if qos etc have made is a tolerable thing to do now. Gpfs I think was barely above v3.5 in mid 2013. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From valdis.kletnieks at vt.edu Wed Mar 22 21:21:42 2017 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Wed, 22 Mar 2017 17:21:42 -0400 Subject: [gpfsug-discuss] LTFS/EE release question.. Message-ID: <37386.1490217702@turing-police.cc.vt.edu> We're currently on ltfs/ee 1.2.1.1. IBM has recommended we upgrade to gpfs 4.2.2.3 to fix a performance issue. I can't seem to find a support matrix that says what ltfs release we need for 4.2.2. Anybody have info on that? -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 484 bytes Desc: not available URL: From christof.schmitt at us.ibm.com Wed Mar 22 22:41:04 2017 From: christof.schmitt at us.ibm.com (Christof Schmitt) Date: Wed, 22 Mar 2017 15:41:04 -0700 Subject: [gpfsug-discuss] Configuration spectrum scale to use SFU schema In-Reply-To: <75a2acf9-e598-89c2-9546-6cedd80186c1@leeds.ac.uk> References: <8d1622e5-99dc-dad4-998b-b0d5a88c8f70@leeds.ac.uk> <75a2acf9-e598-89c2-9546-6cedd80186c1@leeds.ac.uk> Message-ID: > Thanks, would I need to re-run mmuserauth as per > https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.1/ > com.ibm.spectrum.scale.v4r21.doc/bl1adm_adwithrfc2307.htm > or would the schema change be needed after that when configuring the > authentication services? You would still need the domain join and the basic configuration done by mmuserauth. The easiest way would be running mmuserauth with --unix-domains as usually. The only change on top is then the switch to the old attribute format, which requires a winbindd restart to pick-up the change: mmuserauth service create ??data?access?method file --type ad ... /usr/lpp/mmfs/bin/net conf setparm global 'idmap config DS:schema_mode' sfu /usr/lpp/mmfs/bin/mmdsh -N CesNodes systemctl restart gpfs-winbind This also illustrates the support statement: The config through mmuserauth is supported, changing the Samba config under the hood is outside the normal support. Regards, Christof Schmitt || IBM || Spectrum Scale Development || Tucson, AZ christof.schmitt at us.ibm.com || +1-520-799-2469 (T/L: 321-2469) From kevindjo at us.ibm.com Thu Mar 23 13:35:24 2017 From: kevindjo at us.ibm.com (Kevin D Johnson) Date: Thu, 23 Mar 2017 13:35:24 +0000 Subject: [gpfsug-discuss] LTFS/EE release question.. In-Reply-To: <37386.1490217702@turing-police.cc.vt.edu> References: <37386.1490217702@turing-police.cc.vt.edu> Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: att66cs2.dat Type: application/octet-stream Size: 495 bytes Desc: not available URL: From olaf.weiser at de.ibm.com Thu Mar 23 14:24:38 2017 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Thu, 23 Mar 2017 15:24:38 +0100 Subject: [gpfsug-discuss] CES doesn't assign addresses to nodes In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From jonathon.anderson at colorado.edu Thu Mar 23 15:02:10 2017 From: jonathon.anderson at colorado.edu (Jonathon A Anderson) Date: Thu, 23 Mar 2017 15:02:10 +0000 Subject: [gpfsug-discuss] CES doesn't assign addresses to nodes In-Reply-To: References: Message-ID: Thanks! I?m looking forward to upgrading our CES nodes and resuming work on the project. ~jonathon On 3/23/17, 8:24 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Olaf Weiser" wrote: the issue is fixed, an APAR will be released soon - IV93100 From: Olaf Weiser/Germany/IBM at IBMDE To: "gpfsug main discussion list" Cc: "gpfsug main discussion list" Date: 01/31/2017 11:47 PM Subject: Re: [gpfsug-discuss] CES doesn't assign addresses to nodes Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________________ Yeah... depending on the #nodes you 're affected or not. ..... So if your remote ces cluster is small enough in terms of the #nodes ... you'll neuer hit into this issue Gesendet von IBM Verse Simon Thompson (Research Computing - IT Services) --- Re: [gpfsug-discuss] CES doesn't assign addresses to nodes --- Von:"Simon Thompson (Research Computing - IT Services)" An:"gpfsug main discussion list" Datum:Di. 31.01.2017 21:07Betreff:Re: [gpfsug-discuss] CES doesn't assign addresses to nodes ________________________________________ We use multicluster for our environment, storage systems in a separate cluster to hpc nodes on a separate cluster from protocol nodes. According to the docs, this isn't supported, but we haven't seen any issues. Note unsupported as opposed to broken. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Jonathon A Anderson [jonathon.anderson at colorado.edu] Sent: 31 January 2017 17:47 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] CES doesn't assign addresses to nodes Yeah, I searched around for places where ` tsctl shownodes up` appears in the GPFS code I have access to (i.e., the ksh and python stuff); but it?s only in CES. I suspect there just haven?t been that many people exporting CES out of an HPC cluster environment. ~jonathon From: on behalf of Olaf Weiser Reply-To: gpfsug main discussion list Date: Tuesday, January 31, 2017 at 10:45 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] CES doesn't assign addresses to nodes I ll open a pmr here for my env ... the issue may hurt you in a ces env. only... but needs to be fixed in core gpfs.base i thi k Gesendet von IBM Verse Jonathon A Anderson --- Re: [gpfsug-discuss] CES doesn't assign addresses to nodes --- Von: "Jonathon A Anderson" An: "gpfsug main discussion list" Datum: Di. 31.01.2017 17:32 Betreff: Re: [gpfsug-discuss] CES doesn't assign addresses to nodes ________________________________ No, I?m having trouble getting this through DDN support because, while we have a GPFS server license and GRIDScaler support, apparently we don?t have ?protocol node? support, so they?ve pushed back on supporting this as an overall CES-rooted effort. I do have a DDN case open, though: 78804. If you are (as I suspect) a GPFS developer, do you mind if I cite your info from here in my DDN case to get them to open a PMR? Thanks. ~jonathon From: on behalf of Olaf Weiser Reply-To: gpfsug main discussion list Date: Tuesday, January 31, 2017 at 8:42 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] CES doesn't assign addresses to nodes ok.. so obviously ... it seems , that we have several issues.. the 3983 characters is obviously a defect have you already raised a PMR , if so , can you send me the number ? From: Jonathon A Anderson To: gpfsug main discussion list Date: 01/31/2017 04:14 PM Subject: Re: [gpfsug-discuss] CES doesn't assign addresses to nodes Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ The tail isn?t the issue; that? my addition, so that I didn?t have to paste the hundred or so line nodelist into the thread. The actual command is tsctl shownodes up | $tr ',' '\n' | $sort -o $upnodefile But you can see in my tailed output that the last hostname listed is cut-off halfway through the hostname. Less obvious in the example, but true, is the fact that it?s only showing the first 120 hosts, when we have 403 nodes in our gpfs cluster. [root at sgate2 ~]# tsctl shownodes up | tr ',' '\n' | wc -l 120 [root at sgate2 ~]# mmlscluster | grep '\-opa' | wc -l 403 Perhaps more explicitly, it looks like `tsctl shownodes up` can only transmit 3983 characters. [root at sgate2 ~]# tsctl shownodes up | wc -c 3983 Again, I?m convinced this is a bug not only because the command doesn?t actually produce a list of all of the up nodes in our cluster; but because the last name listed is incomplete. [root at sgate2 ~]# tsctl shownodes up | tr ',' '\n' | tail -n 1 shas0260-opa.rc.int.col[root at sgate2 ~]# I?d continue my investigation within tsctl itself but, alas, it?s a binary with no source code available to me. :) I?m trying to get this opened as a bug / PMR; but I?m still working through the DDN support infrastructure. Thanks for reporting it, though. For the record: [root at sgate2 ~]# rpm -qa | grep -i gpfs gpfs.base-4.2.1-2.x86_64 gpfs.msg.en_US-4.2.1-2.noarch gpfs.gplbin-3.10.0-327.el7.x86_64-4.2.1-0.x86_64 gpfs.gskit-8.0.50-57.x86_64 gpfs.gpl-4.2.1-2.noarch nfs-ganesha-gpfs-2.3.2-0.ibm24.el7.x86_64 gpfs.ext-4.2.1-2.x86_64 gpfs.gplbin-3.10.0-327.36.3.el7.x86_64-4.2.1-2.x86_64 gpfs.docs-4.2.1-2.noarch ~jonathon From: on behalf of Olaf Weiser Reply-To: gpfsug main discussion list Date: Tuesday, January 31, 2017 at 1:30 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] CES doesn't assign addresses to nodes Hi ...same thing here.. everything after 10 nodes will be truncated.. though I don't have an issue with it ... I 'll open a PMR .. and I recommend you to do the same thing.. ;-) the reason seems simple.. it is the "| tail" .at the end of the command.. .. which truncates the output to the last 10 items... should be easy to fix.. cheers olaf From: Jonathon A Anderson To: "gpfsug-discuss at spectrumscale.org" Date: 01/30/2017 11:11 PM Subject: Re: [gpfsug-discuss] CES doesn't assign addresses to nodes Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ In trying to figure this out on my own, I?m relatively certain I?ve found a bug in GPFS related to the truncation of output from `tsctl shownodes up`. Any chance someone in development can confirm? Here are the details of my investigation: ## GPFS is up on sgate2 [root at sgate2 ~]# mmgetstate Node number Node name GPFS state ------------------------------------------ 414 sgate2-opa active ## but if I tell ces to explicitly put one of our ces addresses on that node, it says that GPFS is down [root at sgate2 ~]# mmces address move --ces-ip 10.225.71.102 --ces-node sgate2-opa mmces address move: GPFS is down on this node. mmces address move: Command failed. Examine previous error messages to determine cause. ## the ?GPFS is down on this node? message is defined as code 109 in mmglobfuncs [root at sgate2 ~]# grep --before-context=1 "GPFS is down on this node." /usr/lpp/mmfs/bin/mmglobfuncs 109 ) msgTxt=\ "%s: GPFS is down on this node." ## and is generated by printErrorMsg in mmcesnetmvaddress when it detects that the current node is identified as ?down? by getDownCesNodeList [root at sgate2 ~]# grep --before-context=5 'printErrorMsg 109' /usr/lpp/mmfs/bin/mmcesnetmvaddress downNodeList=$(getDownCesNodeList) for downNode in $downNodeList do if [[ $toNodeName == $downNode ]] then printErrorMsg 109 "$mmcmd" ## getDownCesNodeList is the intersection of all ces nodes with GPFS cluster nodes listed in `tsctl shownodes up` [root at sgate2 ~]# grep --after-context=16 '^function getDownCesNodeList' /usr/lpp/mmfs/bin/mmcesfuncs function getDownCesNodeList { typeset sourceFile="mmcesfuncs.sh" [[ -n $DEBUG || -n $DEBUGgetDownCesNodeList ]] &&set -x $mmTRACE_ENTER "$*" typeset upnodefile=${cmdTmpDir}upnodefile typeset downNodeList # get all CES nodes $sort -o $nodefile $mmfsCesNodes.dae $tsctl shownodes up | $tr ',' '\n' | $sort -o $upnodefile downNodeList=$($comm -23 $nodefile $upnodefile) print -- $downNodeList } #----- end of function getDownCesNodeList -------------------- ## but not only are the sgate nodes not listed by `tsctl shownodes up`; its output is obviously and erroneously truncated [root at sgate2 ~]# tsctl shownodes up | tr ',' '\n' | tail shas0251-opa.rc.int.colorado.edu shas0252-opa.rc.int.colorado.edu shas0253-opa.rc.int.colorado.edu shas0254-opa.rc.int.colorado.edu shas0255-opa.rc.int.colorado.edu shas0256-opa.rc.int.colorado.edu shas0257-opa.rc.int.colorado.edu shas0258-opa.rc.int.colorado.edu shas0259-opa.rc.int.colorado.edu shas0260-opa.rc.int.col[root at sgate2 ~]# ## I expect that this is a bug in GPFS, likely related to a maximum output buffer for `tsctl shownodes up`. On 1/24/17, 12:48 PM, "Jonathon A Anderson" wrote: I think I'm having the same issue described here: http://www.spectrumscale.org/pipermail/gpfsug-discuss/2016-October/002288.html Any advice or further troubleshooting steps would be much appreciated. Full disclosure: I also have a DDN case open. (78804) We've got a four-node (snsd{1..4}) DDN gridscaler system. I'm trying to add two CES protocol nodes (sgate{1,2}) to serve NFS. Here's the steps I took: --- mmcrnodeclass protocol -N sgate1-opa,sgate2-opa mmcrnodeclass nfs -N sgate1-opa,sgate2-opa mmchconfig cesSharedRoot=/gpfs/summit/ces mmchcluster --ccr-enable mmchnode --ces-enable -N protocol mmces service enable NFS mmces service start NFS -N nfs mmces address add --ces-ip 10.225.71.104,10.225.71.105 mmces address policy even-coverage mmces address move --rebalance --- This worked the very first time I ran it, but the CES addresses weren't re-distributed after restarting GPFS or a node reboot. Things I've tried: * disabling ces on the sgate nodes and re-running the above procedure * moving the cluster and filesystem managers to different snsd nodes * deleting and re-creating the cesSharedRoot directory Meanwhile, the following log entry appears in mmfs.log.latest every ~30s: --- Mon Jan 23 20:31:20 MST 2017: mmcesnetworkmonitor: Found unassigned address 10.225.71.104 Mon Jan 23 20:31:20 MST 2017: mmcesnetworkmonitor: Found unassigned address 10.225.71.105 Mon Jan 23 20:31:20 MST 2017: mmcesnetworkmonitor: handleNetworkProblem with lock held: assignIP 10.225.71.104_0-_+,10.225.71.105_0-_+ 1 Mon Jan 23 20:31:20 MST 2017: mmcesnetworkmonitor: Assigning addresses: 10.225.71.104_0-_+,10.225.71.105_0-_+ Mon Jan 23 20:31:20 MST 2017: mmcesnetworkmonitor: moveCesIPs: 10.225.71.104_0-_+,10.225.71.105_0-_+ --- Also notable, whenever I add or remove addresses now, I see this in mmsysmonitor.log (among a lot of other entries): --- 2017-01-23T20:40:56.363 sgate1 D ET_cesnetwork Entity state without requireUnique: ces_network_ips_down WARNING No CES relevant NICs detected - Service.calculateAndUpdateState:275 2017-01-23T20:40:11.364 sgate1 D ET_cesnetwork Update multiple entities at once {'p2p2': 1, 'bond0': 1, 'p2p1': 1} - Service.setLocalState:333 --- For the record, here's the interface I expect to get the address on sgate1: --- 11: bond0: mtu 9000 qdisc noqueue state UP link/ether 3c:fd:fe:08:a7:c0 brd ff:ff:ff:ff:ff:ff inet 10.225.71.107/20 brd 10.225.79.255 scope global bond0 valid_lft forever preferred_lft forever inet6 fe80::3efd:feff:fe08:a7c0/64 scope link valid_lft forever preferred_lft forever --- which is a bond of p2p1 and p2p2. --- 6: p2p1: mtu 9000 qdisc mq master bond0 state UP qlen 1000 link/ether 3c:fd:fe:08:a7:c0 brd ff:ff:ff:ff:ff:ff 7: p2p2: mtu 9000 qdisc mq master bond0 state UP qlen 1000 link/ether 3c:fd:fe:08:a7:c0 brd ff:ff:ff:ff:ff:ff --- A similar bond0 exists on sgate2. I crawled around in /usr/lpp/mmfs/lib/mmsysmon/CESNetworkService.py for a while trying to figure it out, but have been unsuccessful so far. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From mweil at wustl.edu Thu Mar 23 15:23:56 2017 From: mweil at wustl.edu (Matt Weil) Date: Thu, 23 Mar 2017 10:23:56 -0500 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: <47e18bb0062544f4b510ce0e87049fd3@jumptrading.com> References: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu> <47e18bb0062544f4b510ce0e87049fd3@jumptrading.com> Message-ID: <643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> FYI all We also ran into this after bumping maxFilesToCache. Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: unexpected error: Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many open files in system fix sysctl -w fs.file-max=1000000 On 3/22/17 12:01 PM, Bryan Banister wrote: > We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. > > Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil > Sent: Wednesday, March 22, 2017 11:43 AM > To: gpfsug main discussion list > Subject: [gpfsug-discuss] CES node slow to respond > > All, > > We had an indecent yesterday where one of our CES nodes slowed to a > crawl. GPFS waiters showed pre fetch threads going after inodes. > iohist also showed lots of inode fetching. Then we noticed that the CES > host had 5.4 million files open. > > The change I made was to set maxStatCache=DEFAULT because it is linux. > And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. > > Is there something else we should change as well. > > Thanks > > Matt > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. From bbanister at jumptrading.com Thu Mar 23 15:26:57 2017 From: bbanister at jumptrading.com (Bryan Banister) Date: Thu, 23 Mar 2017 15:26:57 +0000 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: <643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> References: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu> <47e18bb0062544f4b510ce0e87049fd3@jumptrading.com> <643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> Message-ID: Anybody from IBM willing/able to give us some explanation of why ganesha is holding open so many files? Is this expected/needed/etc? Or do we have to open a PMR to get some kind of explanation? -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Thursday, March 23, 2017 10:24 AM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] CES node slow to respond FYI all We also ran into this after bumping maxFilesToCache. Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: unexpected error: Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many open files in system fix sysctl -w fs.file-max=1000000 On 3/22/17 12:01 PM, Bryan Banister wrote: > We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. > > Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil > Sent: Wednesday, March 22, 2017 11:43 AM > To: gpfsug main discussion list > Subject: [gpfsug-discuss] CES node slow to respond > > All, > > We had an indecent yesterday where one of our CES nodes slowed to a > crawl. GPFS waiters showed pre fetch threads going after inodes. > iohist also showed lots of inode fetching. Then we noticed that the CES > host had 5.4 million files open. > > The change I made was to set maxStatCache=DEFAULT because it is linux. > And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. > > Is there something else we should change as well. > > Thanks > > Matt > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. From mweil at wustl.edu Thu Mar 23 15:35:30 2017 From: mweil at wustl.edu (Matt Weil) Date: Thu, 23 Mar 2017 10:35:30 -0500 Subject: [gpfsug-discuss] Dual homing CES nodes Message-ID: Hello all, Are there any issues with connecting CES nodes to multiple networks? Thanks Matt ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. From kevindjo at us.ibm.com Thu Mar 23 15:39:48 2017 From: kevindjo at us.ibm.com (Kevin D Johnson) Date: Thu, 23 Mar 2017 15:39:48 +0000 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: References: , <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu><47e18bb0062544f4b510ce0e87049fd3@jumptrading.com><643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> Message-ID: An HTML attachment was scrubbed... URL: From jonathon.anderson at colorado.edu Thu Mar 23 15:41:47 2017 From: jonathon.anderson at colorado.edu (Jonathon A Anderson) Date: Thu, 23 Mar 2017 15:41:47 +0000 Subject: [gpfsug-discuss] Dual homing CES nodes In-Reply-To: References: Message-ID: <0FE05202-B444-4EBB-994C-AB2CC229F1DA@colorado.edu> We?ve run CES with addresses on a different network than the GPFS admin or daemon interface; but I haven?t tried running CES with addresses on multiple networks itself. ~jonathon On 3/23/17, 9:35 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Matt Weil" wrote: Hello all, Are there any issues with connecting CES nodes to multiple networks? Thanks Matt ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From skylar2 at u.washington.edu Thu Mar 23 15:42:18 2017 From: skylar2 at u.washington.edu (Skylar Thompson) Date: Thu, 23 Mar 2017 15:42:18 +0000 Subject: [gpfsug-discuss] Dual homing CES nodes In-Reply-To: References: Message-ID: <20170323154218.wqjhjaszffinajy5@utumno.gs.washington.edu> The only thing I can think of is you should be careful that you have distinct CES groups so that addresses will failover to the right networks. On Thu, Mar 23, 2017 at 10:35:30AM -0500, Matt Weil wrote: > Hello all, > > Are there any issues with connecting CES nodes to multiple networks? -- -- Skylar Thompson (skylar2 at u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine From mweil at wustl.edu Thu Mar 23 15:44:47 2017 From: mweil at wustl.edu (Matt Weil) Date: Thu, 23 Mar 2017 10:44:47 -0500 Subject: [gpfsug-discuss] Dual homing CES nodes In-Reply-To: <20170323154218.wqjhjaszffinajy5@utumno.gs.washington.edu> References: <20170323154218.wqjhjaszffinajy5@utumno.gs.washington.edu> Message-ID: <43fe47d2-f3a7-2b52-4a21-b135c7141e9f@wustl.edu> yeah it seems to know which interface to use. here is my quick test.. no failure groups. > em2: flags=4099 mtu 1500 > inet 192.168.1.2 netmask 255.255.255.0 broadcast 192.168.1.255 > ether 84:2b:2b:47:70:35 txqueuelen 1000 (Ethernet) > RX packets 0 bytes 0 (0.0 B) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 0 bytes 0 (0.0 B) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > em2:0: flags=4099 mtu 1500 > inet 192.168.1.5 netmask 255.255.255.0 broadcast 192.168.1.255 > ether 84:2b:2b:47:70:35 txqueuelen 1000 (Ethernet) On 3/23/17 10:42 AM, Skylar Thompson wrote: > The only thing I can think of is you should be careful that you have > distinct CES groups so that addresses will failover to the right networks. > > On Thu, Mar 23, 2017 at 10:35:30AM -0500, Matt Weil wrote: >> Hello all, >> >> Are there any issues with connecting CES nodes to multiple networks? ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. From S.J.Thompson at bham.ac.uk Thu Mar 23 15:46:38 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Thu, 23 Mar 2017 15:46:38 +0000 Subject: [gpfsug-discuss] Dual homing CES nodes In-Reply-To: References: Message-ID: We've done this in an unsupported manner when we were testing connectivity to private VM networks - manually attach VXLAN interface to host, then enable CES on the segment. Worked fine for us in this, so don't see why it should;t work fine for real/VLAN interfaces. Just be mindful of asymmetric routing of course. Simon >Hello all, > >Are there any issues with connecting CES nodes to multiple networks? > >Thanks >Matt > >________________________________ >The materials in this message are private and may contain Protected >Healthcare Information or other information of a sensitive nature. If you >are not the intended recipient, be advised that any unauthorized use, >disclosure, copying or the taking of any action in reliance on the >contents of this information is strictly prohibited. If you have received >this email in error, please immediately notify the sender via telephone >or return mail. >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss From ulmer at ulmer.org Thu Mar 23 18:04:17 2017 From: ulmer at ulmer.org (Stephen Ulmer) Date: Thu, 23 Mar 2017 14:04:17 -0400 Subject: [gpfsug-discuss] GPFS on PowerVM Shared Storage Pools Message-ID: <91792F99-B4EB-4B96-9F89-01B30F49108A@ulmer.org> I have a client who is very enamored with PowerVM Shared Storage Pools (because they work great for them). Has anyone here implemented GPFS on such? I think that technically there?s no reason that it wouldn?t work, but I?ve never actually "owned" an installation with GPFS on SSPs. Any thoughts? NB: There is an entry on the SSP FAQ that lists "What disk types are NOT supported for Shared Storage Pool use?", among which GPFS is listed. I take that to mean that you can?t deploy an SSP on GPFS storage (not the other way around). This entry also lists iSCSI, for example, which is definitely not supported for SSP use. Liberty, -- Stephen From aaron.s.knister at nasa.gov Fri Mar 24 16:43:02 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 24 Mar 2017 12:43:02 -0400 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock Message-ID: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Since yesterday morning we've noticed some deadlocks on one of our filesystems that seem to be triggered by writing to it. The waiters on the clients look like this: 0x19450B0 ( 6730) waiting 2063.294589599 seconds, SyncHandlerThread: on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) (InodeFlushCondVar), reason 'waiting for the flush flag to commit metadata' 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 (0x7FFFDAC7FE28) (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRelinquishRegion on node 10.1.52.33 0x197EE70 ( 6776) waiting 0.000198354 seconds, FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRequestRegion on node 10.1.52.33 (10.1.52.33/c0n3271 is the fs manager for the filesystem in question) there's a single process running on this node writing to the filesystem in question (well, trying to write, it's been blocked doing nothing for half an hour now). There are ~10 other client nodes in this situation right now. We had many more last night before the problem seemed to disappear in the early hours of the morning and now its back. Waiters on the fs manager look like this. While the individual waiter is short it's a near constant stream: 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg handler allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg handler allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg handler allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) I don't know if this data point is useful but both yesterday and today the metadata NSDs for this filesystem have had a constant aggregate stream of 25MB/s 4kop/s reads during each episode (very low latency though so I don't believe the storage is a bottleneck here). Writes are only a few hundred ops and didn't strike me as odd. I have a PMR open for this but I'm curious if folks have seen this in the wild and what it might mean. -Aaron -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From jfosburg at mdanderson.org Fri Mar 24 16:50:03 2017 From: jfosburg at mdanderson.org (Fosburgh,Jonathan) Date: Fri, 24 Mar 2017 16:50:03 +0000 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: <1490374200.1901.0.camel@mdanderson.org> This is one of the more annoying long waiter problems. We've seen it several times and I'm not sure they all had the same cause. What version of GPFS? Do you have anything like Tivoli HSM or LTFSEE? -- Jonathan Fosburgh Principal Application Systems Analyst Storage Team IT Operations jfosburg at mdanderson.org (713) 745-9346 -----Original Message----- Date: Fri, 24 Mar 2017 12:43:02 -0400 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock To: gpfsug main discussion list > Reply-to: gpfsug main discussion list From: Aaron Knister > Since yesterday morning we've noticed some deadlocks on one of our filesystems that seem to be triggered by writing to it. The waiters on the clients look like this: 0x19450B0 ( 6730) waiting 2063.294589599 seconds, SyncHandlerThread: on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) (InodeFlushCondVar), reason 'waiting for the flush flag to commit metadata' 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 (0x7FFFDAC7FE28) (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRelinquishRegion on node 10.1.52.33 0x197EE70 ( 6776) waiting 0.000198354 seconds, FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRequestRegion on node 10.1.52.33 (10.1.52.33/c0n3271 is the fs manager for the filesystem in question) there's a single process running on this node writing to the filesystem in question (well, trying to write, it's been blocked doing nothing for half an hour now). There are ~10 other client nodes in this situation right now. We had many more last night before the problem seemed to disappear in the early hours of the morning and now its back. Waiters on the fs manager look like this. While the individual waiter is short it's a near constant stream: 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg handler allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg handler allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg handler allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) I don't know if this data point is useful but both yesterday and today the metadata NSDs for this filesystem have had a constant aggregate stream of 25MB/s 4kop/s reads during each episode (very low latency though so I don't believe the storage is a bottleneck here). Writes are only a few hundred ops and didn't strike me as odd. I have a PMR open for this but I'm curious if folks have seen this in the wild and what it might mean. -Aaron The information contained in this e-mail message may be privileged, confidential, and/or protected from disclosure. This e-mail message may contain protected health information (PHI); dissemination of PHI should comply with applicable federal and state laws. If you are not the intended recipient, or an authorized representative of the intended recipient, any further review, disclosure, use, dissemination, distribution, or copying of this message or any attachment (or the information contained therein) is strictly prohibited. If you think that you have received this e-mail message in error, please notify the sender by return e-mail and delete all references to it and its contents from your systems. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Fri Mar 24 16:50:47 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Fri, 24 Mar 2017 16:50:47 +0000 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock Message-ID: <29B113F2-BDE9-456A-BBEB-9B1C203CB5F4@nuance.com> Hi Aaron Yes, I have seen this several times over the last 6 months. I opened at least one PMR on it and they never could track it down. I did some snap dumps but without some traces, they did not have enough. I ended up getting out of it by selectively rebooting some of my NSD servers. My suspicion is that one of them had a thread deadlocked and was holding up IO to the rest of the filesystem. I haven?t seen it since I updated to 4.2.2-2, but I?m not convinced (yet) that it?s not lurking in the background. Bob Oesterlin Sr Principal Storage Engineer, Nuance On 3/24/17, 11:43 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Aaron Knister" wrote: Since yesterday morning we've noticed some deadlocks on one of our filesystems that seem to be triggered by writing to it. The waiters on the clients look like this: 0x19450B0 ( 6730) waiting 2063.294589599 seconds, SyncHandlerThread: on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) (InodeFlushCondVar), reason 'waiting for the flush flag to commit metadata' 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 (0x7FFFDAC7FE28) (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRelinquishRegion on node 10.1.52.33 0x197EE70 ( 6776) waiting 0.000198354 seconds, FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRequestRegion on node 10.1.52.33 (10.1.52.33/c0n3271 is the fs manager for the filesystem in question) there's a single process running on this node writing to the filesystem in question (well, trying to write, it's been blocked doing nothing for half an hour now). There are ~10 other client nodes in this situation right now. We had many more last night before the problem seemed to disappear in the early hours of the morning and now its back. Waiters on the fs manager look like this. While the individual waiter is short it's a near constant stream: 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg handler allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg handler allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg handler allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) I don't know if this data point is useful but both yesterday and today the metadata NSDs for this filesystem have had a constant aggregate stream of 25MB/s 4kop/s reads during each episode (very low latency though so I don't believe the storage is a bottleneck here). Writes are only a few hundred ops and didn't strike me as odd. I have a PMR open for this but I'm curious if folks have seen this in the wild and what it might mean. -Aaron -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=fUSq1Wdz1p_yJQVOgOoqe7fu7nsPXewvz8BUeKJyiRg&s=9sXk_xPuEtEyJgVhsZ7FCgM-rfytQGDDC2EyqwYgLhQ&e= From oehmes at gmail.com Fri Mar 24 17:04:02 2017 From: oehmes at gmail.com (Sven Oehme) Date: Fri, 24 Mar 2017 17:04:02 +0000 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: while this is happening run top and see if there is very high cpu utilization at this time on the NSD Server. if there is , run perf top (you might need to install perf command) and see if the top cpu contender is a spinlock . if so send a screenshot of perf top as i may know what that is and how to fix. sven On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister wrote: > Since yesterday morning we've noticed some deadlocks on one of our > filesystems that seem to be triggered by writing to it. The waiters on > the clients look like this: > > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, SyncHandlerThread: > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) (InodeFlushCondVar), reason > 'waiting for the flush flag to commit metadata' > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 (0x7FFFDAC7FE28) > (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRelinquishRegion > on node 10.1.52.33 > 0x197EE70 ( 6776) waiting 0.000198354 seconds, > FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for > allocMsgTypeRequestRegion on node 10.1.52.33 > > (10.1.52.33/c0n3271 is the fs manager for the filesystem in question) > > there's a single process running on this node writing to the filesystem > in question (well, trying to write, it's been blocked doing nothing for > half an hour now). There are ~10 other client nodes in this situation > right now. We had many more last night before the problem seemed to > disappear in the early hours of the morning and now its back. > > Waiters on the fs manager look like this. While the individual waiter is > short it's a near constant stream: > > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > I don't know if this data point is useful but both yesterday and today > the metadata NSDs for this filesystem have had a constant aggregate > stream of 25MB/s 4kop/s reads during each episode (very low latency > though so I don't believe the storage is a bottleneck here). Writes are > only a few hundred ops and didn't strike me as odd. > > I have a PMR open for this but I'm curious if folks have seen this in > the wild and what it might mean. > > -Aaron > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Fri Mar 24 17:07:58 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 24 Mar 2017 13:07:58 -0400 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: Hi Sven, Which NSD server should I run top on, the fs manager? If so the CPU load is about 155%. I'm working on perf top but not off to a great start... # perf top PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% [1000Hz cycles], (all, 28 CPUs) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Segmentation fault -Aaron On 3/24/17 1:04 PM, Sven Oehme wrote: > while this is happening run top and see if there is very high cpu > utilization at this time on the NSD Server. > > if there is , run perf top (you might need to install perf command) and > see if the top cpu contender is a spinlock . if so send a screenshot of > perf top as i may know what that is and how to fix. > > sven > > > On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister > wrote: > > Since yesterday morning we've noticed some deadlocks on one of our > filesystems that seem to be triggered by writing to it. The waiters on > the clients look like this: > > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, SyncHandlerThread: > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) (InodeFlushCondVar), reason > 'waiting for the flush flag to commit metadata' > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 (0x7FFFDAC7FE28) > (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRelinquishRegion > on node 10.1.52.33 > 0x197EE70 ( 6776) waiting 0.000198354 seconds, > FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for > allocMsgTypeRequestRegion on node 10.1.52.33 > > (10.1.52.33/c0n3271 is the fs manager > for the filesystem in question) > > there's a single process running on this node writing to the filesystem > in question (well, trying to write, it's been blocked doing nothing for > half an hour now). There are ~10 other client nodes in this situation > right now. We had many more last night before the problem seemed to > disappear in the early hours of the morning and now its back. > > Waiters on the fs manager look like this. While the individual waiter is > short it's a near constant stream: > > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > I don't know if this data point is useful but both yesterday and today > the metadata NSDs for this filesystem have had a constant aggregate > stream of 25MB/s 4kop/s reads during each episode (very low latency > though so I don't believe the storage is a bottleneck here). Writes are > only a few hundred ops and didn't strike me as odd. > > I have a PMR open for this but I'm curious if folks have seen this in > the wild and what it might mean. > > -Aaron > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From scale at us.ibm.com Fri Mar 24 17:17:33 2017 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Fri, 24 Mar 2017 09:17:33 -0800 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: References: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu><47e18bb0062544f4b510ce0e87049fd3@jumptrading.com><643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> Message-ID: Hi Bryan, Making sure Malahal's reply was received by the user group. >> Then we noticed that the CES host had 5.4 million files open This is technically not possible with ganesha alone. A process can only open 1 million files on RHEL distro. Either we have leaks in kernel or some other processes contributing to this. Ganesha does keep NFSv3 files open and keep open for performance. It doesn't have a good logic to close them after some inactivity. It does close them if the number is close to max open files which is configurable. PS: kNFS does open, read/write, and then close. No caching in older versions. They did have a feature in a recent code to cache open files in NFSv3 as well. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Bryan Banister To: gpfsug main discussion list Date: 03/23/2017 08:27 AM Subject: Re: [gpfsug-discuss] CES node slow to respond Sent by: gpfsug-discuss-bounces at spectrumscale.org Anybody from IBM willing/able to give us some explanation of why ganesha is holding open so many files? Is this expected/needed/etc? Or do we have to open a PMR to get some kind of explanation? -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Thursday, March 23, 2017 10:24 AM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] CES node slow to respond FYI all We also ran into this after bumping maxFilesToCache. Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: unexpected error: Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many open files in system fix sysctl -w fs.file-max=1000000 On 3/22/17 12:01 PM, Bryan Banister wrote: > We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. > > Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil > Sent: Wednesday, March 22, 2017 11:43 AM > To: gpfsug main discussion list > Subject: [gpfsug-discuss] CES node slow to respond > > All, > > We had an indecent yesterday where one of our CES nodes slowed to a > crawl. GPFS waiters showed pre fetch threads going after inodes. > iohist also showed lots of inode fetching. Then we noticed that the CES > host had 5.4 million files open. > > The change I made was to set maxStatCache=DEFAULT because it is linux. > And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. > > Is there something else we should change as well. > > Thanks > > Matt > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From oehmes at gmail.com Fri Mar 24 17:22:12 2017 From: oehmes at gmail.com (Sven Oehme) Date: Fri, 24 Mar 2017 17:22:12 +0000 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: you must be on sles as this segfaults only on sles to my knowledge :-) i am looking for a NSD or manager node in your cluster that runs at 100% cpu usage. do you have zimon deployed to look at cpu utilization across your nodes ? sven On Fri, Mar 24, 2017 at 10:08 AM Aaron Knister wrote: Hi Sven, Which NSD server should I run top on, the fs manager? If so the CPU load is about 155%. I'm working on perf top but not off to a great start... # perf top PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% [1000Hz cycles], (all, 28 CPUs) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Segmentation fault -Aaron On 3/24/17 1:04 PM, Sven Oehme wrote: > while this is happening run top and see if there is very high cpu > utilization at this time on the NSD Server. > > if there is , run perf top (you might need to install perf command) and > see if the top cpu contender is a spinlock . if so send a screenshot of > perf top as i may know what that is and how to fix. > > sven > > > On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister > wrote: > > Since yesterday morning we've noticed some deadlocks on one of our > filesystems that seem to be triggered by writing to it. The waiters on > the clients look like this: > > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, SyncHandlerThread: > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) (InodeFlushCondVar), reason > 'waiting for the flush flag to commit metadata' > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 (0x7FFFDAC7FE28) > (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRelinquishRegion > on node 10.1.52.33 > 0x197EE70 ( 6776) waiting 0.000198354 seconds, > FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for > allocMsgTypeRequestRegion on node 10.1.52.33 > > (10.1.52.33/c0n3271 is the fs manager > for the filesystem in question) > > there's a single process running on this node writing to the filesystem > in question (well, trying to write, it's been blocked doing nothing for > half an hour now). There are ~10 other client nodes in this situation > right now. We had many more last night before the problem seemed to > disappear in the early hours of the morning and now its back. > > Waiters on the fs manager look like this. While the individual waiter is > short it's a near constant stream: > > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > I don't know if this data point is useful but both yesterday and today > the metadata NSDs for this filesystem have had a constant aggregate > stream of 25MB/s 4kop/s reads during each episode (very low latency > though so I don't believe the storage is a bottleneck here). Writes are > only a few hundred ops and didn't strike me as odd. > > I have a PMR open for this but I'm curious if folks have seen this in > the wild and what it might mean. > > -Aaron > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Fri Mar 24 17:32:26 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 24 Mar 2017 13:32:26 -0400 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: heh, yep we're on sles :) here's a screenshot of the fs manager from the deadlocked filesystem. I don't think there's an nsd server or manager node that's running full throttle across all cpus. There is one that's got relatively high CPU utilization though (300-400%). I'll send a screenshot of it in a sec. no zimon yet but we do have other tools to see cpu utilization. -Aaron On 3/24/17 1:22 PM, Sven Oehme wrote: > you must be on sles as this segfaults only on sles to my knowledge :-) > > i am looking for a NSD or manager node in your cluster that runs at 100% > cpu usage. > > do you have zimon deployed to look at cpu utilization across your nodes ? > > sven > > > > On Fri, Mar 24, 2017 at 10:08 AM Aaron Knister > wrote: > > Hi Sven, > > Which NSD server should I run top on, the fs manager? If so the CPU load > is about 155%. I'm working on perf top but not off to a great start... > > # perf top > PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% [1000Hz > cycles], (all, 28 CPUs) > --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Segmentation fault > > -Aaron > > On 3/24/17 1:04 PM, Sven Oehme wrote: > > while this is happening run top and see if there is very high cpu > > utilization at this time on the NSD Server. > > > > if there is , run perf top (you might need to install perf command) and > > see if the top cpu contender is a spinlock . if so send a screenshot of > > perf top as i may know what that is and how to fix. > > > > sven > > > > > > On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister > > >> wrote: > > > > Since yesterday morning we've noticed some deadlocks on one of our > > filesystems that seem to be triggered by writing to it. The waiters on > > the clients look like this: > > > > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, SyncHandlerThread: > > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) (InodeFlushCondVar), reason > > 'waiting for the flush flag to commit metadata' > > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, > > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 (0x7FFFDAC7FE28) > > (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRelinquishRegion > > on node 10.1.52.33 > > 0x197EE70 ( 6776) waiting 0.000198354 seconds, > > FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 > > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for > > allocMsgTypeRequestRegion on node 10.1.52.33 > > > > (10.1.52.33/c0n3271 > is the fs manager > > for the filesystem in question) > > > > there's a single process running on this node writing to the filesystem > > in question (well, trying to write, it's been blocked doing nothing for > > half an hour now). There are ~10 other client nodes in this situation > > right now. We had many more last night before the problem seemed to > > disappear in the early hours of the morning and now its back. > > > > Waiters on the fs manager look like this. While the individual waiter is > > short it's a near constant stream: > > > > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg handler > > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > > (AllocManagerMutex) > > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg handler > > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg handler > > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > > (AllocManagerMutex) > > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg handler > > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > > (AllocManagerMutex) > > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg handler > > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg handler > > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > > (AllocManagerMutex) > > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg handler > > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > > (AllocManagerMutex) > > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg handler > > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > > > I don't know if this data point is useful but both yesterday and today > > the metadata NSDs for this filesystem have had a constant aggregate > > stream of 25MB/s 4kop/s reads during each episode (very low latency > > though so I don't believe the storage is a bottleneck here). Writes are > > only a few hundred ops and didn't strike me as odd. > > > > I have a PMR open for this but I'm curious if folks have seen this in > > the wild and what it might mean. > > > > -Aaron > > > > -- > > Aaron Knister > > NASA Center for Climate Simulation (Code 606.2) > > Goddard Space Flight Center > > (301) 286-2776 > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 -------------- next part -------------- A non-text attachment was scrubbed... Name: perf_top.png Type: image/png Size: 218901 bytes Desc: not available URL: From aaron.s.knister at nasa.gov Fri Mar 24 17:34:30 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 24 Mar 2017 13:34:30 -0400 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: Here's the screenshot from the other node with the high cpu utilization. On 3/24/17 1:32 PM, Aaron Knister wrote: > heh, yep we're on sles :) > > here's a screenshot of the fs manager from the deadlocked filesystem. I > don't think there's an nsd server or manager node that's running full > throttle across all cpus. There is one that's got relatively high CPU > utilization though (300-400%). I'll send a screenshot of it in a sec. > > no zimon yet but we do have other tools to see cpu utilization. > > -Aaron > > On 3/24/17 1:22 PM, Sven Oehme wrote: >> you must be on sles as this segfaults only on sles to my knowledge :-) >> >> i am looking for a NSD or manager node in your cluster that runs at 100% >> cpu usage. >> >> do you have zimon deployed to look at cpu utilization across your nodes ? >> >> sven >> >> >> >> On Fri, Mar 24, 2017 at 10:08 AM Aaron Knister > > wrote: >> >> Hi Sven, >> >> Which NSD server should I run top on, the fs manager? If so the >> CPU load >> is about 155%. I'm working on perf top but not off to a great >> start... >> >> # perf top >> PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% [1000Hz >> cycles], (all, 28 CPUs) >> >> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >> >> Segmentation fault >> >> -Aaron >> >> On 3/24/17 1:04 PM, Sven Oehme wrote: >> > while this is happening run top and see if there is very high cpu >> > utilization at this time on the NSD Server. >> > >> > if there is , run perf top (you might need to install perf >> command) and >> > see if the top cpu contender is a spinlock . if so send a >> screenshot of >> > perf top as i may know what that is and how to fix. >> > >> > sven >> > >> > >> > On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister >> >> > > >> wrote: >> > >> > Since yesterday morning we've noticed some deadlocks on one >> of our >> > filesystems that seem to be triggered by writing to it. The >> waiters on >> > the clients look like this: >> > >> > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, >> SyncHandlerThread: >> > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) >> (InodeFlushCondVar), reason >> > 'waiting for the flush flag to commit metadata' >> > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, >> > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 >> (0x7FFFDAC7FE28) >> > (MsgRecordCondvar), reason 'RPC wait' for >> allocMsgTypeRelinquishRegion >> > on node 10.1.52.33 >> > 0x197EE70 ( 6776) waiting 0.000198354 seconds, >> > FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 >> > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for >> > allocMsgTypeRequestRegion on node 10.1.52.33 >> > >> > (10.1.52.33/c0n3271 >> is the fs manager >> > for the filesystem in question) >> > >> > there's a single process running on this node writing to the >> filesystem >> > in question (well, trying to write, it's been blocked doing >> nothing for >> > half an hour now). There are ~10 other client nodes in this >> situation >> > right now. We had many more last night before the problem >> seemed to >> > disappear in the early hours of the morning and now its back. >> > >> > Waiters on the fs manager look like this. While the >> individual waiter is >> > short it's a near constant stream: >> > >> > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg >> handler >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) >> > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg >> handler >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) >> > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg >> handler >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) >> > >> > I don't know if this data point is useful but both yesterday >> and today >> > the metadata NSDs for this filesystem have had a constant >> aggregate >> > stream of 25MB/s 4kop/s reads during each episode (very low >> latency >> > though so I don't believe the storage is a bottleneck here). >> Writes are >> > only a few hundred ops and didn't strike me as odd. >> > >> > I have a PMR open for this but I'm curious if folks have >> seen this in >> > the wild and what it might mean. >> > >> > -Aaron >> > >> > -- >> > Aaron Knister >> > NASA Center for Climate Simulation (Code 606.2) >> > Goddard Space Flight Center >> > (301) 286-2776 >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > >> > >> > >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > >> >> -- >> Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) >> Goddard Space Flight Center >> (301) 286-2776 >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 -------------- next part -------------- A non-text attachment was scrubbed... Name: nsd35.png Type: image/png Size: 243009 bytes Desc: not available URL: From oehmes at gmail.com Fri Mar 24 17:41:10 2017 From: oehmes at gmail.com (Sven Oehme) Date: Fri, 24 Mar 2017 17:41:10 +0000 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: ok, that seems a different problem then i was thinking. can you send output of mmlscluster, mmlsconfig, mmlsfs all ? also are you getting close to fill grade on inodes or capacity on any of the filesystems ? sven On Fri, Mar 24, 2017 at 10:34 AM Aaron Knister wrote: > Here's the screenshot from the other node with the high cpu utilization. > > On 3/24/17 1:32 PM, Aaron Knister wrote: > > heh, yep we're on sles :) > > > > here's a screenshot of the fs manager from the deadlocked filesystem. I > > don't think there's an nsd server or manager node that's running full > > throttle across all cpus. There is one that's got relatively high CPU > > utilization though (300-400%). I'll send a screenshot of it in a sec. > > > > no zimon yet but we do have other tools to see cpu utilization. > > > > -Aaron > > > > On 3/24/17 1:22 PM, Sven Oehme wrote: > >> you must be on sles as this segfaults only on sles to my knowledge :-) > >> > >> i am looking for a NSD or manager node in your cluster that runs at 100% > >> cpu usage. > >> > >> do you have zimon deployed to look at cpu utilization across your nodes > ? > >> > >> sven > >> > >> > >> > >> On Fri, Mar 24, 2017 at 10:08 AM Aaron Knister < > aaron.s.knister at nasa.gov > >> > wrote: > >> > >> Hi Sven, > >> > >> Which NSD server should I run top on, the fs manager? If so the > >> CPU load > >> is about 155%. I'm working on perf top but not off to a great > >> start... > >> > >> # perf top > >> PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% [1000Hz > >> cycles], (all, 28 CPUs) > >> > >> > --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > >> > >> Segmentation fault > >> > >> -Aaron > >> > >> On 3/24/17 1:04 PM, Sven Oehme wrote: > >> > while this is happening run top and see if there is very high cpu > >> > utilization at this time on the NSD Server. > >> > > >> > if there is , run perf top (you might need to install perf > >> command) and > >> > see if the top cpu contender is a spinlock . if so send a > >> screenshot of > >> > perf top as i may know what that is and how to fix. > >> > > >> > sven > >> > > >> > > >> > On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister > >> > >> > >> >> wrote: > >> > > >> > Since yesterday morning we've noticed some deadlocks on one > >> of our > >> > filesystems that seem to be triggered by writing to it. The > >> waiters on > >> > the clients look like this: > >> > > >> > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, > >> SyncHandlerThread: > >> > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) > >> (InodeFlushCondVar), reason > >> > 'waiting for the flush flag to commit metadata' > >> > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, > >> > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 > >> (0x7FFFDAC7FE28) > >> > (MsgRecordCondvar), reason 'RPC wait' for > >> allocMsgTypeRelinquishRegion > >> > on node 10.1.52.33 > >> > 0x197EE70 ( 6776) waiting 0.000198354 seconds, > >> > FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 > >> > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for > >> > allocMsgTypeRequestRegion on node 10.1.52.33 > >> > > >> > (10.1.52.33/c0n3271 > >> is the fs manager > >> > for the filesystem in question) > >> > > >> > there's a single process running on this node writing to the > >> filesystem > >> > in question (well, trying to write, it's been blocked doing > >> nothing for > >> > half an hour now). There are ~10 other client nodes in this > >> situation > >> > right now. We had many more last night before the problem > >> seemed to > >> > disappear in the early hours of the morning and now its back. > >> > > >> > Waiters on the fs manager look like this. While the > >> individual waiter is > >> > short it's a near constant stream: > >> > > >> > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg > >> handler > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > >> (0xFFFFC9002163A2E0) > >> > (AllocManagerMutex) > >> > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg > >> handler > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > >> > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg > >> handler > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > >> (0xFFFFC9002163A2E0) > >> > (AllocManagerMutex) > >> > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg > >> handler > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > >> (0xFFFFC9002163A2E0) > >> > (AllocManagerMutex) > >> > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg > >> handler > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > >> > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg > >> handler > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > >> (0xFFFFC9002163A2E0) > >> > (AllocManagerMutex) > >> > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg > >> handler > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > >> (0xFFFFC9002163A2E0) > >> > (AllocManagerMutex) > >> > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg > >> handler > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > >> > > >> > I don't know if this data point is useful but both yesterday > >> and today > >> > the metadata NSDs for this filesystem have had a constant > >> aggregate > >> > stream of 25MB/s 4kop/s reads during each episode (very low > >> latency > >> > though so I don't believe the storage is a bottleneck here). > >> Writes are > >> > only a few hundred ops and didn't strike me as odd. > >> > > >> > I have a PMR open for this but I'm curious if folks have > >> seen this in > >> > the wild and what it might mean. > >> > > >> > -Aaron > >> > > >> > -- > >> > Aaron Knister > >> > NASA Center for Climate Simulation (Code 606.2) > >> > Goddard Space Flight Center > >> > (301) 286-2776 > >> > _______________________________________________ > >> > gpfsug-discuss mailing list > >> > gpfsug-discuss at spectrumscale.org > >> > >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > >> > > >> > > >> > _______________________________________________ > >> > gpfsug-discuss mailing list > >> > gpfsug-discuss at spectrumscale.org > >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > >> > >> -- > >> Aaron Knister > >> NASA Center for Climate Simulation (Code 606.2) > >> Goddard Space Flight Center > >> (301) 286-2776 > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > >> > >> > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Fri Mar 24 17:53:02 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 24 Mar 2017 13:53:02 -0400 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: <29B113F2-BDE9-456A-BBEB-9B1C203CB5F4@nuance.com> References: <29B113F2-BDE9-456A-BBEB-9B1C203CB5F4@nuance.com> Message-ID: Thanks Bob, Jonathan. We're running GPFS 4.1.1.10 and no HSM/LTFSEE. I'm currently gathering, as requested, a snap from all nodes (with traces). With 3500 nodes this ought to be entertaining. -Aaron On 3/24/17 12:50 PM, Oesterlin, Robert wrote: > Hi Aaron > > Yes, I have seen this several times over the last 6 months. I opened at least one PMR on it and they never could track it down. I did some snap dumps but without some traces, they did not have enough. I ended up getting out of it by selectively rebooting some of my NSD servers. My suspicion is that one of them had a thread deadlocked and was holding up IO to the rest of the filesystem. > > I haven?t seen it since I updated to 4.2.2-2, but I?m not convinced (yet) that it?s not lurking in the background. > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > > On 3/24/17, 11:43 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Aaron Knister" wrote: > > Since yesterday morning we've noticed some deadlocks on one of our > filesystems that seem to be triggered by writing to it. The waiters on > the clients look like this: > > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, SyncHandlerThread: > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) (InodeFlushCondVar), reason > 'waiting for the flush flag to commit metadata' > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 (0x7FFFDAC7FE28) > (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRelinquishRegion > on node 10.1.52.33 > 0x197EE70 ( 6776) waiting 0.000198354 seconds, > FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for > allocMsgTypeRequestRegion on node 10.1.52.33 > > (10.1.52.33/c0n3271 is the fs manager for the filesystem in question) > > there's a single process running on this node writing to the filesystem > in question (well, trying to write, it's been blocked doing nothing for > half an hour now). There are ~10 other client nodes in this situation > right now. We had many more last night before the problem seemed to > disappear in the early hours of the morning and now its back. > > Waiters on the fs manager look like this. While the individual waiter is > short it's a near constant stream: > > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > I don't know if this data point is useful but both yesterday and today > the metadata NSDs for this filesystem have had a constant aggregate > stream of 25MB/s 4kop/s reads during each episode (very low latency > though so I don't believe the storage is a bottleneck here). Writes are > only a few hundred ops and didn't strike me as odd. > > I have a PMR open for this but I'm curious if folks have seen this in > the wild and what it might mean. > > -Aaron > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=fUSq1Wdz1p_yJQVOgOoqe7fu7nsPXewvz8BUeKJyiRg&s=9sXk_xPuEtEyJgVhsZ7FCgM-rfytQGDDC2EyqwYgLhQ&e= > > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From aaron.s.knister at nasa.gov Fri Mar 24 17:58:18 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 24 Mar 2017 13:58:18 -0400 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: I feel a little awkward about posting wlists of IP's and hostnames on the mailing list (even though they're all internal) but I'm happy to send to you directly. I've attached both an lsfs and an mmdf output of the fs in question here since that may be useful for others to see. Just a note about disk d23_02_021-- it's been evacuated for several weeks now due to a hardware issue in the disk enclosure. The fs is rather full percentage wise (93%) but in terms of capacity there's a good amount free. 93% full of a 7PB filesystem still leaves 551T. Metadata, as you'll see, is 31% free (roughly 800GB). The fs has 40M inodes allocated and 12M free. -Aaron On 3/24/17 1:41 PM, Sven Oehme wrote: > ok, that seems a different problem then i was thinking. > can you send output of mmlscluster, mmlsconfig, mmlsfs all ? > also are you getting close to fill grade on inodes or capacity on any of > the filesystems ? > > sven > > > On Fri, Mar 24, 2017 at 10:34 AM Aaron Knister > wrote: > > Here's the screenshot from the other node with the high cpu utilization. > > On 3/24/17 1:32 PM, Aaron Knister wrote: > > heh, yep we're on sles :) > > > > here's a screenshot of the fs manager from the deadlocked filesystem. I > > don't think there's an nsd server or manager node that's running full > > throttle across all cpus. There is one that's got relatively high CPU > > utilization though (300-400%). I'll send a screenshot of it in a sec. > > > > no zimon yet but we do have other tools to see cpu utilization. > > > > -Aaron > > > > On 3/24/17 1:22 PM, Sven Oehme wrote: > >> you must be on sles as this segfaults only on sles to my knowledge :-) > >> > >> i am looking for a NSD or manager node in your cluster that runs at 100% > >> cpu usage. > >> > >> do you have zimon deployed to look at cpu utilization across your nodes ? > >> > >> sven > >> > >> > >> > >> On Fri, Mar 24, 2017 at 10:08 AM Aaron Knister > >> >> wrote: > >> > >> Hi Sven, > >> > >> Which NSD server should I run top on, the fs manager? If so the > >> CPU load > >> is about 155%. I'm working on perf top but not off to a great > >> start... > >> > >> # perf top > >> PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% [1000Hz > >> cycles], (all, 28 CPUs) > >> > >> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > >> > >> Segmentation fault > >> > >> -Aaron > >> > >> On 3/24/17 1:04 PM, Sven Oehme wrote: > >> > while this is happening run top and see if there is very high cpu > >> > utilization at this time on the NSD Server. > >> > > >> > if there is , run perf top (you might need to install perf > >> command) and > >> > see if the top cpu contender is a spinlock . if so send a > >> screenshot of > >> > perf top as i may know what that is and how to fix. > >> > > >> > sven > >> > > >> > > >> > On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister > >> > > > >> > > >> >>> wrote: > >> > > >> > Since yesterday morning we've noticed some deadlocks on one > >> of our > >> > filesystems that seem to be triggered by writing to it. The > >> waiters on > >> > the clients look like this: > >> > > >> > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, > >> SyncHandlerThread: > >> > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) > >> (InodeFlushCondVar), reason > >> > 'waiting for the flush flag to commit metadata' > >> > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, > >> > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 > >> (0x7FFFDAC7FE28) > >> > (MsgRecordCondvar), reason 'RPC wait' for > >> allocMsgTypeRelinquishRegion > >> > on node 10.1.52.33 > >> > 0x197EE70 ( 6776) waiting 0.000198354 seconds, > >> > FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 > >> > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for > >> > allocMsgTypeRequestRegion on node 10.1.52.33 > >> > > >> > (10.1.52.33/c0n3271 > > >> is the fs manager > >> > for the filesystem in question) > >> > > >> > there's a single process running on this node writing to the > >> filesystem > >> > in question (well, trying to write, it's been blocked doing > >> nothing for > >> > half an hour now). There are ~10 other client nodes in this > >> situation > >> > right now. We had many more last night before the problem > >> seemed to > >> > disappear in the early hours of the morning and now its back. > >> > > >> > Waiters on the fs manager look like this. While the > >> individual waiter is > >> > short it's a near constant stream: > >> > > >> > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg > >> handler > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > >> (0xFFFFC9002163A2E0) > >> > (AllocManagerMutex) > >> > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg > >> handler > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > >> > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg > >> handler > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > >> (0xFFFFC9002163A2E0) > >> > (AllocManagerMutex) > >> > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg > >> handler > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > >> (0xFFFFC9002163A2E0) > >> > (AllocManagerMutex) > >> > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg > >> handler > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > >> > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg > >> handler > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > >> (0xFFFFC9002163A2E0) > >> > (AllocManagerMutex) > >> > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg > >> handler > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > >> (0xFFFFC9002163A2E0) > >> > (AllocManagerMutex) > >> > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg > >> handler > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > >> > > >> > I don't know if this data point is useful but both yesterday > >> and today > >> > the metadata NSDs for this filesystem have had a constant > >> aggregate > >> > stream of 25MB/s 4kop/s reads during each episode (very low > >> latency > >> > though so I don't believe the storage is a bottleneck here). > >> Writes are > >> > only a few hundred ops and didn't strike me as odd. > >> > > >> > I have a PMR open for this but I'm curious if folks have > >> seen this in > >> > the wild and what it might mean. > >> > > >> > -Aaron > >> > > >> > -- > >> > Aaron Knister > >> > NASA Center for Climate Simulation (Code 606.2) > >> > Goddard Space Flight Center > >> > (301) 286-2776 > > >> > _______________________________________________ > >> > gpfsug-discuss mailing list > >> > gpfsug-discuss at spectrumscale.org > >> > >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > >> > > >> > > >> > _______________________________________________ > >> > gpfsug-discuss mailing list > >> > gpfsug-discuss at spectrumscale.org > >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > >> > >> -- > >> Aaron Knister > >> NASA Center for Climate Simulation (Code 606.2) > >> Goddard Space Flight Center > >> (301) 286-2776 > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > >> > >> > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 -------------- next part -------------- disk disk size failure holds holds free KB free KB name in KB group metadata data in full blocks in fragments --------------- ------------- -------- -------- ----- -------------------- ------------------- Disks in storage pool: system (Maximum disk size allowed is 8.4 TB) m01_12_061 52428800 30 yes no 40660992 ( 78%) 689168 ( 1%) m01_12_062 52428800 30 yes no 40662528 ( 78%) 687168 ( 1%) m01_12_063 52428800 30 yes no 40657920 ( 78%) 694256 ( 1%) m01_12_064 52428800 30 yes no 40654080 ( 78%) 703464 ( 1%) m01_12_096 52428800 30 yes no 15453184 ( 29%) 1138392 ( 2%) m01_12_095 52428800 30 yes no 15514112 ( 30%) 1139072 ( 2%) m01_12_094 52428800 30 yes no 15425536 ( 29%) 1111776 ( 2%) m01_12_093 52428800 30 yes no 15340544 ( 29%) 1126584 ( 2%) m01_12_068 52428800 30 yes no 40716544 ( 78%) 688752 ( 1%) m01_12_067 52428800 30 yes no 40690944 ( 78%) 692200 ( 1%) m01_12_066 52428800 30 yes no 40687104 ( 78%) 692976 ( 1%) m01_12_065 52428800 30 yes no 40674304 ( 78%) 690848 ( 1%) m01_12_081 52428800 30 yes no 137728 ( 0%) 487760 ( 1%) m01_12_082 52428800 30 yes no 60416 ( 0%) 512632 ( 1%) m01_12_083 52428800 30 yes no 102144 ( 0%) 674152 ( 1%) m01_12_084 52428800 30 yes no 126208 ( 0%) 684296 ( 1%) m01_12_085 52428800 30 yes no 117504 ( 0%) 705952 ( 1%) m01_12_086 52428800 30 yes no 119296 ( 0%) 691056 ( 1%) m01_12_087 52428800 30 yes no 57344 ( 0%) 493992 ( 1%) m01_12_088 52428800 30 yes no 60672 ( 0%) 547360 ( 1%) m01_12_089 52428800 30 yes no 1455616 ( 3%) 888688 ( 2%) m01_12_090 52428800 30 yes no 1467392 ( 3%) 919312 ( 2%) m01_12_091 52428800 30 yes no 190464 ( 0%) 745456 ( 1%) m01_12_092 52428800 30 yes no 1367296 ( 3%) 771400 ( 1%) m02_22_081 52428800 40 yes no 1245696 ( 2%) 855992 ( 2%) m02_22_082 52428800 40 yes no 1261056 ( 2%) 869336 ( 2%) m02_22_083 52428800 40 yes no 1254912 ( 2%) 865656 ( 2%) m02_22_084 52428800 40 yes no 62464 ( 0%) 698480 ( 1%) m02_22_085 52428800 40 yes no 64256 ( 0%) 703016 ( 1%) m02_22_086 52428800 40 yes no 62208 ( 0%) 690032 ( 1%) m02_22_087 52428800 40 yes no 62464 ( 0%) 687584 ( 1%) m02_22_088 52428800 40 yes no 68608 ( 0%) 699848 ( 1%) m02_22_089 52428800 40 yes no 84480 ( 0%) 698144 ( 1%) m02_22_090 52428800 40 yes no 85248 ( 0%) 720216 ( 1%) m02_22_091 52428800 40 yes no 98816 ( 0%) 711824 ( 1%) m02_22_092 52428800 40 yes no 104448 ( 0%) 732808 ( 1%) m02_22_068 52428800 40 yes no 40727552 ( 78%) 702472 ( 1%) m02_22_067 52428800 40 yes no 40713728 ( 78%) 688576 ( 1%) m02_22_066 52428800 40 yes no 40694272 ( 78%) 700960 ( 1%) m02_22_065 52428800 40 yes no 40694016 ( 78%) 689936 ( 1%) m02_22_064 52428800 40 yes no 40683264 ( 78%) 695144 ( 1%) m02_22_063 52428800 40 yes no 40676864 ( 78%) 701288 ( 1%) m02_22_062 52428800 40 yes no 40670976 ( 78%) 692984 ( 1%) m02_22_061 52428800 40 yes no 40672512 ( 78%) 690024 ( 1%) m02_22_096 52428800 40 yes no 15327232 ( 29%) 1149064 ( 2%) m02_22_095 52428800 40 yes no 15363584 ( 29%) 1146384 ( 2%) m02_22_094 52428800 40 yes no 15397376 ( 29%) 1172856 ( 2%) m02_22_093 52428800 40 yes no 15374336 ( 29%) 1163832 ( 2%) ------------- -------------------- ------------------- (pool total) 2516582400 783850240 ( 31%) 37303168 ( 1%) Disks in storage pool: sp_1620 (Maximum disk size allowed is 5.4 PB) d23_01_001 46028292096 1620 no yes 3541176320 ( 8%) 37724768 ( 0%) d23_01_002 46028292096 1620 no yes 3542331392 ( 8%) 37545024 ( 0%) d23_01_003 46028292096 1620 no yes 3541968896 ( 8%) 37765344 ( 0%) d23_01_004 46028292096 1620 no yes 3544687616 ( 8%) 37720576 ( 0%) d23_01_005 46028292096 1620 no yes 3543368704 ( 8%) 37647456 ( 0%) d23_01_006 46028292096 1620 no yes 3542778880 ( 8%) 37695232 ( 0%) d23_01_007 46028292096 1620 no yes 3543220224 ( 8%) 37539712 ( 0%) d23_01_008 46028292096 1620 no yes 3540293632 ( 8%) 37548928 ( 0%) d23_01_009 46028292096 1620 no yes 3544590336 ( 8%) 37547424 ( 0%) d23_01_010 46028292096 1620 no yes 3542993920 ( 8%) 37865728 ( 0%) d23_01_011 46028292096 1620 no yes 3542859776 ( 8%) 37889408 ( 0%) d23_01_012 46028292096 1620 no yes 3542452224 ( 8%) 37721440 ( 0%) d23_01_013 46028292096 1620 no yes 3542286336 ( 8%) 37797824 ( 0%) d23_01_014 46028292096 1620 no yes 3543352320 ( 8%) 37647456 ( 0%) d23_01_015 46028292096 1620 no yes 3542906880 ( 8%) 37776960 ( 0%) d23_01_016 46028292096 1620 no yes 3540386816 ( 8%) 37521632 ( 0%) d23_01_017 46028292096 1620 no yes 3543212032 ( 8%) 37568480 ( 0%) d23_01_018 46028292096 1620 no yes 3542416384 ( 8%) 37467648 ( 0%) d23_01_019 46028292096 1620 no yes 3542659072 ( 8%) 37865344 ( 0%) d23_01_020 46028292096 1620 no yes 3542518784 ( 8%) 37623840 ( 0%) d23_01_021 46028292096 1620 no yes 3543202816 ( 8%) 37630848 ( 0%) d23_01_022 46028292096 1620 no yes 3544535040 ( 8%) 37723968 ( 0%) d23_01_023 46028292096 1620 no yes 3543248896 ( 8%) 37542656 ( 0%) d23_01_024 46028292096 1620 no yes 3541811200 ( 8%) 37775360 ( 0%) d23_01_025 46028292096 1620 no yes 3544839168 ( 8%) 37887744 ( 0%) d23_01_026 46028292096 1620 no yes 3542474752 ( 8%) 37820672 ( 0%) d23_01_027 46028292096 1620 no yes 3542050816 ( 8%) 37847296 ( 0%) d23_01_028 46028292096 1620 no yes 3540822016 ( 8%) 37578400 ( 0%) d23_01_029 46028292096 1620 no yes 3542011904 ( 8%) 37423328 ( 0%) d23_01_030 46028292096 1620 no yes 3542572032 ( 8%) 37751840 ( 0%) d23_01_031 46028292096 1620 no yes 3541582848 ( 8%) 37648896 ( 0%) d23_01_032 46028292096 1620 no yes 3542650880 ( 8%) 37715840 ( 0%) d23_01_033 46028292096 1620 no yes 3542251520 ( 8%) 37598432 ( 0%) d23_01_034 46028292096 1620 no yes 3542195200 ( 8%) 37582944 ( 0%) d23_01_035 46028292096 1620 no yes 3541298176 ( 8%) 37694848 ( 0%) d23_01_036 46028292096 1620 no yes 3541215232 ( 8%) 37869568 ( 0%) d23_01_037 46028292096 1620 no yes 3542111232 ( 8%) 37601088 ( 0%) d23_01_038 46028292096 1620 no yes 3541210112 ( 8%) 37474400 ( 0%) d23_01_039 46028292096 1620 no yes 3540457472 ( 8%) 37654656 ( 0%) d23_01_040 46028292096 1620 no yes 3541776384 ( 8%) 37645760 ( 0%) d23_01_041 46028292096 1620 no yes 3542624256 ( 8%) 37798880 ( 0%) d23_01_042 46028292096 1620 no yes 3541653504 ( 8%) 37595936 ( 0%) d23_01_043 46028292096 1620 no yes 3540583424 ( 8%) 37751936 ( 0%) d23_01_044 46028292096 1620 no yes 3542136832 ( 8%) 37793536 ( 0%) d23_01_045 46028292096 1620 no yes 3543443456 ( 8%) 37683872 ( 0%) d23_01_046 46028292096 1620 no yes 3540705280 ( 8%) 37896096 ( 0%) d23_01_047 46028292096 1620 no yes 3541550080 ( 8%) 37577760 ( 0%) d23_01_048 46028292096 1620 no yes 3542068224 ( 8%) 37724960 ( 0%) d23_01_049 46028292096 1620 no yes 3544568832 ( 8%) 37687264 ( 0%) d23_01_050 46028292096 1620 no yes 3543891968 ( 8%) 37737824 ( 0%) d23_01_051 46028292096 1620 no yes 3541944320 ( 8%) 37787904 ( 0%) d23_01_052 46028292096 1620 no yes 3542128640 ( 8%) 37960704 ( 0%) d23_01_053 46028292096 1620 no yes 3542494208 ( 8%) 37823104 ( 0%) d23_01_054 46028292096 1620 no yes 3541776384 ( 8%) 37652064 ( 0%) d23_01_055 46028292096 1620 no yes 3543655424 ( 8%) 37802656 ( 0%) d23_01_056 46028292096 1620 no yes 3541664768 ( 8%) 37694272 ( 0%) d23_01_057 46028292096 1620 no yes 3542197248 ( 8%) 37798272 ( 0%) d23_01_058 46028292096 1620 no yes 3543078912 ( 8%) 37740448 ( 0%) d23_01_059 46028292096 1620 no yes 3544783872 ( 8%) 37741248 ( 0%) d23_01_060 46028292096 1620 no yes 3542276096 ( 8%) 37818304 ( 0%) d23_01_061 46028292096 1620 no yes 3543452672 ( 8%) 37727104 ( 0%) d23_01_062 46028292096 1620 no yes 3543225344 ( 8%) 37754720 ( 0%) d23_01_063 46028292096 1620 no yes 3543173120 ( 8%) 37685280 ( 0%) d23_01_064 46028292096 1620 no yes 3541703680 ( 8%) 37711424 ( 0%) d23_01_065 46028292096 1620 no yes 3541797888 ( 8%) 37836992 ( 0%) d23_01_066 46028292096 1620 no yes 3542709248 ( 8%) 37780864 ( 0%) d23_01_067 46028292096 1620 no yes 3542996992 ( 8%) 37798976 ( 0%) d23_01_068 46028292096 1620 no yes 3542989824 ( 8%) 37672352 ( 0%) d23_01_069 46028292096 1620 no yes 3542004736 ( 8%) 37688608 ( 0%) d23_01_070 46028292096 1620 no yes 3541458944 ( 8%) 37648320 ( 0%) d23_01_071 46028292096 1620 no yes 3542049792 ( 8%) 37874368 ( 0%) d23_01_072 46028292096 1620 no yes 3541520384 ( 8%) 37650368 ( 0%) d23_01_073 46028292096 1620 no yes 3542274048 ( 8%) 37759776 ( 0%) d23_01_074 46028292096 1620 no yes 3541511168 ( 8%) 37569472 ( 0%) d23_01_075 46028292096 1620 no yes 3544001536 ( 8%) 37685952 ( 0%) d23_01_076 46028292096 1620 no yes 3543203840 ( 8%) 37690880 ( 0%) d23_01_077 46028292096 1620 no yes 3541925888 ( 8%) 37710848 ( 0%) d23_01_078 46028292096 1620 no yes 3543930880 ( 8%) 37588672 ( 0%) d23_01_079 46028292096 1620 no yes 3541520384 ( 8%) 37626432 ( 0%) d23_01_080 46028292096 1620 no yes 3541615616 ( 8%) 37796576 ( 0%) d23_01_081 46028292096 1620 no yes 3542212608 ( 8%) 37773056 ( 0%) d23_01_082 46028292096 1620 no yes 3541496832 ( 8%) 37863200 ( 0%) d23_01_083 46028292096 1620 no yes 3541881856 ( 8%) 37822016 ( 0%) d23_01_084 46028292096 1620 no yes 3543436288 ( 8%) 37838144 ( 0%) d23_02_001 46028292096 1620 no yes 3543580672 ( 8%) 37784480 ( 0%) d23_02_002 46028292096 1620 no yes 3541958656 ( 8%) 38029312 ( 0%) d23_02_003 46028292096 1620 no yes 3542037504 ( 8%) 37781888 ( 0%) d23_02_004 46028292096 1620 no yes 3541141504 ( 8%) 37535936 ( 0%) d23_02_005 46028292096 1620 no yes 3541710848 ( 8%) 37585504 ( 0%) d23_02_006 46028292096 1620 no yes 3542758400 ( 8%) 37699968 ( 0%) d23_02_007 46028292096 1620 no yes 3541051392 ( 8%) 37609824 ( 0%) d23_02_008 46028292096 1620 no yes 3541925888 ( 8%) 37791872 ( 0%) d23_02_009 46028292096 1620 no yes 3542461440 ( 8%) 37854464 ( 0%) d23_02_010 46028292096 1620 no yes 3544000512 ( 8%) 37642048 ( 0%) d23_02_011 46028292096 1620 no yes 3542000640 ( 8%) 37811840 ( 0%) d23_02_012 46028292096 1620 no yes 3543025664 ( 8%) 37802784 ( 0%) d23_02_013 46028292096 1620 no yes 3541744640 ( 8%) 37776608 ( 0%) d23_02_014 46028292096 1620 no yes 3542261760 ( 8%) 37699648 ( 0%) d23_02_015 46028292096 1620 no yes 3542729728 ( 8%) 37690944 ( 0%) d23_02_016 46028292096 1620 no yes 3543721984 ( 8%) 37657472 ( 0%) d23_02_017 46028292096 1620 no yes 3540802560 ( 8%) 37531328 ( 0%) d23_02_018 46028292096 1620 no yes 3542657024 ( 8%) 37860768 ( 0%) d23_02_019 46028292096 1620 no yes 3543438336 ( 8%) 37573760 ( 0%) d23_02_020 46028292096 1620 no yes 3543243776 ( 8%) 37662976 ( 0%) d23_02_021 46028292096 1620 no yes 46028197888 (100%) 32576 ( 0%) * d23_02_022 46028292096 1620 no yes 3543544832 ( 8%) 37620160 ( 0%) d23_02_023 46028292096 1620 no yes 3540716544 ( 8%) 37649536 ( 0%) d23_02_024 46028292096 1620 no yes 3542181888 ( 8%) 37553760 ( 0%) d23_02_025 46028292096 1620 no yes 3540486144 ( 8%) 37529376 ( 0%) d23_02_026 46028292096 1620 no yes 3541583872 ( 8%) 37833792 ( 0%) d23_02_027 46028292096 1620 no yes 3542169600 ( 8%) 37778912 ( 0%) d23_02_028 46028292096 1620 no yes 3541048320 ( 8%) 37696864 ( 0%) d23_02_029 46028292096 1620 no yes 3542336512 ( 8%) 37859264 ( 0%) d23_02_030 46028292096 1620 no yes 3542102016 ( 8%) 38059168 ( 0%) d23_02_031 46028292096 1620 no yes 3541311488 ( 8%) 37784480 ( 0%) d23_02_032 46028292096 1620 no yes 3542036480 ( 8%) 37783456 ( 0%) d23_02_033 46028292096 1620 no yes 3541478400 ( 8%) 37753792 ( 0%) d23_02_034 46028292096 1620 no yes 3540772864 ( 8%) 37725312 ( 0%) d23_02_035 46028292096 1620 no yes 3541840896 ( 8%) 37709664 ( 0%) d23_02_036 46028292096 1620 no yes 3542415360 ( 8%) 37580448 ( 0%) d23_02_037 46028292096 1620 no yes 3542515712 ( 8%) 37587808 ( 0%) d23_02_038 46028292096 1620 no yes 3541250048 ( 8%) 37550976 ( 0%) d23_02_039 46028292096 1620 no yes 3542627328 ( 8%) 37389952 ( 0%) d23_02_040 46028292096 1620 no yes 3541750784 ( 8%) 37709216 ( 0%) d23_02_041 46028292096 1620 no yes 3542558720 ( 8%) 37760288 ( 0%) d23_02_042 46028292096 1620 no yes 3540994048 ( 8%) 37491680 ( 0%) d23_02_043 46028292096 1620 no yes 3542491136 ( 8%) 37768576 ( 0%) d23_02_044 46028292096 1620 no yes 3542805504 ( 8%) 37638144 ( 0%) d23_02_045 46028292096 1620 no yes 3540658176 ( 8%) 37613120 ( 0%) d23_02_046 46028292096 1620 no yes 3543098368 ( 8%) 37920320 ( 0%) d23_02_047 46028292096 1620 no yes 3543087104 ( 8%) 37590432 ( 0%) d23_02_048 46028292096 1620 no yes 3541494784 ( 8%) 37468224 ( 0%) d23_02_049 46028292096 1620 no yes 3541920768 ( 8%) 37675968 ( 0%) d23_02_050 46028292096 1620 no yes 3542463488 ( 8%) 37670816 ( 0%) d23_02_051 46028292096 1620 no yes 3542427648 ( 8%) 37678176 ( 0%) d23_02_052 46028292096 1620 no yes 3539824640 ( 8%) 37590176 ( 0%) d23_02_053 46028292096 1620 no yes 3542251520 ( 8%) 37835200 ( 0%) d23_02_054 46028292096 1620 no yes 3541064704 ( 8%) 37636224 ( 0%) d23_02_055 46028292096 1620 no yes 3540130816 ( 8%) 37703360 ( 0%) d23_02_056 46028292096 1620 no yes 3545320448 ( 8%) 37767712 ( 0%) d23_02_057 46028292096 1620 no yes 3543144448 ( 8%) 37658208 ( 0%) d23_02_058 46028292096 1620 no yes 3541233664 ( 8%) 37720640 ( 0%) d23_02_059 46028292096 1620 no yes 3541435392 ( 8%) 37680896 ( 0%) d23_02_060 46028292096 1620 no yes 3542780928 ( 8%) 37567360 ( 0%) d23_02_061 46028292096 1620 no yes 3542073344 ( 8%) 37420992 ( 0%) d23_02_062 46028292096 1620 no yes 3543192576 ( 8%) 37575232 ( 0%) d23_02_063 46028292096 1620 no yes 3542048768 ( 8%) 37696192 ( 0%) d23_02_064 46028292096 1620 no yes 3541509120 ( 8%) 37536224 ( 0%) d23_02_065 46028292096 1620 no yes 3542299648 ( 8%) 37648736 ( 0%) d23_02_066 46028292096 1620 no yes 3541864448 ( 8%) 37710976 ( 0%) d23_02_067 46028292096 1620 no yes 3538870272 ( 8%) 37532288 ( 0%) d23_02_068 46028292096 1620 no yes 3541297152 ( 8%) 37829184 ( 0%) d23_02_069 46028292096 1620 no yes 3542179840 ( 8%) 37583424 ( 0%) d23_02_070 46028292096 1620 no yes 3541616640 ( 8%) 37742080 ( 0%) d23_02_071 46028292096 1620 no yes 3541706752 ( 8%) 37687328 ( 0%) d23_02_072 46028292096 1620 no yes 3542240256 ( 8%) 37582112 ( 0%) d23_02_073 46028292096 1620 no yes 3542102016 ( 8%) 37752736 ( 0%) d23_02_074 46028292096 1620 no yes 3542754304 ( 8%) 37687456 ( 0%) d23_02_075 46028292096 1620 no yes 3542194176 ( 8%) 37729344 ( 0%) d23_02_076 46028292096 1620 no yes 3542798336 ( 8%) 37672096 ( 0%) d23_02_077 46028292096 1620 no yes 3543918592 ( 8%) 37709024 ( 0%) d23_02_078 46028292096 1620 no yes 3540992000 ( 8%) 37540992 ( 0%) d23_02_079 46028292096 1620 no yes 3543330816 ( 8%) 37573888 ( 0%) d23_02_080 46028292096 1620 no yes 3543735296 ( 8%) 37818208 ( 0%) d23_02_081 46028292096 1620 no yes 3542054912 ( 8%) 37698016 ( 0%) d23_02_082 46028292096 1620 no yes 3540747264 ( 8%) 37643776 ( 0%) d23_02_083 46028292096 1620 no yes 3542846464 ( 8%) 37715680 ( 0%) d23_02_084 46028292096 1620 no yes 3542061056 ( 8%) 37703904 ( 0%) ------------- -------------------- ------------------- (pool total) 7686724780032 591558139904 ( 8%) 6295334976 ( 0%) ============= ==================== =================== (data) 7686724780032 591558139904 ( 8%) 6295334976 ( 0%) (metadata) 2516582400 783850240 ( 31%) 37303168 ( 1%) ============= ==================== =================== (total) 7689241362432 592341990144 ( 8%) 6332638144 ( 0%) Inode Information ----------------- Number of used inodes: 30247288 Number of free inodes: 11695752 Number of allocated inodes: 41943040 Maximum number of inodes: 41943040 -------------- next part -------------- flag value description ------------------- ------------------------ ----------------------------------- -f 8192 Minimum fragment size in bytes (system pool) 32768 Minimum fragment size in bytes (other pools) -i 512 Inode size in bytes -I 32768 Indirect block size in bytes -m 2 Default number of metadata replicas -M 2 Maximum number of metadata replicas -r 1 Default number of data replicas -R 2 Maximum number of data replicas -j cluster Block allocation type -D nfs4 File locking semantics in effect -k posix ACL semantics in effect -n 5000 Estimated number of nodes that will mount file system -B 262144 Block size (system pool) 1048576 Block size (other pools) -Q user;group;fileset Quotas accounting enabled user;group;fileset Quotas enforced none Default quotas enabled --perfileset-quota no Per-fileset quota enforcement --filesetdf no Fileset df enabled? -V 13.23 (3.5.0.7) File system version --create-time Fri Dec 6 15:23:28 2013 File system creation time -z no Is DMAPI enabled? -L 8388608 Logfile size -E yes Exact mtime mount option -S no Suppress atime mount option -K whenpossible Strict replica allocation option --fastea yes Fast external attributes enabled? --encryption no Encryption enabled? --inode-limit 41943040 Maximum number of inodes --log-replicas 0 Number of log replicas --is4KAligned no is4KAligned? --rapid-repair no rapidRepair enabled? --write-cache-threshold 0 HAWC Threshold (max 65536) -P system;sp_1620 Disk storage pools in file system -d m01_12_093;m01_12_094;m01_12_095;m01_12_096;m02_22_093;m02_22_094;m02_22_095;m02_22_096;m01_12_061;m01_12_062;m01_12_063;m01_12_064;m01_12_065;m01_12_066;m01_12_067; -d m01_12_068;m02_22_061;m02_22_062;m02_22_063;m02_22_064;m02_22_065;m02_22_066;m02_22_067;m02_22_068;m01_12_081;m01_12_082;m01_12_083;m01_12_084;m01_12_085;m01_12_086; -d m01_12_087;m01_12_088;m01_12_089;m01_12_090;m01_12_091;m01_12_092;m02_22_081;m02_22_082;m02_22_083;m02_22_084;m02_22_085;m02_22_086;m02_22_087;m02_22_088;m02_22_089; -d m02_22_090;m02_22_091;m02_22_092;d23_01_001;d23_01_002;d23_01_003;d23_01_004;d23_01_005;d23_01_006;d23_01_007;d23_01_008;d23_01_009;d23_01_010;d23_01_011;d23_01_012; -d d23_01_013;d23_01_014;d23_01_015;d23_01_016;d23_01_017;d23_01_018;d23_01_019;d23_01_020;d23_01_021;d23_01_022;d23_01_023;d23_01_024;d23_01_025;d23_01_026;d23_01_027; -d d23_01_028;d23_01_029;d23_01_030;d23_01_031;d23_01_032;d23_01_033;d23_01_034;d23_01_035;d23_01_036;d23_01_037;d23_01_038;d23_01_039;d23_01_040;d23_01_041;d23_01_042; -d d23_01_043;d23_01_044;d23_01_045;d23_01_046;d23_01_047;d23_01_048;d23_01_049;d23_01_050;d23_01_051;d23_01_052;d23_01_053;d23_01_054;d23_01_055;d23_01_056;d23_01_057; -d d23_01_058;d23_01_059;d23_01_060;d23_01_061;d23_01_062;d23_01_063;d23_01_064;d23_01_065;d23_01_066;d23_01_067;d23_01_068;d23_01_069;d23_01_070;d23_01_071;d23_01_072; -d d23_01_073;d23_01_074;d23_01_075;d23_01_076;d23_01_077;d23_01_078;d23_01_079;d23_01_080;d23_01_081;d23_01_082;d23_01_083;d23_01_084;d23_02_001;d23_02_002;d23_02_003; -d d23_02_004;d23_02_005;d23_02_006;d23_02_007;d23_02_008;d23_02_009;d23_02_010;d23_02_011;d23_02_012;d23_02_013;d23_02_014;d23_02_015;d23_02_016;d23_02_017;d23_02_018; -d d23_02_019;d23_02_020;d23_02_021;d23_02_022;d23_02_023;d23_02_024;d23_02_025;d23_02_026;d23_02_027;d23_02_028;d23_02_029;d23_02_030;d23_02_031;d23_02_032;d23_02_033; -d d23_02_034;d23_02_035;d23_02_036;d23_02_037;d23_02_038;d23_02_039;d23_02_040;d23_02_041;d23_02_042;d23_02_043;d23_02_044;d23_02_045;d23_02_046;d23_02_047;d23_02_048; -d d23_02_049;d23_02_050;d23_02_051;d23_02_052;d23_02_053;d23_02_054;d23_02_055;d23_02_056;d23_02_057;d23_02_058;d23_02_059;d23_02_060;d23_02_061;d23_02_062;d23_02_063; -d d23_02_064;d23_02_065;d23_02_066;d23_02_067;d23_02_068;d23_02_069;d23_02_070;d23_02_071;d23_02_072;d23_02_073;d23_02_074;d23_02_075;d23_02_076;d23_02_077;d23_02_078; -d d23_02_079;d23_02_080;d23_02_081;d23_02_082;d23_02_083;d23_02_084 Disks in file system -A no Automatic mount option -o nodev,nosuid Additional mount options -T /gpfsm/dnb03 Default mount point --mount-priority 0 Mount priority From jfosburg at mdanderson.org Fri Mar 24 18:03:22 2017 From: jfosburg at mdanderson.org (Fosburgh,Jonathan) Date: Fri, 24 Mar 2017 18:03:22 +0000 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: <1490378600.1901.4.camel@mdanderson.org> 7PB filesystem and only 28 million inodes in use? What is your average file size? Our large filesystem is 7.5P (currently 71% used) with over 1 billion inodes in use. -- Jonathan Fosburgh Principal Application Systems Analyst Storage Team IT Operations jfosburg at mdanderson.org (713) 745-9346 -----Original Message----- Date: Fri, 24 Mar 2017 13:58:18 -0400 Subject: Re: [gpfsug-discuss] strange waiters + filesystem deadlock To: gpfsug-discuss at spectrumscale.org Reply-to: gpfsug main discussion list From: Aaron Knister > I feel a little awkward about posting wlists of IP's and hostnames on the mailing list (even though they're all internal) but I'm happy to send to you directly. I've attached both an lsfs and an mmdf output of the fs in question here since that may be useful for others to see. Just a note about disk d23_02_021-- it's been evacuated for several weeks now due to a hardware issue in the disk enclosure. The fs is rather full percentage wise (93%) but in terms of capacity there's a good amount free. 93% full of a 7PB filesystem still leaves 551T. Metadata, as you'll see, is 31% free (roughly 800GB). The fs has 40M inodes allocated and 12M free. -Aaron On 3/24/17 1:41 PM, Sven Oehme wrote: ok, that seems a different problem then i was thinking. can you send output of mmlscluster, mmlsconfig, mmlsfs all ? also are you getting close to fill grade on inodes or capacity on any of the filesystems ? sven On Fri, Mar 24, 2017 at 10:34 AM Aaron Knister > wrote: Here's the screenshot from the other node with the high cpu utilization. On 3/24/17 1:32 PM, Aaron Knister wrote: > heh, yep we're on sles :) > > here's a screenshot of the fs manager from the deadlocked filesystem. I > don't think there's an nsd server or manager node that's running full > throttle across all cpus. There is one that's got relatively high CPU > utilization though (300-400%). I'll send a screenshot of it in a sec. > > no zimon yet but we do have other tools to see cpu utilization. > > -Aaron > > On 3/24/17 1:22 PM, Sven Oehme wrote: >> you must be on sles as this segfaults only on sles to my knowledge :-) >> >> i am looking for a NSD or manager node in your cluster that runs at 100% >> cpu usage. >> >> do you have zimon deployed to look at cpu utilization across your nodes ? >> >> sven >> >> >> >> On Fri, Mar 24, 2017 at 10:08 AM Aaron Knister >> >> wrote: >> >> Hi Sven, >> >> Which NSD server should I run top on, the fs manager? If so the >> CPU load >> is about 155%. I'm working on perf top but not off to a great >> start... >> >> # perf top >> PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% [1000Hz >> cycles], (all, 28 CPUs) >> >> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >> >> Segmentation fault >> >> -Aaron >> >> On 3/24/17 1:04 PM, Sven Oehme wrote: >> > while this is happening run top and see if there is very high cpu >> > utilization at this time on the NSD Server. >> > >> > if there is , run perf top (you might need to install perf >> command) and >> > see if the top cpu contender is a spinlock . if so send a >> screenshot of >> > perf top as i may know what that is and how to fix. >> > >> > sven >> > >> > >> > On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister >> > >> > >> >>> wrote: >> > >> > Since yesterday morning we've noticed some deadlocks on one >> of our >> > filesystems that seem to be triggered by writing to it. The >> waiters on >> > the clients look like this: >> > >> > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, >> SyncHandlerThread: >> > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) >> (InodeFlushCondVar), reason >> > 'waiting for the flush flag to commit metadata' >> > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, >> > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 >> (0x7FFFDAC7FE28) >> > (MsgRecordCondvar), reason 'RPC wait' for >> allocMsgTypeRelinquishRegion >> > on node 10.1.52.33 >> > 0x197EE70 ( 6776) waiting 0.000198354 seconds, >> > FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 >> > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for >> > allocMsgTypeRequestRegion on node 10.1.52.33 >> > >> > (10.1.52.33/c0n3271 >> is the fs manager >> > for the filesystem in question) >> > >> > there's a single process running on this node writing to the >> filesystem >> > in question (well, trying to write, it's been blocked doing >> nothing for >> > half an hour now). There are ~10 other client nodes in this >> situation >> > right now. We had many more last night before the problem >> seemed to >> > disappear in the early hours of the morning and now its back. >> > >> > Waiters on the fs manager look like this. While the >> individual waiter is >> > short it's a near constant stream: >> > >> > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg >> handler >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) >> > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg >> handler >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) >> > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg >> handler >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) >> > >> > I don't know if this data point is useful but both yesterday >> and today >> > the metadata NSDs for this filesystem have had a constant >> aggregate >> > stream of 25MB/s 4kop/s reads during each episode (very low >> latency >> > though so I don't believe the storage is a bottleneck here). >> Writes are >> > only a few hundred ops and didn't strike me as odd. >> > >> > I have a PMR open for this but I'm curious if folks have >> seen this in >> > the wild and what it might mean. >> > >> > -Aaron >> > >> > -- >> > Aaron Knister >> > NASA Center for Climate Simulation (Code 606.2) >> > Goddard Space Flight Center >> > (301) 286-2776 >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > >> > >> > >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > >> >> -- >> Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) >> Goddard Space Flight Center >> (301) 286-2776 >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss The information contained in this e-mail message may be privileged, confidential, and/or protected from disclosure. This e-mail message may contain protected health information (PHI); dissemination of PHI should comply with applicable federal and state laws. If you are not the intended recipient, or an authorized representative of the intended recipient, any further review, disclosure, use, dissemination, distribution, or copying of this message or any attachment (or the information contained therein) is strictly prohibited. If you think that you have received this e-mail message in error, please notify the sender by return e-mail and delete all references to it and its contents from your systems. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Fri Mar 24 18:05:26 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 24 Mar 2017 14:05:26 -0400 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: <1490378600.1901.4.camel@mdanderson.org> References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> <1490378600.1901.4.camel@mdanderson.org> Message-ID: <5e630e9a-5f6f-5def-7e92-5bc7c6c01a4f@nasa.gov> It's large, I do know that much. I'll defer to one of our other storage admins. Jordan, do you have that number handy? -Aaron On 3/24/17 2:03 PM, Fosburgh,Jonathan wrote: > 7PB filesystem and only 28 million inodes in use? What is your average > file size? Our large filesystem is 7.5P (currently 71% used) with over 1 > billion inodes in use. > > -- > > Jonathan Fosburgh > Principal Application Systems Analyst > Storage Team > IT Operations > jfosburg at mdanderson.org > (713) 745-9346 > > -----Original Message----- > > *Date*: Fri, 24 Mar 2017 13:58:18 -0400 > *Subject*: Re: [gpfsug-discuss] strange waiters + filesystem deadlock > *To*: gpfsug-discuss at spectrumscale.org > > Reply-to: gpfsug main discussion list > *From*: Aaron Knister > > > I feel a little awkward about posting wlists of IP's and hostnames on > the mailing list (even though they're all internal) but I'm happy to > send to you directly. I've attached both an lsfs and an mmdf output of > the fs in question here since that may be useful for others to see. Just > a note about disk d23_02_021-- it's been evacuated for several weeks now > due to a hardware issue in the disk enclosure. > > The fs is rather full percentage wise (93%) but in terms of capacity > there's a good amount free. 93% full of a 7PB filesystem still leaves > 551T. Metadata, as you'll see, is 31% free (roughly 800GB). > > The fs has 40M inodes allocated and 12M free. > > -Aaron > > On 3/24/17 1:41 PM, Sven Oehme wrote: >> ok, that seems a different problem then i was thinking. can you send >> output of mmlscluster, mmlsconfig, mmlsfs all ? also are you getting >> close to fill grade on inodes or capacity on any of the filesystems ? >> sven On Fri, Mar 24, 2017 at 10:34 AM Aaron Knister >> >> > wrote: Here's the screenshot from >> the other node with the high cpu utilization. On 3/24/17 1:32 PM, >> Aaron Knister wrote: > heh, yep we're on sles :) > > here's a >> screenshot of the fs manager from the deadlocked filesystem. I > don't >> think there's an nsd server or manager node that's running full > >> throttle across all cpus. There is one that's got relatively high CPU >> > utilization though (300-400%). I'll send a screenshot of it in a >> sec. > > no zimon yet but we do have other tools to see cpu >> utilization. > > -Aaron > > On 3/24/17 1:22 PM, Sven Oehme wrote: >> >> you must be on sles as this segfaults only on sles to my knowledge :-) >> >> >> i am looking for a NSD or manager node in your cluster that runs >> at 100% >> cpu usage. >> >> do you have zimon deployed to look at cpu >> utilization across your nodes ? >> >> sven >> >> >> >> On Fri, Mar 24, >> 2017 at 10:08 AM Aaron Knister > >> >> >> >> wrote: >> >> Hi Sven, >> >> Which NSD server should I run top on, the >> fs manager? If so the >> CPU load >> is about 155%. I'm working on >> perf top but not off to a great >> start... >> >> # perf top >> >> PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% [1000Hz >> cycles], >> (all, 28 CPUs) >> >> >> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >> >> >> Segmentation fault >> >> -Aaron >> >> On 3/24/17 1:04 PM, Sven >> Oehme wrote: >> > while this is happening run top and see if there is >> very high cpu >> > utilization at this time on the NSD Server. >> > >> >> > if there is , run perf top (you might need to install perf >> >> command) and >> > see if the top cpu contender is a spinlock . if so >> send a >> screenshot of >> > perf top as i may know what that is and >> how to fix. >> > >> > sven >> > >> > >> > On Fri, Mar 24, 2017 at 9:43 >> AM Aaron Knister >> > >> > >> >> > >> >> > >>> wrote: >> > >> > Since yesterday >> morning we've noticed some deadlocks on one >> of our >> > filesystems >> that seem to be triggered by writing to it. The >> waiters on >> > the >> clients look like this: >> > >> > 0x19450B0 ( 6730) waiting >> 2063.294589599 seconds, >> SyncHandlerThread: >> > on ThCond >> 0x1802585CB10 (0xFFFFC9002585CB10) >> (InodeFlushCondVar), reason >> > >> 'waiting for the flush flag to commit metadata' >> > 0x7FFFDA65E200 ( >> 22850) waiting 0.000246257 seconds, >> > AllocReduceHelperThread: on >> ThCond 0x7FFFDAC7FE28 >> (0x7FFFDAC7FE28) >> > (MsgRecordCondvar), >> reason 'RPC wait' for >> allocMsgTypeRelinquishRegion >> > on node >> 10.1.52.33 >> > 0x197EE70 ( 6776) waiting 0.000198354 >> seconds, >> > FileBlockWriteFetchHandlerThread: on ThCond >> 0x7FFFF00CD598 >> > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC >> wait' for >> > allocMsgTypeRequestRegion on node 10.1.52.33 >> >> > >> > (10.1.52.33/c0n3271 >> >> is the fs >> manager >> > for the filesystem in question) >> > >> > there's a >> single process running on this node writing to the >> filesystem >> > >> in question (well, trying to write, it's been blocked doing >> nothing >> for >> > half an hour now). There are ~10 other client nodes in this >> >> situation >> > right now. We had many more last night before the >> problem >> seemed to >> > disappear in the early hours of the morning >> and now its back. >> > >> > Waiters on the fs manager look like this. >> While the >> individual waiter is >> > short it's a near constant >> stream: >> > >> > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, >> Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex >> 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > >> 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg >> handler >> >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 >> > >> (0xFFFFC9002163A2E0) (AllocManagerMutex) >> > 0x7FFF91C10080 ( 14723) >> waiting 0.000959649 seconds, Msg >> handler >> > >> allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFFB03C2910 ( >> 12636) waiting 0.000769611 seconds, Msg >> handler >> > >> allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF8C092850 ( >> 18215) waiting 0.000682275 seconds, Msg >> handler >> > >> allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 >> > >> (0xFFFFC9002163A2E0) (AllocManagerMutex) >> > 0x7FFF9423F730 ( 12652) >> waiting 0.000641915 seconds, Msg >> handler >> > >> allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF9422D770 ( >> 12625) waiting 0.000494256 seconds, Msg >> handler >> > >> allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF9423E310 ( >> 12651) waiting 0.000437760 seconds, Msg >> handler >> > >> allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 >> > >> (0xFFFFC9002163A2E0) (AllocManagerMutex) >> > >> > I don't know if >> this data point is useful but both yesterday >> and today >> > the >> metadata NSDs for this filesystem have had a constant >> aggregate >> >> > stream of 25MB/s 4kop/s reads during each episode (very low >> >> latency >> > though so I don't believe the storage is a bottleneck >> here). >> Writes are >> > only a few hundred ops and didn't strike me >> as odd. >> > >> > I have a PMR open for this but I'm curious if folks >> have >> seen this in >> > the wild and what it might mean. >> > >> > >> -Aaron >> > >> > -- >> > Aaron Knister >> > NASA Center for Climate >> Simulation (Code 606.2) >> > Goddard Space Flight Center >> > (301) >> 286-2776 >> >> > >> _______________________________________________ >> > gpfsug-discuss >> mailing list >> > gpfsug-discuss at spectrumscale.org >> >> >> >> > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > >> > >> > >> > >> _______________________________________________ >> > gpfsug-discuss >> mailing list >> > gpfsug-discuss at spectrumscale.org >> >> > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > >> >> -- >> >> Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) >> >> Goddard Space Flight Center >> (301) 286-2776 >> >> >> _______________________________________________ >> gpfsug-discuss >> mailing list >> gpfsug-discuss at spectrumscale.org >> >> >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> >> >> _______________________________________________ >> gpfsug-discuss >> mailing list >> gpfsug-discuss at spectrumscale.org >> >> >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > > > >> _______________________________________________ > gpfsug-discuss >> mailing list > gpfsug-discuss at spectrumscale.org >> > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight >> Center (301) 286-2776 >> _______________________________________________ gpfsug-discuss mailing >> list gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> _______________________________________________ gpfsug-discuss mailing >> list gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > The information contained in this e-mail message may be privileged, > confidential, and/or protected from disclosure. This e-mail message may > contain protected health information (PHI); dissemination of PHI should > comply with applicable federal and state laws. If you are not the > intended recipient, or an authorized representative of the intended > recipient, any further review, disclosure, use, dissemination, > distribution, or copying of this message or any attachment (or the > information contained therein) is strictly prohibited. If you think that > you have received this e-mail message in error, please notify the sender > by return e-mail and delete all references to it and its contents from > your systems. > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From oehmes at gmail.com Fri Mar 24 18:05:58 2017 From: oehmes at gmail.com (Sven Oehme) Date: Fri, 24 Mar 2017 18:05:58 +0000 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: was this filesystem creates with -n 5000 ? or was that changed later with mmchfs ? please send the mmlsconfig/mmlscluster output to me at oehmes at us.ibm.com On Fri, Mar 24, 2017 at 10:58 AM Aaron Knister wrote: > I feel a little awkward about posting wlists of IP's and hostnames on > the mailing list (even though they're all internal) but I'm happy to > send to you directly. I've attached both an lsfs and an mmdf output of > the fs in question here since that may be useful for others to see. Just > a note about disk d23_02_021-- it's been evacuated for several weeks now > due to a hardware issue in the disk enclosure. > > The fs is rather full percentage wise (93%) but in terms of capacity > there's a good amount free. 93% full of a 7PB filesystem still leaves > 551T. Metadata, as you'll see, is 31% free (roughly 800GB). > > The fs has 40M inodes allocated and 12M free. > > -Aaron > > On 3/24/17 1:41 PM, Sven Oehme wrote: > > ok, that seems a different problem then i was thinking. > > can you send output of mmlscluster, mmlsconfig, mmlsfs all ? > > also are you getting close to fill grade on inodes or capacity on any of > > the filesystems ? > > > > sven > > > > > > On Fri, Mar 24, 2017 at 10:34 AM Aaron Knister > > wrote: > > > > Here's the screenshot from the other node with the high cpu > utilization. > > > > On 3/24/17 1:32 PM, Aaron Knister wrote: > > > heh, yep we're on sles :) > > > > > > here's a screenshot of the fs manager from the deadlocked > filesystem. I > > > don't think there's an nsd server or manager node that's running > full > > > throttle across all cpus. There is one that's got relatively high > CPU > > > utilization though (300-400%). I'll send a screenshot of it in a > sec. > > > > > > no zimon yet but we do have other tools to see cpu utilization. > > > > > > -Aaron > > > > > > On 3/24/17 1:22 PM, Sven Oehme wrote: > > >> you must be on sles as this segfaults only on sles to my > knowledge :-) > > >> > > >> i am looking for a NSD or manager node in your cluster that runs > at 100% > > >> cpu usage. > > >> > > >> do you have zimon deployed to look at cpu utilization across your > nodes ? > > >> > > >> sven > > >> > > >> > > >> > > >> On Fri, Mar 24, 2017 at 10:08 AM Aaron Knister < > aaron.s.knister at nasa.gov > > >> >> > wrote: > > >> > > >> Hi Sven, > > >> > > >> Which NSD server should I run top on, the fs manager? If so > the > > >> CPU load > > >> is about 155%. I'm working on perf top but not off to a great > > >> start... > > >> > > >> # perf top > > >> PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% > [1000Hz > > >> cycles], (all, 28 CPUs) > > >> > > >> > --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > >> > > >> Segmentation fault > > >> > > >> -Aaron > > >> > > >> On 3/24/17 1:04 PM, Sven Oehme wrote: > > >> > while this is happening run top and see if there is very > high cpu > > >> > utilization at this time on the NSD Server. > > >> > > > >> > if there is , run perf top (you might need to install perf > > >> command) and > > >> > see if the top cpu contender is a spinlock . if so send a > > >> screenshot of > > >> > perf top as i may know what that is and how to fix. > > >> > > > >> > sven > > >> > > > >> > > > >> > On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister > > >> > > > > > >> > aaron.s.knister at nasa.gov> > > >> >>> > wrote: > > >> > > > >> > Since yesterday morning we've noticed some deadlocks on > one > > >> of our > > >> > filesystems that seem to be triggered by writing to it. > The > > >> waiters on > > >> > the clients look like this: > > >> > > > >> > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, > > >> SyncHandlerThread: > > >> > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) > > >> (InodeFlushCondVar), reason > > >> > 'waiting for the flush flag to commit metadata' > > >> > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, > > >> > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 > > >> (0x7FFFDAC7FE28) > > >> > (MsgRecordCondvar), reason 'RPC wait' for > > >> allocMsgTypeRelinquishRegion > > >> > on node 10.1.52.33 > > >> > 0x197EE70 ( 6776) waiting 0.000198354 seconds, > > >> > FileBlockWriteFetchHandlerThread: on ThCond > 0x7FFFF00CD598 > > >> > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' > for > > >> > allocMsgTypeRequestRegion on node 10.1.52.33 > > >> > > > >> > (10.1.52.33/c0n3271 > > > > >> is the fs manager > > >> > for the filesystem in question) > > >> > > > >> > there's a single process running on this node writing > to the > > >> filesystem > > >> > in question (well, trying to write, it's been blocked > doing > > >> nothing for > > >> > half an hour now). There are ~10 other client nodes in > this > > >> situation > > >> > right now. We had many more last night before the > problem > > >> seemed to > > >> > disappear in the early hours of the morning and now its > back. > > >> > > > >> > Waiters on the fs manager look like this. While the > > >> individual waiter is > > >> > short it's a near constant stream: > > >> > > > >> > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, > Msg > > >> handler > > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > > >> (0xFFFFC9002163A2E0) > > >> > (AllocManagerMutex) > > >> > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, > Msg > > >> handler > > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > >> > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, > Msg > > >> handler > > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > > >> (0xFFFFC9002163A2E0) > > >> > (AllocManagerMutex) > > >> > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, > Msg > > >> handler > > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > > >> (0xFFFFC9002163A2E0) > > >> > (AllocManagerMutex) > > >> > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, > Msg > > >> handler > > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > >> > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, > Msg > > >> handler > > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > > >> (0xFFFFC9002163A2E0) > > >> > (AllocManagerMutex) > > >> > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, > Msg > > >> handler > > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > > >> (0xFFFFC9002163A2E0) > > >> > (AllocManagerMutex) > > >> > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, > Msg > > >> handler > > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > >> > > > >> > I don't know if this data point is useful but both > yesterday > > >> and today > > >> > the metadata NSDs for this filesystem have had a > constant > > >> aggregate > > >> > stream of 25MB/s 4kop/s reads during each episode (very > low > > >> latency > > >> > though so I don't believe the storage is a bottleneck > here). > > >> Writes are > > >> > only a few hundred ops and didn't strike me as odd. > > >> > > > >> > I have a PMR open for this but I'm curious if folks have > > >> seen this in > > >> > the wild and what it might mean. > > >> > > > >> > -Aaron > > >> > > > >> > -- > > >> > Aaron Knister > > >> > NASA Center for Climate Simulation (Code 606.2) > > >> > Goddard Space Flight Center > > >> > (301) 286-2776 > > > > > >> > _______________________________________________ > > >> > gpfsug-discuss mailing list > > >> > gpfsug-discuss at spectrumscale.org < > http://spectrumscale.org> > > >> > > >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > >> > > > >> > > > >> > > > >> > _______________________________________________ > > >> > gpfsug-discuss mailing list > > >> > gpfsug-discuss at spectrumscale.org < > http://spectrumscale.org> > > >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > >> > > > >> > > >> -- > > >> Aaron Knister > > >> NASA Center for Climate Simulation (Code 606.2) > > >> Goddard Space Flight Center > > >> (301) 286-2776 > > >> _______________________________________________ > > >> gpfsug-discuss mailing list > > >> gpfsug-discuss at spectrumscale.org > > > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > >> > > >> > > >> > > >> _______________________________________________ > > >> gpfsug-discuss mailing list > > >> gpfsug-discuss at spectrumscale.org > > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > >> > > > > > > > > > > > > _______________________________________________ > > > gpfsug-discuss mailing list > > > gpfsug-discuss at spectrumscale.org > > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > -- > > Aaron Knister > > NASA Center for Climate Simulation (Code 606.2) > > Goddard Space Flight Center > > (301) 286-2776 > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Fri Mar 24 18:08:44 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 24 Mar 2017 14:08:44 -0400 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: <47206c08-671f-efab-d094-aa1200627b22@nasa.gov> I believe it was created with -n 5000. Here's the exact command that was used: /usr/lpp/mmfs/bin/mmcrfs dnb03 -F ./disc_mmcrnsd_dnb03.lst -T /gpfsm/dnb03 -j cluster -B 1M -n 5000 -N 20M -r1 -R2 -m2 -M2 -A no -Q yes -v yes -i 512 --metadata-block-size=256K -L 8388608 -Aaron On 3/24/17 2:05 PM, Sven Oehme wrote: > was this filesystem creates with -n 5000 ? or was that changed later > with mmchfs ? > please send the mmlsconfig/mmlscluster output to me at oehmes at us.ibm.com > > > > > On Fri, Mar 24, 2017 at 10:58 AM Aaron Knister > wrote: > > I feel a little awkward about posting wlists of IP's and hostnames on > the mailing list (even though they're all internal) but I'm happy to > send to you directly. I've attached both an lsfs and an mmdf output of > the fs in question here since that may be useful for others to see. Just > a note about disk d23_02_021-- it's been evacuated for several weeks now > due to a hardware issue in the disk enclosure. > > The fs is rather full percentage wise (93%) but in terms of capacity > there's a good amount free. 93% full of a 7PB filesystem still leaves > 551T. Metadata, as you'll see, is 31% free (roughly 800GB). > > The fs has 40M inodes allocated and 12M free. > > -Aaron > > On 3/24/17 1:41 PM, Sven Oehme wrote: > > ok, that seems a different problem then i was thinking. > > can you send output of mmlscluster, mmlsconfig, mmlsfs all ? > > also are you getting close to fill grade on inodes or capacity on any of > > the filesystems ? > > > > sven > > > > > > On Fri, Mar 24, 2017 at 10:34 AM Aaron Knister > > >> wrote: > > > > Here's the screenshot from the other node with the high cpu utilization. > > > > On 3/24/17 1:32 PM, Aaron Knister wrote: > > > heh, yep we're on sles :) > > > > > > here's a screenshot of the fs manager from the deadlocked filesystem. I > > > don't think there's an nsd server or manager node that's running full > > > throttle across all cpus. There is one that's got relatively high CPU > > > utilization though (300-400%). I'll send a screenshot of it in a sec. > > > > > > no zimon yet but we do have other tools to see cpu utilization. > > > > > > -Aaron > > > > > > On 3/24/17 1:22 PM, Sven Oehme wrote: > > >> you must be on sles as this segfaults only on sles to my knowledge :-) > > >> > > >> i am looking for a NSD or manager node in your cluster that runs at 100% > > >> cpu usage. > > >> > > >> do you have zimon deployed to look at cpu utilization across your nodes ? > > >> > > >> sven > > >> > > >> > > >> > > >> On Fri, Mar 24, 2017 at 10:08 AM Aaron Knister > > > > >> > >>> wrote: > > >> > > >> Hi Sven, > > >> > > >> Which NSD server should I run top on, the fs manager? If so the > > >> CPU load > > >> is about 155%. I'm working on perf top but not off to a great > > >> start... > > >> > > >> # perf top > > >> PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% [1000Hz > > >> cycles], (all, 28 CPUs) > > >> > > >> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > >> > > >> Segmentation fault > > >> > > >> -Aaron > > >> > > >> On 3/24/17 1:04 PM, Sven Oehme wrote: > > >> > while this is happening run top and see if there is very high cpu > > >> > utilization at this time on the NSD Server. > > >> > > > >> > if there is , run perf top (you might need to install perf > > >> command) and > > >> > see if the top cpu contender is a spinlock . if so send a > > >> screenshot of > > >> > perf top as i may know what that is and how to fix. > > >> > > > >> > sven > > >> > > > >> > > > >> > On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister > > >> > > > > > >> > > >> > > > > > >> > >>>> wrote: > > >> > > > >> > Since yesterday morning we've noticed some deadlocks on one > > >> of our > > >> > filesystems that seem to be triggered by writing to it. The > > >> waiters on > > >> > the clients look like this: > > >> > > > >> > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, > > >> SyncHandlerThread: > > >> > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) > > >> (InodeFlushCondVar), reason > > >> > 'waiting for the flush flag to commit metadata' > > >> > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, > > >> > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 > > >> (0x7FFFDAC7FE28) > > >> > (MsgRecordCondvar), reason 'RPC wait' for > > >> allocMsgTypeRelinquishRegion > > >> > on node 10.1.52.33 > > >> > 0x197EE70 ( 6776) waiting 0.000198354 seconds, > > >> > FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 > > >> > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for > > >> > allocMsgTypeRequestRegion on node 10.1.52.33 > > >> > > > >> > (10.1.52.33/c0n3271 > > > > > >> is the fs manager > > >> > for the filesystem in question) > > >> > > > >> > there's a single process running on this node writing to the > > >> filesystem > > >> > in question (well, trying to write, it's been blocked doing > > >> nothing for > > >> > half an hour now). There are ~10 other client nodes in this > > >> situation > > >> > right now. We had many more last night before the problem > > >> seemed to > > >> > disappear in the early hours of the morning and now its back. > > >> > > > >> > Waiters on the fs manager look like this. While the > > >> individual waiter is > > >> > short it's a near constant stream: > > >> > > > >> > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg > > >> handler > > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > > >> (0xFFFFC9002163A2E0) > > >> > (AllocManagerMutex) > > >> > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg > > >> handler > > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > >> > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg > > >> handler > > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > > >> (0xFFFFC9002163A2E0) > > >> > (AllocManagerMutex) > > >> > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg > > >> handler > > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > > >> (0xFFFFC9002163A2E0) > > >> > (AllocManagerMutex) > > >> > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg > > >> handler > > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > >> > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg > > >> handler > > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > > >> (0xFFFFC9002163A2E0) > > >> > (AllocManagerMutex) > > >> > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg > > >> handler > > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > > >> (0xFFFFC9002163A2E0) > > >> > (AllocManagerMutex) > > >> > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg > > >> handler > > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > >> > > > >> > I don't know if this data point is useful but both yesterday > > >> and today > > >> > the metadata NSDs for this filesystem have had a constant > > >> aggregate > > >> > stream of 25MB/s 4kop/s reads during each episode (very low > > >> latency > > >> > though so I don't believe the storage is a bottleneck here). > > >> Writes are > > >> > only a few hundred ops and didn't strike me as odd. > > >> > > > >> > I have a PMR open for this but I'm curious if folks have > > >> seen this in > > >> > the wild and what it might mean. > > >> > > > >> > -Aaron > > >> > > > >> > -- > > >> > Aaron Knister > > >> > NASA Center for Climate Simulation (Code 606.2) > > >> > Goddard Space Flight Center > > >> > (301) 286-2776 > > > > > >> > _______________________________________________ > > >> > gpfsug-discuss mailing list > > >> > gpfsug-discuss at spectrumscale.org > > >> > > >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > >> > > > >> > > > >> > > > >> > _______________________________________________ > > >> > gpfsug-discuss mailing list > > >> > gpfsug-discuss at spectrumscale.org > > > >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > >> > > > >> > > >> -- > > >> Aaron Knister > > >> NASA Center for Climate Simulation (Code 606.2) > > >> Goddard Space Flight Center > > >> (301) 286-2776 > > > >> _______________________________________________ > > >> gpfsug-discuss mailing list > > >> gpfsug-discuss at spectrumscale.org > > > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > >> > > >> > > >> > > >> _______________________________________________ > > >> gpfsug-discuss mailing list > > >> gpfsug-discuss at spectrumscale.org > > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > >> > > > > > > > > > > > > _______________________________________________ > > > gpfsug-discuss mailing list > > > gpfsug-discuss at spectrumscale.org > > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > -- > > Aaron Knister > > NASA Center for Climate Simulation (Code 606.2) > > Goddard Space Flight Center > > (301) 286-2776 > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From bbanister at jumptrading.com Fri Mar 24 18:13:33 2017 From: bbanister at jumptrading.com (Bryan Banister) Date: Fri, 24 Mar 2017 18:13:33 +0000 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: References: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu><47e18bb0062544f4b510ce0e87049fd3@jumptrading.com><643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> Message-ID: Hi Vipul, Hmm... interesting. We have dedicated systems running CES and nothing else, so the only thing opening files on GPFS is ganesha. IBM Support recommended we massively increase the maxFilesToCache to fix the performance issues we were having. I could try to reproduce the problem to investigate further, but the impact to users would not be appreciated. Setting such a high value for maxFilesToCache has direct impact on the token manager so fixing this would be very good. Perhaps IBM could take a closer look at this without our involvement? -Bryan From: Vipul Paul [mailto:vpaul at us.ibm.com] On Behalf Of IBM Spectrum Scale Sent: Friday, March 24, 2017 12:18 PM To: gpfsug main discussion list ; Bryan Banister Subject: Re: CES node slow to respond Hi Bryan, Making sure Malahal's reply was received by the user group. >> Then we noticed that the CES host had 5.4 million files open This is technically not possible with ganesha alone. A process can only open 1 million files on RHEL distro. Either we have leaks in kernel or some other processes contributing to this. Ganesha does keep NFSv3 files open and keep open for performance. It doesn't have a good logic to close them after some inactivity. It does close them if the number is close to max open files which is configurable. PS: kNFS does open, read/write, and then close. No caching in older versions. They did have a feature in a recent code to cache open files in NFSv3 as well. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Bryan Banister > To: gpfsug main discussion list > Date: 03/23/2017 08:27 AM Subject: Re: [gpfsug-discuss] CES node slow to respond Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Anybody from IBM willing/able to give us some explanation of why ganesha is holding open so many files? Is this expected/needed/etc? Or do we have to open a PMR to get some kind of explanation? -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Thursday, March 23, 2017 10:24 AM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] CES node slow to respond FYI all We also ran into this after bumping maxFilesToCache. Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: unexpected error: Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many open files in system fix sysctl -w fs.file-max=1000000 On 3/22/17 12:01 PM, Bryan Banister wrote: > We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. > > Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil > Sent: Wednesday, March 22, 2017 11:43 AM > To: gpfsug main discussion list > > Subject: [gpfsug-discuss] CES node slow to respond > > All, > > We had an indecent yesterday where one of our CES nodes slowed to a > crawl. GPFS waiters showed pre fetch threads going after inodes. > iohist also showed lots of inode fetching. Then we noticed that the CES > host had 5.4 million files open. > > The change I made was to set maxStatCache=DEFAULT because it is linux. > And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. > > Is there something else we should change as well. > > Thanks > > Matt > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From oehmes at gmail.com Fri Mar 24 18:19:52 2017 From: oehmes at gmail.com (Sven Oehme) Date: Fri, 24 Mar 2017 18:19:52 +0000 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: References: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu> <47e18bb0062544f4b510ce0e87049fd3@jumptrading.com> <643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> Message-ID: changes in ganesha management code were made in April 2016 to reduce the need for high maxfilestocache value, the ganesha daemon adjusts it allowed file cache by reading the maxfilestocache value and then reducing its allowed NOFILE value . the code shipped with 4.2.2 release. you want a high maxfilestocache value on CES nodes for various reasons. just FYI we have some customers who have MFTC set to 15 Million :-) sven On Fri, Mar 24, 2017 at 11:13 AM Bryan Banister wrote: > Hi Vipul, > > > > Hmm? interesting. We have dedicated systems running CES and nothing else, > so the only thing opening files on GPFS is ganesha. IBM Support > recommended we massively increase the maxFilesToCache to fix the > performance issues we were having. I could try to reproduce the problem to > investigate further, but the impact to users would not be appreciated. > > > > Setting such a high value for maxFilesToCache has direct impact on the > token manager so fixing this would be very good. > > > > Perhaps IBM could take a closer look at this without our involvement? > > -Bryan > > > > *From:* Vipul Paul [mailto:vpaul at us.ibm.com] *On Behalf Of *IBM Spectrum > Scale > *Sent:* Friday, March 24, 2017 12:18 PM > *To:* gpfsug main discussion list ; > Bryan Banister > *Subject:* Re: CES node slow to respond > > > > Hi Bryan, > > Making sure Malahal's reply was received by the user group. > > >> Then we noticed that the CES host had 5.4 million files open > > This is technically not possible with ganesha alone. A process can only > open 1 million files on RHEL distro. Either we have leaks in kernel or some > other processes contributing to this. > > Ganesha does keep NFSv3 files open and keep open for performance. It > doesn't have a good logic to close them after some inactivity. It does > close them if the number is close to max open files which is configurable. > > PS: kNFS does open, read/write, and then close. No caching in older > versions. They did have a feature in a recent code to cache open files in > NFSv3 as well. > > Regards, The Spectrum Scale (GPFS) team > > > ------------------------------------------------------------------------------------------------------------------ > If you feel that your question can benefit other users of Spectrum Scale > (GPFS), then please post it to the public IBM developerWroks Forum at > https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. > > > If your query concerns a potential software error in Spectrum Scale (GPFS) > and you have an IBM software maintenance contract please contact > 1-800-237-5511 <(800)%20237-5511> in the United States or your local IBM > Service Center in other countries. > > The forum is informally monitored as time permits and should not be used > for priority messages to the Spectrum Scale (GPFS) team. > > > > From: Bryan Banister > To: gpfsug main discussion list > Date: 03/23/2017 08:27 AM > Subject: Re: [gpfsug-discuss] CES node slow to respond > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------ > > > > > Anybody from IBM willing/able to give us some explanation of why ganesha > is holding open so many files? Is this expected/needed/etc? > > Or do we have to open a PMR to get some kind of explanation? > -B > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [ > mailto:gpfsug-discuss-bounces at spectrumscale.org > ] On Behalf Of Matt Weil > Sent: Thursday, March 23, 2017 10:24 AM > To: gpfsug-discuss at spectrumscale.org > Subject: Re: [gpfsug-discuss] CES node slow to respond > > FYI all > > We also ran into this after bumping maxFilesToCache. > > Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: > unexpected error: > Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many > open files in system > > fix > > sysctl -w fs.file-max=1000000 > > > On 3/22/17 12:01 PM, Bryan Banister wrote: > > We had a similar issue and also were instructed by IBM Support to > increase the maxFilesToCache to an insane value... Basically when the file > cache gets full then the host will spend all of its cycles looking for a > file to evict every time a new file is opened... baaaah. > > > > Not sure why Ganesha has to keep so many files open... I can't believe > our NFS clients actually keep that many open. cNFS never needed this. > > -Bryan > > > > -----Original Message----- > > From: gpfsug-discuss-bounces at spectrumscale.org [ > mailto:gpfsug-discuss-bounces at spectrumscale.org > ] On Behalf Of Matt Weil > > Sent: Wednesday, March 22, 2017 11:43 AM > > To: gpfsug main discussion list > > Subject: [gpfsug-discuss] CES node slow to respond > > > > All, > > > > We had an indecent yesterday where one of our CES nodes slowed to a > > crawl. GPFS waiters showed pre fetch threads going after inodes. > > iohist also showed lots of inode fetching. Then we noticed that the CES > > host had 5.4 million files open. > > > > The change I made was to set maxStatCache=DEFAULT because it is linux. > > And set maxFilesToCache=10000000 it was set to 500000. Then restarted > GPFS. > > > > Is there something else we should change as well. > > > > Thanks > > > > Matt > > > > > > ________________________________ > > The materials in this message are private and may contain Protected > Healthcare Information or other information of a sensitive nature. If you > are not the intended recipient, be advised that any unauthorized use, > disclosure, copying or the taking of any action in reliance on the contents > of this information is strictly prohibited. If you have received this email > in error, please immediately notify the sender via telephone or return mail. > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > ________________________________ > > > > Note: This email is for the confidential use of the named addressee(s) > only and may contain proprietary, confidential or privileged information. > If you are not the intended recipient, you are hereby notified that any > review, dissemination or copying of this email is strictly prohibited, and > to please notify the sender immediately and destroy this email and any > attachments. Email transmission cannot be guaranteed to be secure or > error-free. The Company, therefore, does not make any guarantees as to the > completeness or accuracy of this email or any attachments. This email is > for informational purposes only and does not constitute a recommendation, > offer, request or solicitation of any kind to buy, sell, subscribe, redeem > or perform any type of transaction of a financial product. > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > ________________________________ > The materials in this message are private and may contain Protected > Healthcare Information or other information of a sensitive nature. If you > are not the intended recipient, be advised that any unauthorized use, > disclosure, copying or the taking of any action in reliance on the contents > of this information is strictly prohibited. If you have received this email > in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) > only and may contain proprietary, confidential or privileged information. > If you are not the intended recipient, you are hereby notified that any > review, dissemination or copying of this email is strictly prohibited, and > to please notify the sender immediately and destroy this email and any > attachments. Email transmission cannot be guaranteed to be secure or > error-free. The Company, therefore, does not make any guarantees as to the > completeness or accuracy of this email or any attachments. This email is > for informational purposes only and does not constitute a recommendation, > offer, request or solicitation of any kind to buy, sell, subscribe, redeem > or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > ------------------------------ > > Note: This email is for the confidential use of the named addressee(s) > only and may contain proprietary, confidential or privileged information. > If you are not the intended recipient, you are hereby notified that any > review, dissemination or copying of this email is strictly prohibited, and > to please notify the sender immediately and destroy this email and any > attachments. Email transmission cannot be guaranteed to be secure or > error-free. The Company, therefore, does not make any guarantees as to the > completeness or accuracy of this email or any attachments. This email is > for informational purposes only and does not constitute a recommendation, > offer, request or solicitation of any kind to buy, sell, subscribe, redeem > or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Fri Mar 24 18:22:50 2017 From: bbanister at jumptrading.com (Bryan Banister) Date: Fri, 24 Mar 2017 18:22:50 +0000 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: References: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu> <47e18bb0062544f4b510ce0e87049fd3@jumptrading.com> <643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> Message-ID: <47c157d6cb5747698d3ccf11bd7a27ab@jumptrading.com> Thanks Sven! We recently upgrade to 4.2.2 and will see about lowering the maxFilesToCache to something more appropriate. We?re not offering NFS access as a performance solution? but it can?t come to a crawl either! Your help is greatly appreciated as always, -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sven Oehme Sent: Friday, March 24, 2017 1:20 PM To: gpfsug main discussion list ; IBM Spectrum Scale Subject: Re: [gpfsug-discuss] CES node slow to respond changes in ganesha management code were made in April 2016 to reduce the need for high maxfilestocache value, the ganesha daemon adjusts it allowed file cache by reading the maxfilestocache value and then reducing its allowed NOFILE value . the code shipped with 4.2.2 release. you want a high maxfilestocache value on CES nodes for various reasons. just FYI we have some customers who have MFTC set to 15 Million :-) sven On Fri, Mar 24, 2017 at 11:13 AM Bryan Banister > wrote: Hi Vipul, Hmm? interesting. We have dedicated systems running CES and nothing else, so the only thing opening files on GPFS is ganesha. IBM Support recommended we massively increase the maxFilesToCache to fix the performance issues we were having. I could try to reproduce the problem to investigate further, but the impact to users would not be appreciated. Setting such a high value for maxFilesToCache has direct impact on the token manager so fixing this would be very good. Perhaps IBM could take a closer look at this without our involvement? -Bryan From: Vipul Paul [mailto:vpaul at us.ibm.com] On Behalf Of IBM Spectrum Scale Sent: Friday, March 24, 2017 12:18 PM To: gpfsug main discussion list >; Bryan Banister > Subject: Re: CES node slow to respond Hi Bryan, Making sure Malahal's reply was received by the user group. >> Then we noticed that the CES host had 5.4 million files open This is technically not possible with ganesha alone. A process can only open 1 million files on RHEL distro. Either we have leaks in kernel or some other processes contributing to this. Ganesha does keep NFSv3 files open and keep open for performance. It doesn't have a good logic to close them after some inactivity. It does close them if the number is close to max open files which is configurable. PS: kNFS does open, read/write, and then close. No caching in older versions. They did have a feature in a recent code to cache open files in NFSv3 as well. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Bryan Banister > To: gpfsug main discussion list > Date: 03/23/2017 08:27 AM Subject: Re: [gpfsug-discuss] CES node slow to respond Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Anybody from IBM willing/able to give us some explanation of why ganesha is holding open so many files? Is this expected/needed/etc? Or do we have to open a PMR to get some kind of explanation? -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Thursday, March 23, 2017 10:24 AM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] CES node slow to respond FYI all We also ran into this after bumping maxFilesToCache. Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: unexpected error: Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many open files in system fix sysctl -w fs.file-max=1000000 On 3/22/17 12:01 PM, Bryan Banister wrote: > We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. > > Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil > Sent: Wednesday, March 22, 2017 11:43 AM > To: gpfsug main discussion list > > Subject: [gpfsug-discuss] CES node slow to respond > > All, > > We had an indecent yesterday where one of our CES nodes slowed to a > crawl. GPFS waiters showed pre fetch threads going after inodes. > iohist also showed lots of inode fetching. Then we noticed that the CES > host had 5.4 million files open. > > The change I made was to set maxStatCache=DEFAULT because it is linux. > And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. > > Is there something else we should change as well. > > Thanks > > Matt > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mweil at wustl.edu Fri Mar 24 19:57:42 2017 From: mweil at wustl.edu (Matt Weil) Date: Fri, 24 Mar 2017 14:57:42 -0500 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: References: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu> <47e18bb0062544f4b510ce0e87049fd3@jumptrading.com> <643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> Message-ID: <5a3aeddb-3e09-4003-8e57-299e999fe62d@wustl.edu> On 3/24/17 1:13 PM, Bryan Banister wrote: Hi Vipul, Hmm? interesting. We have dedicated systems running CES and nothing else, so the only thing opening files on GPFS is ganesha. IBM Support recommended we massively increase the maxFilesToCache to fix the performance issues we were having. I could try to reproduce the problem to investigate further, but the impact to users would not be appreciated. Same with us the system only runs CES and NFS. All open files 5.4 million where tied to it. This excluded anything the system had open. Setting such a high value for maxFilesToCache has direct impact on the token manager so fixing this would be very good. Perhaps IBM could take a closer look at this without our involvement? -Bryan From: Vipul Paul [mailto:vpaul at us.ibm.com] On Behalf Of IBM Spectrum Scale Sent: Friday, March 24, 2017 12:18 PM To: gpfsug main discussion list ; Bryan Banister Subject: Re: CES node slow to respond Hi Bryan, Making sure Malahal's reply was received by the user group. >> Then we noticed that the CES host had 5.4 million files open This is technically not possible with ganesha alone. A process can only open 1 million files on RHEL distro. Either we have leaks in kernel or some other processes contributing to this. Ganesha does keep NFSv3 files open and keep open for performance. It doesn't have a good logic to close them after some inactivity. It does close them if the number is close to max open files which is configurable. PS: kNFS does open, read/write, and then close. No caching in older versions. They did have a feature in a recent code to cache open files in NFSv3 as well. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Bryan Banister > To: gpfsug main discussion list > Date: 03/23/2017 08:27 AM Subject: Re: [gpfsug-discuss] CES node slow to respond Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Anybody from IBM willing/able to give us some explanation of why ganesha is holding open so many files? Is this expected/needed/etc? Or do we have to open a PMR to get some kind of explanation? -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Thursday, March 23, 2017 10:24 AM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] CES node slow to respond FYI all We also ran into this after bumping maxFilesToCache. Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: unexpected error: Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many open files in system fix sysctl -w fs.file-max=1000000 On 3/22/17 12:01 PM, Bryan Banister wrote: > We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. > > Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil > Sent: Wednesday, March 22, 2017 11:43 AM > To: gpfsug main discussion list > > Subject: [gpfsug-discuss] CES node slow to respond > > All, > > We had an indecent yesterday where one of our CES nodes slowed to a > crawl. GPFS waiters showed pre fetch threads going after inodes. > iohist also showed lots of inode fetching. Then we noticed that the CES > host had 5.4 million files open. > > The change I made was to set maxStatCache=DEFAULT because it is linux. > And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. > > Is there something else we should change as well. > > Thanks > > Matt > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mweil at wustl.edu Fri Mar 24 20:03:22 2017 From: mweil at wustl.edu (Matt Weil) Date: Fri, 24 Mar 2017 15:03:22 -0500 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: <5a3aeddb-3e09-4003-8e57-299e999fe62d@wustl.edu> References: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu> <47e18bb0062544f4b510ce0e87049fd3@jumptrading.com> <643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> <5a3aeddb-3e09-4003-8e57-299e999fe62d@wustl.edu> Message-ID: also running version 4.2.2.2. On 3/24/17 2:57 PM, Matt Weil wrote: On 3/24/17 1:13 PM, Bryan Banister wrote: Hi Vipul, Hmm? interesting. We have dedicated systems running CES and nothing else, so the only thing opening files on GPFS is ganesha. IBM Support recommended we massively increase the maxFilesToCache to fix the performance issues we were having. I could try to reproduce the problem to investigate further, but the impact to users would not be appreciated. Same with us the system only runs CES and NFS. All open files 5.4 million where tied to it. This excluded anything the system had open. Setting such a high value for maxFilesToCache has direct impact on the token manager so fixing this would be very good. Perhaps IBM could take a closer look at this without our involvement? -Bryan From: Vipul Paul [mailto:vpaul at us.ibm.com] On Behalf Of IBM Spectrum Scale Sent: Friday, March 24, 2017 12:18 PM To: gpfsug main discussion list ; Bryan Banister Subject: Re: CES node slow to respond Hi Bryan, Making sure Malahal's reply was received by the user group. >> Then we noticed that the CES host had 5.4 million files open This is technically not possible with ganesha alone. A process can only open 1 million files on RHEL distro. Either we have leaks in kernel or some other processes contributing to this. Ganesha does keep NFSv3 files open and keep open for performance. It doesn't have a good logic to close them after some inactivity. It does close them if the number is close to max open files which is configurable. PS: kNFS does open, read/write, and then close. No caching in older versions. They did have a feature in a recent code to cache open files in NFSv3 as well. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Bryan Banister > To: gpfsug main discussion list > Date: 03/23/2017 08:27 AM Subject: Re: [gpfsug-discuss] CES node slow to respond Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Anybody from IBM willing/able to give us some explanation of why ganesha is holding open so many files? Is this expected/needed/etc? Or do we have to open a PMR to get some kind of explanation? -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Thursday, March 23, 2017 10:24 AM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] CES node slow to respond FYI all We also ran into this after bumping maxFilesToCache. Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: unexpected error: Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many open files in system fix sysctl -w fs.file-max=1000000 On 3/22/17 12:01 PM, Bryan Banister wrote: > We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. > > Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil > Sent: Wednesday, March 22, 2017 11:43 AM > To: gpfsug main discussion list > > Subject: [gpfsug-discuss] CES node slow to respond > > All, > > We had an indecent yesterday where one of our CES nodes slowed to a > crawl. GPFS waiters showed pre fetch threads going after inodes. > iohist also showed lots of inode fetching. Then we noticed that the CES > host had 5.4 million files open. > > The change I made was to set maxStatCache=DEFAULT because it is linux. > And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. > > Is there something else we should change as well. > > Thanks > > Matt > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mweil at wustl.edu Fri Mar 24 20:20:18 2017 From: mweil at wustl.edu (Matt Weil) Date: Fri, 24 Mar 2017 15:20:18 -0500 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: References: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu> <47e18bb0062544f4b510ce0e87049fd3@jumptrading.com> <643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> Message-ID: On 3/24/17 12:17 PM, IBM Spectrum Scale wrote: Hi Bryan, Making sure Malahal's reply was received by the user group. >> Then we noticed that the CES host had 5.4 million files open This is technically not possible with ganesha alone. A process can only open 1 million files on RHEL distro. Either we have leaks in kernel or some other processes contributing to this. Ganesha does keep NFSv3 files open and keep open for performance. It doesn't have a good logic to close them after some inactivity. It does close them if the number is close to max open files which is configurable. how do you set the max open files for Ganesha? PS: kNFS does open, read/write, and then close. No caching in older versions. They did have a feature in a recent code to cache open files in NFSv3 as well. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Bryan Banister To: gpfsug main discussion list Date: 03/23/2017 08:27 AM Subject: Re: [gpfsug-discuss] CES node slow to respond Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Anybody from IBM willing/able to give us some explanation of why ganesha is holding open so many files? Is this expected/needed/etc? Or do we have to open a PMR to get some kind of explanation? -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Thursday, March 23, 2017 10:24 AM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] CES node slow to respond FYI all We also ran into this after bumping maxFilesToCache. Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: unexpected error: Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many open files in system fix sysctl -w fs.file-max=1000000 On 3/22/17 12:01 PM, Bryan Banister wrote: > We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. > > Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil > Sent: Wednesday, March 22, 2017 11:43 AM > To: gpfsug main discussion list > Subject: [gpfsug-discuss] CES node slow to respond > > All, > > We had an indecent yesterday where one of our CES nodes slowed to a > crawl. GPFS waiters showed pre fetch threads going after inodes. > iohist also showed lots of inode fetching. Then we noticed that the CES > host had 5.4 million files open. > > The change I made was to set maxStatCache=DEFAULT because it is linux. > And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. > > Is there something else we should change as well. > > Thanks > > Matt > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Sat Mar 25 05:46:34 2017 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Fri, 24 Mar 2017 21:46:34 -0800 Subject: [gpfsug-discuss] CES node slow to respond Message-ID: Forwarding Malahal's reply. Max open files of the ganesha process is set internally (and written to /etc/sysconfig/ganesha file as NOFILE parameter) based on MFTC (max files to cache) gpfs parameter. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. >> From: Matt Weil To: Date: 03/24/2017 01:20 PM Subject: Re: [gpfsug-discuss] CES node slow to respond Sent by: gpfsug-discuss-bounces at spectrumscale.org On 3/24/17 12:17 PM, IBM Spectrum Scale wrote: Hi Bryan, Making sure Malahal's reply was received by the user group. >> Then we noticed that the CES host had 5.4 million files open This is technically not possible with ganesha alone. A process can only open 1 million files on RHEL distro. Either we have leaks in kernel or some other processes contributing to this. Ganesha does keep NFSv3 files open and keep open for performance. It doesn't have a good logic to close them after some inactivity. It does close them if the number is close to max open files which is configurable. how do you set the max open files for Ganesha? PS: kNFS does open, read/write, and then close. No caching in older versions. They did have a feature in a recent code to cache open files in NFSv3 as well. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Bryan Banister To: gpfsug main discussion list Date: 03/23/2017 08:27 AM Subject: Re: [gpfsug-discuss] CES node slow to respond Sent by: gpfsug-discuss-bounces at spectrumscale.org Anybody from IBM willing/able to give us some explanation of why ganesha is holding open so many files? Is this expected/needed/etc? Or do we have to open a PMR to get some kind of explanation? -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Thursday, March 23, 2017 10:24 AM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] CES node slow to respond FYI all We also ran into this after bumping maxFilesToCache. Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: unexpected error: Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many open files in system fix sysctl -w fs.file-max=1000000 On 3/22/17 12:01 PM, Bryan Banister wrote: > We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. > > Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil > Sent: Wednesday, March 22, 2017 11:43 AM > To: gpfsug main discussion list > Subject: [gpfsug-discuss] CES node slow to respond > > All, > > We had an indecent yesterday where one of our CES nodes slowed to a > crawl. GPFS waiters showed pre fetch threads going after inodes. > iohist also showed lots of inode fetching. Then we noticed that the CES > host had 5.4 million files open. > > The change I made was to set maxStatCache=DEFAULT because it is linux. > And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. > > Is there something else we should change as well. > > Thanks > > Matt > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail._______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Sat Mar 25 05:55:57 2017 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Fri, 24 Mar 2017 21:55:57 -0800 Subject: [gpfsug-discuss] Fw: CES node slow to respond Message-ID: Caching of file descriptors can be disabled with "Cache_FDs = FALSE;" in cacheinode{} block. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. >> From: Bryan Banister To: IBM Spectrum Scale/Poughkeepsie/IBM at IBMUS, gpfsug main discussion list Date: 03/24/2017 11:13 AM Subject: RE: CES node slow to respond Hi Vipul, Hmm? interesting. We have dedicated systems running CES and nothing else, so the only thing opening files on GPFS is ganesha. IBM Support recommended we massively increase the maxFilesToCache to fix the performance issues we were having. I could try to reproduce the problem to investigate further, but the impact to users would not be appreciated. Setting such a high value for maxFilesToCache has direct impact on the token manager so fixing this would be very good. Perhaps IBM could take a closer look at this without our involvement? -Bryan From: Vipul Paul [mailto:vpaul at us.ibm.com] On Behalf Of IBM Spectrum Scale Sent: Friday, March 24, 2017 12:18 PM To: gpfsug main discussion list ; Bryan Banister Subject: Re: CES node slow to respond Hi Bryan, Making sure Malahal's reply was received by the user group. >> Then we noticed that the CES host had 5.4 million files open This is technically not possible with ganesha alone. A process can only open 1 million files on RHEL distro. Either we have leaks in kernel or some other processes contributing to this. Ganesha does keep NFSv3 files open and keep open for performance. It doesn't have a good logic to close them after some inactivity. It does close them if the number is close to max open files which is configurable. PS: kNFS does open, read/write, and then close. No caching in older versions. They did have a feature in a recent code to cache open files in NFSv3 as well. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Bryan Banister To: gpfsug main discussion list Date: 03/23/2017 08:27 AM Subject: Re: [gpfsug-discuss] CES node slow to respond Sent by: gpfsug-discuss-bounces at spectrumscale.org Anybody from IBM willing/able to give us some explanation of why ganesha is holding open so many files? Is this expected/needed/etc? Or do we have to open a PMR to get some kind of explanation? -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Thursday, March 23, 2017 10:24 AM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] CES node slow to respond FYI all We also ran into this after bumping maxFilesToCache. Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: unexpected error: Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many open files in system fix sysctl -w fs.file-max=1000000 On 3/22/17 12:01 PM, Bryan Banister wrote: > We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. > > Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil > Sent: Wednesday, March 22, 2017 11:43 AM > To: gpfsug main discussion list > Subject: [gpfsug-discuss] CES node slow to respond > > All, > > We had an indecent yesterday where one of our CES nodes slowed to a > crawl. GPFS waiters showed pre fetch threads going after inodes. > iohist also showed lots of inode fetching. Then we noticed that the CES > host had 5.4 million files open. > > The change I made was to set maxStatCache=DEFAULT because it is linux. > And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. > > Is there something else we should change as well. > > Thanks > > Matt > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chair at spectrumscale.org Mon Mar 27 08:55:04 2017 From: chair at spectrumscale.org (Spectrum Scale UG Chair (Simon Thompson)) Date: Mon, 27 Mar 2017 08:55:04 +0100 Subject: [gpfsug-discuss] UK User Group Registration NOW OPEN! [May 9th / 10th 2017] Message-ID: Registration for the UK Spectrum Scale (GPFS) user group (May 9th/10th) has opened this morning. Registration is via EventBrite at: https://www.eventbrite.com/e/spectrum-scalegpfs-user-group-spring-2017-regi stration-32113696932 This year we have moved to a larger capacity venue (Manchester Metropole Hotel), with thanks to IBM for sponsoring and supporting the event. We would also like to thank our other sponsors ArcaStream, DDN Storage, Ellexus, Mellanox, OCF and Seagate for supporting the event. Their sponsorship goes to help ensure the event remains free for the community and supports the evening social event as well as providing them with a technical (non sales) slot on the agenda. This year, the evening social event will be taking place at Manchester Museum of Science and Industry, and we're very grateful that are sponsors are able to support us to ensure this happens. The agenda is still a work in progress and may be subject to change as we confirm speakers and timings. As usual, we have reserved some slots on the agenda for user speakers, so if you have an interesting use case of Spectrum Scale, please let me know. As sectors we don't really get to hear from, it would be great if one of the members from the finance or sport industry were able to speak! Of course the event isn't limited to people from the UK, and we welcome registrants from across the world! Please note that to ensure attendance from a wide range of organisations, we are currently limiting attendance to 3 people per organisation. The venue is situated just a few minutes walk from Manchester Piccadilly station in Central Manchester with many hotels near by. The venue also has a limited number of special rate rooms, details of how you can book direct with the hotel are on the registration page. Thanks Simon UK Group Chair From p.childs at qmul.ac.uk Mon Mar 27 19:14:34 2017 From: p.childs at qmul.ac.uk (Peter Childs) Date: Mon, 27 Mar 2017 18:14:34 +0000 Subject: [gpfsug-discuss] UK User Group Registration NOW OPEN! [May 9th / 10th 2017] In-Reply-To: References: Message-ID: <0d1hhnpr5cn1dc5n2h7jpcec.1490638472562@email.android.com> Can I verify that this is at the Manchester McDonald hotel as eventbright, and website says, not the Manchester metropole hotel as the mailing list messages says, and I'm having difficulty finding. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ---- Spectrum Scale UG Chair (Simon Thompson) wrote ---- Registration for the UK Spectrum Scale (GPFS) user group (May 9th/10th) has opened this morning. Registration is via EventBrite at: https://www.eventbrite.com/e/spectrum-scalegpfs-user-group-spring-2017-regi stration-32113696932 This year we have moved to a larger capacity venue (Manchester Metropole Hotel), with thanks to IBM for sponsoring and supporting the event. We would also like to thank our other sponsors ArcaStream, DDN Storage, Ellexus, Mellanox, OCF and Seagate for supporting the event. Their sponsorship goes to help ensure the event remains free for the community and supports the evening social event as well as providing them with a technical (non sales) slot on the agenda. This year, the evening social event will be taking place at Manchester Museum of Science and Industry, and we're very grateful that are sponsors are able to support us to ensure this happens. The agenda is still a work in progress and may be subject to change as we confirm speakers and timings. As usual, we have reserved some slots on the agenda for user speakers, so if you have an interesting use case of Spectrum Scale, please let me know. As sectors we don't really get to hear from, it would be great if one of the members from the finance or sport industry were able to speak! Of course the event isn't limited to people from the UK, and we welcome registrants from across the world! Please note that to ensure attendance from a wide range of organisations, we are currently limiting attendance to 3 people per organisation. The venue is situated just a few minutes walk from Manchester Piccadilly station in Central Manchester with many hotels near by. The venue also has a limited number of special rate rooms, details of how you can book direct with the hotel are on the registration page. Thanks Simon UK Group Chair _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Mon Mar 27 19:25:53 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Mon, 27 Mar 2017 18:25:53 +0000 Subject: [gpfsug-discuss] UK User Group Registration NOW OPEN! [May 9th / 10th 2017] In-Reply-To: <0d1hhnpr5cn1dc5n2h7jpcec.1490638472562@email.android.com> References: <0d1hhnpr5cn1dc5n2h7jpcec.1490638472562@email.android.com> Message-ID: Hi Peter, Sorry, my mistake, the eventbrite page is correct - Manchester McDonald hotel. (we looked at a couple of different options in different areas and I obviously got confused!) Simon From: > on behalf of Peter Childs > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Monday, 27 March 2017 at 19:14 To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] UK User Group Registration NOW OPEN! [May 9th / 10th 2017] Can I verify that this is at the Manchester McDonald hotel as eventbright, and website says, not the Manchester metropole hotel as the mailing list messages says, and I'm having difficulty finding. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ---- Spectrum Scale UG Chair (Simon Thompson) wrote ---- Registration for the UK Spectrum Scale (GPFS) user group (May 9th/10th) has opened this morning. Registration is via EventBrite at: https://www.eventbrite.com/e/spectrum-scalegpfs-user-group-spring-2017-regi stration-32113696932 This year we have moved to a larger capacity venue (Manchester Metropole Hotel), with thanks to IBM for sponsoring and supporting the event. We would also like to thank our other sponsors ArcaStream, DDN Storage, Ellexus, Mellanox, OCF and Seagate for supporting the event. Their sponsorship goes to help ensure the event remains free for the community and supports the evening social event as well as providing them with a technical (non sales) slot on the agenda. This year, the evening social event will be taking place at Manchester Museum of Science and Industry, and we're very grateful that are sponsors are able to support us to ensure this happens. The agenda is still a work in progress and may be subject to change as we confirm speakers and timings. As usual, we have reserved some slots on the agenda for user speakers, so if you have an interesting use case of Spectrum Scale, please let me know. As sectors we don't really get to hear from, it would be great if one of the members from the finance or sport industry were able to speak! Of course the event isn't limited to people from the UK, and we welcome registrants from across the world! Please note that to ensure attendance from a wide range of organisations, we are currently limiting attendance to 3 people per organisation. The venue is situated just a few minutes walk from Manchester Piccadilly station in Central Manchester with many hotels near by. The venue also has a limited number of special rate rooms, details of how you can book direct with the hotel are on the registration page. Thanks Simon UK Group Chair _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From pinto at scinet.utoronto.ca Tue Mar 28 15:47:44 2017 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Tue, 28 Mar 2017 10:47:44 -0400 Subject: [gpfsug-discuss] fix mmrepquota report format during grace periods Message-ID: <20170328104744.138874ih550oncls@support.scinet.utoronto.ca> Any chance you guys in the GPFS devel team could patch the mmrepquota code so that during grace periods the report column for "none" would still be replaced with >>>*ONE*<<< word? By that I mean, instead of "2 days" for example, just print "2-days" or "2days" or "2_days", and so on. I have a number of scripts that fail for users when they are over their quotas under grace periods, because the report shifts the remaining information for that user 1 column to the right. Obviously it would cost me absolutely nothing to patch my scripts to deal with this, however the principle here is that the reports generated by GPFS should be the ones keeping consistence. Thanks Jaime ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. From Robert.Oesterlin at nuance.com Tue Mar 28 15:54:42 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Tue, 28 Mar 2017 14:54:42 +0000 Subject: [gpfsug-discuss] fix mmrepquota report format during grace periods Message-ID: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> Try running it with the ?-Y? option, it returns an easily to read output: mmrepquota -Y dns mmrepquota::HEADER:version:reserved:reserved:filesystemName:quotaType:id:name:blockUsage:blockQuota:blockLimit:blockInDoubt:blockGrace:filesUsage:filesQuota:filesLimit:filesInDoubt:filesGrace:remarks:quota:defQuota:fid:filesetname: mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:root: mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:users: mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:root: mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:users: mmrepquota::0:1:::dns:FILESET:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off::: mmrepquota::0:1:::dns:FILESET:1:users:0:4294967296:4294967296:0:none:1:0:0:0:none:e:on:off::: Bob Oesterlin Sr Principal Storage Engineer, Nuance On 3/28/17, 9:47 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Jaime Pinto" wrote: Any chance you guys in the GPFS devel team could patch the mmrepquota code so that during grace periods the report column for "none" would still be replaced with >>>*ONE*<<< word? By that I mean, instead of "2 days" for example, just print "2-days" or "2days" or "2_days", and so on. I have a number of scripts that fail for users when they are over their quotas under grace periods, because the report shifts the remaining information for that user 1 column to the right. Obviously it would cost me absolutely nothing to patch my scripts to deal with this, however the principle here is that the reports generated by GPFS should be the ones keeping consistence. Thanks Jaime ************************************ TELL US ABOUT YOUR SUCCESS STORIES https://urldefense.proofpoint.com/v2/url?u=http-3A__www.scinethpc.ca_testimonials&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzNsKbADKWCeLhA&s=TVGnqMwSWqNI1Vu1BlCcwXiVGsLUO9ZnbqlasVmT2HU&e= ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzNsKbADKWCeLhA&s=AXRwDMAVkYdwEaSFzejLQzNnS-KXoKj9GauzeEuu2H8&e= From Kevin.Buterbaugh at Vanderbilt.Edu Tue Mar 28 16:00:26 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 28 Mar 2017 15:00:26 +0000 Subject: [gpfsug-discuss] fix mmrepquota report format during grace periods In-Reply-To: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> Message-ID: <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> Hi Bob, Jaime, and GPFS team, That?s great for mmrepquota, but mmlsquota does not have a similar option AFAICT. That has really caused me grief ? for example, I?ve got a Perl script that takes mmlsquota output for a user and does two things: 1) converts it into something easier for them to parse, and 2) doesn?t display anything for the several dozen filesets they don?t have access to. That Perl script is ~300 lines and probably about a third of that is dealing with the grace period spacing issue? Kevin > On Mar 28, 2017, at 9:54 AM, Oesterlin, Robert wrote: > > Try running it with the ?-Y? option, it returns an easily to read output: > mmrepquota -Y dns > mmrepquota::HEADER:version:reserved:reserved:filesystemName:quotaType:id:name:blockUsage:blockQuota:blockLimit:blockInDoubt:blockGrace:filesUsage:filesQuota:filesLimit:filesInDoubt:filesGrace:remarks:quota:defQuota:fid:filesetname: > mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:root: > mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:users: > mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:root: > mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:users: > mmrepquota::0:1:::dns:FILESET:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off::: > mmrepquota::0:1:::dns:FILESET:1:users:0:4294967296:4294967296:0:none:1:0:0:0:none:e:on:off::: > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > > On 3/28/17, 9:47 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Jaime Pinto" wrote: > > Any chance you guys in the GPFS devel team could patch the mmrepquota > code so that during grace periods the report column for "none" would > still be replaced with >>>*ONE*<<< word? By that I mean, instead of "2 > days" for example, just print "2-days" or "2days" or "2_days", and so > on. > > I have a number of scripts that fail for users when they are over > their quotas under grace periods, because the report shifts the > remaining information for that user 1 column to the right. > > Obviously it would cost me absolutely nothing to patch my scripts to > deal with this, however the principle here is that the reports > generated by GPFS should be the ones keeping consistence. > > Thanks > Jaime > > > > > ************************************ > TELL US ABOUT YOUR SUCCESS STORIES > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.scinethpc.ca_testimonials&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzNsKbADKWCeLhA&s=TVGnqMwSWqNI1Vu1BlCcwXiVGsLUO9ZnbqlasVmT2HU&e= > ************************************ > --- > Jaime Pinto > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.ca > University of Toronto > 661 University Ave. (MaRS), Suite 1140 > Toronto, ON, M5G1M1 > P: 416-978-2755 > C: 416-505-1477 > > ---------------------------------------------------------------- > This message was sent using IMP at SciNet Consortium, University of Toronto. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzNsKbADKWCeLhA&s=AXRwDMAVkYdwEaSFzejLQzNnS-KXoKj9GauzeEuu2H8&e= > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Kevin.Buterbaugh at Vanderbilt.Edu Tue Mar 28 16:04:58 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 28 Mar 2017 15:04:58 +0000 Subject: [gpfsug-discuss] fix mmrepquota report format during grace periods In-Reply-To: <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> Message-ID: <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> Ugh! Of course, I?m wrong ? mmlsquota does support the ?-Y? option ? it?s just not documented. Why not???? Kevin > On Mar 28, 2017, at 10:00 AM, Buterbaugh, Kevin L wrote: > > Hi Bob, Jaime, and GPFS team, > > That?s great for mmrepquota, but mmlsquota does not have a similar option AFAICT. > > That has really caused me grief ? for example, I?ve got a Perl script that takes mmlsquota output for a user and does two things: 1) converts it into something easier for them to parse, and 2) doesn?t display anything for the several dozen filesets they don?t have access to. That Perl script is ~300 lines and probably about a third of that is dealing with the grace period spacing issue? > > Kevin > >> On Mar 28, 2017, at 9:54 AM, Oesterlin, Robert wrote: >> >> Try running it with the ?-Y? option, it returns an easily to read output: >> mmrepquota -Y dns >> mmrepquota::HEADER:version:reserved:reserved:filesystemName:quotaType:id:name:blockUsage:blockQuota:blockLimit:blockInDoubt:blockGrace:filesUsage:filesQuota:filesLimit:filesInDoubt:filesGrace:remarks:quota:defQuota:fid:filesetname: >> mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:root: >> mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:users: >> mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:root: >> mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:users: >> mmrepquota::0:1:::dns:FILESET:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off::: >> mmrepquota::0:1:::dns:FILESET:1:users:0:4294967296:4294967296:0:none:1:0:0:0:none:e:on:off::: >> >> Bob Oesterlin >> Sr Principal Storage Engineer, Nuance >> >> >> >> On 3/28/17, 9:47 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Jaime Pinto" wrote: >> >> Any chance you guys in the GPFS devel team could patch the mmrepquota >> code so that during grace periods the report column for "none" would >> still be replaced with >>>*ONE*<<< word? By that I mean, instead of "2 >> days" for example, just print "2-days" or "2days" or "2_days", and so >> on. >> >> I have a number of scripts that fail for users when they are over >> their quotas under grace periods, because the report shifts the >> remaining information for that user 1 column to the right. >> >> Obviously it would cost me absolutely nothing to patch my scripts to >> deal with this, however the principle here is that the reports >> generated by GPFS should be the ones keeping consistence. >> >> Thanks >> Jaime >> >> >> >> >> ************************************ >> TELL US ABOUT YOUR SUCCESS STORIES >> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.scinethpc.ca_testimonials&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzNsKbADKWCeLhA&s=TVGnqMwSWqNI1Vu1BlCcwXiVGsLUO9ZnbqlasVmT2HU&e= >> ************************************ >> --- >> Jaime Pinto >> SciNet HPC Consortium - Compute/Calcul Canada >> www.scinet.utoronto.ca - www.computecanada.ca >> University of Toronto >> 661 University Ave. (MaRS), Suite 1140 >> Toronto, ON, M5G1M1 >> P: 416-978-2755 >> C: 416-505-1477 >> >> ---------------------------------------------------------------- >> This message was sent using IMP at SciNet Consortium, University of Toronto. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzNsKbADKWCeLhA&s=AXRwDMAVkYdwEaSFzejLQzNnS-KXoKj9GauzeEuu2H8&e= >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From pinto at scinet.utoronto.ca Tue Mar 28 16:09:45 2017 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Tue, 28 Mar 2017 11:09:45 -0400 Subject: [gpfsug-discuss] fix mmrepquota report format during grace periods In-Reply-To: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> Message-ID: <20170328110945.18511am6iq6gf1dl@support.scinet.utoronto.ca> Aah! Another one of those options not so well documented or exposed: Usage: mmrepquota [-u] [-g] [-e] [-q] [-n] [-v] [-t] [--block-size {BlockSize | auto}] {-a | Device[:Fileset] ...} or mmrepquota -j [-e] [-q] [-n] [-v] [-t] [--block-size {BlockSize | auto}] {-a | Device ...} I agree, in this way it would be easier for a script to deal with fields that have spaces, using ':' as a field separator. However it mangles all the information together, making it very difficult for human eyes of a sysadmin to deal with in its original format. I'll take it under consideration for the scripts version (many of them to be revised), however the best is for the original and plain reports to have consistence. Thanks Jaime Quoting "Oesterlin, Robert" : > Try running it with the ?-Y? option, it returns an easily to read output: > mmrepquota -Y dns > mmrepquota::HEADER:version:reserved:reserved:filesystemName:quotaType:id:name:blockUsage:blockQuota:blockLimit:blockInDoubt:blockGrace:filesUsage:filesQuota:filesLimit:filesInDoubt:filesGrace:remarks:quota:defQuota:fid:filesetname: > mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:root: > mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:users: > mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:root: > mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:users: > mmrepquota::0:1:::dns:FILESET:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off::: > mmrepquota::0:1:::dns:FILESET:1:users:0:4294967296:4294967296:0:none:1:0:0:0:none:e:on:off::: > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > > On 3/28/17, 9:47 AM, "gpfsug-discuss-bounces at spectrumscale.org on > behalf of Jaime Pinto" behalf of pinto at scinet.utoronto.ca> wrote: > > Any chance you guys in the GPFS devel team could patch the mmrepquota > code so that during grace periods the report column for "none" would > still be replaced with >>>*ONE*<<< word? By that I mean, instead of "2 > days" for example, just print "2-days" or "2days" or "2_days", and so > on. > > I have a number of scripts that fail for users when they are over > their quotas under grace periods, because the report shifts the > remaining information for that user 1 column to the right. > > Obviously it would cost me absolutely nothing to patch my scripts to > deal with this, however the principle here is that the reports > generated by GPFS should be the ones keeping consistence. > > Thanks > Jaime > > > > > ************************************ > TELL US ABOUT YOUR SUCCESS STORIES > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.scinethpc.ca_testimonials&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzNsKbADKWCeLhA&s=TVGnqMwSWqNI1Vu1BlCcwXiVGsLUO9ZnbqlasVmT2HU&e= > ************************************ > --- > Jaime Pinto > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.ca > University of Toronto > 661 University Ave. (MaRS), Suite 1140 > Toronto, ON, M5G1M1 > P: 416-978-2755 > C: 416-505-1477 > > ---------------------------------------------------------------- > This message was sent using IMP at SciNet Consortium, University > of Toronto. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzNsKbADKWCeLhA&s=AXRwDMAVkYdwEaSFzejLQzNnS-KXoKj9GauzeEuu2H8&e= > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. From S.J.Thompson at bham.ac.uk Tue Mar 28 16:11:58 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Tue, 28 Mar 2017 15:11:58 +0000 Subject: [gpfsug-discuss] fix mmrepquota report format during grace periods In-Reply-To: <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> Message-ID: I thought this was because the -Y flag was going into new commands and being added to older commands during later releases. So it might be that it was added, but the docs not updated. Simon On 28/03/2017, 16:04, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Buterbaugh, Kevin L" wrote: >Ugh! Of course, I?m wrong ? mmlsquota does support the ?-Y? option ? >it?s just not documented. Why not???? > >Kevin > >> On Mar 28, 2017, at 10:00 AM, Buterbaugh, Kevin L >> wrote: >> >> Hi Bob, Jaime, and GPFS team, >> >> That?s great for mmrepquota, but mmlsquota does not have a similar >>option AFAICT. >> >> That has really caused me grief ? for example, I?ve got a Perl script >>that takes mmlsquota output for a user and does two things: 1) converts >>it into something easier for them to parse, and 2) doesn?t display >>anything for the several dozen filesets they don?t have access to. That >>Perl script is ~300 lines and probably about a third of that is dealing >>with the grace period spacing issue? >> >> Kevin >> >>> On Mar 28, 2017, at 9:54 AM, Oesterlin, Robert >>> wrote: >>> >>> Try running it with the ?-Y? option, it returns an easily to read >>>output: >>> mmrepquota -Y dns >>> >>>mmrepquota::HEADER:version:reserved:reserved:filesystemName:quotaType:id >>>:name:blockUsage:blockQuota:blockLimit:blockInDoubt:blockGrace:filesUsag >>>e:filesQuota:filesLimit:filesInDoubt:filesGrace:remarks:quota:defQuota:f >>>id:filesetname: >>> >>>mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:ro >>>ot: >>> >>>mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:us >>>ers: >>> >>>mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:ro >>>ot: >>> >>>mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:us >>>ers: >>> >>>mmrepquota::0:1:::dns:FILESET:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off: >>>:: >>> >>>mmrepquota::0:1:::dns:FILESET:1:users:0:4294967296:4294967296:0:none:1:0 >>>:0:0:none:e:on:off::: >>> >>> Bob Oesterlin >>> Sr Principal Storage Engineer, Nuance >>> >>> >>> >>> On 3/28/17, 9:47 AM, "gpfsug-discuss-bounces at spectrumscale.org on >>>behalf of Jaime Pinto" >>behalf of pinto at scinet.utoronto.ca> wrote: >>> >>> Any chance you guys in the GPFS devel team could patch the >>>mmrepquota >>> code so that during grace periods the report column for "none" would >>> >>> still be replaced with >>>*ONE*<<< word? By that I mean, instead of >>>"2 >>> days" for example, just print "2-days" or "2days" or "2_days", and >>>so >>> on. >>> >>> I have a number of scripts that fail for users when they are over >>> their quotas under grace periods, because the report shifts the >>> remaining information for that user 1 column to the right. >>> >>> Obviously it would cost me absolutely nothing to patch my scripts to >>> >>> deal with this, however the principle here is that the reports >>> generated by GPFS should be the ones keeping consistence. >>> >>> Thanks >>> Jaime >>> >>> >>> >>> >>> ************************************ >>> TELL US ABOUT YOUR SUCCESS STORIES >>> >>>https://urldefense.proofpoint.com/v2/url?u=http-3A__www.scinethpc.ca_tes >>>timonials&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDew >>>t1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzN >>>sKbADKWCeLhA&s=TVGnqMwSWqNI1Vu1BlCcwXiVGsLUO9ZnbqlasVmT2HU&e= >>> ************************************ >>> --- >>> Jaime Pinto >>> SciNet HPC Consortium - Compute/Calcul Canada >>> www.scinet.utoronto.ca - www.computecanada.ca >>> University of Toronto >>> 661 University Ave. (MaRS), Suite 1140 >>> Toronto, ON, M5G1M1 >>> P: 416-978-2755 >>> C: 416-505-1477 >>> >>> ---------------------------------------------------------------- >>> This message was sent using IMP at SciNet Consortium, University of >>>Toronto. >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> >>>https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_l >>>istinfo_gpfsug-2Ddiscuss&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0 >>>rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCI >>>ZvUgTr2CN-RqtzNsKbADKWCeLhA&s=AXRwDMAVkYdwEaSFzejLQzNnS-KXoKj9GauzeEuu2H >>>8&e= >>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Kevin.Buterbaugh at Vanderbilt.Edu Tue Mar 28 16:34:33 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 28 Mar 2017 15:34:33 +0000 Subject: [gpfsug-discuss] fix mmrepquota report format during grace periods In-Reply-To: References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> Message-ID: <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> All, Could someone(s) from the GPFS team please: 1) see that the appropriate documentation gets updated (manuals, ?-h? option to commands, the man pages for the commands, etc.)? 2) let us know what version of GPFS introduced the undocumented ?-Y? option for mmrepquota and mmlsquota? I?ve got numerous quota related scripts and I?m just curious to go back and figure out how much time I?ve wasted because I didn?t know about it. These ?unknown unknowns? bite us again? ;-) Kevin > On Mar 28, 2017, at 10:11 AM, Simon Thompson (Research Computing - IT Services) wrote: > > I thought this was because the -Y flag was going into new commands and > being added to older commands during later releases. > > So it might be that it was added, but the docs not updated. > > Simon > > On 28/03/2017, 16:04, "gpfsug-discuss-bounces at spectrumscale.org on behalf > of Buterbaugh, Kevin L" behalf of Kevin.Buterbaugh at Vanderbilt.Edu> wrote: > >> Ugh! Of course, I?m wrong ? mmlsquota does support the ?-Y? option ? >> it?s just not documented. Why not???? >> >> Kevin >> >>> On Mar 28, 2017, at 10:00 AM, Buterbaugh, Kevin L >>> wrote: >>> >>> Hi Bob, Jaime, and GPFS team, >>> >>> That?s great for mmrepquota, but mmlsquota does not have a similar >>> option AFAICT. >>> >>> That has really caused me grief ? for example, I?ve got a Perl script >>> that takes mmlsquota output for a user and does two things: 1) converts >>> it into something easier for them to parse, and 2) doesn?t display >>> anything for the several dozen filesets they don?t have access to. That >>> Perl script is ~300 lines and probably about a third of that is dealing >>> with the grace period spacing issue? >>> >>> Kevin >>> >>>> On Mar 28, 2017, at 9:54 AM, Oesterlin, Robert >>>> wrote: >>>> >>>> Try running it with the ?-Y? option, it returns an easily to read >>>> output: >>>> mmrepquota -Y dns >>>> >>>> mmrepquota::HEADER:version:reserved:reserved:filesystemName:quotaType:id >>>> :name:blockUsage:blockQuota:blockLimit:blockInDoubt:blockGrace:filesUsag >>>> e:filesQuota:filesLimit:filesInDoubt:filesGrace:remarks:quota:defQuota:f >>>> id:filesetname: >>>> >>>> mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:ro >>>> ot: >>>> >>>> mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:us >>>> ers: >>>> >>>> mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:ro >>>> ot: >>>> >>>> mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:us >>>> ers: >>>> >>>> mmrepquota::0:1:::dns:FILESET:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off: >>>> :: >>>> >>>> mmrepquota::0:1:::dns:FILESET:1:users:0:4294967296:4294967296:0:none:1:0 >>>> :0:0:none:e:on:off::: >>>> >>>> Bob Oesterlin >>>> Sr Principal Storage Engineer, Nuance >>>> >>>> >>>> >>>> On 3/28/17, 9:47 AM, "gpfsug-discuss-bounces at spectrumscale.org on >>>> behalf of Jaime Pinto" >>> behalf of pinto at scinet.utoronto.ca> wrote: >>>> >>>> Any chance you guys in the GPFS devel team could patch the >>>> mmrepquota >>>> code so that during grace periods the report column for "none" would >>>> >>>> still be replaced with >>>*ONE*<<< word? By that I mean, instead of >>>> "2 >>>> days" for example, just print "2-days" or "2days" or "2_days", and >>>> so >>>> on. >>>> >>>> I have a number of scripts that fail for users when they are over >>>> their quotas under grace periods, because the report shifts the >>>> remaining information for that user 1 column to the right. >>>> >>>> Obviously it would cost me absolutely nothing to patch my scripts to >>>> >>>> deal with this, however the principle here is that the reports >>>> generated by GPFS should be the ones keeping consistence. >>>> >>>> Thanks >>>> Jaime >>>> >>>> >>>> >>>> >>>> ************************************ >>>> TELL US ABOUT YOUR SUCCESS STORIES >>>> >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.scinethpc.ca_tes >>>> timonials&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDew >>>> t1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzN >>>> sKbADKWCeLhA&s=TVGnqMwSWqNI1Vu1BlCcwXiVGsLUO9ZnbqlasVmT2HU&e= >>>> ************************************ >>>> --- >>>> Jaime Pinto >>>> SciNet HPC Consortium - Compute/Calcul Canada >>>> www.scinet.utoronto.ca - www.computecanada.ca >>>> University of Toronto >>>> 661 University Ave. (MaRS), Suite 1140 >>>> Toronto, ON, M5G1M1 >>>> P: 416-978-2755 >>>> C: 416-505-1477 >>>> >>>> ---------------------------------------------------------------- >>>> This message was sent using IMP at SciNet Consortium, University of >>>> Toronto. >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_l >>>> istinfo_gpfsug-2Ddiscuss&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0 >>>> rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCI >>>> ZvUgTr2CN-RqtzNsKbADKWCeLhA&s=AXRwDMAVkYdwEaSFzejLQzNnS-KXoKj9GauzeEuu2H >>>> 8&e= >>>> >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From makaplan at us.ibm.com Tue Mar 28 17:14:18 2017 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 28 Mar 2017 11:14:18 -0500 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com><4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu><4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> Message-ID: Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and Programming) Reference I only found a few commands that officially are documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. But as many of you have discovered, -Y is accepted and yields "interesting" output for many of the mmXXXX commands. Moreover the output *seems to* have easily discernible patterns that can be parsed by simple programs. I believe there is no guarantee that the exact command output formats will not change from release to release, so, as a practical matter, if you're going to parse command output, you're probably better off parsing the -Y output, even if that is not officially supported. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathon.anderson at colorado.edu Tue Mar 28 17:17:29 2017 From: jonathon.anderson at colorado.edu (Jonathon A Anderson) Date: Tue, 28 Mar 2017 16:17:29 +0000 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> Message-ID: I had a bit of a run-in with a gpfs developer at a conference once when I complained about how command output changes frequently, breaking customer scripts. He was confused why we weren?t using `-Y`, and didn?t believe me that it?s simply not documented anywhere! `-Y` output is essential for interfacing programmatically with GPFS, and I don?t understand why it?s not mentioned in any of the guides or manpages. (Though apparently, according to this thead, it?s since appeared in some newer manpages.) ~jonathon On 3/28/17, 10:14 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Marc A Kaplan" wrote: Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and Programming) Reference I only found a few commands that officially are documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. But as many of you have discovered, -Y is accepted and yields "interesting" output for many of the mmXXXX commands. Moreover the output *seems to* have easily discernible patterns that can be parsed by simple programs. I believe there is no guarantee that the exact command output formats will not change from release to release, so, as a practical matter, if you're going to parse command output, you're probably better off parsing the -Y output, even if that is not officially supported. From stockf at us.ibm.com Tue Mar 28 17:21:57 2017 From: stockf at us.ibm.com (Frederick Stock) Date: Tue, 28 Mar 2017 11:21:57 -0500 Subject: [gpfsug-discuss] fix mmrepquota report format during grace periods In-Reply-To: <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com><4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu><4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> Message-ID: My understanding is that with the upcoming 4.2.3 release the -Y option will be documented for many commands, but perhaps not all. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 03/28/2017 11:35 AM Subject: Re: [gpfsug-discuss] fix mmrepquota report format during grace periods Sent by: gpfsug-discuss-bounces at spectrumscale.org All, Could someone(s) from the GPFS team please: 1) see that the appropriate documentation gets updated (manuals, ?-h? option to commands, the man pages for the commands, etc.)? 2) let us know what version of GPFS introduced the undocumented ?-Y? option for mmrepquota and mmlsquota? I?ve got numerous quota related scripts and I?m just curious to go back and figure out how much time I?ve wasted because I didn?t know about it. These ?unknown unknowns? bite us again? ;-) Kevin > On Mar 28, 2017, at 10:11 AM, Simon Thompson (Research Computing - IT Services) wrote: > > I thought this was because the -Y flag was going into new commands and > being added to older commands during later releases. > > So it might be that it was added, but the docs not updated. > > Simon > > On 28/03/2017, 16:04, "gpfsug-discuss-bounces at spectrumscale.org on behalf > of Buterbaugh, Kevin L" behalf of Kevin.Buterbaugh at Vanderbilt.Edu> wrote: > >> Ugh! Of course, I?m wrong ? mmlsquota does support the ?-Y? option ? >> it?s just not documented. Why not???? >> >> Kevin >> >>> On Mar 28, 2017, at 10:00 AM, Buterbaugh, Kevin L >>> wrote: >>> >>> Hi Bob, Jaime, and GPFS team, >>> >>> That?s great for mmrepquota, but mmlsquota does not have a similar >>> option AFAICT. >>> >>> That has really caused me grief ? for example, I?ve got a Perl script >>> that takes mmlsquota output for a user and does two things: 1) converts >>> it into something easier for them to parse, and 2) doesn?t display >>> anything for the several dozen filesets they don?t have access to. That >>> Perl script is ~300 lines and probably about a third of that is dealing >>> with the grace period spacing issue? >>> >>> Kevin >>> >>>> On Mar 28, 2017, at 9:54 AM, Oesterlin, Robert >>>> wrote: >>>> >>>> Try running it with the ?-Y? option, it returns an easily to read >>>> output: >>>> mmrepquota -Y dns >>>> >>>> mmrepquota::HEADER:version:reserved:reserved:filesystemName:quotaType:id >>>> :name:blockUsage:blockQuota:blockLimit:blockInDoubt:blockGrace:filesUsag >>>> e:filesQuota:filesLimit:filesInDoubt:filesGrace:remarks:quota:defQuota:f >>>> id:filesetname: >>>> >>>> mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:ro >>>> ot: >>>> >>>> mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:us >>>> ers: >>>> >>>> mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:ro >>>> ot: >>>> >>>> mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:us >>>> ers: >>>> >>>> mmrepquota::0:1:::dns:FILESET:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off: >>>> :: >>>> >>>> mmrepquota::0:1:::dns:FILESET:1:users:0:4294967296:4294967296:0:none:1:0 >>>> :0:0:none:e:on:off::: >>>> >>>> Bob Oesterlin >>>> Sr Principal Storage Engineer, Nuance >>>> >>>> >>>> >>>> On 3/28/17, 9:47 AM, "gpfsug-discuss-bounces at spectrumscale.org on >>>> behalf of Jaime Pinto" >>> behalf of pinto at scinet.utoronto.ca> wrote: >>>> >>>> Any chance you guys in the GPFS devel team could patch the >>>> mmrepquota >>>> code so that during grace periods the report column for "none" would >>>> >>>> still be replaced with >>>*ONE*<<< word? By that I mean, instead of >>>> "2 >>>> days" for example, just print "2-days" or "2days" or "2_days", and >>>> so >>>> on. >>>> >>>> I have a number of scripts that fail for users when they are over >>>> their quotas under grace periods, because the report shifts the >>>> remaining information for that user 1 column to the right. >>>> >>>> Obviously it would cost me absolutely nothing to patch my scripts to >>>> >>>> deal with this, however the principle here is that the reports >>>> generated by GPFS should be the ones keeping consistence. >>>> >>>> Thanks >>>> Jaime >>>> >>>> >>>> >>>> >>>> ************************************ >>>> TELL US ABOUT YOUR SUCCESS STORIES >>>> >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.scinethpc.ca_tes >>>> timonials&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDew >>>> t1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzN >>>> sKbADKWCeLhA&s=TVGnqMwSWqNI1Vu1BlCcwXiVGsLUO9ZnbqlasVmT2HU&e= >>>> ************************************ >>>> --- >>>> Jaime Pinto >>>> SciNet HPC Consortium - Compute/Calcul Canada >>>> www.scinet.utoronto.ca - www.computecanada.ca >>>> University of Toronto >>>> 661 University Ave. (MaRS), Suite 1140 >>>> Toronto, ON, M5G1M1 >>>> P: 416-978-2755 >>>> C: 416-505-1477 >>>> >>>> ---------------------------------------------------------------- >>>> This message was sent using IMP at SciNet Consortium, University of >>>> Toronto. >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_l >>>> istinfo_gpfsug-2Ddiscuss&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0 >>>> rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCI >>>> ZvUgTr2CN-RqtzNsKbADKWCeLhA&s=AXRwDMAVkYdwEaSFzejLQzNnS-KXoKj9GauzeEuu2H >>>> 8&e= >>>> >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From luis.bolinches at fi.ibm.com Tue Mar 28 17:22:04 2017 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Tue, 28 Mar 2017 16:22:04 +0000 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: References: , <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com><4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu><4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu><5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> Message-ID: An HTML attachment was scrubbed... URL: From jonathon.anderson at colorado.edu Tue Mar 28 17:24:19 2017 From: jonathon.anderson at colorado.edu (Jonathon A Anderson) Date: Tue, 28 Mar 2017 16:24:19 +0000 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> Message-ID: <99B1AC2E-91B4-4A7F-B270-8FDDC49918EA@colorado.edu> Looks great. I look forward to its maturity. ~jonathon On 3/28/17, 10:22 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Luis Bolinches" wrote: Hi While I understand the frustration of tiem that could be used otherwise. depending of what you are plannig with script wrapping I would recommend you seriously take a look to the REST API http://files.gpfsug.org/presentations/2017/Ehningen/23_-_SSUG17DE_-_Alexander_Wolf-Reber_-_Spectrum_Scale_ReST_API.pdf -- Yst?v?llisin terveisin / Kind regards / Saludos cordiales / Salutations Luis Bolinches Lab Services http://www-03.ibm.com/systems/services/labservices/ IBM Laajalahdentie 23 (main Entrance) Helsinki, 00330 Finland Phone: +358 503112585 "If you continually give you will continually have." Anonymous ----- Original message ----- From: Jonathon A Anderson Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list Cc: Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Date: Tue, Mar 28, 2017 7:17 PM I had a bit of a run-in with a gpfs developer at a conference once when I complained about how command output changes frequently, breaking customer scripts. He was confused why we weren?t using `-Y`, and didn?t believe me that it?s simply not documented anywhere! `-Y` output is essential for interfacing programmatically with GPFS, and I don?t understand why it?s not mentioned in any of the guides or manpages. (Though apparently, according to this thead, it?s since appeared in some newer manpages.) ~jonathon On 3/28/17, 10:14 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Marc A Kaplan" wrote: Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and Programming) Reference I only found a few commands that officially are documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. But as many of you have discovered, -Y is accepted and yields "interesting" output for many of the mmXXXX commands. Moreover the output *seems to* have easily discernible patterns that can be parsed by simple programs. I believe there is no guarantee that the exact command output formats will not change from release to release, so, as a practical matter, if you're going to parse command output, you're probably better off parsing the -Y output, even if that is not officially supported. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland From S.J.Thompson at bham.ac.uk Tue Mar 28 17:37:12 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Tue, 28 Mar 2017 16:37:12 +0000 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> Message-ID: I thought the header rows defined what the format of the output was. Its a bit weird as there can be multiple header rows for different content rows ... Simon From: > on behalf of Marc A Kaplan > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Tuesday, 28 March 2017 at 17:14 To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and Programming) Reference I only found a few commands that officially are documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. But as many of you have discovered, -Y is accepted and yields "interesting" output for many of the mmXXXX commands. Moreover the output *seems to* have easily discernible patterns that can be parsed by simple programs. I believe there is no guarantee that the exact command output formats will not change from release to release, so, as a practical matter, if you're going to parse command output, you're probably better off parsing the -Y output, even if that is not officially supported. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathon.anderson at colorado.edu Tue Mar 28 17:37:47 2017 From: jonathon.anderson at colorado.edu (Jonathon A Anderson) Date: Tue, 28 Mar 2017 16:37:47 +0000 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> Message-ID: Sure; but that only helps if you know the flag even exists. ~jonathon On 3/28/17, 10:37 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Simon Thompson (Research Computing - IT Services)" wrote: I thought the header rows defined what the format of the output was. Its a bit weird as there can be multiple header rows for different content rows ... Simon From: on behalf of Marc A Kaplan Reply-To: "gpfsug-discuss at spectrumscale.org" Date: Tuesday, 28 March 2017 at 17:14 To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and Programming) Reference I only found a few commands that officially are documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. But as many of you have discovered, -Y is accepted and yields "interesting" output for many of the mmXXXX commands. Moreover the output *seems to* have easily discernible patterns that can be parsed by simple programs. I believe there is no guarantee that the exact command output formats will not change from release to release, so, as a practical matter, if you're going to parse command output, you're probably better off parsing the -Y output, even if that is not officially supported. From Kevin.Buterbaugh at Vanderbilt.Edu Tue Mar 28 17:47:14 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 28 Mar 2017 16:47:14 +0000 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> Message-ID: <53E9C309-3498-40FB-8018-8CABD9C6C316@vanderbilt.edu> Agreed. From what has been said on this thread about ?-Y? being unsupported and the command output could change from release to release ? well, that?s a ?known unknown? that can be dealt with. But the fact that ?-Y? was completely undocumented (at least as far as mmrepquota / mmlsquota are concerned) makes ?-Y? one of those ?unknown unknowns?? ;-) Kevin > On Mar 28, 2017, at 11:37 AM, Jonathon A Anderson wrote: > > Sure; but that only helps if you know the flag even exists. > > ~jonathon > > > On 3/28/17, 10:37 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Simon Thompson (Research Computing - IT Services)" wrote: > > I thought the header rows defined what the format of the output was. Its a bit weird as there can be multiple header rows for different content rows ... > > > Simon > > > From: on behalf of Marc A Kaplan > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Tuesday, 28 March 2017 at 17:14 > To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! > > > > Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and Programming) Reference I only found a few commands that officially are documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. > > But as many of you have discovered, -Y is accepted and yields "interesting" output for many of the mmXXXX commands. > Moreover the output *seems to* have easily discernible patterns that can be parsed by simple programs. > > I believe there is no guarantee that the exact command output formats will not change from release to release, so, as a practical matter, if you're going to parse command output, you're probably better off parsing the -Y output, > even if that is not officially supported. > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From S.J.Thompson at bham.ac.uk Tue Mar 28 17:51:12 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Tue, 28 Mar 2017 16:51:12 +0000 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> Message-ID: Come to one of the user group events going on around the world in the next few months. NERSC, Berkeley, USA, 4th/5th April: https://www.nersc.gov/research-and-development/data-analytics/spectrum-user -group-meeting/ *** REGISTRATION CLOSES TOMORROW 29th *** Sydney, Australia, 27th April: https://www.eventbrite.com/e/spectrum-scale-user-group-australia-gpfsugaus- april-2017-tickets-29136246297 Manchester, UK, 9th/10th May: https://www.eventbrite.com/e/spectrum-scalegpfs-user-group-spring-2017-regi stration-32113696932 I know the presence of the -Y flag has been discussed there, so its a great place to pick up tips! Simon On 28/03/2017, 17:37, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Jonathon A Anderson" wrote: >Sure; but that only helps if you know the flag even exists. > >~jonathon > > >On 3/28/17, 10:37 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf >of Simon Thompson (Research Computing - IT Services)" >S.J.Thompson at bham.ac.uk> wrote: > > I thought the header rows defined what the format of the output was. >Its a bit weird as there can be multiple header rows for different >content rows ... > > > Simon > > > From: on behalf of Marc A >Kaplan > Reply-To: "gpfsug-discuss at spectrumscale.org" > > Date: Tuesday, 28 March 2017 at 17:14 > To: "gpfsug-discuss at spectrumscale.org" > > Subject: Re: [gpfsug-discuss] -Y option for many commands, precious >few officially! > > > > Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and >Programming) Reference I only found a few commands that officially are >documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. > > But as many of you have discovered, -Y is accepted and yields >"interesting" output for many of the mmXXXX commands. > Moreover the output *seems to* have easily discernible patterns that >can be parsed by simple programs. > > I believe there is no guarantee that the exact command output formats >will not change from release to release, so, as a practical matter, if >you're going to parse command output, you're probably better off parsing >the -Y output, > even if that is not officially supported. > > > > > >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss From carlz at us.ibm.com Tue Mar 28 18:02:32 2017 From: carlz at us.ibm.com (Carl Zetie) Date: Tue, 28 Mar 2017 17:02:32 +0000 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From jonathon.anderson at colorado.edu Tue Mar 28 18:08:20 2017 From: jonathon.anderson at colorado.edu (Jonathon A Anderson) Date: Tue, 28 Mar 2017 17:08:20 +0000 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> Message-ID: <67E4C386-8884-4E92-B262-089FDFA32C94@colorado.edu> > I know the presence of the -Y flag has been discussed there, so it?s a great place to pick up tips! Sure, and I ?picked up this tip? at a community event; but really, it shouldn?t have to be picked up as arcana. It should be documented. Glad to hear that that?s the plan. ~jonathon On 3/28/17, 10:51 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Simon Thompson (Research Computing - IT Services)" wrote: Come to one of the user group events going on around the world in the next few months. NERSC, Berkeley, USA, 4th/5th April: https://www.nersc.gov/research-and-development/data-analytics/spectrum-user -group-meeting/ *** REGISTRATION CLOSES TOMORROW 29th *** Sydney, Australia, 27th April: https://www.eventbrite.com/e/spectrum-scale-user-group-australia-gpfsugaus- april-2017-tickets-29136246297 Manchester, UK, 9th/10th May: https://www.eventbrite.com/e/spectrum-scalegpfs-user-group-spring-2017-regi stration-32113696932 I know the presence of the -Y flag has been discussed there, so its a great place to pick up tips! Simon On 28/03/2017, 17:37, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Jonathon A Anderson" wrote: >Sure; but that only helps if you know the flag even exists. > >~jonathon > > >On 3/28/17, 10:37 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf >of Simon Thompson (Research Computing - IT Services)" >S.J.Thompson at bham.ac.uk> wrote: > > I thought the header rows defined what the format of the output was. >Its a bit weird as there can be multiple header rows for different >content rows ... > > > Simon > > > From: on behalf of Marc A >Kaplan > Reply-To: "gpfsug-discuss at spectrumscale.org" > > Date: Tuesday, 28 March 2017 at 17:14 > To: "gpfsug-discuss at spectrumscale.org" > > Subject: Re: [gpfsug-discuss] -Y option for many commands, precious >few officially! > > > > Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and >Programming) Reference I only found a few commands that officially are >documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. > > But as many of you have discovered, -Y is accepted and yields >"interesting" output for many of the mmXXXX commands. > Moreover the output *seems to* have easily discernible patterns that >can be parsed by simple programs. > > I believe there is no guarantee that the exact command output formats >will not change from release to release, so, as a practical matter, if >you're going to parse command output, you're probably better off parsing >the -Y output, > even if that is not officially supported. > > > > > >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From aleciarm at us.ibm.com Tue Mar 28 18:17:11 2017 From: aleciarm at us.ibm.com (Alecia A Ramsay) Date: Tue, 28 Mar 2017 12:17:11 -0500 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: References: Message-ID: For the benefit of those on digest, here's Carl's list again in plain text (I hope). I have also notified our documentation team that this is of great interest to clients, so they are aware. In the 4.2.3 draft documentation, the -y option information has been added to a number of commands: -Y option Added the -Y option to the following commands: mmblock mmhealth mmlsfileset mmlsnodeclass mmnetverify mmcloudgateway mmkeyserv mmlsfs mmlsnsd mmnfs mmdf mmlscluster mmlslicense mmlspolicy mmrepquota mmdiag mmlsconfig mmlsmgr mmlsquota mmsmb mmgetstate mmlsdisk mmlsmount mmlssnapshot mmuserauth Alecia A. Ramsay, PMP? Program Manager, New Technology Introduction IBM Systems - Storage aleciarm at us.ibm.com work: 919-435-6494; mobile: 651-260-4928 http://ibm.biz/NewTechnologyIntroduction From: Carl Zetie/Fairfax/IBM at IBMUS To: gpfsug-discuss at spectrumscale.org Date: 03/28/2017 01:03 PM Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Sent by: gpfsug-discuss-bounces at spectrumscale.org I checked the draft documentation for 4.2.3, and here is the list of what is planned. As usual -- please remember this is an Intention not a commitment, and subject to change between now and the release. regards, Carl -Y option Added the -Y option to the following commands: mmblock mmhealth mmlsfileset mmlsnodeclass mmnetverify mmcloudgateway mmkeyserv mmlsfs mmlsnsd mmnfs mmdf mmlscluster mmlslicense mmlspolicy mmrepquota mmdiag mmlsconfig mmlsmgr mmlsquota mmsmb mmgetstate mmlsdisk mmlsmount mmlssnapshot mmuserauth Carl Zetie Offering Manager for Spectrum Scale, IBM carlz at us.ibm.com ----- Original message ----- From: gpfsug-discuss-request at spectrumscale.org Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Cc: Subject: gpfsug-discuss Digest, Vol 62, Issue 70 Date: Tue, Mar 28, 2017 12:38 PM Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: -Y option for many commands, precious few officially! (Luis Bolinches) 2. Re: -Y option for many commands, precious few officially! (Jonathon A Anderson) 3. Re: -Y option for many commands, precious few officially! (Simon Thompson (Research Computing - IT Services)) 4. Re: -Y option for many commands, precious few officially! (Jonathon A Anderson) ---------------------------------------------------------------------- Message: 1 Date: Tue, 28 Mar 2017 16:22:04 +0000 From: "Luis Bolinches" To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Message-ID: Content-Type: text/plain; charset="us-ascii" An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20170328/694119bd/attachment-0001.html > ------------------------------ Message: 2 Date: Tue, 28 Mar 2017 16:24:19 +0000 From: Jonathon A Anderson To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Message-ID: <99B1AC2E-91B4-4A7F-B270-8FDDC49918EA at colorado.edu> Content-Type: text/plain; charset="utf-8" Looks great. I look forward to its maturity. ~jonathon On 3/28/17, 10:22 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Luis Bolinches" wrote: Hi While I understand the frustration of tiem that could be used otherwise. depending of what you are plannig with script wrapping I would recommend you seriously take a look to the REST API http://files.gpfsug.org/presentations/2017/Ehningen/23_-_SSUG17DE_-_Alexander_Wolf-Reber_-_Spectrum_Scale_ReST_API.pdf -- Yst?v?llisin terveisin / Kind regards / Saludos cordiales / Salutations Luis Bolinches Lab Services http://www-03.ibm.com/systems/services/labservices/ IBM Laajalahdentie 23 (main Entrance) Helsinki, 00330 Finland Phone: +358 503112585 "If you continually give you will continually have." Anonymous ----- Original message ----- From: Jonathon A Anderson Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list Cc: Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Date: Tue, Mar 28, 2017 7:17 PM I had a bit of a run-in with a gpfs developer at a conference once when I complained about how command output changes frequently, breaking customer scripts. He was confused why we weren?t using `-Y`, and didn?t believe me that it?s simply not documented anywhere! `-Y` output is essential for interfacing programmatically with GPFS, and I don?t understand why it?s not mentioned in any of the guides or manpages. (Though apparently, according to this thead, it?s since appeared in some newer manpages.) ~jonathon On 3/28/17, 10:14 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Marc A Kaplan" wrote: Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and Programming) Reference I only found a few commands that officially are documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. But as many of you have discovered, -Y is accepted and yields "interesting" output for many of the mmXXXX commands. Moreover the output *seems to* have easily discernible patterns that can be parsed by simple programs. I believe there is no guarantee that the exact command output formats will not change from release to release, so, as a practical matter, if you're going to parse command output, you're probably better off parsing the -Y output, even if that is not officially supported. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland ------------------------------ Message: 3 Date: Tue, 28 Mar 2017 16:37:12 +0000 From: "Simon Thompson (Research Computing - IT Services)" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Message-ID: Content-Type: text/plain; charset="us-ascii" I thought the header rows defined what the format of the output was. Its a bit weird as there can be multiple header rows for different content rows ... Simon From: > on behalf of Marc A Kaplan > Reply-To: "gpfsug-discuss at spectrumscale.org< mailto:gpfsug-discuss at spectrumscale.org>" > Date: Tuesday, 28 March 2017 at 17:14 To: "gpfsug-discuss at spectrumscale.org< mailto:gpfsug-discuss at spectrumscale.org>" > Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and Programming) Reference I only found a few commands that officially are documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. But as many of you have discovered, -Y is accepted and yields "interesting" output for many of the mmXXXX commands. Moreover the output *seems to* have easily discernible patterns that can be parsed by simple programs. I believe there is no guarantee that the exact command output formats will not change from release to release, so, as a practical matter, if you're going to parse command output, you're probably better off parsing the -Y output, even if that is not officially supported. -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20170328/2c551c41/attachment-0001.html > ------------------------------ Message: 4 Date: Tue, 28 Mar 2017 16:37:47 +0000 From: Jonathon A Anderson To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Message-ID: Content-Type: text/plain; charset="utf-8" Sure; but that only helps if you know the flag even exists. ~jonathon On 3/28/17, 10:37 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Simon Thompson (Research Computing - IT Services)" wrote: I thought the header rows defined what the format of the output was. Its a bit weird as there can be multiple header rows for different content rows ... Simon From: on behalf of Marc A Kaplan Reply-To: "gpfsug-discuss at spectrumscale.org" Date: Tuesday, 28 March 2017 at 17:14 To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and Programming) Reference I only found a few commands that officially are documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. But as many of you have discovered, -Y is accepted and yields "interesting" output for many of the mmXXXX commands. Moreover the output *seems to* have easily discernible patterns that can be parsed by simple programs. I believe there is no guarantee that the exact command output formats will not change from release to release, so, as a practical matter, if you're going to parse command output, you're probably better off parsing the -Y output, even if that is not officially supported. ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 62, Issue 70 ********************************************** _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From ckrafft at de.ibm.com Wed Mar 29 12:34:29 2017 From: ckrafft at de.ibm.com (Christoph Krafft) Date: Wed, 29 Mar 2017 12:34:29 +0100 Subject: [gpfsug-discuss] GPFS / Spectrum Scale support with x86 SUSE SLES 12 SP2 to be expected when? Message-ID: Folks, does anyone have any tentative information when the above SUSE OS Level will be supported? Thank you in advance for hints and tips! Mit freundlichen Gr??en / Sincerely Christoph Krafft Client Technical Specialist - Power Systems, IBM Systems Certified IT Specialist @ The Open Group Phone: +49 (0) 7034 643 2171 IBM Deutschland GmbH Mobile: +49 (0) 160 97 81 86 12 Am Weiher 24 Email: ckrafft at de.ibm.com 65451 Kelsterbach Germany IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter Gesch?ftsf?hrung: Martina Koederitz (Vorsitzende), Nicole Reimer, Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Stefan Lutz Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecblank.gif Type: image/gif Size: 45 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 16373446.gif Type: image/gif Size: 1851 bytes Desc: not available URL: From aaron.knister at gmail.com Wed Mar 29 14:18:25 2017 From: aaron.knister at gmail.com (Aaron Knister) Date: Wed, 29 Mar 2017 09:18:25 -0400 Subject: [gpfsug-discuss] GPFS / Spectrum Scale support with x86 SUSE SLES 12 SP2 to be expected when? In-Reply-To: References: Message-ID: <57030974-8C80-4808-87F1-3540E1271D6E@gmail.com> I could be wrong but I believe it is currently supported (as in it works, at least that's my recollection). Did you try with a particular version that wouldn't work? Sent from my iPhone > On Mar 29, 2017, at 07:34, Christoph Krafft wrote: > > Folks, > > does anyone have any tentative information when the above SUSE OS Level will be supported? > > Thank you in advance for hints and tips! > > > Mit freundlichen Gr??en / Sincerely > > Christoph Krafft > > Client Technical Specialist - Power Systems, IBM Systems > Certified IT Specialist @ The Open Group > > > Phone: +49 (0) 7034 643 2171 IBM Deutschland GmbH > <16373446.gif> > Mobile: +49 (0) 160 97 81 86 12 Am Weiher 24 > Email: ckrafft at de.ibm.com 65451 Kelsterbach > Germany > IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter > Gesch?ftsf?hrung: Martina Koederitz (Vorsitzende), Nicole Reimer, Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Stefan Lutz > Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Wed Mar 29 14:55:17 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Wed, 29 Mar 2017 09:55:17 -0400 Subject: [gpfsug-discuss] GPFS / Spectrum Scale support with x86 SUSE SLES 12 SP2 to be expected when? In-Reply-To: <57030974-8C80-4808-87F1-3540E1271D6E@gmail.com> References: <57030974-8C80-4808-87F1-3540E1271D6E@gmail.com> Message-ID: Just to make sure I wasn't mis-remembering I just installed 4.1.1.10 on a SLES12 SP1 machine (it has a SLES12 SP2 kernel though), built and loaded the kernel modules. There could be something significantly different in user space between SLES12 SP1 and SP2 that prevents RPM installation but I'd be surprised. -Aaron On 3/29/17 9:18 AM, Aaron Knister wrote: > I could be wrong but I believe it is currently supported (as in it > works, at least that's my recollection). Did you try with a particular > version that wouldn't work? > > Sent from my iPhone > > On Mar 29, 2017, at 07:34, Christoph Krafft > wrote: > >> Folks, >> >> does anyone have any tentative information when the above SUSE OS >> Level will be supported? >> >> Thank you in advance for hints and tips! >> >> >> Mit freundlichen Gr??en / Sincerely >> >> *Christoph Krafft* >> >> Client Technical Specialist - Power Systems, IBM Systems >> Certified IT Specialist @ The Open Group >> >> ------------------------------------------------------------------------ >> >> Phone: +49 (0) 7034 643 2171 IBM Deutschland GmbH >> <16373446.gif> >> Mobile: +49 (0) 160 97 81 86 12 Am Weiher 24 >> Email: ckrafft at de.ibm.com 65451 Kelsterbach >> Germany >> ------------------------------------------------------------------------ >> IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter >> Gesch?ftsf?hrung: Martina Koederitz (Vorsitzende), Nicole Reimer, >> Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Stefan Lutz >> Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht >> Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From S.J.Thompson at bham.ac.uk Wed Mar 29 15:01:33 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Wed, 29 Mar 2017 14:01:33 +0000 Subject: [gpfsug-discuss] GPFS / Spectrum Scale support with x86 SUSE SLES 12 SP2 to be expected when? In-Reply-To: References: <57030974-8C80-4808-87F1-3540E1271D6E@gmail.com>, Message-ID: I think the question was supported, not works though. We've just been round something similar with TSM support, works fine on el7.3, but not supported, the docs actually said 7.1 or higher. We still got support and IBM now officially support 7.3, but I can see people may not want to deploy on something unsupported. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Aaron Knister [aaron.s.knister at nasa.gov] Sent: 29 March 2017 14:55 To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] GPFS / Spectrum Scale support with x86 SUSE SLES 12 SP2 to be expected when? Just to make sure I wasn't mis-remembering I just installed 4.1.1.10 on a SLES12 SP1 machine (it has a SLES12 SP2 kernel though), built and loaded the kernel modules. There could be something significantly different in user space between SLES12 SP1 and SP2 that prevents RPM installation but I'd be surprised. -Aaron On 3/29/17 9:18 AM, Aaron Knister wrote: > I could be wrong but I believe it is currently supported (as in it > works, at least that's my recollection). Did you try with a particular > version that wouldn't work? > > Sent from my iPhone > > On Mar 29, 2017, at 07:34, Christoph Krafft > wrote: > >> Folks, >> >> does anyone have any tentative information when the above SUSE OS >> Level will be supported? >> >> Thank you in advance for hints and tips! >> >> >> Mit freundlichen Gr??en / Sincerely >> >> *Christoph Krafft* >> >> Client Technical Specialist - Power Systems, IBM Systems >> Certified IT Specialist @ The Open Group >> >> ------------------------------------------------------------------------ >> >> Phone: +49 (0) 7034 643 2171 IBM Deutschland GmbH >> <16373446.gif> >> Mobile: +49 (0) 160 97 81 86 12 Am Weiher 24 >> Email: ckrafft at de.ibm.com 65451 Kelsterbach >> Germany >> ------------------------------------------------------------------------ >> IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter >> Gesch?ftsf?hrung: Martina Koederitz (Vorsitzende), Nicole Reimer, >> Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Stefan Lutz >> Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht >> Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From aaron.s.knister at nasa.gov Wed Mar 29 15:17:19 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Wed, 29 Mar 2017 10:17:19 -0400 Subject: [gpfsug-discuss] GPFS / Spectrum Scale support with x86 SUSE SLES 12 SP2 to be expected when? In-Reply-To: References: <57030974-8C80-4808-87F1-3540E1271D6E@gmail.com> Message-ID: Ah, good point. Support means different things to different people, I suppose. The FAQ does say this which I'd not seen before: "IBM Spectrum Scale is not supported on SLES 12 SP2" That's actually a problem for me so I second Christoph's question, then. -Aaron On 3/29/17 10:01 AM, Simon Thompson (Research Computing - IT Services) wrote: > I think the question was supported, not works though. > > We've just been round something similar with TSM support, works fine on el7.3, but not supported, the docs actually said 7.1 or higher. We still got support and IBM now officially support 7.3, but I can see people may not want to deploy on something unsupported. > > Simon > ________________________________________ > From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Aaron Knister [aaron.s.knister at nasa.gov] > Sent: 29 March 2017 14:55 > To: gpfsug-discuss at spectrumscale.org > Subject: Re: [gpfsug-discuss] GPFS / Spectrum Scale support with x86 SUSE SLES 12 SP2 to be expected when? > > Just to make sure I wasn't mis-remembering I just installed 4.1.1.10 on > a SLES12 SP1 machine (it has a SLES12 SP2 kernel though), built and > loaded the kernel modules. There could be something significantly > different in user space between SLES12 SP1 and SP2 that prevents RPM > installation but I'd be surprised. > > -Aaron > > On 3/29/17 9:18 AM, Aaron Knister wrote: >> I could be wrong but I believe it is currently supported (as in it >> works, at least that's my recollection). Did you try with a particular >> version that wouldn't work? >> >> Sent from my iPhone >> >> On Mar 29, 2017, at 07:34, Christoph Krafft > > wrote: >> >>> Folks, >>> >>> does anyone have any tentative information when the above SUSE OS >>> Level will be supported? >>> >>> Thank you in advance for hints and tips! >>> >>> >>> Mit freundlichen Gr??en / Sincerely >>> >>> *Christoph Krafft* >>> >>> Client Technical Specialist - Power Systems, IBM Systems >>> Certified IT Specialist @ The Open Group >>> >>> ------------------------------------------------------------------------ >>> >>> Phone: +49 (0) 7034 643 2171 IBM Deutschland GmbH >>> <16373446.gif> >>> Mobile: +49 (0) 160 97 81 86 12 Am Weiher 24 >>> Email: ckrafft at de.ibm.com 65451 Kelsterbach >>> Germany >>> ------------------------------------------------------------------------ >>> IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter >>> Gesch?ftsf?hrung: Martina Koederitz (Vorsitzende), Nicole Reimer, >>> Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Stefan Lutz >>> Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht >>> Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 >>> >>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From abeattie at au1.ibm.com Wed Mar 29 22:53:06 2017 From: abeattie at au1.ibm.com (Andrew Beattie) Date: Wed, 29 Mar 2017 21:53:06 +0000 Subject: [gpfsug-discuss] GPFS / Spectrum Scale support with x86 SUSE SLES 12 SP2 to be expected when? In-Reply-To: References: , <57030974-8C80-4808-87F1-3540E1271D6E@gmail.com> Message-ID: An HTML attachment was scrubbed... URL: From Greg.Lehmann at csiro.au Thu Mar 30 00:12:35 2017 From: Greg.Lehmann at csiro.au (Greg.Lehmann at csiro.au) Date: Wed, 29 Mar 2017 23:12:35 +0000 Subject: [gpfsug-discuss] question on viewing block distribution across NSDs Message-ID: <2c0ba2b9466b42b4b3e5b3202ab26825@exch1-cdc.nexus.csiro.au> Hi All, I added some NSDs to an existing filesystem and ran mmrestripefs. I was sort of curious to see what the distribution looked like before and after the restripe. Is there any way of looking at it? Cheers, Greg. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Thu Mar 30 00:45:23 2017 From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]) Date: Wed, 29 Mar 2017 23:45:23 +0000 Subject: [gpfsug-discuss] question on viewing block distribution across NSDs References: question on viewing block distribution across NSDs Message-ID: <5F910253243E6A47B81A9A2EB424BBA101F5998D@NDMSMBX404.ndc.nasa.gov> Hi Greg, You could run an mmdf which will show you how full each NSD is. I'm not sure how to look back in time though to see the fs before the restripe. Do you perhaps have a gpfs.snap you took somewhat recently before the restripe? Maybe an internaldump in /tmp/mmfs somewhere? From: greg.lehmann at csiro.au Sent: 3/29/17, 19:21 To: gpfsug main discussion list Subject: [gpfsug-discuss] question on viewing block distribution across NSDs Hi All, I added some NSDs to an existing filesystem and ran mmrestripefs. I was sort of curious to see what the distribution looked like before and after the restripe. Is there any way of looking at it? Cheers, Greg. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Greg.Lehmann at csiro.au Thu Mar 30 00:51:37 2017 From: Greg.Lehmann at csiro.au (Greg.Lehmann at csiro.au) Date: Wed, 29 Mar 2017 23:51:37 +0000 Subject: [gpfsug-discuss] question on viewing block distribution across NSDs In-Reply-To: <5F910253243E6A47B81A9A2EB424BBA101F5998D@NDMSMBX404.ndc.nasa.gov> References: question on viewing block distribution across NSDs <5F910253243E6A47B81A9A2EB424BBA101F5998D@NDMSMBX404.ndc.nasa.gov> Message-ID: <3ec38bd65cad452f974c1d6c0d762853@exch1-cdc.nexus.csiro.au> Thanks. I don't have a snap. I'll keep that in mind for next time I do this. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] Sent: Thursday, 30 March 2017 9:45 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] question on viewing block distribution across NSDs Hi Greg, You could run an mmdf which will show you how full each NSD is. I'm not sure how to look back in time though to see the fs before the restripe. Do you perhaps have a gpfs.snap you took somewhat recently before the restripe? Maybe an internaldump in /tmp/mmfs somewhere? From: greg.lehmann at csiro.au Sent: 3/29/17, 19:21 To: gpfsug main discussion list Subject: [gpfsug-discuss] question on viewing block distribution across NSDs Hi All, I added some NSDs to an existing filesystem and ran mmrestripefs. I was sort of curious to see what the distribution looked like before and after the restripe. Is there any way of looking at it? Cheers, Greg. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Thu Mar 30 00:55:02 2017 From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]) Date: Wed, 29 Mar 2017 23:55:02 +0000 Subject: [gpfsug-discuss] question on viewing block distribution across NSDs References: [gpfsug-discuss] question on viewing block distribution across NSDs Message-ID: <5F910253243E6A47B81A9A2EB424BBA101F59A0A@NDMSMBX404.ndc.nasa.gov> I don't necessarily think you need to run a snap prior, just the output of mmdf should be enough. Something to keep in mind that I should have said before-- an mmdf can be stressful on your system particularly if you have spinning disk for your metadata. We're fortunate enough to have all flash for our metadata and I tend to take it for granted some times :) From: greg.lehmann at csiro.au Sent: 3/29/17, 19:52 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] question on viewing block distribution across NSDs Thanks. I don?t have a snap. I?ll keep that in mind for next time I do this. From: mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] Sent: Thursday, 30 March 2017 9:45 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] question on viewing block distribution across NSDs Hi Greg, You could run an mmdf which will show you how full each NSD is. I'm not sure how to look back in time though to see the fs before the restripe. Do you perhaps have a gpfs.snap you took somewhat recently before the restripe? Maybe an internaldump in /tmp/mmfs somewhere? From: greg.lehmann at csiro.au Sent: 3/29/17, 19:21 To: gpfsug main discussion list Subject: [gpfsug-discuss] question on viewing block distribution across NSDs Hi All, I added some NSDs to an existing filesystem and ran mmrestripefs. I was sort of curious to see what the distribution looked like before and after the restripe. Is there any way of looking at it? Cheers, Greg. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Greg.Lehmann at csiro.au Thu Mar 30 00:59:41 2017 From: Greg.Lehmann at csiro.au (Greg.Lehmann at csiro.au) Date: Wed, 29 Mar 2017 23:59:41 +0000 Subject: [gpfsug-discuss] question on viewing block distribution across NSDs In-Reply-To: <5F910253243E6A47B81A9A2EB424BBA101F59A0A@NDMSMBX404.ndc.nasa.gov> References: [gpfsug-discuss] question on viewing block distribution across NSDs <5F910253243E6A47B81A9A2EB424BBA101F59A0A@NDMSMBX404.ndc.nasa.gov> Message-ID: I was going to keep mmdf in mind, not gpfs.snap. I will now also keep in mind that mmdf can have an impact as at present we have spinning disk for metadata. The system I am playing around on is not production yet, so I am safe for the moment. Thanks again. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] Sent: Thursday, 30 March 2017 9:55 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] question on viewing block distribution across NSDs I don't necessarily think you need to run a snap prior, just the output of mmdf should be enough. Something to keep in mind that I should have said before-- an mmdf can be stressful on your system particularly if you have spinning disk for your metadata. We're fortunate enough to have all flash for our metadata and I tend to take it for granted some times :) From: greg.lehmann at csiro.au Sent: 3/29/17, 19:52 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] question on viewing block distribution across NSDs Thanks. I don?t have a snap. I?ll keep that in mind for next time I do this. From: mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] Sent: Thursday, 30 March 2017 9:45 AM To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] question on viewing block distribution across NSDs Hi Greg, You could run an mmdf which will show you how full each NSD is. I'm not sure how to look back in time though to see the fs before the restripe. Do you perhaps have a gpfs.snap you took somewhat recently before the restripe? Maybe an internaldump in /tmp/mmfs somewhere? From: greg.lehmann at csiro.au Sent: 3/29/17, 19:21 To: gpfsug main discussion list Subject: [gpfsug-discuss] question on viewing block distribution across NSDs Hi All, I added some NSDs to an existing filesystem and ran mmrestripefs. I was sort of curious to see what the distribution looked like before and after the restripe. Is there any way of looking at it? Cheers, Greg. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kums at us.ibm.com Thu Mar 30 14:04:09 2017 From: kums at us.ibm.com (Kumaran Rajaram) Date: Thu, 30 Mar 2017 08:04:09 -0500 Subject: [gpfsug-discuss] question on viewing block distribution across NSDs In-Reply-To: References: [gpfsug-discuss] question on viewing block distribution acrossNSDs <5F910253243E6A47B81A9A2EB424BBA101F59A0A@NDMSMBX404.ndc.nasa.gov> Message-ID: Hi, Yes, you could use "mmdf" to obtain file-system "usage" across the NSDs (comprising the file-system). If you want to obtain "data block distribution corresponding to a file across the NSDs", then there is a utility "mmgetlocation" in /usr/lpp/mmfs/samples/fpo that can be used to get file-data-blocks to NSD mapping. Example: # File-system comprises of single storage pool, all NSDs configured as dataAndMetadata, -m 1 -r 1, FS block-size=2MiB # mmlsfs gpfs1b | grep 'Block size' -B 2097152 Block size # The file-system is comprised of 10 x dataAndMetadata NSDs # mmlsdisk gpfs1b | grep DMD | wc -l 10 # Create a sample file that is 40MiB (20 data blocks) /mnt/sw/benchmarks/gpfsperf/gpfsperf create seq -r 2m -n 40m /mnt/gpfs1b/temp_dir/lf.s.1 # File size is 40 MiB (09:52:49) c25m3n07:~ # ls -lh /mnt/gpfs1b/temp_dir/lf.s.1 -rw-r--r-- 1 root root 40M Mar 17 09:52 /mnt/gpfs1b/temp_dir/lf.s.1 (09:52:54) c25m3n07:~ # du -sh /mnt/gpfs1b/temp_dir/lf.s.1 40M /mnt/gpfs1b/temp_dir/lf.s.1 # Verified through mmgetlocation that the file data blocks is uniformly striped across all the dataAndMetadata NSDs, with each NSD containing 2 file data blocks # In the output below, "DMD_NSDX" is name of the NSDs. (09:53:00) c25m3n07:~ # /usr/lpp/mmfs/samples/fpo/mmgetlocation -f /mnt/gpfs1b/temp_dir/lf.s.1 [FILE INFO] ------------------------------------------------------------------------ blockSize 2 MB blockGroupFactor 1 metadataBlockSize 2M writeAffinityDepth 0 flags: data replication: 1 max 2 storage pool name: system metadata replication: 1 max 2 Chunk 0 (offset 0) is located at disks: [ DMD_NSD09 c25m3n07-ib,c25m3n08-ib ] Chunk 1 (offset 2097152) is located at disks: [ DMD_NSD10 c25m3n08-ib,c25m3n07-ib ] Chunk 2 (offset 4194304) is located at disks: [ DMD_NSD01 c25m3n07-ib,c25m3n08-ib ] Chunk 3 (offset 6291456) is located at disks: [ DMD_NSD02 c25m3n08-ib,c25m3n07-ib ] Chunk 4 (offset 8388608) is located at disks: [ DMD_NSD03 c25m3n07-ib,c25m3n08-ib ] Chunk 5 (offset 10485760) is located at disks: [ DMD_NSD04 c25m3n08-ib,c25m3n07-ib ] Chunk 6 (offset 12582912) is located at disks: [ DMD_NSD05 c25m3n07-ib,c25m3n08-ib ] Chunk 7 (offset 14680064) is located at disks: [ DMD_NSD06 c25m3n08-ib,c25m3n07-ib ] Chunk 8 (offset 16777216) is located at disks: [ DMD_NSD07 c25m3n07-ib,c25m3n08-ib ] Chunk 9 (offset 18874368) is located at disks: [ DMD_NSD08 c25m3n08-ib,c25m3n07-ib ] Chunk 10 (offset 20971520) is located at disks: [ DMD_NSD09 c25m3n07-ib,c25m3n08-ib ] Chunk 11 (offset 23068672) is located at disks: [ DMD_NSD10 c25m3n08-ib,c25m3n07-ib ] Chunk 12 (offset 25165824) is located at disks: [ DMD_NSD01 c25m3n07-ib,c25m3n08-ib ] Chunk 13 (offset 27262976) is located at disks: [ DMD_NSD02 c25m3n08-ib,c25m3n07-ib ] Chunk 14 (offset 29360128) is located at disks: [ DMD_NSD03 c25m3n07-ib,c25m3n08-ib ] Chunk 15 (offset 31457280) is located at disks: [ DMD_NSD04 c25m3n08-ib,c25m3n07-ib ] Chunk 16 (offset 33554432) is located at disks: [ DMD_NSD05 c25m3n07-ib,c25m3n08-ib ] Chunk 17 (offset 35651584) is located at disks: [ DMD_NSD06 c25m3n08-ib,c25m3n07-ib ] Chunk 18 (offset 37748736) is located at disks: [ DMD_NSD07 c25m3n07-ib,c25m3n08-ib ] Chunk 19 (offset 39845888) is located at disks: [ DMD_NSD08 c25m3n08-ib,c25m3n07-ib ] [SUMMARY INFO] ---------------------------------------------------------------------------------------------------------- Replica num Nodename TotalChunkst Replica 1 : c25m3n07-ib,c25m3n08-ib: Total : 10 Replica 1 : c25m3n08-ib,c25m3n07-ib: Total : 10 Best Regards, -Kums From: To: Date: 03/29/2017 08:00 PM Subject: Re: [gpfsug-discuss] question on viewing block distribution across NSDs Sent by: gpfsug-discuss-bounces at spectrumscale.org I was going to keep mmdf in mind, not gpfs.snap. I will now also keep in mind that mmdf can have an impact as at present we have spinning disk for metadata. The system I am playing around on is not production yet, so I am safe for the moment. Thanks again. From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] Sent: Thursday, 30 March 2017 9:55 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] question on viewing block distribution across NSDs I don't necessarily think you need to run a snap prior, just the output of mmdf should be enough. Something to keep in mind that I should have said before-- an mmdf can be stressful on your system particularly if you have spinning disk for your metadata. We're fortunate enough to have all flash for our metadata and I tend to take it for granted some times :) From: greg.lehmann at csiro.au Sent: 3/29/17, 19:52 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] question on viewing block distribution across NSDs Thanks. I don?t have a snap. I?ll keep that in mind for next time I do this. From: mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] Sent: Thursday, 30 March 2017 9:45 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] question on viewing block distribution across NSDs Hi Greg, You could run an mmdf which will show you how full each NSD is. I'm not sure how to look back in time though to see the fs before the restripe. Do you perhaps have a gpfs.snap you took somewhat recently before the restripe? Maybe an internaldump in /tmp/mmfs somewhere? From: greg.lehmann at csiro.au Sent: 3/29/17, 19:21 To: gpfsug main discussion list Subject: [gpfsug-discuss] question on viewing block distribution across NSDs Hi All, I added some NSDs to an existing filesystem and ran mmrestripefs. I was sort of curious to see what the distribution looked like before and after the restripe. Is there any way of looking at it? Cheers, Greg._______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From willi.engeli at id.ethz.ch Thu Mar 30 14:29:26 2017 From: willi.engeli at id.ethz.ch (Engeli Willi (ID SD)) Date: Thu, 30 Mar 2017 13:29:26 +0000 Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Message-ID: Hi everybody, In our organization, the management of AD is strictly separated from management of storage. Since we install spectrum scale with protocol SMB and NFS support, we need to join the systems to AD, and have at least the joining user added as well to the local administrators group. Any idea of how to achieve this? Asking our Domain Admin is not the correct method to add other groups, this needs to be in our hands. Regards Willi -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5461 bytes Desc: not available URL: From r.sobey at imperial.ac.uk Thu Mar 30 14:53:15 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Thu, 30 Mar 2017 13:53:15 +0000 Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group In-Reply-To: References: Message-ID: Last time I checked simply adding a normal computer object to the domain didn't add the account of the adding user to the local administrators group and CES is no exception. Is it a political reason why you cannot ask your Domain Admin team to add you to the admin group for your CES cluster object? From there you can manage it yourself. Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Engeli Willi (ID SD) Sent: 30 March 2017 14:29 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Hi everybody, In our organization, the management of AD is strictly separated from management of storage. Since we install spectrum scale with protocol SMB and NFS support, we need to join the systems to AD, and have at least the joining user added as well to the local administrators group. Any idea of how to achieve this? Asking our Domain Admin is not the correct method to add other groups, this needs to be in our hands. Regards Willi -------------- next part -------------- An HTML attachment was scrubbed... URL: From laurence at qsplace.co.uk Thu Mar 30 15:02:19 2017 From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow) Date: Thu, 30 Mar 2017 15:02:19 +0100 Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group In-Reply-To: References: Message-ID: <2329870e-00f8-258c-187d-feec9589df93@qsplace.co.uk> Hi Willi, Could you just expand on your issue? Are you requiring CES to bind to AD to allow authenticated users to access your NFS/SMB shares. However you require the ability to add additional groups to these users on the CES system? Or are you trying to use your own account that can join the domain as a local admin on a CES node? -- Lauz On 30/03/2017 14:53, Sobey, Richard A wrote: > > Last time I checked simply adding a normal computer object to the > domain didn?t add the account of the adding user to the local > administrators group and CES is no exception. > > Is it a political reason why you cannot ask your Domain Admin team to > add you to the admin group for your CES cluster object? From there you > can manage it yourself. > > Richard > > *From:*gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of > *Engeli Willi (ID SD) > *Sent:* 30 March 2017 14:29 > *To:* gpfsug-discuss at spectrumscale.org > *Subject:* [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin > to local Administrators group > > Hi everybody, > > In our organization, the management of AD is strictly separated from > management of storage. Since we install spectrum scale with protocol > SMB and NFS support, we need to join the systems to AD, and have at > least the joining user added as well to the local administrators group. > > Any idea of how to achieve this? Asking our Domain Admin is not the > correct method to add other groups, this needs to be in our hands. > > Regards Willi > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From jgwood at sandia.gov Thu Mar 30 15:00:50 2017 From: jgwood at sandia.gov (Wood, Justin Gregory) Date: Thu, 30 Mar 2017 14:00:50 +0000 Subject: [gpfsug-discuss] [EXTERNAL] Spectrum Scale CES adds only Domain Admin to local Administrators group In-Reply-To: References: Message-ID: <069E1DC2-8711-4F4E-BD65-250C524ED341@sandia.gov> I recently went through the process of installing some protocol nodes and our organization also has a separate AD/Windows group. All I had to do was to get the AD admins to add a ?stub? computer object, then they made me an administrator of that object. That allowed me to run mmuserauth to do the setup. It was fairly simple and allowed me to do lots of work/testing without interaction with the AD group. -Justin. From: on behalf of "Engeli Willi (ID SD)" Reply-To: gpfsug main discussion list Date: Thursday, March 30, 2017 at 7:29 AM To: "gpfsug-discuss at spectrumscale.org" Subject: [EXTERNAL] [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Hi everybody, In our organization, the management of AD is strictly separated from management of storage. Since we install spectrum scale with protocol SMB and NFS support, we need to join the systems to AD, and have at least the joining user added as well to the local administrators group. Any idea of how to achieve this? Asking our Domain Admin is not the correct method to add other groups, this needs to be in our hands. Regards Willi -------------- next part -------------- An HTML attachment was scrubbed... URL: From willi.engeli at id.ethz.ch Thu Mar 30 15:23:40 2017 From: willi.engeli at id.ethz.ch (Engeli Willi (ID SD)) Date: Thu, 30 Mar 2017 14:23:40 +0000 Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Message-ID: >-Last time I checked simply adding a normal computer object to the domain didn't add the account of the adding user to the local administrators group and CES is no exception. We have been using before a competitor Product as a NAS system. With that system, we were able to define virtual NAS Servers, each one joined as an independent object to AD. When joined, we found the 'Domain Admin' group and the joining user as member of local administrators group of that virtual server. Since out AD is quite big, it is structured into many OU. We as the Storage OU have OU admin rights, but we are not member of "Domain Admin" group. Looking Back, we were able by ourselves to add the required groups as needed to the local Administrators group of the NAS server. Why is this important? Since we have quit a mix of OS accessing our shares, some of the create exclusive access rights at the time they create profiles etc. At the end of the lifecycle, one needs to delete those files via the SMB / NFSV4 protocol, which is difficult if not having access rights. On the other hand, we have seen situations, where one OS corrupted the ACL and could not access anymore. Also this needs to be handled by us, giving us a hard time not being member of the administrators group. I.e. the MS tool subinacl does check the privileges before trying to modify ACLs, and if not being member of the Administrators group, not all required privileges are granted. >-Is it a political reason why you cannot ask your Domain Admin team to add you to the admin group for your CES cluster object? From there you can manage it yourself. Yes and no. We have a clear boundary, where we need to be able to manage the AD Objects, and for security reason it seems to make sense to not use Domain Admin Accounts for such kind of work (statement of our AD Group). So much for the Situation, did I missed something? Willi -----Urspr?ngliche Nachricht----- Von: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] Im Auftrag von gpfsug-discuss-request at spectrumscale.org Gesendet: Donnerstag, 30. M?rz 2017 16:02 An: gpfsug-discuss at spectrumscale.org Betreff: gpfsug-discuss Digest, Vol 62, Issue 77 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Spectrum Scale CES adds only Domain Admin to local Administrators group (Engeli Willi (ID SD)) 2. Re: Spectrum Scale CES adds only Domain Admin to local Administrators group (Sobey, Richard A) 3. Re: Spectrum Scale CES adds only Domain Admin to local Administrators group (Laurence Horrocks-Barlow) ---------------------------------------------------------------------- Message: 1 Date: Thu, 30 Mar 2017 13:29:26 +0000 From: "Engeli Willi (ID SD)" To: "gpfsug-discuss at spectrumscale.org" Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Message-ID: Content-Type: text/plain; charset="us-ascii" Hi everybody, In our organization, the management of AD is strictly separated from management of storage. Since we install spectrum scale with protocol SMB and NFS support, we need to join the systems to AD, and have at least the joining user added as well to the local administrators group. Any idea of how to achieve this? Asking our Domain Admin is not the correct method to add other groups, this needs to be in our hands. Regards Willi -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5461 bytes Desc: not available URL: ------------------------------ Message: 2 Date: Thu, 30 Mar 2017 13:53:15 +0000 From: "Sobey, Richard A" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Message-ID: Content-Type: text/plain; charset="us-ascii" Last time I checked simply adding a normal computer object to the domain didn't add the account of the adding user to the local administrators group and CES is no exception. Is it a political reason why you cannot ask your Domain Admin team to add you to the admin group for your CES cluster object? From there you can manage it yourself. Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Engeli Willi (ID SD) Sent: 30 March 2017 14:29 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Hi everybody, In our organization, the management of AD is strictly separated from management of storage. Since we install spectrum scale with protocol SMB and NFS support, we need to join the systems to AD, and have at least the joining user added as well to the local administrators group. Any idea of how to achieve this? Asking our Domain Admin is not the correct method to add other groups, this needs to be in our hands. Regards Willi -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 3 Date: Thu, 30 Mar 2017 15:02:19 +0100 From: Laurence Horrocks-Barlow To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Message-ID: <2329870e-00f8-258c-187d-feec9589df93 at qsplace.co.uk> Content-Type: text/plain; charset="windows-1252"; Format="flowed" Hi Willi, Could you just expand on your issue? Are you requiring CES to bind to AD to allow authenticated users to access your NFS/SMB shares. However you require the ability to add additional groups to these users on the CES system? Or are you trying to use your own account that can join the domain as a local admin on a CES node? -- Lauz On 30/03/2017 14:53, Sobey, Richard A wrote: > > Last time I checked simply adding a normal computer object to the > domain didn?t add the account of the adding user to the local > administrators group and CES is no exception. > > Is it a political reason why you cannot ask your Domain Admin team to > add you to the admin group for your CES cluster object? From there you > can manage it yourself. > > Richard > > *From:*gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of > *Engeli Willi (ID SD) > *Sent:* 30 March 2017 14:29 > *To:* gpfsug-discuss at spectrumscale.org > *Subject:* [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin > to local Administrators group > > Hi everybody, > > In our organization, the management of AD is strictly separated from > management of storage. Since we install spectrum scale with protocol > SMB and NFS support, we need to join the systems to AD, and have at > least the joining user added as well to the local administrators group. > > Any idea of how to achieve this? Asking our Domain Admin is not the > correct method to add other groups, this needs to be in our hands. > > Regards Willi > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 62, Issue 77 ********************************************** -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5461 bytes Desc: not available URL: From gmcpheeters at anl.gov Thu Mar 30 15:39:45 2017 From: gmcpheeters at anl.gov (McPheeters, Gordon) Date: Thu, 30 Mar 2017 14:39:45 +0000 Subject: [gpfsug-discuss] question on viewing block distribution across NSDs In-Reply-To: <2c0ba2b9466b42b4b3e5b3202ab26825@exch1-cdc.nexus.csiro.au> References: <2c0ba2b9466b42b4b3e5b3202ab26825@exch1-cdc.nexus.csiro.au> Message-ID: <2A55B023-7DE1-417E-BD30-88FD44DF0ADA@anl.gov> mmdf will show you the distribution across NSDs. Gordon On Mar 29, 2017, at 6:12 PM, Greg.Lehmann at csiro.au wrote: Hi All, I added some NSDs to an existing filesystem and ran mmrestripefs. I was sort of curious to see what the distribution looked like before and after the restripe. Is there any way of looking at it? Cheers, Greg. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Thu Mar 30 15:50:13 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Thu, 30 Mar 2017 14:50:13 +0000 Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group In-Reply-To: References: Message-ID: Did your AD team perchance define a group policy on the OU such that any object placed into that OU inherited a specific set of local administrators? That's the only way I can think that your NAS ended up with the calling user in the local admin group. I understand where you're coming from - we do not manage AD ourselves but we do not want Domain Admins to have administrator control of our CES nodes. So once it was joined to AD (with their help) I simply removed Domain Admins and added the storage team DL in its place. But back to the original question, I'm afraid I do not know how to make CES add a specific user to its local administrator group. Richard -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Engeli Willi (ID SD) Sent: 30 March 2017 15:24 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group >-Last time I checked simply adding a normal computer object to the >domain didn't add the account of the adding user to the local administrators group and CES is no exception. We have been using before a competitor Product as a NAS system. With that system, we were able to define virtual NAS Servers, each one joined as an independent object to AD. When joined, we found the 'Domain Admin' group and the joining user as member of local administrators group of that virtual server. Since out AD is quite big, it is structured into many OU. We as the Storage OU have OU admin rights, but we are not member of "Domain Admin" group. Looking Back, we were able by ourselves to add the required groups as needed to the local Administrators group of the NAS server. Why is this important? Since we have quit a mix of OS accessing our shares, some of the create exclusive access rights at the time they create profiles etc. At the end of the lifecycle, one needs to delete those files via the SMB / NFSV4 protocol, which is difficult if not having access rights. On the other hand, we have seen situations, where one OS corrupted the ACL and could not access anymore. Also this needs to be handled by us, giving us a hard time not being member of the administrators group. I.e. the MS tool subinacl does check the privileges before trying to modify ACLs, and if not being member of the Administrators group, not all required privileges are granted. >-Is it a political reason why you cannot ask your Domain Admin team to >add you to the admin group for your CES cluster object? From there you can manage it yourself. Yes and no. We have a clear boundary, where we need to be able to manage the AD Objects, and for security reason it seems to make sense to not use Domain Admin Accounts for such kind of work (statement of our AD Group). So much for the Situation, did I missed something? Willi -----Urspr?ngliche Nachricht----- Von: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] Im Auftrag von gpfsug-discuss-request at spectrumscale.org Gesendet: Donnerstag, 30. M?rz 2017 16:02 An: gpfsug-discuss at spectrumscale.org Betreff: gpfsug-discuss Digest, Vol 62, Issue 77 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Spectrum Scale CES adds only Domain Admin to local Administrators group (Engeli Willi (ID SD)) 2. Re: Spectrum Scale CES adds only Domain Admin to local Administrators group (Sobey, Richard A) 3. Re: Spectrum Scale CES adds only Domain Admin to local Administrators group (Laurence Horrocks-Barlow) ---------------------------------------------------------------------- Message: 1 Date: Thu, 30 Mar 2017 13:29:26 +0000 From: "Engeli Willi (ID SD)" To: "gpfsug-discuss at spectrumscale.org" Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Message-ID: Content-Type: text/plain; charset="us-ascii" Hi everybody, In our organization, the management of AD is strictly separated from management of storage. Since we install spectrum scale with protocol SMB and NFS support, we need to join the systems to AD, and have at least the joining user added as well to the local administrators group. Any idea of how to achieve this? Asking our Domain Admin is not the correct method to add other groups, this needs to be in our hands. Regards Willi -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5461 bytes Desc: not available URL: ------------------------------ Message: 2 Date: Thu, 30 Mar 2017 13:53:15 +0000 From: "Sobey, Richard A" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Message-ID: Content-Type: text/plain; charset="us-ascii" Last time I checked simply adding a normal computer object to the domain didn't add the account of the adding user to the local administrators group and CES is no exception. Is it a political reason why you cannot ask your Domain Admin team to add you to the admin group for your CES cluster object? From there you can manage it yourself. Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Engeli Willi (ID SD) Sent: 30 March 2017 14:29 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Hi everybody, In our organization, the management of AD is strictly separated from management of storage. Since we install spectrum scale with protocol SMB and NFS support, we need to join the systems to AD, and have at least the joining user added as well to the local administrators group. Any idea of how to achieve this? Asking our Domain Admin is not the correct method to add other groups, this needs to be in our hands. Regards Willi -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 3 Date: Thu, 30 Mar 2017 15:02:19 +0100 From: Laurence Horrocks-Barlow To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Message-ID: <2329870e-00f8-258c-187d-feec9589df93 at qsplace.co.uk> Content-Type: text/plain; charset="windows-1252"; Format="flowed" Hi Willi, Could you just expand on your issue? Are you requiring CES to bind to AD to allow authenticated users to access your NFS/SMB shares. However you require the ability to add additional groups to these users on the CES system? Or are you trying to use your own account that can join the domain as a local admin on a CES node? -- Lauz On 30/03/2017 14:53, Sobey, Richard A wrote: > > Last time I checked simply adding a normal computer object to the > domain didn?t add the account of the adding user to the local > administrators group and CES is no exception. > > Is it a political reason why you cannot ask your Domain Admin team to > add you to the admin group for your CES cluster object? From there you > can manage it yourself. > > Richard > > *From:*gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of > *Engeli Willi (ID SD) > *Sent:* 30 March 2017 14:29 > *To:* gpfsug-discuss at spectrumscale.org > *Subject:* [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin > to local Administrators group > > Hi everybody, > > In our organization, the management of AD is strictly separated from > management of storage. Since we install spectrum scale with protocol > SMB and NFS support, we need to join the systems to AD, and have at > least the joining user added as well to the local administrators group. > > Any idea of how to achieve this? Asking our Domain Admin is not the > correct method to add other groups, this needs to be in our hands. > > Regards Willi > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 62, Issue 77 ********************************************** From Mark.Bush at siriuscom.com Thu Mar 30 17:09:15 2017 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Thu, 30 Mar 2017 16:09:15 +0000 Subject: [gpfsug-discuss] AFM vs mmremotefs Message-ID: <5A22811B-7FBD-4091-B6FD-E5E74053DDF2@siriuscom.com> When does it make sense to use mmremotefs vs AFM? With mmremotefs the actual data lives on the remote cluster right so if you need data closer then AFM make more sense to use? If the clusters get disconnected with mmremotefs I have not access to the data with AFM I do. Do I have these concepts right in my head? [id:image001.png at 01D2709D.6EF65720] Mark R. Bush| Storage Architect Mobile: 210-237-8415 Twitter: @bushmr | LinkedIn: /markreedbush 10100 Reunion Place, Suite 500, San Antonio, TX 78216 www.siriuscom.com |mark.bush at siriuscom.com This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 8745 bytes Desc: image001.png URL: From S.J.Thompson at bham.ac.uk Thu Mar 30 17:19:07 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Thu, 30 Mar 2017 16:19:07 +0000 Subject: [gpfsug-discuss] AFM vs mmremotefs Message-ID: We are just planning to deploy AFM as a cache tier for our HPC cluster and bulk data storage. So whilst they are locally connected etc, I can use a different type of storage for performance for HPC work. Bulk data has two copies over two data centres, so the AFM cache would ACK the write to the HPC client and then "push it as soon as possible" to the home (bulk data storage and do the two copies). Similarly for ingest systems, for example streaming data off instruments, you might want to do really really fast (to a flash system) and then have the AFM write to home at some point. (of course your cache capacity needs to be sufficiently large that you are able to push writes/expel). On the flip side, I don't think AFM has remote identity mapping (someone might correct me on that of course) So I don't think its just about data location/connectivity. Simon From: > on behalf of "Mark.Bush at siriuscom.com" > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Thursday, 30 March 2017 at 17:09 To: "gpfsug-discuss at spectrumscale.org" > Subject: [gpfsug-discuss] AFM vs mmremotefs When does it make sense to use mmremotefs vs AFM? With mmremotefs the actual data lives on the remote cluster right so if you need data closer then AFM make more sense to use? If the clusters get disconnected with mmremotefs I have not access to the data with AFM I do. Do I have these concepts right in my head? [id:image001.png at 01D2709D.6EF65720] Mark R. Bush| Storage Architect Mobile: 210-237-8415 Twitter: @bushmr | LinkedIn: /markreedbush 10100 Reunion Place, Suite 500, San Antonio, TX 78216 www.siriuscom.com |mark.bush at siriuscom.com This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 8745 bytes Desc: image001.png URL: From makaplan at us.ibm.com Thu Mar 30 17:26:34 2017 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Thu, 30 Mar 2017 11:26:34 -0500 Subject: [gpfsug-discuss] AFM vs mmremotefs In-Reply-To: <5A22811B-7FBD-4091-B6FD-E5E74053DDF2@siriuscom.com> References: <5A22811B-7FBD-4091-B6FD-E5E74053DDF2@siriuscom.com> Message-ID: I think you "got it" - despite the name "remote" - mmremotefs is an access method and a way of organizing your data and machines into several clusters - which can work great if your network(s) are fast enough. For example you can have one or more client clusters with no GPFS(Spectrum Scale) file systems stored "locally". And one or several clusters that are servers to those client clusters. AFM is about caching and replicating files... You might do that because of "remote-ness" - or for other reasons... --marc From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Date: 03/30/2017 12:09 PM Subject: [gpfsug-discuss] AFM vs mmremotefs Sent by: gpfsug-discuss-bounces at spectrumscale.org When does it make sense to use mmremotefs vs AFM? With mmremotefs the actual data lives on the remote cluster right so if you need data closer then AFM make more sense to use? If the clusters get disconnected with mmremotefs I have not access to the data with AFM I do. Do I have these concepts right in my head? Mark R. Bush| Storage Architect Mobile: 210-237-8415 Twitter: @bushmr | LinkedIn: /markreedbush 10100 Reunion Place, Suite 500, San Antonio, TX 78216 www.siriuscom.com |mark.bush at siriuscom.com This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 8745 bytes Desc: not available URL: From gcorneau at us.ibm.com Thu Mar 30 17:42:38 2017 From: gcorneau at us.ibm.com (Glen Corneau) Date: Thu, 30 Mar 2017 10:42:38 -0600 Subject: [gpfsug-discuss] AFM vs mmremotefs In-Reply-To: References: <5A22811B-7FBD-4091-B6FD-E5E74053DDF2@siriuscom.com> Message-ID: Just a extra note: Multi-cluster (mmremotefs) does not always mean "network" for file system data transfer. If both cluster(s) see the storage over the SAN then the data happens over that path. This is common in a localized scenario for authorization reasons (i.e. nodes 1,2,3 can access file systems A & B, but not C; but nodes 4,5,6 in a separate cluster can access all 3 file systems) ------------------ Glen Corneau Power Systems Washington Systems Center gcorneau at us.ibm.com From: Marc A Kaplan/Watson/IBM at IBMUS To: gpfsug main discussion list Date: 03/30/2017 11:27 AM Subject: Re: [gpfsug-discuss] AFM vs mmremotefs Sent by: gpfsug-discuss-bounces at spectrumscale.org I think you "got it" - despite the name "remote" - mmremotefs is an access method and a way of organizing your data and machines into several clusters - which can work great if your network(s) are fast enough. For example you can have one or more client clusters with no GPFS(Spectrum Scale) file systems stored "locally". And one or several clusters that are servers to those client clusters. AFM is about caching and replicating files... You might do that because of "remote-ness" - or for other reasons... --marc From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Date: 03/30/2017 12:09 PM Subject: [gpfsug-discuss] AFM vs mmremotefs Sent by: gpfsug-discuss-bounces at spectrumscale.org When does it make sense to use mmremotefs vs AFM? With mmremotefs the actual data lives on the remote cluster right so if you need data closer then AFM make more sense to use? If the clusters get disconnected with mmremotefs I have not access to the data with AFM I do. Do I have these concepts right in my head? Mark R. Bush| Storage Architect Mobile: 210-237-8415 Twitter: @bushmr| LinkedIn: /markreedbush 10100 Reunion Place, Suite 500, San Antonio, TX 78216 www.siriuscom.com|mark.bush at siriuscom.com This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 26117 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 8745 bytes Desc: not available URL: From christof.schmitt at us.ibm.com Thu Mar 30 21:18:21 2017 From: christof.schmitt at us.ibm.com (Christof Schmitt) Date: Thu, 30 Mar 2017 13:18:21 -0700 Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group In-Reply-To: References: Message-ID: willi.engeli at id.ethz.ch wrote on 03/30/2017 07:23:40 AM: > >-Last time I checked simply adding a normal computer object to the domain > didn't add the account of the adding user to the local administrators group > and CES is no exception. > > We have been using before a competitor Product as a NAS system. With that > system, we were able to define virtual NAS Servers, each one joined as an > independent object to AD. When joined, we found the 'Domain Admin' group and > the joining user as member of local administrators group of that virtual > server. > Since out AD is quite big, it is structured into many OU. We as the Storage > OU have OU admin rights, but we are not member of "Domain Admin" group. > Looking Back, we were able by ourselves to add the required groups as needed > to the local Administrators group of the NAS server. > Why is this important? Since we have quit a mix of OS accessing our shares, > some of the create exclusive access rights at the time they create profiles > etc. At the end of the lifecycle, one needs to delete those files via the > SMB / NFSV4 protocol, which is difficult if not having access rights. On the > other hand, we have seen situations, where one OS corrupted the ACL and > could not access anymore. Also this needs to be handled by us, giving us a > hard time not being member of the administrators group. I.e. the MS tool > subinacl does check the privileges before trying to modify ACLs, and if not > being member of the Administrators group, not all required privileges are > granted. There is two parts to that in Spectrum Scale: 1) There is an option to declare a user as 'admin users'. The notion there is that this user is mapped to root on access, thus this user can always access files and fix access issues. The user defined here should not be used for normal usage, this is only recommended for data migrations and to fix access issues. 2) When joining Spectrum Scale to an Active Directory domain, the Domain Admins groups is added to the internal Administrators group (sometimes referred to as BUILTIN\Administrators). One way to change the membership in that group would be through the MMC on a Windows client. Initially only Domain Admins are allowed, a member of this group would be required to add other users or groups. Alternatively, the "net sam" interface can be used to modify the group from root access on the protocol nodes: /usr/lpp/mmfs/bin/net sam listmem Administrators to list the members of the Administrators groups. /usr/lpp/mmfs/bin/net sam addmem Administrators DOMAIN\user to add a member. /usr/lpp/mmfs/bin/net sam delmem Administrators DOMAIN\user to remove a member This is currently an untested feature and not exposed through the CLI. If there is a need to have this exposed through the CLI or GUI, that should be requested through a RFE so that it can feed into the planning and prioritization for future releases. Regards, Christof Schmitt || IBM || Spectrum Scale Development || Tucson, AZ christof.schmitt at us.ibm.com || +1-520-799-2469 (T/L: 321-2469) From willi.engeli at id.ethz.ch Fri Mar 31 12:46:02 2017 From: willi.engeli at id.ethz.ch (Engeli Willi (ID SD)) Date: Fri, 31 Mar 2017 11:46:02 +0000 Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Message-ID: Hi Christoph, This solved my issues in most areas. Now I will probably add our Storage Management Group to local Administrators group, this way we are able to use all strong utilities like subinacl etc, and will be able to migrate to Spectrum Scale using robocopy with /ZB option working properly. For our Share-responsible Administrator we probably will add their Management user to the 'admin Users' option of the specific share allowing them to do wat ever they need to do, knowing that some tools may work with limitations. Do you know if we may also add a builtin group named BackupOperators? Regards Willi -----Urspr?ngliche Nachricht----- Von: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] Im Auftrag von gpfsug-discuss-request at spectrumscale.org Gesendet: Freitag, 31. M?rz 2017 13:00 An: gpfsug-discuss at spectrumscale.org Betreff: gpfsug-discuss Digest, Vol 62, Issue 82 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Spectrum Scale CES adds only Domain Admin to local Administrators group (Christof Schmitt) ---------------------------------------------------------------------- Message: 1 Date: Thu, 30 Mar 2017 13:18:21 -0700 From: "Christof Schmitt" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Message-ID: Content-Type: text/plain; charset="US-ASCII" willi.engeli at id.ethz.ch wrote on 03/30/2017 07:23:40 AM: > >-Last time I checked simply adding a normal computer object to the domain > didn't add the account of the adding user to the local administrators group > and CES is no exception. > > We have been using before a competitor Product as a NAS system. With that > system, we were able to define virtual NAS Servers, each one joined as an > independent object to AD. When joined, we found the 'Domain Admin' > group and > the joining user as member of local administrators group of that > virtual server. > Since out AD is quite big, it is structured into many OU. We as the Storage > OU have OU admin rights, but we are not member of "Domain Admin" group. > Looking Back, we were able by ourselves to add the required groups as needed > to the local Administrators group of the NAS server. > Why is this important? Since we have quit a mix of OS accessing our shares, > some of the create exclusive access rights at the time they create profiles > etc. At the end of the lifecycle, one needs to delete those files via the > SMB / NFSV4 protocol, which is difficult if not having access rights. > On the > other hand, we have seen situations, where one OS corrupted the ACL > and could not access anymore. Also this needs to be handled by us, > giving us a > hard time not being member of the administrators group. I.e. the MS > tool subinacl does check the privileges before trying to modify ACLs, > and if not > being member of the Administrators group, not all required privileges are > granted. There is two parts to that in Spectrum Scale: 1) There is an option to declare a user as 'admin users'. The notion there is that this user is mapped to root on access, thus this user can always access files and fix access issues. The user defined here should not be used for normal usage, this is only recommended for data migrations and to fix access issues. 2) When joining Spectrum Scale to an Active Directory domain, the Domain Admins groups is added to the internal Administrators group (sometimes referred to as BUILTIN\Administrators). One way to change the membership in that group would be through the MMC on a Windows client. Initially only Domain Admins are allowed, a member of this group would be required to add other users or groups. Alternatively, the "net sam" interface can be used to modify the group from root access on the protocol nodes: /usr/lpp/mmfs/bin/net sam listmem Administrators to list the members of the Administrators groups. /usr/lpp/mmfs/bin/net sam addmem Administrators DOMAIN\user to add a member. /usr/lpp/mmfs/bin/net sam delmem Administrators DOMAIN\user to remove a member This is currently an untested feature and not exposed through the CLI. If there is a need to have this exposed through the CLI or GUI, that should be requested through a RFE so that it can feed into the planning and prioritization for future releases. Regards, Christof Schmitt || IBM || Spectrum Scale Development || Tucson, AZ christof.schmitt at us.ibm.com || +1-520-799-2469 (T/L: 321-2469) ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 62, Issue 82 ********************************************** -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5461 bytes Desc: not available URL: From pinto at scinet.utoronto.ca Fri Mar 31 15:18:34 2017 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Fri, 31 Mar 2017 10:18:34 -0400 Subject: [gpfsug-discuss] BIG LAG since 3.5 on quota accounting reconciliation Message-ID: <20170331101834.162212izrlj4swu2@support.scinet.utoronto.ca> In the old days of DDN 9900 and gpfs 3.4 I only had to run mmcheckquota once a month, usually after the massive monthly purge. I noticed that starting with the GSS and ESS appliances under 3.5 that I needed to run mmcheckquota more often, at least once a week, or as often as daily, to clear the slippage errors in the accounting information, otherwise users complained that they were hitting their quotas, even throughout they deleted a lot of stuff. More recently we adopted a G200 appliance (1.8PB), with v4.1, and now things have gotten worst, and I have to run it twice daily, just in case. So, what I am missing? Is their a parameter since 3.5 and through 4.1 that we can set, so that GPFS will reconcile the quota accounting internally more often and on its own? Thanks Jaime ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. From billowen at us.ibm.com Fri Mar 31 19:23:35 2017 From: billowen at us.ibm.com (Bill Owen) Date: Fri, 31 Mar 2017 11:23:35 -0700 Subject: [gpfsug-discuss] mmnetverify In-Reply-To: References: , <18729d8a86fc430fac09c1c3a22159d2@mbxtoa1.winmail.deshaw.com> Message-ID: mmnetverify is a new tool that aims to make it easier to identify network problems. Regarding bandwidth commands, in 4.2.2, there are two options: mmnetverify bandwidth-node - [1 to 1] this will communicate from local node (or one or more nodes specified with -N option) to one or more target nodes. The bandwidth tests are executed serially from nodes in node list to target, iterating through each target node, one by one. The serially calculated bandwidth with each node is reported. mmnetverify bandwidth-cluster - [1 to many] this is a measure of parallel communication from the local node (or one or more nodes specified with -N option) to all of the other nodes in the cluster. The concurrent bandwidth with each target node in the cluster is reported. In both of these tests, we establish a socket connection, and pass a fixed number of bytes over the connection and calculate bandwidth based on how long that transmission took. For 4.2.3, there is a new bandwidth test called gnr-bandwidth. It is similar to the bandwidth-cluster [1 to many] except that it uses the following steps: 1. establish connection from node to all other target nodes in the cluster 2. start sending data to target for some ramp up period 3. after ramp up period, continue sending data for test period 4. calculate bandwidth based on bytes transmitted during test period The bandwidth to each node is summed to return a total bandwidth from the command node to the other nodes in the cluster. In future releases, we may modify bandwidth-node & bandwidth-cluster tests to use the gnr-bandwidth methodology (and deprecate gnr-bandwidth). Your feedback on how to improve mmnetverify is appreciated. Regarding: > We found some weird looking numbers that i don't quite understand and not in the places we might expect. > For example between hosts on the same switch, traffic flowing to another switch and traffic flowing to > nodes in another data centre where it's several switch hops. Some nodes over there were significantly > faster than switch local nodes. Note that system load can impact the test results. Is it possible that the slow nodes on the local switch were heavily loaded? Or is it possible they are using an interface that is lower bandwidth? (sorry, i had to ask that one to be sure...) Regards, Bill Owen billowen at us.ibm.com Spectrum Scale Development 520-799-4829 From: "Simon Thompson (Research Computing - IT Services)" To: gpfsug main discussion list Date: 03/17/2017 01:13 PM Subject: Re: [gpfsug-discuss] mmnetverify Sent by: gpfsug-discuss-bounces at spectrumscale.org It looks to run sequential tests to each node one at a time and isn't using NSD protocol but echo server. We found some weird looking numbers that i don't quite understand and not in the places we might expect. For example between hosts on the same switch, traffic flowing to another switch and traffic flowing to nodes in another data centre where it's several switch hops. Some nodes over there were significantly faster than switch local nodes. I think it was only added in 4.2.2 and is listed as "not yet a replacement for nsdperf". I get that is different as it's using NSD protocol, but was struggling a bit with what mmnetverify might be doing. Simon From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Sanchez, Paul [Paul.Sanchez at deshaw.com] Sent: 17 March 2017 19:43 To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] mmnetverify Sven will tell you: "RPC isn't streaming" and that may account for the discrepancy. If the tests are doing any "fan-in" where multiple nodes are sending to single node, then it's also possible that you are exhausting switch buffer memory in a way that a 1:1 iperf wouldn't. For our internal benchmarking we've used /usr/lpp/mmfs/samples/net/nsdperf to more closely estimate the real performance. I haven't played with mmnetverify yet though. -Paul -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (Research Computing - IT Services) Sent: Friday, March 17, 2017 2:50 PM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] mmnetverify Hi all, Just wondering if anyone has used the mmnetverify tool at all? Having made some changes to our internal L3 routing this week, I was interested to see what it claimed. As a side-note, it picked up some DNS resolution issues, though I'm not clear on some of those why it was claiming this as doing a "dig" on the node, it resolved fine (but adding the NSD servers to the hosts files cleared the error). Its actually the bandwidth tests that I'm interested in hearing other people's experience with as the numbers that some out from it are very different (lower) than if we use iperf to test performance between two nodes. Anyone any thoughts at all on this? Thanks Simon _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From christof.schmitt at us.ibm.com Fri Mar 31 21:22:36 2017 From: christof.schmitt at us.ibm.com (Christof Schmitt) Date: Fri, 31 Mar 2017 13:22:36 -0700 Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin tolocal In-Reply-To: References: Message-ID: willi.engeli at id.ethz.ch wrote on 03/31/2017 04:46:02 AM: > Hi Christoph, > This solved my issues in most areas. > Now I will probably add our Storage Management Group to local > Administrators group, this way we are able to use all strong > utilities like subinacl etc, and will be able to migrate to Spectrum > Scale using robocopy with /ZB option working properly. > For our Share-responsible Administrator we probably will add their > Management user to the 'admin Users' option of the specific share > allowing them to do wat ever they need to do, knowing that some > tools may work with limitations. > > Do you know if we may also add a builtin group named BackupOperators? Privileges are not supported in Spectrum Scale: https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.2/com.ibm.spectrum.scale.v4r22.doc/bl1adm_fileauthlimitations.htm "Access privileges defined in Windows are not honored. Those privileges are tied to administrator groups and allow access, where the ACL alone does not grant it." You can create a group and assign the BackupOperator privilege: /usr/lpp/mmfs/bin/net sam createbuiltingroup 'Backup Operators' /usr/lpp/mmfs/bin/net sam rights grant 'Backup Operators' SeBackupPrivilege Without looking at all the details, i would suspect that this does not work. Access for a member of this group would overwrite the internal access check in Samba, but i would expect that it would fail as the file system also enforces the permissions defined in the ACL, and these are not overwritten by the additional privilege. The current workaround is the 'admin users' option. You might want to raise a RFE to request better support of the Backup privilege and the "Backup Operators" group. Regards, Christof Schmitt || IBM || Spectrum Scale Development || Tucson, AZ christof.schmitt at us.ibm.com || +1-520-799-2469 (T/L: 321-2469) From a.g.richmond at leeds.ac.uk Wed Mar 1 11:19:17 2017 From: a.g.richmond at leeds.ac.uk (Aidan Richmond) Date: Wed, 1 Mar 2017 11:19:17 +0000 Subject: [gpfsug-discuss] Issues getting SMB shares working. Message-ID: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> Hello I've recently taken over managing a GPFS installation providing storage to a research facility at Leeds University. The system was setup by a colleague who recently left and before he left he configured NFSV4 access which services an important subset, but not all the users who require access to the storage. SMB access was part of the deployment plan, but he didn't have time to get that working. I've been attempting to get an SMB share up and running myself and have enabled the SMB service with "mmces service enable SMB", have added a share with "mmsmb export add share /path/to/folder" and "mmsmb export list" seems to show everything is okay. However attempts to access the share through the floating protocol addresses fail and I have no idea how to diagnose the issue. Any advice/help would be welcome. -- Aidan Richmond Apple/Unix Support Officer, IT Garstang 10.137 Faculty of Biological Sciences University of Leeds Clarendon Way LS2 9JT Tel:0113 3434252 a.g.richmond at leeds.ac.uk From r.sobey at imperial.ac.uk Wed Mar 1 11:33:13 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Wed, 1 Mar 2017 11:33:13 +0000 Subject: [gpfsug-discuss] Issues getting SMB shares working. In-Reply-To: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> References: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> Message-ID: Have you configured authentication? Mmuserauth service list --data-access-method file -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aidan Richmond Sent: 01 March 2017 11:19 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Issues getting SMB shares working. Hello I've recently taken over managing a GPFS installation providing storage to a research facility at Leeds University. The system was setup by a colleague who recently left and before he left he configured NFSV4 access which services an important subset, but not all the users who require access to the storage. SMB access was part of the deployment plan, but he didn't have time to get that working. I've been attempting to get an SMB share up and running myself and have enabled the SMB service with "mmces service enable SMB", have added a share with "mmsmb export add share /path/to/folder" and "mmsmb export list" seems to show everything is okay. However attempts to access the share through the floating protocol addresses fail and I have no idea how to diagnose the issue. Any advice/help would be welcome. -- Aidan Richmond Apple/Unix Support Officer, IT Garstang 10.137 Faculty of Biological Sciences University of Leeds Clarendon Way LS2 9JT Tel:0113 3434252 a.g.richmond at leeds.ac.uk _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From a.g.richmond at leeds.ac.uk Wed Mar 1 11:36:04 2017 From: a.g.richmond at leeds.ac.uk (Aidan Richmond) Date: Wed, 1 Mar 2017 11:36:04 +0000 Subject: [gpfsug-discuss] Issues getting SMB shares working. In-Reply-To: References: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> Message-ID: <901c6472-1c9c-3551-555e-22bc849fe2b1@leeds.ac.uk> Hello It returns the following: FILE access configuration : USERDEFINED PARAMETERS VALUES ------------------------------------------------- On 01/03/17 11:33, Sobey, Richard A wrote: > Mmuserauth service list --data-access-method file -- Aidan Richmond Apple/Unix Support Officer, IT Garstang 10.137 Faculty of Biological Sciences University of Leeds Clarendon Way LS2 9JT Tel:0113 3434252 a.g.richmond at leeds.ac.uk From r.sobey at imperial.ac.uk Wed Mar 1 11:42:06 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Wed, 1 Mar 2017 11:42:06 +0000 Subject: [gpfsug-discuss] Issues getting SMB shares working. In-Reply-To: <901c6472-1c9c-3551-555e-22bc849fe2b1@leeds.ac.uk> References: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> <901c6472-1c9c-3551-555e-22bc849fe2b1@leeds.ac.uk> Message-ID: That's probably the first thing you need to sort out then. Check out the mmuserauth service create command. There was an example on this list on Monday so depending when you subscribed you may not have seen it. FYI the command cited was: mmuserauth service create --type ad --data-access-method file --servers 192.168.88.3 --user-name administrator --netbios-name scale --idmap-role master --password ********* --idmap-range-size 1000000 --idmap-range 10000000-299999999 --enable-nfs-kerberos --unixmap-domains 'sirius(10000-20000)' Change the parameters to cater to your environment and needs of course. Richard -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aidan Richmond Sent: 01 March 2017 11:36 To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Issues getting SMB shares working. Hello It returns the following: FILE access configuration : USERDEFINED PARAMETERS VALUES ------------------------------------------------- On 01/03/17 11:33, Sobey, Richard A wrote: > Mmuserauth service list --data-access-method file -- Aidan Richmond Apple/Unix Support Officer, IT Garstang 10.137 Faculty of Biological Sciences University of Leeds Clarendon Way LS2 9JT Tel:0113 3434252 a.g.richmond at leeds.ac.uk _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From a.g.richmond at leeds.ac.uk Wed Mar 1 12:07:28 2017 From: a.g.richmond at leeds.ac.uk (Aidan Richmond) Date: Wed, 1 Mar 2017 12:07:28 +0000 Subject: [gpfsug-discuss] Issues getting SMB shares working. In-Reply-To: References: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> <901c6472-1c9c-3551-555e-22bc849fe2b1@leeds.ac.uk> Message-ID: <2ac0911f-65f7-d500-1d01-7ba4b5929a79@leeds.ac.uk> Hello I'm a little hesitant to mess with the authentication as I we are wanting consistent UIDs across our systems and I know my predecessor struggled to get them consistent. Our AD environment stores the UID and GID settings in msSFU30uid and msSFU30gid. I'm also concerned that the system is already in use with nfsv4 access and don't want to break existing access unless I have to. On 01/03/17 11:42, Sobey, Richard A wrote: > That's probably the first thing you need to sort out then. > > Check out the mmuserauth service create command. > > There was an example on this list on Monday so depending when you subscribed you may not have seen it. FYI the command cited was: > > mmuserauth service create --type ad --data-access-method file --servers 192.168.88.3 --user-name administrator --netbios-name scale --idmap-role master --password ********* --idmap-range-size 1000000 --idmap-range 10000000-299999999 --enable-nfs-kerberos --unixmap-domains 'sirius(10000-20000)' > > Change the parameters to cater to your environment and needs of course. > > Richard > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aidan Richmond > Sent: 01 March 2017 11:36 > To: gpfsug-discuss at spectrumscale.org > Subject: Re: [gpfsug-discuss] Issues getting SMB shares working. > > Hello > > It returns the following: > > FILE access configuration : USERDEFINED > PARAMETERS VALUES > ------------------------------------------------- > > On 01/03/17 11:33, Sobey, Richard A wrote: >> Mmuserauth service list --data-access-method file > > -- Aidan Richmond Apple/Unix Support Officer, IT Garstang 10.137 Faculty of Biological Sciences University of Leeds Clarendon Way LS2 9JT Tel:0113 3434252 a.g.richmond at leeds.ac.uk From janfrode at tanso.net Wed Mar 1 14:12:04 2017 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Wed, 1 Mar 2017 15:12:04 +0100 Subject: [gpfsug-discuss] Issues getting SMB shares working. In-Reply-To: <2ac0911f-65f7-d500-1d01-7ba4b5929a79@leeds.ac.uk> References: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> <901c6472-1c9c-3551-555e-22bc849fe2b1@leeds.ac.uk> <2ac0911f-65f7-d500-1d01-7ba4b5929a79@leeds.ac.uk> Message-ID: Lets figure out how your NFS is authenticating then. The userdefined authentication you have, could mean that your linux host is configured to authenticated towards AD --- or it could be that you're using simple sys-authentication for NFS. Could you please post the output of: mmnfs export list mmnfs config list -jf On Wed, Mar 1, 2017 at 1:07 PM, Aidan Richmond wrote: > Hello > > I'm a little hesitant to mess with the authentication as I we are wanting > consistent UIDs across our systems and I know my predecessor struggled to > get them consistent. Our AD environment stores the UID and GID settings in > msSFU30uid and msSFU30gid. > > I'm also concerned that the system is already in use with nfsv4 access and > don't want to break existing access unless I have to. > > > > On 01/03/17 11:42, Sobey, Richard A wrote: > >> That's probably the first thing you need to sort out then. >> >> Check out the mmuserauth service create command. >> >> There was an example on this list on Monday so depending when you >> subscribed you may not have seen it. FYI the command cited was: >> >> mmuserauth service create --type ad --data-access-method file --servers >> 192.168.88.3 --user-name administrator --netbios-name scale --idmap-role >> master --password ********* --idmap-range-size 1000000 --idmap-range >> 10000000-299999999 --enable-nfs-kerberos --unixmap-domains >> 'sirius(10000-20000)' >> >> Change the parameters to cater to your environment and needs of course. >> >> Richard >> >> -----Original Message----- >> From: gpfsug-discuss-bounces at spectrumscale.org [mailto: >> gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aidan Richmond >> Sent: 01 March 2017 11:36 >> To: gpfsug-discuss at spectrumscale.org >> Subject: Re: [gpfsug-discuss] Issues getting SMB shares working. >> >> Hello >> >> It returns the following: >> >> FILE access configuration : USERDEFINED >> PARAMETERS VALUES >> ------------------------------------------------- >> >> On 01/03/17 11:33, Sobey, Richard A wrote: >> >>> Mmuserauth service list --data-access-method file >>> >> >> >> > > -- > Aidan Richmond > Apple/Unix Support Officer, IT > Garstang 10.137 > Faculty of Biological Sciences > University of Leeds > Clarendon Way > LS2 9JT > > Tel:0113 3434252 > a.g.richmond at leeds.ac.uk > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.g.richmond at leeds.ac.uk Wed Mar 1 15:22:36 2017 From: a.g.richmond at leeds.ac.uk (Aidan Richmond) Date: Wed, 1 Mar 2017 15:22:36 +0000 Subject: [gpfsug-discuss] Issues getting SMB shares working. In-Reply-To: References: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> <901c6472-1c9c-3551-555e-22bc849fe2b1@leeds.ac.uk> <2ac0911f-65f7-d500-1d01-7ba4b5929a79@leeds.ac.uk> Message-ID: mmnfs export list Path Delegations Clients ------------------------------------- /absl/SCRATCH none * /absl/SCRATCH none fbscpcu097 mmnfs config list NFS Ganesha Configuration: ========================== NFS_PROTOCOLS: 3,4 NFS_PORT: 2049 MNT_PORT: 0 NLM_PORT: 0 RQUOTA_PORT: 0 SHORT_FILE_HANDLE: FALSE LEASE_LIFETIME: 60 DOMAINNAME: LEEDS.AC.UK DELEGATIONS: Disabled ========================== STATD Configuration ========================== STATD_PORT: 0 ========================== CacheInode Configuration ========================== ENTRIES_HWMARK: 1500000 ========================== Export Defaults ========================== ACCESS_TYPE: NONE PROTOCOLS: 3,4 TRANSPORTS: TCP ANONYMOUS_UID: -2 ANONYMOUS_GID: -2 SECTYPE: SYS PRIVILEGEDPORT: FALSE MANAGE_GIDS: FALSE SQUASH: ROOT_SQUASH NFS_COMMIT: FALSE ========================== Log Configuration ========================== LOG_LEVEL: EVENT ========================== Idmapd Configuration ========================== DOMAIN: DS.LEEDS.AC.UK ========================== On 01/03/17 14:12, Jan-Frode Myklebust wrote: > Lets figure out how your NFS is authenticating then. The userdefined > authentication you have, could mean that your linux host is configured to > authenticated towards AD --- or it could be that you're using simple > sys-authentication for NFS. > > Could you please post the output of: > > mmnfs export list > mmnfs config list > > > -jf > > > On Wed, Mar 1, 2017 at 1:07 PM, Aidan Richmond > wrote: > >> Hello >> >> I'm a little hesitant to mess with the authentication as I we are wanting >> consistent UIDs across our systems and I know my predecessor struggled to >> get them consistent. Our AD environment stores the UID and GID settings in >> msSFU30uid and msSFU30gid. >> >> I'm also concerned that the system is already in use with nfsv4 access and >> don't want to break existing access unless I have to. >> >> >> >> On 01/03/17 11:42, Sobey, Richard A wrote: >> >>> That's probably the first thing you need to sort out then. >>> >>> Check out the mmuserauth service create command. >>> >>> There was an example on this list on Monday so depending when you >>> subscribed you may not have seen it. FYI the command cited was: >>> >>> mmuserauth service create --type ad --data-access-method file --servers >>> 192.168.88.3 --user-name administrator --netbios-name scale --idmap-role >>> master --password ********* --idmap-range-size 1000000 --idmap-range >>> 10000000-299999999 --enable-nfs-kerberos --unixmap-domains >>> 'sirius(10000-20000)' >>> >>> Change the parameters to cater to your environment and needs of course. >>> >>> Richard >>> >>> -----Original Message----- >>> From: gpfsug-discuss-bounces at spectrumscale.org [mailto: >>> gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aidan Richmond >>> Sent: 01 March 2017 11:36 >>> To: gpfsug-discuss at spectrumscale.org >>> Subject: Re: [gpfsug-discuss] Issues getting SMB shares working. >>> >>> Hello >>> >>> It returns the following: >>> >>> FILE access configuration : USERDEFINED >>> PARAMETERS VALUES >>> ------------------------------------------------- >>> >>> On 01/03/17 11:33, Sobey, Richard A wrote: >>> >>>> Mmuserauth service list --data-access-method file >>>> >>> >>> >>> >> >> -- >> Aidan Richmond >> Apple/Unix Support Officer, IT >> Garstang 10.137 >> Faculty of Biological Sciences >> University of Leeds >> Clarendon Way >> LS2 9JT >> >> Tel:0113 3434252 >> a.g.richmond at leeds.ac.uk >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aidan Richmond Apple/Unix Support Officer, IT Garstang 10.137 Faculty of Biological Sciences University of Leeds Clarendon Way LS2 9JT Tel:0113 3434252 a.g.richmond at leeds.ac.uk From Mark.Bush at siriuscom.com Wed Mar 1 19:28:24 2017 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Wed, 1 Mar 2017 19:28:24 +0000 Subject: [gpfsug-discuss] Issues getting SMB shares working. In-Reply-To: References: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> <901c6472-1c9c-3551-555e-22bc849fe2b1@leeds.ac.uk> <2ac0911f-65f7-d500-1d01-7ba4b5929a79@leeds.ac.uk> Message-ID: I?m pretty noob still here but I think what may be going on in your situation is that since you are just now enabling SMB service you need to figure out how you want users to authenticate. If you want to stick with the way things are currently authenticating on your system that is to say, USERDEFINED that really just means it?s delegated to the underlying system for authentication. So, you can change your authentication strategy to use Active Directory (like was mentioned by Richard), or use the local SMB authentication which means you?ll have to add users manually with the /usr/lpp/mmfs/bin/smbpasswd command. This works if you just want a few users to get access to SMB shares but it becomes troublesome when you get loads of users. You can use LDAP or AD to make things more manageable. Mark On 3/1/17, 9:22 AM, "Aidan Richmond" wrote: mmnfs export list Path Delegations Clients ------------------------------------- /absl/SCRATCH none * /absl/SCRATCH none fbscpcu097 mmnfs config list NFS Ganesha Configuration: ========================== NFS_PROTOCOLS: 3,4 NFS_PORT: 2049 MNT_PORT: 0 NLM_PORT: 0 RQUOTA_PORT: 0 SHORT_FILE_HANDLE: FALSE LEASE_LIFETIME: 60 DOMAINNAME: LEEDS.AC.UK DELEGATIONS: Disabled ========================== STATD Configuration ========================== STATD_PORT: 0 ========================== CacheInode Configuration ========================== ENTRIES_HWMARK: 1500000 ========================== Export Defaults ========================== ACCESS_TYPE: NONE PROTOCOLS: 3,4 TRANSPORTS: TCP ANONYMOUS_UID: -2 ANONYMOUS_GID: -2 SECTYPE: SYS PRIVILEGEDPORT: FALSE MANAGE_GIDS: FALSE SQUASH: ROOT_SQUASH NFS_COMMIT: FALSE ========================== Log Configuration ========================== LOG_LEVEL: EVENT ========================== Idmapd Configuration ========================== DOMAIN: DS.LEEDS.AC.UK ========================== On 01/03/17 14:12, Jan-Frode Myklebust wrote: > Lets figure out how your NFS is authenticating then. The userdefined > authentication you have, could mean that your linux host is configured to > authenticated towards AD --- or it could be that you're using simple > sys-authentication for NFS. > > Could you please post the output of: > > mmnfs export list > mmnfs config list > > > -jf > > > On Wed, Mar 1, 2017 at 1:07 PM, Aidan Richmond > wrote: > >> Hello >> >> I'm a little hesitant to mess with the authentication as I we are wanting >> consistent UIDs across our systems and I know my predecessor struggled to >> get them consistent. Our AD environment stores the UID and GID settings in >> msSFU30uid and msSFU30gid. >> >> I'm also concerned that the system is already in use with nfsv4 access and >> don't want to break existing access unless I have to. >> >> >> >> On 01/03/17 11:42, Sobey, Richard A wrote: >> >>> That's probably the first thing you need to sort out then. >>> >>> Check out the mmuserauth service create command. >>> >>> There was an example on this list on Monday so depending when you >>> subscribed you may not have seen it. FYI the command cited was: >>> >>> mmuserauth service create --type ad --data-access-method file --servers >>> 192.168.88.3 --user-name administrator --netbios-name scale --idmap-role >>> master --password ********* --idmap-range-size 1000000 --idmap-range >>> 10000000-299999999 --enable-nfs-kerberos --unixmap-domains >>> 'sirius(10000-20000)' >>> >>> Change the parameters to cater to your environment and needs of course. >>> >>> Richard >>> >>> -----Original Message----- >>> From: gpfsug-discuss-bounces at spectrumscale.org [mailto: >>> gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aidan Richmond >>> Sent: 01 March 2017 11:36 >>> To: gpfsug-discuss at spectrumscale.org >>> Subject: Re: [gpfsug-discuss] Issues getting SMB shares working. >>> >>> Hello >>> >>> It returns the following: >>> >>> FILE access configuration : USERDEFINED >>> PARAMETERS VALUES >>> ------------------------------------------------- >>> >>> On 01/03/17 11:33, Sobey, Richard A wrote: >>> >>>> Mmuserauth service list --data-access-method file >>>> >>> >>> >>> >> >> -- >> Aidan Richmond >> Apple/Unix Support Officer, IT >> Garstang 10.137 >> Faculty of Biological Sciences >> University of Leeds >> Clarendon Way >> LS2 9JT >> >> Tel:0113 3434252 >> a.g.richmond at leeds.ac.uk >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aidan Richmond Apple/Unix Support Officer, IT Garstang 10.137 Faculty of Biological Sciences University of Leeds Clarendon Way LS2 9JT Tel:0113 3434252 a.g.richmond at leeds.ac.uk _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions From janfrode at tanso.net Wed Mar 1 20:21:03 2017 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Wed, 01 Mar 2017 20:21:03 +0000 Subject: [gpfsug-discuss] Issues getting SMB shares working. In-Reply-To: References: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> <901c6472-1c9c-3551-555e-22bc849fe2b1@leeds.ac.uk> <2ac0911f-65f7-d500-1d01-7ba4b5929a79@leeds.ac.uk> Message-ID: This looks to me like a quite plain SYS authorized NFS, maybe also verify that the individual NFS shares has sec=sys with "mmnfs export list -n /absl/SCRATCH". If you check "man mmuserauth" there are quite clear examples for how to connect it to AD populated with unix id and gid. I don't think this will affect your NFS service, since there doesn't seem to be any kerberos involved. But, please beware that mmuserauth will overwrite any customized sssd.conf, krb5.conf, winbind, etc.. As it configures the authentication for the whole host, not just samba/nfs-services. -jf ons. 1. mar. 2017 kl. 16.22 skrev Aidan Richmond : > mmnfs export list > Path Delegations Clients > ------------------------------------- > /absl/SCRATCH none * > /absl/SCRATCH none fbscpcu097 > > mmnfs config list > > NFS Ganesha Configuration: > ========================== > NFS_PROTOCOLS: 3,4 > NFS_PORT: 2049 > MNT_PORT: 0 > NLM_PORT: 0 > RQUOTA_PORT: 0 > SHORT_FILE_HANDLE: FALSE > LEASE_LIFETIME: 60 > DOMAINNAME: LEEDS.AC.UK > DELEGATIONS: Disabled > ========================== > > STATD Configuration > ========================== > STATD_PORT: 0 > ========================== > > CacheInode Configuration > ========================== > ENTRIES_HWMARK: 1500000 > ========================== > > Export Defaults > ========================== > ACCESS_TYPE: NONE > PROTOCOLS: 3,4 > TRANSPORTS: TCP > ANONYMOUS_UID: -2 > ANONYMOUS_GID: -2 > SECTYPE: SYS > PRIVILEGEDPORT: FALSE > MANAGE_GIDS: FALSE > SQUASH: ROOT_SQUASH > NFS_COMMIT: FALSE > ========================== > > Log Configuration > ========================== > LOG_LEVEL: EVENT > ========================== > > Idmapd Configuration > ========================== > DOMAIN: DS.LEEDS.AC.UK > ========================== > > On 01/03/17 14:12, Jan-Frode Myklebust wrote: > > Lets figure out how your NFS is authenticating then. The userdefined > > authentication you have, could mean that your linux host is configured to > > authenticated towards AD --- or it could be that you're using simple > > sys-authentication for NFS. > > > > Could you please post the output of: > > > > mmnfs export list > > mmnfs config list > > > > > > -jf > > > > > > On Wed, Mar 1, 2017 at 1:07 PM, Aidan Richmond > > > wrote: > > > >> Hello > >> > >> I'm a little hesitant to mess with the authentication as I we are > wanting > >> consistent UIDs across our systems and I know my predecessor struggled > to > >> get them consistent. Our AD environment stores the UID and GID settings > in > >> msSFU30uid and msSFU30gid. > >> > >> I'm also concerned that the system is already in use with nfsv4 access > and > >> don't want to break existing access unless I have to. > >> > >> > >> > >> On 01/03/17 11:42, Sobey, Richard A wrote: > >> > >>> That's probably the first thing you need to sort out then. > >>> > >>> Check out the mmuserauth service create command. > >>> > >>> There was an example on this list on Monday so depending when you > >>> subscribed you may not have seen it. FYI the command cited was: > >>> > >>> mmuserauth service create --type ad --data-access-method file --servers > >>> 192.168.88.3 --user-name administrator --netbios-name scale > --idmap-role > >>> master --password ********* --idmap-range-size 1000000 --idmap-range > >>> 10000000-299999999 --enable-nfs-kerberos --unixmap-domains > >>> 'sirius(10000-20000)' > >>> > >>> Change the parameters to cater to your environment and needs of course. > >>> > >>> Richard > >>> > >>> -----Original Message----- > >>> From: gpfsug-discuss-bounces at spectrumscale.org [mailto: > >>> gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aidan Richmond > >>> Sent: 01 March 2017 11:36 > >>> To: gpfsug-discuss at spectrumscale.org > >>> Subject: Re: [gpfsug-discuss] Issues getting SMB shares working. > >>> > >>> Hello > >>> > >>> It returns the following: > >>> > >>> FILE access configuration : USERDEFINED > >>> PARAMETERS VALUES > >>> ------------------------------------------------- > >>> > >>> On 01/03/17 11:33, Sobey, Richard A wrote: > >>> > >>>> Mmuserauth service list --data-access-method file > >>>> > >>> > >>> > >>> > >> > >> -- > >> Aidan Richmond > >> Apple/Unix Support Officer, IT > >> Garstang 10.137 > >> Faculty of Biological Sciences > >> University of Leeds > >> Clarendon Way > >> LS2 9JT > >> > >> Tel:0113 3434252 > >> a.g.richmond at leeds.ac.uk > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > -- > Aidan Richmond > Apple/Unix Support Officer, IT > Garstang 10.137 > Faculty of Biological Sciences > University of Leeds > Clarendon Way > LS2 9JT > > Tel:0113 3434252 > a.g.richmond at leeds.ac.uk > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Thu Mar 2 14:17:45 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Thu, 2 Mar 2017 14:17:45 +0000 Subject: [gpfsug-discuss] GPFSUG-USA Meeting at NERSC April 4-5th - Registration Reminder and Agenda Details Message-ID: Here?s a reminder to register for the upcoming user group meeting! Registration deadline is March 24th, only 3 weeks away. You won?t want to miss the 2 hour session on Tuesday on problem determination by Yuri Volobuev. Any long time GPFS admin has no doubt interacted with Yuri. He?s moved on to new challenges from GPFS, but has a wealth of problem determination tips that he wants to pass along. We?re happy to have him back in the GPFS community. We also have a few slots (20 mins each) for customer and partner talks ? please contact me if you want to do a short presentation. Register here: https://www.nersc.gov/research-and-development/data-analytics/spectrum-user-group-meeting/ Tuesday, April 4th 09:00-10:00 Registration/Coffee/Networking 10:00-10:30 Welcome - Bob Oesterlin, Kristy Kallback, Doris Conti 10:30-11:30 Spectrum Scale & ESS Update - Scott Fadden / Puneet Chaudhary 11:30-12:30 Problem Avoidance - Best Practices in the Field - IBM Support 12:30-13:30 Lunch and Networking 13:30-15:30 Problem Determination - Deep Dive - Yuri Volobuev 15:30-16:00 Break/Networking 16:00-17:00 Problem Determination - What's New? - Matthias Dietz 17:00-18:00 Bring your favorite problem or performance issue. We will fix it now! Wednesday April 5th 08:30-09:00 Coffee and Networking 09:00-10:00 Transparent Cloud Tiering - Introduction and Use Cases (Meet the Devs) - Nikhil Khandelwal or Rob Basham Life Sciences and Next Generation Sequencing with Spectrum Scale (Solutions) 10:00-11:00 AFM - Introduction and Use Cases (Meet the Devs) - Scott Fadden Spectrum Protect on Spectrum Scale (Solutions) - Jason Basler 11:00-12:00 Meet the Devs - IBM Development Hadoop Integration and Support for HortonWorks HDP (Solutions) - Ted Hoover 12:00-13:00 Lunch and Networking 13:00-13:20 Customer/Partner Talk 13:20-13:40 Customer/Partner Talk 13:40-14:00 Customer/Partner Talk 14:00-14:30 Break 14:30-15:00 Challenges in Tracing - Vasily Tarasov 15:00-15:30 Application Integration with Containers - Dean Hildebrand 15:30-16:00 Wrap up - Bob Oesterlin, Kristy Kallback, Doris Conti Bob Oesterlin Sr Principal Storage Engineer, Nuance Spectrum Scale UG-USA Co-Principal -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.g.richmond at leeds.ac.uk Thu Mar 2 15:41:40 2017 From: a.g.richmond at leeds.ac.uk (Aidan Richmond) Date: Thu, 2 Mar 2017 15:41:40 +0000 Subject: [gpfsug-discuss] Issues getting SMB shares working. In-Reply-To: References: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> <901c6472-1c9c-3551-555e-22bc849fe2b1@leeds.ac.uk> <2ac0911f-65f7-d500-1d01-7ba4b5929a79@leeds.ac.uk> Message-ID: <97b56c55-58d4-3400-f8cf-5f6745583fc9@leeds.ac.uk> mmnfs export list -n /absl/SCRATCH Path Delegations Clients Access_Type Protocols Transports Squash Anonymous_uid Anonymous_gid SecType PrivilegedPort DefaultDelegations Manage_Gids NFS_Commit ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- /absl/SCRATCH none * RW 4 TCP ROOT_SQUASH -2 -2 KRB5I FALSE none FALSE FALSE /absl/SCRATCH none fbscpcu097 RW 3 TCP ROOT_SQUASH -2 -2 sys FALSE none FALSE FALSE Ignore the fbscpcu097 entry, I think my departed colleague was just using that for testing, the NFS clients all access it though nfsv4 which looks to be using kerberos from this. On 01/03/17 20:21, Jan-Frode Myklebust wrote: > This looks to me like a quite plain SYS authorized NFS, maybe also verify > that the individual NFS shares has sec=sys with "mmnfs export list -n > /absl/SCRATCH". > > > If you check "man mmuserauth" there are quite clear examples for how to > connect it to AD populated with unix id and gid. I don't think this will > affect your NFS service, since there doesn't seem to be any kerberos > involved. > > But, please beware that mmuserauth will overwrite any customized sssd.conf, > krb5.conf, winbind, etc.. As it configures the authentication for the whole > host, not just samba/nfs-services. > > > -jf > > ons. 1. mar. 2017 kl. 16.22 skrev Aidan Richmond : > >> mmnfs export list >> Path Delegations Clients >> ------------------------------------- >> /absl/SCRATCH none * >> /absl/SCRATCH none fbscpcu097 >> >> mmnfs config list >> >> NFS Ganesha Configuration: >> ========================== >> NFS_PROTOCOLS: 3,4 >> NFS_PORT: 2049 >> MNT_PORT: 0 >> NLM_PORT: 0 >> RQUOTA_PORT: 0 >> SHORT_FILE_HANDLE: FALSE >> LEASE_LIFETIME: 60 >> DOMAINNAME: LEEDS.AC.UK >> DELEGATIONS: Disabled >> ========================== >> >> STATD Configuration >> ========================== >> STATD_PORT: 0 >> ========================== >> >> CacheInode Configuration >> ========================== >> ENTRIES_HWMARK: 1500000 >> ========================== >> >> Export Defaults >> ========================== >> ACCESS_TYPE: NONE >> PROTOCOLS: 3,4 >> TRANSPORTS: TCP >> ANONYMOUS_UID: -2 >> ANONYMOUS_GID: -2 >> SECTYPE: SYS >> PRIVILEGEDPORT: FALSE >> MANAGE_GIDS: FALSE >> SQUASH: ROOT_SQUASH >> NFS_COMMIT: FALSE >> ========================== >> >> Log Configuration >> ========================== >> LOG_LEVEL: EVENT >> ========================== >> >> Idmapd Configuration >> ========================== >> DOMAIN: DS.LEEDS.AC.UK >> ========================== >> -- Aidan Richmond Apple/Unix Support Officer, IT Garstang 10.137 Faculty of Biological Sciences University of Leeds Clarendon Way LS2 9JT Tel:0113 3434252 a.g.richmond at leeds.ac.uk From p.childs at qmul.ac.uk Thu Mar 2 16:34:09 2017 From: p.childs at qmul.ac.uk (Peter Childs) Date: Thu, 2 Mar 2017 16:34:09 +0000 Subject: [gpfsug-discuss] mmbackup logging issue In-Reply-To: References: Message-ID: We had that issue. we had to export MMBACKUP_PROGRESS_CONTENT=5 export MMBACKUP_PROGRESS_INTERVAL=300 before we run it to get it back. Lets just say IBM changed the behaviour, We ended up opening a PRM to get that answer ;) We also set -L 1 you can change how often the messages are displayed by changing MMBACKUP_PROGRESS_INTERVAL flexable but the default is different;) Peter Childs ITS Research Infrastructure Queen Mary, University of London ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Ashish Thandavan Sent: Tuesday, February 28, 2017 4:10:44 PM To: gpfsug main discussion list Subject: [gpfsug-discuss] mmbackup logging issue Dear all, We have a small GPFS cluster and a separate server running TSM and one of the three NSD servers backs up our GPFS filesystem to the TSM server using mmbackup. After a recent upgrade from v3.5 to 4.1.1, we've noticed that mmbackup no longer logs stuff like it used to : ... Thu Jan 19 05:45:41 2017 mmbackup:Backing up files: 0 backed up, 870532 expired, 2 failed. Thu Jan 19 06:15:41 2017 mmbackup:Backing up files: 0 backed up, 870532 expired, 3 failed. Thu Jan 19 06:45:41 2017 mmbackup:Backing up files: 0 backed up, 870532 expired, 3 failed. ... instead of ... Sat Dec 3 12:01:00 2016 mmbackup:Backing up files: 105030 backed up, 635456 expired, 30 failed. Sat Dec 3 12:31:00 2016 mmbackup:Backing up files: 205934 backed up, 635456 expired, 57 failed. Sat Dec 3 13:01:00 2016 mmbackup:Backing up files: 321702 backed up, 635456 expired, 169 failed. ... like it used to pre-upgrade. I am therefore unable to see how far long it has got, and indeed if it completed successfully, as this is what it logs at the end of a job : ... Tue Jan 17 18:07:31 2017 mmbackup:Completed policy backup run with 0 policy errors, 10012 files failed, 0 severe errors, returning rc=9. Tue Jan 17 18:07:31 2017 mmbackup:Policy for backup returned 9 Highest TSM error 12 mmbackup: TSM Summary Information: Total number of objects inspected: 20617273 Total number of objects backed up: 0 Total number of objects updated: 0 Total number of objects rebound: 0 Total number of objects deleted: 0 Total number of objects expired: 1 Total number of objects failed: 10012 Total number of objects encrypted: 0 Total number of bytes inspected: 3821624716861 Total number of bytes transferred: 3712040943672 Tue Jan 17 18:07:31 2017 mmbackup:Audit files /cs/mmbackup.audit.gpfs* contain 0 failed paths but there were 10012 failures. Cannot reconcile shadow database. Unable to compensate for all TSM errors in new shadow database. Preserving previous shadow database. Run next mmbackup with -q to synchronize shadow database. exit 12 If it helps, the mmbackup job is kicked off with the following options : /usr/lpp/mmfs/bin/mmbackup gpfs -n 8 -t full -B 20000 -L 1 --tsm-servers gpfs_weekly_stanza -N glossop1a | /usr/bin/tee /var/log/mmbackup/gpfs_weekly/backup_log.`date +%Y%m%d_%H_%M` (The excerpts above are from the backup_log. file.) Our NSD servers are running GPFS 4.1.1-11, TSM is at 7.1.1.100 and the File system version is 12.06 (3.4.0.3). Has anyone else seen this behaviour with mmbackup and if so, found a fix? Thanks, Regards, Ash -- ------------------------- Ashish Thandavan UNIX Support Computing Officer Department of Computer Science University of Oxford Wolfson Building Parks Road Oxford OX1 3QD Phone: 01865 610733 Email: ashish.thandavan at cs.ox.ac.uk _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From ashish.thandavan at cs.ox.ac.uk Thu Mar 2 16:50:23 2017 From: ashish.thandavan at cs.ox.ac.uk (Ashish Thandavan) Date: Thu, 2 Mar 2017 16:50:23 +0000 Subject: [gpfsug-discuss] mmbackup logging issue In-Reply-To: References: Message-ID: <6cd84459-ecc3-1ef9-da22-dea843cff154@cs.ox.ac.uk> Dear Peter, On 02/03/17 16:34, Peter Childs wrote: > We had that issue. > > we had to > > export MMBACKUP_PROGRESS_CONTENT=5 > export MMBACKUP_PROGRESS_INTERVAL=300 > > before we run it to get it back. > > Lets just say IBM changed the behaviour, We ended up opening a PRM to get that answer ;) We also set -L 1 > > you can change how often the messages are displayed by changing MMBACKUP_PROGRESS_INTERVAL flexable but the default is different;) > I'll set those variables before kicking off the next mmbackup and hope that fixes it. Thank you!! Regards, Ash -- ------------------------- Ashish Thandavan UNIX Support Computing Officer Department of Computer Science University of Oxford Wolfson Building Parks Road Oxford OX1 3QD Phone: 01865 610733 Email: ashish.thandavan at cs.ox.ac.uk From janfrode at tanso.net Thu Mar 2 20:42:23 2017 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Thu, 02 Mar 2017 20:42:23 +0000 Subject: [gpfsug-discuss] Issues getting SMB shares working. In-Reply-To: <97b56c55-58d4-3400-f8cf-5f6745583fc9@leeds.ac.uk> References: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> <901c6472-1c9c-3551-555e-22bc849fe2b1@leeds.ac.uk> <2ac0911f-65f7-d500-1d01-7ba4b5929a79@leeds.ac.uk> <97b56c55-58d4-3400-f8cf-5f6745583fc9@leeds.ac.uk> Message-ID: Ouch, yes.. Then the switch to mmuserauth is more difficult. I would recommend setting up a lab cluster (only need a single node), and use mmuserauth to connect it to AD and see that you get both kerberized NFS and SMB working by default there, before doing the same on your production cluster. -jf tor. 2. mar. 2017 kl. 16.42 skrev Aidan Richmond : > mmnfs export list -n /absl/SCRATCH > Path Delegations Clients Access_Type Protocols Transports > Squash Anonymous_uid Anonymous_gid SecType PrivilegedPort > DefaultDelegations Manage_Gids NFS_Commit > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > /absl/SCRATCH none * RW 4 TCP > ROOT_SQUASH -2 -2 KRB5I FALSE none > FALSE FALSE > /absl/SCRATCH none fbscpcu097 RW 3 TCP > ROOT_SQUASH -2 -2 sys FALSE none > FALSE FALSE > > > Ignore the fbscpcu097 entry, I think my departed colleague was just > using that for testing, the NFS clients all access it though nfsv4 which > looks to be using kerberos from this. > > On 01/03/17 20:21, Jan-Frode Myklebust wrote: > > This looks to me like a quite plain SYS authorized NFS, maybe also verify > > that the individual NFS shares has sec=sys with "mmnfs export list -n > > /absl/SCRATCH". > > > > > > If you check "man mmuserauth" there are quite clear examples for how to > > connect it to AD populated with unix id and gid. I don't think this will > > affect your NFS service, since there doesn't seem to be any kerberos > > involved. > > > > But, please beware that mmuserauth will overwrite any customized > sssd.conf, > > krb5.conf, winbind, etc.. As it configures the authentication for the > whole > > host, not just samba/nfs-services. > > > > > > -jf > > > > ons. 1. mar. 2017 kl. 16.22 skrev Aidan Richmond < > a.g.richmond at leeds.ac.uk>: > > > >> mmnfs export list > >> Path Delegations Clients > >> ------------------------------------- > >> /absl/SCRATCH none * > >> /absl/SCRATCH none fbscpcu097 > >> > >> mmnfs config list > >> > >> NFS Ganesha Configuration: > >> ========================== > >> NFS_PROTOCOLS: 3,4 > >> NFS_PORT: 2049 > >> MNT_PORT: 0 > >> NLM_PORT: 0 > >> RQUOTA_PORT: 0 > >> SHORT_FILE_HANDLE: FALSE > >> LEASE_LIFETIME: 60 > >> DOMAINNAME: LEEDS.AC.UK > >> DELEGATIONS: Disabled > >> ========================== > >> > >> STATD Configuration > >> ========================== > >> STATD_PORT: 0 > >> ========================== > >> > >> CacheInode Configuration > >> ========================== > >> ENTRIES_HWMARK: 1500000 > >> ========================== > >> > >> Export Defaults > >> ========================== > >> ACCESS_TYPE: NONE > >> PROTOCOLS: 3,4 > >> TRANSPORTS: TCP > >> ANONYMOUS_UID: -2 > >> ANONYMOUS_GID: -2 > >> SECTYPE: SYS > >> PRIVILEGEDPORT: FALSE > >> MANAGE_GIDS: FALSE > >> SQUASH: ROOT_SQUASH > >> NFS_COMMIT: FALSE > >> ========================== > >> > >> Log Configuration > >> ========================== > >> LOG_LEVEL: EVENT > >> ========================== > >> > >> Idmapd Configuration > >> ========================== > >> DOMAIN: DS.LEEDS.AC.UK > >> ========================== > >> > > -- > Aidan Richmond > Apple/Unix Support Officer, IT > Garstang 10.137 > Faculty of Biological Sciences > University of Leeds > Clarendon Way > LS2 9JT > > Tel:0113 3434252 > a.g.richmond at leeds.ac.uk > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Fri Mar 3 09:20:24 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Fri, 3 Mar 2017 09:20:24 +0000 Subject: [gpfsug-discuss] mmbackup logging issue In-Reply-To: <6cd84459-ecc3-1ef9-da22-dea843cff154@cs.ox.ac.uk> References: <6cd84459-ecc3-1ef9-da22-dea843cff154@cs.ox.ac.uk> Message-ID: HI all We have the same problem (less of a problem, more lack of visibilitiy). Can I just add those lines to the top of our mmbackup.sh script? -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Ashish Thandavan Sent: 02 March 2017 16:50 To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] mmbackup logging issue Dear Peter, On 02/03/17 16:34, Peter Childs wrote: > We had that issue. > > we had to > > export MMBACKUP_PROGRESS_CONTENT=5 > export MMBACKUP_PROGRESS_INTERVAL=300 > > before we run it to get it back. > > Lets just say IBM changed the behaviour, We ended up opening a PRM to > get that answer ;) We also set -L 1 > > you can change how often the messages are displayed by changing > MMBACKUP_PROGRESS_INTERVAL flexable but the default is different;) > I'll set those variables before kicking off the next mmbackup and hope that fixes it. Thank you!! Regards, Ash -- ------------------------- Ashish Thandavan UNIX Support Computing Officer Department of Computer Science University of Oxford Wolfson Building Parks Road Oxford OX1 3QD Phone: 01865 610733 Email: ashish.thandavan at cs.ox.ac.uk _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From p.childs at qmul.ac.uk Fri Mar 3 10:44:03 2017 From: p.childs at qmul.ac.uk (Peter Childs) Date: Fri, 3 Mar 2017 10:44:03 +0000 Subject: [gpfsug-discuss] mmbackup logging issue In-Reply-To: References: <6cd84459-ecc3-1ef9-da22-dea843cff154@cs.ox.ac.uk>, Message-ID: That's basically what we did, They are only environment variables, so if you not using bash to call mmbackup you will need to change the lines accordingly. What they do is in the manual the issue is the default changed between versions..... Peter Childs ITS Research Infrastructure Queen Mary, University of London ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Sobey, Richard A Sent: Friday, March 3, 2017 9:20:24 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] mmbackup logging issue HI all We have the same problem (less of a problem, more lack of visibilitiy). Can I just add those lines to the top of our mmbackup.sh script? -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Ashish Thandavan Sent: 02 March 2017 16:50 To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] mmbackup logging issue Dear Peter, On 02/03/17 16:34, Peter Childs wrote: > We had that issue. > > we had to > > export MMBACKUP_PROGRESS_CONTENT=5 > export MMBACKUP_PROGRESS_INTERVAL=300 > > before we run it to get it back. > > Lets just say IBM changed the behaviour, We ended up opening a PRM to > get that answer ;) We also set -L 1 > > you can change how often the messages are displayed by changing > MMBACKUP_PROGRESS_INTERVAL flexable but the default is different;) > I'll set those variables before kicking off the next mmbackup and hope that fixes it. Thank you!! Regards, Ash -- ------------------------- Ashish Thandavan UNIX Support Computing Officer Department of Computer Science University of Oxford Wolfson Building Parks Road Oxford OX1 3QD Phone: 01865 610733 Email: ashish.thandavan at cs.ox.ac.uk _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From jtucker at pixitmedia.com Fri Mar 3 12:59:48 2017 From: jtucker at pixitmedia.com (Jez Tucker) Date: Fri, 3 Mar 2017 12:59:48 +0000 Subject: [gpfsug-discuss] mmbackup logging issue In-Reply-To: References: Message-ID: <7ca99092-63e6-c1a1-d9a7-e1907bcef7c8@pixitmedia.com> Hi Ash, This is the primary reason for using snapshots for mmbackup ( -S snapname ) and making sure that only mmbackup sends data to backup rather than an oob dsmc: Tue Jan 17 18:07:31 2017 mmbackup:Audit files /cs/mmbackup.audit.gpfs* contain 0 failed paths but there were 10012 failures. Cannot reconcile shadow database. Unable to compensate for all TSM errors in new shadow database. Preserving previous shadow database. Run next mmbackup with -q to synchronize shadow database. exit 12 <------------ ick! Other obvious causes are very, very, odd filenames / paths. Jez On 28/02/17 16:10, Ashish Thandavan wrote: > Dear all, > > We have a small GPFS cluster and a separate server running TSM and one > of the three NSD servers backs up our GPFS filesystem to the TSM > server using mmbackup. After a recent upgrade from v3.5 to 4.1.1, > we've noticed that mmbackup no longer logs stuff like it used to : > > ... > Thu Jan 19 05:45:41 2017 mmbackup:Backing up files: 0 backed up, > 870532 expired, 2 failed. > Thu Jan 19 06:15:41 2017 mmbackup:Backing up files: 0 backed up, > 870532 expired, 3 failed. > Thu Jan 19 06:45:41 2017 mmbackup:Backing up files: 0 backed up, > 870532 expired, 3 failed. > ... > > > instead of > > ... > Sat Dec 3 12:01:00 2016 mmbackup:Backing up files: 105030 backed up, > 635456 expired, 30 failed. > Sat Dec 3 12:31:00 2016 mmbackup:Backing up files: 205934 backed up, > 635456 expired, 57 failed. > Sat Dec 3 13:01:00 2016 mmbackup:Backing up files: 321702 backed up, > 635456 expired, 169 failed. > ... > > like it used to pre-upgrade. > > I am therefore unable to see how far long it has got, and indeed if it > completed successfully, as this is what it logs at the end of a job : > > ... > Tue Jan 17 18:07:31 2017 mmbackup:Completed policy backup run with 0 > policy errors, 10012 files failed, 0 severe errors, returning rc=9. > Tue Jan 17 18:07:31 2017 mmbackup:Policy for backup returned 9 Highest > TSM error 12 > mmbackup: TSM Summary Information: > Total number of objects inspected: 20617273 > Total number of objects backed up: 0 > Total number of objects updated: 0 > Total number of objects rebound: 0 > Total number of objects deleted: 0 > Total number of objects expired: 1 > Total number of objects failed: 10012 > Total number of objects encrypted: 0 > Total number of bytes inspected: 3821624716861 > Total number of bytes transferred: 3712040943672 > Tue Jan 17 18:07:31 2017 mmbackup:Audit files /cs/mmbackup.audit.gpfs* > contain 0 failed paths but there were 10012 failures. > Cannot reconcile shadow database. > Unable to compensate for all TSM errors in new shadow database. > Preserving previous shadow database. > Run next mmbackup with -q to synchronize shadow database. exit 12 > > If it helps, the mmbackup job is kicked off with the following options : > /usr/lpp/mmfs/bin/mmbackup gpfs -n 8 -t full -B 20000 -L 1 > --tsm-servers gpfs_weekly_stanza -N glossop1a | /usr/bin/tee > /var/log/mmbackup/gpfs_weekly/backup_log.`date +%Y%m%d_%H_%M` > > (The excerpts above are from the backup_log. file.) > > Our NSD servers are running GPFS 4.1.1-11, TSM is at 7.1.1.100 and the > File system version is 12.06 (3.4.0.3). Has anyone else seen this > behaviour with mmbackup and if so, found a fix? > > Thanks, > > Regards, > Ash > -- *Jez Tucker* Head of Research and Development, Pixit Media 07764193820 | jtucker at pixitmedia.com www.pixitmedia.com | Tw:@pixitmedia.com -- This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ashish.thandavan at cs.ox.ac.uk Fri Mar 3 13:05:56 2017 From: ashish.thandavan at cs.ox.ac.uk (Ashish Thandavan) Date: Fri, 3 Mar 2017 13:05:56 +0000 Subject: [gpfsug-discuss] mmbackup logging issue In-Reply-To: <7ca99092-63e6-c1a1-d9a7-e1907bcef7c8@pixitmedia.com> References: <7ca99092-63e6-c1a1-d9a7-e1907bcef7c8@pixitmedia.com> Message-ID: <5882c991-4cb0-4658-e25b-b2a1bbc26a48@cs.ox.ac.uk> Hi Jez, > > This is the primary reason for using snapshots for mmbackup ( -S > snapname ) and making sure that only mmbackup sends data to backup > rather than an oob dsmc: > > Tue Jan 17 18:07:31 2017 mmbackup:Audit files /cs/mmbackup.audit.gpfs* > contain 0 failed paths but there were 10012 failures. > Cannot reconcile shadow database. > Unable to compensate for all TSM errors in new shadow database. > Preserving previous shadow database. > Run next mmbackup with -q to synchronize shadow database. exit 12 > <------------ ick! > I was going to send a separate email about this, but as you mention it -- I was wondering if the TSM server requires the shadow files to also be backed up? And that reconciliation of the shadow database will fail if they are not? (Unless of course, you mean above that the shadow database failure is purely on account of the 10012 failures...) Re the use of snapshots for mmbackup, are you saying dsmc does not get involved in sending the files to the TSM server? I haven't used snapshots for mmbackup, but will certainly try! Regards, Ash -- ------------------------- Ashish Thandavan UNIX Support Computing Officer Department of Computer Science University of Oxford Wolfson Building Parks Road Oxford OX1 3QD Phone: 01865 610733 Email: ashish.thandavan at cs.ox.ac.uk From jez at rib-it.org Fri Mar 3 13:19:24 2017 From: jez at rib-it.org (Jez Tucker) Date: Fri, 3 Mar 2017 13:19:24 +0000 Subject: [gpfsug-discuss] mmbackup logging issue In-Reply-To: <5882c991-4cb0-4658-e25b-b2a1bbc26a48@cs.ox.ac.uk> References: <7ca99092-63e6-c1a1-d9a7-e1907bcef7c8@pixitmedia.com> <5882c991-4cb0-4658-e25b-b2a1bbc26a48@cs.ox.ac.uk> Message-ID: <754ae5a7-82e0-17a1-2ab6-b1538bcf4e73@rib-it.org> Comments inline... On 03/03/17 13:05, Ashish Thandavan wrote: > Hi Jez, > >> >> This is the primary reason for using snapshots for mmbackup ( -S >> snapname ) and making sure that only mmbackup sends data to backup >> rather than an oob dsmc: >> >> Tue Jan 17 18:07:31 2017 mmbackup:Audit files >> /cs/mmbackup.audit.gpfs* contain 0 failed paths but there were 10012 >> failures. >> Cannot reconcile shadow database. >> Unable to compensate for all TSM errors in new shadow database. >> Preserving previous shadow database. >> Run next mmbackup with -q to synchronize shadow database. exit 12 >> <------------ ick! >> > > I was going to send a separate email about this, but as you mention it > -- I was wondering if the TSM server requires the shadow files to also > be backed up? And that reconciliation of the shadow database will fail > if they are not? No, that's not a requirement. > (Unless of course, you mean above that the shadow database failure is > purely on account of the 10012 failures...) More than likely. Check the dsmerror.log and the mmbackup logs. There are also other possibilities which could exhibit the similar cases. > > Re the use of snapshots for mmbackup, are you saying dsmc does not get > involved in sending the files to the TSM server? I haven't used > snapshots for mmbackup, but will certainly try! Snapshots provide a consistent dataset to backup. I.E. no errors from 'trying to backup a file which has been deleted since I first knew about it in the scan phase'. But as per previous related thread 'Tracking deleted files', YMMV depending on your workload and environment. > > Regards, > Ash > From chair at spectrumscale.org Fri Mar 3 15:23:00 2017 From: chair at spectrumscale.org (GPFS UG Chair (Simon Thompson)) Date: Fri, 03 Mar 2017 15:23:00 +0000 Subject: [gpfsug-discuss] Save the Date 9th/10th May - UK/European/Global (?) Spectrum Scale user group meeting Message-ID: Hi All, Save the date for the next UK Spectrum Scale user group meeting on 9th/10th May 2017. As last year, this will be a 2 day meeting and having out-grown IBM South Bank client centre who have hosted us over the past few years, we're heading off to Manchester to a bigger venue. Further details on registration and agenda will be available in the next few weeks. Thanks to IBM for supporting the event funding the venue, we are working up our sponsorship package for other companies for an evening function. This year we are able to take up to 200 delegates and we're expecting that it will be expanded out to other European customers looking for an English language event, and we're flattered that Doug listed us a the Global event :-D Thanks Simon (Group Chair) From ewahl at osc.edu Fri Mar 3 18:50:55 2017 From: ewahl at osc.edu (Edward Wahl) Date: Fri, 3 Mar 2017 13:50:55 -0500 Subject: [gpfsug-discuss] mmbackup logging issue In-Reply-To: <7ca99092-63e6-c1a1-d9a7-e1907bcef7c8@pixitmedia.com> References: <7ca99092-63e6-c1a1-d9a7-e1907bcef7c8@pixitmedia.com> Message-ID: <20170303135055.7f046983@osc.edu> Comments inline On Fri, 3 Mar 2017 12:59:48 +0000 Jez Tucker wrote: > Hi Ash, > > This is the primary reason for using snapshots for mmbackup ( -S > snapname ) and making sure that only mmbackup sends data to backup > rather than an oob dsmc: > > Tue Jan 17 18:07:31 2017 mmbackup:Audit files /cs/mmbackup.audit.gpfs* > contain 0 failed paths but there were 10012 failures. > Cannot reconcile shadow database. > Unable to compensate for all TSM errors in new shadow database. > Preserving previous shadow database. > Run next mmbackup with -q to synchronize shadow database. exit 12 > <------------ ick! > > Other obvious causes are very, very, odd filenames / paths. Yes, GPFS allows much more lenient filenames than TSM does. You can attempt to cut this back a bit with 'WILDCARDSARELITERAL yes' and 'QUOTESARELITERAL yes' in your dsm.opt (and I recommend SKIPACLUPDATECHECK yes and UPDATECTIME yes ) But this still won't catch everything. We still tend to find foreign character sets, control sequences that look like people trying to exit emacs, etc. in file names. Valid for the Filesystem, not so much for TSM. Ed > > Jez > > > On 28/02/17 16:10, Ashish Thandavan wrote: > > Dear all, > > > > We have a small GPFS cluster and a separate server running TSM and one > > of the three NSD servers backs up our GPFS filesystem to the TSM > > server using mmbackup. After a recent upgrade from v3.5 to 4.1.1, > > we've noticed that mmbackup no longer logs stuff like it used to : > > > > ... > > Thu Jan 19 05:45:41 2017 mmbackup:Backing up files: 0 backed up, > > 870532 expired, 2 failed. > > Thu Jan 19 06:15:41 2017 mmbackup:Backing up files: 0 backed up, > > 870532 expired, 3 failed. > > Thu Jan 19 06:45:41 2017 mmbackup:Backing up files: 0 backed up, > > 870532 expired, 3 failed. > > ... > > > > > > instead of > > > > ... > > Sat Dec 3 12:01:00 2016 mmbackup:Backing up files: 105030 backed up, > > 635456 expired, 30 failed. > > Sat Dec 3 12:31:00 2016 mmbackup:Backing up files: 205934 backed up, > > 635456 expired, 57 failed. > > Sat Dec 3 13:01:00 2016 mmbackup:Backing up files: 321702 backed up, > > 635456 expired, 169 failed. > > ... > > > > like it used to pre-upgrade. > > > > I am therefore unable to see how far long it has got, and indeed if it > > completed successfully, as this is what it logs at the end of a job : > > > > ... > > Tue Jan 17 18:07:31 2017 mmbackup:Completed policy backup run with 0 > > policy errors, 10012 files failed, 0 severe errors, returning rc=9. > > Tue Jan 17 18:07:31 2017 mmbackup:Policy for backup returned 9 Highest > > TSM error 12 > > mmbackup: TSM Summary Information: > > Total number of objects inspected: 20617273 > > Total number of objects backed up: 0 > > Total number of objects updated: 0 > > Total number of objects rebound: 0 > > Total number of objects deleted: 0 > > Total number of objects expired: 1 > > Total number of objects failed: 10012 > > Total number of objects encrypted: 0 > > Total number of bytes inspected: 3821624716861 > > Total number of bytes transferred: 3712040943672 > > Tue Jan 17 18:07:31 2017 mmbackup:Audit files /cs/mmbackup.audit.gpfs* > > contain 0 failed paths but there were 10012 failures. > > Cannot reconcile shadow database. > > Unable to compensate for all TSM errors in new shadow database. > > Preserving previous shadow database. > > Run next mmbackup with -q to synchronize shadow database. exit 12 > > > > If it helps, the mmbackup job is kicked off with the following options : > > /usr/lpp/mmfs/bin/mmbackup gpfs -n 8 -t full -B 20000 -L 1 > > --tsm-servers gpfs_weekly_stanza -N glossop1a | /usr/bin/tee > > /var/log/mmbackup/gpfs_weekly/backup_log.`date +%Y%m%d_%H_%M` > > > > (The excerpts above are from the backup_log. file.) > > > > Our NSD servers are running GPFS 4.1.1-11, TSM is at 7.1.1.100 and the > > File system version is 12.06 (3.4.0.3). Has anyone else seen this > > behaviour with mmbackup and if so, found a fix? > > > > Thanks, > > > > Regards, > > Ash > > > > -- Ed Wahl Ohio Supercomputer Center 614-292-9302 From joseph_grace at us.ibm.com Fri Mar 3 19:35:48 2017 From: joseph_grace at us.ibm.com (Joseph Grace) Date: Fri, 3 Mar 2017 14:35:48 -0500 Subject: [gpfsug-discuss] shutdown Message-ID: An HTML attachment was scrubbed... URL: From janfrode at tanso.net Fri Mar 3 20:54:23 2017 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Fri, 03 Mar 2017 20:54:23 +0000 Subject: [gpfsug-discuss] shutdown In-Reply-To: References: Message-ID: Unmount filesystems cleanly "mmumount all -a", stop gpfs "mmshutdown -N gss_ppc64" and poweroff "xdsh gss_ppc64 poweroff". -jf fre. 3. mar. 2017 kl. 20.55 skrev Joseph Grace : > Please excuse the newb question but we have a planned power outage coming > and I can't locate a procedure to shutdown an ESSp GL2 system? > Thanks, > > Joe Grace > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jake.carroll at uq.edu.au Sat Mar 4 20:34:51 2017 From: jake.carroll at uq.edu.au (Jake Carroll) Date: Sat, 4 Mar 2017 20:34:51 +0000 Subject: [gpfsug-discuss] Quota issues, eviction, AFM won't stop throwing data to a full location - probably a rookie AFM mistake? Message-ID: <168ADEDD-5712-42E8-9003-CAE63B2F1192@uq.edu.au> Hi all, I think I need some help with GPFS quotas and hard limits vs soft limits + eviction in AFM scenarios. We?ve got a couple of issues: One: ------- We?ve come across a scenario where if a user hits the hard quota while ingesting into cache in an AFM ?home to cache? relationship whilst an eviction loop is being triggered, things seem to go wrong ? and the filesystem runs off into locking up territory. The report I have on the last incident is that a file-set got stuck at 100% (capacity utilisation), the eviction loop either failed or blocked and the IO requests blocked and/or failed (this one I'm a little fuzzy on). Maybe it isn?t a bug and our guess is that someone on here will probably tell us that the likely ?fix? is to right-size our high and low water marks appropriately. We considered a potential bug mechanism or race condition if the eviction loop uses space in the file-set in the .afm directory ? but I then thought better of it and though ?Nah, surely IBM would have thought of that!?. Two: ------- We witness a scenario where AFM doesn't back off if it gets a filesystem full error code when trying to make the cache clean in migrating data to ?home?. If this takes a couple of seconds to raise the error each attempt, gpfs/mmfsd will deplete NFS daemons causing a DoS against the NFS server that is powering the cache/home relationship for the AFM transport. We had a mental model that AFM cache wouldn?t or shouldn?t overload hard and soft quota as the high and low watermarks for cache eviction policies. I guess in our heads, we?d like caches to also enforce and respect quotas based on requests received from home. There are probably lots of reasons this doesn?t make sense programmatically, or to the rest of scale ? but it would (seem to us) that it would clean up this problem or at least some of it. Happy to chat through his further and explain it more if anyone is interested. If there are any AFM users out there, we?d love to hear about how you deal with quotas, hwm/lwm and eviction over-flow scenarios, if they exist in your environment. Thank you as always, list. -jc -------------- next part -------------- An HTML attachment was scrubbed... URL: From mweil at wustl.edu Tue Mar 7 20:21:42 2017 From: mweil at wustl.edu (Matt Weil) Date: Tue, 7 Mar 2017 14:21:42 -0600 Subject: [gpfsug-discuss] numaMemoryInterleave=yes Message-ID: <85f1c29c-c71d-2d43-bda7-65a7278c1c40@wustl.edu> Hello all Is this necessary any more? numastat -p mmfsd seems to spread it out without it. Thanks Matt ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. From Robert.Oesterlin at nuance.com Tue Mar 7 20:32:32 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Tue, 7 Mar 2017 20:32:32 +0000 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? Message-ID: I?m considering enabling trace on all nodes all the time, doing something like this: mmtracectl --set --trace=def --trace-recycle=global --tracedev-write-mode=overwrite --tracedev-overwrite-buffer-size=256M mmtracectl --start My questions are: - What is the performance penalty of leaving this on 100% of the time on a node? - Does anyone have any suggestions on automation on stopping trace when a particular event occurs? - What other issues, if any? Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Tue Mar 7 20:53:44 2017 From: bbanister at jumptrading.com (Bryan Banister) Date: Tue, 7 Mar 2017 20:53:44 +0000 Subject: [gpfsug-discuss] numaMemoryInterleave=yes In-Reply-To: <85f1c29c-c71d-2d43-bda7-65a7278c1c40@wustl.edu> References: <85f1c29c-c71d-2d43-bda7-65a7278c1c40@wustl.edu> Message-ID: I believe the last I saw from Yuri was that this should always be enabled... but don't recall if there are other things this tunes outside of the numactl stuff, -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Tuesday, March 07, 2017 2:22 PM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] numaMemoryInterleave=yes Hello all Is this necessary any more? numastat -p mmfsd seems to spread it out without it. Thanks Matt ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. From bbanister at jumptrading.com Tue Mar 7 20:58:15 2017 From: bbanister at jumptrading.com (Bryan Banister) Date: Tue, 7 Mar 2017 20:58:15 +0000 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? In-Reply-To: References: Message-ID: The performance impact can be quite significant depending on what you are tracing. We even having monitoring that looks for long running traces and the recommended action is to ?kill with impunity!!? I believe IBM recommends never running clusters with continuous tracing. -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Oesterlin, Robert Sent: Tuesday, March 07, 2017 2:33 PM To: gpfsug main discussion list Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? I?m considering enabling trace on all nodes all the time, doing something like this: mmtracectl --set --trace=def --trace-recycle=global --tracedev-write-mode=overwrite --tracedev-overwrite-buffer-size=256M mmtracectl --start My questions are: - What is the performance penalty of leaving this on 100% of the time on a node? - Does anyone have any suggestions on automation on stopping trace when a particular event occurs? - What other issues, if any? Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Tue Mar 7 21:11:33 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Tue, 7 Mar 2017 21:11:33 +0000 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? Message-ID: I?ve been told that V70o0 unified nodes (GPFS under the covers) run with tracing enabled all the time.. but I agree with you Brian on the potential impacts. But when you must catch a trace for a problem that occurs once every few weeks ? how else would I do it? Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 From: on behalf of Bryan Banister Reply-To: gpfsug main discussion list Date: Tuesday, March 7, 2017 at 2:58 PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? I believe IBM recommends never running clusters with continuous tracing. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Tue Mar 7 21:17:35 2017 From: bbanister at jumptrading.com (Bryan Banister) Date: Tue, 7 Mar 2017 21:17:35 +0000 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? In-Reply-To: References: Message-ID: <098c950ba76e454d861d784f7f66cba1@jumptrading.com> Just depends on how your problem is detected? is it in a log? Is it found by running a command (.e.g mm*)? Is it discovered in `ps` output? Is your scheduler failing jobs? We have ongoing monitoring of most of these types of problem detection points and an automated process to capture a trace could be added to trigger upon detection depending on how the problem is detected? -B From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Oesterlin, Robert Sent: Tuesday, March 07, 2017 3:12 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? I?ve been told that V70o0 unified nodes (GPFS under the covers) run with tracing enabled all the time.. but I agree with you Brian on the potential impacts. But when you must catch a trace for a problem that occurs once every few weeks ? how else would I do it? Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 From: > on behalf of Bryan Banister > Reply-To: gpfsug main discussion list > Date: Tuesday, March 7, 2017 at 2:58 PM To: gpfsug main discussion list > Subject: [EXTERNAL] Re: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? I believe IBM recommends never running clusters with continuous tracing. ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Tue Mar 7 21:25:05 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Tue, 7 Mar 2017 21:25:05 +0000 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? Message-ID: mmfsd crash - IBM says, ?we need a trace to debug the issue?. Sigh Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Bryan Banister Reply-To: gpfsug main discussion list Date: Tuesday, March 7, 2017 at 3:17 PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? Just depends on how your problem is detected? is it in a log? Is it found by running a command (.e.g mm*)? Is it discovered in `ps` output? Is your scheduler failing jobs? -------------- next part -------------- An HTML attachment was scrubbed... URL: From valdis.kletnieks at vt.edu Tue Mar 7 21:36:44 2017 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Tue, 07 Mar 2017 16:36:44 -0500 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? In-Reply-To: <098c950ba76e454d861d784f7f66cba1@jumptrading.com> References: <098c950ba76e454d861d784f7f66cba1@jumptrading.com> Message-ID: <19135.1488922604@turing-police.cc.vt.edu> On Tue, 07 Mar 2017 21:17:35 +0000, Bryan Banister said: > Just depends on how your problem is detected??? is it in a log? Is it found by > running a command (.e.g mm*)? Is it discovered in `ps` output? Is your > scheduler failing jobs? I think the problem here is that if you have a sudden cataclysmic event, you want to have been in flight-recorder mode and be able to look at the last 5 or 10 seconds of trace *before* you became aware that your filesystem just went walkies. Sure, you can start tracing when the filesystem dies - but at that point you just get a whole mess of failed I/O requests in the trace, and no hint of where things went south... -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 484 bytes Desc: not available URL: From aaron.s.knister at nasa.gov Tue Mar 7 21:51:23 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 7 Mar 2017 16:51:23 -0500 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? In-Reply-To: References: Message-ID: <6bc4cf00-7904-0ab3-5f93-963beaca1dbc@nasa.gov> Hi Bob, I have the impression the biggest impact is to metadata-type operations rather than throughput but don't quote me on that because I have very little data to back it up. In the process of testing upgrading from GPFS 3.5 to 4.1 we ran fio on 1000 some nodes against an FS in our test environment which sustained about 60-80k iops on the filesystem's metadata LUNs. At one point I couldn't understand why I was struggling to get about 13k iops and realized tracing was turned on on some subset of nsd servers (which are also manager nodes). After turning it off the throughput immediately shot back up to where I was expecting it to be. Also during testing we were tracking down a bug for which I needed to run tracing *everywhere* and then turn it off when one of the manager nodes saw a particular error. I used a script IBM had sent me a while back to help with this that I made some tweaks to. I've attached it in case its helpful. In a nutshell the process looks like: - start tracing everywhere (/usr/lpp/mmfs/bin/mmdsh -Nall /usr/lpp/mmfs/bin/mmtrace start). Doing it this way avoids the need to change the sdrfs file which depending on your cluster size may or may not have some benefits. - run a command to watch for the event in question that when triggered runs /usr/lpp/mmfs/bin/mmdsh -Nall /usr/lpp/mmfs/bin/mmtrace stop If the condition could present itself on multiple nodes within quick succession (as was the case for me) you could wrap the mmdsh for stopping tracing in an flock, using an arbitrary node that stores the lock locally: ssh $stopHost flock -xn /tmp/mmfsTraceStopLock -c "'/usr/lpp/mmfs/bin/mmdsh -N all /usr/lpp/mmfs/bin/mmtrace stop'" Wrapping it in an flock avoids multiple trace format format attempts. -Aaron On 3/7/17 3:32 PM, Oesterlin, Robert wrote: > I?m considering enabling trace on all nodes all the time, doing > something like this: > > > > mmtracectl --set --trace=def --trace-recycle=global > --tracedev-write-mode=overwrite --tracedev-overwrite-buffer-size=256M > mmtracectl --start > > > > My questions are: > > > > - What is the performance penalty of leaving this on 100% of the time on > a node? > > - Does anyone have any suggestions on automation on stopping trace when > a particular event occurs? > > - What other issues, if any? > > > > > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > 507-269-0413 > > > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 -------------- next part -------------- #!/usr/bin/ksh stopHost=loremds20 mmtrace=/usr/lpp/mmfs/bin/mmtrace mmtracectl=/usr/lpp/mmfs/bin/mmtracectl # No automatic start of mmtrace. # Second to sleep between checking. secondsToSleep=2 # Flag to know when tripped or stopped tripped=0 # mmfs log file to monitor logToGrep=/var/log/messages # Path to mmfs bin directory MMFSbin=/usr/lpp/mmfs/bin # Trip file. Will exist if trap is sprung trapHasSprung=/tmp/mmfsTrapHasSprung rm $trapHasSprung 2>/dev/null # Start tracing on this node #${mmtrace} start # Initial count of expelled message in mmfs log baseCount=$(grep "unmounted by the system with return code 301 reason code" $logToGrep | wc -l) # do this loop while the trip file does not exist while [[ ! -f $trapHasSprung ]] do sleep $secondsToSleep # Get current count of expelled to check against the initial. currentCount=$(grep "unmounted by the system with return code 301 reason code" $logToGrep | wc -l) if [[ $currentCount > $baseCount ]] then tripped=1 /usr/lpp/mmfs/bin/mmdsh -N managernodes,quorumnodes touch $trapHasSprung # cluster manager? #stopHost=$(/usr/lpp/mmfs/bin/tslsmgr | grep '^Cluster manager' | awk '{ print $NF }' | sed -e 's/[()]//g') ssh $stopHost flock -xn /tmp/mmfsTraceStopLock -c "'/usr/lpp/mmfs/bin/mmdsh -N all -f128 /usr/lpp/mmfs/bin/mmtrace stop noformat'" fi done From r.sobey at imperial.ac.uk Wed Mar 8 06:53:17 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Wed, 8 Mar 2017 06:53:17 +0000 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? In-Reply-To: <19135.1488922604@turing-police.cc.vt.edu> References: <098c950ba76e454d861d784f7f66cba1@jumptrading.com> <19135.1488922604@turing-police.cc.vt.edu> Message-ID: We're in the same boat...gpfs snap hangs when the cluster / node is unresponsive but they don't know how to give us a root cause without one. Very frustrating. -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of valdis.kletnieks at vt.edu Sent: 07 March 2017 21:37 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? On Tue, 07 Mar 2017 21:17:35 +0000, Bryan Banister said: > Just depends on how your problem is detected??? is it in a log? Is it > found by running a command (.e.g mm*)? Is it discovered in `ps` > output? Is your scheduler failing jobs? I think the problem here is that if you have a sudden cataclysmic event, you want to have been in flight-recorder mode and be able to look at the last 5 or 10 seconds of trace *before* you became aware that your filesystem just went walkies. Sure, you can start tracing when the filesystem dies - but at that point you just get a whole mess of failed I/O requests in the trace, and no hint of where things went south... From vpuvvada at in.ibm.com Wed Mar 8 10:28:35 2017 From: vpuvvada at in.ibm.com (Venkateswara R Puvvada) Date: Wed, 8 Mar 2017 15:58:35 +0530 Subject: [gpfsug-discuss] Quota issues, eviction, AFM won't stop throwing data to a full location - probably a rookie AFM mistake? In-Reply-To: <168ADEDD-5712-42E8-9003-CAE63B2F1192@uq.edu.au> References: <168ADEDD-5712-42E8-9003-CAE63B2F1192@uq.edu.au> Message-ID: 1. What is the version of GPFS ? Eviction should not be blocking the applications. Was partial file caching enabled ? Eviction cannot evict partially cached files in recent releases. Eviction does not use space inside .afm directory, and its logs are stored under /var/mmfs/tmp by default. 2. I did not understand this requirement. a. When IO to home fails with quota exceeded or no space error , the messages are requeued at gateway node and will be retried later (usually 15 minutes). Cache cannot read home quotas today, and in most of the cases this is not valid. b. When soft quota is exceeded on AFM fileset, auto eviction clears data blocks on files based on LRU policy to bring quota below soft limit. These evicted files are uncached and there is no real migration of data to home during eviction. Eviction should get triggered before fileset usage nearing hard quota and applications getting errors. ~Venkat (vpuvvada at in.ibm.com) From: Jake Carroll To: "gpfsug-discuss at spectrumscale.org" Date: 03/05/2017 02:05 AM Subject: [gpfsug-discuss] Quota issues, eviction, AFM won't stop throwing data to a full location - probably a rookie AFM mistake? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi all, I think I need some help with GPFS quotas and hard limits vs soft limits + eviction in AFM scenarios. We?ve got a couple of issues: One: ------- We?ve come across a scenario where if a user hits the hard quota while ingesting into cache in an AFM ?home to cache? relationship whilst an eviction loop is being triggered, things seem to go wrong ? and the filesystem runs off into locking up territory. The report I have on the last incident is that a file-set got stuck at 100% (capacity utilisation), the eviction loop either failed or blocked and the IO requests blocked and/or failed (this one I'm a little fuzzy on). Maybe it isn?t a bug and our guess is that someone on here will probably tell us that the likely ?fix? is to right-size our high and low water marks appropriately. We considered a potential bug mechanism or race condition if the eviction loop uses space in the file-set in the .afm directory ? but I then thought better of it and though ?Nah, surely IBM would have thought of that!?. Two: ------- We witness a scenario where AFM doesn't back off if it gets a filesystem full error code when trying to make the cache clean in migrating data to ?home?. If this takes a couple of seconds to raise the error each attempt, gpfs/mmfsd will deplete NFS daemons causing a DoS against the NFS server that is powering the cache/home relationship for the AFM transport. We had a mental model that AFM cache wouldn?t or shouldn?t overload hard and soft quota as the high and low watermarks for cache eviction policies. I guess in our heads, we?d like caches to also enforce and respect quotas based on requests received from home. There are probably lots of reasons this doesn?t make sense programmatically, or to the rest of scale ? but it would (seem to us) that it would clean up this problem or at least some of it. Happy to chat through his further and explain it more if anyone is interested. If there are any AFM users out there, we?d love to hear about how you deal with quotas, hwm/lwm and eviction over-flow scenarios, if they exist in your environment. Thank you as always, list. -jc _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From duersch at us.ibm.com Wed Mar 8 13:32:37 2017 From: duersch at us.ibm.com (Steve Duersch) Date: Wed, 8 Mar 2017 08:32:37 -0500 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? In-Reply-To: References: Message-ID: You are correct. There is some level of tracing on the v7000U. Also, if the gpfs daemon hits an assert or signal and tracing is running then traces are recycled automatically. There is no need to have another script monitoring for such events. You probably want to have mmtracectl --trace-recycle=global so that an assert on one node grabs traces on all nodes (ie fs manager). As for performance, this one is more difficult and I don't have details. I can ping a colleague to see if he has numbers. Obviously there will be a performance penalty for running tracing. Steve Duersch Spectrum Scale IBM Poughkeepsie, New York > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 7 Mar 2017 21:11:33 +0000 > From: "Oesterlin, Robert" > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Potential problems - leaving trace > enabled in over-write mode? > Message-ID: > Content-Type: text/plain; charset="utf-8" > > I?ve been told that V70o0 unified nodes (GPFS under the covers) run > with tracing enabled all the time.. but I agree with you Brian on > the potential impacts. But when you must catch a trace for a problem > that occurs once every few weeks ? how else would I do it? > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > 507-269-0413 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oehmes at gmail.com Wed Mar 8 13:37:15 2017 From: oehmes at gmail.com (Sven Oehme) Date: Wed, 08 Mar 2017 13:37:15 +0000 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? In-Reply-To: References: Message-ID: given that i love data points, let me put some behind this thread :-) starting in version 3.4 we enhanced the trace code of scale significant. this went on release to release all the way up to 4.2.1. since 4.2.1 we made further improvements, but much smaller changes, more optimization , e.g. reducing of trace levels verbosity, etc . with 4.2.2 we switched from blocking traces to in memory traces as the default trace infrastructure, this infrastructure was designed to be turned on all the time with minimal impact on performance. i just took a 4.2.3 build and did load it on one of my faster NVMe nodes and did a 100% random read test with 64 threads against very fast storage. while this was running i started trace with mmtracectl --start ; sleep 10 ; mmtracectl --off , attached is a screenshot of this run : [image: pasted1] the trace started at 14:19:15 and stopped at 14:20:00 (that includes 10 command processing, format of traces, etc) as one can see on the chart the dip in performance is very small, measured looking at raw data its 8%. the system runs at ~920k 4k random read iops /sec and the workload running on the node pushes the system almost to 100% cpu time even without trace running, reason i tested under this condition is because also on a HPC compute oriented workload you will run without spare cpu cycles, so the 8% is under extreme conditions. said all this, the workload i am running is not causing a lot of metadata i/o as i am running within a small number of files, its all read, so no write, therefore this isn't representative to a lot of real world workloads, but covers one very extreme case , high iops requirements under heavy cpu contention. therefore i repeated the test under a heavy metadata workload , SpecSFS SWBUILD. when i run the test with and without traces turned on i see a ~15% loss on max operations/sec/node. again this is another extreme case as there are only few workloads which have a metadata workload component as heavy as SWBUILD. but given that between the 2 extreme cases we are talking about 8 - 15% overhead its is very likely that all real world workloads suffer less than 15% when traces are turned on. only a test in your environment will really show . i would not recommend to try this with any version prior 4.2.1 as the impact will be significant higher than what i measured. hope this helps make a informed decision. Sven On Tue, Mar 7, 2017 at 9:32 PM Oesterlin, Robert < Robert.Oesterlin at nuance.com> wrote: > I?m considering enabling trace on all nodes all the time, doing something > like this: > > > > mmtracectl --set --trace=def --trace-recycle=global > --tracedev-write-mode=overwrite --tracedev-overwrite-buffer-size=256M > mmtracectl --start > > > > My questions are: > > > > - What is the performance penalty of leaving this on 100% of the time on a > node? > > - Does anyone have any suggestions on automation on stopping trace when a > particular event occurs? > > - What other issues, if any? > > > > > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > 507-269-0413 <(507)%20269-0413> > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pasted1 Type: image/png Size: 216901 bytes Desc: not available URL: From Robert.Oesterlin at nuance.com Wed Mar 8 13:56:32 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Wed, 8 Mar 2017 13:56:32 +0000 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? Message-ID: <90CE7D81-D1F0-4E29-B9F2-02D467911E6C@nuance.com> As always, Sven comes in to back this up with real data :) To net this out, Sven ? I should be able enable trace on my NSD servers running 4.2.2 without much impact, correct? Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Sven Oehme Reply-To: gpfsug main discussion list Date: Wednesday, March 8, 2017 at 7:37 AM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? starting in version 3.4 we enhanced the trace code of scale significant. this went on release to release all the way up to 4.2.1. since 4.2.1 we made further improvements, but much smaller changes, more optimization , e.g. reducing of trace levels verbosity, etc . with 4.2.2 we switched from blocking traces to in memory traces as the default trace infrastructure, this infrastructure was designed to be turned on all the time with minimal impact on performance. -------------- next part -------------- An HTML attachment was scrubbed... URL: From oehmes at gmail.com Wed Mar 8 14:15:55 2017 From: oehmes at gmail.com (Sven Oehme) Date: Wed, 08 Mar 2017 14:15:55 +0000 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? In-Reply-To: <90CE7D81-D1F0-4E29-B9F2-02D467911E6C@nuance.com> References: <90CE7D81-D1F0-4E29-B9F2-02D467911E6C@nuance.com> Message-ID: yes , but i would do this in stages given how large your system is. pick one set of nodes (lets say) 100 out of 200 that do similar things and turn tracing on there. this will give you data you can compare between the 2 set of nodes. let it run for a week and if the data with your real workload doesn't show any significant degradation (which is what i expect) turn it on everywhere. the one thing i am not 100% sure about is size of trace buffer as well as the global cut config. what this means is if you apply the settings as mentioned in this first post, if one node asserts in your cluster you will cut a trace on all nodes that will write a 256M buffer into your dump file location. if you have a node thats in an assert loop (asserts, restarts , asserts) this can cause significant load on all nodes. therefore i would probably start without cutting a global trace and reduce the trace size to 64M. i (and i am sure other dev folks) would be very interested in the outcome as we have this debate on a yearly basis if we shouldn't just turn tracing on by default, in the past performance was the biggest hurdle, this is solved now (my claim) . next big questions is how well does that work on larger scale systems with production workloads. as more feedback we will get in this area as better we can make informed decision how and if it could be turned on all the time and work harder on handling cases like i mentioned above to mitigate the risks . sven On Wed, Mar 8, 2017 at 2:56 PM Oesterlin, Robert < Robert.Oesterlin at nuance.com> wrote: > As always, Sven comes in to back this up with real data :) > > > > To net this out, Sven ? I should be able enable trace on my NSD servers > running 4.2.2 without much impact, correct? > > > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > > > > > > *From: * on behalf of Sven > Oehme > *Reply-To: *gpfsug main discussion list > *Date: *Wednesday, March 8, 2017 at 7:37 AM > > > *To: *gpfsug main discussion list > > *Subject: *[EXTERNAL] Re: [gpfsug-discuss] Potential problems - leaving > trace enabled in over-write mode? > > > > starting in version 3.4 we enhanced the trace code of scale significant. > this went on release to release all the way up to 4.2.1. since 4.2.1 we > made further improvements, but much smaller changes, more optimization , > e.g. reducing of trace levels verbosity, etc . > > with 4.2.2 we switched from blocking traces to in memory traces as the > default trace infrastructure, this infrastructure was designed to be turned > on all the time with minimal impact on performance. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Wed Mar 8 15:25:16 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Wed, 8 Mar 2017 15:25:16 +0000 Subject: [gpfsug-discuss] Registration Reminder: GPFSUG-USA Meeting at NERSC April 4-5th Message-ID: Here?s a reminder to register for the upcoming user group meeting! Registration deadline is March 24th! We Still have slots (20 mins each) for customer and partner talks ? please contact me if you want to do a short presentation. Register here: https://www.nersc.gov/research-and-development/data-analytics/spectrum-user-group-meeting/ Bob Oesterlin Sr Principal Storage Engineer, Nuance -------------- next part -------------- An HTML attachment was scrubbed... URL: From jenocram at gmail.com Wed Mar 8 20:15:44 2017 From: jenocram at gmail.com (Jeno Cram) Date: Wed, 8 Mar 2017 15:15:44 -0500 Subject: [gpfsug-discuss] CES, NFSv4 and Kerberos with AD authentication Message-ID: I'm running into an issue and I can't find the answer anywhere in the Spectrum Scale documentation. I'm trying to keberize nfsv4 with AD authentication for file access. I have the instructions for the server (--enable-nfs-kerberos) and also ran "mmnfs configuration change From jenocram at gmail.com Wed Mar 8 20:18:07 2017 From: jenocram at gmail.com (Jeno Cram) Date: Wed, 8 Mar 2017 15:18:07 -0500 Subject: [gpfsug-discuss] CES, NFSv4 and Kerberos with AD authentication In-Reply-To: References: Message-ID: My apologies.. I hit send too soon. I'm running into an issue and I can't find the answer anywhere in the Spectrum Scale documentation. I'm trying to keberize nfsv4 with AD authentication for file access. I have the instructions for the server (--enable-nfs-kerberos) and also ran "mmnfs configuration change idmapd_domain=example.com". Is there anything else that needs to be done on the server side and what needs to be configured on the client to allow this? On Wed, Mar 8, 2017 at 3:15 PM, Jeno Cram wrote: > I'm running into an issue and I can't find the answer anywhere in the > Spectrum Scale documentation. > > I'm trying to keberize nfsv4 with AD authentication for file access. > I have the instructions for the server (--enable-nfs-kerberos) and > also ran "mmnfs configuration change From Robert.Oesterlin at nuance.com Wed Mar 8 21:54:33 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Wed, 8 Mar 2017 21:54:33 +0000 Subject: [gpfsug-discuss] Problem installin CES - 4.2.2-2 Message-ID: <19C1E64C-76E3-4C59-914E-199EA5D37CCF@nuance.com> OK, I?ll admit I?m not protocols install expert. Using the ?spectrumscale? installer command, it?s failing to install the object packages due to some internal dependencies from the IBM supplied repo. Who can help me? Excerpt from the install log: 2017-03-08 16:47:34,319 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com * log[IBM SPECTRUM SCALE: Installing Object packages (SS50).] action write 2017-03-08 16:47:34,319 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,554 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com * yum_package[spectrum-scale-object] action install 2017-03-08 16:47:59,555 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,555 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com ================================================================================ 2017-03-08 16:47:59,556 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error executing action `install` on resource 'yum_package[spectrum-scale-object]' 2017-03-08 16:47:59,556 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com ================================================================================ 2017-03-08 16:47:59,557 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,557 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Chef::Exceptions::Exec 2017-03-08 16:47:59,558 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com ---------------------- 2017-03-08 16:47:59,558 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com yum -d0 -e0 -y install spectrum-scale-object-4.2.2-2 returned 1: 2017-03-08 16:47:59,558 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com STDOUT: Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,560 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,560 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,560 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,563 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,563 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,563 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com You could try using --skip-broken to work around the problem 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com You could try running: rpm -Va --nofiles --nodigest 2017-03-08 16:47:59,565 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,565 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com STDERR: Error: Package: 1:python-novaclient-2.30.3-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,565 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error: Package: 1:python-glanceclient-1.1.2-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error: Package: python-openstackclient-1.7.3-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,569 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,569 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,569 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error: Package: python-neutronclient-3.1.2-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,571 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,571 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,571 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 Bob Oesterlin Sr Principal Storage Engineer, Nuance -------------- next part -------------- An HTML attachment was scrubbed... URL: From stockf at us.ibm.com Wed Mar 8 22:06:18 2017 From: stockf at us.ibm.com (Frederick Stock) Date: Wed, 8 Mar 2017 17:06:18 -0500 Subject: [gpfsug-discuss] Problem installin CES - 4.2.2-2 In-Reply-To: <19C1E64C-76E3-4C59-914E-199EA5D37CCF@nuance.com> References: <19C1E64C-76E3-4C59-914E-199EA5D37CCF@nuance.com> Message-ID: What version of python do you have installed? I think you need at least version 2.7. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 03/08/2017 04:55 PM Subject: [gpfsug-discuss] Problem installin CES - 4.2.2-2 Sent by: gpfsug-discuss-bounces at spectrumscale.org OK, I?ll admit I?m not protocols install expert. Using the ?spectrumscale? installer command, it?s failing to install the object packages due to some internal dependencies from the IBM supplied repo. Who can help me? Excerpt from the install log: 2017-03-08 16:47:34,319 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com * log[IBM SPECTRUM SCALE: Installing Object packages (SS50).] action write 2017-03-08 16:47:34,319 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,554 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com * yum_package[spectrum-scale-object] action install 2017-03-08 16:47:59,555 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,555 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com ================================================================================ 2017-03-08 16:47:59,556 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error executing action `install` on resource 'yum_package[spectrum-scale-object]' 2017-03-08 16:47:59,556 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com ================================================================================ 2017-03-08 16:47:59,557 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,557 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Chef::Exceptions::Exec 2017-03-08 16:47:59,558 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com ---------------------- 2017-03-08 16:47:59,558 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com yum -d0 -e0 -y install spectrum-scale-object-4.2.2-2 returned 1: 2017-03-08 16:47:59,558 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com STDOUT: Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,560 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,560 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,560 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,563 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,563 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,563 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com You could try using --skip-broken to work around the problem 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com You could try running: rpm -Va --nofiles --nodigest 2017-03-08 16:47:59,565 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,565 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com STDERR: Error: Package: 1:python-novaclient-2.30.3-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,565 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error: Package: 1:python-glanceclient-1.1.2-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error: Package: python-openstackclient-1.7.3-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,569 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,569 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,569 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error: Package: python-neutronclient-3.1.2-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,571 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,571 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,571 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 Bob Oesterlin Sr Principal Storage Engineer, Nuance _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mark.Bush at siriuscom.com Wed Mar 8 23:25:07 2017 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Wed, 8 Mar 2017 23:25:07 +0000 Subject: [gpfsug-discuss] Problem installin CES - 4.2.2-2 Message-ID: <4FA8EF66-4E49-4C65-AA6E-5166AF8F45C7@siriuscom.com> What version of RHEL/CENTOS? From: "Oesterlin, Robert" Reply-To: gpfsug main discussion list Date: Wednesday, March 8, 2017 at 3:54 PM To: gpfsug main discussion list Subject: [gpfsug-discuss] Problem installin CES - 4.2.2-2 OK, I?ll admit I?m not protocols install expert. Using the ?spectrumscale? installer command, it?s failing to install the object packages due to some internal dependencies from the IBM supplied repo. Who can help me? Excerpt from the install log: 2017-03-08 16:47:34,319 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com * log[IBM SPECTRUM SCALE: Installing Object packages (SS50).] action write 2017-03-08 16:47:34,319 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,554 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com * yum_package[spectrum-scale-object] action install 2017-03-08 16:47:59,555 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,555 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com ================================================================================ 2017-03-08 16:47:59,556 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error executing action `install` on resource 'yum_package[spectrum-scale-object]' 2017-03-08 16:47:59,556 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com ================================================================================ 2017-03-08 16:47:59,557 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,557 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Chef::Exceptions::Exec 2017-03-08 16:47:59,558 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com ---------------------- 2017-03-08 16:47:59,558 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com yum -d0 -e0 -y install spectrum-scale-object-4.2.2-2 returned 1: 2017-03-08 16:47:59,558 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com STDOUT: Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,560 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,560 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,560 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,563 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,563 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,563 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com You could try using --skip-broken to work around the problem 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com You could try running: rpm -Va --nofiles --nodigest 2017-03-08 16:47:59,565 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,565 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com STDERR: Error: Package: 1:python-novaclient-2.30.3-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,565 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error: Package: 1:python-glanceclient-1.1.2-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error: Package: python-openstackclient-1.7.3-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,569 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,569 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,569 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error: Package: python-neutronclient-3.1.2-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,571 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,571 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,571 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 Bob Oesterlin Sr Principal Storage Engineer, Nuance This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Thu Mar 9 00:25:41 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Thu, 9 Mar 2017 00:25:41 +0000 Subject: [gpfsug-discuss] Problem installin CES - 4.2.2-2 In-Reply-To: <4FA8EF66-4E49-4C65-AA6E-5166AF8F45C7@siriuscom.com> References: <4FA8EF66-4E49-4C65-AA6E-5166AF8F45C7@siriuscom.com> Message-ID: 7.2 No other conflicting ?openstack|keystone|swift|glance|nova|neutron? package installed that I can see. Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 From: on behalf of "Mark.Bush at siriuscom.com" Reply-To: gpfsug main discussion list Date: Wednesday, March 8, 2017 at 5:25 PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Problem installin CES - 4.2.2-2 What version of RHEL/CENTOS? -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mark.Bush at siriuscom.com Thu Mar 9 14:12:40 2017 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Thu, 9 Mar 2017 14:12:40 +0000 Subject: [gpfsug-discuss] Problem installin CES - 4.2.2-2 In-Reply-To: References: <4FA8EF66-4E49-4C65-AA6E-5166AF8F45C7@siriuscom.com> Message-ID: I don?t normally use the chef based installer but I have had issues getting OBJ installed on 7.x before and it was all based on the fact that the packages were moved to subdirs /usr/local/mmfs/4.2.2.0/object_rpms/rhel7 where before they were just in the root of object_rpms. Not sure that applies here since I would think the installer was updated to look there instead of the root. It sure seems from your log output that it?s not happy trying to install packages. Worse case you can install manually. Here?s how I do that in my automated script cat </etc/yum.repos.d/IBM-SpecScaleOBJ.repo [spectrum_scale] name=spectrum_scale baseurl=file:///vagrant/4.2.2.0/object_rpms/rhel7/ enabled=1 EOL yum update -y 2>&1 yum install -y /vagrant/4.2.2.0/zimon_rpms/rhel7/gpfs.gss.pm{sensors,collector}*.rpm > /dev/null 2>&1 yum install -y /vagrant/4.2.2.0/gpfs_rpms/gpfs.{base,ext,gskit,gpl,msg,docs,adv,crypto,java,gui}*.rpm > /dev/null 2>&1 yum install -y /vagrant/4.2.2.0/smb_rpms/rhel7/gpfs.*.rpm > /dev/null 2>&1 yum install -y /vagrant/4.2.2.0/ganesha_rpms/rhel7/nfs-*.rpm > /dev/null 2>&1 yum install -y /vagrant/4.2.2.0/object_rpms/rhel7/*.rpm >/dev/null 2>&1 yum install -y spectrum-scale-object 2>&1 From: "Oesterlin, Robert" Reply-To: gpfsug main discussion list Date: Wednesday, March 8, 2017 at 6:25 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Problem installin CES - 4.2.2-2 7.2 No other conflicting ?openstack|keystone|swift|glance|nova|neutron? package installed that I can see. Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 From: on behalf of "Mark.Bush at siriuscom.com" Reply-To: gpfsug main discussion list Date: Wednesday, March 8, 2017 at 5:25 PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Problem installin CES - 4.2.2-2 What version of RHEL/CENTOS? This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Thu Mar 9 14:53:33 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Thu, 9 Mar 2017 14:53:33 +0000 Subject: [gpfsug-discuss] Problem installin CES - 4.2.2-2 Message-ID: Bingo! That worked flawlessly. You are a lifesaver, thanks! Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of "Mark.Bush at siriuscom.com" Reply-To: gpfsug main discussion list Date: Thursday, March 9, 2017 at 8:12 AM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Problem installin CES - 4.2.2-2 I don?t normally use the chef based installer but I have had issues getting OBJ installed on 7.x before and it was all based on the fact that the packages were moved to subdirs /usr/local/mmfs/4.2.2.0/object_rpms/rhel7 where before they were just in the root of object_rpms. Not sure that applies here since I would think the installer was updated to look there instead of the root. It sure seems from your log output that it?s not happy trying to install packages. Worse case you can install manually. Here?s how I do that in my automated script cat </etc/yum.repos.d/IBM-SpecScaleOBJ.repo [spectrum_scale] name=spectrum_scale baseurl=file:///vagrant/4.2.2.0/object_rpms/rhel7/ enabled=1 EOL yum update -y 2>&1 yum install -y /vagrant/4.2.2.0/zimon_rpms/rhel7/gpfs.gss.pm{sensors,collector}*.rpm > /dev/null 2>&1 yum install -y /vagrant/4.2.2.0/gpfs_rpms/gpfs.{base,ext,gskit,gpl,msg,docs,adv,crypto,java,gui}*.rpm > /dev/null 2>&1 yum install -y /vagrant/4.2.2.0/smb_rpms/rhel7/gpfs.*.rpm > /dev/null 2>&1 yum install -y /vagrant/4.2.2.0/ganesha_rpms/rhel7/nfs-*.rpm > /dev/null 2>&1 yum install -y /vagrant/4.2.2.0/object_rpms/rhel7/*.rpm >/dev/null 2>&1 yum install -y spectrum-scale-object 2>&1 From: "Oesterlin, Robert" Reply-To: gpfsug main discussion list Date: Wednesday, March 8, 2017 at 6:25 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Problem installin CES - 4.2.2-2 7.2 No other conflicting ?openstack|keystone|swift|glance|nova|neutron? package installed that I can see. Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 From: on behalf of "Mark.Bush at siriuscom.com" Reply-To: gpfsug main discussion list Date: Wednesday, March 8, 2017 at 5:25 PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Problem installin CES - 4.2.2-2 What version of RHEL/CENTOS? This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Thu Mar 9 16:11:56 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Thu, 9 Mar 2017 16:11:56 +0000 Subject: [gpfsug-discuss] Running gpfs.snap outside of problems Message-ID: Hi all, Is there any best practice as to when to run a gpfs.snap, and the impact it will have on a healthy cluster? Some of my colleagues are keen to gather a snap for example every 2 hours to assist in problem determination. I can elaborate on the "why" part of this if needed but I'm not fully convinced it would help. Said I'd look into it though so I'm asking the question. Cheers Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Thu Mar 9 16:29:52 2017 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Thu, 09 Mar 2017 16:29:52 +0000 Subject: [gpfsug-discuss] Running gpfs.snap outside of problems In-Reply-To: References: Message-ID: There's a manual for it now.. and it points out "The tool impacts performance.." Also it has caused mmfsd crashes for me earlier, so I've learned to be weary of running it.. The manual also says it's collecting using mmfsadm, and the mmfsadm manual warns that it might cause GPFS to fail ("in certain rare cases"). I wouldn't dare running it every few hours. -jf tor. 9. mar. 2017 kl. 17.12 skrev Sobey, Richard A : > Hi all, > > > > Is there any best practice as to when to run a gpfs.snap, and the impact > it will have on a healthy cluster? Some of my colleagues are keen to gather > a snap for example every 2 hours to assist in problem determination. > > > > I can elaborate on the ?why? part of this if needed but I?m not fully > convinced it would help. Said I?d look into it though so I?m asking the > question. > > > > Cheers > > Richard > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mxcheng at uchicago.edu Thu Mar 9 19:23:38 2017 From: mxcheng at uchicago.edu (Mengxing Cheng) Date: Thu, 9 Mar 2017 19:23:38 +0000 Subject: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem Message-ID: Dear all, My name is Mengxing Cheng and I am a HPC system administrator at the University of Chicago. We have a GSS26 running gss2.5.10.3-3b and gpfs-4.2.0.3. Recently, we run mmfsck on a relatively small filesystem with 14TB block and 73863102 inodes but it was unusually slow so as to not be able to finish in 48 hours. In contrast, mmfsck run on a filesystem with the same size and inodes but sitting on a traditional IBM DS3512 storage took only 2 hours to complete. In particular, the mmfsck run in parallel using 3 nodes within the GSS storage cluster, we notice that one gss storage server scans inodes much slower than the other gss storage server and the quorum service node. Has anyone experience the same mmfsck performance issue? Could anyone make recommendation to troubleshoot and improve mmfsck performance? Thank you! Mengxing --- Mengxing Cheng, Ph.D. HPC System Administrator Research Computing Center The University of Chicago 5607 S. Drexel Ave. Chicago, IL 60637 email: mxcheng at uchicago.edu phone: (773) 702-4070 -------------- next part -------------- An HTML attachment was scrubbed... URL: From kevindjo at us.ibm.com Thu Mar 9 19:47:56 2017 From: kevindjo at us.ibm.com (Kevin D Johnson) Date: Thu, 9 Mar 2017 19:47:56 +0000 Subject: [gpfsug-discuss] Running gpfs.snap outside of problems In-Reply-To: References: , Message-ID: An HTML attachment was scrubbed... URL: From esperle at us.ibm.com Thu Mar 9 21:13:08 2017 From: esperle at us.ibm.com (Eric Sperley) Date: Thu, 9 Mar 2017 13:13:08 -0800 Subject: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem In-Reply-To: References: Message-ID: Mengxing, It is nice meeting you. I have seen a situation where the amount of RAM on a node can affect mmfsck times. Do all the nodes have the same amount of RAM, or does the slow running node have less RAM? Best Regards, Eric Eric Sperley SDI Architect "Carpe Diem" IBM Systems esperle at us.ibm.com +15033088721 From: Mengxing Cheng To: "gpfsug-discuss at spectrumscale.org" Date: 03/09/2017 11:24 AM Subject: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem Sent by: gpfsug-discuss-bounces at spectrumscale.org Dear all, My name is Mengxing Cheng and I am a HPC system administrator at the University of Chicago. We have a GSS26 running gss2.5.10.3-3b and gpfs-4.2.0.3. Recently, we run mmfsck on a relatively small filesystem with 14TB block and 73863102 inodes but it was unusually slow so as to not be able to finish in 48 hours. In contrast, mmfsck run on a filesystem with the same size and inodes but sitting on a traditional IBM DS3512 storage took only 2 hours to complete. In particular, the mmfsck run in parallel using 3 nodes within the GSS storage cluster, we notice that one gss storage server scans inodes much slower than the other gss storage server and the quorum service node. Has anyone experience the same mmfsck performance issue? Could anyone make recommendation to troubleshoot and improve mmfsck performance? Thank you! Mengxing --- Mengxing Cheng, Ph.D. HPC System Administrator Research Computing Center The University of Chicago 5607 S. Drexel Ave. Chicago, IL 60637 email: mxcheng at uchicago.edu phone: (773) 702-4070 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 1C803747.gif Type: image/gif Size: 481 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecblank.gif Type: image/gif Size: 45 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 1C166252.gif Type: image/gif Size: 2322 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From mxcheng at uchicago.edu Thu Mar 9 21:24:21 2017 From: mxcheng at uchicago.edu (Mengxing Cheng) Date: Thu, 9 Mar 2017 21:24:21 +0000 Subject: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem In-Reply-To: References: , Message-ID: Eric, thank you very much for replying. Here is the memory configuration and current usage. Note that mmfsck is not running now. The two gss servers have the same 256GB memory and the service node has 128GB. 1. service node: total used free shared buffers cached Mem: 125 58 66 0 0 4 -/+ buffers/cache: 53 71 Swap: 7 0 7 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12990 root 0 -20 71.0g 43g 885m S 7.6 34.4 9306:00 mmfsd 2. gss nodes: ==================================== total used free shared buff/cache available Mem: 251 210 37 0 4 36 Swap: 3 0 3 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 36770 root 0 -20 0.216t 0.192t 2.667g S 48.9 78.1 75684:09 /usr/lpp/mmfs/bin/mmfsd The gss nodes' memory usage is so high because their pagepool is set to 192GB while the service node has 16GB pagepool. Mengxing --- Mengxing Cheng, Ph.D. HPC System Administrator Research Computing Center The University of Chicago 5607 S. Drexel Ave. Chicago, IL 60637 email: mxcheng at uchicago.edu phone: (773) 702-4070 ________________________________ From: Eric Sperley [esperle at us.ibm.com] Sent: Thursday, March 09, 2017 3:13 PM To: Mengxing Cheng Cc: gpfsug main discussion list Subject: Re: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem Mengxing, It is nice meeting you. I have seen a situation where the amount of RAM on a node can affect mmfsck times. Do all the nodes have the same amount of RAM, or does the slow running node have less RAM? Best Regards, Eric [cid:3__=8FBB0A4DDFE7DC3F8f9e8a93df938690918c8FB@] Eric Sperley SDI Architect "Carpe Diem" IBM Systems esperle at us.ibm.com +15033088721 [Inactive hide details for Mengxing Cheng ---03/09/2017 11:24:02 AM---Dear all, My name is Mengxing Cheng and I am a HPC system]Mengxing Cheng ---03/09/2017 11:24:02 AM---Dear all, My name is Mengxing Cheng and I am a HPC system administrator at the University of Chicago From: Mengxing Cheng To: "gpfsug-discuss at spectrumscale.org" Date: 03/09/2017 11:24 AM Subject: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Dear all, My name is Mengxing Cheng and I am a HPC system administrator at the University of Chicago. We have a GSS26 running gss2.5.10.3-3b and gpfs-4.2.0.3. Recently, we run mmfsck on a relatively small filesystem with 14TB block and 73863102 inodes but it was unusually slow so as to not be able to finish in 48 hours. In contrast, mmfsck run on a filesystem with the same size and inodes but sitting on a traditional IBM DS3512 storage took only 2 hours to complete. In particular, the mmfsck run in parallel using 3 nodes within the GSS storage cluster, we notice that one gss storage server scans inodes much slower than the other gss storage server and the quorum service node. Has anyone experience the same mmfsck performance issue? Could anyone make recommendation to troubleshoot and improve mmfsck performance? Thank you! Mengxing --- Mengxing Cheng, Ph.D. HPC System Administrator Research Computing Center The University of Chicago 5607 S. Drexel Ave. Chicago, IL 60637 email: mxcheng at uchicago.edu phone: (773) 702-4070 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 1C803747.gif Type: image/gif Size: 481 bytes Desc: 1C803747.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecblank.gif Type: image/gif Size: 45 bytes Desc: ecblank.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 1C166252.gif Type: image/gif Size: 2322 bytes Desc: 1C166252.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: graycol.gif URL: From r.sobey at imperial.ac.uk Fri Mar 10 09:07:08 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Fri, 10 Mar 2017 09:07:08 +0000 Subject: [gpfsug-discuss] Running gpfs.snap outside of problems In-Reply-To: References: , Message-ID: Thanks Kevin and J-F, I appreciate your thoughts. I?ll go with ?no? then as my answer ? Cheers Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Kevin D Johnson Sent: 09 March 2017 19:48 To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Running gpfs.snap outside of problems I've never heard of running a gpfs.snap every few hours, but it is technically possible. Probably not advisable except in the most limited of circumstances. I wouldn't do it unless Support or another technical resource you trust says to do so. You can limit the snap to run on a few nodes versus a whole cluster and that way limit the effect it has overall. You can also specify a directory outside GPFS to write the snap file. Kevin D. Johnson, MBA, MAFM Spectrum Computing, Senior Managing Consultant IBM Certified Deployment Professional - Spectrum Scale V4.1.1 IBM Certified Deployment Professional - Cloud Object Storage V3.8 IBM Certified Solution Advisor - Spectrum Computing V1 720.349.6199 - kevindjo at us.ibm.com ----- Original message ----- From: Jan-Frode Myklebust Sent by: gpfsug-discuss-bounces at spectrumscale.org To: "gpfsug-discuss at spectrumscale.org" Cc: Subject: Re: [gpfsug-discuss] Running gpfs.snap outside of problems Date: Thu, Mar 9, 2017 11:30 AM There's a manual for it now.. and it points out "The tool impacts performance.." Also it has caused mmfsd crashes for me earlier, so I've learned to be weary of running it.. The manual also says it's collecting using mmfsadm, and the mmfsadm manual warns that it might cause GPFS to fail ("in certain rare cases"). I wouldn't dare running it every few hours. -jf tor. 9. mar. 2017 kl. 17.12 skrev Sobey, Richard A >: Hi all, Is there any best practice as to when to run a gpfs.snap, and the impact it will have on a healthy cluster? Some of my colleagues are keen to gather a snap for example every 2 hours to assist in problem determination. I can elaborate on the ?why? part of this if needed but I?m not fully convinced it would help. Said I?d look into it though so I?m asking the question. Cheers Richard _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From chair at spectrumscale.org Fri Mar 10 10:22:24 2017 From: chair at spectrumscale.org (GPFS UG Chair (Simon Thompson)) Date: Fri, 10 Mar 2017 10:22:24 +0000 Subject: [gpfsug-discuss] UK Spectrum Scale User Group Sponsorship packages Message-ID: Hi All, As we are developing the programme for the UK Spectrum Scale UG on 9th/10th May, we are again looking for commercial sponsors to support the evening event we hope to run. I have already contacted a number of contacts who kindly supported last year's event, however if you are interested in being a sponsor for the event, please get in touch with me directly and I can provide you with more information. Simon From Achim.Rehor at de.ibm.com Fri Mar 10 16:56:30 2017 From: Achim.Rehor at de.ibm.com (Achim Rehor) Date: Fri, 10 Mar 2017 17:56:30 +0100 Subject: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem In-Reply-To: References: , Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 7182 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 481 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 45 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 2322 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 105 bytes Desc: not available URL: From mxcheng at uchicago.edu Fri Mar 10 16:59:43 2017 From: mxcheng at uchicago.edu (Mengxing Cheng) Date: Fri, 10 Mar 2017 16:59:43 +0000 Subject: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem In-Reply-To: References: , , Message-ID: Achim, thank you very much! Mengxing --- Mengxing Cheng, Ph.D. HPC System Administrator Research Computing Center The University of Chicago 5607 S. Drexel Ave. Chicago, IL 60637 email: mxcheng at uchicago.edu phone: (773) 702-4070 ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Achim Rehor [Achim.Rehor at de.ibm.com] Sent: Friday, March 10, 2017 10:56 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem Hi, mmfsck is highly dependent on pagepool, so my best guess would be, the the EMS node with only 16GB of pagepool is sort of a bottleneck for the slow progress. You might want to try to temporarily raise the pagepool on the service node to .. lets say 64GB, or restrict the mmfsck to run only on the storage nodes. note: the fs mgr node will always be part of the mmfsck nodes. so if it is located on the service node, mmchmgr to a storage node first Mit freundlichen Gr??en / Kind regards Achim Rehor ________________________________ Software Technical Support Specialist AIX/ Emea HPC Support [cid:_1_D975D64CD975D0CC005D0FE8C12580DF] IBM Certified Advanced Technical Expert - Power Systems with AIX TSCC Software Service, Dept. 7922 Global Technology Services ________________________________ Phone: +49-7034-274-7862 IBM Deutschland E-Mail: Achim.Rehor at de.ibm.com Am Weiher 24 65451 Kelsterbach Germany ________________________________ IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter Gesch?ftsf?hrung: Martina Koederitz (Vorsitzende), Reinhard Reschke, Dieter Scholz, Gregor Pillen, Ivo Koerner, Christian Noll Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 WEEE-Reg.-Nr. DE 99369940 From: Mengxing Cheng To: Eric Sperley Cc: gpfsug main discussion list Date: 03/09/2017 10:24 PM Subject: Re: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Eric, thank you very much for replying. Here is the memory configuration and current usage. Note that mmfsck is not running now. The two gss servers have the same 256GB memory and the service node has 128GB. 1. service node: total used free shared buffers cached Mem: 125 58 66 0 0 4 -/+ buffers/cache: 53 71 Swap: 7 0 7 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12990 root 0 -20 71.0g 43g 885m S 7.6 34.4 9306:00 mmfsd 2. gss nodes: ==================================== total used free shared buff/cache available Mem: 251 210 37 0 4 36 Swap: 3 0 3 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 36770 root 0 -20 0.216t 0.192t 2.667g S 48.9 78.1 75684:09 /usr/lpp/mmfs/bin/mmfsd The gss nodes' memory usage is so high because their pagepool is set to 192GB while the service node has 16GB pagepool. Mengxing --- Mengxing Cheng, Ph.D. HPC System Administrator Research Computing Center The University of Chicago 5607 S. Drexel Ave. Chicago, IL 60637 email: mxcheng at uchicago.edu phone: (773) 702-4070 ________________________________ From: Eric Sperley [esperle at us.ibm.com] Sent: Thursday, March 09, 2017 3:13 PM To: Mengxing Cheng Cc: gpfsug main discussion list Subject: Re: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem Mengxing, It is nice meeting you. I have seen a situation where the amount of RAM on a node can affect mmfsck times. Do all the nodes have the same amount of RAM, or does the slow running node have less RAM? Best Regards, Eric [cid:_4_0B8EBB580B8EB73C005D0FE8C12580DF] Eric Sperley SDI Architect "Carpe Diem" IBM Systems esperle at us.ibm.com +15033088721 [Inactive hide details for Mengxing Cheng ---03/09/2017 11:24:02 AM---Dear all, My name is Mengxing Cheng and I am a HPC system]Mengxing Cheng ---03/09/2017 11:24:02 AM---Dear all, My name is Mengxing Cheng and I am a HPC system administrator at the University of Chicago From: Mengxing Cheng To: "gpfsug-discuss at spectrumscale.org" Date: 03/09/2017 11:24 AM Subject: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Dear all, My name is Mengxing Cheng and I am a HPC system administrator at the University of Chicago. We have a GSS26 running gss2.5.10.3-3b and gpfs-4.2.0.3. Recently, we run mmfsck on a relatively small filesystem with 14TB block and 73863102 inodes but it was unusually slow so as to not be able to finish in 48 hours. In contrast, mmfsck run on a filesystem with the same size and inodes but sitting on a traditional IBM DS3512 storage took only 2 hours to complete. In particular, the mmfsck run in parallel using 3 nodes within the GSS storage cluster, we notice that one gss storage server scans inodes much slower than the other gss storage server and the quorum service node. Has anyone experience the same mmfsck performance issue? Could anyone make recommendation to troubleshoot and improve mmfsck performance? Thank you! Mengxing --- Mengxing Cheng, Ph.D. HPC System Administrator Research Computing Center The University of Chicago 5607 S. Drexel Ave. Chicago, IL 60637 email: mxcheng at uchicago.edu phone: (773) 702-4070 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00007.gif Type: image/gif Size: 7182 bytes Desc: ATT00007.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00008.png Type: image/png Size: 481 bytes Desc: ATT00008.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00009.gif Type: image/gif Size: 45 bytes Desc: ATT00009.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00010.png Type: image/png Size: 2322 bytes Desc: ATT00010.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00011.gif Type: image/gif Size: 105 bytes Desc: ATT00011.gif URL: From Kevin.Buterbaugh at Vanderbilt.Edu Fri Mar 10 20:43:37 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 10 Mar 2017 20:43:37 +0000 Subject: [gpfsug-discuss] mmcrfs issue Message-ID: <43922F46-548D-4D34-89D4-74D35C6EB447@vanderbilt.edu> Hi All, We are testing out some flash storage. I created a couple of NSDs successfully (?): root at nec:~/gpfs# mmlsnsd -F File system Disk name NSD servers --------------------------------------------------------------------------- (free disk) nsd1 nec (free disk) nsd2 nec root at nec:~/gpfs# So I tried to create a filesystem: root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 GPFS: 6027-441 Unable to open disk 'nsd2' on node nec. GPFS: 6027-441 Unable to open disk 'nsd1' on node nec. No such device No such device GPFS: 6027-538 Error accessing disks. mmcrfs: 6027-1200 tscrfs failed. Cannot create gpfs0 mmcrfs: 6027-1639 Command failed. Examine previous error messages to determine cause. root at nec:~/gpfs# Does this output from readdescraw look normal? root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd16 NSD descriptor in sector 64 of /dev/zd16 NSDid: 0A0023D258C1C02C format version: 1403 Label: Paxos sector: -1931478434 number of sectors: 8192 isPdisk: 0 Comment: NSD descriptor for Thu Mar 9 14:50:52 2017 No Disk descriptor in sector 96 of /dev/zd16 No FS descriptor in sector 2048 of /dev/zd16 root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd32 NSD descriptor in sector 64 of /dev/zd32 NSDid: 0A0023D258C1C02B format version: 1403 Label: Paxos sector: -1880562609 number of sectors: 8192 isPdisk: 0 Comment: NSD descriptor for Thu Mar 9 14:50:51 2017 No Disk descriptor in sector 96 of /dev/zd32 No FS descriptor in sector 2048 of /dev/zd32 root at nec:~/gpfs# Thanks in advance, all? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From valdis.kletnieks at vt.edu Fri Mar 10 20:54:42 2017 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Fri, 10 Mar 2017 15:54:42 -0500 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: <43922F46-548D-4D34-89D4-74D35C6EB447@vanderbilt.edu> References: <43922F46-548D-4D34-89D4-74D35C6EB447@vanderbilt.edu> Message-ID: <16136.1489179282@turing-police.cc.vt.edu> On Fri, 10 Mar 2017 20:43:37 +0000, "Buterbaugh, Kevin L" said: > So I tried to create a filesystem: > > root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 What was in flash.stanza? -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 484 bytes Desc: not available URL: From Kevin.Buterbaugh at Vanderbilt.Edu Fri Mar 10 20:59:11 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 10 Mar 2017 20:59:11 +0000 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: <16136.1489179282@turing-police.cc.vt.edu> References: <43922F46-548D-4D34-89D4-74D35C6EB447@vanderbilt.edu> <16136.1489179282@turing-police.cc.vt.edu> Message-ID: <9FEBE666-48CA-438B-B0AE-9B954491A9FF@vanderbilt.edu> %nsd: nsd=nsd1 usage=metadataOnly failureGroup=1 pool=system servers=nec device=/dev/zd32 %nsd: nsd=nsd2 usage=dataOnly failureGroup=1 pool=gpfsdata servers=nec device=/dev/zd16 %pool: pool=system blockSize=1M usage=metadataOnly layoutMap=scatter allowWriteAffinity=no %pool: pool=gpfsdata blockSize=1M usage=dataOnly layoutMap=scatter allowWriteAffinity=no > On Mar 10, 2017, at 2:54 PM, valdis.kletnieks at vt.edu wrote: > > On Fri, 10 Mar 2017 20:43:37 +0000, "Buterbaugh, Kevin L" said: > >> So I tried to create a filesystem: >> >> root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 > > What was in flash.stanza? > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Paul.Sanchez at deshaw.com Fri Mar 10 21:36:10 2017 From: Paul.Sanchez at deshaw.com (Sanchez, Paul) Date: Fri, 10 Mar 2017 21:36:10 +0000 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: <43922F46-548D-4D34-89D4-74D35C6EB447@vanderbilt.edu> References: <43922F46-548D-4D34-89D4-74D35C6EB447@vanderbilt.edu> Message-ID: <09e39c995afc410c822c6e66665cb98b@mbxtoa1.winmail.deshaw.com> See: https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm GPFS has a limited set of device search specs it uses to find connected NSDs. When using exotic devices, you need to whitelist the devices yourself using the user exit script at /var/mmfs/etc/nsddevices. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Buterbaugh, Kevin L Sent: Friday, March 10, 2017 3:44 PM To: gpfsug main discussion list Subject: [gpfsug-discuss] mmcrfs issue Hi All, We are testing out some flash storage. I created a couple of NSDs successfully (?): root at nec:~/gpfs# mmlsnsd -F File system Disk name NSD servers --------------------------------------------------------------------------- (free disk) nsd1 nec (free disk) nsd2 nec root at nec:~/gpfs# So I tried to create a filesystem: root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 GPFS: 6027-441 Unable to open disk 'nsd2' on node nec. GPFS: 6027-441 Unable to open disk 'nsd1' on node nec. No such device No such device GPFS: 6027-538 Error accessing disks. mmcrfs: 6027-1200 tscrfs failed. Cannot create gpfs0 mmcrfs: 6027-1639 Command failed. Examine previous error messages to determine cause. root at nec:~/gpfs# Does this output from readdescraw look normal? root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd16 NSD descriptor in sector 64 of /dev/zd16 NSDid: 0A0023D258C1C02C format version: 1403 Label: Paxos sector: -1931478434 number of sectors: 8192 isPdisk: 0 Comment: NSD descriptor for Thu Mar 9 14:50:52 2017 No Disk descriptor in sector 96 of /dev/zd16 No FS descriptor in sector 2048 of /dev/zd16 root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd32 NSD descriptor in sector 64 of /dev/zd32 NSDid: 0A0023D258C1C02B format version: 1403 Label: Paxos sector: -1880562609 number of sectors: 8192 isPdisk: 0 Comment: NSD descriptor for Thu Mar 9 14:50:51 2017 No Disk descriptor in sector 96 of /dev/zd32 No FS descriptor in sector 2048 of /dev/zd32 root at nec:~/gpfs# Thanks in advance, all? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Fri Mar 10 21:44:05 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 10 Mar 2017 16:44:05 -0500 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: <09e39c995afc410c822c6e66665cb98b@mbxtoa1.winmail.deshaw.com> References: <43922F46-548D-4D34-89D4-74D35C6EB447@vanderbilt.edu> <09e39c995afc410c822c6e66665cb98b@mbxtoa1.winmail.deshaw.com> Message-ID: <4b1fbfca-95c4-a15a-ef70-fcf3e20d936d@nasa.gov> Those look like zvol's. Out of curiosity have you set sync=always on the filesystem root or zvols themselves? It's my understanding that without that you risk data loss since GPFS won't ever cause a sync to be issued to the zvol for zfs to flush acknowledged but uncommitted writes. -Aaron On 3/10/17 4:36 PM, Sanchez, Paul wrote: > See: > https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm > > > > GPFS has a limited set of device search specs it uses to find connected > NSDs. When using exotic devices, you need to whitelist the devices > yourself using the user exit script at /var/mmfs/etc/nsddevices. > > > > *From:*gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of > *Buterbaugh, Kevin L > *Sent:* Friday, March 10, 2017 3:44 PM > *To:* gpfsug main discussion list > *Subject:* [gpfsug-discuss] mmcrfs issue > > > > Hi All, > > > > We are testing out some flash storage. I created a couple of NSDs > successfully (?): > > > > root at nec:~/gpfs# mmlsnsd -F > > > > File system Disk name NSD servers > > --------------------------------------------------------------------------- > > (free disk) nsd1 nec > > (free disk) nsd2 nec > > > > root at nec:~/gpfs# > > > > So I tried to create a filesystem: > > > > root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j > scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 > > GPFS: 6027-441 Unable to open disk 'nsd2' on node nec. > > GPFS: 6027-441 Unable to open disk 'nsd1' on node nec. > > No such device > > No such device > > GPFS: 6027-538 Error accessing disks. > > mmcrfs: 6027-1200 tscrfs failed. Cannot create gpfs0 > > mmcrfs: 6027-1639 Command failed. Examine previous error messages to > determine cause. > > root at nec:~/gpfs# > > > > Does this output from readdescraw look normal? > > > > root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd16 > > NSD descriptor in sector 64 of /dev/zd16 > > NSDid: 0A0023D258C1C02C format version: 1403 Label: > > Paxos sector: -1931478434 number of sectors: 8192 isPdisk: 0 > > Comment: NSD descriptor for Thu Mar 9 14:50:52 2017 > > No Disk descriptor in sector 96 of /dev/zd16 > > No FS descriptor in sector 2048 of /dev/zd16 > > root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd32 > > NSD descriptor in sector 64 of /dev/zd32 > > NSDid: 0A0023D258C1C02B format version: 1403 Label: > > Paxos sector: -1880562609 number of sectors: 8192 isPdisk: 0 > > Comment: NSD descriptor for Thu Mar 9 14:50:51 2017 > > No Disk descriptor in sector 96 of /dev/zd32 > > No FS descriptor in sector 2048 of /dev/zd32 > > root at nec:~/gpfs# > > > > Thanks in advance, all? > > > > Kevin > > ? > > Kevin Buterbaugh - Senior System Administrator > > Vanderbilt University - Advanced Computing Center for Research and Education > > Kevin.Buterbaugh at vanderbilt.edu > - (615)875-9633 > > > > > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From daniel.kidger at uk.ibm.com Sat Mar 11 10:37:47 2017 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Sat, 11 Mar 2017 10:37:47 +0000 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: <4b1fbfca-95c4-a15a-ef70-fcf3e20d936d@nasa.gov> Message-ID: On the subject of using zvols for software Raid/ replication, can ask as a quick poll, how many people are doing this? And any feedback on stability, tuning and performance? Daniel IBM Technical Presales > On 10 Mar 2017, at 22:44, Aaron Knister wrote: > > Those look like zvol's. Out of curiosity have you set sync=always on the > filesystem root or zvols themselves? It's my understanding that without > that you risk data loss since GPFS won't ever cause a sync to be issued > to the zvol for zfs to flush acknowledged but uncommitted writes. > > -Aaron > >> On 3/10/17 4:36 PM, Sanchez, Paul wrote: >> See: >> https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm >> >> >> >> GPFS has a limited set of device search specs it uses to find connected >> NSDs. When using exotic devices, you need to whitelist the devices >> yourself using the user exit script at /var/mmfs/etc/nsddevices. >> >> >> >> *From:*gpfsug-discuss-bounces at spectrumscale.org >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of >> *Buterbaugh, Kevin L >> *Sent:* Friday, March 10, 2017 3:44 PM >> *To:* gpfsug main discussion list >> *Subject:* [gpfsug-discuss] mmcrfs issue >> >> >> >> Hi All, >> >> >> >> We are testing out some flash storage. I created a couple of NSDs >> successfully (?): >> >> >> >> root at nec:~/gpfs# mmlsnsd -F >> >> >> >> File system Disk name NSD servers >> >> --------------------------------------------------------------------------- >> >> (free disk) nsd1 nec >> >> (free disk) nsd2 nec >> >> >> >> root at nec:~/gpfs# >> >> >> >> So I tried to create a filesystem: >> >> >> >> root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j >> scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 >> >> GPFS: 6027-441 Unable to open disk 'nsd2' on node nec. >> >> GPFS: 6027-441 Unable to open disk 'nsd1' on node nec. >> >> No such device >> >> No such device >> >> GPFS: 6027-538 Error accessing disks. >> >> mmcrfs: 6027-1200 tscrfs failed. Cannot create gpfs0 >> >> mmcrfs: 6027-1639 Command failed. Examine previous error messages to >> determine cause. >> >> root at nec:~/gpfs# >> >> >> >> Does this output from readdescraw look normal? >> >> >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd16 >> >> NSD descriptor in sector 64 of /dev/zd16 >> >> NSDid: 0A0023D258C1C02C format version: 1403 Label: >> >> Paxos sector: -1931478434 number of sectors: 8192 isPdisk: 0 >> >> Comment: NSD descriptor for Thu Mar 9 14:50:52 2017 >> >> No Disk descriptor in sector 96 of /dev/zd16 >> >> No FS descriptor in sector 2048 of /dev/zd16 >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd32 >> >> NSD descriptor in sector 64 of /dev/zd32 >> >> NSDid: 0A0023D258C1C02B format version: 1403 Label: >> >> Paxos sector: -1880562609 number of sectors: 8192 isPdisk: 0 >> >> Comment: NSD descriptor for Thu Mar 9 14:50:51 2017 >> >> No Disk descriptor in sector 96 of /dev/zd32 >> >> No FS descriptor in sector 2048 of /dev/zd32 >> >> root at nec:~/gpfs# >> >> >> >> Thanks in advance, all? >> >> >> >> Kevin >> >> ? >> >> Kevin Buterbaugh - Senior System Administrator >> >> Vanderbilt University - Advanced Computing Center for Research and Education >> >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss >Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Mon Mar 13 13:50:04 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 13 Mar 2017 13:50:04 +0000 Subject: [gpfsug-discuss] Registration Reminder: GPFSUG-USA Meeting at NERSC April 4-5th In-Reply-To: References: Message-ID: Here?s a reminder to register for the upcoming US user group meeting- Registration deadline is March 24th! Register here: https://www.nersc.gov/research-and-development/data-analytics/spectrum-user-group-meeting/ Bob Oesterlin Sr Principal Storage Engineer, Nuance -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Mon Mar 13 20:44:01 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Mon, 13 Mar 2017 20:44:01 +0000 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: References: Message-ID: <357B0FF5-4B7C-4941-8496-BA70D011A898@vanderbilt.edu> Hi All, Two things: 1) Paul?s suggestion to look at the nsddevices script was the answer I needed to fix my mmcrfs issue. Thanks. 2) I am also interested in hearing if anyone is using ZFS to create the equivalent of RAID 8+2P hardware RAID 6 LUNs and presenting that to GPFS to use as disks? Thanks? Kevin On Mar 11, 2017, at 4:37 AM, Daniel Kidger > wrote: On the subject of using zvols for software Raid/ replication, can ask as a quick poll, how many people are doing this? And any feedback on stability, tuning and performance? Daniel IBM Technical Presales > On 10 Mar 2017, at 22:44, Aaron Knister > wrote: > > Those look like zvol's. Out of curiosity have you set sync=always on the > filesystem root or zvols themselves? It's my understanding that without > that you risk data loss since GPFS won't ever cause a sync to be issued > to the zvol for zfs to flush acknowledged but uncommitted writes. > > -Aaron > >> On 3/10/17 4:36 PM, Sanchez, Paul wrote: >> See: >> https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm >> >> >> >> GPFS has a limited set of device search specs it uses to find connected >> NSDs. When using exotic devices, you need to whitelist the devices >> yourself using the user exit script at /var/mmfs/etc/nsddevices. >> >> >> >> *From:*gpfsug-discuss-bounces at spectrumscale.org >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of >> *Buterbaugh, Kevin L >> *Sent:* Friday, March 10, 2017 3:44 PM >> *To:* gpfsug main discussion list >> *Subject:* [gpfsug-discuss] mmcrfs issue >> >> >> >> Hi All, >> >> >> >> We are testing out some flash storage. I created a couple of NSDs >> successfully (?): >> >> >> >> root at nec:~/gpfs# mmlsnsd -F >> >> >> >> File system Disk name NSD servers >> >> --------------------------------------------------------------------------- >> >> (free disk) nsd1 nec >> >> (free disk) nsd2 nec >> >> >> >> root at nec:~/gpfs# >> >> >> >> So I tried to create a filesystem: >> >> >> >> root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j >> scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 >> >> GPFS: 6027-441 Unable to open disk 'nsd2' on node nec. >> >> GPFS: 6027-441 Unable to open disk 'nsd1' on node nec. >> >> No such device >> >> No such device >> >> GPFS: 6027-538 Error accessing disks. >> >> mmcrfs: 6027-1200 tscrfs failed. Cannot create gpfs0 >> >> mmcrfs: 6027-1639 Command failed. Examine previous error messages to >> determine cause. >> >> root at nec:~/gpfs# >> >> >> >> Does this output from readdescraw look normal? >> >> >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd16 >> >> NSD descriptor in sector 64 of /dev/zd16 >> >> NSDid: 0A0023D258C1C02C format version: 1403 Label: >> >> Paxos sector: -1931478434 number of sectors: 8192 isPdisk: 0 >> >> Comment: NSD descriptor for Thu Mar 9 14:50:52 2017 >> >> No Disk descriptor in sector 96 of /dev/zd16 >> >> No FS descriptor in sector 2048 of /dev/zd16 >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd32 >> >> NSD descriptor in sector 64 of /dev/zd32 >> >> NSDid: 0A0023D258C1C02B format version: 1403 Label: >> >> Paxos sector: -1880562609 number of sectors: 8192 isPdisk: 0 >> >> Comment: NSD descriptor for Thu Mar 9 14:50:51 2017 >> >> No Disk descriptor in sector 96 of /dev/zd32 >> >> No FS descriptor in sector 2048 of /dev/zd32 >> >> root at nec:~/gpfs# >> >> >> >> Thanks in advance, all? >> >> >> >> Kevin >> >> ? >> >> Kevin Buterbaugh - Senior System Administrator >> >> Vanderbilt University - Advanced Computing Center for Research and Education >> >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Tue Mar 14 00:06:29 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Mon, 13 Mar 2017 20:06:29 -0400 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: <357B0FF5-4B7C-4941-8496-BA70D011A898@vanderbilt.edu> References: <357B0FF5-4B7C-4941-8496-BA70D011A898@vanderbilt.edu> Message-ID: I was doing this in testing (with fantastic performance too) until I realized the issue with ZFS's behavior with direct io on zvols (e.g. not flushing a write to stable storage after acknowledging it to GPFS). After setting the sync=always parameter to not lose data in the event of a crash or power outage the write performance became unbearably slow (under 100MB/s of writes for an 8+2 RAIDZ2 if I recall correctly). I even tried adding a battery-backed PCIe write cache (http://www.netlist.com/products/vault-memory-storage/expressvault-pcIe-ev3/default.aspx) as a log device to the zpool but the performance was still really slow. I posted to the ZFS mailing list asking about how to optimize for a large block streaming workload but I didn't many bites (http://list.zfsonlinux.org/pipermail/zfs-discuss/2016-February/024851.html). I've got an RFE open with IBM (https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84994) to see if the behavior of GPFS could be changed such that it would issue explicit cache flushes that would allow it to work with ZFS (it might even be beneficial in FPO environments too). -Aaron On 3/13/17 4:44 PM, Buterbaugh, Kevin L wrote: > Hi All, > > Two things: > > 1) Paul?s suggestion to look at the nsddevices script was the answer I > needed to fix my mmcrfs issue. Thanks. > > 2) I am also interested in hearing if anyone is using ZFS to create the > equivalent of RAID 8+2P hardware RAID 6 LUNs and presenting that to GPFS > to use as disks? > > Thanks? > > Kevin > >> On Mar 11, 2017, at 4:37 AM, Daniel Kidger > > wrote: >> >> On the subject of using zvols for software Raid/ replication, can ask >> as a quick poll, how many people are doing this? >> >> And any feedback on stability, tuning and performance? >> >> Daniel >> IBM Technical Presales >> >> > On 10 Mar 2017, at 22:44, Aaron Knister > wrote: >> > >> > Those look like zvol's. Out of curiosity have you set sync=always on the >> > filesystem root or zvols themselves? It's my understanding that without >> > that you risk data loss since GPFS won't ever cause a sync to be issued >> > to the zvol for zfs to flush acknowledged but uncommitted writes. >> > >> > -Aaron >> > >> >> On 3/10/17 4:36 PM, Sanchez, Paul wrote: >> >> See: >> >> https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm >> >> >> >> >> >> >> >> GPFS has a limited set of device search specs it uses to find connected >> >> NSDs. When using exotic devices, you need to whitelist the devices >> >> yourself using the user exit script at /var/mmfs/etc/nsddevices. >> >> >> >> >> >> >> >> *From:*gpfsug-discuss-bounces at spectrumscale.org >> >> >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of >> >> *Buterbaugh, Kevin L >> >> *Sent:* Friday, March 10, 2017 3:44 PM >> >> *To:* gpfsug main discussion list >> >> *Subject:* [gpfsug-discuss] mmcrfs issue >> >> >> >> >> >> >> >> Hi All, >> >> >> >> >> >> >> >> We are testing out some flash storage. I created a couple of NSDs >> >> successfully (?): >> >> >> >> >> >> >> >> root at nec:~/gpfs# mmlsnsd -F >> >> >> >> >> >> >> >> File system Disk name NSD servers >> >> >> >> --------------------------------------------------------------------------- >> >> >> >> (free disk) nsd1 nec >> >> >> >> (free disk) nsd2 nec >> >> >> >> >> >> >> >> root at nec:~/gpfs# >> >> >> >> >> >> >> >> So I tried to create a filesystem: >> >> >> >> >> >> >> >> root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j >> >> scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 >> >> >> >> GPFS: 6027-441 Unable to open disk 'nsd2' on node nec. >> >> >> >> GPFS: 6027-441 Unable to open disk 'nsd1' on node nec. >> >> >> >> No such device >> >> >> >> No such device >> >> >> >> GPFS: 6027-538 Error accessing disks. >> >> >> >> mmcrfs: 6027-1200 tscrfs failed. Cannot create gpfs0 >> >> >> >> mmcrfs: 6027-1639 Command failed. Examine previous error messages to >> >> determine cause. >> >> >> >> root at nec:~/gpfs# >> >> >> >> >> >> >> >> Does this output from readdescraw look normal? >> >> >> >> >> >> >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd16 >> >> >> >> NSD descriptor in sector 64 of /dev/zd16 >> >> >> >> NSDid: 0A0023D258C1C02C format version: 1403 Label: >> >> >> >> Paxos sector: -1931478434 number of sectors: 8192 isPdisk: 0 >> >> >> >> Comment: NSD descriptor for Thu Mar 9 14:50:52 2017 >> >> >> >> No Disk descriptor in sector 96 of /dev/zd16 >> >> >> >> No FS descriptor in sector 2048 of /dev/zd16 >> >> >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd32 >> >> >> >> NSD descriptor in sector 64 of /dev/zd32 >> >> >> >> NSDid: 0A0023D258C1C02B format version: 1403 Label: >> >> >> >> Paxos sector: -1880562609 number of sectors: 8192 isPdisk: 0 >> >> >> >> Comment: NSD descriptor for Thu Mar 9 14:50:51 2017 >> >> >> >> No Disk descriptor in sector 96 of /dev/zd32 >> >> >> >> No FS descriptor in sector 2048 of /dev/zd32 >> >> >> >> root at nec:~/gpfs# >> >> >> >> >> >> >> >> Thanks in advance, all? >> >> >> >> >> >> >> >> Kevin >> >> >> >> ? >> >> >> >> Kevin Buterbaugh - Senior System Administrator >> >> >> >> Vanderbilt University - Advanced Computing Center for Research and Education >> >> >> >> Kevin.Buterbaugh at vanderbilt.edu >> >> - (615)875-9633 >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> gpfsug-discuss mailing list >> >> gpfsug-discuss at spectrumscale.org >> >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> > >> > -- >> > Aaron Knister >> > NASA Center for Climate Simulation (Code 606.2) >> > Goddard Space Flight Center >> > (301) 286-2776 >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > >> Unless stated otherwise above: >> IBM United Kingdom Limited - Registered in England and Wales with >> number 741598. >> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From valdis.kletnieks at vt.edu Tue Mar 14 18:15:17 2017 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Tue, 14 Mar 2017 14:15:17 -0400 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: References: <357B0FF5-4B7C-4941-8496-BA70D011A898@vanderbilt.edu> Message-ID: <16774.1489515317@turing-police.cc.vt.edu> On Mon, 13 Mar 2017 20:06:29 -0400, Aaron Knister said: > After setting the sync=always parameter to not lose data in the event of > a crash or power outage the write performance became unbearably slow > (under 100MB/s of writes for an 8+2 RAIDZ2 if I recall correctly). I Not at all surprising. Forcing a 'sync every write' has been painful for pretty much everything - first time I saw it was on a SunOS 3.2 box doing NFS over 3 decades ago. > I've got an RFE open with IBM > (https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84994) > to see if the behavior of GPFS could be changed such that it would issue > explicit cache flushes that would allow it to work with ZFS (it might > even be beneficial in FPO environments too). Do you have a suggestion for how to do this without it turning into 'sync=always' just done at a different layer in the stack? -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 484 bytes Desc: not available URL: From aaron.s.knister at nasa.gov Tue Mar 14 18:31:55 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 14 Mar 2017 14:31:55 -0400 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: <16774.1489515317@turing-police.cc.vt.edu> References: <357B0FF5-4B7C-4941-8496-BA70D011A898@vanderbilt.edu> <16774.1489515317@turing-police.cc.vt.edu> Message-ID: <411ef927-533e-c2b6-289f-44f0d8845c4f@nasa.gov> On 3/14/17 2:15 PM, valdis.kletnieks at vt.edu wrote: > On Mon, 13 Mar 2017 20:06:29 -0400, Aaron Knister said: >> After setting the sync=always parameter to not lose data in the event of >> a crash or power outage the write performance became unbearably slow >> (under 100MB/s of writes for an 8+2 RAIDZ2 if I recall correctly). I > > Not at all surprising. Forcing a 'sync every write' has been painful for pretty > much everything - first time I saw it was on a SunOS 3.2 box doing NFS over 3 > decades ago. > You know, what's interesting is when you do direct I/O to a linux md RAID device each acknowledged I/O to the md device results in SCSI SYNCHRONIZE CACHE commands being sent to the underlying member devices. I consider that doing a sync of sorts after each write and as long as everything is properly aligned you can get pretty fantastic performance from this. No data checksums, though :( >> I've got an RFE open with IBM >> (https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84994) >> to see if the behavior of GPFS could be changed such that it would issue >> explicit cache flushes that would allow it to work with ZFS (it might >> even be beneficial in FPO environments too). > > Do you have a suggestion for how to do this without it turning into 'sync=always' > just done at a different layer in the stack? > I don't think that GPFS would need to issue a sync after *every* write. I think the key is more that GPFS is aware of the state of acknowledged but uncommitted writes so that if there's something *really* important to be written (i.e. an internal piece of metadata) GPFS can force it to be committed to avoid GPFS itself becoming inconsistent. You don't want to have to mmfsck after a power outage/crash if there were lost writes that GPFS assumed were committed. -Aaron > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: OpenPGP digital signature URL: From luke.raimbach at googlemail.com Tue Mar 14 21:59:50 2017 From: luke.raimbach at googlemail.com (Luke Raimbach) Date: Tue, 14 Mar 2017 21:59:50 +0000 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: References: <357B0FF5-4B7C-4941-8496-BA70D011A898@vanderbilt.edu> Message-ID: Can I ask what the fascination with zvols is? Using a copy-on-write file system to underpin another block based file system seems counterintuitive. Perhaps I've missed something vital, in which case I'd be delighted to have my eyes opened! On Tue, 14 Mar 2017, 00:06 Aaron Knister, wrote: > I was doing this in testing (with fantastic performance too) until I > realized the issue with ZFS's behavior with direct io on zvols (e.g. not > flushing a write to stable storage after acknowledging it to GPFS). > After setting the sync=always parameter to not lose data in the event of > a crash or power outage the write performance became unbearably slow > (under 100MB/s of writes for an 8+2 RAIDZ2 if I recall correctly). I > even tried adding a battery-backed PCIe write cache > ( > http://www.netlist.com/products/vault-memory-storage/expressvault-pcIe-ev3/default.aspx > ) > as a log device to the zpool but the performance was still really slow. > I posted to the ZFS mailing list asking about how to optimize for a > large block streaming workload but I didn't many bites > ( > http://list.zfsonlinux.org/pipermail/zfs-discuss/2016-February/024851.html > ). > > I've got an RFE open with IBM > ( > https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84994 > ) > to see if the behavior of GPFS could be changed such that it would issue > explicit cache flushes that would allow it to work with ZFS (it might > even be beneficial in FPO environments too). > > -Aaron > > On 3/13/17 4:44 PM, Buterbaugh, Kevin L wrote: > > Hi All, > > > > Two things: > > > > 1) Paul?s suggestion to look at the nsddevices script was the answer I > > needed to fix my mmcrfs issue. Thanks. > > > > 2) I am also interested in hearing if anyone is using ZFS to create the > > equivalent of RAID 8+2P hardware RAID 6 LUNs and presenting that to GPFS > > to use as disks? > > > > Thanks? > > > > Kevin > > > >> On Mar 11, 2017, at 4:37 AM, Daniel Kidger >> > wrote: > >> > >> On the subject of using zvols for software Raid/ replication, can ask > >> as a quick poll, how many people are doing this? > >> > >> And any feedback on stability, tuning and performance? > >> > >> Daniel > >> IBM Technical Presales > >> > >> > On 10 Mar 2017, at 22:44, Aaron Knister > wrote: > >> > > >> > Those look like zvol's. Out of curiosity have you set sync=always on > the > >> > filesystem root or zvols themselves? It's my understanding that > without > >> > that you risk data loss since GPFS won't ever cause a sync to be > issued > >> > to the zvol for zfs to flush acknowledged but uncommitted writes. > >> > > >> > -Aaron > >> > > >> >> On 3/10/17 4:36 PM, Sanchez, Paul wrote: > >> >> See: > >> >> > https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm > >> >> > >> >> > >> >> > >> >> GPFS has a limited set of device search specs it uses to find > connected > >> >> NSDs. When using exotic devices, you need to whitelist the devices > >> >> yourself using the user exit script at /var/mmfs/etc/nsddevices. > >> >> > >> >> > >> >> > >> >> *From:*gpfsug-discuss-bounces at spectrumscale.org > >> > >> >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of > >> >> *Buterbaugh, Kevin L > >> >> *Sent:* Friday, March 10, 2017 3:44 PM > >> >> *To:* gpfsug main discussion list > >> >> *Subject:* [gpfsug-discuss] mmcrfs issue > >> >> > >> >> > >> >> > >> >> Hi All, > >> >> > >> >> > >> >> > >> >> We are testing out some flash storage. I created a couple of NSDs > >> >> successfully (?): > >> >> > >> >> > >> >> > >> >> root at nec:~/gpfs# mmlsnsd -F > >> >> > >> >> > >> >> > >> >> File system Disk name NSD servers > >> >> > >> >> > --------------------------------------------------------------------------- > >> >> > >> >> (free disk) nsd1 nec > >> >> > >> >> (free disk) nsd2 nec > >> >> > >> >> > >> >> > >> >> root at nec:~/gpfs# > >> >> > >> >> > >> >> > >> >> So I tried to create a filesystem: > >> >> > >> >> > >> >> > >> >> root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j > >> >> scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 > >> >> > >> >> GPFS: 6027-441 Unable to open disk 'nsd2' on node nec. > >> >> > >> >> GPFS: 6027-441 Unable to open disk 'nsd1' on node nec. > >> >> > >> >> No such device > >> >> > >> >> No such device > >> >> > >> >> GPFS: 6027-538 Error accessing disks. > >> >> > >> >> mmcrfs: 6027-1200 tscrfs failed. Cannot create gpfs0 > >> >> > >> >> mmcrfs: 6027-1639 Command failed. Examine previous error messages to > >> >> determine cause. > >> >> > >> >> root at nec:~/gpfs# > >> >> > >> >> > >> >> > >> >> Does this output from readdescraw look normal? > >> >> > >> >> > >> >> > >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd16 > >> >> > >> >> NSD descriptor in sector 64 of /dev/zd16 > >> >> > >> >> NSDid: 0A0023D258C1C02C format version: 1403 Label: > >> >> > >> >> Paxos sector: -1931478434 number of sectors: 8192 isPdisk: 0 > >> >> > >> >> Comment: NSD descriptor for Thu Mar 9 14:50:52 2017 > >> >> > >> >> No Disk descriptor in sector 96 of /dev/zd16 > >> >> > >> >> No FS descriptor in sector 2048 of /dev/zd16 > >> >> > >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd32 > >> >> > >> >> NSD descriptor in sector 64 of /dev/zd32 > >> >> > >> >> NSDid: 0A0023D258C1C02B format version: 1403 Label: > >> >> > >> >> Paxos sector: -1880562609 number of sectors: 8192 isPdisk: 0 > >> >> > >> >> Comment: NSD descriptor for Thu Mar 9 14:50:51 2017 > >> >> > >> >> No Disk descriptor in sector 96 of /dev/zd32 > >> >> > >> >> No FS descriptor in sector 2048 of /dev/zd32 > >> >> > >> >> root at nec:~/gpfs# > >> >> > >> >> > >> >> > >> >> Thanks in advance, all? > >> >> > >> >> > >> >> > >> >> Kevin > >> >> > >> >> ? > >> >> > >> >> Kevin Buterbaugh - Senior System Administrator > >> >> > >> >> Vanderbilt University - Advanced Computing Center for Research and > Education > >> >> > >> >> Kevin.Buterbaugh at vanderbilt.edu Kevin.Buterbaugh at vanderbilt.edu> > >> >> - (615)875-9633 > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> _______________________________________________ > >> >> gpfsug-discuss mailing list > >> >> gpfsug-discuss at spectrumscale.org > >> >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> >> > >> > > >> > -- > >> > Aaron Knister > >> > NASA Center for Climate Simulation (Code 606.2) > >> > Goddard Space Flight Center > >> > (301) 286-2776 > >> > _______________________________________________ > >> > gpfsug-discuss mailing list > >> > gpfsug-discuss at spectrumscale.org > >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > >> Unless stated otherwise above: > >> IBM United Kingdom Limited - Registered in England and Wales with > >> number 741598. > >> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 > 3AU > >> > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Tue Mar 14 22:16:43 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 14 Mar 2017 18:16:43 -0400 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: References: <357B0FF5-4B7C-4941-8496-BA70D011A898@vanderbilt.edu> Message-ID: <2d4fa452-3137-0a44-fd18-bed4b9923279@nasa.gov> For me it's the protection against bitrot and added protection against silent data corruption and in theory the write caching offered by adding log devices that could help with small random writes (although there are other problems with ZFS + synchronous workloads that stop this from actually materializing). -Aaron On 3/14/17 5:59 PM, Luke Raimbach wrote: > Can I ask what the fascination with zvols is? Using a copy-on-write file > system to underpin another block based file system seems > counterintuitive. Perhaps I've missed something vital, in which case I'd > be delighted to have my eyes opened! > > On Tue, 14 Mar 2017, 00:06 Aaron Knister, > wrote: > > I was doing this in testing (with fantastic performance too) until I > realized the issue with ZFS's behavior with direct io on zvols (e.g. not > flushing a write to stable storage after acknowledging it to GPFS). > After setting the sync=always parameter to not lose data in the event of > a crash or power outage the write performance became unbearably slow > (under 100MB/s of writes for an 8+2 RAIDZ2 if I recall correctly). I > even tried adding a battery-backed PCIe write cache > (http://www.netlist.com/products/vault-memory-storage/expressvault-pcIe-ev3/default.aspx) > as a log device to the zpool but the performance was still really slow. > I posted to the ZFS mailing list asking about how to optimize for a > large block streaming workload but I didn't many bites > (http://list.zfsonlinux.org/pipermail/zfs-discuss/2016-February/024851.html). > > I've got an RFE open with IBM > (https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84994) > to see if the behavior of GPFS could be changed such that it would issue > explicit cache flushes that would allow it to work with ZFS (it might > even be beneficial in FPO environments too). > > -Aaron > > On 3/13/17 4:44 PM, Buterbaugh, Kevin L wrote: > > Hi All, > > > > Two things: > > > > 1) Paul?s suggestion to look at the nsddevices script was the answer I > > needed to fix my mmcrfs issue. Thanks. > > > > 2) I am also interested in hearing if anyone is using ZFS to create the > > equivalent of RAID 8+2P hardware RAID 6 LUNs and presenting that to GPFS > > to use as disks? > > > > Thanks? > > > > Kevin > > > >> On Mar 11, 2017, at 4:37 AM, Daniel Kidger > >> >> wrote: > >> > >> On the subject of using zvols for software Raid/ replication, can ask > >> as a quick poll, how many people are doing this? > >> > >> And any feedback on stability, tuning and performance? > >> > >> Daniel > >> IBM Technical Presales > >> > >> > On 10 Mar 2017, at 22:44, Aaron Knister > >> > wrote: > >> > > >> > Those look like zvol's. Out of curiosity have you set sync=always on the > >> > filesystem root or zvols themselves? It's my understanding that without > >> > that you risk data loss since GPFS won't ever cause a sync to be issued > >> > to the zvol for zfs to flush acknowledged but uncommitted writes. > >> > > >> > -Aaron > >> > > >> >> On 3/10/17 4:36 PM, Sanchez, Paul wrote: > >> >> See: > >> >> https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm > >> >> > >> >> > >> >> > >> >> GPFS has a limited set of device search specs it uses to find connected > >> >> NSDs. When using exotic devices, you need to whitelist the devices > >> >> yourself using the user exit script at /var/mmfs/etc/nsddevices. > >> >> > >> >> > >> >> > >> >> *From:*gpfsug-discuss-bounces at spectrumscale.org > > >> > > >> >> [mailto:gpfsug-discuss-bounces at spectrumscale.org > ] *On Behalf Of > >> >> *Buterbaugh, Kevin L > >> >> *Sent:* Friday, March 10, 2017 3:44 PM > >> >> *To:* gpfsug main discussion list > >> >> *Subject:* [gpfsug-discuss] mmcrfs issue > >> >> > >> >> > >> >> > >> >> Hi All, > >> >> > >> >> > >> >> > >> >> We are testing out some flash storage. I created a couple of NSDs > >> >> successfully (?): > >> >> > >> >> > >> >> > >> >> root at nec:~/gpfs# mmlsnsd -F > >> >> > >> >> > >> >> > >> >> File system Disk name NSD servers > >> >> > >> >> --------------------------------------------------------------------------- > >> >> > >> >> (free disk) nsd1 nec > >> >> > >> >> (free disk) nsd2 nec > >> >> > >> >> > >> >> > >> >> root at nec:~/gpfs# > >> >> > >> >> > >> >> > >> >> So I tried to create a filesystem: > >> >> > >> >> > >> >> > >> >> root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j > >> >> scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 > >> >> > >> >> GPFS: 6027-441 Unable to open disk 'nsd2' on node nec. > >> >> > >> >> GPFS: 6027-441 Unable to open disk 'nsd1' on node nec. > >> >> > >> >> No such device > >> >> > >> >> No such device > >> >> > >> >> GPFS: 6027-538 Error accessing disks. > >> >> > >> >> mmcrfs: 6027-1200 tscrfs failed. Cannot create gpfs0 > >> >> > >> >> mmcrfs: 6027-1639 Command failed. Examine previous error messages to > >> >> determine cause. > >> >> > >> >> root at nec:~/gpfs# > >> >> > >> >> > >> >> > >> >> Does this output from readdescraw look normal? > >> >> > >> >> > >> >> > >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd16 > >> >> > >> >> NSD descriptor in sector 64 of /dev/zd16 > >> >> > >> >> NSDid: 0A0023D258C1C02C format version: 1403 Label: > >> >> > >> >> Paxos sector: -1931478434 number of sectors: 8192 isPdisk: 0 > >> >> > >> >> Comment: NSD descriptor for Thu Mar 9 14:50:52 2017 > >> >> > >> >> No Disk descriptor in sector 96 of /dev/zd16 > >> >> > >> >> No FS descriptor in sector 2048 of /dev/zd16 > >> >> > >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd32 > >> >> > >> >> NSD descriptor in sector 64 of /dev/zd32 > >> >> > >> >> NSDid: 0A0023D258C1C02B format version: 1403 Label: > >> >> > >> >> Paxos sector: -1880562609 number of sectors: 8192 isPdisk: 0 > >> >> > >> >> Comment: NSD descriptor for Thu Mar 9 14:50:51 2017 > >> >> > >> >> No Disk descriptor in sector 96 of /dev/zd32 > >> >> > >> >> No FS descriptor in sector 2048 of /dev/zd32 > >> >> > >> >> root at nec:~/gpfs# > >> >> > >> >> > >> >> > >> >> Thanks in advance, all? > >> >> > >> >> > >> >> > >> >> Kevin > >> >> > >> >> ? > >> >> > >> >> Kevin Buterbaugh - Senior System Administrator > >> >> > >> >> Vanderbilt University - Advanced Computing Center for Research and Education > >> >> > >> >> Kevin.Buterbaugh at vanderbilt.edu > > > > >> >> > - (615)875-9633 > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> _______________________________________________ > >> >> gpfsug-discuss mailing list > >> >> gpfsug-discuss at spectrumscale.org > >> >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> >> > >> > > >> > -- > >> > Aaron Knister > >> > NASA Center for Climate Simulation (Code 606.2) > >> > Goddard Space Flight Center > >> > (301) 286-2776 > >> > _______________________________________________ > >> > gpfsug-discuss mailing list > >> > gpfsug-discuss at spectrumscale.org > >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > >> Unless stated otherwise above: > >> IBM United Kingdom Limited - Registered in England and Wales with > >> number 741598. > >> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > >> > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From xhejtman at ics.muni.cz Wed Mar 15 09:50:31 2017 From: xhejtman at ics.muni.cz (Lukas Hejtmanek) Date: Wed, 15 Mar 2017 10:50:31 +0100 Subject: [gpfsug-discuss] default inode size Message-ID: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> Hello, does anyone know why with GPFS 4.x, there is inode size 4096B by default. Is there any problem with, e.g., only 1024B inode size? -- Luk?? Hejtm?nek From ulmer at ulmer.org Wed Mar 15 12:22:22 2017 From: ulmer at ulmer.org (Stephen Ulmer) Date: Wed, 15 Mar 2017 08:22:22 -0400 Subject: [gpfsug-discuss] default inode size In-Reply-To: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> References: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> Message-ID: <0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> You need 4K nodes to store encryption keys. You can also put other useful things in there, like extended attributes and (possibly) the entire file. Are you worried about wasting space? -- Stephen > On Mar 15, 2017, at 5:50 AM, Lukas Hejtmanek > wrote: > > Hello, > > does anyone know why with GPFS 4.x, there is inode size 4096B by default. > > Is there any problem with, e.g., only 1024B inode size? > > -- > Luk?? Hejtm?nek > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From xhejtman at ics.muni.cz Wed Mar 15 12:37:00 2017 From: xhejtman at ics.muni.cz (Lukas Hejtmanek) Date: Wed, 15 Mar 2017 13:37:00 +0100 Subject: [gpfsug-discuss] default inode size In-Reply-To: <0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> References: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> <0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> Message-ID: <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> On Wed, Mar 15, 2017 at 08:22:22AM -0400, Stephen Ulmer wrote: > You need 4K nodes to store encryption keys. You can also put other useful things in there, like extended attributes and (possibly) the entire file. > > Are you worried about wasting space? well, I have 1PB of capacity for data disks and 3TB of capacity for metadata (SSD). With 512B inodes, this ration seemed to be quite ok, while with 4k inodes, I run out of free space on SSD pretty fast. So I'm thinking about re-creating file system with smaller inode size. I don't think I will ever need encryption keys. -- Luk?? Hejtm?nek From luis.bolinches at fi.ibm.com Wed Mar 15 13:09:07 2017 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Wed, 15 Mar 2017 13:09:07 +0000 Subject: [gpfsug-discuss] default inode size In-Reply-To: <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> References: <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz>, <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz><0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> Message-ID: An HTML attachment was scrubbed... URL: From dieter.gorecki at atos.net Wed Mar 15 13:12:44 2017 From: dieter.gorecki at atos.net (GORECKI, DIETER) Date: Wed, 15 Mar 2017 13:12:44 +0000 Subject: [gpfsug-discuss] default inode size In-Reply-To: <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> References: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> <0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> Message-ID: Lukas, One other thing to consider is the storage of data inside the inode itself for very small files. GPFS has the ability to use the remaining [kilo]bytes of the inode to store the data of the file whenever the file is small enough to fit in. Anyone correct me if I am wrong, but with 4k inodes, you can store up to (4096-128 header) 3968 bytes of data. (without ILM) So regarding the size of the files you intend to store into your filesystem, it might be very interesting to take advantage of the performance of your SSD's to store small files. Regards, Dieter -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Lukas Hejtmanek Sent: Wednesday, March 15, 2017 1:37 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] default inode size On Wed, Mar 15, 2017 at 08:22:22AM -0400, Stephen Ulmer wrote: > You need 4K nodes to store encryption keys. You can also put other useful things in there, like extended attributes and (possibly) the entire file. > > Are you worried about wasting space? well, I have 1PB of capacity for data disks and 3TB of capacity for metadata (SSD). With 512B inodes, this ration seemed to be quite ok, while with 4k inodes, I run out of free space on SSD pretty fast. So I'm thinking about re-creating file system with smaller inode size. I don't think I will ever need encryption keys. -- Luk?? Hejtm?nek _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From xhejtman at ics.muni.cz Wed Mar 15 13:21:40 2017 From: xhejtman at ics.muni.cz (Lukas Hejtmanek) Date: Wed, 15 Mar 2017 14:21:40 +0100 Subject: [gpfsug-discuss] default inode size In-Reply-To: References: <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> <0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> Message-ID: <20170315132140.v5rfagqy4hqzqsfh@ics.muni.cz> Hello, On Wed, Mar 15, 2017 at 01:09:07PM +0000, Luis Bolinches wrote: > My 2 cents. Before even thinking too much on that path I would check the > following > ? > - What the is physical size on those SSD if they are already 4K you won't > "save" anything size of what? > - Do you use small (>3.5Kb) files? If so I would still keep 4K inodes > - Check if 512 can hold NSDv2 format, from top of my head it cannot but > pls check. If so, do you want to use a format that anyway is going to fade > away and lose the goodies that format brings? > ? > I am sure few other pros (and cons) can be put on but I would start with > those 3 I was thinking about 1024B inodes. Is this still not enough for NSDv2 format? -- Luk?? Hejtm?nek From luis.bolinches at fi.ibm.com Wed Mar 15 13:24:11 2017 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Wed, 15 Mar 2017 13:24:11 +0000 Subject: [gpfsug-discuss] default inode size In-Reply-To: <20170315132140.v5rfagqy4hqzqsfh@ics.muni.cz> References: <20170315132140.v5rfagqy4hqzqsfh@ics.muni.cz>, <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz><20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz><0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> Message-ID: An HTML attachment was scrubbed... URL: From xhejtman at ics.muni.cz Wed Mar 15 13:26:09 2017 From: xhejtman at ics.muni.cz (Lukas Hejtmanek) Date: Wed, 15 Mar 2017 14:26:09 +0100 Subject: [gpfsug-discuss] default inode size In-Reply-To: References: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> <0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> Message-ID: <20170315132609.wjplicnvhizlwc3q@ics.muni.cz> On Wed, Mar 15, 2017 at 01:12:44PM +0000, GORECKI, DIETER wrote: > One other thing to consider is the storage of data inside the inode itself for very small files. GPFS has the ability to use the remaining [kilo]bytes of the inode to store the data of the file whenever the file is small enough to fit in. > > Anyone correct me if I am wrong, but with 4k inodes, you can store up to (4096-128 header) 3968 bytes of data. (without ILM) > > So regarding the size of the files you intend to store into your filesystem, it might be very interesting to take advantage of the performance of your SSD's to store small files. I agree it would, however, I have 30 % free capacity on SSDs only right now (with some snapshots and ~70M files). So I'm afraid I have to either change the rotational disks to hold metadata as well or do not create more files/snapshots or decrease the inode size. I cannot add more SSDs as I do not have free disk slots in HW. -- Luk?? Hejtm?nek From ulmer at ulmer.org Wed Mar 15 13:36:24 2017 From: ulmer at ulmer.org (Stephen Ulmer) Date: Wed, 15 Mar 2017 09:36:24 -0400 Subject: [gpfsug-discuss] default inode size In-Reply-To: <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> References: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> <0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> Message-ID: > On Mar 15, 2017, at 8:37 AM, Lukas Hejtmanek wrote: > > On Wed, Mar 15, 2017 at 08:22:22AM -0400, Stephen Ulmer wrote: >> You need 4K nodes to store encryption keys. You can also put other useful things in there, like extended attributes and (possibly) the entire file. >> >> Are you worried about wasting space? > > well, I have 1PB of capacity for data disks and 3TB of capacity for metadata > (SSD). > > With 512B inodes, this ration seemed to be quite ok, while with 4k inodes, > I run out of free space on SSD pretty fast. So I'm thinking about re-creating > file system with smaller inode size. > > I don't think I will ever need encryption keys. What?s the average file size? Every file that is less than *about* 3.5K will wind up in the (4k) inode itself. Also check the HW sector size of the SSDs (many are 4K now). If that?s the case, you?ll actually be much more efficient from an access point of view with 4K inodes. If you don? t have 1PB of data yet (or won?t in the next year or so), you can always expand the system pool with more SSDs in the future. It?s not scary to pick another size ? it works just fine, but you?re about to create a filesystem that will be a pain in the rear to change. It will take a long time to move/backfill even a few hundred TB of data if you change you mind later. You?ll definitely give up (possibly future) features by choosing not 4K, but you?ve got the flexibility to do so. :) Liberty, -- Stephen From xhejtman at ics.muni.cz Wed Mar 15 13:46:54 2017 From: xhejtman at ics.muni.cz (Lukas Hejtmanek) Date: Wed, 15 Mar 2017 14:46:54 +0100 Subject: [gpfsug-discuss] default inode size In-Reply-To: References: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> <0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> Message-ID: <20170315134654.335p4rsrnaxgrjrq@ics.muni.cz> On Wed, Mar 15, 2017 at 09:36:24AM -0400, Stephen Ulmer wrote: > What?s the average file size? Every file that is less than *about* 3.5K will wind up in the (4k) inode itself. average file size is about 4MB. > Also check the HW sector size of the SSDs (many are 4K now). If that?s the case, you?ll actually be much more efficient from an access point of view with 4K inodes. HW sector size reported by disk array is 512B. SSD itself uses 4k though, however, Linux thinks it is only 512B. > If you don? t have 1PB of data yet (or won?t in the next year or so), you can always expand the system pool with more SSDs in the future. I cannot. I don't have free HW slots. > It?s not scary to pick another size ? it works just fine, but you?re about to create a filesystem that will be a pain in the rear to change. It will take a long time to move/backfill even a few hundred TB of data if you change you mind later. You?ll definitely give up (possibly future) features by choosing not 4K, but you?ve got the flexibility to do so. :) But at least I will have space to store metadata :) -- Luk?? Hejtm?nek From aaron.s.knister at nasa.gov Wed Mar 15 13:59:52 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Wed, 15 Mar 2017 09:59:52 -0400 Subject: [gpfsug-discuss] default inode size In-Reply-To: <20170315134654.335p4rsrnaxgrjrq@ics.muni.cz> References: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> <0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> <20170315134654.335p4rsrnaxgrjrq@ics.muni.cz> Message-ID: <4bb5ec26-bf98-76b4-5399-0d315b502458@nasa.gov> I'm not sure because I've not tried it but if you chose an inode size less than 4K does the FS still show as being 4K aligned? If it doesn't this could prevent you from expanding the metadata capacity of the FS in the future because in a non 4k aligned fs GPFS won't let you add 4K sector size LUNS that contain metadata. -Aaron On 3/15/17 9:46 AM, Lukas Hejtmanek wrote: > On Wed, Mar 15, 2017 at 09:36:24AM -0400, Stephen Ulmer wrote: >> What?s the average file size? Every file that is less than *about* 3.5K will wind up in the (4k) inode itself. > > average file size is about 4MB. > >> Also check the HW sector size of the SSDs (many are 4K now). If that?s the case, you?ll actually be much more efficient from an access point of view with 4K inodes. > > HW sector size reported by disk array is 512B. SSD itself uses 4k though, > however, Linux thinks it is only 512B. > >> If you don? t have 1PB of data yet (or won?t in the next year or so), you can always expand the system pool with more SSDs in the future. > > I cannot. I don't have free HW slots. > >> It?s not scary to pick another size ? it works just fine, but you?re about to create a filesystem that will be a pain in the rear to change. It will take a long time to move/backfill even a few hundred TB of data if you change you mind later. You?ll definitely give up (possibly future) features by choosing not 4K, but you?ve got the flexibility to do so. :) > > But at least I will have space to store metadata :) > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From janfrode at tanso.net Wed Mar 15 14:22:36 2017 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Wed, 15 Mar 2017 15:22:36 +0100 Subject: [gpfsug-discuss] default inode size In-Reply-To: <20170315132609.wjplicnvhizlwc3q@ics.muni.cz> References: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> <0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> <20170315132609.wjplicnvhizlwc3q@ics.muni.cz> Message-ID: Another thing to consider is how many disk block pointers you have room for in the inode, and when you'll need to add additional indirect blocks. Ref: http://files.gpfsug.org/presentations/2016/south-bank/ D2_P2_A_spectrum_scale_metadata_dark_V2a.pdf If I understand that presentation correctly.. a block pointer is 12 bytes, so a 1 KB inode can hold approximately 70 block pointers -- ie. any file larger than 70 blocks will need additional indirect blocks to hold the addresses, requiring minimum one additional sub-block. With a 1 MB blocksize filesystem (data and metadata), files between 75 MB and 3 GB will need 1x inode and 1x indirect block = 1 KB + 32 KB metadata space. If your files are smaller than 75 MB, the address pointers fit in the 1K inode. If you have 4 KB inode, files smaller than 340 MB will fit all pointers in the inode. -jf On Wed, Mar 15, 2017 at 2:26 PM, Lukas Hejtmanek wrote: > On Wed, Mar 15, 2017 at 01:12:44PM +0000, GORECKI, DIETER wrote: > > One other thing to consider is the storage of data inside the inode > itself for very small files. GPFS has the ability to use the remaining > [kilo]bytes of the inode to store the data of the file whenever the file is > small enough to fit in. > > > > Anyone correct me if I am wrong, but with 4k inodes, you can store up to > (4096-128 header) 3968 bytes of data. (without ILM) > > > > So regarding the size of the files you intend to store into your > filesystem, it might be very interesting to take advantage of the > performance of your SSD's to store small files. > > I agree it would, however, I have 30 % free capacity on SSDs only right > now (with > some snapshots and ~70M files). So I'm afraid I have to either change the > rotational disks to hold metadata as well or do not create more > files/snapshots or decrease the inode size. I cannot add more SSDs as I do > not > have free disk slots in HW. > > -- > Luk?? Hejtm?nek > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Wed Mar 15 14:25:41 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Wed, 15 Mar 2017 14:25:41 +0000 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: <2d4fa452-3137-0a44-fd18-bed4b9923279@nasa.gov> References: <357B0FF5-4B7C-4941-8496-BA70D011A898@vanderbilt.edu> <2d4fa452-3137-0a44-fd18-bed4b9923279@nasa.gov> Message-ID: Hi All, Since I started this thread I guess I should chime in, too ? for us it was simply that we were testing a device that did not have hardware RAID controllers and we were wanting to implement something roughly equivalent to RAID 6 LUNs. Kevin > On Mar 14, 2017, at 5:16 PM, Aaron Knister wrote: > > For me it's the protection against bitrot and added protection against silent data corruption and in theory the write caching offered by adding log devices that could help with small random writes (although there are other problems with ZFS + synchronous workloads that stop this from actually materializing). > > -Aaron > > On 3/14/17 5:59 PM, Luke Raimbach wrote: >> Can I ask what the fascination with zvols is? Using a copy-on-write file >> system to underpin another block based file system seems >> counterintuitive. Perhaps I've missed something vital, in which case I'd >> be delighted to have my eyes opened! >> >> On Tue, 14 Mar 2017, 00:06 Aaron Knister, > > wrote: >> >> I was doing this in testing (with fantastic performance too) until I >> realized the issue with ZFS's behavior with direct io on zvols (e.g. not >> flushing a write to stable storage after acknowledging it to GPFS). >> After setting the sync=always parameter to not lose data in the event of >> a crash or power outage the write performance became unbearably slow >> (under 100MB/s of writes for an 8+2 RAIDZ2 if I recall correctly). I >> even tried adding a battery-backed PCIe write cache >> (http://www.netlist.com/products/vault-memory-storage/expressvault-pcIe-ev3/default.aspx) >> as a log device to the zpool but the performance was still really slow. >> I posted to the ZFS mailing list asking about how to optimize for a >> large block streaming workload but I didn't many bites >> (http://list.zfsonlinux.org/pipermail/zfs-discuss/2016-February/024851.html). >> >> I've got an RFE open with IBM >> (https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84994) >> to see if the behavior of GPFS could be changed such that it would issue >> explicit cache flushes that would allow it to work with ZFS (it might >> even be beneficial in FPO environments too). >> >> -Aaron >> >> On 3/13/17 4:44 PM, Buterbaugh, Kevin L wrote: >> > Hi All, >> > >> > Two things: >> > >> > 1) Paul?s suggestion to look at the nsddevices script was the answer I >> > needed to fix my mmcrfs issue. Thanks. >> > >> > 2) I am also interested in hearing if anyone is using ZFS to create the >> > equivalent of RAID 8+2P hardware RAID 6 LUNs and presenting that to GPFS >> > to use as disks? >> > >> > Thanks? >> > >> > Kevin >> > >> >> On Mar 11, 2017, at 4:37 AM, Daniel Kidger >> >> >> wrote: >> >> >> >> On the subject of using zvols for software Raid/ replication, can ask >> >> as a quick poll, how many people are doing this? >> >> >> >> And any feedback on stability, tuning and performance? >> >> >> >> Daniel >> >> IBM Technical Presales >> >> >> >> > On 10 Mar 2017, at 22:44, Aaron Knister >> >> >> wrote: >> >> > >> >> > Those look like zvol's. Out of curiosity have you set sync=always on the >> >> > filesystem root or zvols themselves? It's my understanding that without >> >> > that you risk data loss since GPFS won't ever cause a sync to be issued >> >> > to the zvol for zfs to flush acknowledged but uncommitted writes. >> >> > >> >> > -Aaron >> >> > >> >> >> On 3/10/17 4:36 PM, Sanchez, Paul wrote: >> >> >> See: >> >> >> https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm >> >> >> >> >> >> >> >> >> >> >> >> GPFS has a limited set of device search specs it uses to find connected >> >> >> NSDs. When using exotic devices, you need to whitelist the devices >> >> >> yourself using the user exit script at /var/mmfs/etc/nsddevices. >> >> >> >> >> >> >> >> >> >> >> >> *From:*gpfsug-discuss-bounces at spectrumscale.org >> >> >> > > >> >> >> [mailto:gpfsug-discuss-bounces at spectrumscale.org >> ] *On Behalf Of >> >> >> *Buterbaugh, Kevin L >> >> >> *Sent:* Friday, March 10, 2017 3:44 PM >> >> >> *To:* gpfsug main discussion list >> >> >> *Subject:* [gpfsug-discuss] mmcrfs issue >> >> >> >> >> >> >> >> >> >> >> >> Hi All, >> >> >> >> >> >> >> >> >> >> >> >> We are testing out some flash storage. I created a couple of NSDs >> >> >> successfully (?): >> >> >> >> >> >> >> >> >> >> >> >> root at nec:~/gpfs# mmlsnsd -F >> >> >> >> >> >> >> >> >> >> >> >> File system Disk name NSD servers >> >> >> >> >> >> --------------------------------------------------------------------------- >> >> >> >> >> >> (free disk) nsd1 nec >> >> >> >> >> >> (free disk) nsd2 nec >> >> >> >> >> >> >> >> >> >> >> >> root at nec:~/gpfs# >> >> >> >> >> >> >> >> >> >> >> >> So I tried to create a filesystem: >> >> >> >> >> >> >> >> >> >> >> >> root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j >> >> >> scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 >> >> >> >> >> >> GPFS: 6027-441 Unable to open disk 'nsd2' on node nec. >> >> >> >> >> >> GPFS: 6027-441 Unable to open disk 'nsd1' on node nec. >> >> >> >> >> >> No such device >> >> >> >> >> >> No such device >> >> >> >> >> >> GPFS: 6027-538 Error accessing disks. >> >> >> >> >> >> mmcrfs: 6027-1200 tscrfs failed. Cannot create gpfs0 >> >> >> >> >> >> mmcrfs: 6027-1639 Command failed. Examine previous error messages to >> >> >> determine cause. >> >> >> >> >> >> root at nec:~/gpfs# >> >> >> >> >> >> >> >> >> >> >> >> Does this output from readdescraw look normal? >> >> >> >> >> >> >> >> >> >> >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd16 >> >> >> >> >> >> NSD descriptor in sector 64 of /dev/zd16 >> >> >> >> >> >> NSDid: 0A0023D258C1C02C format version: 1403 Label: >> >> >> >> >> >> Paxos sector: -1931478434 number of sectors: 8192 isPdisk: 0 >> >> >> >> >> >> Comment: NSD descriptor for Thu Mar 9 14:50:52 2017 >> >> >> >> >> >> No Disk descriptor in sector 96 of /dev/zd16 >> >> >> >> >> >> No FS descriptor in sector 2048 of /dev/zd16 >> >> >> >> >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd32 >> >> >> >> >> >> NSD descriptor in sector 64 of /dev/zd32 >> >> >> >> >> >> NSDid: 0A0023D258C1C02B format version: 1403 Label: >> >> >> >> >> >> Paxos sector: -1880562609 number of sectors: 8192 isPdisk: 0 >> >> >> >> >> >> Comment: NSD descriptor for Thu Mar 9 14:50:51 2017 >> >> >> >> >> >> No Disk descriptor in sector 96 of /dev/zd32 >> >> >> >> >> >> No FS descriptor in sector 2048 of /dev/zd32 >> >> >> >> >> >> root at nec:~/gpfs# >> >> >> >> >> >> >> >> >> >> >> >> Thanks in advance, all? >> >> >> >> >> >> >> >> >> >> >> >> Kevin >> >> >> >> >> >> ? >> >> >> >> >> >> Kevin Buterbaugh - Senior System Administrator >> >> >> >> >> >> Vanderbilt University - Advanced Computing Center for Research and Education >> >> >> >> >> >> Kevin.Buterbaugh at vanderbilt.edu >> >> > > >> >> >> > > - (615)875-9633 >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> gpfsug-discuss mailing list >> >> >> gpfsug-discuss at spectrumscale.org >> >> >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> >> >> > >> >> > -- >> >> > Aaron Knister >> >> > NASA Center for Climate Simulation (Code 606.2) >> >> > Goddard Space Flight Center >> >> > (301) 286-2776 >> >> > _______________________________________________ >> >> > gpfsug-discuss mailing list >> >> > gpfsug-discuss at spectrumscale.org >> >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> > >> >> Unless stated otherwise above: >> >> IBM United Kingdom Limited - Registered in England and Wales with >> >> number 741598. >> >> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >> >> >> >> _______________________________________________ >> >> gpfsug-discuss mailing list >> >> gpfsug-discuss at spectrumscale.org >> >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > >> > >> > >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > >> >> -- >> Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) >> Goddard Space Flight Center >> (301) 286-2776 >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Robert.Oesterlin at nuance.com Wed Mar 15 14:27:20 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Wed, 15 Mar 2017 14:27:20 +0000 Subject: [gpfsug-discuss] Running multiple mmrestripefs in a single cluster? Message-ID: I?m looking at migrating multiple file systems from one set of NSDs to another. Assuming I put aside any potential IO bottlenecks, has anyone tried running multiple ?mmrestripefs? commands in a single cluster? Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Wed Mar 15 14:53:44 2017 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Wed, 15 Mar 2017 09:53:44 -0500 Subject: [gpfsug-discuss] default inode size - remarks In-Reply-To: <20170315132609.wjplicnvhizlwc3q@ics.muni.cz> References: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz><0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org><20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> <20170315132609.wjplicnvhizlwc3q@ics.muni.cz> Message-ID: You can set inode size to any one of 512, 1024, 2048 or 4096 when you mmcrfs. Try it on a test system. You want all the metadata of a file to fit into the inode. Also small files, and small directories, can be stored in their inodes. On a test file system you can examine how much of an inode is occupied with which data and metadata using the inode subcommand of the tsdbfs command. (Root only, and usual cautions -- use only to look, if you make a mistake patching - I told you not to do that!) inode number of any path is easily discovered with the standard ls -ild pathname command -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Wed Mar 15 20:03:35 2017 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Wed, 15 Mar 2017 21:03:35 +0100 Subject: [gpfsug-discuss] Running multiple mmrestripefs in a single cluster? In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Wed Mar 15 20:33:19 2017 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Wed, 15 Mar 2017 15:33:19 -0500 Subject: [gpfsug-discuss] Running multiple mmrestripefs in a singlecluster? In-Reply-To: References: Message-ID: You can control the load of mmrestripefs (and all maintenance commands) on your system using mmchqos ... From: "Olaf Weiser" To: gpfsug main discussion list Date: 03/15/2017 04:04 PM Subject: Re: [gpfsug-discuss] Running multiple mmrestripefs in a single cluster? Sent by: gpfsug-discuss-bounces at spectrumscale.org yes.. and please be carefully about the number of nodes , doing the job because of multiple PIT worker hammering against your data if you limit the restripe to 2 nodes (-N ......) of adjust the PITworker down to 8 or even 4 ... you can run multiple restripes.. without hurting the application workload to much ... but the final duration of your restripe then will be affected cheers From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 03/15/2017 03:27 PM Subject: [gpfsug-discuss] Running multiple mmrestripefs in a single cluster? Sent by: gpfsug-discuss-bounces at spectrumscale.org I?m looking at migrating multiple file systems from one set of NSDs to another. Assuming I put aside any potential IO bottlenecks, has anyone tried running multiple ?mmrestripefs? commands in a single cluster? Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From duersch at us.ibm.com Thu Mar 16 00:26:28 2017 From: duersch at us.ibm.com (Steve Duersch) Date: Wed, 15 Mar 2017 19:26:28 -0500 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 62, Issue 33 In-Reply-To: References: Message-ID: >>For me it's the protection against bitrot and added protection against silent data corruption GNR has this functionality. Right now that is available through ESS though. Not yet as software only. Steve Duersch Spectrum Scale 845-433-7902 IBM Poughkeepsie, New York gpfsug-discuss-bounces at spectrumscale.org wrote on 03/15/2017 10:25:59 AM: > > Message: 6 > Date: Wed, 15 Mar 2017 14:25:41 +0000 > From: "Buterbaugh, Kevin L" > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] mmcrfs issue > Message-ID: > Content-Type: text/plain; charset="utf-8" > > Hi All, > > Since I started this thread I guess I should chime in, too ? for us > it was simply that we were testing a device that did not have > hardware RAID controllers and we were wanting to implement something > roughly equivalent to RAID 6 LUNs. > > Kevin > > > On Mar 14, 2017, at 5:16 PM, Aaron Knister wrote: > > > > For me it's the protection against bitrot and added protection > against silent data corruption and in theory the write caching > offered by adding log devices that could help with small random > writes (although there are other problems with ZFS + synchronous > workloads that stop this from actually materializing). > > > > -Aaron > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.knister at gmail.com Thu Mar 16 00:52:58 2017 From: aaron.knister at gmail.com (Aaron Knister) Date: Wed, 15 Mar 2017 20:52:58 -0400 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 62, Issue 33 In-Reply-To: References: Message-ID: *drags out soapbox* Sorry in advance for the rant, this is one of my huge pet peeves :) There are some serious blockers for GNR adoption in my environment. It drives me up a wall that the only way to get end to end checksums in GPFS is with vendor hardware lock-in. I find it infuriating. Lustre can do this for free with ZFS. Historically it has also offered various other features too like eating your data so I guess it's a tradeoff ;-) I believe that either GNR should be available for any hardware that passes a validation suite or GPFS should support checksums on non-GNR NSDs either by leveraging T10-PI information or by checksumming blocks/subblocks and storing that somewhere. I opened an RFE for this and it was rejected and I was effectively told to go use GNR/ESS, but well... can't do GNR. But lets say I could run GNR on any hardware of my choosing after perhaps paying some modest licensing fee and passing a hardware validation test there's another blocker for me. Because GPFS doesn't support anything like an LNet router I'm fairly limited on the number of high speed verbs rdma fabrics I can connect GNR to. Furthermore even if I had enough PCIe slots the configuration may not be supported (e.g. a site with an OPA and an IB fabric that would like to use rdma verbs on both). There could even be a situation where a vendor of an HPC solution requires a specific OFED version for support purposes that's not the version running on the GNR nodes. If an NSD protocol router were available I could perhaps use ethernet as a common medium to work around this. I'd really like IBM to *do* something about this situation but I've not gotten any traction on it so far. -Aaron On Wed, Mar 15, 2017 at 8:26 PM, Steve Duersch wrote: > >>For me it's the protection against bitrot and added protection against > silent data corruption > GNR has this functionality. Right now that is available through ESS > though. Not yet as software only. > > Steve Duersch > Spectrum Scale > 845-433-7902 <(845)%20433-7902> > IBM Poughkeepsie, New York > > > > > gpfsug-discuss-bounces at spectrumscale.org wrote on 03/15/2017 10:25:59 AM: > > > > > > Message: 6 > > Date: Wed, 15 Mar 2017 14:25:41 +0000 > > From: "Buterbaugh, Kevin L" > > To: gpfsug main discussion list > > Subject: Re: [gpfsug-discuss] mmcrfs issue > > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > > > Hi All, > > > > Since I started this thread I guess I should chime in, too ? for us > > it was simply that we were testing a device that did not have > > hardware RAID controllers and we were wanting to implement something > > roughly equivalent to RAID 6 LUNs. > > > > Kevin > > > > > On Mar 14, 2017, at 5:16 PM, Aaron Knister > wrote: > > > > > > For me it's the protection against bitrot and added protection > > against silent data corruption and in theory the write caching > > offered by adding log devices that could help with small random > > writes (although there are other problems with ZFS + synchronous > > workloads that stop this from actually materializing). > > > > > > -Aaron > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan at buzzard.me.uk Thu Mar 16 11:19:30 2017 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Thu, 16 Mar 2017 11:19:30 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 62, Issue 33 In-Reply-To: References: Message-ID: <1489663170.17464.39.camel@buzzard.me.uk> On Wed, 2017-03-15 at 20:52 -0400, Aaron Knister wrote: > *drags out soapbox* > > > Sorry in advance for the rant, this is one of my huge pet peeves :) > > > > There are some serious blockers for GNR adoption in my environment. It > drives me up a wall that the only way to get end to end checksums in > GPFS is with vendor hardware lock-in. As I understand it that is a factually inaccurate statement. You are free to use a DIF/DIX solution which is part of the T10 SCSI specification and as such an open(ish) standard. That is if you pay a suitable amount you will get a copy of the standard and are free to implement it. It is arguably more open than ZFS that has licensing issues. Further as I understand it the DIF/DIX solution is actually better than the ZFS solution on a number of counts, that revolve around it being baked into the hardware. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From janfrode at tanso.net Thu Mar 16 13:50:48 2017 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Thu, 16 Mar 2017 14:50:48 +0100 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 62, Issue 33 In-Reply-To: References: Message-ID: Why would you need a NSD protocol router when the NSD servers can have a mix of infiniband and ethernet adapters? F.ex. 4x EDR + 2x 100GbE per io-node in an ESS should give you lots of bandwidth for your common ethernet medium. -jf On Thu, Mar 16, 2017 at 1:52 AM, Aaron Knister wrote: > *drags out soapbox* > > Sorry in advance for the rant, this is one of my huge pet peeves :) > > There are some serious blockers for GNR adoption in my environment. It > drives me up a wall that the only way to get end to end checksums in GPFS > is with vendor hardware lock-in. I find it infuriating. Lustre can do this > for free with ZFS. Historically it has also offered various other features > too like eating your data so I guess it's a tradeoff ;-) I believe that > either GNR should be available for any hardware that passes a validation > suite or GPFS should support checksums on non-GNR NSDs either by leveraging > T10-PI information or by checksumming blocks/subblocks and storing that > somewhere. I opened an RFE for this and it was rejected and I was > effectively told to go use GNR/ESS, but well... can't do GNR. > > But lets say I could run GNR on any hardware of my choosing after perhaps > paying some modest licensing fee and passing a hardware validation test > there's another blocker for me. Because GPFS doesn't support anything like > an LNet router I'm fairly limited on the number of high speed verbs rdma > fabrics I can connect GNR to. Furthermore even if I had enough PCIe slots > the configuration may not be supported (e.g. a site with an OPA and an IB > fabric that would like to use rdma verbs on both). There could even be a > situation where a vendor of an HPC solution requires a specific OFED > version for support purposes that's not the version running on the GNR > nodes. If an NSD protocol router were available I could perhaps use > ethernet as a common medium to work around this. > > I'd really like IBM to *do* something about this situation but I've not > gotten any traction on it so far. > > -Aaron > > > > On Wed, Mar 15, 2017 at 8:26 PM, Steve Duersch wrote: > >> >>For me it's the protection against bitrot and added protection against >> silent data corruption >> GNR has this functionality. Right now that is available through ESS >> though. Not yet as software only. >> >> Steve Duersch >> Spectrum Scale >> 845-433-7902 <(845)%20433-7902> >> IBM Poughkeepsie, New York >> >> >> >> >> gpfsug-discuss-bounces at spectrumscale.org wrote on 03/15/2017 10:25:59 AM: >> >> >> > >> > Message: 6 >> > Date: Wed, 15 Mar 2017 14:25:41 +0000 >> > From: "Buterbaugh, Kevin L" >> > To: gpfsug main discussion list >> > Subject: Re: [gpfsug-discuss] mmcrfs issue >> > Message-ID: >> > Content-Type: text/plain; charset="utf-8" >> > >> > Hi All, >> > >> > Since I started this thread I guess I should chime in, too ? for us >> > it was simply that we were testing a device that did not have >> > hardware RAID controllers and we were wanting to implement something >> > roughly equivalent to RAID 6 LUNs. >> > >> > Kevin >> > >> > > On Mar 14, 2017, at 5:16 PM, Aaron Knister >> wrote: >> > > >> > > For me it's the protection against bitrot and added protection >> > against silent data corruption and in theory the write caching >> > offered by adding log devices that could help with small random >> > writes (although there are other problems with ZFS + synchronous >> > workloads that stop this from actually materializing). >> > > >> > > -Aaron >> > > >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.knister at gmail.com Thu Mar 16 14:39:13 2017 From: aaron.knister at gmail.com (Aaron Knister) Date: Thu, 16 Mar 2017 10:39:13 -0400 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 62, Issue 33 In-Reply-To: <1489663170.17464.39.camel@buzzard.me.uk> References: <1489663170.17464.39.camel@buzzard.me.uk> Message-ID: In all seriousness, I'd love to be wrong about this. I'm not sure which part(s) of what I said was inaccurate-- the vendor lock in and/or that GNR is the only way to get end to end checksums. When I say end to end checksums I mean that from the moment an FS write is submitted to mmfsd a checksum is calculated that is passed through every layer (memory, network, sas, fibre channel etc.) down to individual disk drives (understanding that the RAID controller may need to derive the checksum based on whatever RAID algorithm it's using). It's my understanding that the only way to achieve this with GPFS today is with GNR which is only available via purchasing a GSS or ESS solution from IBM/Lenovo. One is of course free to by hardware from any vendor but you don't get GNR. I should really have said "if you want GNR you have to buy hardware from IBM or Lenovo" which to me is being locked in to a vendor as long as you'd like end to end checksums. If there's another way to get end-to-end checksums, could you help me understand how? Regarding DIF/DIX, it's my understanding that I can/could use T10-DIF today (with the correct hardware) without purchasing any standards which would checksum data from the HBA to the RAID controller (and in theory disks). However, in an environment with NSD servers the origin of a read/write could in theory be quite a bit further away from the HBA in terms of hops. T10-DIF would be completely separate from GPFS as I understand it. I'm not aware of any integration (T10-DIF + DIX). What I'm really looking for, I think, is T10-DIF + DIX where the application (GPFS) generates protection information that's then passed along to the layers below it. -Aaron On Thu, Mar 16, 2017 at 7:19 AM, Jonathan Buzzard wrote: > On Wed, 2017-03-15 at 20:52 -0400, Aaron Knister wrote: > > *drags out soapbox* > > > > > > Sorry in advance for the rant, this is one of my huge pet peeves :) > > > > > > > > There are some serious blockers for GNR adoption in my environment. It > > drives me up a wall that the only way to get end to end checksums in > > GPFS is with vendor hardware lock-in. > > As I understand it that is a factually inaccurate statement. > > You are free to use a DIF/DIX solution which is part of the T10 SCSI > specification and as such an open(ish) standard. That is if you pay a > suitable amount you will get a copy of the standard and are free to > implement it. It is arguably more open than ZFS that has licensing > issues. > > Further as I understand it the DIF/DIX solution is actually better than > the ZFS solution on a number of counts, that revolve around it being > baked into the hardware. > > JAB. > > -- > Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk > Fife, United Kingdom. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Thu Mar 16 14:43:47 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Thu, 16 Mar 2017 10:43:47 -0400 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 62, Issue 33 In-Reply-To: References: Message-ID: <26bc0188-b5de-a128-9122-e6ed145aba69@nasa.gov> Perhaps an environment where one has OPA and IB fabrics. Taken from here (https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html): RDMA is not supported on a node when both Mellanox HCAs and Intel Omni-Path HFIs are enabled for RDMA. The alternative being a situation where multiple IB fabrics exist that require different OFED versions from each other (and most likely from ESS) for support reasons (speaking from experience). That is to say if $VENDOR supports OFED version X on an IB fabric, and ESS/GSS ships with version Y and there's a problem on the IB fabric $VENDOR may point at the different OFED version on the ESS/GSS and say they don't support it and then one is in a bad spot. -Aaron On 3/16/17 9:50 AM, Jan-Frode Myklebust wrote: > Why would you need a NSD protocol router when the NSD servers can have a > mix of infiniband and ethernet adapters? F.ex. 4x EDR + 2x 100GbE per > io-node in an ESS should give you lots of bandwidth for your common > ethernet medium. > > > -jf > > On Thu, Mar 16, 2017 at 1:52 AM, Aaron Knister > wrote: > > *drags out soapbox* > > Sorry in advance for the rant, this is one of my huge pet peeves :) > > There are some serious blockers for GNR adoption in my environment. > It drives me up a wall that the only way to get end to end checksums > in GPFS is with vendor hardware lock-in. I find it infuriating. > Lustre can do this for free with ZFS. Historically it has also > offered various other features too like eating your data so I guess > it's a tradeoff ;-) I believe that either GNR should be available > for any hardware that passes a validation suite or GPFS should > support checksums on non-GNR NSDs either by leveraging T10-PI > information or by checksumming blocks/subblocks and storing that > somewhere. I opened an RFE for this and it was rejected and I was > effectively told to go use GNR/ESS, but well... can't do GNR. > > But lets say I could run GNR on any hardware of my choosing after > perhaps paying some modest licensing fee and passing a hardware > validation test there's another blocker for me. Because GPFS doesn't > support anything like an LNet router I'm fairly limited on the > number of high speed verbs rdma fabrics I can connect GNR to. > Furthermore even if I had enough PCIe slots the configuration may > not be supported (e.g. a site with an OPA and an IB fabric that > would like to use rdma verbs on both). There could even be a > situation where a vendor of an HPC solution requires a specific OFED > version for support purposes that's not the version running on the > GNR nodes. If an NSD protocol router were available I could perhaps > use ethernet as a common medium to work around this. > > I'd really like IBM to *do* something about this situation but I've > not gotten any traction on it so far. > > -Aaron > > > > On Wed, Mar 15, 2017 at 8:26 PM, Steve Duersch > wrote: > > >>For me it's the protection against bitrot and added protection > against silent data corruption > GNR has this functionality. Right now that is available through > ESS though. Not yet as software only. > > Steve Duersch > Spectrum Scale > 845-433-7902 > IBM Poughkeepsie, New York > > > > > gpfsug-discuss-bounces at spectrumscale.org > wrote on > 03/15/2017 10:25:59 AM: > > > > > > Message: 6 > > Date: Wed, 15 Mar 2017 14:25:41 +0000 > > From: "Buterbaugh, Kevin L" > > To: gpfsug main discussion list > > > Subject: Re: [gpfsug-discuss] mmcrfs issue > > Message-ID: > > > Content-Type: text/plain; charset="utf-8" > > > > Hi All, > > > > Since I started this thread I guess I should chime in, too ? for us > > it was simply that we were testing a device that did not have > > hardware RAID controllers and we were wanting to implement something > > roughly equivalent to RAID 6 LUNs. > > > > Kevin > > > > > On Mar 14, 2017, at 5:16 PM, Aaron Knister > wrote: > > > > > > For me it's the protection against bitrot and added protection > > against silent data corruption and in theory the write caching > > offered by adding log devices that could help with small random > > writes (although there are other problems with ZFS + synchronous > > workloads that stop this from actually materializing). > > > > > > -Aaron > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From jonathan at buzzard.me.uk Thu Mar 16 16:54:32 2017 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Thu, 16 Mar 2017 16:54:32 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 62, Issue 33 In-Reply-To: References: <1489663170.17464.39.camel@buzzard.me.uk> Message-ID: <1489683272.17464.67.camel@buzzard.me.uk> On Thu, 2017-03-16 at 10:39 -0400, Aaron Knister wrote: > In all seriousness, I'd love to be wrong about this. I'm not sure > which part(s) of what I said was inaccurate-- the vendor lock in > and/or that GNR is the only way to get end to end checksums. The end to end checksums, or at least the ability to protect against silent corruption by having the data checksummed at all stages. > > When I say end to end checksums I mean that from the moment an FS > write is submitted to mmfsd a checksum is calculated that is passed > through every layer (memory, network, sas, fibre channel etc.) down to > individual disk drives (understanding that the RAID controller may > need to derive the checksum based on whatever RAID algorithm it's > using). It's my understanding that the only way to achieve this with > GPFS today is with GNR which is only available via purchasing a GSS or > ESS solution from IBM/Lenovo. One is of course free to by hardware > from any vendor but you don't get GNR. I should really have said "if > you want GNR you have to buy hardware from IBM or Lenovo" which to me > is being locked in to a vendor as long as you'd like end to end > checksums. > > If there's another way to get end-to-end checksums, could you help me > understand how? > Bear in mind the purpose of the checksums is to ensure the data is not corrupted. Noting you don't get true end to end because the checksums are never exposed to the application even in ZFS, at least as I understand it; you just get a read failure. > Regarding DIF/DIX, it's my understanding that I can/could use T10-DIF > today (with the correct hardware) without purchasing any standards > which would checksum data from the HBA to the RAID controller (and in > theory disks). However, in an environment with NSD servers the origin > of a read/write could in theory be quite a bit further away from the > HBA in terms of hops. T10-DIF would be completely separate from GPFS > as I understand it. I'm not aware of any integration (T10-DIF + DIX). > What I'm really looking for, I think, is T10-DIF + DIX where the > application (GPFS) generates protection information that's then passed > along to the layers below it. > Well yes an end user does not need to purchase a copy of the standard. However some people take the view that as you need to spend money to get the standard it is not truly open, so I was just making that clear. Right, so bearing in mind that the purpose of the checksums is to ensure data is not corrupted if the NSD servers are using DIF/DIX, then the options for silent data corruption are very limited if you are using ECC memory and you are using standard TCP/IP to communicate between the nodes. That is the TCP/IP checksums will protect your data between the clients and the NSD nodes, and once at the NSD node the DIF/DIX will protect your data as it makes it way to the disk. In between all that the ECC memory in your machines will protect everything once in RAM, and the PCIe bus has a 32bit CRC protecting everything that moves over the PCIe bus. The only "hole" in this is as far as I am aware is that the CPU itself, because at least x86 ones (as far as I am aware) do not "checksum" themselves as they do calculations, but you have the same problem with ZFS for exactly the same reason. So the point is that what you should be interested in as an admin; ensuring your data is protected against silent corruption, is achievable without hardware vendor lock in using open standards. The advantage of DIF/DIX over ZFS is as I understand it unless you do an immediate read on written blocks with ZFS and take a large performance penalty on writes you have no idea if your data has been corrupted until you try and read it back which could be months if not years down the line. Where DIF/DIX should try and fix it immediately by a retry and if that does not work pass the failure back up the line immediately. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From jonathan at buzzard.me.uk Thu Mar 16 17:18:22 2017 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Thu, 16 Mar 2017 17:18:22 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 62, Issue 33 In-Reply-To: <26bc0188-b5de-a128-9122-e6ed145aba69@nasa.gov> References: <26bc0188-b5de-a128-9122-e6ed145aba69@nasa.gov> Message-ID: <1489684702.17464.84.camel@buzzard.me.uk> On Thu, 2017-03-16 at 10:43 -0400, Aaron Knister wrote: > Perhaps an environment where one has OPA and IB fabrics. Taken from here > (https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html): > > RDMA is not supported on a node when both Mellanox HCAs and Intel > Omni-Path HFIs are enabled for RDMA. > > The alternative being a situation where multiple IB fabrics exist that > require different OFED versions from each other (and most likely from > ESS) for support reasons (speaking from experience). That is to say if > $VENDOR supports OFED version X on an IB fabric, and ESS/GSS ships with > version Y and there's a problem on the IB fabric $VENDOR may point at > the different OFED version on the ESS/GSS and say they don't support it > and then one is in a bad spot. > Or just use Ethernet for the GPFS traffic everywhere. It's 2017, 10GbE to compute nodes is cheap enough to be the norm, and you can use 40GbE and 100GbE everywhere else as required, and note unless you are a pure SSD system (which must be vanishingly rare at the moment) the latency is all in the disks anyway. I can't imagine why you would use IB and/or Omni-Path on anything other than compute clusters, so using Ethernet for your storage has advantages too especially if you are using IB. One of the features of Omni-Path over IB is that it will prioritize MPI traffic over storage, but given current core counts in CPU's separating storage out onto Ethernet is not a bad thing anyway as it keeps the MPI bandwidth per core up. My feeling is that in a compute cluster 10Gb for storage is more than enough for a node because much more and it would only take a handful of nodes to denial of service your storage, which is not good. A 500 node compute cluster with 10GbE for storage can in theory try and hit your storage for 600GB/s which very very few storage systems will be able to keep up with. On a storage GPFS cluster then you can up to 40/100GbE and put in multiple links for redundancy which is not possible with IB/Omni-path as far as I am aware which makes it a better solution. Even in a compute cluster with Ethernet I can stick in multiple connections for my NSD nodes for redundancy which is an improvement over the IB/Omni-path options. Especially given in my experience IB links go wonky orders of magnitude more often than Ethernet ones do. Loose a link to a compute node I loose one job on the cluster. Loose a link to the NSD server and I face loosing all the jobs on the cluster... JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From Robert.Oesterlin at nuance.com Thu Mar 16 18:43:59 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Thu, 16 Mar 2017 18:43:59 +0000 Subject: [gpfsug-discuss] Registration Reminder: GPFSUG-USA Meeting at NERSC April 4-5th Message-ID: Here?s a reminder to register for the upcoming user group meeting! Registration deadline extended to March 29th! Registration is filling up ? don?t wait. We Still have slots (20 mins each) for customer and partner talks ? please contact me if you want to do a short presentation. Register here: https://www.nersc.gov/research-and-development/data-analytics/spectrum-user-group-meeting/ Bob Oesterlin Sr Principal Storage Engineer, Nuance -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Fri Mar 17 18:50:11 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Fri, 17 Mar 2017 18:50:11 +0000 Subject: [gpfsug-discuss] mmnetverify Message-ID: Hi all, Just wondering if anyone has used the mmnetverify tool at all? Having made some changes to our internal L3 routing this week, I was interested to see what it claimed. As a side-note, it picked up some DNS resolution issues, though I'm not clear on some of those why it was claiming this as doing a "dig" on the node, it resolved fine (but adding the NSD servers to the hosts files cleared the error). Its actually the bandwidth tests that I'm interested in hearing other people's experience with as the numbers that some out from it are very different (lower) than if we use iperf to test performance between two nodes. Anyone any thoughts at all on this? Thanks Simon From Paul.Sanchez at deshaw.com Fri Mar 17 19:43:20 2017 From: Paul.Sanchez at deshaw.com (Sanchez, Paul) Date: Fri, 17 Mar 2017 19:43:20 +0000 Subject: [gpfsug-discuss] mmnetverify In-Reply-To: References: Message-ID: <18729d8a86fc430fac09c1c3a22159d2@mbxtoa1.winmail.deshaw.com> Sven will tell you: "RPC isn't streaming" and that may account for the discrepancy. If the tests are doing any "fan-in" where multiple nodes are sending to single node, then it's also possible that you are exhausting switch buffer memory in a way that a 1:1 iperf wouldn't. For our internal benchmarking we've used /usr/lpp/mmfs/samples/net/nsdperf to more closely estimate the real performance. I haven't played with mmnetverify yet though. -Paul -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (Research Computing - IT Services) Sent: Friday, March 17, 2017 2:50 PM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] mmnetverify Hi all, Just wondering if anyone has used the mmnetverify tool at all? Having made some changes to our internal L3 routing this week, I was interested to see what it claimed. As a side-note, it picked up some DNS resolution issues, though I'm not clear on some of those why it was claiming this as doing a "dig" on the node, it resolved fine (but adding the NSD servers to the hosts files cleared the error). Its actually the bandwidth tests that I'm interested in hearing other people's experience with as the numbers that some out from it are very different (lower) than if we use iperf to test performance between two nodes. Anyone any thoughts at all on this? Thanks Simon _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From S.J.Thompson at bham.ac.uk Fri Mar 17 20:13:22 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Fri, 17 Mar 2017 20:13:22 +0000 Subject: [gpfsug-discuss] mmnetverify In-Reply-To: <18729d8a86fc430fac09c1c3a22159d2@mbxtoa1.winmail.deshaw.com> References: , <18729d8a86fc430fac09c1c3a22159d2@mbxtoa1.winmail.deshaw.com> Message-ID: It looks to run sequential tests to each node one at a time and isn't using NSD protocol but echo server. We found some weird looking numbers that i don't quite understand and not in the places we might expect. For example between hosts on the same switch, traffic flowing to another switch and traffic flowing to nodes in another data centre where it's several switch hops. Some nodes over there were significantly faster than switch local nodes. I think it was only added in 4.2.2 and is listed as "not yet a replacement for nsdperf". I get that is different as it's using NSD protocol, but was struggling a bit with what mmnetverify might be doing. Simon From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Sanchez, Paul [Paul.Sanchez at deshaw.com] Sent: 17 March 2017 19:43 To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] mmnetverify Sven will tell you: "RPC isn't streaming" and that may account for the discrepancy. If the tests are doing any "fan-in" where multiple nodes are sending to single node, then it's also possible that you are exhausting switch buffer memory in a way that a 1:1 iperf wouldn't. For our internal benchmarking we've used /usr/lpp/mmfs/samples/net/nsdperf to more closely estimate the real performance. I haven't played with mmnetverify yet though. -Paul -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (Research Computing - IT Services) Sent: Friday, March 17, 2017 2:50 PM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] mmnetverify Hi all, Just wondering if anyone has used the mmnetverify tool at all? Having made some changes to our internal L3 routing this week, I was interested to see what it claimed. As a side-note, it picked up some DNS resolution issues, though I'm not clear on some of those why it was claiming this as doing a "dig" on the node, it resolved fine (but adding the NSD servers to the hosts files cleared the error). Its actually the bandwidth tests that I'm interested in hearing other people's experience with as the numbers that some out from it are very different (lower) than if we use iperf to test performance between two nodes. Anyone any thoughts at all on this? Thanks Simon _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Robert.Oesterlin at nuance.com Mon Mar 20 13:07:43 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 20 Mar 2017 13:07:43 +0000 Subject: [gpfsug-discuss] SSUG-USA: Registration Reminder and Agenda Update Message-ID: Only two weeks until the user group meeting! Registration is filling up fast ? Deadline is March 29th. We Still have 2 slots (20 mins each) for customer and partner talks ? please contact me if you want to do a short presentation. Register here: https://www.nersc.gov/research-and-development/data-analytics/spectrum-user-group-meeting/ Duration Start End Title Speaker Tues April 4th 60 9:00 10:00 Registration and Coffee n/a 30 10:00 10:30 Welcome Bob Oesterlin, Kristy Kallback, Doris Conti 60 10:30 11:30 Spectrum Scale & ESS Update Ulf Troppens / Scott Fadden 60 11:30 12:30 Problem Avoidance - Best Practices in the Field Brian Yaeger 60 12:30 13:30 Lunch and Networking n/a 120 13:30 15:30 Problem Determination - Deep Dive Yuri Volobuev 30 15:30 16:00 Coffee and Networking n/a 60 16:00 17:00 Problem Determination - What's New? Matthias Dietz 60 17:00 18:00 Bring your favorite problem or performance issue. We will fix it now! All Wed April 5th 30 8:30 9:00 Coffee and Networking n/a 60 9:00 10:00 1) AFM - Introduction and Use Cases (Meet the Devs) Scott Fadden 2) Life Sciences and Next Generation Sequencing with Spectrum Scale (Solutions) Madhav Ponamgi 3) Cloud and Virtualization Discussion (BOF) John Lewars 60 10:00 11:00 1) Transparent Cloud Tiering - Introduction and Use Cases (Meet the Devs) Rob Basham 2) Spectrum Protect on Spectrum Scale (Solutions) Jason Basler 3) Open User BOF TBD 60 11:00 12:00 1) Spectrum Scale Security (Meet the Devs) Felipe Knop 2) Hadoop Integration and Support for HortonWorks HDP (Solutions) Ted Hoover 3) Open User BOF TBD 60 12:00 13:00 Lunch and Networking n/a 20 13:00 13:20 System Analytics/IO Profiling TBD Customer 20 13:20 13:40 Placeholder - Customer or Partner talk TBD Customer 20 13:40 14:00 Placeholder - Customer or Partner talk TBD Customer 20 14:00 14:20 REST API Ulf Troppens 30 14:20 14:50 Coffee and Networking n/a 30 14:50 15:20 Application Integration with Containers Dean Hildebrand 30 15:20 15:50 Challenges in Tracing Vasily Tarasov 10 15:50 16:00 Wrap-up Bob Oesterlin, Kristy Kallback, Doris Conti Bob Oesterlin Sr Principal Storage Engineer, Nuance -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.wonderley at vt.edu Mon Mar 20 14:02:56 2017 From: eric.wonderley at vt.edu (J. Eric Wonderley) Date: Mon, 20 Mar 2017 10:02:56 -0400 Subject: [gpfsug-discuss] snapshots & tiering in a busy filesystem Message-ID: I found this link and it didn't give me much hope for doing snapshots & backup in a home(busy) filesystem: http://www.spectrumscale.org/pipermail/gpfsug-discuss/2013- February/000200.html I realize this is dated and I wondered if qos etc have made is a tolerable thing to do now. Gpfs I think was barely above v3.5 in mid 2013. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Mon Mar 20 14:49:07 2017 From: bbanister at jumptrading.com (Bryan Banister) Date: Mon, 20 Mar 2017 14:49:07 +0000 Subject: [gpfsug-discuss] snapshots & tiering in a busy filesystem In-Reply-To: References: Message-ID: We run snapshots every 4 hours on a busy home file system without issues for many years now, -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of J. Eric Wonderley Sent: Monday, March 20, 2017 9:03 AM To: gpfsug main discussion list Subject: [gpfsug-discuss] snapshots & tiering in a busy filesystem I found this link and it didn't give me much hope for doing snapshots & backup in a home(busy) filesystem: http://www.spectrumscale.org/pipermail/gpfsug-discuss/2013-February/000200.html I realize this is dated and I wondered if qos etc have made is a tolerable thing to do now. Gpfs I think was barely above v3.5 in mid 2013. ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Mon Mar 20 14:53:51 2017 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 20 Mar 2017 09:53:51 -0500 Subject: [gpfsug-discuss] snapshots & tiering in a busy filesystem In-Reply-To: References: Message-ID: Yes. QOS (see mmchqos) is now the way (4.2.x) to control maintenance commands like snapshots, backup, restripe. From: "J. Eric Wonderley" To: gpfsug main discussion list Date: 03/20/2017 10:03 AM Subject: [gpfsug-discuss] snapshots & tiering in a busy filesystem Sent by: gpfsug-discuss-bounces at spectrumscale.org I found this link and it didn't give me much hope for doing snapshots & backup in a home(busy) filesystem: http://www.spectrumscale.org/pipermail/gpfsug-discuss/2013-February/000200.html I realize this is dated and I wondered if qos etc have made is a tolerable thing to do now. Gpfs I think was barely above v3.5 in mid 2013. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.g.richmond at leeds.ac.uk Tue Mar 21 10:56:41 2017 From: a.g.richmond at leeds.ac.uk (Aidan Richmond) Date: Tue, 21 Mar 2017 10:56:41 +0000 Subject: [gpfsug-discuss] Configuration spectrum scale to use SFU schema Message-ID: <8d1622e5-99dc-dad4-998b-b0d5a88c8f70@leeds.ac.uk> Hello Is it possible to configure authentication services to bind to AD using the older SFU schema for ID mappings? We have standard samba and nfs servers that use this ID mapping scheme with "idmap config DS:schema_mode = sfu" in the samba configuration files, but if its possible using the mmuserauth command it doesn't seem to be documented. Thanks. -- Aidan Richmond Apple/Unix Support Officer, IT Garstang 10.137 Faculty of Biological Sciences University of Leeds Clarendon Way LS2 9JT Tel:0113 3434252 a.g.richmond at leeds.ac.uk From chair at spectrumscale.org Tue Mar 21 14:00:50 2017 From: chair at spectrumscale.org (Spectrum Scale UG Chair (Simon Thompson)) Date: Tue, 21 Mar 2017 14:00:50 +0000 Subject: [gpfsug-discuss] Sydney April Spectrum Scale UG Meeting In-Reply-To: References: Message-ID: Hi All, If you are located in the Australia region, you may be interested in the Spectrum Scale User Group meeting being hosted in Sydney, Australia: >We have a good part of the day for a Meet the Developers team with Kedar >Karmarkar form India and his team. >Plus Mellanox, Pawsey Supercomputing Centre, and a lessons learn from DDN >on IB networking at the Australian Bureau of Meteorology (subject to >disclosure agreement). Agenda and registration are at: >https://www.eventbrite.com/e/spectrum-scale-user-group-australia-gpfsugaus >-april-2017-tickets-29136246297 There's also a slack channel for discussion at: >https://ibmau-spectrumscale.slack.com/messages/C3C10AC9H/details/ This is being organised by Chris Schlipalius from Pawsey Supercomputing Centre, please contact Chris directly for more details (see link on Eventbrite page). Simon UK Group Chair From christof.schmitt at us.ibm.com Tue Mar 21 17:24:03 2017 From: christof.schmitt at us.ibm.com (Christof Schmitt) Date: Tue, 21 Mar 2017 10:24:03 -0700 Subject: [gpfsug-discuss] Configuration spectrum scale to use SFU schema In-Reply-To: <8d1622e5-99dc-dad4-998b-b0d5a88c8f70@leeds.ac.uk> References: <8d1622e5-99dc-dad4-998b-b0d5a88c8f70@leeds.ac.uk> Message-ID: gpfsug-discuss-bounces at spectrumscale.org wrote on 03/21/2017 03:56:41 AM: > Is it possible to configure authentication services to bind to AD using > the older SFU schema for ID mappings? We have standard samba and nfs > servers that use this ID mapping scheme with "idmap config > DS:schema_mode = sfu" in the samba configuration files, but if its > possible using the mmuserauth command it doesn't seem to be documented. This falls into the category that we use Samba to provide these services, but cannot test and support every possible configuration. The feature to read the old MS style id attributes has not been tested with Spectrum Scale and is not supported. That said, it can still be set in the internal config: /usr/lpp/mmfs/bin/net conf setparm global 'idmap config DS:schema_mode' sfu Activating this change probably would require a restart of the process using this config: /usr/lpp/mmfs/bin/mmdsh -N CesNodes systemctl restart gpfs-winbind Opening a PMR on this will likely result in the answer that this is unsupported. The best way to request official support would be opening a RFE, let others vote and then product management can decide on the priority of the request. Regards, Christof Schmitt || IBM || Spectrum Scale Development || Tucson, AZ christof.schmitt at us.ibm.com || +1-520-799-2469 (T/L: 321-2469) From TROPPENS at de.ibm.com Tue Mar 21 17:51:09 2017 From: TROPPENS at de.ibm.com (Ulf Troppens) Date: Tue, 21 Mar 2017 18:51:09 +0100 Subject: [gpfsug-discuss] Charts of German User Meeting now online Message-ID: Thanks a lot to all speaker. Event report will follow later. http://www.spectrumscale.org/presentations/ -- IBM Spectrum Scale Development - Client Engagements & Solutions Delivery Consulting IT Specialist Author "Storage Networks Explained" IBM Deutschland Research & Development GmbH Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.g.richmond at leeds.ac.uk Wed Mar 22 08:01:33 2017 From: a.g.richmond at leeds.ac.uk (Aidan Richmond) Date: Wed, 22 Mar 2017 08:01:33 +0000 Subject: [gpfsug-discuss] Configuration spectrum scale to use SFU schema In-Reply-To: References: <8d1622e5-99dc-dad4-998b-b0d5a88c8f70@leeds.ac.uk> Message-ID: <75a2acf9-e598-89c2-9546-6cedd80186c1@leeds.ac.uk> On 21/03/17 17:24, Christof Schmitt wrote: > gpfsug-discuss-bounces at spectrumscale.org wrote on 03/21/2017 03:56:41 AM: > >> Is it possible to configure authentication services to bind to AD using >> the older SFU schema for ID mappings? We have standard samba and nfs >> servers that use this ID mapping scheme with "idmap config >> DS:schema_mode = sfu" in the samba configuration files, but if its >> possible using the mmuserauth command it doesn't seem to be documented. > > This falls into the category that we use Samba to provide these services, > but cannot test and support > every possible configuration. The feature to read the old MS style id > attributes has not been tested > with Spectrum Scale and is not supported. That said, it can still be set > in the internal config: > > /usr/lpp/mmfs/bin/net conf setparm global 'idmap config DS:schema_mode' > sfu > > Activating this change probably would require a restart of the process > using this config: > > /usr/lpp/mmfs/bin/mmdsh -N CesNodes systemctl restart gpfs-winbind > > Opening a PMR on this will likely result in the answer that this is > unsupported. The best way > to request official support would be opening a RFE, let others vote and > then product > management can decide on the priority of the request. > > Regards, > > Christof Schmitt || IBM || Spectrum Scale Development || Tucson, AZ > christof.schmitt at us.ibm.com || +1-520-799-2469 (T/L: 321-2469) > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > Thanks, would I need to re-run mmuserauth as per https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.1/com.ibm.spectrum.scale.v4r21.doc/bl1adm_adwithrfc2307.htm or would the schema change be needed after that when configuring the authentication services? -- Aidan Richmond Apple/Unix Support Officer, IT Garstang 10.137 Faculty of Biological Sciences University of Leeds Clarendon Way LS2 9JT Tel:0113 3434252 a.g.richmond at leeds.ac.uk From r.sobey at imperial.ac.uk Wed Mar 22 10:47:37 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Wed, 22 Mar 2017 10:47:37 +0000 Subject: [gpfsug-discuss] snapshots & tiering in a busy filesystem In-Reply-To: References: Message-ID: We?re also snapshotting 4 times a day. Filesystem isn?t tremendously busy at all but we?re creating snaps for each fileset. [root at cesnode tmp]# mmlssnapshot gpfs | wc -l 6916 From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of J. Eric Wonderley Sent: 20 March 2017 14:03 To: gpfsug main discussion list Subject: [gpfsug-discuss] snapshots & tiering in a busy filesystem I found this link and it didn't give me much hope for doing snapshots & backup in a home(busy) filesystem: http://www.spectrumscale.org/pipermail/gpfsug-discuss/2013-February/000200.html I realize this is dated and I wondered if qos etc have made is a tolerable thing to do now. Gpfs I think was barely above v3.5 in mid 2013. -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.wonderley at vt.edu Wed Mar 22 13:41:26 2017 From: eric.wonderley at vt.edu (J. Eric Wonderley) Date: Wed, 22 Mar 2017 09:41:26 -0400 Subject: [gpfsug-discuss] snapshots & tiering in a busy filesystem In-Reply-To: References: Message-ID: The filesystem I'm working with has about 100M files and 80Tb of data. What kind of metadata latency do you observe? I did a mmdiag --iohist and filtered out all of the md devices and averaged over reads and writes. I'm seeing ~.28ms on a one off dump. The pure array which we have is 10G iscsi connected and is reporting average .25ms. On Wed, Mar 22, 2017 at 6:47 AM, Sobey, Richard A wrote: > We?re also snapshotting 4 times a day. Filesystem isn?t tremendously busy > at all but we?re creating snaps for each fileset. > > > > [root at cesnode tmp]# mmlssnapshot gpfs | wc -l > > 6916 > > > > *From:* gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss- > bounces at spectrumscale.org] *On Behalf Of *J. Eric Wonderley > *Sent:* 20 March 2017 14:03 > *To:* gpfsug main discussion list > *Subject:* [gpfsug-discuss] snapshots & tiering in a busy filesystem > > > > I found this link and it didn't give me much hope for doing snapshots & > backup in a home(busy) filesystem: > > http://www.spectrumscale.org/pipermail/gpfsug-discuss/2013- > February/000200.html > > I realize this is dated and I wondered if qos etc have made is a tolerable > thing to do now. Gpfs I think was barely above v3.5 in mid 2013. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mweil at wustl.edu Wed Mar 22 16:42:35 2017 From: mweil at wustl.edu (Matt Weil) Date: Wed, 22 Mar 2017 11:42:35 -0500 Subject: [gpfsug-discuss] CES node slow to respond Message-ID: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu> All, We had an indecent yesterday where one of our CES nodes slowed to a crawl. GPFS waiters showed pre fetch threads going after inodes. iohist also showed lots of inode fetching. Then we noticed that the CES host had 5.4 million files open. The change I made was to set maxStatCache=DEFAULT because it is linux. And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. Is there something else we should change as well. Thanks Matt ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. From bbanister at jumptrading.com Wed Mar 22 17:01:26 2017 From: bbanister at jumptrading.com (Bryan Banister) Date: Wed, 22 Mar 2017 17:01:26 +0000 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu> References: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu> Message-ID: <47e18bb0062544f4b510ce0e87049fd3@jumptrading.com> We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. -Bryan -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Wednesday, March 22, 2017 11:43 AM To: gpfsug main discussion list Subject: [gpfsug-discuss] CES node slow to respond All, We had an indecent yesterday where one of our CES nodes slowed to a crawl. GPFS waiters showed pre fetch threads going after inodes. iohist also showed lots of inode fetching. Then we noticed that the CES host had 5.4 million files open. The change I made was to set maxStatCache=DEFAULT because it is linux. And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. Is there something else we should change as well. Thanks Matt ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. From r.sobey at imperial.ac.uk Wed Mar 22 17:44:56 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Wed, 22 Mar 2017 17:44:56 +0000 Subject: [gpfsug-discuss] snapshots & tiering in a busy filesystem In-Reply-To: References: Message-ID: Embarrassingly I don?t know how to do that scientifically. I can tell you latency of the storage system which at a very simplistic snapshot is averaging around 2ms. We are not using SSD for metadata but 10K SAS. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of J. Eric Wonderley Sent: 22 March 2017 13:41 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] snapshots & tiering in a busy filesystem The filesystem I'm working with has about 100M files and 80Tb of data. What kind of metadata latency do you observe? I did a mmdiag --iohist and filtered out all of the md devices and averaged over reads and writes. I'm seeing ~.28ms on a one off dump. The pure array which we have is 10G iscsi connected and is reporting average .25ms. On Wed, Mar 22, 2017 at 6:47 AM, Sobey, Richard A > wrote: We?re also snapshotting 4 times a day. Filesystem isn?t tremendously busy at all but we?re creating snaps for each fileset. [root at cesnode tmp]# mmlssnapshot gpfs | wc -l 6916 From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of J. Eric Wonderley Sent: 20 March 2017 14:03 To: gpfsug main discussion list > Subject: [gpfsug-discuss] snapshots & tiering in a busy filesystem I found this link and it didn't give me much hope for doing snapshots & backup in a home(busy) filesystem: http://www.spectrumscale.org/pipermail/gpfsug-discuss/2013-February/000200.html I realize this is dated and I wondered if qos etc have made is a tolerable thing to do now. Gpfs I think was barely above v3.5 in mid 2013. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From valdis.kletnieks at vt.edu Wed Mar 22 21:21:42 2017 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Wed, 22 Mar 2017 17:21:42 -0400 Subject: [gpfsug-discuss] LTFS/EE release question.. Message-ID: <37386.1490217702@turing-police.cc.vt.edu> We're currently on ltfs/ee 1.2.1.1. IBM has recommended we upgrade to gpfs 4.2.2.3 to fix a performance issue. I can't seem to find a support matrix that says what ltfs release we need for 4.2.2. Anybody have info on that? -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 484 bytes Desc: not available URL: From christof.schmitt at us.ibm.com Wed Mar 22 22:41:04 2017 From: christof.schmitt at us.ibm.com (Christof Schmitt) Date: Wed, 22 Mar 2017 15:41:04 -0700 Subject: [gpfsug-discuss] Configuration spectrum scale to use SFU schema In-Reply-To: <75a2acf9-e598-89c2-9546-6cedd80186c1@leeds.ac.uk> References: <8d1622e5-99dc-dad4-998b-b0d5a88c8f70@leeds.ac.uk> <75a2acf9-e598-89c2-9546-6cedd80186c1@leeds.ac.uk> Message-ID: > Thanks, would I need to re-run mmuserauth as per > https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.1/ > com.ibm.spectrum.scale.v4r21.doc/bl1adm_adwithrfc2307.htm > or would the schema change be needed after that when configuring the > authentication services? You would still need the domain join and the basic configuration done by mmuserauth. The easiest way would be running mmuserauth with --unix-domains as usually. The only change on top is then the switch to the old attribute format, which requires a winbindd restart to pick-up the change: mmuserauth service create ??data?access?method file --type ad ... /usr/lpp/mmfs/bin/net conf setparm global 'idmap config DS:schema_mode' sfu /usr/lpp/mmfs/bin/mmdsh -N CesNodes systemctl restart gpfs-winbind This also illustrates the support statement: The config through mmuserauth is supported, changing the Samba config under the hood is outside the normal support. Regards, Christof Schmitt || IBM || Spectrum Scale Development || Tucson, AZ christof.schmitt at us.ibm.com || +1-520-799-2469 (T/L: 321-2469) From kevindjo at us.ibm.com Thu Mar 23 13:35:24 2017 From: kevindjo at us.ibm.com (Kevin D Johnson) Date: Thu, 23 Mar 2017 13:35:24 +0000 Subject: [gpfsug-discuss] LTFS/EE release question.. In-Reply-To: <37386.1490217702@turing-police.cc.vt.edu> References: <37386.1490217702@turing-police.cc.vt.edu> Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: att66cs2.dat Type: application/octet-stream Size: 495 bytes Desc: not available URL: From olaf.weiser at de.ibm.com Thu Mar 23 14:24:38 2017 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Thu, 23 Mar 2017 15:24:38 +0100 Subject: [gpfsug-discuss] CES doesn't assign addresses to nodes In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From jonathon.anderson at colorado.edu Thu Mar 23 15:02:10 2017 From: jonathon.anderson at colorado.edu (Jonathon A Anderson) Date: Thu, 23 Mar 2017 15:02:10 +0000 Subject: [gpfsug-discuss] CES doesn't assign addresses to nodes In-Reply-To: References: Message-ID: Thanks! I?m looking forward to upgrading our CES nodes and resuming work on the project. ~jonathon On 3/23/17, 8:24 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Olaf Weiser" wrote: the issue is fixed, an APAR will be released soon - IV93100 From: Olaf Weiser/Germany/IBM at IBMDE To: "gpfsug main discussion list" Cc: "gpfsug main discussion list" Date: 01/31/2017 11:47 PM Subject: Re: [gpfsug-discuss] CES doesn't assign addresses to nodes Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________________ Yeah... depending on the #nodes you 're affected or not. ..... So if your remote ces cluster is small enough in terms of the #nodes ... you'll neuer hit into this issue Gesendet von IBM Verse Simon Thompson (Research Computing - IT Services) --- Re: [gpfsug-discuss] CES doesn't assign addresses to nodes --- Von:"Simon Thompson (Research Computing - IT Services)" An:"gpfsug main discussion list" Datum:Di. 31.01.2017 21:07Betreff:Re: [gpfsug-discuss] CES doesn't assign addresses to nodes ________________________________________ We use multicluster for our environment, storage systems in a separate cluster to hpc nodes on a separate cluster from protocol nodes. According to the docs, this isn't supported, but we haven't seen any issues. Note unsupported as opposed to broken. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Jonathon A Anderson [jonathon.anderson at colorado.edu] Sent: 31 January 2017 17:47 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] CES doesn't assign addresses to nodes Yeah, I searched around for places where ` tsctl shownodes up` appears in the GPFS code I have access to (i.e., the ksh and python stuff); but it?s only in CES. I suspect there just haven?t been that many people exporting CES out of an HPC cluster environment. ~jonathon From: on behalf of Olaf Weiser Reply-To: gpfsug main discussion list Date: Tuesday, January 31, 2017 at 10:45 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] CES doesn't assign addresses to nodes I ll open a pmr here for my env ... the issue may hurt you in a ces env. only... but needs to be fixed in core gpfs.base i thi k Gesendet von IBM Verse Jonathon A Anderson --- Re: [gpfsug-discuss] CES doesn't assign addresses to nodes --- Von: "Jonathon A Anderson" An: "gpfsug main discussion list" Datum: Di. 31.01.2017 17:32 Betreff: Re: [gpfsug-discuss] CES doesn't assign addresses to nodes ________________________________ No, I?m having trouble getting this through DDN support because, while we have a GPFS server license and GRIDScaler support, apparently we don?t have ?protocol node? support, so they?ve pushed back on supporting this as an overall CES-rooted effort. I do have a DDN case open, though: 78804. If you are (as I suspect) a GPFS developer, do you mind if I cite your info from here in my DDN case to get them to open a PMR? Thanks. ~jonathon From: on behalf of Olaf Weiser Reply-To: gpfsug main discussion list Date: Tuesday, January 31, 2017 at 8:42 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] CES doesn't assign addresses to nodes ok.. so obviously ... it seems , that we have several issues.. the 3983 characters is obviously a defect have you already raised a PMR , if so , can you send me the number ? From: Jonathon A Anderson To: gpfsug main discussion list Date: 01/31/2017 04:14 PM Subject: Re: [gpfsug-discuss] CES doesn't assign addresses to nodes Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ The tail isn?t the issue; that? my addition, so that I didn?t have to paste the hundred or so line nodelist into the thread. The actual command is tsctl shownodes up | $tr ',' '\n' | $sort -o $upnodefile But you can see in my tailed output that the last hostname listed is cut-off halfway through the hostname. Less obvious in the example, but true, is the fact that it?s only showing the first 120 hosts, when we have 403 nodes in our gpfs cluster. [root at sgate2 ~]# tsctl shownodes up | tr ',' '\n' | wc -l 120 [root at sgate2 ~]# mmlscluster | grep '\-opa' | wc -l 403 Perhaps more explicitly, it looks like `tsctl shownodes up` can only transmit 3983 characters. [root at sgate2 ~]# tsctl shownodes up | wc -c 3983 Again, I?m convinced this is a bug not only because the command doesn?t actually produce a list of all of the up nodes in our cluster; but because the last name listed is incomplete. [root at sgate2 ~]# tsctl shownodes up | tr ',' '\n' | tail -n 1 shas0260-opa.rc.int.col[root at sgate2 ~]# I?d continue my investigation within tsctl itself but, alas, it?s a binary with no source code available to me. :) I?m trying to get this opened as a bug / PMR; but I?m still working through the DDN support infrastructure. Thanks for reporting it, though. For the record: [root at sgate2 ~]# rpm -qa | grep -i gpfs gpfs.base-4.2.1-2.x86_64 gpfs.msg.en_US-4.2.1-2.noarch gpfs.gplbin-3.10.0-327.el7.x86_64-4.2.1-0.x86_64 gpfs.gskit-8.0.50-57.x86_64 gpfs.gpl-4.2.1-2.noarch nfs-ganesha-gpfs-2.3.2-0.ibm24.el7.x86_64 gpfs.ext-4.2.1-2.x86_64 gpfs.gplbin-3.10.0-327.36.3.el7.x86_64-4.2.1-2.x86_64 gpfs.docs-4.2.1-2.noarch ~jonathon From: on behalf of Olaf Weiser Reply-To: gpfsug main discussion list Date: Tuesday, January 31, 2017 at 1:30 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] CES doesn't assign addresses to nodes Hi ...same thing here.. everything after 10 nodes will be truncated.. though I don't have an issue with it ... I 'll open a PMR .. and I recommend you to do the same thing.. ;-) the reason seems simple.. it is the "| tail" .at the end of the command.. .. which truncates the output to the last 10 items... should be easy to fix.. cheers olaf From: Jonathon A Anderson To: "gpfsug-discuss at spectrumscale.org" Date: 01/30/2017 11:11 PM Subject: Re: [gpfsug-discuss] CES doesn't assign addresses to nodes Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ In trying to figure this out on my own, I?m relatively certain I?ve found a bug in GPFS related to the truncation of output from `tsctl shownodes up`. Any chance someone in development can confirm? Here are the details of my investigation: ## GPFS is up on sgate2 [root at sgate2 ~]# mmgetstate Node number Node name GPFS state ------------------------------------------ 414 sgate2-opa active ## but if I tell ces to explicitly put one of our ces addresses on that node, it says that GPFS is down [root at sgate2 ~]# mmces address move --ces-ip 10.225.71.102 --ces-node sgate2-opa mmces address move: GPFS is down on this node. mmces address move: Command failed. Examine previous error messages to determine cause. ## the ?GPFS is down on this node? message is defined as code 109 in mmglobfuncs [root at sgate2 ~]# grep --before-context=1 "GPFS is down on this node." /usr/lpp/mmfs/bin/mmglobfuncs 109 ) msgTxt=\ "%s: GPFS is down on this node." ## and is generated by printErrorMsg in mmcesnetmvaddress when it detects that the current node is identified as ?down? by getDownCesNodeList [root at sgate2 ~]# grep --before-context=5 'printErrorMsg 109' /usr/lpp/mmfs/bin/mmcesnetmvaddress downNodeList=$(getDownCesNodeList) for downNode in $downNodeList do if [[ $toNodeName == $downNode ]] then printErrorMsg 109 "$mmcmd" ## getDownCesNodeList is the intersection of all ces nodes with GPFS cluster nodes listed in `tsctl shownodes up` [root at sgate2 ~]# grep --after-context=16 '^function getDownCesNodeList' /usr/lpp/mmfs/bin/mmcesfuncs function getDownCesNodeList { typeset sourceFile="mmcesfuncs.sh" [[ -n $DEBUG || -n $DEBUGgetDownCesNodeList ]] &&set -x $mmTRACE_ENTER "$*" typeset upnodefile=${cmdTmpDir}upnodefile typeset downNodeList # get all CES nodes $sort -o $nodefile $mmfsCesNodes.dae $tsctl shownodes up | $tr ',' '\n' | $sort -o $upnodefile downNodeList=$($comm -23 $nodefile $upnodefile) print -- $downNodeList } #----- end of function getDownCesNodeList -------------------- ## but not only are the sgate nodes not listed by `tsctl shownodes up`; its output is obviously and erroneously truncated [root at sgate2 ~]# tsctl shownodes up | tr ',' '\n' | tail shas0251-opa.rc.int.colorado.edu shas0252-opa.rc.int.colorado.edu shas0253-opa.rc.int.colorado.edu shas0254-opa.rc.int.colorado.edu shas0255-opa.rc.int.colorado.edu shas0256-opa.rc.int.colorado.edu shas0257-opa.rc.int.colorado.edu shas0258-opa.rc.int.colorado.edu shas0259-opa.rc.int.colorado.edu shas0260-opa.rc.int.col[root at sgate2 ~]# ## I expect that this is a bug in GPFS, likely related to a maximum output buffer for `tsctl shownodes up`. On 1/24/17, 12:48 PM, "Jonathon A Anderson" wrote: I think I'm having the same issue described here: http://www.spectrumscale.org/pipermail/gpfsug-discuss/2016-October/002288.html Any advice or further troubleshooting steps would be much appreciated. Full disclosure: I also have a DDN case open. (78804) We've got a four-node (snsd{1..4}) DDN gridscaler system. I'm trying to add two CES protocol nodes (sgate{1,2}) to serve NFS. Here's the steps I took: --- mmcrnodeclass protocol -N sgate1-opa,sgate2-opa mmcrnodeclass nfs -N sgate1-opa,sgate2-opa mmchconfig cesSharedRoot=/gpfs/summit/ces mmchcluster --ccr-enable mmchnode --ces-enable -N protocol mmces service enable NFS mmces service start NFS -N nfs mmces address add --ces-ip 10.225.71.104,10.225.71.105 mmces address policy even-coverage mmces address move --rebalance --- This worked the very first time I ran it, but the CES addresses weren't re-distributed after restarting GPFS or a node reboot. Things I've tried: * disabling ces on the sgate nodes and re-running the above procedure * moving the cluster and filesystem managers to different snsd nodes * deleting and re-creating the cesSharedRoot directory Meanwhile, the following log entry appears in mmfs.log.latest every ~30s: --- Mon Jan 23 20:31:20 MST 2017: mmcesnetworkmonitor: Found unassigned address 10.225.71.104 Mon Jan 23 20:31:20 MST 2017: mmcesnetworkmonitor: Found unassigned address 10.225.71.105 Mon Jan 23 20:31:20 MST 2017: mmcesnetworkmonitor: handleNetworkProblem with lock held: assignIP 10.225.71.104_0-_+,10.225.71.105_0-_+ 1 Mon Jan 23 20:31:20 MST 2017: mmcesnetworkmonitor: Assigning addresses: 10.225.71.104_0-_+,10.225.71.105_0-_+ Mon Jan 23 20:31:20 MST 2017: mmcesnetworkmonitor: moveCesIPs: 10.225.71.104_0-_+,10.225.71.105_0-_+ --- Also notable, whenever I add or remove addresses now, I see this in mmsysmonitor.log (among a lot of other entries): --- 2017-01-23T20:40:56.363 sgate1 D ET_cesnetwork Entity state without requireUnique: ces_network_ips_down WARNING No CES relevant NICs detected - Service.calculateAndUpdateState:275 2017-01-23T20:40:11.364 sgate1 D ET_cesnetwork Update multiple entities at once {'p2p2': 1, 'bond0': 1, 'p2p1': 1} - Service.setLocalState:333 --- For the record, here's the interface I expect to get the address on sgate1: --- 11: bond0: mtu 9000 qdisc noqueue state UP link/ether 3c:fd:fe:08:a7:c0 brd ff:ff:ff:ff:ff:ff inet 10.225.71.107/20 brd 10.225.79.255 scope global bond0 valid_lft forever preferred_lft forever inet6 fe80::3efd:feff:fe08:a7c0/64 scope link valid_lft forever preferred_lft forever --- which is a bond of p2p1 and p2p2. --- 6: p2p1: mtu 9000 qdisc mq master bond0 state UP qlen 1000 link/ether 3c:fd:fe:08:a7:c0 brd ff:ff:ff:ff:ff:ff 7: p2p2: mtu 9000 qdisc mq master bond0 state UP qlen 1000 link/ether 3c:fd:fe:08:a7:c0 brd ff:ff:ff:ff:ff:ff --- A similar bond0 exists on sgate2. I crawled around in /usr/lpp/mmfs/lib/mmsysmon/CESNetworkService.py for a while trying to figure it out, but have been unsuccessful so far. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From mweil at wustl.edu Thu Mar 23 15:23:56 2017 From: mweil at wustl.edu (Matt Weil) Date: Thu, 23 Mar 2017 10:23:56 -0500 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: <47e18bb0062544f4b510ce0e87049fd3@jumptrading.com> References: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu> <47e18bb0062544f4b510ce0e87049fd3@jumptrading.com> Message-ID: <643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> FYI all We also ran into this after bumping maxFilesToCache. Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: unexpected error: Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many open files in system fix sysctl -w fs.file-max=1000000 On 3/22/17 12:01 PM, Bryan Banister wrote: > We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. > > Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil > Sent: Wednesday, March 22, 2017 11:43 AM > To: gpfsug main discussion list > Subject: [gpfsug-discuss] CES node slow to respond > > All, > > We had an indecent yesterday where one of our CES nodes slowed to a > crawl. GPFS waiters showed pre fetch threads going after inodes. > iohist also showed lots of inode fetching. Then we noticed that the CES > host had 5.4 million files open. > > The change I made was to set maxStatCache=DEFAULT because it is linux. > And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. > > Is there something else we should change as well. > > Thanks > > Matt > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. From bbanister at jumptrading.com Thu Mar 23 15:26:57 2017 From: bbanister at jumptrading.com (Bryan Banister) Date: Thu, 23 Mar 2017 15:26:57 +0000 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: <643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> References: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu> <47e18bb0062544f4b510ce0e87049fd3@jumptrading.com> <643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> Message-ID: Anybody from IBM willing/able to give us some explanation of why ganesha is holding open so many files? Is this expected/needed/etc? Or do we have to open a PMR to get some kind of explanation? -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Thursday, March 23, 2017 10:24 AM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] CES node slow to respond FYI all We also ran into this after bumping maxFilesToCache. Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: unexpected error: Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many open files in system fix sysctl -w fs.file-max=1000000 On 3/22/17 12:01 PM, Bryan Banister wrote: > We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. > > Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil > Sent: Wednesday, March 22, 2017 11:43 AM > To: gpfsug main discussion list > Subject: [gpfsug-discuss] CES node slow to respond > > All, > > We had an indecent yesterday where one of our CES nodes slowed to a > crawl. GPFS waiters showed pre fetch threads going after inodes. > iohist also showed lots of inode fetching. Then we noticed that the CES > host had 5.4 million files open. > > The change I made was to set maxStatCache=DEFAULT because it is linux. > And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. > > Is there something else we should change as well. > > Thanks > > Matt > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. From mweil at wustl.edu Thu Mar 23 15:35:30 2017 From: mweil at wustl.edu (Matt Weil) Date: Thu, 23 Mar 2017 10:35:30 -0500 Subject: [gpfsug-discuss] Dual homing CES nodes Message-ID: Hello all, Are there any issues with connecting CES nodes to multiple networks? Thanks Matt ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. From kevindjo at us.ibm.com Thu Mar 23 15:39:48 2017 From: kevindjo at us.ibm.com (Kevin D Johnson) Date: Thu, 23 Mar 2017 15:39:48 +0000 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: References: , <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu><47e18bb0062544f4b510ce0e87049fd3@jumptrading.com><643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> Message-ID: An HTML attachment was scrubbed... URL: From jonathon.anderson at colorado.edu Thu Mar 23 15:41:47 2017 From: jonathon.anderson at colorado.edu (Jonathon A Anderson) Date: Thu, 23 Mar 2017 15:41:47 +0000 Subject: [gpfsug-discuss] Dual homing CES nodes In-Reply-To: References: Message-ID: <0FE05202-B444-4EBB-994C-AB2CC229F1DA@colorado.edu> We?ve run CES with addresses on a different network than the GPFS admin or daemon interface; but I haven?t tried running CES with addresses on multiple networks itself. ~jonathon On 3/23/17, 9:35 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Matt Weil" wrote: Hello all, Are there any issues with connecting CES nodes to multiple networks? Thanks Matt ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From skylar2 at u.washington.edu Thu Mar 23 15:42:18 2017 From: skylar2 at u.washington.edu (Skylar Thompson) Date: Thu, 23 Mar 2017 15:42:18 +0000 Subject: [gpfsug-discuss] Dual homing CES nodes In-Reply-To: References: Message-ID: <20170323154218.wqjhjaszffinajy5@utumno.gs.washington.edu> The only thing I can think of is you should be careful that you have distinct CES groups so that addresses will failover to the right networks. On Thu, Mar 23, 2017 at 10:35:30AM -0500, Matt Weil wrote: > Hello all, > > Are there any issues with connecting CES nodes to multiple networks? -- -- Skylar Thompson (skylar2 at u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine From mweil at wustl.edu Thu Mar 23 15:44:47 2017 From: mweil at wustl.edu (Matt Weil) Date: Thu, 23 Mar 2017 10:44:47 -0500 Subject: [gpfsug-discuss] Dual homing CES nodes In-Reply-To: <20170323154218.wqjhjaszffinajy5@utumno.gs.washington.edu> References: <20170323154218.wqjhjaszffinajy5@utumno.gs.washington.edu> Message-ID: <43fe47d2-f3a7-2b52-4a21-b135c7141e9f@wustl.edu> yeah it seems to know which interface to use. here is my quick test.. no failure groups. > em2: flags=4099 mtu 1500 > inet 192.168.1.2 netmask 255.255.255.0 broadcast 192.168.1.255 > ether 84:2b:2b:47:70:35 txqueuelen 1000 (Ethernet) > RX packets 0 bytes 0 (0.0 B) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 0 bytes 0 (0.0 B) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > em2:0: flags=4099 mtu 1500 > inet 192.168.1.5 netmask 255.255.255.0 broadcast 192.168.1.255 > ether 84:2b:2b:47:70:35 txqueuelen 1000 (Ethernet) On 3/23/17 10:42 AM, Skylar Thompson wrote: > The only thing I can think of is you should be careful that you have > distinct CES groups so that addresses will failover to the right networks. > > On Thu, Mar 23, 2017 at 10:35:30AM -0500, Matt Weil wrote: >> Hello all, >> >> Are there any issues with connecting CES nodes to multiple networks? ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. From S.J.Thompson at bham.ac.uk Thu Mar 23 15:46:38 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Thu, 23 Mar 2017 15:46:38 +0000 Subject: [gpfsug-discuss] Dual homing CES nodes In-Reply-To: References: Message-ID: We've done this in an unsupported manner when we were testing connectivity to private VM networks - manually attach VXLAN interface to host, then enable CES on the segment. Worked fine for us in this, so don't see why it should;t work fine for real/VLAN interfaces. Just be mindful of asymmetric routing of course. Simon >Hello all, > >Are there any issues with connecting CES nodes to multiple networks? > >Thanks >Matt > >________________________________ >The materials in this message are private and may contain Protected >Healthcare Information or other information of a sensitive nature. If you >are not the intended recipient, be advised that any unauthorized use, >disclosure, copying or the taking of any action in reliance on the >contents of this information is strictly prohibited. If you have received >this email in error, please immediately notify the sender via telephone >or return mail. >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss From ulmer at ulmer.org Thu Mar 23 18:04:17 2017 From: ulmer at ulmer.org (Stephen Ulmer) Date: Thu, 23 Mar 2017 14:04:17 -0400 Subject: [gpfsug-discuss] GPFS on PowerVM Shared Storage Pools Message-ID: <91792F99-B4EB-4B96-9F89-01B30F49108A@ulmer.org> I have a client who is very enamored with PowerVM Shared Storage Pools (because they work great for them). Has anyone here implemented GPFS on such? I think that technically there?s no reason that it wouldn?t work, but I?ve never actually "owned" an installation with GPFS on SSPs. Any thoughts? NB: There is an entry on the SSP FAQ that lists "What disk types are NOT supported for Shared Storage Pool use?", among which GPFS is listed. I take that to mean that you can?t deploy an SSP on GPFS storage (not the other way around). This entry also lists iSCSI, for example, which is definitely not supported for SSP use. Liberty, -- Stephen From aaron.s.knister at nasa.gov Fri Mar 24 16:43:02 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 24 Mar 2017 12:43:02 -0400 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock Message-ID: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Since yesterday morning we've noticed some deadlocks on one of our filesystems that seem to be triggered by writing to it. The waiters on the clients look like this: 0x19450B0 ( 6730) waiting 2063.294589599 seconds, SyncHandlerThread: on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) (InodeFlushCondVar), reason 'waiting for the flush flag to commit metadata' 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 (0x7FFFDAC7FE28) (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRelinquishRegion on node 10.1.52.33 0x197EE70 ( 6776) waiting 0.000198354 seconds, FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRequestRegion on node 10.1.52.33 (10.1.52.33/c0n3271 is the fs manager for the filesystem in question) there's a single process running on this node writing to the filesystem in question (well, trying to write, it's been blocked doing nothing for half an hour now). There are ~10 other client nodes in this situation right now. We had many more last night before the problem seemed to disappear in the early hours of the morning and now its back. Waiters on the fs manager look like this. While the individual waiter is short it's a near constant stream: 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg handler allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg handler allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg handler allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) I don't know if this data point is useful but both yesterday and today the metadata NSDs for this filesystem have had a constant aggregate stream of 25MB/s 4kop/s reads during each episode (very low latency though so I don't believe the storage is a bottleneck here). Writes are only a few hundred ops and didn't strike me as odd. I have a PMR open for this but I'm curious if folks have seen this in the wild and what it might mean. -Aaron -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From jfosburg at mdanderson.org Fri Mar 24 16:50:03 2017 From: jfosburg at mdanderson.org (Fosburgh,Jonathan) Date: Fri, 24 Mar 2017 16:50:03 +0000 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: <1490374200.1901.0.camel@mdanderson.org> This is one of the more annoying long waiter problems. We've seen it several times and I'm not sure they all had the same cause. What version of GPFS? Do you have anything like Tivoli HSM or LTFSEE? -- Jonathan Fosburgh Principal Application Systems Analyst Storage Team IT Operations jfosburg at mdanderson.org (713) 745-9346 -----Original Message----- Date: Fri, 24 Mar 2017 12:43:02 -0400 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock To: gpfsug main discussion list > Reply-to: gpfsug main discussion list From: Aaron Knister > Since yesterday morning we've noticed some deadlocks on one of our filesystems that seem to be triggered by writing to it. The waiters on the clients look like this: 0x19450B0 ( 6730) waiting 2063.294589599 seconds, SyncHandlerThread: on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) (InodeFlushCondVar), reason 'waiting for the flush flag to commit metadata' 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 (0x7FFFDAC7FE28) (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRelinquishRegion on node 10.1.52.33 0x197EE70 ( 6776) waiting 0.000198354 seconds, FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRequestRegion on node 10.1.52.33 (10.1.52.33/c0n3271 is the fs manager for the filesystem in question) there's a single process running on this node writing to the filesystem in question (well, trying to write, it's been blocked doing nothing for half an hour now). There are ~10 other client nodes in this situation right now. We had many more last night before the problem seemed to disappear in the early hours of the morning and now its back. Waiters on the fs manager look like this. While the individual waiter is short it's a near constant stream: 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg handler allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg handler allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg handler allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) I don't know if this data point is useful but both yesterday and today the metadata NSDs for this filesystem have had a constant aggregate stream of 25MB/s 4kop/s reads during each episode (very low latency though so I don't believe the storage is a bottleneck here). Writes are only a few hundred ops and didn't strike me as odd. I have a PMR open for this but I'm curious if folks have seen this in the wild and what it might mean. -Aaron The information contained in this e-mail message may be privileged, confidential, and/or protected from disclosure. This e-mail message may contain protected health information (PHI); dissemination of PHI should comply with applicable federal and state laws. If you are not the intended recipient, or an authorized representative of the intended recipient, any further review, disclosure, use, dissemination, distribution, or copying of this message or any attachment (or the information contained therein) is strictly prohibited. If you think that you have received this e-mail message in error, please notify the sender by return e-mail and delete all references to it and its contents from your systems. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Fri Mar 24 16:50:47 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Fri, 24 Mar 2017 16:50:47 +0000 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock Message-ID: <29B113F2-BDE9-456A-BBEB-9B1C203CB5F4@nuance.com> Hi Aaron Yes, I have seen this several times over the last 6 months. I opened at least one PMR on it and they never could track it down. I did some snap dumps but without some traces, they did not have enough. I ended up getting out of it by selectively rebooting some of my NSD servers. My suspicion is that one of them had a thread deadlocked and was holding up IO to the rest of the filesystem. I haven?t seen it since I updated to 4.2.2-2, but I?m not convinced (yet) that it?s not lurking in the background. Bob Oesterlin Sr Principal Storage Engineer, Nuance On 3/24/17, 11:43 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Aaron Knister" wrote: Since yesterday morning we've noticed some deadlocks on one of our filesystems that seem to be triggered by writing to it. The waiters on the clients look like this: 0x19450B0 ( 6730) waiting 2063.294589599 seconds, SyncHandlerThread: on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) (InodeFlushCondVar), reason 'waiting for the flush flag to commit metadata' 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 (0x7FFFDAC7FE28) (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRelinquishRegion on node 10.1.52.33 0x197EE70 ( 6776) waiting 0.000198354 seconds, FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRequestRegion on node 10.1.52.33 (10.1.52.33/c0n3271 is the fs manager for the filesystem in question) there's a single process running on this node writing to the filesystem in question (well, trying to write, it's been blocked doing nothing for half an hour now). There are ~10 other client nodes in this situation right now. We had many more last night before the problem seemed to disappear in the early hours of the morning and now its back. Waiters on the fs manager look like this. While the individual waiter is short it's a near constant stream: 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg handler allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg handler allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg handler allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) I don't know if this data point is useful but both yesterday and today the metadata NSDs for this filesystem have had a constant aggregate stream of 25MB/s 4kop/s reads during each episode (very low latency though so I don't believe the storage is a bottleneck here). Writes are only a few hundred ops and didn't strike me as odd. I have a PMR open for this but I'm curious if folks have seen this in the wild and what it might mean. -Aaron -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=fUSq1Wdz1p_yJQVOgOoqe7fu7nsPXewvz8BUeKJyiRg&s=9sXk_xPuEtEyJgVhsZ7FCgM-rfytQGDDC2EyqwYgLhQ&e= From oehmes at gmail.com Fri Mar 24 17:04:02 2017 From: oehmes at gmail.com (Sven Oehme) Date: Fri, 24 Mar 2017 17:04:02 +0000 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: while this is happening run top and see if there is very high cpu utilization at this time on the NSD Server. if there is , run perf top (you might need to install perf command) and see if the top cpu contender is a spinlock . if so send a screenshot of perf top as i may know what that is and how to fix. sven On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister wrote: > Since yesterday morning we've noticed some deadlocks on one of our > filesystems that seem to be triggered by writing to it. The waiters on > the clients look like this: > > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, SyncHandlerThread: > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) (InodeFlushCondVar), reason > 'waiting for the flush flag to commit metadata' > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 (0x7FFFDAC7FE28) > (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRelinquishRegion > on node 10.1.52.33 > 0x197EE70 ( 6776) waiting 0.000198354 seconds, > FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for > allocMsgTypeRequestRegion on node 10.1.52.33 > > (10.1.52.33/c0n3271 is the fs manager for the filesystem in question) > > there's a single process running on this node writing to the filesystem > in question (well, trying to write, it's been blocked doing nothing for > half an hour now). There are ~10 other client nodes in this situation > right now. We had many more last night before the problem seemed to > disappear in the early hours of the morning and now its back. > > Waiters on the fs manager look like this. While the individual waiter is > short it's a near constant stream: > > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > I don't know if this data point is useful but both yesterday and today > the metadata NSDs for this filesystem have had a constant aggregate > stream of 25MB/s 4kop/s reads during each episode (very low latency > though so I don't believe the storage is a bottleneck here). Writes are > only a few hundred ops and didn't strike me as odd. > > I have a PMR open for this but I'm curious if folks have seen this in > the wild and what it might mean. > > -Aaron > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Fri Mar 24 17:07:58 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 24 Mar 2017 13:07:58 -0400 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: Hi Sven, Which NSD server should I run top on, the fs manager? If so the CPU load is about 155%. I'm working on perf top but not off to a great start... # perf top PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% [1000Hz cycles], (all, 28 CPUs) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Segmentation fault -Aaron On 3/24/17 1:04 PM, Sven Oehme wrote: > while this is happening run top and see if there is very high cpu > utilization at this time on the NSD Server. > > if there is , run perf top (you might need to install perf command) and > see if the top cpu contender is a spinlock . if so send a screenshot of > perf top as i may know what that is and how to fix. > > sven > > > On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister > wrote: > > Since yesterday morning we've noticed some deadlocks on one of our > filesystems that seem to be triggered by writing to it. The waiters on > the clients look like this: > > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, SyncHandlerThread: > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) (InodeFlushCondVar), reason > 'waiting for the flush flag to commit metadata' > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 (0x7FFFDAC7FE28) > (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRelinquishRegion > on node 10.1.52.33 > 0x197EE70 ( 6776) waiting 0.000198354 seconds, > FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for > allocMsgTypeRequestRegion on node 10.1.52.33 > > (10.1.52.33/c0n3271 is the fs manager > for the filesystem in question) > > there's a single process running on this node writing to the filesystem > in question (well, trying to write, it's been blocked doing nothing for > half an hour now). There are ~10 other client nodes in this situation > right now. We had many more last night before the problem seemed to > disappear in the early hours of the morning and now its back. > > Waiters on the fs manager look like this. While the individual waiter is > short it's a near constant stream: > > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > I don't know if this data point is useful but both yesterday and today > the metadata NSDs for this filesystem have had a constant aggregate > stream of 25MB/s 4kop/s reads during each episode (very low latency > though so I don't believe the storage is a bottleneck here). Writes are > only a few hundred ops and didn't strike me as odd. > > I have a PMR open for this but I'm curious if folks have seen this in > the wild and what it might mean. > > -Aaron > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From scale at us.ibm.com Fri Mar 24 17:17:33 2017 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Fri, 24 Mar 2017 09:17:33 -0800 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: References: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu><47e18bb0062544f4b510ce0e87049fd3@jumptrading.com><643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> Message-ID: Hi Bryan, Making sure Malahal's reply was received by the user group. >> Then we noticed that the CES host had 5.4 million files open This is technically not possible with ganesha alone. A process can only open 1 million files on RHEL distro. Either we have leaks in kernel or some other processes contributing to this. Ganesha does keep NFSv3 files open and keep open for performance. It doesn't have a good logic to close them after some inactivity. It does close them if the number is close to max open files which is configurable. PS: kNFS does open, read/write, and then close. No caching in older versions. They did have a feature in a recent code to cache open files in NFSv3 as well. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Bryan Banister To: gpfsug main discussion list Date: 03/23/2017 08:27 AM Subject: Re: [gpfsug-discuss] CES node slow to respond Sent by: gpfsug-discuss-bounces at spectrumscale.org Anybody from IBM willing/able to give us some explanation of why ganesha is holding open so many files? Is this expected/needed/etc? Or do we have to open a PMR to get some kind of explanation? -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Thursday, March 23, 2017 10:24 AM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] CES node slow to respond FYI all We also ran into this after bumping maxFilesToCache. Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: unexpected error: Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many open files in system fix sysctl -w fs.file-max=1000000 On 3/22/17 12:01 PM, Bryan Banister wrote: > We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. > > Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil > Sent: Wednesday, March 22, 2017 11:43 AM > To: gpfsug main discussion list > Subject: [gpfsug-discuss] CES node slow to respond > > All, > > We had an indecent yesterday where one of our CES nodes slowed to a > crawl. GPFS waiters showed pre fetch threads going after inodes. > iohist also showed lots of inode fetching. Then we noticed that the CES > host had 5.4 million files open. > > The change I made was to set maxStatCache=DEFAULT because it is linux. > And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. > > Is there something else we should change as well. > > Thanks > > Matt > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From oehmes at gmail.com Fri Mar 24 17:22:12 2017 From: oehmes at gmail.com (Sven Oehme) Date: Fri, 24 Mar 2017 17:22:12 +0000 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: you must be on sles as this segfaults only on sles to my knowledge :-) i am looking for a NSD or manager node in your cluster that runs at 100% cpu usage. do you have zimon deployed to look at cpu utilization across your nodes ? sven On Fri, Mar 24, 2017 at 10:08 AM Aaron Knister wrote: Hi Sven, Which NSD server should I run top on, the fs manager? If so the CPU load is about 155%. I'm working on perf top but not off to a great start... # perf top PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% [1000Hz cycles], (all, 28 CPUs) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Segmentation fault -Aaron On 3/24/17 1:04 PM, Sven Oehme wrote: > while this is happening run top and see if there is very high cpu > utilization at this time on the NSD Server. > > if there is , run perf top (you might need to install perf command) and > see if the top cpu contender is a spinlock . if so send a screenshot of > perf top as i may know what that is and how to fix. > > sven > > > On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister > wrote: > > Since yesterday morning we've noticed some deadlocks on one of our > filesystems that seem to be triggered by writing to it. The waiters on > the clients look like this: > > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, SyncHandlerThread: > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) (InodeFlushCondVar), reason > 'waiting for the flush flag to commit metadata' > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 (0x7FFFDAC7FE28) > (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRelinquishRegion > on node 10.1.52.33 > 0x197EE70 ( 6776) waiting 0.000198354 seconds, > FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for > allocMsgTypeRequestRegion on node 10.1.52.33 > > (10.1.52.33/c0n3271 is the fs manager > for the filesystem in question) > > there's a single process running on this node writing to the filesystem > in question (well, trying to write, it's been blocked doing nothing for > half an hour now). There are ~10 other client nodes in this situation > right now. We had many more last night before the problem seemed to > disappear in the early hours of the morning and now its back. > > Waiters on the fs manager look like this. While the individual waiter is > short it's a near constant stream: > > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > I don't know if this data point is useful but both yesterday and today > the metadata NSDs for this filesystem have had a constant aggregate > stream of 25MB/s 4kop/s reads during each episode (very low latency > though so I don't believe the storage is a bottleneck here). Writes are > only a few hundred ops and didn't strike me as odd. > > I have a PMR open for this but I'm curious if folks have seen this in > the wild and what it might mean. > > -Aaron > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Fri Mar 24 17:32:26 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 24 Mar 2017 13:32:26 -0400 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: heh, yep we're on sles :) here's a screenshot of the fs manager from the deadlocked filesystem. I don't think there's an nsd server or manager node that's running full throttle across all cpus. There is one that's got relatively high CPU utilization though (300-400%). I'll send a screenshot of it in a sec. no zimon yet but we do have other tools to see cpu utilization. -Aaron On 3/24/17 1:22 PM, Sven Oehme wrote: > you must be on sles as this segfaults only on sles to my knowledge :-) > > i am looking for a NSD or manager node in your cluster that runs at 100% > cpu usage. > > do you have zimon deployed to look at cpu utilization across your nodes ? > > sven > > > > On Fri, Mar 24, 2017 at 10:08 AM Aaron Knister > wrote: > > Hi Sven, > > Which NSD server should I run top on, the fs manager? If so the CPU load > is about 155%. I'm working on perf top but not off to a great start... > > # perf top > PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% [1000Hz > cycles], (all, 28 CPUs) > --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Segmentation fault > > -Aaron > > On 3/24/17 1:04 PM, Sven Oehme wrote: > > while this is happening run top and see if there is very high cpu > > utilization at this time on the NSD Server. > > > > if there is , run perf top (you might need to install perf command) and > > see if the top cpu contender is a spinlock . if so send a screenshot of > > perf top as i may know what that is and how to fix. > > > > sven > > > > > > On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister > > >> wrote: > > > > Since yesterday morning we've noticed some deadlocks on one of our > > filesystems that seem to be triggered by writing to it. The waiters on > > the clients look like this: > > > > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, SyncHandlerThread: > > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) (InodeFlushCondVar), reason > > 'waiting for the flush flag to commit metadata' > > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, > > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 (0x7FFFDAC7FE28) > > (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRelinquishRegion > > on node 10.1.52.33 > > 0x197EE70 ( 6776) waiting 0.000198354 seconds, > > FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 > > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for > > allocMsgTypeRequestRegion on node 10.1.52.33 > > > > (10.1.52.33/c0n3271 > is the fs manager > > for the filesystem in question) > > > > there's a single process running on this node writing to the filesystem > > in question (well, trying to write, it's been blocked doing nothing for > > half an hour now). There are ~10 other client nodes in this situation > > right now. We had many more last night before the problem seemed to > > disappear in the early hours of the morning and now its back. > > > > Waiters on the fs manager look like this. While the individual waiter is > > short it's a near constant stream: > > > > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg handler > > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > > (AllocManagerMutex) > > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg handler > > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg handler > > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > > (AllocManagerMutex) > > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg handler > > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > > (AllocManagerMutex) > > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg handler > > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg handler > > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > > (AllocManagerMutex) > > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg handler > > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > > (AllocManagerMutex) > > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg handler > > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > > > I don't know if this data point is useful but both yesterday and today > > the metadata NSDs for this filesystem have had a constant aggregate > > stream of 25MB/s 4kop/s reads during each episode (very low latency > > though so I don't believe the storage is a bottleneck here). Writes are > > only a few hundred ops and didn't strike me as odd. > > > > I have a PMR open for this but I'm curious if folks have seen this in > > the wild and what it might mean. > > > > -Aaron > > > > -- > > Aaron Knister > > NASA Center for Climate Simulation (Code 606.2) > > Goddard Space Flight Center > > (301) 286-2776 > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 -------------- next part -------------- A non-text attachment was scrubbed... Name: perf_top.png Type: image/png Size: 218901 bytes Desc: not available URL: From aaron.s.knister at nasa.gov Fri Mar 24 17:34:30 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 24 Mar 2017 13:34:30 -0400 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: Here's the screenshot from the other node with the high cpu utilization. On 3/24/17 1:32 PM, Aaron Knister wrote: > heh, yep we're on sles :) > > here's a screenshot of the fs manager from the deadlocked filesystem. I > don't think there's an nsd server or manager node that's running full > throttle across all cpus. There is one that's got relatively high CPU > utilization though (300-400%). I'll send a screenshot of it in a sec. > > no zimon yet but we do have other tools to see cpu utilization. > > -Aaron > > On 3/24/17 1:22 PM, Sven Oehme wrote: >> you must be on sles as this segfaults only on sles to my knowledge :-) >> >> i am looking for a NSD or manager node in your cluster that runs at 100% >> cpu usage. >> >> do you have zimon deployed to look at cpu utilization across your nodes ? >> >> sven >> >> >> >> On Fri, Mar 24, 2017 at 10:08 AM Aaron Knister > > wrote: >> >> Hi Sven, >> >> Which NSD server should I run top on, the fs manager? If so the >> CPU load >> is about 155%. I'm working on perf top but not off to a great >> start... >> >> # perf top >> PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% [1000Hz >> cycles], (all, 28 CPUs) >> >> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >> >> Segmentation fault >> >> -Aaron >> >> On 3/24/17 1:04 PM, Sven Oehme wrote: >> > while this is happening run top and see if there is very high cpu >> > utilization at this time on the NSD Server. >> > >> > if there is , run perf top (you might need to install perf >> command) and >> > see if the top cpu contender is a spinlock . if so send a >> screenshot of >> > perf top as i may know what that is and how to fix. >> > >> > sven >> > >> > >> > On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister >> >> > > >> wrote: >> > >> > Since yesterday morning we've noticed some deadlocks on one >> of our >> > filesystems that seem to be triggered by writing to it. The >> waiters on >> > the clients look like this: >> > >> > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, >> SyncHandlerThread: >> > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) >> (InodeFlushCondVar), reason >> > 'waiting for the flush flag to commit metadata' >> > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, >> > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 >> (0x7FFFDAC7FE28) >> > (MsgRecordCondvar), reason 'RPC wait' for >> allocMsgTypeRelinquishRegion >> > on node 10.1.52.33 >> > 0x197EE70 ( 6776) waiting 0.000198354 seconds, >> > FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 >> > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for >> > allocMsgTypeRequestRegion on node 10.1.52.33 >> > >> > (10.1.52.33/c0n3271 >> is the fs manager >> > for the filesystem in question) >> > >> > there's a single process running on this node writing to the >> filesystem >> > in question (well, trying to write, it's been blocked doing >> nothing for >> > half an hour now). There are ~10 other client nodes in this >> situation >> > right now. We had many more last night before the problem >> seemed to >> > disappear in the early hours of the morning and now its back. >> > >> > Waiters on the fs manager look like this. While the >> individual waiter is >> > short it's a near constant stream: >> > >> > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg >> handler >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) >> > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg >> handler >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) >> > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg >> handler >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) >> > >> > I don't know if this data point is useful but both yesterday >> and today >> > the metadata NSDs for this filesystem have had a constant >> aggregate >> > stream of 25MB/s 4kop/s reads during each episode (very low >> latency >> > though so I don't believe the storage is a bottleneck here). >> Writes are >> > only a few hundred ops and didn't strike me as odd. >> > >> > I have a PMR open for this but I'm curious if folks have >> seen this in >> > the wild and what it might mean. >> > >> > -Aaron >> > >> > -- >> > Aaron Knister >> > NASA Center for Climate Simulation (Code 606.2) >> > Goddard Space Flight Center >> > (301) 286-2776 >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > >> > >> > >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > >> >> -- >> Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) >> Goddard Space Flight Center >> (301) 286-2776 >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 -------------- next part -------------- A non-text attachment was scrubbed... Name: nsd35.png Type: image/png Size: 243009 bytes Desc: not available URL: From oehmes at gmail.com Fri Mar 24 17:41:10 2017 From: oehmes at gmail.com (Sven Oehme) Date: Fri, 24 Mar 2017 17:41:10 +0000 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: ok, that seems a different problem then i was thinking. can you send output of mmlscluster, mmlsconfig, mmlsfs all ? also are you getting close to fill grade on inodes or capacity on any of the filesystems ? sven On Fri, Mar 24, 2017 at 10:34 AM Aaron Knister wrote: > Here's the screenshot from the other node with the high cpu utilization. > > On 3/24/17 1:32 PM, Aaron Knister wrote: > > heh, yep we're on sles :) > > > > here's a screenshot of the fs manager from the deadlocked filesystem. I > > don't think there's an nsd server or manager node that's running full > > throttle across all cpus. There is one that's got relatively high CPU > > utilization though (300-400%). I'll send a screenshot of it in a sec. > > > > no zimon yet but we do have other tools to see cpu utilization. > > > > -Aaron > > > > On 3/24/17 1:22 PM, Sven Oehme wrote: > >> you must be on sles as this segfaults only on sles to my knowledge :-) > >> > >> i am looking for a NSD or manager node in your cluster that runs at 100% > >> cpu usage. > >> > >> do you have zimon deployed to look at cpu utilization across your nodes > ? > >> > >> sven > >> > >> > >> > >> On Fri, Mar 24, 2017 at 10:08 AM Aaron Knister < > aaron.s.knister at nasa.gov > >> > wrote: > >> > >> Hi Sven, > >> > >> Which NSD server should I run top on, the fs manager? If so the > >> CPU load > >> is about 155%. I'm working on perf top but not off to a great > >> start... > >> > >> # perf top > >> PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% [1000Hz > >> cycles], (all, 28 CPUs) > >> > >> > --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > >> > >> Segmentation fault > >> > >> -Aaron > >> > >> On 3/24/17 1:04 PM, Sven Oehme wrote: > >> > while this is happening run top and see if there is very high cpu > >> > utilization at this time on the NSD Server. > >> > > >> > if there is , run perf top (you might need to install perf > >> command) and > >> > see if the top cpu contender is a spinlock . if so send a > >> screenshot of > >> > perf top as i may know what that is and how to fix. > >> > > >> > sven > >> > > >> > > >> > On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister > >> > >> > >> >> wrote: > >> > > >> > Since yesterday morning we've noticed some deadlocks on one > >> of our > >> > filesystems that seem to be triggered by writing to it. The > >> waiters on > >> > the clients look like this: > >> > > >> > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, > >> SyncHandlerThread: > >> > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) > >> (InodeFlushCondVar), reason > >> > 'waiting for the flush flag to commit metadata' > >> > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, > >> > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 > >> (0x7FFFDAC7FE28) > >> > (MsgRecordCondvar), reason 'RPC wait' for > >> allocMsgTypeRelinquishRegion > >> > on node 10.1.52.33 > >> > 0x197EE70 ( 6776) waiting 0.000198354 seconds, > >> > FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 > >> > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for > >> > allocMsgTypeRequestRegion on node 10.1.52.33 > >> > > >> > (10.1.52.33/c0n3271 > >> is the fs manager > >> > for the filesystem in question) > >> > > >> > there's a single process running on this node writing to the > >> filesystem > >> > in question (well, trying to write, it's been blocked doing > >> nothing for > >> > half an hour now). There are ~10 other client nodes in this > >> situation > >> > right now. We had many more last night before the problem > >> seemed to > >> > disappear in the early hours of the morning and now its back. > >> > > >> > Waiters on the fs manager look like this. While the > >> individual waiter is > >> > short it's a near constant stream: > >> > > >> > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg > >> handler > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > >> (0xFFFFC9002163A2E0) > >> > (AllocManagerMutex) > >> > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg > >> handler > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > >> > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg > >> handler > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > >> (0xFFFFC9002163A2E0) > >> > (AllocManagerMutex) > >> > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg > >> handler > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > >> (0xFFFFC9002163A2E0) > >> > (AllocManagerMutex) > >> > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg > >> handler > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > >> > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg > >> handler > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > >> (0xFFFFC9002163A2E0) > >> > (AllocManagerMutex) > >> > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg > >> handler > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > >> (0xFFFFC9002163A2E0) > >> > (AllocManagerMutex) > >> > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg > >> handler > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > >> > > >> > I don't know if this data point is useful but both yesterday > >> and today > >> > the metadata NSDs for this filesystem have had a constant > >> aggregate > >> > stream of 25MB/s 4kop/s reads during each episode (very low > >> latency > >> > though so I don't believe the storage is a bottleneck here). > >> Writes are > >> > only a few hundred ops and didn't strike me as odd. > >> > > >> > I have a PMR open for this but I'm curious if folks have > >> seen this in > >> > the wild and what it might mean. > >> > > >> > -Aaron > >> > > >> > -- > >> > Aaron Knister > >> > NASA Center for Climate Simulation (Code 606.2) > >> > Goddard Space Flight Center > >> > (301) 286-2776 > >> > _______________________________________________ > >> > gpfsug-discuss mailing list > >> > gpfsug-discuss at spectrumscale.org > >> > >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > >> > > >> > > >> > _______________________________________________ > >> > gpfsug-discuss mailing list > >> > gpfsug-discuss at spectrumscale.org > >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > >> > >> -- > >> Aaron Knister > >> NASA Center for Climate Simulation (Code 606.2) > >> Goddard Space Flight Center > >> (301) 286-2776 > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > >> > >> > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Fri Mar 24 17:53:02 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 24 Mar 2017 13:53:02 -0400 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: <29B113F2-BDE9-456A-BBEB-9B1C203CB5F4@nuance.com> References: <29B113F2-BDE9-456A-BBEB-9B1C203CB5F4@nuance.com> Message-ID: Thanks Bob, Jonathan. We're running GPFS 4.1.1.10 and no HSM/LTFSEE. I'm currently gathering, as requested, a snap from all nodes (with traces). With 3500 nodes this ought to be entertaining. -Aaron On 3/24/17 12:50 PM, Oesterlin, Robert wrote: > Hi Aaron > > Yes, I have seen this several times over the last 6 months. I opened at least one PMR on it and they never could track it down. I did some snap dumps but without some traces, they did not have enough. I ended up getting out of it by selectively rebooting some of my NSD servers. My suspicion is that one of them had a thread deadlocked and was holding up IO to the rest of the filesystem. > > I haven?t seen it since I updated to 4.2.2-2, but I?m not convinced (yet) that it?s not lurking in the background. > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > > On 3/24/17, 11:43 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Aaron Knister" wrote: > > Since yesterday morning we've noticed some deadlocks on one of our > filesystems that seem to be triggered by writing to it. The waiters on > the clients look like this: > > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, SyncHandlerThread: > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) (InodeFlushCondVar), reason > 'waiting for the flush flag to commit metadata' > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 (0x7FFFDAC7FE28) > (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRelinquishRegion > on node 10.1.52.33 > 0x197EE70 ( 6776) waiting 0.000198354 seconds, > FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for > allocMsgTypeRequestRegion on node 10.1.52.33 > > (10.1.52.33/c0n3271 is the fs manager for the filesystem in question) > > there's a single process running on this node writing to the filesystem > in question (well, trying to write, it's been blocked doing nothing for > half an hour now). There are ~10 other client nodes in this situation > right now. We had many more last night before the problem seemed to > disappear in the early hours of the morning and now its back. > > Waiters on the fs manager look like this. While the individual waiter is > short it's a near constant stream: > > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > I don't know if this data point is useful but both yesterday and today > the metadata NSDs for this filesystem have had a constant aggregate > stream of 25MB/s 4kop/s reads during each episode (very low latency > though so I don't believe the storage is a bottleneck here). Writes are > only a few hundred ops and didn't strike me as odd. > > I have a PMR open for this but I'm curious if folks have seen this in > the wild and what it might mean. > > -Aaron > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=fUSq1Wdz1p_yJQVOgOoqe7fu7nsPXewvz8BUeKJyiRg&s=9sXk_xPuEtEyJgVhsZ7FCgM-rfytQGDDC2EyqwYgLhQ&e= > > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From aaron.s.knister at nasa.gov Fri Mar 24 17:58:18 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 24 Mar 2017 13:58:18 -0400 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: I feel a little awkward about posting wlists of IP's and hostnames on the mailing list (even though they're all internal) but I'm happy to send to you directly. I've attached both an lsfs and an mmdf output of the fs in question here since that may be useful for others to see. Just a note about disk d23_02_021-- it's been evacuated for several weeks now due to a hardware issue in the disk enclosure. The fs is rather full percentage wise (93%) but in terms of capacity there's a good amount free. 93% full of a 7PB filesystem still leaves 551T. Metadata, as you'll see, is 31% free (roughly 800GB). The fs has 40M inodes allocated and 12M free. -Aaron On 3/24/17 1:41 PM, Sven Oehme wrote: > ok, that seems a different problem then i was thinking. > can you send output of mmlscluster, mmlsconfig, mmlsfs all ? > also are you getting close to fill grade on inodes or capacity on any of > the filesystems ? > > sven > > > On Fri, Mar 24, 2017 at 10:34 AM Aaron Knister > wrote: > > Here's the screenshot from the other node with the high cpu utilization. > > On 3/24/17 1:32 PM, Aaron Knister wrote: > > heh, yep we're on sles :) > > > > here's a screenshot of the fs manager from the deadlocked filesystem. I > > don't think there's an nsd server or manager node that's running full > > throttle across all cpus. There is one that's got relatively high CPU > > utilization though (300-400%). I'll send a screenshot of it in a sec. > > > > no zimon yet but we do have other tools to see cpu utilization. > > > > -Aaron > > > > On 3/24/17 1:22 PM, Sven Oehme wrote: > >> you must be on sles as this segfaults only on sles to my knowledge :-) > >> > >> i am looking for a NSD or manager node in your cluster that runs at 100% > >> cpu usage. > >> > >> do you have zimon deployed to look at cpu utilization across your nodes ? > >> > >> sven > >> > >> > >> > >> On Fri, Mar 24, 2017 at 10:08 AM Aaron Knister > >> >> wrote: > >> > >> Hi Sven, > >> > >> Which NSD server should I run top on, the fs manager? If so the > >> CPU load > >> is about 155%. I'm working on perf top but not off to a great > >> start... > >> > >> # perf top > >> PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% [1000Hz > >> cycles], (all, 28 CPUs) > >> > >> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > >> > >> Segmentation fault > >> > >> -Aaron > >> > >> On 3/24/17 1:04 PM, Sven Oehme wrote: > >> > while this is happening run top and see if there is very high cpu > >> > utilization at this time on the NSD Server. > >> > > >> > if there is , run perf top (you might need to install perf > >> command) and > >> > see if the top cpu contender is a spinlock . if so send a > >> screenshot of > >> > perf top as i may know what that is and how to fix. > >> > > >> > sven > >> > > >> > > >> > On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister > >> > > > >> > > >> >>> wrote: > >> > > >> > Since yesterday morning we've noticed some deadlocks on one > >> of our > >> > filesystems that seem to be triggered by writing to it. The > >> waiters on > >> > the clients look like this: > >> > > >> > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, > >> SyncHandlerThread: > >> > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) > >> (InodeFlushCondVar), reason > >> > 'waiting for the flush flag to commit metadata' > >> > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, > >> > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 > >> (0x7FFFDAC7FE28) > >> > (MsgRecordCondvar), reason 'RPC wait' for > >> allocMsgTypeRelinquishRegion > >> > on node 10.1.52.33 > >> > 0x197EE70 ( 6776) waiting 0.000198354 seconds, > >> > FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 > >> > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for > >> > allocMsgTypeRequestRegion on node 10.1.52.33 > >> > > >> > (10.1.52.33/c0n3271 > > >> is the fs manager > >> > for the filesystem in question) > >> > > >> > there's a single process running on this node writing to the > >> filesystem > >> > in question (well, trying to write, it's been blocked doing > >> nothing for > >> > half an hour now). There are ~10 other client nodes in this > >> situation > >> > right now. We had many more last night before the problem > >> seemed to > >> > disappear in the early hours of the morning and now its back. > >> > > >> > Waiters on the fs manager look like this. While the > >> individual waiter is > >> > short it's a near constant stream: > >> > > >> > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg > >> handler > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > >> (0xFFFFC9002163A2E0) > >> > (AllocManagerMutex) > >> > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg > >> handler > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > >> > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg > >> handler > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > >> (0xFFFFC9002163A2E0) > >> > (AllocManagerMutex) > >> > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg > >> handler > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > >> (0xFFFFC9002163A2E0) > >> > (AllocManagerMutex) > >> > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg > >> handler > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > >> > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg > >> handler > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > >> (0xFFFFC9002163A2E0) > >> > (AllocManagerMutex) > >> > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg > >> handler > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > >> (0xFFFFC9002163A2E0) > >> > (AllocManagerMutex) > >> > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg > >> handler > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > >> > > >> > I don't know if this data point is useful but both yesterday > >> and today > >> > the metadata NSDs for this filesystem have had a constant > >> aggregate > >> > stream of 25MB/s 4kop/s reads during each episode (very low > >> latency > >> > though so I don't believe the storage is a bottleneck here). > >> Writes are > >> > only a few hundred ops and didn't strike me as odd. > >> > > >> > I have a PMR open for this but I'm curious if folks have > >> seen this in > >> > the wild and what it might mean. > >> > > >> > -Aaron > >> > > >> > -- > >> > Aaron Knister > >> > NASA Center for Climate Simulation (Code 606.2) > >> > Goddard Space Flight Center > >> > (301) 286-2776 > > >> > _______________________________________________ > >> > gpfsug-discuss mailing list > >> > gpfsug-discuss at spectrumscale.org > >> > >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > >> > > >> > > >> > _______________________________________________ > >> > gpfsug-discuss mailing list > >> > gpfsug-discuss at spectrumscale.org > >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > >> > >> -- > >> Aaron Knister > >> NASA Center for Climate Simulation (Code 606.2) > >> Goddard Space Flight Center > >> (301) 286-2776 > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > >> > >> > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 -------------- next part -------------- disk disk size failure holds holds free KB free KB name in KB group metadata data in full blocks in fragments --------------- ------------- -------- -------- ----- -------------------- ------------------- Disks in storage pool: system (Maximum disk size allowed is 8.4 TB) m01_12_061 52428800 30 yes no 40660992 ( 78%) 689168 ( 1%) m01_12_062 52428800 30 yes no 40662528 ( 78%) 687168 ( 1%) m01_12_063 52428800 30 yes no 40657920 ( 78%) 694256 ( 1%) m01_12_064 52428800 30 yes no 40654080 ( 78%) 703464 ( 1%) m01_12_096 52428800 30 yes no 15453184 ( 29%) 1138392 ( 2%) m01_12_095 52428800 30 yes no 15514112 ( 30%) 1139072 ( 2%) m01_12_094 52428800 30 yes no 15425536 ( 29%) 1111776 ( 2%) m01_12_093 52428800 30 yes no 15340544 ( 29%) 1126584 ( 2%) m01_12_068 52428800 30 yes no 40716544 ( 78%) 688752 ( 1%) m01_12_067 52428800 30 yes no 40690944 ( 78%) 692200 ( 1%) m01_12_066 52428800 30 yes no 40687104 ( 78%) 692976 ( 1%) m01_12_065 52428800 30 yes no 40674304 ( 78%) 690848 ( 1%) m01_12_081 52428800 30 yes no 137728 ( 0%) 487760 ( 1%) m01_12_082 52428800 30 yes no 60416 ( 0%) 512632 ( 1%) m01_12_083 52428800 30 yes no 102144 ( 0%) 674152 ( 1%) m01_12_084 52428800 30 yes no 126208 ( 0%) 684296 ( 1%) m01_12_085 52428800 30 yes no 117504 ( 0%) 705952 ( 1%) m01_12_086 52428800 30 yes no 119296 ( 0%) 691056 ( 1%) m01_12_087 52428800 30 yes no 57344 ( 0%) 493992 ( 1%) m01_12_088 52428800 30 yes no 60672 ( 0%) 547360 ( 1%) m01_12_089 52428800 30 yes no 1455616 ( 3%) 888688 ( 2%) m01_12_090 52428800 30 yes no 1467392 ( 3%) 919312 ( 2%) m01_12_091 52428800 30 yes no 190464 ( 0%) 745456 ( 1%) m01_12_092 52428800 30 yes no 1367296 ( 3%) 771400 ( 1%) m02_22_081 52428800 40 yes no 1245696 ( 2%) 855992 ( 2%) m02_22_082 52428800 40 yes no 1261056 ( 2%) 869336 ( 2%) m02_22_083 52428800 40 yes no 1254912 ( 2%) 865656 ( 2%) m02_22_084 52428800 40 yes no 62464 ( 0%) 698480 ( 1%) m02_22_085 52428800 40 yes no 64256 ( 0%) 703016 ( 1%) m02_22_086 52428800 40 yes no 62208 ( 0%) 690032 ( 1%) m02_22_087 52428800 40 yes no 62464 ( 0%) 687584 ( 1%) m02_22_088 52428800 40 yes no 68608 ( 0%) 699848 ( 1%) m02_22_089 52428800 40 yes no 84480 ( 0%) 698144 ( 1%) m02_22_090 52428800 40 yes no 85248 ( 0%) 720216 ( 1%) m02_22_091 52428800 40 yes no 98816 ( 0%) 711824 ( 1%) m02_22_092 52428800 40 yes no 104448 ( 0%) 732808 ( 1%) m02_22_068 52428800 40 yes no 40727552 ( 78%) 702472 ( 1%) m02_22_067 52428800 40 yes no 40713728 ( 78%) 688576 ( 1%) m02_22_066 52428800 40 yes no 40694272 ( 78%) 700960 ( 1%) m02_22_065 52428800 40 yes no 40694016 ( 78%) 689936 ( 1%) m02_22_064 52428800 40 yes no 40683264 ( 78%) 695144 ( 1%) m02_22_063 52428800 40 yes no 40676864 ( 78%) 701288 ( 1%) m02_22_062 52428800 40 yes no 40670976 ( 78%) 692984 ( 1%) m02_22_061 52428800 40 yes no 40672512 ( 78%) 690024 ( 1%) m02_22_096 52428800 40 yes no 15327232 ( 29%) 1149064 ( 2%) m02_22_095 52428800 40 yes no 15363584 ( 29%) 1146384 ( 2%) m02_22_094 52428800 40 yes no 15397376 ( 29%) 1172856 ( 2%) m02_22_093 52428800 40 yes no 15374336 ( 29%) 1163832 ( 2%) ------------- -------------------- ------------------- (pool total) 2516582400 783850240 ( 31%) 37303168 ( 1%) Disks in storage pool: sp_1620 (Maximum disk size allowed is 5.4 PB) d23_01_001 46028292096 1620 no yes 3541176320 ( 8%) 37724768 ( 0%) d23_01_002 46028292096 1620 no yes 3542331392 ( 8%) 37545024 ( 0%) d23_01_003 46028292096 1620 no yes 3541968896 ( 8%) 37765344 ( 0%) d23_01_004 46028292096 1620 no yes 3544687616 ( 8%) 37720576 ( 0%) d23_01_005 46028292096 1620 no yes 3543368704 ( 8%) 37647456 ( 0%) d23_01_006 46028292096 1620 no yes 3542778880 ( 8%) 37695232 ( 0%) d23_01_007 46028292096 1620 no yes 3543220224 ( 8%) 37539712 ( 0%) d23_01_008 46028292096 1620 no yes 3540293632 ( 8%) 37548928 ( 0%) d23_01_009 46028292096 1620 no yes 3544590336 ( 8%) 37547424 ( 0%) d23_01_010 46028292096 1620 no yes 3542993920 ( 8%) 37865728 ( 0%) d23_01_011 46028292096 1620 no yes 3542859776 ( 8%) 37889408 ( 0%) d23_01_012 46028292096 1620 no yes 3542452224 ( 8%) 37721440 ( 0%) d23_01_013 46028292096 1620 no yes 3542286336 ( 8%) 37797824 ( 0%) d23_01_014 46028292096 1620 no yes 3543352320 ( 8%) 37647456 ( 0%) d23_01_015 46028292096 1620 no yes 3542906880 ( 8%) 37776960 ( 0%) d23_01_016 46028292096 1620 no yes 3540386816 ( 8%) 37521632 ( 0%) d23_01_017 46028292096 1620 no yes 3543212032 ( 8%) 37568480 ( 0%) d23_01_018 46028292096 1620 no yes 3542416384 ( 8%) 37467648 ( 0%) d23_01_019 46028292096 1620 no yes 3542659072 ( 8%) 37865344 ( 0%) d23_01_020 46028292096 1620 no yes 3542518784 ( 8%) 37623840 ( 0%) d23_01_021 46028292096 1620 no yes 3543202816 ( 8%) 37630848 ( 0%) d23_01_022 46028292096 1620 no yes 3544535040 ( 8%) 37723968 ( 0%) d23_01_023 46028292096 1620 no yes 3543248896 ( 8%) 37542656 ( 0%) d23_01_024 46028292096 1620 no yes 3541811200 ( 8%) 37775360 ( 0%) d23_01_025 46028292096 1620 no yes 3544839168 ( 8%) 37887744 ( 0%) d23_01_026 46028292096 1620 no yes 3542474752 ( 8%) 37820672 ( 0%) d23_01_027 46028292096 1620 no yes 3542050816 ( 8%) 37847296 ( 0%) d23_01_028 46028292096 1620 no yes 3540822016 ( 8%) 37578400 ( 0%) d23_01_029 46028292096 1620 no yes 3542011904 ( 8%) 37423328 ( 0%) d23_01_030 46028292096 1620 no yes 3542572032 ( 8%) 37751840 ( 0%) d23_01_031 46028292096 1620 no yes 3541582848 ( 8%) 37648896 ( 0%) d23_01_032 46028292096 1620 no yes 3542650880 ( 8%) 37715840 ( 0%) d23_01_033 46028292096 1620 no yes 3542251520 ( 8%) 37598432 ( 0%) d23_01_034 46028292096 1620 no yes 3542195200 ( 8%) 37582944 ( 0%) d23_01_035 46028292096 1620 no yes 3541298176 ( 8%) 37694848 ( 0%) d23_01_036 46028292096 1620 no yes 3541215232 ( 8%) 37869568 ( 0%) d23_01_037 46028292096 1620 no yes 3542111232 ( 8%) 37601088 ( 0%) d23_01_038 46028292096 1620 no yes 3541210112 ( 8%) 37474400 ( 0%) d23_01_039 46028292096 1620 no yes 3540457472 ( 8%) 37654656 ( 0%) d23_01_040 46028292096 1620 no yes 3541776384 ( 8%) 37645760 ( 0%) d23_01_041 46028292096 1620 no yes 3542624256 ( 8%) 37798880 ( 0%) d23_01_042 46028292096 1620 no yes 3541653504 ( 8%) 37595936 ( 0%) d23_01_043 46028292096 1620 no yes 3540583424 ( 8%) 37751936 ( 0%) d23_01_044 46028292096 1620 no yes 3542136832 ( 8%) 37793536 ( 0%) d23_01_045 46028292096 1620 no yes 3543443456 ( 8%) 37683872 ( 0%) d23_01_046 46028292096 1620 no yes 3540705280 ( 8%) 37896096 ( 0%) d23_01_047 46028292096 1620 no yes 3541550080 ( 8%) 37577760 ( 0%) d23_01_048 46028292096 1620 no yes 3542068224 ( 8%) 37724960 ( 0%) d23_01_049 46028292096 1620 no yes 3544568832 ( 8%) 37687264 ( 0%) d23_01_050 46028292096 1620 no yes 3543891968 ( 8%) 37737824 ( 0%) d23_01_051 46028292096 1620 no yes 3541944320 ( 8%) 37787904 ( 0%) d23_01_052 46028292096 1620 no yes 3542128640 ( 8%) 37960704 ( 0%) d23_01_053 46028292096 1620 no yes 3542494208 ( 8%) 37823104 ( 0%) d23_01_054 46028292096 1620 no yes 3541776384 ( 8%) 37652064 ( 0%) d23_01_055 46028292096 1620 no yes 3543655424 ( 8%) 37802656 ( 0%) d23_01_056 46028292096 1620 no yes 3541664768 ( 8%) 37694272 ( 0%) d23_01_057 46028292096 1620 no yes 3542197248 ( 8%) 37798272 ( 0%) d23_01_058 46028292096 1620 no yes 3543078912 ( 8%) 37740448 ( 0%) d23_01_059 46028292096 1620 no yes 3544783872 ( 8%) 37741248 ( 0%) d23_01_060 46028292096 1620 no yes 3542276096 ( 8%) 37818304 ( 0%) d23_01_061 46028292096 1620 no yes 3543452672 ( 8%) 37727104 ( 0%) d23_01_062 46028292096 1620 no yes 3543225344 ( 8%) 37754720 ( 0%) d23_01_063 46028292096 1620 no yes 3543173120 ( 8%) 37685280 ( 0%) d23_01_064 46028292096 1620 no yes 3541703680 ( 8%) 37711424 ( 0%) d23_01_065 46028292096 1620 no yes 3541797888 ( 8%) 37836992 ( 0%) d23_01_066 46028292096 1620 no yes 3542709248 ( 8%) 37780864 ( 0%) d23_01_067 46028292096 1620 no yes 3542996992 ( 8%) 37798976 ( 0%) d23_01_068 46028292096 1620 no yes 3542989824 ( 8%) 37672352 ( 0%) d23_01_069 46028292096 1620 no yes 3542004736 ( 8%) 37688608 ( 0%) d23_01_070 46028292096 1620 no yes 3541458944 ( 8%) 37648320 ( 0%) d23_01_071 46028292096 1620 no yes 3542049792 ( 8%) 37874368 ( 0%) d23_01_072 46028292096 1620 no yes 3541520384 ( 8%) 37650368 ( 0%) d23_01_073 46028292096 1620 no yes 3542274048 ( 8%) 37759776 ( 0%) d23_01_074 46028292096 1620 no yes 3541511168 ( 8%) 37569472 ( 0%) d23_01_075 46028292096 1620 no yes 3544001536 ( 8%) 37685952 ( 0%) d23_01_076 46028292096 1620 no yes 3543203840 ( 8%) 37690880 ( 0%) d23_01_077 46028292096 1620 no yes 3541925888 ( 8%) 37710848 ( 0%) d23_01_078 46028292096 1620 no yes 3543930880 ( 8%) 37588672 ( 0%) d23_01_079 46028292096 1620 no yes 3541520384 ( 8%) 37626432 ( 0%) d23_01_080 46028292096 1620 no yes 3541615616 ( 8%) 37796576 ( 0%) d23_01_081 46028292096 1620 no yes 3542212608 ( 8%) 37773056 ( 0%) d23_01_082 46028292096 1620 no yes 3541496832 ( 8%) 37863200 ( 0%) d23_01_083 46028292096 1620 no yes 3541881856 ( 8%) 37822016 ( 0%) d23_01_084 46028292096 1620 no yes 3543436288 ( 8%) 37838144 ( 0%) d23_02_001 46028292096 1620 no yes 3543580672 ( 8%) 37784480 ( 0%) d23_02_002 46028292096 1620 no yes 3541958656 ( 8%) 38029312 ( 0%) d23_02_003 46028292096 1620 no yes 3542037504 ( 8%) 37781888 ( 0%) d23_02_004 46028292096 1620 no yes 3541141504 ( 8%) 37535936 ( 0%) d23_02_005 46028292096 1620 no yes 3541710848 ( 8%) 37585504 ( 0%) d23_02_006 46028292096 1620 no yes 3542758400 ( 8%) 37699968 ( 0%) d23_02_007 46028292096 1620 no yes 3541051392 ( 8%) 37609824 ( 0%) d23_02_008 46028292096 1620 no yes 3541925888 ( 8%) 37791872 ( 0%) d23_02_009 46028292096 1620 no yes 3542461440 ( 8%) 37854464 ( 0%) d23_02_010 46028292096 1620 no yes 3544000512 ( 8%) 37642048 ( 0%) d23_02_011 46028292096 1620 no yes 3542000640 ( 8%) 37811840 ( 0%) d23_02_012 46028292096 1620 no yes 3543025664 ( 8%) 37802784 ( 0%) d23_02_013 46028292096 1620 no yes 3541744640 ( 8%) 37776608 ( 0%) d23_02_014 46028292096 1620 no yes 3542261760 ( 8%) 37699648 ( 0%) d23_02_015 46028292096 1620 no yes 3542729728 ( 8%) 37690944 ( 0%) d23_02_016 46028292096 1620 no yes 3543721984 ( 8%) 37657472 ( 0%) d23_02_017 46028292096 1620 no yes 3540802560 ( 8%) 37531328 ( 0%) d23_02_018 46028292096 1620 no yes 3542657024 ( 8%) 37860768 ( 0%) d23_02_019 46028292096 1620 no yes 3543438336 ( 8%) 37573760 ( 0%) d23_02_020 46028292096 1620 no yes 3543243776 ( 8%) 37662976 ( 0%) d23_02_021 46028292096 1620 no yes 46028197888 (100%) 32576 ( 0%) * d23_02_022 46028292096 1620 no yes 3543544832 ( 8%) 37620160 ( 0%) d23_02_023 46028292096 1620 no yes 3540716544 ( 8%) 37649536 ( 0%) d23_02_024 46028292096 1620 no yes 3542181888 ( 8%) 37553760 ( 0%) d23_02_025 46028292096 1620 no yes 3540486144 ( 8%) 37529376 ( 0%) d23_02_026 46028292096 1620 no yes 3541583872 ( 8%) 37833792 ( 0%) d23_02_027 46028292096 1620 no yes 3542169600 ( 8%) 37778912 ( 0%) d23_02_028 46028292096 1620 no yes 3541048320 ( 8%) 37696864 ( 0%) d23_02_029 46028292096 1620 no yes 3542336512 ( 8%) 37859264 ( 0%) d23_02_030 46028292096 1620 no yes 3542102016 ( 8%) 38059168 ( 0%) d23_02_031 46028292096 1620 no yes 3541311488 ( 8%) 37784480 ( 0%) d23_02_032 46028292096 1620 no yes 3542036480 ( 8%) 37783456 ( 0%) d23_02_033 46028292096 1620 no yes 3541478400 ( 8%) 37753792 ( 0%) d23_02_034 46028292096 1620 no yes 3540772864 ( 8%) 37725312 ( 0%) d23_02_035 46028292096 1620 no yes 3541840896 ( 8%) 37709664 ( 0%) d23_02_036 46028292096 1620 no yes 3542415360 ( 8%) 37580448 ( 0%) d23_02_037 46028292096 1620 no yes 3542515712 ( 8%) 37587808 ( 0%) d23_02_038 46028292096 1620 no yes 3541250048 ( 8%) 37550976 ( 0%) d23_02_039 46028292096 1620 no yes 3542627328 ( 8%) 37389952 ( 0%) d23_02_040 46028292096 1620 no yes 3541750784 ( 8%) 37709216 ( 0%) d23_02_041 46028292096 1620 no yes 3542558720 ( 8%) 37760288 ( 0%) d23_02_042 46028292096 1620 no yes 3540994048 ( 8%) 37491680 ( 0%) d23_02_043 46028292096 1620 no yes 3542491136 ( 8%) 37768576 ( 0%) d23_02_044 46028292096 1620 no yes 3542805504 ( 8%) 37638144 ( 0%) d23_02_045 46028292096 1620 no yes 3540658176 ( 8%) 37613120 ( 0%) d23_02_046 46028292096 1620 no yes 3543098368 ( 8%) 37920320 ( 0%) d23_02_047 46028292096 1620 no yes 3543087104 ( 8%) 37590432 ( 0%) d23_02_048 46028292096 1620 no yes 3541494784 ( 8%) 37468224 ( 0%) d23_02_049 46028292096 1620 no yes 3541920768 ( 8%) 37675968 ( 0%) d23_02_050 46028292096 1620 no yes 3542463488 ( 8%) 37670816 ( 0%) d23_02_051 46028292096 1620 no yes 3542427648 ( 8%) 37678176 ( 0%) d23_02_052 46028292096 1620 no yes 3539824640 ( 8%) 37590176 ( 0%) d23_02_053 46028292096 1620 no yes 3542251520 ( 8%) 37835200 ( 0%) d23_02_054 46028292096 1620 no yes 3541064704 ( 8%) 37636224 ( 0%) d23_02_055 46028292096 1620 no yes 3540130816 ( 8%) 37703360 ( 0%) d23_02_056 46028292096 1620 no yes 3545320448 ( 8%) 37767712 ( 0%) d23_02_057 46028292096 1620 no yes 3543144448 ( 8%) 37658208 ( 0%) d23_02_058 46028292096 1620 no yes 3541233664 ( 8%) 37720640 ( 0%) d23_02_059 46028292096 1620 no yes 3541435392 ( 8%) 37680896 ( 0%) d23_02_060 46028292096 1620 no yes 3542780928 ( 8%) 37567360 ( 0%) d23_02_061 46028292096 1620 no yes 3542073344 ( 8%) 37420992 ( 0%) d23_02_062 46028292096 1620 no yes 3543192576 ( 8%) 37575232 ( 0%) d23_02_063 46028292096 1620 no yes 3542048768 ( 8%) 37696192 ( 0%) d23_02_064 46028292096 1620 no yes 3541509120 ( 8%) 37536224 ( 0%) d23_02_065 46028292096 1620 no yes 3542299648 ( 8%) 37648736 ( 0%) d23_02_066 46028292096 1620 no yes 3541864448 ( 8%) 37710976 ( 0%) d23_02_067 46028292096 1620 no yes 3538870272 ( 8%) 37532288 ( 0%) d23_02_068 46028292096 1620 no yes 3541297152 ( 8%) 37829184 ( 0%) d23_02_069 46028292096 1620 no yes 3542179840 ( 8%) 37583424 ( 0%) d23_02_070 46028292096 1620 no yes 3541616640 ( 8%) 37742080 ( 0%) d23_02_071 46028292096 1620 no yes 3541706752 ( 8%) 37687328 ( 0%) d23_02_072 46028292096 1620 no yes 3542240256 ( 8%) 37582112 ( 0%) d23_02_073 46028292096 1620 no yes 3542102016 ( 8%) 37752736 ( 0%) d23_02_074 46028292096 1620 no yes 3542754304 ( 8%) 37687456 ( 0%) d23_02_075 46028292096 1620 no yes 3542194176 ( 8%) 37729344 ( 0%) d23_02_076 46028292096 1620 no yes 3542798336 ( 8%) 37672096 ( 0%) d23_02_077 46028292096 1620 no yes 3543918592 ( 8%) 37709024 ( 0%) d23_02_078 46028292096 1620 no yes 3540992000 ( 8%) 37540992 ( 0%) d23_02_079 46028292096 1620 no yes 3543330816 ( 8%) 37573888 ( 0%) d23_02_080 46028292096 1620 no yes 3543735296 ( 8%) 37818208 ( 0%) d23_02_081 46028292096 1620 no yes 3542054912 ( 8%) 37698016 ( 0%) d23_02_082 46028292096 1620 no yes 3540747264 ( 8%) 37643776 ( 0%) d23_02_083 46028292096 1620 no yes 3542846464 ( 8%) 37715680 ( 0%) d23_02_084 46028292096 1620 no yes 3542061056 ( 8%) 37703904 ( 0%) ------------- -------------------- ------------------- (pool total) 7686724780032 591558139904 ( 8%) 6295334976 ( 0%) ============= ==================== =================== (data) 7686724780032 591558139904 ( 8%) 6295334976 ( 0%) (metadata) 2516582400 783850240 ( 31%) 37303168 ( 1%) ============= ==================== =================== (total) 7689241362432 592341990144 ( 8%) 6332638144 ( 0%) Inode Information ----------------- Number of used inodes: 30247288 Number of free inodes: 11695752 Number of allocated inodes: 41943040 Maximum number of inodes: 41943040 -------------- next part -------------- flag value description ------------------- ------------------------ ----------------------------------- -f 8192 Minimum fragment size in bytes (system pool) 32768 Minimum fragment size in bytes (other pools) -i 512 Inode size in bytes -I 32768 Indirect block size in bytes -m 2 Default number of metadata replicas -M 2 Maximum number of metadata replicas -r 1 Default number of data replicas -R 2 Maximum number of data replicas -j cluster Block allocation type -D nfs4 File locking semantics in effect -k posix ACL semantics in effect -n 5000 Estimated number of nodes that will mount file system -B 262144 Block size (system pool) 1048576 Block size (other pools) -Q user;group;fileset Quotas accounting enabled user;group;fileset Quotas enforced none Default quotas enabled --perfileset-quota no Per-fileset quota enforcement --filesetdf no Fileset df enabled? -V 13.23 (3.5.0.7) File system version --create-time Fri Dec 6 15:23:28 2013 File system creation time -z no Is DMAPI enabled? -L 8388608 Logfile size -E yes Exact mtime mount option -S no Suppress atime mount option -K whenpossible Strict replica allocation option --fastea yes Fast external attributes enabled? --encryption no Encryption enabled? --inode-limit 41943040 Maximum number of inodes --log-replicas 0 Number of log replicas --is4KAligned no is4KAligned? --rapid-repair no rapidRepair enabled? --write-cache-threshold 0 HAWC Threshold (max 65536) -P system;sp_1620 Disk storage pools in file system -d m01_12_093;m01_12_094;m01_12_095;m01_12_096;m02_22_093;m02_22_094;m02_22_095;m02_22_096;m01_12_061;m01_12_062;m01_12_063;m01_12_064;m01_12_065;m01_12_066;m01_12_067; -d m01_12_068;m02_22_061;m02_22_062;m02_22_063;m02_22_064;m02_22_065;m02_22_066;m02_22_067;m02_22_068;m01_12_081;m01_12_082;m01_12_083;m01_12_084;m01_12_085;m01_12_086; -d m01_12_087;m01_12_088;m01_12_089;m01_12_090;m01_12_091;m01_12_092;m02_22_081;m02_22_082;m02_22_083;m02_22_084;m02_22_085;m02_22_086;m02_22_087;m02_22_088;m02_22_089; -d m02_22_090;m02_22_091;m02_22_092;d23_01_001;d23_01_002;d23_01_003;d23_01_004;d23_01_005;d23_01_006;d23_01_007;d23_01_008;d23_01_009;d23_01_010;d23_01_011;d23_01_012; -d d23_01_013;d23_01_014;d23_01_015;d23_01_016;d23_01_017;d23_01_018;d23_01_019;d23_01_020;d23_01_021;d23_01_022;d23_01_023;d23_01_024;d23_01_025;d23_01_026;d23_01_027; -d d23_01_028;d23_01_029;d23_01_030;d23_01_031;d23_01_032;d23_01_033;d23_01_034;d23_01_035;d23_01_036;d23_01_037;d23_01_038;d23_01_039;d23_01_040;d23_01_041;d23_01_042; -d d23_01_043;d23_01_044;d23_01_045;d23_01_046;d23_01_047;d23_01_048;d23_01_049;d23_01_050;d23_01_051;d23_01_052;d23_01_053;d23_01_054;d23_01_055;d23_01_056;d23_01_057; -d d23_01_058;d23_01_059;d23_01_060;d23_01_061;d23_01_062;d23_01_063;d23_01_064;d23_01_065;d23_01_066;d23_01_067;d23_01_068;d23_01_069;d23_01_070;d23_01_071;d23_01_072; -d d23_01_073;d23_01_074;d23_01_075;d23_01_076;d23_01_077;d23_01_078;d23_01_079;d23_01_080;d23_01_081;d23_01_082;d23_01_083;d23_01_084;d23_02_001;d23_02_002;d23_02_003; -d d23_02_004;d23_02_005;d23_02_006;d23_02_007;d23_02_008;d23_02_009;d23_02_010;d23_02_011;d23_02_012;d23_02_013;d23_02_014;d23_02_015;d23_02_016;d23_02_017;d23_02_018; -d d23_02_019;d23_02_020;d23_02_021;d23_02_022;d23_02_023;d23_02_024;d23_02_025;d23_02_026;d23_02_027;d23_02_028;d23_02_029;d23_02_030;d23_02_031;d23_02_032;d23_02_033; -d d23_02_034;d23_02_035;d23_02_036;d23_02_037;d23_02_038;d23_02_039;d23_02_040;d23_02_041;d23_02_042;d23_02_043;d23_02_044;d23_02_045;d23_02_046;d23_02_047;d23_02_048; -d d23_02_049;d23_02_050;d23_02_051;d23_02_052;d23_02_053;d23_02_054;d23_02_055;d23_02_056;d23_02_057;d23_02_058;d23_02_059;d23_02_060;d23_02_061;d23_02_062;d23_02_063; -d d23_02_064;d23_02_065;d23_02_066;d23_02_067;d23_02_068;d23_02_069;d23_02_070;d23_02_071;d23_02_072;d23_02_073;d23_02_074;d23_02_075;d23_02_076;d23_02_077;d23_02_078; -d d23_02_079;d23_02_080;d23_02_081;d23_02_082;d23_02_083;d23_02_084 Disks in file system -A no Automatic mount option -o nodev,nosuid Additional mount options -T /gpfsm/dnb03 Default mount point --mount-priority 0 Mount priority From jfosburg at mdanderson.org Fri Mar 24 18:03:22 2017 From: jfosburg at mdanderson.org (Fosburgh,Jonathan) Date: Fri, 24 Mar 2017 18:03:22 +0000 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: <1490378600.1901.4.camel@mdanderson.org> 7PB filesystem and only 28 million inodes in use? What is your average file size? Our large filesystem is 7.5P (currently 71% used) with over 1 billion inodes in use. -- Jonathan Fosburgh Principal Application Systems Analyst Storage Team IT Operations jfosburg at mdanderson.org (713) 745-9346 -----Original Message----- Date: Fri, 24 Mar 2017 13:58:18 -0400 Subject: Re: [gpfsug-discuss] strange waiters + filesystem deadlock To: gpfsug-discuss at spectrumscale.org Reply-to: gpfsug main discussion list From: Aaron Knister > I feel a little awkward about posting wlists of IP's and hostnames on the mailing list (even though they're all internal) but I'm happy to send to you directly. I've attached both an lsfs and an mmdf output of the fs in question here since that may be useful for others to see. Just a note about disk d23_02_021-- it's been evacuated for several weeks now due to a hardware issue in the disk enclosure. The fs is rather full percentage wise (93%) but in terms of capacity there's a good amount free. 93% full of a 7PB filesystem still leaves 551T. Metadata, as you'll see, is 31% free (roughly 800GB). The fs has 40M inodes allocated and 12M free. -Aaron On 3/24/17 1:41 PM, Sven Oehme wrote: ok, that seems a different problem then i was thinking. can you send output of mmlscluster, mmlsconfig, mmlsfs all ? also are you getting close to fill grade on inodes or capacity on any of the filesystems ? sven On Fri, Mar 24, 2017 at 10:34 AM Aaron Knister > wrote: Here's the screenshot from the other node with the high cpu utilization. On 3/24/17 1:32 PM, Aaron Knister wrote: > heh, yep we're on sles :) > > here's a screenshot of the fs manager from the deadlocked filesystem. I > don't think there's an nsd server or manager node that's running full > throttle across all cpus. There is one that's got relatively high CPU > utilization though (300-400%). I'll send a screenshot of it in a sec. > > no zimon yet but we do have other tools to see cpu utilization. > > -Aaron > > On 3/24/17 1:22 PM, Sven Oehme wrote: >> you must be on sles as this segfaults only on sles to my knowledge :-) >> >> i am looking for a NSD or manager node in your cluster that runs at 100% >> cpu usage. >> >> do you have zimon deployed to look at cpu utilization across your nodes ? >> >> sven >> >> >> >> On Fri, Mar 24, 2017 at 10:08 AM Aaron Knister >> >> wrote: >> >> Hi Sven, >> >> Which NSD server should I run top on, the fs manager? If so the >> CPU load >> is about 155%. I'm working on perf top but not off to a great >> start... >> >> # perf top >> PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% [1000Hz >> cycles], (all, 28 CPUs) >> >> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >> >> Segmentation fault >> >> -Aaron >> >> On 3/24/17 1:04 PM, Sven Oehme wrote: >> > while this is happening run top and see if there is very high cpu >> > utilization at this time on the NSD Server. >> > >> > if there is , run perf top (you might need to install perf >> command) and >> > see if the top cpu contender is a spinlock . if so send a >> screenshot of >> > perf top as i may know what that is and how to fix. >> > >> > sven >> > >> > >> > On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister >> > >> > >> >>> wrote: >> > >> > Since yesterday morning we've noticed some deadlocks on one >> of our >> > filesystems that seem to be triggered by writing to it. The >> waiters on >> > the clients look like this: >> > >> > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, >> SyncHandlerThread: >> > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) >> (InodeFlushCondVar), reason >> > 'waiting for the flush flag to commit metadata' >> > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, >> > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 >> (0x7FFFDAC7FE28) >> > (MsgRecordCondvar), reason 'RPC wait' for >> allocMsgTypeRelinquishRegion >> > on node 10.1.52.33 >> > 0x197EE70 ( 6776) waiting 0.000198354 seconds, >> > FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 >> > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for >> > allocMsgTypeRequestRegion on node 10.1.52.33 >> > >> > (10.1.52.33/c0n3271 >> is the fs manager >> > for the filesystem in question) >> > >> > there's a single process running on this node writing to the >> filesystem >> > in question (well, trying to write, it's been blocked doing >> nothing for >> > half an hour now). There are ~10 other client nodes in this >> situation >> > right now. We had many more last night before the problem >> seemed to >> > disappear in the early hours of the morning and now its back. >> > >> > Waiters on the fs manager look like this. While the >> individual waiter is >> > short it's a near constant stream: >> > >> > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg >> handler >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) >> > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg >> handler >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) >> > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg >> handler >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) >> > >> > I don't know if this data point is useful but both yesterday >> and today >> > the metadata NSDs for this filesystem have had a constant >> aggregate >> > stream of 25MB/s 4kop/s reads during each episode (very low >> latency >> > though so I don't believe the storage is a bottleneck here). >> Writes are >> > only a few hundred ops and didn't strike me as odd. >> > >> > I have a PMR open for this but I'm curious if folks have >> seen this in >> > the wild and what it might mean. >> > >> > -Aaron >> > >> > -- >> > Aaron Knister >> > NASA Center for Climate Simulation (Code 606.2) >> > Goddard Space Flight Center >> > (301) 286-2776 >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > >> > >> > >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > >> >> -- >> Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) >> Goddard Space Flight Center >> (301) 286-2776 >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss The information contained in this e-mail message may be privileged, confidential, and/or protected from disclosure. This e-mail message may contain protected health information (PHI); dissemination of PHI should comply with applicable federal and state laws. If you are not the intended recipient, or an authorized representative of the intended recipient, any further review, disclosure, use, dissemination, distribution, or copying of this message or any attachment (or the information contained therein) is strictly prohibited. If you think that you have received this e-mail message in error, please notify the sender by return e-mail and delete all references to it and its contents from your systems. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Fri Mar 24 18:05:26 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 24 Mar 2017 14:05:26 -0400 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: <1490378600.1901.4.camel@mdanderson.org> References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> <1490378600.1901.4.camel@mdanderson.org> Message-ID: <5e630e9a-5f6f-5def-7e92-5bc7c6c01a4f@nasa.gov> It's large, I do know that much. I'll defer to one of our other storage admins. Jordan, do you have that number handy? -Aaron On 3/24/17 2:03 PM, Fosburgh,Jonathan wrote: > 7PB filesystem and only 28 million inodes in use? What is your average > file size? Our large filesystem is 7.5P (currently 71% used) with over 1 > billion inodes in use. > > -- > > Jonathan Fosburgh > Principal Application Systems Analyst > Storage Team > IT Operations > jfosburg at mdanderson.org > (713) 745-9346 > > -----Original Message----- > > *Date*: Fri, 24 Mar 2017 13:58:18 -0400 > *Subject*: Re: [gpfsug-discuss] strange waiters + filesystem deadlock > *To*: gpfsug-discuss at spectrumscale.org > > Reply-to: gpfsug main discussion list > *From*: Aaron Knister > > > I feel a little awkward about posting wlists of IP's and hostnames on > the mailing list (even though they're all internal) but I'm happy to > send to you directly. I've attached both an lsfs and an mmdf output of > the fs in question here since that may be useful for others to see. Just > a note about disk d23_02_021-- it's been evacuated for several weeks now > due to a hardware issue in the disk enclosure. > > The fs is rather full percentage wise (93%) but in terms of capacity > there's a good amount free. 93% full of a 7PB filesystem still leaves > 551T. Metadata, as you'll see, is 31% free (roughly 800GB). > > The fs has 40M inodes allocated and 12M free. > > -Aaron > > On 3/24/17 1:41 PM, Sven Oehme wrote: >> ok, that seems a different problem then i was thinking. can you send >> output of mmlscluster, mmlsconfig, mmlsfs all ? also are you getting >> close to fill grade on inodes or capacity on any of the filesystems ? >> sven On Fri, Mar 24, 2017 at 10:34 AM Aaron Knister >> >> > wrote: Here's the screenshot from >> the other node with the high cpu utilization. On 3/24/17 1:32 PM, >> Aaron Knister wrote: > heh, yep we're on sles :) > > here's a >> screenshot of the fs manager from the deadlocked filesystem. I > don't >> think there's an nsd server or manager node that's running full > >> throttle across all cpus. There is one that's got relatively high CPU >> > utilization though (300-400%). I'll send a screenshot of it in a >> sec. > > no zimon yet but we do have other tools to see cpu >> utilization. > > -Aaron > > On 3/24/17 1:22 PM, Sven Oehme wrote: >> >> you must be on sles as this segfaults only on sles to my knowledge :-) >> >> >> i am looking for a NSD or manager node in your cluster that runs >> at 100% >> cpu usage. >> >> do you have zimon deployed to look at cpu >> utilization across your nodes ? >> >> sven >> >> >> >> On Fri, Mar 24, >> 2017 at 10:08 AM Aaron Knister > >> >> >> >> wrote: >> >> Hi Sven, >> >> Which NSD server should I run top on, the >> fs manager? If so the >> CPU load >> is about 155%. I'm working on >> perf top but not off to a great >> start... >> >> # perf top >> >> PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% [1000Hz >> cycles], >> (all, 28 CPUs) >> >> >> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >> >> >> Segmentation fault >> >> -Aaron >> >> On 3/24/17 1:04 PM, Sven >> Oehme wrote: >> > while this is happening run top and see if there is >> very high cpu >> > utilization at this time on the NSD Server. >> > >> >> > if there is , run perf top (you might need to install perf >> >> command) and >> > see if the top cpu contender is a spinlock . if so >> send a >> screenshot of >> > perf top as i may know what that is and >> how to fix. >> > >> > sven >> > >> > >> > On Fri, Mar 24, 2017 at 9:43 >> AM Aaron Knister >> > >> > >> >> > >> >> > >>> wrote: >> > >> > Since yesterday >> morning we've noticed some deadlocks on one >> of our >> > filesystems >> that seem to be triggered by writing to it. The >> waiters on >> > the >> clients look like this: >> > >> > 0x19450B0 ( 6730) waiting >> 2063.294589599 seconds, >> SyncHandlerThread: >> > on ThCond >> 0x1802585CB10 (0xFFFFC9002585CB10) >> (InodeFlushCondVar), reason >> > >> 'waiting for the flush flag to commit metadata' >> > 0x7FFFDA65E200 ( >> 22850) waiting 0.000246257 seconds, >> > AllocReduceHelperThread: on >> ThCond 0x7FFFDAC7FE28 >> (0x7FFFDAC7FE28) >> > (MsgRecordCondvar), >> reason 'RPC wait' for >> allocMsgTypeRelinquishRegion >> > on node >> 10.1.52.33 >> > 0x197EE70 ( 6776) waiting 0.000198354 >> seconds, >> > FileBlockWriteFetchHandlerThread: on ThCond >> 0x7FFFF00CD598 >> > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC >> wait' for >> > allocMsgTypeRequestRegion on node 10.1.52.33 >> >> > >> > (10.1.52.33/c0n3271 >> >> is the fs >> manager >> > for the filesystem in question) >> > >> > there's a >> single process running on this node writing to the >> filesystem >> > >> in question (well, trying to write, it's been blocked doing >> nothing >> for >> > half an hour now). There are ~10 other client nodes in this >> >> situation >> > right now. We had many more last night before the >> problem >> seemed to >> > disappear in the early hours of the morning >> and now its back. >> > >> > Waiters on the fs manager look like this. >> While the >> individual waiter is >> > short it's a near constant >> stream: >> > >> > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, >> Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex >> 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > >> 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg >> handler >> >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 >> > >> (0xFFFFC9002163A2E0) (AllocManagerMutex) >> > 0x7FFF91C10080 ( 14723) >> waiting 0.000959649 seconds, Msg >> handler >> > >> allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFFB03C2910 ( >> 12636) waiting 0.000769611 seconds, Msg >> handler >> > >> allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF8C092850 ( >> 18215) waiting 0.000682275 seconds, Msg >> handler >> > >> allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 >> > >> (0xFFFFC9002163A2E0) (AllocManagerMutex) >> > 0x7FFF9423F730 ( 12652) >> waiting 0.000641915 seconds, Msg >> handler >> > >> allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF9422D770 ( >> 12625) waiting 0.000494256 seconds, Msg >> handler >> > >> allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF9423E310 ( >> 12651) waiting 0.000437760 seconds, Msg >> handler >> > >> allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 >> > >> (0xFFFFC9002163A2E0) (AllocManagerMutex) >> > >> > I don't know if >> this data point is useful but both yesterday >> and today >> > the >> metadata NSDs for this filesystem have had a constant >> aggregate >> >> > stream of 25MB/s 4kop/s reads during each episode (very low >> >> latency >> > though so I don't believe the storage is a bottleneck >> here). >> Writes are >> > only a few hundred ops and didn't strike me >> as odd. >> > >> > I have a PMR open for this but I'm curious if folks >> have >> seen this in >> > the wild and what it might mean. >> > >> > >> -Aaron >> > >> > -- >> > Aaron Knister >> > NASA Center for Climate >> Simulation (Code 606.2) >> > Goddard Space Flight Center >> > (301) >> 286-2776 >> >> > >> _______________________________________________ >> > gpfsug-discuss >> mailing list >> > gpfsug-discuss at spectrumscale.org >> >> >> >> > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > >> > >> > >> > >> _______________________________________________ >> > gpfsug-discuss >> mailing list >> > gpfsug-discuss at spectrumscale.org >> >> > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > >> >> -- >> >> Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) >> >> Goddard Space Flight Center >> (301) 286-2776 >> >> >> _______________________________________________ >> gpfsug-discuss >> mailing list >> gpfsug-discuss at spectrumscale.org >> >> >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> >> >> _______________________________________________ >> gpfsug-discuss >> mailing list >> gpfsug-discuss at spectrumscale.org >> >> >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > > > >> _______________________________________________ > gpfsug-discuss >> mailing list > gpfsug-discuss at spectrumscale.org >> > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight >> Center (301) 286-2776 >> _______________________________________________ gpfsug-discuss mailing >> list gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> _______________________________________________ gpfsug-discuss mailing >> list gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > The information contained in this e-mail message may be privileged, > confidential, and/or protected from disclosure. This e-mail message may > contain protected health information (PHI); dissemination of PHI should > comply with applicable federal and state laws. If you are not the > intended recipient, or an authorized representative of the intended > recipient, any further review, disclosure, use, dissemination, > distribution, or copying of this message or any attachment (or the > information contained therein) is strictly prohibited. If you think that > you have received this e-mail message in error, please notify the sender > by return e-mail and delete all references to it and its contents from > your systems. > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From oehmes at gmail.com Fri Mar 24 18:05:58 2017 From: oehmes at gmail.com (Sven Oehme) Date: Fri, 24 Mar 2017 18:05:58 +0000 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: was this filesystem creates with -n 5000 ? or was that changed later with mmchfs ? please send the mmlsconfig/mmlscluster output to me at oehmes at us.ibm.com On Fri, Mar 24, 2017 at 10:58 AM Aaron Knister wrote: > I feel a little awkward about posting wlists of IP's and hostnames on > the mailing list (even though they're all internal) but I'm happy to > send to you directly. I've attached both an lsfs and an mmdf output of > the fs in question here since that may be useful for others to see. Just > a note about disk d23_02_021-- it's been evacuated for several weeks now > due to a hardware issue in the disk enclosure. > > The fs is rather full percentage wise (93%) but in terms of capacity > there's a good amount free. 93% full of a 7PB filesystem still leaves > 551T. Metadata, as you'll see, is 31% free (roughly 800GB). > > The fs has 40M inodes allocated and 12M free. > > -Aaron > > On 3/24/17 1:41 PM, Sven Oehme wrote: > > ok, that seems a different problem then i was thinking. > > can you send output of mmlscluster, mmlsconfig, mmlsfs all ? > > also are you getting close to fill grade on inodes or capacity on any of > > the filesystems ? > > > > sven > > > > > > On Fri, Mar 24, 2017 at 10:34 AM Aaron Knister > > wrote: > > > > Here's the screenshot from the other node with the high cpu > utilization. > > > > On 3/24/17 1:32 PM, Aaron Knister wrote: > > > heh, yep we're on sles :) > > > > > > here's a screenshot of the fs manager from the deadlocked > filesystem. I > > > don't think there's an nsd server or manager node that's running > full > > > throttle across all cpus. There is one that's got relatively high > CPU > > > utilization though (300-400%). I'll send a screenshot of it in a > sec. > > > > > > no zimon yet but we do have other tools to see cpu utilization. > > > > > > -Aaron > > > > > > On 3/24/17 1:22 PM, Sven Oehme wrote: > > >> you must be on sles as this segfaults only on sles to my > knowledge :-) > > >> > > >> i am looking for a NSD or manager node in your cluster that runs > at 100% > > >> cpu usage. > > >> > > >> do you have zimon deployed to look at cpu utilization across your > nodes ? > > >> > > >> sven > > >> > > >> > > >> > > >> On Fri, Mar 24, 2017 at 10:08 AM Aaron Knister < > aaron.s.knister at nasa.gov > > >> >> > wrote: > > >> > > >> Hi Sven, > > >> > > >> Which NSD server should I run top on, the fs manager? If so > the > > >> CPU load > > >> is about 155%. I'm working on perf top but not off to a great > > >> start... > > >> > > >> # perf top > > >> PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% > [1000Hz > > >> cycles], (all, 28 CPUs) > > >> > > >> > --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > >> > > >> Segmentation fault > > >> > > >> -Aaron > > >> > > >> On 3/24/17 1:04 PM, Sven Oehme wrote: > > >> > while this is happening run top and see if there is very > high cpu > > >> > utilization at this time on the NSD Server. > > >> > > > >> > if there is , run perf top (you might need to install perf > > >> command) and > > >> > see if the top cpu contender is a spinlock . if so send a > > >> screenshot of > > >> > perf top as i may know what that is and how to fix. > > >> > > > >> > sven > > >> > > > >> > > > >> > On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister > > >> > > > > > >> > aaron.s.knister at nasa.gov> > > >> >>> > wrote: > > >> > > > >> > Since yesterday morning we've noticed some deadlocks on > one > > >> of our > > >> > filesystems that seem to be triggered by writing to it. > The > > >> waiters on > > >> > the clients look like this: > > >> > > > >> > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, > > >> SyncHandlerThread: > > >> > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) > > >> (InodeFlushCondVar), reason > > >> > 'waiting for the flush flag to commit metadata' > > >> > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, > > >> > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 > > >> (0x7FFFDAC7FE28) > > >> > (MsgRecordCondvar), reason 'RPC wait' for > > >> allocMsgTypeRelinquishRegion > > >> > on node 10.1.52.33 > > >> > 0x197EE70 ( 6776) waiting 0.000198354 seconds, > > >> > FileBlockWriteFetchHandlerThread: on ThCond > 0x7FFFF00CD598 > > >> > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' > for > > >> > allocMsgTypeRequestRegion on node 10.1.52.33 > > >> > > > >> > (10.1.52.33/c0n3271 > > > > >> is the fs manager > > >> > for the filesystem in question) > > >> > > > >> > there's a single process running on this node writing > to the > > >> filesystem > > >> > in question (well, trying to write, it's been blocked > doing > > >> nothing for > > >> > half an hour now). There are ~10 other client nodes in > this > > >> situation > > >> > right now. We had many more last night before the > problem > > >> seemed to > > >> > disappear in the early hours of the morning and now its > back. > > >> > > > >> > Waiters on the fs manager look like this. While the > > >> individual waiter is > > >> > short it's a near constant stream: > > >> > > > >> > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, > Msg > > >> handler > > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > > >> (0xFFFFC9002163A2E0) > > >> > (AllocManagerMutex) > > >> > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, > Msg > > >> handler > > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > >> > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, > Msg > > >> handler > > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > > >> (0xFFFFC9002163A2E0) > > >> > (AllocManagerMutex) > > >> > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, > Msg > > >> handler > > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > > >> (0xFFFFC9002163A2E0) > > >> > (AllocManagerMutex) > > >> > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, > Msg > > >> handler > > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > >> > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, > Msg > > >> handler > > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > > >> (0xFFFFC9002163A2E0) > > >> > (AllocManagerMutex) > > >> > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, > Msg > > >> handler > > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > > >> (0xFFFFC9002163A2E0) > > >> > (AllocManagerMutex) > > >> > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, > Msg > > >> handler > > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > >> > > > >> > I don't know if this data point is useful but both > yesterday > > >> and today > > >> > the metadata NSDs for this filesystem have had a > constant > > >> aggregate > > >> > stream of 25MB/s 4kop/s reads during each episode (very > low > > >> latency > > >> > though so I don't believe the storage is a bottleneck > here). > > >> Writes are > > >> > only a few hundred ops and didn't strike me as odd. > > >> > > > >> > I have a PMR open for this but I'm curious if folks have > > >> seen this in > > >> > the wild and what it might mean. > > >> > > > >> > -Aaron > > >> > > > >> > -- > > >> > Aaron Knister > > >> > NASA Center for Climate Simulation (Code 606.2) > > >> > Goddard Space Flight Center > > >> > (301) 286-2776 > > > > > >> > _______________________________________________ > > >> > gpfsug-discuss mailing list > > >> > gpfsug-discuss at spectrumscale.org < > http://spectrumscale.org> > > >> > > >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > >> > > > >> > > > >> > > > >> > _______________________________________________ > > >> > gpfsug-discuss mailing list > > >> > gpfsug-discuss at spectrumscale.org < > http://spectrumscale.org> > > >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > >> > > > >> > > >> -- > > >> Aaron Knister > > >> NASA Center for Climate Simulation (Code 606.2) > > >> Goddard Space Flight Center > > >> (301) 286-2776 > > >> _______________________________________________ > > >> gpfsug-discuss mailing list > > >> gpfsug-discuss at spectrumscale.org > > > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > >> > > >> > > >> > > >> _______________________________________________ > > >> gpfsug-discuss mailing list > > >> gpfsug-discuss at spectrumscale.org > > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > >> > > > > > > > > > > > > _______________________________________________ > > > gpfsug-discuss mailing list > > > gpfsug-discuss at spectrumscale.org > > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > -- > > Aaron Knister > > NASA Center for Climate Simulation (Code 606.2) > > Goddard Space Flight Center > > (301) 286-2776 > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Fri Mar 24 18:08:44 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 24 Mar 2017 14:08:44 -0400 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: <47206c08-671f-efab-d094-aa1200627b22@nasa.gov> I believe it was created with -n 5000. Here's the exact command that was used: /usr/lpp/mmfs/bin/mmcrfs dnb03 -F ./disc_mmcrnsd_dnb03.lst -T /gpfsm/dnb03 -j cluster -B 1M -n 5000 -N 20M -r1 -R2 -m2 -M2 -A no -Q yes -v yes -i 512 --metadata-block-size=256K -L 8388608 -Aaron On 3/24/17 2:05 PM, Sven Oehme wrote: > was this filesystem creates with -n 5000 ? or was that changed later > with mmchfs ? > please send the mmlsconfig/mmlscluster output to me at oehmes at us.ibm.com > > > > > On Fri, Mar 24, 2017 at 10:58 AM Aaron Knister > wrote: > > I feel a little awkward about posting wlists of IP's and hostnames on > the mailing list (even though they're all internal) but I'm happy to > send to you directly. I've attached both an lsfs and an mmdf output of > the fs in question here since that may be useful for others to see. Just > a note about disk d23_02_021-- it's been evacuated for several weeks now > due to a hardware issue in the disk enclosure. > > The fs is rather full percentage wise (93%) but in terms of capacity > there's a good amount free. 93% full of a 7PB filesystem still leaves > 551T. Metadata, as you'll see, is 31% free (roughly 800GB). > > The fs has 40M inodes allocated and 12M free. > > -Aaron > > On 3/24/17 1:41 PM, Sven Oehme wrote: > > ok, that seems a different problem then i was thinking. > > can you send output of mmlscluster, mmlsconfig, mmlsfs all ? > > also are you getting close to fill grade on inodes or capacity on any of > > the filesystems ? > > > > sven > > > > > > On Fri, Mar 24, 2017 at 10:34 AM Aaron Knister > > >> wrote: > > > > Here's the screenshot from the other node with the high cpu utilization. > > > > On 3/24/17 1:32 PM, Aaron Knister wrote: > > > heh, yep we're on sles :) > > > > > > here's a screenshot of the fs manager from the deadlocked filesystem. I > > > don't think there's an nsd server or manager node that's running full > > > throttle across all cpus. There is one that's got relatively high CPU > > > utilization though (300-400%). I'll send a screenshot of it in a sec. > > > > > > no zimon yet but we do have other tools to see cpu utilization. > > > > > > -Aaron > > > > > > On 3/24/17 1:22 PM, Sven Oehme wrote: > > >> you must be on sles as this segfaults only on sles to my knowledge :-) > > >> > > >> i am looking for a NSD or manager node in your cluster that runs at 100% > > >> cpu usage. > > >> > > >> do you have zimon deployed to look at cpu utilization across your nodes ? > > >> > > >> sven > > >> > > >> > > >> > > >> On Fri, Mar 24, 2017 at 10:08 AM Aaron Knister > > > > >> > >>> wrote: > > >> > > >> Hi Sven, > > >> > > >> Which NSD server should I run top on, the fs manager? If so the > > >> CPU load > > >> is about 155%. I'm working on perf top but not off to a great > > >> start... > > >> > > >> # perf top > > >> PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% [1000Hz > > >> cycles], (all, 28 CPUs) > > >> > > >> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > >> > > >> Segmentation fault > > >> > > >> -Aaron > > >> > > >> On 3/24/17 1:04 PM, Sven Oehme wrote: > > >> > while this is happening run top and see if there is very high cpu > > >> > utilization at this time on the NSD Server. > > >> > > > >> > if there is , run perf top (you might need to install perf > > >> command) and > > >> > see if the top cpu contender is a spinlock . if so send a > > >> screenshot of > > >> > perf top as i may know what that is and how to fix. > > >> > > > >> > sven > > >> > > > >> > > > >> > On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister > > >> > > > > > >> > > >> > > > > > >> > >>>> wrote: > > >> > > > >> > Since yesterday morning we've noticed some deadlocks on one > > >> of our > > >> > filesystems that seem to be triggered by writing to it. The > > >> waiters on > > >> > the clients look like this: > > >> > > > >> > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, > > >> SyncHandlerThread: > > >> > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) > > >> (InodeFlushCondVar), reason > > >> > 'waiting for the flush flag to commit metadata' > > >> > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, > > >> > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 > > >> (0x7FFFDAC7FE28) > > >> > (MsgRecordCondvar), reason 'RPC wait' for > > >> allocMsgTypeRelinquishRegion > > >> > on node 10.1.52.33 > > >> > 0x197EE70 ( 6776) waiting 0.000198354 seconds, > > >> > FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 > > >> > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for > > >> > allocMsgTypeRequestRegion on node 10.1.52.33 > > >> > > > >> > (10.1.52.33/c0n3271 > > > > > >> is the fs manager > > >> > for the filesystem in question) > > >> > > > >> > there's a single process running on this node writing to the > > >> filesystem > > >> > in question (well, trying to write, it's been blocked doing > > >> nothing for > > >> > half an hour now). There are ~10 other client nodes in this > > >> situation > > >> > right now. We had many more last night before the problem > > >> seemed to > > >> > disappear in the early hours of the morning and now its back. > > >> > > > >> > Waiters on the fs manager look like this. While the > > >> individual waiter is > > >> > short it's a near constant stream: > > >> > > > >> > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg > > >> handler > > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > > >> (0xFFFFC9002163A2E0) > > >> > (AllocManagerMutex) > > >> > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg > > >> handler > > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > >> > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg > > >> handler > > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > > >> (0xFFFFC9002163A2E0) > > >> > (AllocManagerMutex) > > >> > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg > > >> handler > > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > > >> (0xFFFFC9002163A2E0) > > >> > (AllocManagerMutex) > > >> > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg > > >> handler > > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > >> > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg > > >> handler > > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > > >> (0xFFFFC9002163A2E0) > > >> > (AllocManagerMutex) > > >> > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg > > >> handler > > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > > >> (0xFFFFC9002163A2E0) > > >> > (AllocManagerMutex) > > >> > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg > > >> handler > > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > >> > > > >> > I don't know if this data point is useful but both yesterday > > >> and today > > >> > the metadata NSDs for this filesystem have had a constant > > >> aggregate > > >> > stream of 25MB/s 4kop/s reads during each episode (very low > > >> latency > > >> > though so I don't believe the storage is a bottleneck here). > > >> Writes are > > >> > only a few hundred ops and didn't strike me as odd. > > >> > > > >> > I have a PMR open for this but I'm curious if folks have > > >> seen this in > > >> > the wild and what it might mean. > > >> > > > >> > -Aaron > > >> > > > >> > -- > > >> > Aaron Knister > > >> > NASA Center for Climate Simulation (Code 606.2) > > >> > Goddard Space Flight Center > > >> > (301) 286-2776 > > > > > >> > _______________________________________________ > > >> > gpfsug-discuss mailing list > > >> > gpfsug-discuss at spectrumscale.org > > >> > > >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > >> > > > >> > > > >> > > > >> > _______________________________________________ > > >> > gpfsug-discuss mailing list > > >> > gpfsug-discuss at spectrumscale.org > > > >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > >> > > > >> > > >> -- > > >> Aaron Knister > > >> NASA Center for Climate Simulation (Code 606.2) > > >> Goddard Space Flight Center > > >> (301) 286-2776 > > > >> _______________________________________________ > > >> gpfsug-discuss mailing list > > >> gpfsug-discuss at spectrumscale.org > > > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > >> > > >> > > >> > > >> _______________________________________________ > > >> gpfsug-discuss mailing list > > >> gpfsug-discuss at spectrumscale.org > > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > >> > > > > > > > > > > > > _______________________________________________ > > > gpfsug-discuss mailing list > > > gpfsug-discuss at spectrumscale.org > > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > -- > > Aaron Knister > > NASA Center for Climate Simulation (Code 606.2) > > Goddard Space Flight Center > > (301) 286-2776 > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From bbanister at jumptrading.com Fri Mar 24 18:13:33 2017 From: bbanister at jumptrading.com (Bryan Banister) Date: Fri, 24 Mar 2017 18:13:33 +0000 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: References: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu><47e18bb0062544f4b510ce0e87049fd3@jumptrading.com><643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> Message-ID: Hi Vipul, Hmm... interesting. We have dedicated systems running CES and nothing else, so the only thing opening files on GPFS is ganesha. IBM Support recommended we massively increase the maxFilesToCache to fix the performance issues we were having. I could try to reproduce the problem to investigate further, but the impact to users would not be appreciated. Setting such a high value for maxFilesToCache has direct impact on the token manager so fixing this would be very good. Perhaps IBM could take a closer look at this without our involvement? -Bryan From: Vipul Paul [mailto:vpaul at us.ibm.com] On Behalf Of IBM Spectrum Scale Sent: Friday, March 24, 2017 12:18 PM To: gpfsug main discussion list ; Bryan Banister Subject: Re: CES node slow to respond Hi Bryan, Making sure Malahal's reply was received by the user group. >> Then we noticed that the CES host had 5.4 million files open This is technically not possible with ganesha alone. A process can only open 1 million files on RHEL distro. Either we have leaks in kernel or some other processes contributing to this. Ganesha does keep NFSv3 files open and keep open for performance. It doesn't have a good logic to close them after some inactivity. It does close them if the number is close to max open files which is configurable. PS: kNFS does open, read/write, and then close. No caching in older versions. They did have a feature in a recent code to cache open files in NFSv3 as well. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Bryan Banister > To: gpfsug main discussion list > Date: 03/23/2017 08:27 AM Subject: Re: [gpfsug-discuss] CES node slow to respond Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Anybody from IBM willing/able to give us some explanation of why ganesha is holding open so many files? Is this expected/needed/etc? Or do we have to open a PMR to get some kind of explanation? -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Thursday, March 23, 2017 10:24 AM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] CES node slow to respond FYI all We also ran into this after bumping maxFilesToCache. Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: unexpected error: Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many open files in system fix sysctl -w fs.file-max=1000000 On 3/22/17 12:01 PM, Bryan Banister wrote: > We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. > > Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil > Sent: Wednesday, March 22, 2017 11:43 AM > To: gpfsug main discussion list > > Subject: [gpfsug-discuss] CES node slow to respond > > All, > > We had an indecent yesterday where one of our CES nodes slowed to a > crawl. GPFS waiters showed pre fetch threads going after inodes. > iohist also showed lots of inode fetching. Then we noticed that the CES > host had 5.4 million files open. > > The change I made was to set maxStatCache=DEFAULT because it is linux. > And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. > > Is there something else we should change as well. > > Thanks > > Matt > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From oehmes at gmail.com Fri Mar 24 18:19:52 2017 From: oehmes at gmail.com (Sven Oehme) Date: Fri, 24 Mar 2017 18:19:52 +0000 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: References: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu> <47e18bb0062544f4b510ce0e87049fd3@jumptrading.com> <643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> Message-ID: changes in ganesha management code were made in April 2016 to reduce the need for high maxfilestocache value, the ganesha daemon adjusts it allowed file cache by reading the maxfilestocache value and then reducing its allowed NOFILE value . the code shipped with 4.2.2 release. you want a high maxfilestocache value on CES nodes for various reasons. just FYI we have some customers who have MFTC set to 15 Million :-) sven On Fri, Mar 24, 2017 at 11:13 AM Bryan Banister wrote: > Hi Vipul, > > > > Hmm? interesting. We have dedicated systems running CES and nothing else, > so the only thing opening files on GPFS is ganesha. IBM Support > recommended we massively increase the maxFilesToCache to fix the > performance issues we were having. I could try to reproduce the problem to > investigate further, but the impact to users would not be appreciated. > > > > Setting such a high value for maxFilesToCache has direct impact on the > token manager so fixing this would be very good. > > > > Perhaps IBM could take a closer look at this without our involvement? > > -Bryan > > > > *From:* Vipul Paul [mailto:vpaul at us.ibm.com] *On Behalf Of *IBM Spectrum > Scale > *Sent:* Friday, March 24, 2017 12:18 PM > *To:* gpfsug main discussion list ; > Bryan Banister > *Subject:* Re: CES node slow to respond > > > > Hi Bryan, > > Making sure Malahal's reply was received by the user group. > > >> Then we noticed that the CES host had 5.4 million files open > > This is technically not possible with ganesha alone. A process can only > open 1 million files on RHEL distro. Either we have leaks in kernel or some > other processes contributing to this. > > Ganesha does keep NFSv3 files open and keep open for performance. It > doesn't have a good logic to close them after some inactivity. It does > close them if the number is close to max open files which is configurable. > > PS: kNFS does open, read/write, and then close. No caching in older > versions. They did have a feature in a recent code to cache open files in > NFSv3 as well. > > Regards, The Spectrum Scale (GPFS) team > > > ------------------------------------------------------------------------------------------------------------------ > If you feel that your question can benefit other users of Spectrum Scale > (GPFS), then please post it to the public IBM developerWroks Forum at > https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. > > > If your query concerns a potential software error in Spectrum Scale (GPFS) > and you have an IBM software maintenance contract please contact > 1-800-237-5511 <(800)%20237-5511> in the United States or your local IBM > Service Center in other countries. > > The forum is informally monitored as time permits and should not be used > for priority messages to the Spectrum Scale (GPFS) team. > > > > From: Bryan Banister > To: gpfsug main discussion list > Date: 03/23/2017 08:27 AM > Subject: Re: [gpfsug-discuss] CES node slow to respond > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------ > > > > > Anybody from IBM willing/able to give us some explanation of why ganesha > is holding open so many files? Is this expected/needed/etc? > > Or do we have to open a PMR to get some kind of explanation? > -B > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [ > mailto:gpfsug-discuss-bounces at spectrumscale.org > ] On Behalf Of Matt Weil > Sent: Thursday, March 23, 2017 10:24 AM > To: gpfsug-discuss at spectrumscale.org > Subject: Re: [gpfsug-discuss] CES node slow to respond > > FYI all > > We also ran into this after bumping maxFilesToCache. > > Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: > unexpected error: > Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many > open files in system > > fix > > sysctl -w fs.file-max=1000000 > > > On 3/22/17 12:01 PM, Bryan Banister wrote: > > We had a similar issue and also were instructed by IBM Support to > increase the maxFilesToCache to an insane value... Basically when the file > cache gets full then the host will spend all of its cycles looking for a > file to evict every time a new file is opened... baaaah. > > > > Not sure why Ganesha has to keep so many files open... I can't believe > our NFS clients actually keep that many open. cNFS never needed this. > > -Bryan > > > > -----Original Message----- > > From: gpfsug-discuss-bounces at spectrumscale.org [ > mailto:gpfsug-discuss-bounces at spectrumscale.org > ] On Behalf Of Matt Weil > > Sent: Wednesday, March 22, 2017 11:43 AM > > To: gpfsug main discussion list > > Subject: [gpfsug-discuss] CES node slow to respond > > > > All, > > > > We had an indecent yesterday where one of our CES nodes slowed to a > > crawl. GPFS waiters showed pre fetch threads going after inodes. > > iohist also showed lots of inode fetching. Then we noticed that the CES > > host had 5.4 million files open. > > > > The change I made was to set maxStatCache=DEFAULT because it is linux. > > And set maxFilesToCache=10000000 it was set to 500000. Then restarted > GPFS. > > > > Is there something else we should change as well. > > > > Thanks > > > > Matt > > > > > > ________________________________ > > The materials in this message are private and may contain Protected > Healthcare Information or other information of a sensitive nature. If you > are not the intended recipient, be advised that any unauthorized use, > disclosure, copying or the taking of any action in reliance on the contents > of this information is strictly prohibited. If you have received this email > in error, please immediately notify the sender via telephone or return mail. > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > ________________________________ > > > > Note: This email is for the confidential use of the named addressee(s) > only and may contain proprietary, confidential or privileged information. > If you are not the intended recipient, you are hereby notified that any > review, dissemination or copying of this email is strictly prohibited, and > to please notify the sender immediately and destroy this email and any > attachments. Email transmission cannot be guaranteed to be secure or > error-free. The Company, therefore, does not make any guarantees as to the > completeness or accuracy of this email or any attachments. This email is > for informational purposes only and does not constitute a recommendation, > offer, request or solicitation of any kind to buy, sell, subscribe, redeem > or perform any type of transaction of a financial product. > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > ________________________________ > The materials in this message are private and may contain Protected > Healthcare Information or other information of a sensitive nature. If you > are not the intended recipient, be advised that any unauthorized use, > disclosure, copying or the taking of any action in reliance on the contents > of this information is strictly prohibited. If you have received this email > in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) > only and may contain proprietary, confidential or privileged information. > If you are not the intended recipient, you are hereby notified that any > review, dissemination or copying of this email is strictly prohibited, and > to please notify the sender immediately and destroy this email and any > attachments. Email transmission cannot be guaranteed to be secure or > error-free. The Company, therefore, does not make any guarantees as to the > completeness or accuracy of this email or any attachments. This email is > for informational purposes only and does not constitute a recommendation, > offer, request or solicitation of any kind to buy, sell, subscribe, redeem > or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > ------------------------------ > > Note: This email is for the confidential use of the named addressee(s) > only and may contain proprietary, confidential or privileged information. > If you are not the intended recipient, you are hereby notified that any > review, dissemination or copying of this email is strictly prohibited, and > to please notify the sender immediately and destroy this email and any > attachments. Email transmission cannot be guaranteed to be secure or > error-free. The Company, therefore, does not make any guarantees as to the > completeness or accuracy of this email or any attachments. This email is > for informational purposes only and does not constitute a recommendation, > offer, request or solicitation of any kind to buy, sell, subscribe, redeem > or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Fri Mar 24 18:22:50 2017 From: bbanister at jumptrading.com (Bryan Banister) Date: Fri, 24 Mar 2017 18:22:50 +0000 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: References: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu> <47e18bb0062544f4b510ce0e87049fd3@jumptrading.com> <643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> Message-ID: <47c157d6cb5747698d3ccf11bd7a27ab@jumptrading.com> Thanks Sven! We recently upgrade to 4.2.2 and will see about lowering the maxFilesToCache to something more appropriate. We?re not offering NFS access as a performance solution? but it can?t come to a crawl either! Your help is greatly appreciated as always, -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sven Oehme Sent: Friday, March 24, 2017 1:20 PM To: gpfsug main discussion list ; IBM Spectrum Scale Subject: Re: [gpfsug-discuss] CES node slow to respond changes in ganesha management code were made in April 2016 to reduce the need for high maxfilestocache value, the ganesha daemon adjusts it allowed file cache by reading the maxfilestocache value and then reducing its allowed NOFILE value . the code shipped with 4.2.2 release. you want a high maxfilestocache value on CES nodes for various reasons. just FYI we have some customers who have MFTC set to 15 Million :-) sven On Fri, Mar 24, 2017 at 11:13 AM Bryan Banister > wrote: Hi Vipul, Hmm? interesting. We have dedicated systems running CES and nothing else, so the only thing opening files on GPFS is ganesha. IBM Support recommended we massively increase the maxFilesToCache to fix the performance issues we were having. I could try to reproduce the problem to investigate further, but the impact to users would not be appreciated. Setting such a high value for maxFilesToCache has direct impact on the token manager so fixing this would be very good. Perhaps IBM could take a closer look at this without our involvement? -Bryan From: Vipul Paul [mailto:vpaul at us.ibm.com] On Behalf Of IBM Spectrum Scale Sent: Friday, March 24, 2017 12:18 PM To: gpfsug main discussion list >; Bryan Banister > Subject: Re: CES node slow to respond Hi Bryan, Making sure Malahal's reply was received by the user group. >> Then we noticed that the CES host had 5.4 million files open This is technically not possible with ganesha alone. A process can only open 1 million files on RHEL distro. Either we have leaks in kernel or some other processes contributing to this. Ganesha does keep NFSv3 files open and keep open for performance. It doesn't have a good logic to close them after some inactivity. It does close them if the number is close to max open files which is configurable. PS: kNFS does open, read/write, and then close. No caching in older versions. They did have a feature in a recent code to cache open files in NFSv3 as well. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Bryan Banister > To: gpfsug main discussion list > Date: 03/23/2017 08:27 AM Subject: Re: [gpfsug-discuss] CES node slow to respond Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Anybody from IBM willing/able to give us some explanation of why ganesha is holding open so many files? Is this expected/needed/etc? Or do we have to open a PMR to get some kind of explanation? -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Thursday, March 23, 2017 10:24 AM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] CES node slow to respond FYI all We also ran into this after bumping maxFilesToCache. Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: unexpected error: Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many open files in system fix sysctl -w fs.file-max=1000000 On 3/22/17 12:01 PM, Bryan Banister wrote: > We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. > > Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil > Sent: Wednesday, March 22, 2017 11:43 AM > To: gpfsug main discussion list > > Subject: [gpfsug-discuss] CES node slow to respond > > All, > > We had an indecent yesterday where one of our CES nodes slowed to a > crawl. GPFS waiters showed pre fetch threads going after inodes. > iohist also showed lots of inode fetching. Then we noticed that the CES > host had 5.4 million files open. > > The change I made was to set maxStatCache=DEFAULT because it is linux. > And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. > > Is there something else we should change as well. > > Thanks > > Matt > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mweil at wustl.edu Fri Mar 24 19:57:42 2017 From: mweil at wustl.edu (Matt Weil) Date: Fri, 24 Mar 2017 14:57:42 -0500 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: References: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu> <47e18bb0062544f4b510ce0e87049fd3@jumptrading.com> <643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> Message-ID: <5a3aeddb-3e09-4003-8e57-299e999fe62d@wustl.edu> On 3/24/17 1:13 PM, Bryan Banister wrote: Hi Vipul, Hmm? interesting. We have dedicated systems running CES and nothing else, so the only thing opening files on GPFS is ganesha. IBM Support recommended we massively increase the maxFilesToCache to fix the performance issues we were having. I could try to reproduce the problem to investigate further, but the impact to users would not be appreciated. Same with us the system only runs CES and NFS. All open files 5.4 million where tied to it. This excluded anything the system had open. Setting such a high value for maxFilesToCache has direct impact on the token manager so fixing this would be very good. Perhaps IBM could take a closer look at this without our involvement? -Bryan From: Vipul Paul [mailto:vpaul at us.ibm.com] On Behalf Of IBM Spectrum Scale Sent: Friday, March 24, 2017 12:18 PM To: gpfsug main discussion list ; Bryan Banister Subject: Re: CES node slow to respond Hi Bryan, Making sure Malahal's reply was received by the user group. >> Then we noticed that the CES host had 5.4 million files open This is technically not possible with ganesha alone. A process can only open 1 million files on RHEL distro. Either we have leaks in kernel or some other processes contributing to this. Ganesha does keep NFSv3 files open and keep open for performance. It doesn't have a good logic to close them after some inactivity. It does close them if the number is close to max open files which is configurable. PS: kNFS does open, read/write, and then close. No caching in older versions. They did have a feature in a recent code to cache open files in NFSv3 as well. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Bryan Banister > To: gpfsug main discussion list > Date: 03/23/2017 08:27 AM Subject: Re: [gpfsug-discuss] CES node slow to respond Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Anybody from IBM willing/able to give us some explanation of why ganesha is holding open so many files? Is this expected/needed/etc? Or do we have to open a PMR to get some kind of explanation? -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Thursday, March 23, 2017 10:24 AM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] CES node slow to respond FYI all We also ran into this after bumping maxFilesToCache. Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: unexpected error: Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many open files in system fix sysctl -w fs.file-max=1000000 On 3/22/17 12:01 PM, Bryan Banister wrote: > We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. > > Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil > Sent: Wednesday, March 22, 2017 11:43 AM > To: gpfsug main discussion list > > Subject: [gpfsug-discuss] CES node slow to respond > > All, > > We had an indecent yesterday where one of our CES nodes slowed to a > crawl. GPFS waiters showed pre fetch threads going after inodes. > iohist also showed lots of inode fetching. Then we noticed that the CES > host had 5.4 million files open. > > The change I made was to set maxStatCache=DEFAULT because it is linux. > And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. > > Is there something else we should change as well. > > Thanks > > Matt > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mweil at wustl.edu Fri Mar 24 20:03:22 2017 From: mweil at wustl.edu (Matt Weil) Date: Fri, 24 Mar 2017 15:03:22 -0500 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: <5a3aeddb-3e09-4003-8e57-299e999fe62d@wustl.edu> References: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu> <47e18bb0062544f4b510ce0e87049fd3@jumptrading.com> <643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> <5a3aeddb-3e09-4003-8e57-299e999fe62d@wustl.edu> Message-ID: also running version 4.2.2.2. On 3/24/17 2:57 PM, Matt Weil wrote: On 3/24/17 1:13 PM, Bryan Banister wrote: Hi Vipul, Hmm? interesting. We have dedicated systems running CES and nothing else, so the only thing opening files on GPFS is ganesha. IBM Support recommended we massively increase the maxFilesToCache to fix the performance issues we were having. I could try to reproduce the problem to investigate further, but the impact to users would not be appreciated. Same with us the system only runs CES and NFS. All open files 5.4 million where tied to it. This excluded anything the system had open. Setting such a high value for maxFilesToCache has direct impact on the token manager so fixing this would be very good. Perhaps IBM could take a closer look at this without our involvement? -Bryan From: Vipul Paul [mailto:vpaul at us.ibm.com] On Behalf Of IBM Spectrum Scale Sent: Friday, March 24, 2017 12:18 PM To: gpfsug main discussion list ; Bryan Banister Subject: Re: CES node slow to respond Hi Bryan, Making sure Malahal's reply was received by the user group. >> Then we noticed that the CES host had 5.4 million files open This is technically not possible with ganesha alone. A process can only open 1 million files on RHEL distro. Either we have leaks in kernel or some other processes contributing to this. Ganesha does keep NFSv3 files open and keep open for performance. It doesn't have a good logic to close them after some inactivity. It does close them if the number is close to max open files which is configurable. PS: kNFS does open, read/write, and then close. No caching in older versions. They did have a feature in a recent code to cache open files in NFSv3 as well. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Bryan Banister > To: gpfsug main discussion list > Date: 03/23/2017 08:27 AM Subject: Re: [gpfsug-discuss] CES node slow to respond Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Anybody from IBM willing/able to give us some explanation of why ganesha is holding open so many files? Is this expected/needed/etc? Or do we have to open a PMR to get some kind of explanation? -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Thursday, March 23, 2017 10:24 AM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] CES node slow to respond FYI all We also ran into this after bumping maxFilesToCache. Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: unexpected error: Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many open files in system fix sysctl -w fs.file-max=1000000 On 3/22/17 12:01 PM, Bryan Banister wrote: > We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. > > Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil > Sent: Wednesday, March 22, 2017 11:43 AM > To: gpfsug main discussion list > > Subject: [gpfsug-discuss] CES node slow to respond > > All, > > We had an indecent yesterday where one of our CES nodes slowed to a > crawl. GPFS waiters showed pre fetch threads going after inodes. > iohist also showed lots of inode fetching. Then we noticed that the CES > host had 5.4 million files open. > > The change I made was to set maxStatCache=DEFAULT because it is linux. > And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. > > Is there something else we should change as well. > > Thanks > > Matt > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mweil at wustl.edu Fri Mar 24 20:20:18 2017 From: mweil at wustl.edu (Matt Weil) Date: Fri, 24 Mar 2017 15:20:18 -0500 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: References: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu> <47e18bb0062544f4b510ce0e87049fd3@jumptrading.com> <643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> Message-ID: On 3/24/17 12:17 PM, IBM Spectrum Scale wrote: Hi Bryan, Making sure Malahal's reply was received by the user group. >> Then we noticed that the CES host had 5.4 million files open This is technically not possible with ganesha alone. A process can only open 1 million files on RHEL distro. Either we have leaks in kernel or some other processes contributing to this. Ganesha does keep NFSv3 files open and keep open for performance. It doesn't have a good logic to close them after some inactivity. It does close them if the number is close to max open files which is configurable. how do you set the max open files for Ganesha? PS: kNFS does open, read/write, and then close. No caching in older versions. They did have a feature in a recent code to cache open files in NFSv3 as well. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Bryan Banister To: gpfsug main discussion list Date: 03/23/2017 08:27 AM Subject: Re: [gpfsug-discuss] CES node slow to respond Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Anybody from IBM willing/able to give us some explanation of why ganesha is holding open so many files? Is this expected/needed/etc? Or do we have to open a PMR to get some kind of explanation? -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Thursday, March 23, 2017 10:24 AM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] CES node slow to respond FYI all We also ran into this after bumping maxFilesToCache. Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: unexpected error: Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many open files in system fix sysctl -w fs.file-max=1000000 On 3/22/17 12:01 PM, Bryan Banister wrote: > We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. > > Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil > Sent: Wednesday, March 22, 2017 11:43 AM > To: gpfsug main discussion list > Subject: [gpfsug-discuss] CES node slow to respond > > All, > > We had an indecent yesterday where one of our CES nodes slowed to a > crawl. GPFS waiters showed pre fetch threads going after inodes. > iohist also showed lots of inode fetching. Then we noticed that the CES > host had 5.4 million files open. > > The change I made was to set maxStatCache=DEFAULT because it is linux. > And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. > > Is there something else we should change as well. > > Thanks > > Matt > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Sat Mar 25 05:46:34 2017 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Fri, 24 Mar 2017 21:46:34 -0800 Subject: [gpfsug-discuss] CES node slow to respond Message-ID: Forwarding Malahal's reply. Max open files of the ganesha process is set internally (and written to /etc/sysconfig/ganesha file as NOFILE parameter) based on MFTC (max files to cache) gpfs parameter. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. >> From: Matt Weil To: Date: 03/24/2017 01:20 PM Subject: Re: [gpfsug-discuss] CES node slow to respond Sent by: gpfsug-discuss-bounces at spectrumscale.org On 3/24/17 12:17 PM, IBM Spectrum Scale wrote: Hi Bryan, Making sure Malahal's reply was received by the user group. >> Then we noticed that the CES host had 5.4 million files open This is technically not possible with ganesha alone. A process can only open 1 million files on RHEL distro. Either we have leaks in kernel or some other processes contributing to this. Ganesha does keep NFSv3 files open and keep open for performance. It doesn't have a good logic to close them after some inactivity. It does close them if the number is close to max open files which is configurable. how do you set the max open files for Ganesha? PS: kNFS does open, read/write, and then close. No caching in older versions. They did have a feature in a recent code to cache open files in NFSv3 as well. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Bryan Banister To: gpfsug main discussion list Date: 03/23/2017 08:27 AM Subject: Re: [gpfsug-discuss] CES node slow to respond Sent by: gpfsug-discuss-bounces at spectrumscale.org Anybody from IBM willing/able to give us some explanation of why ganesha is holding open so many files? Is this expected/needed/etc? Or do we have to open a PMR to get some kind of explanation? -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Thursday, March 23, 2017 10:24 AM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] CES node slow to respond FYI all We also ran into this after bumping maxFilesToCache. Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: unexpected error: Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many open files in system fix sysctl -w fs.file-max=1000000 On 3/22/17 12:01 PM, Bryan Banister wrote: > We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. > > Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil > Sent: Wednesday, March 22, 2017 11:43 AM > To: gpfsug main discussion list > Subject: [gpfsug-discuss] CES node slow to respond > > All, > > We had an indecent yesterday where one of our CES nodes slowed to a > crawl. GPFS waiters showed pre fetch threads going after inodes. > iohist also showed lots of inode fetching. Then we noticed that the CES > host had 5.4 million files open. > > The change I made was to set maxStatCache=DEFAULT because it is linux. > And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. > > Is there something else we should change as well. > > Thanks > > Matt > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail._______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Sat Mar 25 05:55:57 2017 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Fri, 24 Mar 2017 21:55:57 -0800 Subject: [gpfsug-discuss] Fw: CES node slow to respond Message-ID: Caching of file descriptors can be disabled with "Cache_FDs = FALSE;" in cacheinode{} block. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. >> From: Bryan Banister To: IBM Spectrum Scale/Poughkeepsie/IBM at IBMUS, gpfsug main discussion list Date: 03/24/2017 11:13 AM Subject: RE: CES node slow to respond Hi Vipul, Hmm? interesting. We have dedicated systems running CES and nothing else, so the only thing opening files on GPFS is ganesha. IBM Support recommended we massively increase the maxFilesToCache to fix the performance issues we were having. I could try to reproduce the problem to investigate further, but the impact to users would not be appreciated. Setting such a high value for maxFilesToCache has direct impact on the token manager so fixing this would be very good. Perhaps IBM could take a closer look at this without our involvement? -Bryan From: Vipul Paul [mailto:vpaul at us.ibm.com] On Behalf Of IBM Spectrum Scale Sent: Friday, March 24, 2017 12:18 PM To: gpfsug main discussion list ; Bryan Banister Subject: Re: CES node slow to respond Hi Bryan, Making sure Malahal's reply was received by the user group. >> Then we noticed that the CES host had 5.4 million files open This is technically not possible with ganesha alone. A process can only open 1 million files on RHEL distro. Either we have leaks in kernel or some other processes contributing to this. Ganesha does keep NFSv3 files open and keep open for performance. It doesn't have a good logic to close them after some inactivity. It does close them if the number is close to max open files which is configurable. PS: kNFS does open, read/write, and then close. No caching in older versions. They did have a feature in a recent code to cache open files in NFSv3 as well. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Bryan Banister To: gpfsug main discussion list Date: 03/23/2017 08:27 AM Subject: Re: [gpfsug-discuss] CES node slow to respond Sent by: gpfsug-discuss-bounces at spectrumscale.org Anybody from IBM willing/able to give us some explanation of why ganesha is holding open so many files? Is this expected/needed/etc? Or do we have to open a PMR to get some kind of explanation? -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Thursday, March 23, 2017 10:24 AM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] CES node slow to respond FYI all We also ran into this after bumping maxFilesToCache. Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: unexpected error: Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many open files in system fix sysctl -w fs.file-max=1000000 On 3/22/17 12:01 PM, Bryan Banister wrote: > We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. > > Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil > Sent: Wednesday, March 22, 2017 11:43 AM > To: gpfsug main discussion list > Subject: [gpfsug-discuss] CES node slow to respond > > All, > > We had an indecent yesterday where one of our CES nodes slowed to a > crawl. GPFS waiters showed pre fetch threads going after inodes. > iohist also showed lots of inode fetching. Then we noticed that the CES > host had 5.4 million files open. > > The change I made was to set maxStatCache=DEFAULT because it is linux. > And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. > > Is there something else we should change as well. > > Thanks > > Matt > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chair at spectrumscale.org Mon Mar 27 08:55:04 2017 From: chair at spectrumscale.org (Spectrum Scale UG Chair (Simon Thompson)) Date: Mon, 27 Mar 2017 08:55:04 +0100 Subject: [gpfsug-discuss] UK User Group Registration NOW OPEN! [May 9th / 10th 2017] Message-ID: Registration for the UK Spectrum Scale (GPFS) user group (May 9th/10th) has opened this morning. Registration is via EventBrite at: https://www.eventbrite.com/e/spectrum-scalegpfs-user-group-spring-2017-regi stration-32113696932 This year we have moved to a larger capacity venue (Manchester Metropole Hotel), with thanks to IBM for sponsoring and supporting the event. We would also like to thank our other sponsors ArcaStream, DDN Storage, Ellexus, Mellanox, OCF and Seagate for supporting the event. Their sponsorship goes to help ensure the event remains free for the community and supports the evening social event as well as providing them with a technical (non sales) slot on the agenda. This year, the evening social event will be taking place at Manchester Museum of Science and Industry, and we're very grateful that are sponsors are able to support us to ensure this happens. The agenda is still a work in progress and may be subject to change as we confirm speakers and timings. As usual, we have reserved some slots on the agenda for user speakers, so if you have an interesting use case of Spectrum Scale, please let me know. As sectors we don't really get to hear from, it would be great if one of the members from the finance or sport industry were able to speak! Of course the event isn't limited to people from the UK, and we welcome registrants from across the world! Please note that to ensure attendance from a wide range of organisations, we are currently limiting attendance to 3 people per organisation. The venue is situated just a few minutes walk from Manchester Piccadilly station in Central Manchester with many hotels near by. The venue also has a limited number of special rate rooms, details of how you can book direct with the hotel are on the registration page. Thanks Simon UK Group Chair From p.childs at qmul.ac.uk Mon Mar 27 19:14:34 2017 From: p.childs at qmul.ac.uk (Peter Childs) Date: Mon, 27 Mar 2017 18:14:34 +0000 Subject: [gpfsug-discuss] UK User Group Registration NOW OPEN! [May 9th / 10th 2017] In-Reply-To: References: Message-ID: <0d1hhnpr5cn1dc5n2h7jpcec.1490638472562@email.android.com> Can I verify that this is at the Manchester McDonald hotel as eventbright, and website says, not the Manchester metropole hotel as the mailing list messages says, and I'm having difficulty finding. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ---- Spectrum Scale UG Chair (Simon Thompson) wrote ---- Registration for the UK Spectrum Scale (GPFS) user group (May 9th/10th) has opened this morning. Registration is via EventBrite at: https://www.eventbrite.com/e/spectrum-scalegpfs-user-group-spring-2017-regi stration-32113696932 This year we have moved to a larger capacity venue (Manchester Metropole Hotel), with thanks to IBM for sponsoring and supporting the event. We would also like to thank our other sponsors ArcaStream, DDN Storage, Ellexus, Mellanox, OCF and Seagate for supporting the event. Their sponsorship goes to help ensure the event remains free for the community and supports the evening social event as well as providing them with a technical (non sales) slot on the agenda. This year, the evening social event will be taking place at Manchester Museum of Science and Industry, and we're very grateful that are sponsors are able to support us to ensure this happens. The agenda is still a work in progress and may be subject to change as we confirm speakers and timings. As usual, we have reserved some slots on the agenda for user speakers, so if you have an interesting use case of Spectrum Scale, please let me know. As sectors we don't really get to hear from, it would be great if one of the members from the finance or sport industry were able to speak! Of course the event isn't limited to people from the UK, and we welcome registrants from across the world! Please note that to ensure attendance from a wide range of organisations, we are currently limiting attendance to 3 people per organisation. The venue is situated just a few minutes walk from Manchester Piccadilly station in Central Manchester with many hotels near by. The venue also has a limited number of special rate rooms, details of how you can book direct with the hotel are on the registration page. Thanks Simon UK Group Chair _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Mon Mar 27 19:25:53 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Mon, 27 Mar 2017 18:25:53 +0000 Subject: [gpfsug-discuss] UK User Group Registration NOW OPEN! [May 9th / 10th 2017] In-Reply-To: <0d1hhnpr5cn1dc5n2h7jpcec.1490638472562@email.android.com> References: <0d1hhnpr5cn1dc5n2h7jpcec.1490638472562@email.android.com> Message-ID: Hi Peter, Sorry, my mistake, the eventbrite page is correct - Manchester McDonald hotel. (we looked at a couple of different options in different areas and I obviously got confused!) Simon From: > on behalf of Peter Childs > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Monday, 27 March 2017 at 19:14 To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] UK User Group Registration NOW OPEN! [May 9th / 10th 2017] Can I verify that this is at the Manchester McDonald hotel as eventbright, and website says, not the Manchester metropole hotel as the mailing list messages says, and I'm having difficulty finding. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ---- Spectrum Scale UG Chair (Simon Thompson) wrote ---- Registration for the UK Spectrum Scale (GPFS) user group (May 9th/10th) has opened this morning. Registration is via EventBrite at: https://www.eventbrite.com/e/spectrum-scalegpfs-user-group-spring-2017-regi stration-32113696932 This year we have moved to a larger capacity venue (Manchester Metropole Hotel), with thanks to IBM for sponsoring and supporting the event. We would also like to thank our other sponsors ArcaStream, DDN Storage, Ellexus, Mellanox, OCF and Seagate for supporting the event. Their sponsorship goes to help ensure the event remains free for the community and supports the evening social event as well as providing them with a technical (non sales) slot on the agenda. This year, the evening social event will be taking place at Manchester Museum of Science and Industry, and we're very grateful that are sponsors are able to support us to ensure this happens. The agenda is still a work in progress and may be subject to change as we confirm speakers and timings. As usual, we have reserved some slots on the agenda for user speakers, so if you have an interesting use case of Spectrum Scale, please let me know. As sectors we don't really get to hear from, it would be great if one of the members from the finance or sport industry were able to speak! Of course the event isn't limited to people from the UK, and we welcome registrants from across the world! Please note that to ensure attendance from a wide range of organisations, we are currently limiting attendance to 3 people per organisation. The venue is situated just a few minutes walk from Manchester Piccadilly station in Central Manchester with many hotels near by. The venue also has a limited number of special rate rooms, details of how you can book direct with the hotel are on the registration page. Thanks Simon UK Group Chair _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From pinto at scinet.utoronto.ca Tue Mar 28 15:47:44 2017 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Tue, 28 Mar 2017 10:47:44 -0400 Subject: [gpfsug-discuss] fix mmrepquota report format during grace periods Message-ID: <20170328104744.138874ih550oncls@support.scinet.utoronto.ca> Any chance you guys in the GPFS devel team could patch the mmrepquota code so that during grace periods the report column for "none" would still be replaced with >>>*ONE*<<< word? By that I mean, instead of "2 days" for example, just print "2-days" or "2days" or "2_days", and so on. I have a number of scripts that fail for users when they are over their quotas under grace periods, because the report shifts the remaining information for that user 1 column to the right. Obviously it would cost me absolutely nothing to patch my scripts to deal with this, however the principle here is that the reports generated by GPFS should be the ones keeping consistence. Thanks Jaime ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. From Robert.Oesterlin at nuance.com Tue Mar 28 15:54:42 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Tue, 28 Mar 2017 14:54:42 +0000 Subject: [gpfsug-discuss] fix mmrepquota report format during grace periods Message-ID: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> Try running it with the ?-Y? option, it returns an easily to read output: mmrepquota -Y dns mmrepquota::HEADER:version:reserved:reserved:filesystemName:quotaType:id:name:blockUsage:blockQuota:blockLimit:blockInDoubt:blockGrace:filesUsage:filesQuota:filesLimit:filesInDoubt:filesGrace:remarks:quota:defQuota:fid:filesetname: mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:root: mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:users: mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:root: mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:users: mmrepquota::0:1:::dns:FILESET:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off::: mmrepquota::0:1:::dns:FILESET:1:users:0:4294967296:4294967296:0:none:1:0:0:0:none:e:on:off::: Bob Oesterlin Sr Principal Storage Engineer, Nuance On 3/28/17, 9:47 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Jaime Pinto" wrote: Any chance you guys in the GPFS devel team could patch the mmrepquota code so that during grace periods the report column for "none" would still be replaced with >>>*ONE*<<< word? By that I mean, instead of "2 days" for example, just print "2-days" or "2days" or "2_days", and so on. I have a number of scripts that fail for users when they are over their quotas under grace periods, because the report shifts the remaining information for that user 1 column to the right. Obviously it would cost me absolutely nothing to patch my scripts to deal with this, however the principle here is that the reports generated by GPFS should be the ones keeping consistence. Thanks Jaime ************************************ TELL US ABOUT YOUR SUCCESS STORIES https://urldefense.proofpoint.com/v2/url?u=http-3A__www.scinethpc.ca_testimonials&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzNsKbADKWCeLhA&s=TVGnqMwSWqNI1Vu1BlCcwXiVGsLUO9ZnbqlasVmT2HU&e= ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzNsKbADKWCeLhA&s=AXRwDMAVkYdwEaSFzejLQzNnS-KXoKj9GauzeEuu2H8&e= From Kevin.Buterbaugh at Vanderbilt.Edu Tue Mar 28 16:00:26 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 28 Mar 2017 15:00:26 +0000 Subject: [gpfsug-discuss] fix mmrepquota report format during grace periods In-Reply-To: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> Message-ID: <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> Hi Bob, Jaime, and GPFS team, That?s great for mmrepquota, but mmlsquota does not have a similar option AFAICT. That has really caused me grief ? for example, I?ve got a Perl script that takes mmlsquota output for a user and does two things: 1) converts it into something easier for them to parse, and 2) doesn?t display anything for the several dozen filesets they don?t have access to. That Perl script is ~300 lines and probably about a third of that is dealing with the grace period spacing issue? Kevin > On Mar 28, 2017, at 9:54 AM, Oesterlin, Robert wrote: > > Try running it with the ?-Y? option, it returns an easily to read output: > mmrepquota -Y dns > mmrepquota::HEADER:version:reserved:reserved:filesystemName:quotaType:id:name:blockUsage:blockQuota:blockLimit:blockInDoubt:blockGrace:filesUsage:filesQuota:filesLimit:filesInDoubt:filesGrace:remarks:quota:defQuota:fid:filesetname: > mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:root: > mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:users: > mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:root: > mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:users: > mmrepquota::0:1:::dns:FILESET:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off::: > mmrepquota::0:1:::dns:FILESET:1:users:0:4294967296:4294967296:0:none:1:0:0:0:none:e:on:off::: > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > > On 3/28/17, 9:47 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Jaime Pinto" wrote: > > Any chance you guys in the GPFS devel team could patch the mmrepquota > code so that during grace periods the report column for "none" would > still be replaced with >>>*ONE*<<< word? By that I mean, instead of "2 > days" for example, just print "2-days" or "2days" or "2_days", and so > on. > > I have a number of scripts that fail for users when they are over > their quotas under grace periods, because the report shifts the > remaining information for that user 1 column to the right. > > Obviously it would cost me absolutely nothing to patch my scripts to > deal with this, however the principle here is that the reports > generated by GPFS should be the ones keeping consistence. > > Thanks > Jaime > > > > > ************************************ > TELL US ABOUT YOUR SUCCESS STORIES > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.scinethpc.ca_testimonials&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzNsKbADKWCeLhA&s=TVGnqMwSWqNI1Vu1BlCcwXiVGsLUO9ZnbqlasVmT2HU&e= > ************************************ > --- > Jaime Pinto > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.ca > University of Toronto > 661 University Ave. (MaRS), Suite 1140 > Toronto, ON, M5G1M1 > P: 416-978-2755 > C: 416-505-1477 > > ---------------------------------------------------------------- > This message was sent using IMP at SciNet Consortium, University of Toronto. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzNsKbADKWCeLhA&s=AXRwDMAVkYdwEaSFzejLQzNnS-KXoKj9GauzeEuu2H8&e= > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Kevin.Buterbaugh at Vanderbilt.Edu Tue Mar 28 16:04:58 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 28 Mar 2017 15:04:58 +0000 Subject: [gpfsug-discuss] fix mmrepquota report format during grace periods In-Reply-To: <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> Message-ID: <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> Ugh! Of course, I?m wrong ? mmlsquota does support the ?-Y? option ? it?s just not documented. Why not???? Kevin > On Mar 28, 2017, at 10:00 AM, Buterbaugh, Kevin L wrote: > > Hi Bob, Jaime, and GPFS team, > > That?s great for mmrepquota, but mmlsquota does not have a similar option AFAICT. > > That has really caused me grief ? for example, I?ve got a Perl script that takes mmlsquota output for a user and does two things: 1) converts it into something easier for them to parse, and 2) doesn?t display anything for the several dozen filesets they don?t have access to. That Perl script is ~300 lines and probably about a third of that is dealing with the grace period spacing issue? > > Kevin > >> On Mar 28, 2017, at 9:54 AM, Oesterlin, Robert wrote: >> >> Try running it with the ?-Y? option, it returns an easily to read output: >> mmrepquota -Y dns >> mmrepquota::HEADER:version:reserved:reserved:filesystemName:quotaType:id:name:blockUsage:blockQuota:blockLimit:blockInDoubt:blockGrace:filesUsage:filesQuota:filesLimit:filesInDoubt:filesGrace:remarks:quota:defQuota:fid:filesetname: >> mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:root: >> mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:users: >> mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:root: >> mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:users: >> mmrepquota::0:1:::dns:FILESET:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off::: >> mmrepquota::0:1:::dns:FILESET:1:users:0:4294967296:4294967296:0:none:1:0:0:0:none:e:on:off::: >> >> Bob Oesterlin >> Sr Principal Storage Engineer, Nuance >> >> >> >> On 3/28/17, 9:47 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Jaime Pinto" wrote: >> >> Any chance you guys in the GPFS devel team could patch the mmrepquota >> code so that during grace periods the report column for "none" would >> still be replaced with >>>*ONE*<<< word? By that I mean, instead of "2 >> days" for example, just print "2-days" or "2days" or "2_days", and so >> on. >> >> I have a number of scripts that fail for users when they are over >> their quotas under grace periods, because the report shifts the >> remaining information for that user 1 column to the right. >> >> Obviously it would cost me absolutely nothing to patch my scripts to >> deal with this, however the principle here is that the reports >> generated by GPFS should be the ones keeping consistence. >> >> Thanks >> Jaime >> >> >> >> >> ************************************ >> TELL US ABOUT YOUR SUCCESS STORIES >> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.scinethpc.ca_testimonials&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzNsKbADKWCeLhA&s=TVGnqMwSWqNI1Vu1BlCcwXiVGsLUO9ZnbqlasVmT2HU&e= >> ************************************ >> --- >> Jaime Pinto >> SciNet HPC Consortium - Compute/Calcul Canada >> www.scinet.utoronto.ca - www.computecanada.ca >> University of Toronto >> 661 University Ave. (MaRS), Suite 1140 >> Toronto, ON, M5G1M1 >> P: 416-978-2755 >> C: 416-505-1477 >> >> ---------------------------------------------------------------- >> This message was sent using IMP at SciNet Consortium, University of Toronto. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzNsKbADKWCeLhA&s=AXRwDMAVkYdwEaSFzejLQzNnS-KXoKj9GauzeEuu2H8&e= >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From pinto at scinet.utoronto.ca Tue Mar 28 16:09:45 2017 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Tue, 28 Mar 2017 11:09:45 -0400 Subject: [gpfsug-discuss] fix mmrepquota report format during grace periods In-Reply-To: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> Message-ID: <20170328110945.18511am6iq6gf1dl@support.scinet.utoronto.ca> Aah! Another one of those options not so well documented or exposed: Usage: mmrepquota [-u] [-g] [-e] [-q] [-n] [-v] [-t] [--block-size {BlockSize | auto}] {-a | Device[:Fileset] ...} or mmrepquota -j [-e] [-q] [-n] [-v] [-t] [--block-size {BlockSize | auto}] {-a | Device ...} I agree, in this way it would be easier for a script to deal with fields that have spaces, using ':' as a field separator. However it mangles all the information together, making it very difficult for human eyes of a sysadmin to deal with in its original format. I'll take it under consideration for the scripts version (many of them to be revised), however the best is for the original and plain reports to have consistence. Thanks Jaime Quoting "Oesterlin, Robert" : > Try running it with the ?-Y? option, it returns an easily to read output: > mmrepquota -Y dns > mmrepquota::HEADER:version:reserved:reserved:filesystemName:quotaType:id:name:blockUsage:blockQuota:blockLimit:blockInDoubt:blockGrace:filesUsage:filesQuota:filesLimit:filesInDoubt:filesGrace:remarks:quota:defQuota:fid:filesetname: > mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:root: > mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:users: > mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:root: > mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:users: > mmrepquota::0:1:::dns:FILESET:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off::: > mmrepquota::0:1:::dns:FILESET:1:users:0:4294967296:4294967296:0:none:1:0:0:0:none:e:on:off::: > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > > On 3/28/17, 9:47 AM, "gpfsug-discuss-bounces at spectrumscale.org on > behalf of Jaime Pinto" behalf of pinto at scinet.utoronto.ca> wrote: > > Any chance you guys in the GPFS devel team could patch the mmrepquota > code so that during grace periods the report column for "none" would > still be replaced with >>>*ONE*<<< word? By that I mean, instead of "2 > days" for example, just print "2-days" or "2days" or "2_days", and so > on. > > I have a number of scripts that fail for users when they are over > their quotas under grace periods, because the report shifts the > remaining information for that user 1 column to the right. > > Obviously it would cost me absolutely nothing to patch my scripts to > deal with this, however the principle here is that the reports > generated by GPFS should be the ones keeping consistence. > > Thanks > Jaime > > > > > ************************************ > TELL US ABOUT YOUR SUCCESS STORIES > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.scinethpc.ca_testimonials&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzNsKbADKWCeLhA&s=TVGnqMwSWqNI1Vu1BlCcwXiVGsLUO9ZnbqlasVmT2HU&e= > ************************************ > --- > Jaime Pinto > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.ca > University of Toronto > 661 University Ave. (MaRS), Suite 1140 > Toronto, ON, M5G1M1 > P: 416-978-2755 > C: 416-505-1477 > > ---------------------------------------------------------------- > This message was sent using IMP at SciNet Consortium, University > of Toronto. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzNsKbADKWCeLhA&s=AXRwDMAVkYdwEaSFzejLQzNnS-KXoKj9GauzeEuu2H8&e= > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. From S.J.Thompson at bham.ac.uk Tue Mar 28 16:11:58 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Tue, 28 Mar 2017 15:11:58 +0000 Subject: [gpfsug-discuss] fix mmrepquota report format during grace periods In-Reply-To: <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> Message-ID: I thought this was because the -Y flag was going into new commands and being added to older commands during later releases. So it might be that it was added, but the docs not updated. Simon On 28/03/2017, 16:04, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Buterbaugh, Kevin L" wrote: >Ugh! Of course, I?m wrong ? mmlsquota does support the ?-Y? option ? >it?s just not documented. Why not???? > >Kevin > >> On Mar 28, 2017, at 10:00 AM, Buterbaugh, Kevin L >> wrote: >> >> Hi Bob, Jaime, and GPFS team, >> >> That?s great for mmrepquota, but mmlsquota does not have a similar >>option AFAICT. >> >> That has really caused me grief ? for example, I?ve got a Perl script >>that takes mmlsquota output for a user and does two things: 1) converts >>it into something easier for them to parse, and 2) doesn?t display >>anything for the several dozen filesets they don?t have access to. That >>Perl script is ~300 lines and probably about a third of that is dealing >>with the grace period spacing issue? >> >> Kevin >> >>> On Mar 28, 2017, at 9:54 AM, Oesterlin, Robert >>> wrote: >>> >>> Try running it with the ?-Y? option, it returns an easily to read >>>output: >>> mmrepquota -Y dns >>> >>>mmrepquota::HEADER:version:reserved:reserved:filesystemName:quotaType:id >>>:name:blockUsage:blockQuota:blockLimit:blockInDoubt:blockGrace:filesUsag >>>e:filesQuota:filesLimit:filesInDoubt:filesGrace:remarks:quota:defQuota:f >>>id:filesetname: >>> >>>mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:ro >>>ot: >>> >>>mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:us >>>ers: >>> >>>mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:ro >>>ot: >>> >>>mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:us >>>ers: >>> >>>mmrepquota::0:1:::dns:FILESET:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off: >>>:: >>> >>>mmrepquota::0:1:::dns:FILESET:1:users:0:4294967296:4294967296:0:none:1:0 >>>:0:0:none:e:on:off::: >>> >>> Bob Oesterlin >>> Sr Principal Storage Engineer, Nuance >>> >>> >>> >>> On 3/28/17, 9:47 AM, "gpfsug-discuss-bounces at spectrumscale.org on >>>behalf of Jaime Pinto" >>behalf of pinto at scinet.utoronto.ca> wrote: >>> >>> Any chance you guys in the GPFS devel team could patch the >>>mmrepquota >>> code so that during grace periods the report column for "none" would >>> >>> still be replaced with >>>*ONE*<<< word? By that I mean, instead of >>>"2 >>> days" for example, just print "2-days" or "2days" or "2_days", and >>>so >>> on. >>> >>> I have a number of scripts that fail for users when they are over >>> their quotas under grace periods, because the report shifts the >>> remaining information for that user 1 column to the right. >>> >>> Obviously it would cost me absolutely nothing to patch my scripts to >>> >>> deal with this, however the principle here is that the reports >>> generated by GPFS should be the ones keeping consistence. >>> >>> Thanks >>> Jaime >>> >>> >>> >>> >>> ************************************ >>> TELL US ABOUT YOUR SUCCESS STORIES >>> >>>https://urldefense.proofpoint.com/v2/url?u=http-3A__www.scinethpc.ca_tes >>>timonials&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDew >>>t1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzN >>>sKbADKWCeLhA&s=TVGnqMwSWqNI1Vu1BlCcwXiVGsLUO9ZnbqlasVmT2HU&e= >>> ************************************ >>> --- >>> Jaime Pinto >>> SciNet HPC Consortium - Compute/Calcul Canada >>> www.scinet.utoronto.ca - www.computecanada.ca >>> University of Toronto >>> 661 University Ave. (MaRS), Suite 1140 >>> Toronto, ON, M5G1M1 >>> P: 416-978-2755 >>> C: 416-505-1477 >>> >>> ---------------------------------------------------------------- >>> This message was sent using IMP at SciNet Consortium, University of >>>Toronto. >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> >>>https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_l >>>istinfo_gpfsug-2Ddiscuss&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0 >>>rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCI >>>ZvUgTr2CN-RqtzNsKbADKWCeLhA&s=AXRwDMAVkYdwEaSFzejLQzNnS-KXoKj9GauzeEuu2H >>>8&e= >>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Kevin.Buterbaugh at Vanderbilt.Edu Tue Mar 28 16:34:33 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 28 Mar 2017 15:34:33 +0000 Subject: [gpfsug-discuss] fix mmrepquota report format during grace periods In-Reply-To: References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> Message-ID: <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> All, Could someone(s) from the GPFS team please: 1) see that the appropriate documentation gets updated (manuals, ?-h? option to commands, the man pages for the commands, etc.)? 2) let us know what version of GPFS introduced the undocumented ?-Y? option for mmrepquota and mmlsquota? I?ve got numerous quota related scripts and I?m just curious to go back and figure out how much time I?ve wasted because I didn?t know about it. These ?unknown unknowns? bite us again? ;-) Kevin > On Mar 28, 2017, at 10:11 AM, Simon Thompson (Research Computing - IT Services) wrote: > > I thought this was because the -Y flag was going into new commands and > being added to older commands during later releases. > > So it might be that it was added, but the docs not updated. > > Simon > > On 28/03/2017, 16:04, "gpfsug-discuss-bounces at spectrumscale.org on behalf > of Buterbaugh, Kevin L" behalf of Kevin.Buterbaugh at Vanderbilt.Edu> wrote: > >> Ugh! Of course, I?m wrong ? mmlsquota does support the ?-Y? option ? >> it?s just not documented. Why not???? >> >> Kevin >> >>> On Mar 28, 2017, at 10:00 AM, Buterbaugh, Kevin L >>> wrote: >>> >>> Hi Bob, Jaime, and GPFS team, >>> >>> That?s great for mmrepquota, but mmlsquota does not have a similar >>> option AFAICT. >>> >>> That has really caused me grief ? for example, I?ve got a Perl script >>> that takes mmlsquota output for a user and does two things: 1) converts >>> it into something easier for them to parse, and 2) doesn?t display >>> anything for the several dozen filesets they don?t have access to. That >>> Perl script is ~300 lines and probably about a third of that is dealing >>> with the grace period spacing issue? >>> >>> Kevin >>> >>>> On Mar 28, 2017, at 9:54 AM, Oesterlin, Robert >>>> wrote: >>>> >>>> Try running it with the ?-Y? option, it returns an easily to read >>>> output: >>>> mmrepquota -Y dns >>>> >>>> mmrepquota::HEADER:version:reserved:reserved:filesystemName:quotaType:id >>>> :name:blockUsage:blockQuota:blockLimit:blockInDoubt:blockGrace:filesUsag >>>> e:filesQuota:filesLimit:filesInDoubt:filesGrace:remarks:quota:defQuota:f >>>> id:filesetname: >>>> >>>> mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:ro >>>> ot: >>>> >>>> mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:us >>>> ers: >>>> >>>> mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:ro >>>> ot: >>>> >>>> mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:us >>>> ers: >>>> >>>> mmrepquota::0:1:::dns:FILESET:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off: >>>> :: >>>> >>>> mmrepquota::0:1:::dns:FILESET:1:users:0:4294967296:4294967296:0:none:1:0 >>>> :0:0:none:e:on:off::: >>>> >>>> Bob Oesterlin >>>> Sr Principal Storage Engineer, Nuance >>>> >>>> >>>> >>>> On 3/28/17, 9:47 AM, "gpfsug-discuss-bounces at spectrumscale.org on >>>> behalf of Jaime Pinto" >>> behalf of pinto at scinet.utoronto.ca> wrote: >>>> >>>> Any chance you guys in the GPFS devel team could patch the >>>> mmrepquota >>>> code so that during grace periods the report column for "none" would >>>> >>>> still be replaced with >>>*ONE*<<< word? By that I mean, instead of >>>> "2 >>>> days" for example, just print "2-days" or "2days" or "2_days", and >>>> so >>>> on. >>>> >>>> I have a number of scripts that fail for users when they are over >>>> their quotas under grace periods, because the report shifts the >>>> remaining information for that user 1 column to the right. >>>> >>>> Obviously it would cost me absolutely nothing to patch my scripts to >>>> >>>> deal with this, however the principle here is that the reports >>>> generated by GPFS should be the ones keeping consistence. >>>> >>>> Thanks >>>> Jaime >>>> >>>> >>>> >>>> >>>> ************************************ >>>> TELL US ABOUT YOUR SUCCESS STORIES >>>> >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.scinethpc.ca_tes >>>> timonials&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDew >>>> t1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzN >>>> sKbADKWCeLhA&s=TVGnqMwSWqNI1Vu1BlCcwXiVGsLUO9ZnbqlasVmT2HU&e= >>>> ************************************ >>>> --- >>>> Jaime Pinto >>>> SciNet HPC Consortium - Compute/Calcul Canada >>>> www.scinet.utoronto.ca - www.computecanada.ca >>>> University of Toronto >>>> 661 University Ave. (MaRS), Suite 1140 >>>> Toronto, ON, M5G1M1 >>>> P: 416-978-2755 >>>> C: 416-505-1477 >>>> >>>> ---------------------------------------------------------------- >>>> This message was sent using IMP at SciNet Consortium, University of >>>> Toronto. >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_l >>>> istinfo_gpfsug-2Ddiscuss&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0 >>>> rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCI >>>> ZvUgTr2CN-RqtzNsKbADKWCeLhA&s=AXRwDMAVkYdwEaSFzejLQzNnS-KXoKj9GauzeEuu2H >>>> 8&e= >>>> >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From makaplan at us.ibm.com Tue Mar 28 17:14:18 2017 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 28 Mar 2017 11:14:18 -0500 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com><4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu><4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> Message-ID: Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and Programming) Reference I only found a few commands that officially are documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. But as many of you have discovered, -Y is accepted and yields "interesting" output for many of the mmXXXX commands. Moreover the output *seems to* have easily discernible patterns that can be parsed by simple programs. I believe there is no guarantee that the exact command output formats will not change from release to release, so, as a practical matter, if you're going to parse command output, you're probably better off parsing the -Y output, even if that is not officially supported. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathon.anderson at colorado.edu Tue Mar 28 17:17:29 2017 From: jonathon.anderson at colorado.edu (Jonathon A Anderson) Date: Tue, 28 Mar 2017 16:17:29 +0000 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> Message-ID: I had a bit of a run-in with a gpfs developer at a conference once when I complained about how command output changes frequently, breaking customer scripts. He was confused why we weren?t using `-Y`, and didn?t believe me that it?s simply not documented anywhere! `-Y` output is essential for interfacing programmatically with GPFS, and I don?t understand why it?s not mentioned in any of the guides or manpages. (Though apparently, according to this thead, it?s since appeared in some newer manpages.) ~jonathon On 3/28/17, 10:14 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Marc A Kaplan" wrote: Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and Programming) Reference I only found a few commands that officially are documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. But as many of you have discovered, -Y is accepted and yields "interesting" output for many of the mmXXXX commands. Moreover the output *seems to* have easily discernible patterns that can be parsed by simple programs. I believe there is no guarantee that the exact command output formats will not change from release to release, so, as a practical matter, if you're going to parse command output, you're probably better off parsing the -Y output, even if that is not officially supported. From stockf at us.ibm.com Tue Mar 28 17:21:57 2017 From: stockf at us.ibm.com (Frederick Stock) Date: Tue, 28 Mar 2017 11:21:57 -0500 Subject: [gpfsug-discuss] fix mmrepquota report format during grace periods In-Reply-To: <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com><4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu><4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> Message-ID: My understanding is that with the upcoming 4.2.3 release the -Y option will be documented for many commands, but perhaps not all. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 03/28/2017 11:35 AM Subject: Re: [gpfsug-discuss] fix mmrepquota report format during grace periods Sent by: gpfsug-discuss-bounces at spectrumscale.org All, Could someone(s) from the GPFS team please: 1) see that the appropriate documentation gets updated (manuals, ?-h? option to commands, the man pages for the commands, etc.)? 2) let us know what version of GPFS introduced the undocumented ?-Y? option for mmrepquota and mmlsquota? I?ve got numerous quota related scripts and I?m just curious to go back and figure out how much time I?ve wasted because I didn?t know about it. These ?unknown unknowns? bite us again? ;-) Kevin > On Mar 28, 2017, at 10:11 AM, Simon Thompson (Research Computing - IT Services) wrote: > > I thought this was because the -Y flag was going into new commands and > being added to older commands during later releases. > > So it might be that it was added, but the docs not updated. > > Simon > > On 28/03/2017, 16:04, "gpfsug-discuss-bounces at spectrumscale.org on behalf > of Buterbaugh, Kevin L" behalf of Kevin.Buterbaugh at Vanderbilt.Edu> wrote: > >> Ugh! Of course, I?m wrong ? mmlsquota does support the ?-Y? option ? >> it?s just not documented. Why not???? >> >> Kevin >> >>> On Mar 28, 2017, at 10:00 AM, Buterbaugh, Kevin L >>> wrote: >>> >>> Hi Bob, Jaime, and GPFS team, >>> >>> That?s great for mmrepquota, but mmlsquota does not have a similar >>> option AFAICT. >>> >>> That has really caused me grief ? for example, I?ve got a Perl script >>> that takes mmlsquota output for a user and does two things: 1) converts >>> it into something easier for them to parse, and 2) doesn?t display >>> anything for the several dozen filesets they don?t have access to. That >>> Perl script is ~300 lines and probably about a third of that is dealing >>> with the grace period spacing issue? >>> >>> Kevin >>> >>>> On Mar 28, 2017, at 9:54 AM, Oesterlin, Robert >>>> wrote: >>>> >>>> Try running it with the ?-Y? option, it returns an easily to read >>>> output: >>>> mmrepquota -Y dns >>>> >>>> mmrepquota::HEADER:version:reserved:reserved:filesystemName:quotaType:id >>>> :name:blockUsage:blockQuota:blockLimit:blockInDoubt:blockGrace:filesUsag >>>> e:filesQuota:filesLimit:filesInDoubt:filesGrace:remarks:quota:defQuota:f >>>> id:filesetname: >>>> >>>> mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:ro >>>> ot: >>>> >>>> mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:us >>>> ers: >>>> >>>> mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:ro >>>> ot: >>>> >>>> mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:us >>>> ers: >>>> >>>> mmrepquota::0:1:::dns:FILESET:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off: >>>> :: >>>> >>>> mmrepquota::0:1:::dns:FILESET:1:users:0:4294967296:4294967296:0:none:1:0 >>>> :0:0:none:e:on:off::: >>>> >>>> Bob Oesterlin >>>> Sr Principal Storage Engineer, Nuance >>>> >>>> >>>> >>>> On 3/28/17, 9:47 AM, "gpfsug-discuss-bounces at spectrumscale.org on >>>> behalf of Jaime Pinto" >>> behalf of pinto at scinet.utoronto.ca> wrote: >>>> >>>> Any chance you guys in the GPFS devel team could patch the >>>> mmrepquota >>>> code so that during grace periods the report column for "none" would >>>> >>>> still be replaced with >>>*ONE*<<< word? By that I mean, instead of >>>> "2 >>>> days" for example, just print "2-days" or "2days" or "2_days", and >>>> so >>>> on. >>>> >>>> I have a number of scripts that fail for users when they are over >>>> their quotas under grace periods, because the report shifts the >>>> remaining information for that user 1 column to the right. >>>> >>>> Obviously it would cost me absolutely nothing to patch my scripts to >>>> >>>> deal with this, however the principle here is that the reports >>>> generated by GPFS should be the ones keeping consistence. >>>> >>>> Thanks >>>> Jaime >>>> >>>> >>>> >>>> >>>> ************************************ >>>> TELL US ABOUT YOUR SUCCESS STORIES >>>> >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.scinethpc.ca_tes >>>> timonials&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDew >>>> t1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzN >>>> sKbADKWCeLhA&s=TVGnqMwSWqNI1Vu1BlCcwXiVGsLUO9ZnbqlasVmT2HU&e= >>>> ************************************ >>>> --- >>>> Jaime Pinto >>>> SciNet HPC Consortium - Compute/Calcul Canada >>>> www.scinet.utoronto.ca - www.computecanada.ca >>>> University of Toronto >>>> 661 University Ave. (MaRS), Suite 1140 >>>> Toronto, ON, M5G1M1 >>>> P: 416-978-2755 >>>> C: 416-505-1477 >>>> >>>> ---------------------------------------------------------------- >>>> This message was sent using IMP at SciNet Consortium, University of >>>> Toronto. >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_l >>>> istinfo_gpfsug-2Ddiscuss&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0 >>>> rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCI >>>> ZvUgTr2CN-RqtzNsKbADKWCeLhA&s=AXRwDMAVkYdwEaSFzejLQzNnS-KXoKj9GauzeEuu2H >>>> 8&e= >>>> >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From luis.bolinches at fi.ibm.com Tue Mar 28 17:22:04 2017 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Tue, 28 Mar 2017 16:22:04 +0000 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: References: , <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com><4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu><4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu><5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> Message-ID: An HTML attachment was scrubbed... URL: From jonathon.anderson at colorado.edu Tue Mar 28 17:24:19 2017 From: jonathon.anderson at colorado.edu (Jonathon A Anderson) Date: Tue, 28 Mar 2017 16:24:19 +0000 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> Message-ID: <99B1AC2E-91B4-4A7F-B270-8FDDC49918EA@colorado.edu> Looks great. I look forward to its maturity. ~jonathon On 3/28/17, 10:22 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Luis Bolinches" wrote: Hi While I understand the frustration of tiem that could be used otherwise. depending of what you are plannig with script wrapping I would recommend you seriously take a look to the REST API http://files.gpfsug.org/presentations/2017/Ehningen/23_-_SSUG17DE_-_Alexander_Wolf-Reber_-_Spectrum_Scale_ReST_API.pdf -- Yst?v?llisin terveisin / Kind regards / Saludos cordiales / Salutations Luis Bolinches Lab Services http://www-03.ibm.com/systems/services/labservices/ IBM Laajalahdentie 23 (main Entrance) Helsinki, 00330 Finland Phone: +358 503112585 "If you continually give you will continually have." Anonymous ----- Original message ----- From: Jonathon A Anderson Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list Cc: Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Date: Tue, Mar 28, 2017 7:17 PM I had a bit of a run-in with a gpfs developer at a conference once when I complained about how command output changes frequently, breaking customer scripts. He was confused why we weren?t using `-Y`, and didn?t believe me that it?s simply not documented anywhere! `-Y` output is essential for interfacing programmatically with GPFS, and I don?t understand why it?s not mentioned in any of the guides or manpages. (Though apparently, according to this thead, it?s since appeared in some newer manpages.) ~jonathon On 3/28/17, 10:14 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Marc A Kaplan" wrote: Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and Programming) Reference I only found a few commands that officially are documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. But as many of you have discovered, -Y is accepted and yields "interesting" output for many of the mmXXXX commands. Moreover the output *seems to* have easily discernible patterns that can be parsed by simple programs. I believe there is no guarantee that the exact command output formats will not change from release to release, so, as a practical matter, if you're going to parse command output, you're probably better off parsing the -Y output, even if that is not officially supported. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland From S.J.Thompson at bham.ac.uk Tue Mar 28 17:37:12 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Tue, 28 Mar 2017 16:37:12 +0000 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> Message-ID: I thought the header rows defined what the format of the output was. Its a bit weird as there can be multiple header rows for different content rows ... Simon From: > on behalf of Marc A Kaplan > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Tuesday, 28 March 2017 at 17:14 To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and Programming) Reference I only found a few commands that officially are documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. But as many of you have discovered, -Y is accepted and yields "interesting" output for many of the mmXXXX commands. Moreover the output *seems to* have easily discernible patterns that can be parsed by simple programs. I believe there is no guarantee that the exact command output formats will not change from release to release, so, as a practical matter, if you're going to parse command output, you're probably better off parsing the -Y output, even if that is not officially supported. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathon.anderson at colorado.edu Tue Mar 28 17:37:47 2017 From: jonathon.anderson at colorado.edu (Jonathon A Anderson) Date: Tue, 28 Mar 2017 16:37:47 +0000 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> Message-ID: Sure; but that only helps if you know the flag even exists. ~jonathon On 3/28/17, 10:37 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Simon Thompson (Research Computing - IT Services)" wrote: I thought the header rows defined what the format of the output was. Its a bit weird as there can be multiple header rows for different content rows ... Simon From: on behalf of Marc A Kaplan Reply-To: "gpfsug-discuss at spectrumscale.org" Date: Tuesday, 28 March 2017 at 17:14 To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and Programming) Reference I only found a few commands that officially are documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. But as many of you have discovered, -Y is accepted and yields "interesting" output for many of the mmXXXX commands. Moreover the output *seems to* have easily discernible patterns that can be parsed by simple programs. I believe there is no guarantee that the exact command output formats will not change from release to release, so, as a practical matter, if you're going to parse command output, you're probably better off parsing the -Y output, even if that is not officially supported. From Kevin.Buterbaugh at Vanderbilt.Edu Tue Mar 28 17:47:14 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 28 Mar 2017 16:47:14 +0000 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> Message-ID: <53E9C309-3498-40FB-8018-8CABD9C6C316@vanderbilt.edu> Agreed. From what has been said on this thread about ?-Y? being unsupported and the command output could change from release to release ? well, that?s a ?known unknown? that can be dealt with. But the fact that ?-Y? was completely undocumented (at least as far as mmrepquota / mmlsquota are concerned) makes ?-Y? one of those ?unknown unknowns?? ;-) Kevin > On Mar 28, 2017, at 11:37 AM, Jonathon A Anderson wrote: > > Sure; but that only helps if you know the flag even exists. > > ~jonathon > > > On 3/28/17, 10:37 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Simon Thompson (Research Computing - IT Services)" wrote: > > I thought the header rows defined what the format of the output was. Its a bit weird as there can be multiple header rows for different content rows ... > > > Simon > > > From: on behalf of Marc A Kaplan > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Tuesday, 28 March 2017 at 17:14 > To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! > > > > Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and Programming) Reference I only found a few commands that officially are documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. > > But as many of you have discovered, -Y is accepted and yields "interesting" output for many of the mmXXXX commands. > Moreover the output *seems to* have easily discernible patterns that can be parsed by simple programs. > > I believe there is no guarantee that the exact command output formats will not change from release to release, so, as a practical matter, if you're going to parse command output, you're probably better off parsing the -Y output, > even if that is not officially supported. > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From S.J.Thompson at bham.ac.uk Tue Mar 28 17:51:12 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Tue, 28 Mar 2017 16:51:12 +0000 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> Message-ID: Come to one of the user group events going on around the world in the next few months. NERSC, Berkeley, USA, 4th/5th April: https://www.nersc.gov/research-and-development/data-analytics/spectrum-user -group-meeting/ *** REGISTRATION CLOSES TOMORROW 29th *** Sydney, Australia, 27th April: https://www.eventbrite.com/e/spectrum-scale-user-group-australia-gpfsugaus- april-2017-tickets-29136246297 Manchester, UK, 9th/10th May: https://www.eventbrite.com/e/spectrum-scalegpfs-user-group-spring-2017-regi stration-32113696932 I know the presence of the -Y flag has been discussed there, so its a great place to pick up tips! Simon On 28/03/2017, 17:37, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Jonathon A Anderson" wrote: >Sure; but that only helps if you know the flag even exists. > >~jonathon > > >On 3/28/17, 10:37 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf >of Simon Thompson (Research Computing - IT Services)" >S.J.Thompson at bham.ac.uk> wrote: > > I thought the header rows defined what the format of the output was. >Its a bit weird as there can be multiple header rows for different >content rows ... > > > Simon > > > From: on behalf of Marc A >Kaplan > Reply-To: "gpfsug-discuss at spectrumscale.org" > > Date: Tuesday, 28 March 2017 at 17:14 > To: "gpfsug-discuss at spectrumscale.org" > > Subject: Re: [gpfsug-discuss] -Y option for many commands, precious >few officially! > > > > Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and >Programming) Reference I only found a few commands that officially are >documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. > > But as many of you have discovered, -Y is accepted and yields >"interesting" output for many of the mmXXXX commands. > Moreover the output *seems to* have easily discernible patterns that >can be parsed by simple programs. > > I believe there is no guarantee that the exact command output formats >will not change from release to release, so, as a practical matter, if >you're going to parse command output, you're probably better off parsing >the -Y output, > even if that is not officially supported. > > > > > >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss From carlz at us.ibm.com Tue Mar 28 18:02:32 2017 From: carlz at us.ibm.com (Carl Zetie) Date: Tue, 28 Mar 2017 17:02:32 +0000 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From jonathon.anderson at colorado.edu Tue Mar 28 18:08:20 2017 From: jonathon.anderson at colorado.edu (Jonathon A Anderson) Date: Tue, 28 Mar 2017 17:08:20 +0000 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> Message-ID: <67E4C386-8884-4E92-B262-089FDFA32C94@colorado.edu> > I know the presence of the -Y flag has been discussed there, so it?s a great place to pick up tips! Sure, and I ?picked up this tip? at a community event; but really, it shouldn?t have to be picked up as arcana. It should be documented. Glad to hear that that?s the plan. ~jonathon On 3/28/17, 10:51 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Simon Thompson (Research Computing - IT Services)" wrote: Come to one of the user group events going on around the world in the next few months. NERSC, Berkeley, USA, 4th/5th April: https://www.nersc.gov/research-and-development/data-analytics/spectrum-user -group-meeting/ *** REGISTRATION CLOSES TOMORROW 29th *** Sydney, Australia, 27th April: https://www.eventbrite.com/e/spectrum-scale-user-group-australia-gpfsugaus- april-2017-tickets-29136246297 Manchester, UK, 9th/10th May: https://www.eventbrite.com/e/spectrum-scalegpfs-user-group-spring-2017-regi stration-32113696932 I know the presence of the -Y flag has been discussed there, so its a great place to pick up tips! Simon On 28/03/2017, 17:37, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Jonathon A Anderson" wrote: >Sure; but that only helps if you know the flag even exists. > >~jonathon > > >On 3/28/17, 10:37 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf >of Simon Thompson (Research Computing - IT Services)" >S.J.Thompson at bham.ac.uk> wrote: > > I thought the header rows defined what the format of the output was. >Its a bit weird as there can be multiple header rows for different >content rows ... > > > Simon > > > From: on behalf of Marc A >Kaplan > Reply-To: "gpfsug-discuss at spectrumscale.org" > > Date: Tuesday, 28 March 2017 at 17:14 > To: "gpfsug-discuss at spectrumscale.org" > > Subject: Re: [gpfsug-discuss] -Y option for many commands, precious >few officially! > > > > Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and >Programming) Reference I only found a few commands that officially are >documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. > > But as many of you have discovered, -Y is accepted and yields >"interesting" output for many of the mmXXXX commands. > Moreover the output *seems to* have easily discernible patterns that >can be parsed by simple programs. > > I believe there is no guarantee that the exact command output formats >will not change from release to release, so, as a practical matter, if >you're going to parse command output, you're probably better off parsing >the -Y output, > even if that is not officially supported. > > > > > >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From aleciarm at us.ibm.com Tue Mar 28 18:17:11 2017 From: aleciarm at us.ibm.com (Alecia A Ramsay) Date: Tue, 28 Mar 2017 12:17:11 -0500 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: References: Message-ID: For the benefit of those on digest, here's Carl's list again in plain text (I hope). I have also notified our documentation team that this is of great interest to clients, so they are aware. In the 4.2.3 draft documentation, the -y option information has been added to a number of commands: -Y option Added the -Y option to the following commands: mmblock mmhealth mmlsfileset mmlsnodeclass mmnetverify mmcloudgateway mmkeyserv mmlsfs mmlsnsd mmnfs mmdf mmlscluster mmlslicense mmlspolicy mmrepquota mmdiag mmlsconfig mmlsmgr mmlsquota mmsmb mmgetstate mmlsdisk mmlsmount mmlssnapshot mmuserauth Alecia A. Ramsay, PMP? Program Manager, New Technology Introduction IBM Systems - Storage aleciarm at us.ibm.com work: 919-435-6494; mobile: 651-260-4928 http://ibm.biz/NewTechnologyIntroduction From: Carl Zetie/Fairfax/IBM at IBMUS To: gpfsug-discuss at spectrumscale.org Date: 03/28/2017 01:03 PM Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Sent by: gpfsug-discuss-bounces at spectrumscale.org I checked the draft documentation for 4.2.3, and here is the list of what is planned. As usual -- please remember this is an Intention not a commitment, and subject to change between now and the release. regards, Carl -Y option Added the -Y option to the following commands: mmblock mmhealth mmlsfileset mmlsnodeclass mmnetverify mmcloudgateway mmkeyserv mmlsfs mmlsnsd mmnfs mmdf mmlscluster mmlslicense mmlspolicy mmrepquota mmdiag mmlsconfig mmlsmgr mmlsquota mmsmb mmgetstate mmlsdisk mmlsmount mmlssnapshot mmuserauth Carl Zetie Offering Manager for Spectrum Scale, IBM carlz at us.ibm.com ----- Original message ----- From: gpfsug-discuss-request at spectrumscale.org Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Cc: Subject: gpfsug-discuss Digest, Vol 62, Issue 70 Date: Tue, Mar 28, 2017 12:38 PM Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: -Y option for many commands, precious few officially! (Luis Bolinches) 2. Re: -Y option for many commands, precious few officially! (Jonathon A Anderson) 3. Re: -Y option for many commands, precious few officially! (Simon Thompson (Research Computing - IT Services)) 4. Re: -Y option for many commands, precious few officially! (Jonathon A Anderson) ---------------------------------------------------------------------- Message: 1 Date: Tue, 28 Mar 2017 16:22:04 +0000 From: "Luis Bolinches" To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Message-ID: Content-Type: text/plain; charset="us-ascii" An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20170328/694119bd/attachment-0001.html > ------------------------------ Message: 2 Date: Tue, 28 Mar 2017 16:24:19 +0000 From: Jonathon A Anderson To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Message-ID: <99B1AC2E-91B4-4A7F-B270-8FDDC49918EA at colorado.edu> Content-Type: text/plain; charset="utf-8" Looks great. I look forward to its maturity. ~jonathon On 3/28/17, 10:22 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Luis Bolinches" wrote: Hi While I understand the frustration of tiem that could be used otherwise. depending of what you are plannig with script wrapping I would recommend you seriously take a look to the REST API http://files.gpfsug.org/presentations/2017/Ehningen/23_-_SSUG17DE_-_Alexander_Wolf-Reber_-_Spectrum_Scale_ReST_API.pdf -- Yst?v?llisin terveisin / Kind regards / Saludos cordiales / Salutations Luis Bolinches Lab Services http://www-03.ibm.com/systems/services/labservices/ IBM Laajalahdentie 23 (main Entrance) Helsinki, 00330 Finland Phone: +358 503112585 "If you continually give you will continually have." Anonymous ----- Original message ----- From: Jonathon A Anderson Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list Cc: Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Date: Tue, Mar 28, 2017 7:17 PM I had a bit of a run-in with a gpfs developer at a conference once when I complained about how command output changes frequently, breaking customer scripts. He was confused why we weren?t using `-Y`, and didn?t believe me that it?s simply not documented anywhere! `-Y` output is essential for interfacing programmatically with GPFS, and I don?t understand why it?s not mentioned in any of the guides or manpages. (Though apparently, according to this thead, it?s since appeared in some newer manpages.) ~jonathon On 3/28/17, 10:14 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Marc A Kaplan" wrote: Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and Programming) Reference I only found a few commands that officially are documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. But as many of you have discovered, -Y is accepted and yields "interesting" output for many of the mmXXXX commands. Moreover the output *seems to* have easily discernible patterns that can be parsed by simple programs. I believe there is no guarantee that the exact command output formats will not change from release to release, so, as a practical matter, if you're going to parse command output, you're probably better off parsing the -Y output, even if that is not officially supported. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland ------------------------------ Message: 3 Date: Tue, 28 Mar 2017 16:37:12 +0000 From: "Simon Thompson (Research Computing - IT Services)" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Message-ID: Content-Type: text/plain; charset="us-ascii" I thought the header rows defined what the format of the output was. Its a bit weird as there can be multiple header rows for different content rows ... Simon From: > on behalf of Marc A Kaplan > Reply-To: "gpfsug-discuss at spectrumscale.org< mailto:gpfsug-discuss at spectrumscale.org>" > Date: Tuesday, 28 March 2017 at 17:14 To: "gpfsug-discuss at spectrumscale.org< mailto:gpfsug-discuss at spectrumscale.org>" > Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and Programming) Reference I only found a few commands that officially are documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. But as many of you have discovered, -Y is accepted and yields "interesting" output for many of the mmXXXX commands. Moreover the output *seems to* have easily discernible patterns that can be parsed by simple programs. I believe there is no guarantee that the exact command output formats will not change from release to release, so, as a practical matter, if you're going to parse command output, you're probably better off parsing the -Y output, even if that is not officially supported. -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20170328/2c551c41/attachment-0001.html > ------------------------------ Message: 4 Date: Tue, 28 Mar 2017 16:37:47 +0000 From: Jonathon A Anderson To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Message-ID: Content-Type: text/plain; charset="utf-8" Sure; but that only helps if you know the flag even exists. ~jonathon On 3/28/17, 10:37 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Simon Thompson (Research Computing - IT Services)" wrote: I thought the header rows defined what the format of the output was. Its a bit weird as there can be multiple header rows for different content rows ... Simon From: on behalf of Marc A Kaplan Reply-To: "gpfsug-discuss at spectrumscale.org" Date: Tuesday, 28 March 2017 at 17:14 To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and Programming) Reference I only found a few commands that officially are documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. But as many of you have discovered, -Y is accepted and yields "interesting" output for many of the mmXXXX commands. Moreover the output *seems to* have easily discernible patterns that can be parsed by simple programs. I believe there is no guarantee that the exact command output formats will not change from release to release, so, as a practical matter, if you're going to parse command output, you're probably better off parsing the -Y output, even if that is not officially supported. ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 62, Issue 70 ********************************************** _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From ckrafft at de.ibm.com Wed Mar 29 12:34:29 2017 From: ckrafft at de.ibm.com (Christoph Krafft) Date: Wed, 29 Mar 2017 12:34:29 +0100 Subject: [gpfsug-discuss] GPFS / Spectrum Scale support with x86 SUSE SLES 12 SP2 to be expected when? Message-ID: Folks, does anyone have any tentative information when the above SUSE OS Level will be supported? Thank you in advance for hints and tips! Mit freundlichen Gr??en / Sincerely Christoph Krafft Client Technical Specialist - Power Systems, IBM Systems Certified IT Specialist @ The Open Group Phone: +49 (0) 7034 643 2171 IBM Deutschland GmbH Mobile: +49 (0) 160 97 81 86 12 Am Weiher 24 Email: ckrafft at de.ibm.com 65451 Kelsterbach Germany IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter Gesch?ftsf?hrung: Martina Koederitz (Vorsitzende), Nicole Reimer, Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Stefan Lutz Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecblank.gif Type: image/gif Size: 45 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 16373446.gif Type: image/gif Size: 1851 bytes Desc: not available URL: From aaron.knister at gmail.com Wed Mar 29 14:18:25 2017 From: aaron.knister at gmail.com (Aaron Knister) Date: Wed, 29 Mar 2017 09:18:25 -0400 Subject: [gpfsug-discuss] GPFS / Spectrum Scale support with x86 SUSE SLES 12 SP2 to be expected when? In-Reply-To: References: Message-ID: <57030974-8C80-4808-87F1-3540E1271D6E@gmail.com> I could be wrong but I believe it is currently supported (as in it works, at least that's my recollection). Did you try with a particular version that wouldn't work? Sent from my iPhone > On Mar 29, 2017, at 07:34, Christoph Krafft wrote: > > Folks, > > does anyone have any tentative information when the above SUSE OS Level will be supported? > > Thank you in advance for hints and tips! > > > Mit freundlichen Gr??en / Sincerely > > Christoph Krafft > > Client Technical Specialist - Power Systems, IBM Systems > Certified IT Specialist @ The Open Group > > > Phone: +49 (0) 7034 643 2171 IBM Deutschland GmbH > <16373446.gif> > Mobile: +49 (0) 160 97 81 86 12 Am Weiher 24 > Email: ckrafft at de.ibm.com 65451 Kelsterbach > Germany > IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter > Gesch?ftsf?hrung: Martina Koederitz (Vorsitzende), Nicole Reimer, Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Stefan Lutz > Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Wed Mar 29 14:55:17 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Wed, 29 Mar 2017 09:55:17 -0400 Subject: [gpfsug-discuss] GPFS / Spectrum Scale support with x86 SUSE SLES 12 SP2 to be expected when? In-Reply-To: <57030974-8C80-4808-87F1-3540E1271D6E@gmail.com> References: <57030974-8C80-4808-87F1-3540E1271D6E@gmail.com> Message-ID: Just to make sure I wasn't mis-remembering I just installed 4.1.1.10 on a SLES12 SP1 machine (it has a SLES12 SP2 kernel though), built and loaded the kernel modules. There could be something significantly different in user space between SLES12 SP1 and SP2 that prevents RPM installation but I'd be surprised. -Aaron On 3/29/17 9:18 AM, Aaron Knister wrote: > I could be wrong but I believe it is currently supported (as in it > works, at least that's my recollection). Did you try with a particular > version that wouldn't work? > > Sent from my iPhone > > On Mar 29, 2017, at 07:34, Christoph Krafft > wrote: > >> Folks, >> >> does anyone have any tentative information when the above SUSE OS >> Level will be supported? >> >> Thank you in advance for hints and tips! >> >> >> Mit freundlichen Gr??en / Sincerely >> >> *Christoph Krafft* >> >> Client Technical Specialist - Power Systems, IBM Systems >> Certified IT Specialist @ The Open Group >> >> ------------------------------------------------------------------------ >> >> Phone: +49 (0) 7034 643 2171 IBM Deutschland GmbH >> <16373446.gif> >> Mobile: +49 (0) 160 97 81 86 12 Am Weiher 24 >> Email: ckrafft at de.ibm.com 65451 Kelsterbach >> Germany >> ------------------------------------------------------------------------ >> IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter >> Gesch?ftsf?hrung: Martina Koederitz (Vorsitzende), Nicole Reimer, >> Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Stefan Lutz >> Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht >> Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From S.J.Thompson at bham.ac.uk Wed Mar 29 15:01:33 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Wed, 29 Mar 2017 14:01:33 +0000 Subject: [gpfsug-discuss] GPFS / Spectrum Scale support with x86 SUSE SLES 12 SP2 to be expected when? In-Reply-To: References: <57030974-8C80-4808-87F1-3540E1271D6E@gmail.com>, Message-ID: I think the question was supported, not works though. We've just been round something similar with TSM support, works fine on el7.3, but not supported, the docs actually said 7.1 or higher. We still got support and IBM now officially support 7.3, but I can see people may not want to deploy on something unsupported. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Aaron Knister [aaron.s.knister at nasa.gov] Sent: 29 March 2017 14:55 To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] GPFS / Spectrum Scale support with x86 SUSE SLES 12 SP2 to be expected when? Just to make sure I wasn't mis-remembering I just installed 4.1.1.10 on a SLES12 SP1 machine (it has a SLES12 SP2 kernel though), built and loaded the kernel modules. There could be something significantly different in user space between SLES12 SP1 and SP2 that prevents RPM installation but I'd be surprised. -Aaron On 3/29/17 9:18 AM, Aaron Knister wrote: > I could be wrong but I believe it is currently supported (as in it > works, at least that's my recollection). Did you try with a particular > version that wouldn't work? > > Sent from my iPhone > > On Mar 29, 2017, at 07:34, Christoph Krafft > wrote: > >> Folks, >> >> does anyone have any tentative information when the above SUSE OS >> Level will be supported? >> >> Thank you in advance for hints and tips! >> >> >> Mit freundlichen Gr??en / Sincerely >> >> *Christoph Krafft* >> >> Client Technical Specialist - Power Systems, IBM Systems >> Certified IT Specialist @ The Open Group >> >> ------------------------------------------------------------------------ >> >> Phone: +49 (0) 7034 643 2171 IBM Deutschland GmbH >> <16373446.gif> >> Mobile: +49 (0) 160 97 81 86 12 Am Weiher 24 >> Email: ckrafft at de.ibm.com 65451 Kelsterbach >> Germany >> ------------------------------------------------------------------------ >> IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter >> Gesch?ftsf?hrung: Martina Koederitz (Vorsitzende), Nicole Reimer, >> Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Stefan Lutz >> Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht >> Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From aaron.s.knister at nasa.gov Wed Mar 29 15:17:19 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Wed, 29 Mar 2017 10:17:19 -0400 Subject: [gpfsug-discuss] GPFS / Spectrum Scale support with x86 SUSE SLES 12 SP2 to be expected when? In-Reply-To: References: <57030974-8C80-4808-87F1-3540E1271D6E@gmail.com> Message-ID: Ah, good point. Support means different things to different people, I suppose. The FAQ does say this which I'd not seen before: "IBM Spectrum Scale is not supported on SLES 12 SP2" That's actually a problem for me so I second Christoph's question, then. -Aaron On 3/29/17 10:01 AM, Simon Thompson (Research Computing - IT Services) wrote: > I think the question was supported, not works though. > > We've just been round something similar with TSM support, works fine on el7.3, but not supported, the docs actually said 7.1 or higher. We still got support and IBM now officially support 7.3, but I can see people may not want to deploy on something unsupported. > > Simon > ________________________________________ > From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Aaron Knister [aaron.s.knister at nasa.gov] > Sent: 29 March 2017 14:55 > To: gpfsug-discuss at spectrumscale.org > Subject: Re: [gpfsug-discuss] GPFS / Spectrum Scale support with x86 SUSE SLES 12 SP2 to be expected when? > > Just to make sure I wasn't mis-remembering I just installed 4.1.1.10 on > a SLES12 SP1 machine (it has a SLES12 SP2 kernel though), built and > loaded the kernel modules. There could be something significantly > different in user space between SLES12 SP1 and SP2 that prevents RPM > installation but I'd be surprised. > > -Aaron > > On 3/29/17 9:18 AM, Aaron Knister wrote: >> I could be wrong but I believe it is currently supported (as in it >> works, at least that's my recollection). Did you try with a particular >> version that wouldn't work? >> >> Sent from my iPhone >> >> On Mar 29, 2017, at 07:34, Christoph Krafft > > wrote: >> >>> Folks, >>> >>> does anyone have any tentative information when the above SUSE OS >>> Level will be supported? >>> >>> Thank you in advance for hints and tips! >>> >>> >>> Mit freundlichen Gr??en / Sincerely >>> >>> *Christoph Krafft* >>> >>> Client Technical Specialist - Power Systems, IBM Systems >>> Certified IT Specialist @ The Open Group >>> >>> ------------------------------------------------------------------------ >>> >>> Phone: +49 (0) 7034 643 2171 IBM Deutschland GmbH >>> <16373446.gif> >>> Mobile: +49 (0) 160 97 81 86 12 Am Weiher 24 >>> Email: ckrafft at de.ibm.com 65451 Kelsterbach >>> Germany >>> ------------------------------------------------------------------------ >>> IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter >>> Gesch?ftsf?hrung: Martina Koederitz (Vorsitzende), Nicole Reimer, >>> Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Stefan Lutz >>> Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht >>> Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 >>> >>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From abeattie at au1.ibm.com Wed Mar 29 22:53:06 2017 From: abeattie at au1.ibm.com (Andrew Beattie) Date: Wed, 29 Mar 2017 21:53:06 +0000 Subject: [gpfsug-discuss] GPFS / Spectrum Scale support with x86 SUSE SLES 12 SP2 to be expected when? In-Reply-To: References: , <57030974-8C80-4808-87F1-3540E1271D6E@gmail.com> Message-ID: An HTML attachment was scrubbed... URL: From Greg.Lehmann at csiro.au Thu Mar 30 00:12:35 2017 From: Greg.Lehmann at csiro.au (Greg.Lehmann at csiro.au) Date: Wed, 29 Mar 2017 23:12:35 +0000 Subject: [gpfsug-discuss] question on viewing block distribution across NSDs Message-ID: <2c0ba2b9466b42b4b3e5b3202ab26825@exch1-cdc.nexus.csiro.au> Hi All, I added some NSDs to an existing filesystem and ran mmrestripefs. I was sort of curious to see what the distribution looked like before and after the restripe. Is there any way of looking at it? Cheers, Greg. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Thu Mar 30 00:45:23 2017 From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]) Date: Wed, 29 Mar 2017 23:45:23 +0000 Subject: [gpfsug-discuss] question on viewing block distribution across NSDs References: question on viewing block distribution across NSDs Message-ID: <5F910253243E6A47B81A9A2EB424BBA101F5998D@NDMSMBX404.ndc.nasa.gov> Hi Greg, You could run an mmdf which will show you how full each NSD is. I'm not sure how to look back in time though to see the fs before the restripe. Do you perhaps have a gpfs.snap you took somewhat recently before the restripe? Maybe an internaldump in /tmp/mmfs somewhere? From: greg.lehmann at csiro.au Sent: 3/29/17, 19:21 To: gpfsug main discussion list Subject: [gpfsug-discuss] question on viewing block distribution across NSDs Hi All, I added some NSDs to an existing filesystem and ran mmrestripefs. I was sort of curious to see what the distribution looked like before and after the restripe. Is there any way of looking at it? Cheers, Greg. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Greg.Lehmann at csiro.au Thu Mar 30 00:51:37 2017 From: Greg.Lehmann at csiro.au (Greg.Lehmann at csiro.au) Date: Wed, 29 Mar 2017 23:51:37 +0000 Subject: [gpfsug-discuss] question on viewing block distribution across NSDs In-Reply-To: <5F910253243E6A47B81A9A2EB424BBA101F5998D@NDMSMBX404.ndc.nasa.gov> References: question on viewing block distribution across NSDs <5F910253243E6A47B81A9A2EB424BBA101F5998D@NDMSMBX404.ndc.nasa.gov> Message-ID: <3ec38bd65cad452f974c1d6c0d762853@exch1-cdc.nexus.csiro.au> Thanks. I don't have a snap. I'll keep that in mind for next time I do this. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] Sent: Thursday, 30 March 2017 9:45 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] question on viewing block distribution across NSDs Hi Greg, You could run an mmdf which will show you how full each NSD is. I'm not sure how to look back in time though to see the fs before the restripe. Do you perhaps have a gpfs.snap you took somewhat recently before the restripe? Maybe an internaldump in /tmp/mmfs somewhere? From: greg.lehmann at csiro.au Sent: 3/29/17, 19:21 To: gpfsug main discussion list Subject: [gpfsug-discuss] question on viewing block distribution across NSDs Hi All, I added some NSDs to an existing filesystem and ran mmrestripefs. I was sort of curious to see what the distribution looked like before and after the restripe. Is there any way of looking at it? Cheers, Greg. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Thu Mar 30 00:55:02 2017 From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]) Date: Wed, 29 Mar 2017 23:55:02 +0000 Subject: [gpfsug-discuss] question on viewing block distribution across NSDs References: [gpfsug-discuss] question on viewing block distribution across NSDs Message-ID: <5F910253243E6A47B81A9A2EB424BBA101F59A0A@NDMSMBX404.ndc.nasa.gov> I don't necessarily think you need to run a snap prior, just the output of mmdf should be enough. Something to keep in mind that I should have said before-- an mmdf can be stressful on your system particularly if you have spinning disk for your metadata. We're fortunate enough to have all flash for our metadata and I tend to take it for granted some times :) From: greg.lehmann at csiro.au Sent: 3/29/17, 19:52 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] question on viewing block distribution across NSDs Thanks. I don?t have a snap. I?ll keep that in mind for next time I do this. From: mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] Sent: Thursday, 30 March 2017 9:45 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] question on viewing block distribution across NSDs Hi Greg, You could run an mmdf which will show you how full each NSD is. I'm not sure how to look back in time though to see the fs before the restripe. Do you perhaps have a gpfs.snap you took somewhat recently before the restripe? Maybe an internaldump in /tmp/mmfs somewhere? From: greg.lehmann at csiro.au Sent: 3/29/17, 19:21 To: gpfsug main discussion list Subject: [gpfsug-discuss] question on viewing block distribution across NSDs Hi All, I added some NSDs to an existing filesystem and ran mmrestripefs. I was sort of curious to see what the distribution looked like before and after the restripe. Is there any way of looking at it? Cheers, Greg. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Greg.Lehmann at csiro.au Thu Mar 30 00:59:41 2017 From: Greg.Lehmann at csiro.au (Greg.Lehmann at csiro.au) Date: Wed, 29 Mar 2017 23:59:41 +0000 Subject: [gpfsug-discuss] question on viewing block distribution across NSDs In-Reply-To: <5F910253243E6A47B81A9A2EB424BBA101F59A0A@NDMSMBX404.ndc.nasa.gov> References: [gpfsug-discuss] question on viewing block distribution across NSDs <5F910253243E6A47B81A9A2EB424BBA101F59A0A@NDMSMBX404.ndc.nasa.gov> Message-ID: I was going to keep mmdf in mind, not gpfs.snap. I will now also keep in mind that mmdf can have an impact as at present we have spinning disk for metadata. The system I am playing around on is not production yet, so I am safe for the moment. Thanks again. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] Sent: Thursday, 30 March 2017 9:55 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] question on viewing block distribution across NSDs I don't necessarily think you need to run a snap prior, just the output of mmdf should be enough. Something to keep in mind that I should have said before-- an mmdf can be stressful on your system particularly if you have spinning disk for your metadata. We're fortunate enough to have all flash for our metadata and I tend to take it for granted some times :) From: greg.lehmann at csiro.au Sent: 3/29/17, 19:52 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] question on viewing block distribution across NSDs Thanks. I don?t have a snap. I?ll keep that in mind for next time I do this. From: mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] Sent: Thursday, 30 March 2017 9:45 AM To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] question on viewing block distribution across NSDs Hi Greg, You could run an mmdf which will show you how full each NSD is. I'm not sure how to look back in time though to see the fs before the restripe. Do you perhaps have a gpfs.snap you took somewhat recently before the restripe? Maybe an internaldump in /tmp/mmfs somewhere? From: greg.lehmann at csiro.au Sent: 3/29/17, 19:21 To: gpfsug main discussion list Subject: [gpfsug-discuss] question on viewing block distribution across NSDs Hi All, I added some NSDs to an existing filesystem and ran mmrestripefs. I was sort of curious to see what the distribution looked like before and after the restripe. Is there any way of looking at it? Cheers, Greg. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kums at us.ibm.com Thu Mar 30 14:04:09 2017 From: kums at us.ibm.com (Kumaran Rajaram) Date: Thu, 30 Mar 2017 08:04:09 -0500 Subject: [gpfsug-discuss] question on viewing block distribution across NSDs In-Reply-To: References: [gpfsug-discuss] question on viewing block distribution acrossNSDs <5F910253243E6A47B81A9A2EB424BBA101F59A0A@NDMSMBX404.ndc.nasa.gov> Message-ID: Hi, Yes, you could use "mmdf" to obtain file-system "usage" across the NSDs (comprising the file-system). If you want to obtain "data block distribution corresponding to a file across the NSDs", then there is a utility "mmgetlocation" in /usr/lpp/mmfs/samples/fpo that can be used to get file-data-blocks to NSD mapping. Example: # File-system comprises of single storage pool, all NSDs configured as dataAndMetadata, -m 1 -r 1, FS block-size=2MiB # mmlsfs gpfs1b | grep 'Block size' -B 2097152 Block size # The file-system is comprised of 10 x dataAndMetadata NSDs # mmlsdisk gpfs1b | grep DMD | wc -l 10 # Create a sample file that is 40MiB (20 data blocks) /mnt/sw/benchmarks/gpfsperf/gpfsperf create seq -r 2m -n 40m /mnt/gpfs1b/temp_dir/lf.s.1 # File size is 40 MiB (09:52:49) c25m3n07:~ # ls -lh /mnt/gpfs1b/temp_dir/lf.s.1 -rw-r--r-- 1 root root 40M Mar 17 09:52 /mnt/gpfs1b/temp_dir/lf.s.1 (09:52:54) c25m3n07:~ # du -sh /mnt/gpfs1b/temp_dir/lf.s.1 40M /mnt/gpfs1b/temp_dir/lf.s.1 # Verified through mmgetlocation that the file data blocks is uniformly striped across all the dataAndMetadata NSDs, with each NSD containing 2 file data blocks # In the output below, "DMD_NSDX" is name of the NSDs. (09:53:00) c25m3n07:~ # /usr/lpp/mmfs/samples/fpo/mmgetlocation -f /mnt/gpfs1b/temp_dir/lf.s.1 [FILE INFO] ------------------------------------------------------------------------ blockSize 2 MB blockGroupFactor 1 metadataBlockSize 2M writeAffinityDepth 0 flags: data replication: 1 max 2 storage pool name: system metadata replication: 1 max 2 Chunk 0 (offset 0) is located at disks: [ DMD_NSD09 c25m3n07-ib,c25m3n08-ib ] Chunk 1 (offset 2097152) is located at disks: [ DMD_NSD10 c25m3n08-ib,c25m3n07-ib ] Chunk 2 (offset 4194304) is located at disks: [ DMD_NSD01 c25m3n07-ib,c25m3n08-ib ] Chunk 3 (offset 6291456) is located at disks: [ DMD_NSD02 c25m3n08-ib,c25m3n07-ib ] Chunk 4 (offset 8388608) is located at disks: [ DMD_NSD03 c25m3n07-ib,c25m3n08-ib ] Chunk 5 (offset 10485760) is located at disks: [ DMD_NSD04 c25m3n08-ib,c25m3n07-ib ] Chunk 6 (offset 12582912) is located at disks: [ DMD_NSD05 c25m3n07-ib,c25m3n08-ib ] Chunk 7 (offset 14680064) is located at disks: [ DMD_NSD06 c25m3n08-ib,c25m3n07-ib ] Chunk 8 (offset 16777216) is located at disks: [ DMD_NSD07 c25m3n07-ib,c25m3n08-ib ] Chunk 9 (offset 18874368) is located at disks: [ DMD_NSD08 c25m3n08-ib,c25m3n07-ib ] Chunk 10 (offset 20971520) is located at disks: [ DMD_NSD09 c25m3n07-ib,c25m3n08-ib ] Chunk 11 (offset 23068672) is located at disks: [ DMD_NSD10 c25m3n08-ib,c25m3n07-ib ] Chunk 12 (offset 25165824) is located at disks: [ DMD_NSD01 c25m3n07-ib,c25m3n08-ib ] Chunk 13 (offset 27262976) is located at disks: [ DMD_NSD02 c25m3n08-ib,c25m3n07-ib ] Chunk 14 (offset 29360128) is located at disks: [ DMD_NSD03 c25m3n07-ib,c25m3n08-ib ] Chunk 15 (offset 31457280) is located at disks: [ DMD_NSD04 c25m3n08-ib,c25m3n07-ib ] Chunk 16 (offset 33554432) is located at disks: [ DMD_NSD05 c25m3n07-ib,c25m3n08-ib ] Chunk 17 (offset 35651584) is located at disks: [ DMD_NSD06 c25m3n08-ib,c25m3n07-ib ] Chunk 18 (offset 37748736) is located at disks: [ DMD_NSD07 c25m3n07-ib,c25m3n08-ib ] Chunk 19 (offset 39845888) is located at disks: [ DMD_NSD08 c25m3n08-ib,c25m3n07-ib ] [SUMMARY INFO] ---------------------------------------------------------------------------------------------------------- Replica num Nodename TotalChunkst Replica 1 : c25m3n07-ib,c25m3n08-ib: Total : 10 Replica 1 : c25m3n08-ib,c25m3n07-ib: Total : 10 Best Regards, -Kums From: To: Date: 03/29/2017 08:00 PM Subject: Re: [gpfsug-discuss] question on viewing block distribution across NSDs Sent by: gpfsug-discuss-bounces at spectrumscale.org I was going to keep mmdf in mind, not gpfs.snap. I will now also keep in mind that mmdf can have an impact as at present we have spinning disk for metadata. The system I am playing around on is not production yet, so I am safe for the moment. Thanks again. From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] Sent: Thursday, 30 March 2017 9:55 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] question on viewing block distribution across NSDs I don't necessarily think you need to run a snap prior, just the output of mmdf should be enough. Something to keep in mind that I should have said before-- an mmdf can be stressful on your system particularly if you have spinning disk for your metadata. We're fortunate enough to have all flash for our metadata and I tend to take it for granted some times :) From: greg.lehmann at csiro.au Sent: 3/29/17, 19:52 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] question on viewing block distribution across NSDs Thanks. I don?t have a snap. I?ll keep that in mind for next time I do this. From: mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] Sent: Thursday, 30 March 2017 9:45 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] question on viewing block distribution across NSDs Hi Greg, You could run an mmdf which will show you how full each NSD is. I'm not sure how to look back in time though to see the fs before the restripe. Do you perhaps have a gpfs.snap you took somewhat recently before the restripe? Maybe an internaldump in /tmp/mmfs somewhere? From: greg.lehmann at csiro.au Sent: 3/29/17, 19:21 To: gpfsug main discussion list Subject: [gpfsug-discuss] question on viewing block distribution across NSDs Hi All, I added some NSDs to an existing filesystem and ran mmrestripefs. I was sort of curious to see what the distribution looked like before and after the restripe. Is there any way of looking at it? Cheers, Greg._______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From willi.engeli at id.ethz.ch Thu Mar 30 14:29:26 2017 From: willi.engeli at id.ethz.ch (Engeli Willi (ID SD)) Date: Thu, 30 Mar 2017 13:29:26 +0000 Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Message-ID: Hi everybody, In our organization, the management of AD is strictly separated from management of storage. Since we install spectrum scale with protocol SMB and NFS support, we need to join the systems to AD, and have at least the joining user added as well to the local administrators group. Any idea of how to achieve this? Asking our Domain Admin is not the correct method to add other groups, this needs to be in our hands. Regards Willi -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5461 bytes Desc: not available URL: From r.sobey at imperial.ac.uk Thu Mar 30 14:53:15 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Thu, 30 Mar 2017 13:53:15 +0000 Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group In-Reply-To: References: Message-ID: Last time I checked simply adding a normal computer object to the domain didn't add the account of the adding user to the local administrators group and CES is no exception. Is it a political reason why you cannot ask your Domain Admin team to add you to the admin group for your CES cluster object? From there you can manage it yourself. Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Engeli Willi (ID SD) Sent: 30 March 2017 14:29 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Hi everybody, In our organization, the management of AD is strictly separated from management of storage. Since we install spectrum scale with protocol SMB and NFS support, we need to join the systems to AD, and have at least the joining user added as well to the local administrators group. Any idea of how to achieve this? Asking our Domain Admin is not the correct method to add other groups, this needs to be in our hands. Regards Willi -------------- next part -------------- An HTML attachment was scrubbed... URL: From laurence at qsplace.co.uk Thu Mar 30 15:02:19 2017 From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow) Date: Thu, 30 Mar 2017 15:02:19 +0100 Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group In-Reply-To: References: Message-ID: <2329870e-00f8-258c-187d-feec9589df93@qsplace.co.uk> Hi Willi, Could you just expand on your issue? Are you requiring CES to bind to AD to allow authenticated users to access your NFS/SMB shares. However you require the ability to add additional groups to these users on the CES system? Or are you trying to use your own account that can join the domain as a local admin on a CES node? -- Lauz On 30/03/2017 14:53, Sobey, Richard A wrote: > > Last time I checked simply adding a normal computer object to the > domain didn?t add the account of the adding user to the local > administrators group and CES is no exception. > > Is it a political reason why you cannot ask your Domain Admin team to > add you to the admin group for your CES cluster object? From there you > can manage it yourself. > > Richard > > *From:*gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of > *Engeli Willi (ID SD) > *Sent:* 30 March 2017 14:29 > *To:* gpfsug-discuss at spectrumscale.org > *Subject:* [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin > to local Administrators group > > Hi everybody, > > In our organization, the management of AD is strictly separated from > management of storage. Since we install spectrum scale with protocol > SMB and NFS support, we need to join the systems to AD, and have at > least the joining user added as well to the local administrators group. > > Any idea of how to achieve this? Asking our Domain Admin is not the > correct method to add other groups, this needs to be in our hands. > > Regards Willi > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From jgwood at sandia.gov Thu Mar 30 15:00:50 2017 From: jgwood at sandia.gov (Wood, Justin Gregory) Date: Thu, 30 Mar 2017 14:00:50 +0000 Subject: [gpfsug-discuss] [EXTERNAL] Spectrum Scale CES adds only Domain Admin to local Administrators group In-Reply-To: References: Message-ID: <069E1DC2-8711-4F4E-BD65-250C524ED341@sandia.gov> I recently went through the process of installing some protocol nodes and our organization also has a separate AD/Windows group. All I had to do was to get the AD admins to add a ?stub? computer object, then they made me an administrator of that object. That allowed me to run mmuserauth to do the setup. It was fairly simple and allowed me to do lots of work/testing without interaction with the AD group. -Justin. From: on behalf of "Engeli Willi (ID SD)" Reply-To: gpfsug main discussion list Date: Thursday, March 30, 2017 at 7:29 AM To: "gpfsug-discuss at spectrumscale.org" Subject: [EXTERNAL] [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Hi everybody, In our organization, the management of AD is strictly separated from management of storage. Since we install spectrum scale with protocol SMB and NFS support, we need to join the systems to AD, and have at least the joining user added as well to the local administrators group. Any idea of how to achieve this? Asking our Domain Admin is not the correct method to add other groups, this needs to be in our hands. Regards Willi -------------- next part -------------- An HTML attachment was scrubbed... URL: From willi.engeli at id.ethz.ch Thu Mar 30 15:23:40 2017 From: willi.engeli at id.ethz.ch (Engeli Willi (ID SD)) Date: Thu, 30 Mar 2017 14:23:40 +0000 Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Message-ID: >-Last time I checked simply adding a normal computer object to the domain didn't add the account of the adding user to the local administrators group and CES is no exception. We have been using before a competitor Product as a NAS system. With that system, we were able to define virtual NAS Servers, each one joined as an independent object to AD. When joined, we found the 'Domain Admin' group and the joining user as member of local administrators group of that virtual server. Since out AD is quite big, it is structured into many OU. We as the Storage OU have OU admin rights, but we are not member of "Domain Admin" group. Looking Back, we were able by ourselves to add the required groups as needed to the local Administrators group of the NAS server. Why is this important? Since we have quit a mix of OS accessing our shares, some of the create exclusive access rights at the time they create profiles etc. At the end of the lifecycle, one needs to delete those files via the SMB / NFSV4 protocol, which is difficult if not having access rights. On the other hand, we have seen situations, where one OS corrupted the ACL and could not access anymore. Also this needs to be handled by us, giving us a hard time not being member of the administrators group. I.e. the MS tool subinacl does check the privileges before trying to modify ACLs, and if not being member of the Administrators group, not all required privileges are granted. >-Is it a political reason why you cannot ask your Domain Admin team to add you to the admin group for your CES cluster object? From there you can manage it yourself. Yes and no. We have a clear boundary, where we need to be able to manage the AD Objects, and for security reason it seems to make sense to not use Domain Admin Accounts for such kind of work (statement of our AD Group). So much for the Situation, did I missed something? Willi -----Urspr?ngliche Nachricht----- Von: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] Im Auftrag von gpfsug-discuss-request at spectrumscale.org Gesendet: Donnerstag, 30. M?rz 2017 16:02 An: gpfsug-discuss at spectrumscale.org Betreff: gpfsug-discuss Digest, Vol 62, Issue 77 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Spectrum Scale CES adds only Domain Admin to local Administrators group (Engeli Willi (ID SD)) 2. Re: Spectrum Scale CES adds only Domain Admin to local Administrators group (Sobey, Richard A) 3. Re: Spectrum Scale CES adds only Domain Admin to local Administrators group (Laurence Horrocks-Barlow) ---------------------------------------------------------------------- Message: 1 Date: Thu, 30 Mar 2017 13:29:26 +0000 From: "Engeli Willi (ID SD)" To: "gpfsug-discuss at spectrumscale.org" Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Message-ID: Content-Type: text/plain; charset="us-ascii" Hi everybody, In our organization, the management of AD is strictly separated from management of storage. Since we install spectrum scale with protocol SMB and NFS support, we need to join the systems to AD, and have at least the joining user added as well to the local administrators group. Any idea of how to achieve this? Asking our Domain Admin is not the correct method to add other groups, this needs to be in our hands. Regards Willi -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5461 bytes Desc: not available URL: ------------------------------ Message: 2 Date: Thu, 30 Mar 2017 13:53:15 +0000 From: "Sobey, Richard A" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Message-ID: Content-Type: text/plain; charset="us-ascii" Last time I checked simply adding a normal computer object to the domain didn't add the account of the adding user to the local administrators group and CES is no exception. Is it a political reason why you cannot ask your Domain Admin team to add you to the admin group for your CES cluster object? From there you can manage it yourself. Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Engeli Willi (ID SD) Sent: 30 March 2017 14:29 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Hi everybody, In our organization, the management of AD is strictly separated from management of storage. Since we install spectrum scale with protocol SMB and NFS support, we need to join the systems to AD, and have at least the joining user added as well to the local administrators group. Any idea of how to achieve this? Asking our Domain Admin is not the correct method to add other groups, this needs to be in our hands. Regards Willi -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 3 Date: Thu, 30 Mar 2017 15:02:19 +0100 From: Laurence Horrocks-Barlow To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Message-ID: <2329870e-00f8-258c-187d-feec9589df93 at qsplace.co.uk> Content-Type: text/plain; charset="windows-1252"; Format="flowed" Hi Willi, Could you just expand on your issue? Are you requiring CES to bind to AD to allow authenticated users to access your NFS/SMB shares. However you require the ability to add additional groups to these users on the CES system? Or are you trying to use your own account that can join the domain as a local admin on a CES node? -- Lauz On 30/03/2017 14:53, Sobey, Richard A wrote: > > Last time I checked simply adding a normal computer object to the > domain didn?t add the account of the adding user to the local > administrators group and CES is no exception. > > Is it a political reason why you cannot ask your Domain Admin team to > add you to the admin group for your CES cluster object? From there you > can manage it yourself. > > Richard > > *From:*gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of > *Engeli Willi (ID SD) > *Sent:* 30 March 2017 14:29 > *To:* gpfsug-discuss at spectrumscale.org > *Subject:* [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin > to local Administrators group > > Hi everybody, > > In our organization, the management of AD is strictly separated from > management of storage. Since we install spectrum scale with protocol > SMB and NFS support, we need to join the systems to AD, and have at > least the joining user added as well to the local administrators group. > > Any idea of how to achieve this? Asking our Domain Admin is not the > correct method to add other groups, this needs to be in our hands. > > Regards Willi > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 62, Issue 77 ********************************************** -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5461 bytes Desc: not available URL: From gmcpheeters at anl.gov Thu Mar 30 15:39:45 2017 From: gmcpheeters at anl.gov (McPheeters, Gordon) Date: Thu, 30 Mar 2017 14:39:45 +0000 Subject: [gpfsug-discuss] question on viewing block distribution across NSDs In-Reply-To: <2c0ba2b9466b42b4b3e5b3202ab26825@exch1-cdc.nexus.csiro.au> References: <2c0ba2b9466b42b4b3e5b3202ab26825@exch1-cdc.nexus.csiro.au> Message-ID: <2A55B023-7DE1-417E-BD30-88FD44DF0ADA@anl.gov> mmdf will show you the distribution across NSDs. Gordon On Mar 29, 2017, at 6:12 PM, Greg.Lehmann at csiro.au wrote: Hi All, I added some NSDs to an existing filesystem and ran mmrestripefs. I was sort of curious to see what the distribution looked like before and after the restripe. Is there any way of looking at it? Cheers, Greg. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Thu Mar 30 15:50:13 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Thu, 30 Mar 2017 14:50:13 +0000 Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group In-Reply-To: References: Message-ID: Did your AD team perchance define a group policy on the OU such that any object placed into that OU inherited a specific set of local administrators? That's the only way I can think that your NAS ended up with the calling user in the local admin group. I understand where you're coming from - we do not manage AD ourselves but we do not want Domain Admins to have administrator control of our CES nodes. So once it was joined to AD (with their help) I simply removed Domain Admins and added the storage team DL in its place. But back to the original question, I'm afraid I do not know how to make CES add a specific user to its local administrator group. Richard -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Engeli Willi (ID SD) Sent: 30 March 2017 15:24 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group >-Last time I checked simply adding a normal computer object to the >domain didn't add the account of the adding user to the local administrators group and CES is no exception. We have been using before a competitor Product as a NAS system. With that system, we were able to define virtual NAS Servers, each one joined as an independent object to AD. When joined, we found the 'Domain Admin' group and the joining user as member of local administrators group of that virtual server. Since out AD is quite big, it is structured into many OU. We as the Storage OU have OU admin rights, but we are not member of "Domain Admin" group. Looking Back, we were able by ourselves to add the required groups as needed to the local Administrators group of the NAS server. Why is this important? Since we have quit a mix of OS accessing our shares, some of the create exclusive access rights at the time they create profiles etc. At the end of the lifecycle, one needs to delete those files via the SMB / NFSV4 protocol, which is difficult if not having access rights. On the other hand, we have seen situations, where one OS corrupted the ACL and could not access anymore. Also this needs to be handled by us, giving us a hard time not being member of the administrators group. I.e. the MS tool subinacl does check the privileges before trying to modify ACLs, and if not being member of the Administrators group, not all required privileges are granted. >-Is it a political reason why you cannot ask your Domain Admin team to >add you to the admin group for your CES cluster object? From there you can manage it yourself. Yes and no. We have a clear boundary, where we need to be able to manage the AD Objects, and for security reason it seems to make sense to not use Domain Admin Accounts for such kind of work (statement of our AD Group). So much for the Situation, did I missed something? Willi -----Urspr?ngliche Nachricht----- Von: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] Im Auftrag von gpfsug-discuss-request at spectrumscale.org Gesendet: Donnerstag, 30. M?rz 2017 16:02 An: gpfsug-discuss at spectrumscale.org Betreff: gpfsug-discuss Digest, Vol 62, Issue 77 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Spectrum Scale CES adds only Domain Admin to local Administrators group (Engeli Willi (ID SD)) 2. Re: Spectrum Scale CES adds only Domain Admin to local Administrators group (Sobey, Richard A) 3. Re: Spectrum Scale CES adds only Domain Admin to local Administrators group (Laurence Horrocks-Barlow) ---------------------------------------------------------------------- Message: 1 Date: Thu, 30 Mar 2017 13:29:26 +0000 From: "Engeli Willi (ID SD)" To: "gpfsug-discuss at spectrumscale.org" Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Message-ID: Content-Type: text/plain; charset="us-ascii" Hi everybody, In our organization, the management of AD is strictly separated from management of storage. Since we install spectrum scale with protocol SMB and NFS support, we need to join the systems to AD, and have at least the joining user added as well to the local administrators group. Any idea of how to achieve this? Asking our Domain Admin is not the correct method to add other groups, this needs to be in our hands. Regards Willi -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5461 bytes Desc: not available URL: ------------------------------ Message: 2 Date: Thu, 30 Mar 2017 13:53:15 +0000 From: "Sobey, Richard A" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Message-ID: Content-Type: text/plain; charset="us-ascii" Last time I checked simply adding a normal computer object to the domain didn't add the account of the adding user to the local administrators group and CES is no exception. Is it a political reason why you cannot ask your Domain Admin team to add you to the admin group for your CES cluster object? From there you can manage it yourself. Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Engeli Willi (ID SD) Sent: 30 March 2017 14:29 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Hi everybody, In our organization, the management of AD is strictly separated from management of storage. Since we install spectrum scale with protocol SMB and NFS support, we need to join the systems to AD, and have at least the joining user added as well to the local administrators group. Any idea of how to achieve this? Asking our Domain Admin is not the correct method to add other groups, this needs to be in our hands. Regards Willi -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 3 Date: Thu, 30 Mar 2017 15:02:19 +0100 From: Laurence Horrocks-Barlow To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Message-ID: <2329870e-00f8-258c-187d-feec9589df93 at qsplace.co.uk> Content-Type: text/plain; charset="windows-1252"; Format="flowed" Hi Willi, Could you just expand on your issue? Are you requiring CES to bind to AD to allow authenticated users to access your NFS/SMB shares. However you require the ability to add additional groups to these users on the CES system? Or are you trying to use your own account that can join the domain as a local admin on a CES node? -- Lauz On 30/03/2017 14:53, Sobey, Richard A wrote: > > Last time I checked simply adding a normal computer object to the > domain didn?t add the account of the adding user to the local > administrators group and CES is no exception. > > Is it a political reason why you cannot ask your Domain Admin team to > add you to the admin group for your CES cluster object? From there you > can manage it yourself. > > Richard > > *From:*gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of > *Engeli Willi (ID SD) > *Sent:* 30 March 2017 14:29 > *To:* gpfsug-discuss at spectrumscale.org > *Subject:* [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin > to local Administrators group > > Hi everybody, > > In our organization, the management of AD is strictly separated from > management of storage. Since we install spectrum scale with protocol > SMB and NFS support, we need to join the systems to AD, and have at > least the joining user added as well to the local administrators group. > > Any idea of how to achieve this? Asking our Domain Admin is not the > correct method to add other groups, this needs to be in our hands. > > Regards Willi > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 62, Issue 77 ********************************************** From Mark.Bush at siriuscom.com Thu Mar 30 17:09:15 2017 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Thu, 30 Mar 2017 16:09:15 +0000 Subject: [gpfsug-discuss] AFM vs mmremotefs Message-ID: <5A22811B-7FBD-4091-B6FD-E5E74053DDF2@siriuscom.com> When does it make sense to use mmremotefs vs AFM? With mmremotefs the actual data lives on the remote cluster right so if you need data closer then AFM make more sense to use? If the clusters get disconnected with mmremotefs I have not access to the data with AFM I do. Do I have these concepts right in my head? [id:image001.png at 01D2709D.6EF65720] Mark R. Bush| Storage Architect Mobile: 210-237-8415 Twitter: @bushmr | LinkedIn: /markreedbush 10100 Reunion Place, Suite 500, San Antonio, TX 78216 www.siriuscom.com |mark.bush at siriuscom.com This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 8745 bytes Desc: image001.png URL: From S.J.Thompson at bham.ac.uk Thu Mar 30 17:19:07 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Thu, 30 Mar 2017 16:19:07 +0000 Subject: [gpfsug-discuss] AFM vs mmremotefs Message-ID: We are just planning to deploy AFM as a cache tier for our HPC cluster and bulk data storage. So whilst they are locally connected etc, I can use a different type of storage for performance for HPC work. Bulk data has two copies over two data centres, so the AFM cache would ACK the write to the HPC client and then "push it as soon as possible" to the home (bulk data storage and do the two copies). Similarly for ingest systems, for example streaming data off instruments, you might want to do really really fast (to a flash system) and then have the AFM write to home at some point. (of course your cache capacity needs to be sufficiently large that you are able to push writes/expel). On the flip side, I don't think AFM has remote identity mapping (someone might correct me on that of course) So I don't think its just about data location/connectivity. Simon From: > on behalf of "Mark.Bush at siriuscom.com" > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Thursday, 30 March 2017 at 17:09 To: "gpfsug-discuss at spectrumscale.org" > Subject: [gpfsug-discuss] AFM vs mmremotefs When does it make sense to use mmremotefs vs AFM? With mmremotefs the actual data lives on the remote cluster right so if you need data closer then AFM make more sense to use? If the clusters get disconnected with mmremotefs I have not access to the data with AFM I do. Do I have these concepts right in my head? [id:image001.png at 01D2709D.6EF65720] Mark R. Bush| Storage Architect Mobile: 210-237-8415 Twitter: @bushmr | LinkedIn: /markreedbush 10100 Reunion Place, Suite 500, San Antonio, TX 78216 www.siriuscom.com |mark.bush at siriuscom.com This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 8745 bytes Desc: image001.png URL: From makaplan at us.ibm.com Thu Mar 30 17:26:34 2017 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Thu, 30 Mar 2017 11:26:34 -0500 Subject: [gpfsug-discuss] AFM vs mmremotefs In-Reply-To: <5A22811B-7FBD-4091-B6FD-E5E74053DDF2@siriuscom.com> References: <5A22811B-7FBD-4091-B6FD-E5E74053DDF2@siriuscom.com> Message-ID: I think you "got it" - despite the name "remote" - mmremotefs is an access method and a way of organizing your data and machines into several clusters - which can work great if your network(s) are fast enough. For example you can have one or more client clusters with no GPFS(Spectrum Scale) file systems stored "locally". And one or several clusters that are servers to those client clusters. AFM is about caching and replicating files... You might do that because of "remote-ness" - or for other reasons... --marc From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Date: 03/30/2017 12:09 PM Subject: [gpfsug-discuss] AFM vs mmremotefs Sent by: gpfsug-discuss-bounces at spectrumscale.org When does it make sense to use mmremotefs vs AFM? With mmremotefs the actual data lives on the remote cluster right so if you need data closer then AFM make more sense to use? If the clusters get disconnected with mmremotefs I have not access to the data with AFM I do. Do I have these concepts right in my head? Mark R. Bush| Storage Architect Mobile: 210-237-8415 Twitter: @bushmr | LinkedIn: /markreedbush 10100 Reunion Place, Suite 500, San Antonio, TX 78216 www.siriuscom.com |mark.bush at siriuscom.com This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 8745 bytes Desc: not available URL: From gcorneau at us.ibm.com Thu Mar 30 17:42:38 2017 From: gcorneau at us.ibm.com (Glen Corneau) Date: Thu, 30 Mar 2017 10:42:38 -0600 Subject: [gpfsug-discuss] AFM vs mmremotefs In-Reply-To: References: <5A22811B-7FBD-4091-B6FD-E5E74053DDF2@siriuscom.com> Message-ID: Just a extra note: Multi-cluster (mmremotefs) does not always mean "network" for file system data transfer. If both cluster(s) see the storage over the SAN then the data happens over that path. This is common in a localized scenario for authorization reasons (i.e. nodes 1,2,3 can access file systems A & B, but not C; but nodes 4,5,6 in a separate cluster can access all 3 file systems) ------------------ Glen Corneau Power Systems Washington Systems Center gcorneau at us.ibm.com From: Marc A Kaplan/Watson/IBM at IBMUS To: gpfsug main discussion list Date: 03/30/2017 11:27 AM Subject: Re: [gpfsug-discuss] AFM vs mmremotefs Sent by: gpfsug-discuss-bounces at spectrumscale.org I think you "got it" - despite the name "remote" - mmremotefs is an access method and a way of organizing your data and machines into several clusters - which can work great if your network(s) are fast enough. For example you can have one or more client clusters with no GPFS(Spectrum Scale) file systems stored "locally". And one or several clusters that are servers to those client clusters. AFM is about caching and replicating files... You might do that because of "remote-ness" - or for other reasons... --marc From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Date: 03/30/2017 12:09 PM Subject: [gpfsug-discuss] AFM vs mmremotefs Sent by: gpfsug-discuss-bounces at spectrumscale.org When does it make sense to use mmremotefs vs AFM? With mmremotefs the actual data lives on the remote cluster right so if you need data closer then AFM make more sense to use? If the clusters get disconnected with mmremotefs I have not access to the data with AFM I do. Do I have these concepts right in my head? Mark R. Bush| Storage Architect Mobile: 210-237-8415 Twitter: @bushmr| LinkedIn: /markreedbush 10100 Reunion Place, Suite 500, San Antonio, TX 78216 www.siriuscom.com|mark.bush at siriuscom.com This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 26117 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 8745 bytes Desc: not available URL: From christof.schmitt at us.ibm.com Thu Mar 30 21:18:21 2017 From: christof.schmitt at us.ibm.com (Christof Schmitt) Date: Thu, 30 Mar 2017 13:18:21 -0700 Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group In-Reply-To: References: Message-ID: willi.engeli at id.ethz.ch wrote on 03/30/2017 07:23:40 AM: > >-Last time I checked simply adding a normal computer object to the domain > didn't add the account of the adding user to the local administrators group > and CES is no exception. > > We have been using before a competitor Product as a NAS system. With that > system, we were able to define virtual NAS Servers, each one joined as an > independent object to AD. When joined, we found the 'Domain Admin' group and > the joining user as member of local administrators group of that virtual > server. > Since out AD is quite big, it is structured into many OU. We as the Storage > OU have OU admin rights, but we are not member of "Domain Admin" group. > Looking Back, we were able by ourselves to add the required groups as needed > to the local Administrators group of the NAS server. > Why is this important? Since we have quit a mix of OS accessing our shares, > some of the create exclusive access rights at the time they create profiles > etc. At the end of the lifecycle, one needs to delete those files via the > SMB / NFSV4 protocol, which is difficult if not having access rights. On the > other hand, we have seen situations, where one OS corrupted the ACL and > could not access anymore. Also this needs to be handled by us, giving us a > hard time not being member of the administrators group. I.e. the MS tool > subinacl does check the privileges before trying to modify ACLs, and if not > being member of the Administrators group, not all required privileges are > granted. There is two parts to that in Spectrum Scale: 1) There is an option to declare a user as 'admin users'. The notion there is that this user is mapped to root on access, thus this user can always access files and fix access issues. The user defined here should not be used for normal usage, this is only recommended for data migrations and to fix access issues. 2) When joining Spectrum Scale to an Active Directory domain, the Domain Admins groups is added to the internal Administrators group (sometimes referred to as BUILTIN\Administrators). One way to change the membership in that group would be through the MMC on a Windows client. Initially only Domain Admins are allowed, a member of this group would be required to add other users or groups. Alternatively, the "net sam" interface can be used to modify the group from root access on the protocol nodes: /usr/lpp/mmfs/bin/net sam listmem Administrators to list the members of the Administrators groups. /usr/lpp/mmfs/bin/net sam addmem Administrators DOMAIN\user to add a member. /usr/lpp/mmfs/bin/net sam delmem Administrators DOMAIN\user to remove a member This is currently an untested feature and not exposed through the CLI. If there is a need to have this exposed through the CLI or GUI, that should be requested through a RFE so that it can feed into the planning and prioritization for future releases. Regards, Christof Schmitt || IBM || Spectrum Scale Development || Tucson, AZ christof.schmitt at us.ibm.com || +1-520-799-2469 (T/L: 321-2469) From willi.engeli at id.ethz.ch Fri Mar 31 12:46:02 2017 From: willi.engeli at id.ethz.ch (Engeli Willi (ID SD)) Date: Fri, 31 Mar 2017 11:46:02 +0000 Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Message-ID: Hi Christoph, This solved my issues in most areas. Now I will probably add our Storage Management Group to local Administrators group, this way we are able to use all strong utilities like subinacl etc, and will be able to migrate to Spectrum Scale using robocopy with /ZB option working properly. For our Share-responsible Administrator we probably will add their Management user to the 'admin Users' option of the specific share allowing them to do wat ever they need to do, knowing that some tools may work with limitations. Do you know if we may also add a builtin group named BackupOperators? Regards Willi -----Urspr?ngliche Nachricht----- Von: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] Im Auftrag von gpfsug-discuss-request at spectrumscale.org Gesendet: Freitag, 31. M?rz 2017 13:00 An: gpfsug-discuss at spectrumscale.org Betreff: gpfsug-discuss Digest, Vol 62, Issue 82 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Spectrum Scale CES adds only Domain Admin to local Administrators group (Christof Schmitt) ---------------------------------------------------------------------- Message: 1 Date: Thu, 30 Mar 2017 13:18:21 -0700 From: "Christof Schmitt" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Message-ID: Content-Type: text/plain; charset="US-ASCII" willi.engeli at id.ethz.ch wrote on 03/30/2017 07:23:40 AM: > >-Last time I checked simply adding a normal computer object to the domain > didn't add the account of the adding user to the local administrators group > and CES is no exception. > > We have been using before a competitor Product as a NAS system. With that > system, we were able to define virtual NAS Servers, each one joined as an > independent object to AD. When joined, we found the 'Domain Admin' > group and > the joining user as member of local administrators group of that > virtual server. > Since out AD is quite big, it is structured into many OU. We as the Storage > OU have OU admin rights, but we are not member of "Domain Admin" group. > Looking Back, we were able by ourselves to add the required groups as needed > to the local Administrators group of the NAS server. > Why is this important? Since we have quit a mix of OS accessing our shares, > some of the create exclusive access rights at the time they create profiles > etc. At the end of the lifecycle, one needs to delete those files via the > SMB / NFSV4 protocol, which is difficult if not having access rights. > On the > other hand, we have seen situations, where one OS corrupted the ACL > and could not access anymore. Also this needs to be handled by us, > giving us a > hard time not being member of the administrators group. I.e. the MS > tool subinacl does check the privileges before trying to modify ACLs, > and if not > being member of the Administrators group, not all required privileges are > granted. There is two parts to that in Spectrum Scale: 1) There is an option to declare a user as 'admin users'. The notion there is that this user is mapped to root on access, thus this user can always access files and fix access issues. The user defined here should not be used for normal usage, this is only recommended for data migrations and to fix access issues. 2) When joining Spectrum Scale to an Active Directory domain, the Domain Admins groups is added to the internal Administrators group (sometimes referred to as BUILTIN\Administrators). One way to change the membership in that group would be through the MMC on a Windows client. Initially only Domain Admins are allowed, a member of this group would be required to add other users or groups. Alternatively, the "net sam" interface can be used to modify the group from root access on the protocol nodes: /usr/lpp/mmfs/bin/net sam listmem Administrators to list the members of the Administrators groups. /usr/lpp/mmfs/bin/net sam addmem Administrators DOMAIN\user to add a member. /usr/lpp/mmfs/bin/net sam delmem Administrators DOMAIN\user to remove a member This is currently an untested feature and not exposed through the CLI. If there is a need to have this exposed through the CLI or GUI, that should be requested through a RFE so that it can feed into the planning and prioritization for future releases. Regards, Christof Schmitt || IBM || Spectrum Scale Development || Tucson, AZ christof.schmitt at us.ibm.com || +1-520-799-2469 (T/L: 321-2469) ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 62, Issue 82 ********************************************** -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5461 bytes Desc: not available URL: From pinto at scinet.utoronto.ca Fri Mar 31 15:18:34 2017 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Fri, 31 Mar 2017 10:18:34 -0400 Subject: [gpfsug-discuss] BIG LAG since 3.5 on quota accounting reconciliation Message-ID: <20170331101834.162212izrlj4swu2@support.scinet.utoronto.ca> In the old days of DDN 9900 and gpfs 3.4 I only had to run mmcheckquota once a month, usually after the massive monthly purge. I noticed that starting with the GSS and ESS appliances under 3.5 that I needed to run mmcheckquota more often, at least once a week, or as often as daily, to clear the slippage errors in the accounting information, otherwise users complained that they were hitting their quotas, even throughout they deleted a lot of stuff. More recently we adopted a G200 appliance (1.8PB), with v4.1, and now things have gotten worst, and I have to run it twice daily, just in case. So, what I am missing? Is their a parameter since 3.5 and through 4.1 that we can set, so that GPFS will reconcile the quota accounting internally more often and on its own? Thanks Jaime ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. From billowen at us.ibm.com Fri Mar 31 19:23:35 2017 From: billowen at us.ibm.com (Bill Owen) Date: Fri, 31 Mar 2017 11:23:35 -0700 Subject: [gpfsug-discuss] mmnetverify In-Reply-To: References: , <18729d8a86fc430fac09c1c3a22159d2@mbxtoa1.winmail.deshaw.com> Message-ID: mmnetverify is a new tool that aims to make it easier to identify network problems. Regarding bandwidth commands, in 4.2.2, there are two options: mmnetverify bandwidth-node - [1 to 1] this will communicate from local node (or one or more nodes specified with -N option) to one or more target nodes. The bandwidth tests are executed serially from nodes in node list to target, iterating through each target node, one by one. The serially calculated bandwidth with each node is reported. mmnetverify bandwidth-cluster - [1 to many] this is a measure of parallel communication from the local node (or one or more nodes specified with -N option) to all of the other nodes in the cluster. The concurrent bandwidth with each target node in the cluster is reported. In both of these tests, we establish a socket connection, and pass a fixed number of bytes over the connection and calculate bandwidth based on how long that transmission took. For 4.2.3, there is a new bandwidth test called gnr-bandwidth. It is similar to the bandwidth-cluster [1 to many] except that it uses the following steps: 1. establish connection from node to all other target nodes in the cluster 2. start sending data to target for some ramp up period 3. after ramp up period, continue sending data for test period 4. calculate bandwidth based on bytes transmitted during test period The bandwidth to each node is summed to return a total bandwidth from the command node to the other nodes in the cluster. In future releases, we may modify bandwidth-node & bandwidth-cluster tests to use the gnr-bandwidth methodology (and deprecate gnr-bandwidth). Your feedback on how to improve mmnetverify is appreciated. Regarding: > We found some weird looking numbers that i don't quite understand and not in the places we might expect. > For example between hosts on the same switch, traffic flowing to another switch and traffic flowing to > nodes in another data centre where it's several switch hops. Some nodes over there were significantly > faster than switch local nodes. Note that system load can impact the test results. Is it possible that the slow nodes on the local switch were heavily loaded? Or is it possible they are using an interface that is lower bandwidth? (sorry, i had to ask that one to be sure...) Regards, Bill Owen billowen at us.ibm.com Spectrum Scale Development 520-799-4829 From: "Simon Thompson (Research Computing - IT Services)" To: gpfsug main discussion list Date: 03/17/2017 01:13 PM Subject: Re: [gpfsug-discuss] mmnetverify Sent by: gpfsug-discuss-bounces at spectrumscale.org It looks to run sequential tests to each node one at a time and isn't using NSD protocol but echo server. We found some weird looking numbers that i don't quite understand and not in the places we might expect. For example between hosts on the same switch, traffic flowing to another switch and traffic flowing to nodes in another data centre where it's several switch hops. Some nodes over there were significantly faster than switch local nodes. I think it was only added in 4.2.2 and is listed as "not yet a replacement for nsdperf". I get that is different as it's using NSD protocol, but was struggling a bit with what mmnetverify might be doing. Simon From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Sanchez, Paul [Paul.Sanchez at deshaw.com] Sent: 17 March 2017 19:43 To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] mmnetverify Sven will tell you: "RPC isn't streaming" and that may account for the discrepancy. If the tests are doing any "fan-in" where multiple nodes are sending to single node, then it's also possible that you are exhausting switch buffer memory in a way that a 1:1 iperf wouldn't. For our internal benchmarking we've used /usr/lpp/mmfs/samples/net/nsdperf to more closely estimate the real performance. I haven't played with mmnetverify yet though. -Paul -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (Research Computing - IT Services) Sent: Friday, March 17, 2017 2:50 PM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] mmnetverify Hi all, Just wondering if anyone has used the mmnetverify tool at all? Having made some changes to our internal L3 routing this week, I was interested to see what it claimed. As a side-note, it picked up some DNS resolution issues, though I'm not clear on some of those why it was claiming this as doing a "dig" on the node, it resolved fine (but adding the NSD servers to the hosts files cleared the error). Its actually the bandwidth tests that I'm interested in hearing other people's experience with as the numbers that some out from it are very different (lower) than if we use iperf to test performance between two nodes. Anyone any thoughts at all on this? Thanks Simon _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From christof.schmitt at us.ibm.com Fri Mar 31 21:22:36 2017 From: christof.schmitt at us.ibm.com (Christof Schmitt) Date: Fri, 31 Mar 2017 13:22:36 -0700 Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin tolocal In-Reply-To: References: Message-ID: willi.engeli at id.ethz.ch wrote on 03/31/2017 04:46:02 AM: > Hi Christoph, > This solved my issues in most areas. > Now I will probably add our Storage Management Group to local > Administrators group, this way we are able to use all strong > utilities like subinacl etc, and will be able to migrate to Spectrum > Scale using robocopy with /ZB option working properly. > For our Share-responsible Administrator we probably will add their > Management user to the 'admin Users' option of the specific share > allowing them to do wat ever they need to do, knowing that some > tools may work with limitations. > > Do you know if we may also add a builtin group named BackupOperators? Privileges are not supported in Spectrum Scale: https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.2/com.ibm.spectrum.scale.v4r22.doc/bl1adm_fileauthlimitations.htm "Access privileges defined in Windows are not honored. Those privileges are tied to administrator groups and allow access, where the ACL alone does not grant it." You can create a group and assign the BackupOperator privilege: /usr/lpp/mmfs/bin/net sam createbuiltingroup 'Backup Operators' /usr/lpp/mmfs/bin/net sam rights grant 'Backup Operators' SeBackupPrivilege Without looking at all the details, i would suspect that this does not work. Access for a member of this group would overwrite the internal access check in Samba, but i would expect that it would fail as the file system also enforces the permissions defined in the ACL, and these are not overwritten by the additional privilege. The current workaround is the 'admin users' option. You might want to raise a RFE to request better support of the Backup privilege and the "Backup Operators" group. Regards, Christof Schmitt || IBM || Spectrum Scale Development || Tucson, AZ christof.schmitt at us.ibm.com || +1-520-799-2469 (T/L: 321-2469) From a.g.richmond at leeds.ac.uk Wed Mar 1 11:19:17 2017 From: a.g.richmond at leeds.ac.uk (Aidan Richmond) Date: Wed, 1 Mar 2017 11:19:17 +0000 Subject: [gpfsug-discuss] Issues getting SMB shares working. Message-ID: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> Hello I've recently taken over managing a GPFS installation providing storage to a research facility at Leeds University. The system was setup by a colleague who recently left and before he left he configured NFSV4 access which services an important subset, but not all the users who require access to the storage. SMB access was part of the deployment plan, but he didn't have time to get that working. I've been attempting to get an SMB share up and running myself and have enabled the SMB service with "mmces service enable SMB", have added a share with "mmsmb export add share /path/to/folder" and "mmsmb export list" seems to show everything is okay. However attempts to access the share through the floating protocol addresses fail and I have no idea how to diagnose the issue. Any advice/help would be welcome. -- Aidan Richmond Apple/Unix Support Officer, IT Garstang 10.137 Faculty of Biological Sciences University of Leeds Clarendon Way LS2 9JT Tel:0113 3434252 a.g.richmond at leeds.ac.uk From r.sobey at imperial.ac.uk Wed Mar 1 11:33:13 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Wed, 1 Mar 2017 11:33:13 +0000 Subject: [gpfsug-discuss] Issues getting SMB shares working. In-Reply-To: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> References: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> Message-ID: Have you configured authentication? Mmuserauth service list --data-access-method file -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aidan Richmond Sent: 01 March 2017 11:19 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Issues getting SMB shares working. Hello I've recently taken over managing a GPFS installation providing storage to a research facility at Leeds University. The system was setup by a colleague who recently left and before he left he configured NFSV4 access which services an important subset, but not all the users who require access to the storage. SMB access was part of the deployment plan, but he didn't have time to get that working. I've been attempting to get an SMB share up and running myself and have enabled the SMB service with "mmces service enable SMB", have added a share with "mmsmb export add share /path/to/folder" and "mmsmb export list" seems to show everything is okay. However attempts to access the share through the floating protocol addresses fail and I have no idea how to diagnose the issue. Any advice/help would be welcome. -- Aidan Richmond Apple/Unix Support Officer, IT Garstang 10.137 Faculty of Biological Sciences University of Leeds Clarendon Way LS2 9JT Tel:0113 3434252 a.g.richmond at leeds.ac.uk _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From a.g.richmond at leeds.ac.uk Wed Mar 1 11:36:04 2017 From: a.g.richmond at leeds.ac.uk (Aidan Richmond) Date: Wed, 1 Mar 2017 11:36:04 +0000 Subject: [gpfsug-discuss] Issues getting SMB shares working. In-Reply-To: References: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> Message-ID: <901c6472-1c9c-3551-555e-22bc849fe2b1@leeds.ac.uk> Hello It returns the following: FILE access configuration : USERDEFINED PARAMETERS VALUES ------------------------------------------------- On 01/03/17 11:33, Sobey, Richard A wrote: > Mmuserauth service list --data-access-method file -- Aidan Richmond Apple/Unix Support Officer, IT Garstang 10.137 Faculty of Biological Sciences University of Leeds Clarendon Way LS2 9JT Tel:0113 3434252 a.g.richmond at leeds.ac.uk From r.sobey at imperial.ac.uk Wed Mar 1 11:42:06 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Wed, 1 Mar 2017 11:42:06 +0000 Subject: [gpfsug-discuss] Issues getting SMB shares working. In-Reply-To: <901c6472-1c9c-3551-555e-22bc849fe2b1@leeds.ac.uk> References: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> <901c6472-1c9c-3551-555e-22bc849fe2b1@leeds.ac.uk> Message-ID: That's probably the first thing you need to sort out then. Check out the mmuserauth service create command. There was an example on this list on Monday so depending when you subscribed you may not have seen it. FYI the command cited was: mmuserauth service create --type ad --data-access-method file --servers 192.168.88.3 --user-name administrator --netbios-name scale --idmap-role master --password ********* --idmap-range-size 1000000 --idmap-range 10000000-299999999 --enable-nfs-kerberos --unixmap-domains 'sirius(10000-20000)' Change the parameters to cater to your environment and needs of course. Richard -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aidan Richmond Sent: 01 March 2017 11:36 To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Issues getting SMB shares working. Hello It returns the following: FILE access configuration : USERDEFINED PARAMETERS VALUES ------------------------------------------------- On 01/03/17 11:33, Sobey, Richard A wrote: > Mmuserauth service list --data-access-method file -- Aidan Richmond Apple/Unix Support Officer, IT Garstang 10.137 Faculty of Biological Sciences University of Leeds Clarendon Way LS2 9JT Tel:0113 3434252 a.g.richmond at leeds.ac.uk _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From a.g.richmond at leeds.ac.uk Wed Mar 1 12:07:28 2017 From: a.g.richmond at leeds.ac.uk (Aidan Richmond) Date: Wed, 1 Mar 2017 12:07:28 +0000 Subject: [gpfsug-discuss] Issues getting SMB shares working. In-Reply-To: References: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> <901c6472-1c9c-3551-555e-22bc849fe2b1@leeds.ac.uk> Message-ID: <2ac0911f-65f7-d500-1d01-7ba4b5929a79@leeds.ac.uk> Hello I'm a little hesitant to mess with the authentication as I we are wanting consistent UIDs across our systems and I know my predecessor struggled to get them consistent. Our AD environment stores the UID and GID settings in msSFU30uid and msSFU30gid. I'm also concerned that the system is already in use with nfsv4 access and don't want to break existing access unless I have to. On 01/03/17 11:42, Sobey, Richard A wrote: > That's probably the first thing you need to sort out then. > > Check out the mmuserauth service create command. > > There was an example on this list on Monday so depending when you subscribed you may not have seen it. FYI the command cited was: > > mmuserauth service create --type ad --data-access-method file --servers 192.168.88.3 --user-name administrator --netbios-name scale --idmap-role master --password ********* --idmap-range-size 1000000 --idmap-range 10000000-299999999 --enable-nfs-kerberos --unixmap-domains 'sirius(10000-20000)' > > Change the parameters to cater to your environment and needs of course. > > Richard > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aidan Richmond > Sent: 01 March 2017 11:36 > To: gpfsug-discuss at spectrumscale.org > Subject: Re: [gpfsug-discuss] Issues getting SMB shares working. > > Hello > > It returns the following: > > FILE access configuration : USERDEFINED > PARAMETERS VALUES > ------------------------------------------------- > > On 01/03/17 11:33, Sobey, Richard A wrote: >> Mmuserauth service list --data-access-method file > > -- Aidan Richmond Apple/Unix Support Officer, IT Garstang 10.137 Faculty of Biological Sciences University of Leeds Clarendon Way LS2 9JT Tel:0113 3434252 a.g.richmond at leeds.ac.uk From janfrode at tanso.net Wed Mar 1 14:12:04 2017 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Wed, 1 Mar 2017 15:12:04 +0100 Subject: [gpfsug-discuss] Issues getting SMB shares working. In-Reply-To: <2ac0911f-65f7-d500-1d01-7ba4b5929a79@leeds.ac.uk> References: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> <901c6472-1c9c-3551-555e-22bc849fe2b1@leeds.ac.uk> <2ac0911f-65f7-d500-1d01-7ba4b5929a79@leeds.ac.uk> Message-ID: Lets figure out how your NFS is authenticating then. The userdefined authentication you have, could mean that your linux host is configured to authenticated towards AD --- or it could be that you're using simple sys-authentication for NFS. Could you please post the output of: mmnfs export list mmnfs config list -jf On Wed, Mar 1, 2017 at 1:07 PM, Aidan Richmond wrote: > Hello > > I'm a little hesitant to mess with the authentication as I we are wanting > consistent UIDs across our systems and I know my predecessor struggled to > get them consistent. Our AD environment stores the UID and GID settings in > msSFU30uid and msSFU30gid. > > I'm also concerned that the system is already in use with nfsv4 access and > don't want to break existing access unless I have to. > > > > On 01/03/17 11:42, Sobey, Richard A wrote: > >> That's probably the first thing you need to sort out then. >> >> Check out the mmuserauth service create command. >> >> There was an example on this list on Monday so depending when you >> subscribed you may not have seen it. FYI the command cited was: >> >> mmuserauth service create --type ad --data-access-method file --servers >> 192.168.88.3 --user-name administrator --netbios-name scale --idmap-role >> master --password ********* --idmap-range-size 1000000 --idmap-range >> 10000000-299999999 --enable-nfs-kerberos --unixmap-domains >> 'sirius(10000-20000)' >> >> Change the parameters to cater to your environment and needs of course. >> >> Richard >> >> -----Original Message----- >> From: gpfsug-discuss-bounces at spectrumscale.org [mailto: >> gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aidan Richmond >> Sent: 01 March 2017 11:36 >> To: gpfsug-discuss at spectrumscale.org >> Subject: Re: [gpfsug-discuss] Issues getting SMB shares working. >> >> Hello >> >> It returns the following: >> >> FILE access configuration : USERDEFINED >> PARAMETERS VALUES >> ------------------------------------------------- >> >> On 01/03/17 11:33, Sobey, Richard A wrote: >> >>> Mmuserauth service list --data-access-method file >>> >> >> >> > > -- > Aidan Richmond > Apple/Unix Support Officer, IT > Garstang 10.137 > Faculty of Biological Sciences > University of Leeds > Clarendon Way > LS2 9JT > > Tel:0113 3434252 > a.g.richmond at leeds.ac.uk > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.g.richmond at leeds.ac.uk Wed Mar 1 15:22:36 2017 From: a.g.richmond at leeds.ac.uk (Aidan Richmond) Date: Wed, 1 Mar 2017 15:22:36 +0000 Subject: [gpfsug-discuss] Issues getting SMB shares working. In-Reply-To: References: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> <901c6472-1c9c-3551-555e-22bc849fe2b1@leeds.ac.uk> <2ac0911f-65f7-d500-1d01-7ba4b5929a79@leeds.ac.uk> Message-ID: mmnfs export list Path Delegations Clients ------------------------------------- /absl/SCRATCH none * /absl/SCRATCH none fbscpcu097 mmnfs config list NFS Ganesha Configuration: ========================== NFS_PROTOCOLS: 3,4 NFS_PORT: 2049 MNT_PORT: 0 NLM_PORT: 0 RQUOTA_PORT: 0 SHORT_FILE_HANDLE: FALSE LEASE_LIFETIME: 60 DOMAINNAME: LEEDS.AC.UK DELEGATIONS: Disabled ========================== STATD Configuration ========================== STATD_PORT: 0 ========================== CacheInode Configuration ========================== ENTRIES_HWMARK: 1500000 ========================== Export Defaults ========================== ACCESS_TYPE: NONE PROTOCOLS: 3,4 TRANSPORTS: TCP ANONYMOUS_UID: -2 ANONYMOUS_GID: -2 SECTYPE: SYS PRIVILEGEDPORT: FALSE MANAGE_GIDS: FALSE SQUASH: ROOT_SQUASH NFS_COMMIT: FALSE ========================== Log Configuration ========================== LOG_LEVEL: EVENT ========================== Idmapd Configuration ========================== DOMAIN: DS.LEEDS.AC.UK ========================== On 01/03/17 14:12, Jan-Frode Myklebust wrote: > Lets figure out how your NFS is authenticating then. The userdefined > authentication you have, could mean that your linux host is configured to > authenticated towards AD --- or it could be that you're using simple > sys-authentication for NFS. > > Could you please post the output of: > > mmnfs export list > mmnfs config list > > > -jf > > > On Wed, Mar 1, 2017 at 1:07 PM, Aidan Richmond > wrote: > >> Hello >> >> I'm a little hesitant to mess with the authentication as I we are wanting >> consistent UIDs across our systems and I know my predecessor struggled to >> get them consistent. Our AD environment stores the UID and GID settings in >> msSFU30uid and msSFU30gid. >> >> I'm also concerned that the system is already in use with nfsv4 access and >> don't want to break existing access unless I have to. >> >> >> >> On 01/03/17 11:42, Sobey, Richard A wrote: >> >>> That's probably the first thing you need to sort out then. >>> >>> Check out the mmuserauth service create command. >>> >>> There was an example on this list on Monday so depending when you >>> subscribed you may not have seen it. FYI the command cited was: >>> >>> mmuserauth service create --type ad --data-access-method file --servers >>> 192.168.88.3 --user-name administrator --netbios-name scale --idmap-role >>> master --password ********* --idmap-range-size 1000000 --idmap-range >>> 10000000-299999999 --enable-nfs-kerberos --unixmap-domains >>> 'sirius(10000-20000)' >>> >>> Change the parameters to cater to your environment and needs of course. >>> >>> Richard >>> >>> -----Original Message----- >>> From: gpfsug-discuss-bounces at spectrumscale.org [mailto: >>> gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aidan Richmond >>> Sent: 01 March 2017 11:36 >>> To: gpfsug-discuss at spectrumscale.org >>> Subject: Re: [gpfsug-discuss] Issues getting SMB shares working. >>> >>> Hello >>> >>> It returns the following: >>> >>> FILE access configuration : USERDEFINED >>> PARAMETERS VALUES >>> ------------------------------------------------- >>> >>> On 01/03/17 11:33, Sobey, Richard A wrote: >>> >>>> Mmuserauth service list --data-access-method file >>>> >>> >>> >>> >> >> -- >> Aidan Richmond >> Apple/Unix Support Officer, IT >> Garstang 10.137 >> Faculty of Biological Sciences >> University of Leeds >> Clarendon Way >> LS2 9JT >> >> Tel:0113 3434252 >> a.g.richmond at leeds.ac.uk >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aidan Richmond Apple/Unix Support Officer, IT Garstang 10.137 Faculty of Biological Sciences University of Leeds Clarendon Way LS2 9JT Tel:0113 3434252 a.g.richmond at leeds.ac.uk From Mark.Bush at siriuscom.com Wed Mar 1 19:28:24 2017 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Wed, 1 Mar 2017 19:28:24 +0000 Subject: [gpfsug-discuss] Issues getting SMB shares working. In-Reply-To: References: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> <901c6472-1c9c-3551-555e-22bc849fe2b1@leeds.ac.uk> <2ac0911f-65f7-d500-1d01-7ba4b5929a79@leeds.ac.uk> Message-ID: I?m pretty noob still here but I think what may be going on in your situation is that since you are just now enabling SMB service you need to figure out how you want users to authenticate. If you want to stick with the way things are currently authenticating on your system that is to say, USERDEFINED that really just means it?s delegated to the underlying system for authentication. So, you can change your authentication strategy to use Active Directory (like was mentioned by Richard), or use the local SMB authentication which means you?ll have to add users manually with the /usr/lpp/mmfs/bin/smbpasswd command. This works if you just want a few users to get access to SMB shares but it becomes troublesome when you get loads of users. You can use LDAP or AD to make things more manageable. Mark On 3/1/17, 9:22 AM, "Aidan Richmond" wrote: mmnfs export list Path Delegations Clients ------------------------------------- /absl/SCRATCH none * /absl/SCRATCH none fbscpcu097 mmnfs config list NFS Ganesha Configuration: ========================== NFS_PROTOCOLS: 3,4 NFS_PORT: 2049 MNT_PORT: 0 NLM_PORT: 0 RQUOTA_PORT: 0 SHORT_FILE_HANDLE: FALSE LEASE_LIFETIME: 60 DOMAINNAME: LEEDS.AC.UK DELEGATIONS: Disabled ========================== STATD Configuration ========================== STATD_PORT: 0 ========================== CacheInode Configuration ========================== ENTRIES_HWMARK: 1500000 ========================== Export Defaults ========================== ACCESS_TYPE: NONE PROTOCOLS: 3,4 TRANSPORTS: TCP ANONYMOUS_UID: -2 ANONYMOUS_GID: -2 SECTYPE: SYS PRIVILEGEDPORT: FALSE MANAGE_GIDS: FALSE SQUASH: ROOT_SQUASH NFS_COMMIT: FALSE ========================== Log Configuration ========================== LOG_LEVEL: EVENT ========================== Idmapd Configuration ========================== DOMAIN: DS.LEEDS.AC.UK ========================== On 01/03/17 14:12, Jan-Frode Myklebust wrote: > Lets figure out how your NFS is authenticating then. The userdefined > authentication you have, could mean that your linux host is configured to > authenticated towards AD --- or it could be that you're using simple > sys-authentication for NFS. > > Could you please post the output of: > > mmnfs export list > mmnfs config list > > > -jf > > > On Wed, Mar 1, 2017 at 1:07 PM, Aidan Richmond > wrote: > >> Hello >> >> I'm a little hesitant to mess with the authentication as I we are wanting >> consistent UIDs across our systems and I know my predecessor struggled to >> get them consistent. Our AD environment stores the UID and GID settings in >> msSFU30uid and msSFU30gid. >> >> I'm also concerned that the system is already in use with nfsv4 access and >> don't want to break existing access unless I have to. >> >> >> >> On 01/03/17 11:42, Sobey, Richard A wrote: >> >>> That's probably the first thing you need to sort out then. >>> >>> Check out the mmuserauth service create command. >>> >>> There was an example on this list on Monday so depending when you >>> subscribed you may not have seen it. FYI the command cited was: >>> >>> mmuserauth service create --type ad --data-access-method file --servers >>> 192.168.88.3 --user-name administrator --netbios-name scale --idmap-role >>> master --password ********* --idmap-range-size 1000000 --idmap-range >>> 10000000-299999999 --enable-nfs-kerberos --unixmap-domains >>> 'sirius(10000-20000)' >>> >>> Change the parameters to cater to your environment and needs of course. >>> >>> Richard >>> >>> -----Original Message----- >>> From: gpfsug-discuss-bounces at spectrumscale.org [mailto: >>> gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aidan Richmond >>> Sent: 01 March 2017 11:36 >>> To: gpfsug-discuss at spectrumscale.org >>> Subject: Re: [gpfsug-discuss] Issues getting SMB shares working. >>> >>> Hello >>> >>> It returns the following: >>> >>> FILE access configuration : USERDEFINED >>> PARAMETERS VALUES >>> ------------------------------------------------- >>> >>> On 01/03/17 11:33, Sobey, Richard A wrote: >>> >>>> Mmuserauth service list --data-access-method file >>>> >>> >>> >>> >> >> -- >> Aidan Richmond >> Apple/Unix Support Officer, IT >> Garstang 10.137 >> Faculty of Biological Sciences >> University of Leeds >> Clarendon Way >> LS2 9JT >> >> Tel:0113 3434252 >> a.g.richmond at leeds.ac.uk >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aidan Richmond Apple/Unix Support Officer, IT Garstang 10.137 Faculty of Biological Sciences University of Leeds Clarendon Way LS2 9JT Tel:0113 3434252 a.g.richmond at leeds.ac.uk _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions From janfrode at tanso.net Wed Mar 1 20:21:03 2017 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Wed, 01 Mar 2017 20:21:03 +0000 Subject: [gpfsug-discuss] Issues getting SMB shares working. In-Reply-To: References: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> <901c6472-1c9c-3551-555e-22bc849fe2b1@leeds.ac.uk> <2ac0911f-65f7-d500-1d01-7ba4b5929a79@leeds.ac.uk> Message-ID: This looks to me like a quite plain SYS authorized NFS, maybe also verify that the individual NFS shares has sec=sys with "mmnfs export list -n /absl/SCRATCH". If you check "man mmuserauth" there are quite clear examples for how to connect it to AD populated with unix id and gid. I don't think this will affect your NFS service, since there doesn't seem to be any kerberos involved. But, please beware that mmuserauth will overwrite any customized sssd.conf, krb5.conf, winbind, etc.. As it configures the authentication for the whole host, not just samba/nfs-services. -jf ons. 1. mar. 2017 kl. 16.22 skrev Aidan Richmond : > mmnfs export list > Path Delegations Clients > ------------------------------------- > /absl/SCRATCH none * > /absl/SCRATCH none fbscpcu097 > > mmnfs config list > > NFS Ganesha Configuration: > ========================== > NFS_PROTOCOLS: 3,4 > NFS_PORT: 2049 > MNT_PORT: 0 > NLM_PORT: 0 > RQUOTA_PORT: 0 > SHORT_FILE_HANDLE: FALSE > LEASE_LIFETIME: 60 > DOMAINNAME: LEEDS.AC.UK > DELEGATIONS: Disabled > ========================== > > STATD Configuration > ========================== > STATD_PORT: 0 > ========================== > > CacheInode Configuration > ========================== > ENTRIES_HWMARK: 1500000 > ========================== > > Export Defaults > ========================== > ACCESS_TYPE: NONE > PROTOCOLS: 3,4 > TRANSPORTS: TCP > ANONYMOUS_UID: -2 > ANONYMOUS_GID: -2 > SECTYPE: SYS > PRIVILEGEDPORT: FALSE > MANAGE_GIDS: FALSE > SQUASH: ROOT_SQUASH > NFS_COMMIT: FALSE > ========================== > > Log Configuration > ========================== > LOG_LEVEL: EVENT > ========================== > > Idmapd Configuration > ========================== > DOMAIN: DS.LEEDS.AC.UK > ========================== > > On 01/03/17 14:12, Jan-Frode Myklebust wrote: > > Lets figure out how your NFS is authenticating then. The userdefined > > authentication you have, could mean that your linux host is configured to > > authenticated towards AD --- or it could be that you're using simple > > sys-authentication for NFS. > > > > Could you please post the output of: > > > > mmnfs export list > > mmnfs config list > > > > > > -jf > > > > > > On Wed, Mar 1, 2017 at 1:07 PM, Aidan Richmond > > > wrote: > > > >> Hello > >> > >> I'm a little hesitant to mess with the authentication as I we are > wanting > >> consistent UIDs across our systems and I know my predecessor struggled > to > >> get them consistent. Our AD environment stores the UID and GID settings > in > >> msSFU30uid and msSFU30gid. > >> > >> I'm also concerned that the system is already in use with nfsv4 access > and > >> don't want to break existing access unless I have to. > >> > >> > >> > >> On 01/03/17 11:42, Sobey, Richard A wrote: > >> > >>> That's probably the first thing you need to sort out then. > >>> > >>> Check out the mmuserauth service create command. > >>> > >>> There was an example on this list on Monday so depending when you > >>> subscribed you may not have seen it. FYI the command cited was: > >>> > >>> mmuserauth service create --type ad --data-access-method file --servers > >>> 192.168.88.3 --user-name administrator --netbios-name scale > --idmap-role > >>> master --password ********* --idmap-range-size 1000000 --idmap-range > >>> 10000000-299999999 --enable-nfs-kerberos --unixmap-domains > >>> 'sirius(10000-20000)' > >>> > >>> Change the parameters to cater to your environment and needs of course. > >>> > >>> Richard > >>> > >>> -----Original Message----- > >>> From: gpfsug-discuss-bounces at spectrumscale.org [mailto: > >>> gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aidan Richmond > >>> Sent: 01 March 2017 11:36 > >>> To: gpfsug-discuss at spectrumscale.org > >>> Subject: Re: [gpfsug-discuss] Issues getting SMB shares working. > >>> > >>> Hello > >>> > >>> It returns the following: > >>> > >>> FILE access configuration : USERDEFINED > >>> PARAMETERS VALUES > >>> ------------------------------------------------- > >>> > >>> On 01/03/17 11:33, Sobey, Richard A wrote: > >>> > >>>> Mmuserauth service list --data-access-method file > >>>> > >>> > >>> > >>> > >> > >> -- > >> Aidan Richmond > >> Apple/Unix Support Officer, IT > >> Garstang 10.137 > >> Faculty of Biological Sciences > >> University of Leeds > >> Clarendon Way > >> LS2 9JT > >> > >> Tel:0113 3434252 > >> a.g.richmond at leeds.ac.uk > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > -- > Aidan Richmond > Apple/Unix Support Officer, IT > Garstang 10.137 > Faculty of Biological Sciences > University of Leeds > Clarendon Way > LS2 9JT > > Tel:0113 3434252 > a.g.richmond at leeds.ac.uk > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Thu Mar 2 14:17:45 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Thu, 2 Mar 2017 14:17:45 +0000 Subject: [gpfsug-discuss] GPFSUG-USA Meeting at NERSC April 4-5th - Registration Reminder and Agenda Details Message-ID: Here?s a reminder to register for the upcoming user group meeting! Registration deadline is March 24th, only 3 weeks away. You won?t want to miss the 2 hour session on Tuesday on problem determination by Yuri Volobuev. Any long time GPFS admin has no doubt interacted with Yuri. He?s moved on to new challenges from GPFS, but has a wealth of problem determination tips that he wants to pass along. We?re happy to have him back in the GPFS community. We also have a few slots (20 mins each) for customer and partner talks ? please contact me if you want to do a short presentation. Register here: https://www.nersc.gov/research-and-development/data-analytics/spectrum-user-group-meeting/ Tuesday, April 4th 09:00-10:00 Registration/Coffee/Networking 10:00-10:30 Welcome - Bob Oesterlin, Kristy Kallback, Doris Conti 10:30-11:30 Spectrum Scale & ESS Update - Scott Fadden / Puneet Chaudhary 11:30-12:30 Problem Avoidance - Best Practices in the Field - IBM Support 12:30-13:30 Lunch and Networking 13:30-15:30 Problem Determination - Deep Dive - Yuri Volobuev 15:30-16:00 Break/Networking 16:00-17:00 Problem Determination - What's New? - Matthias Dietz 17:00-18:00 Bring your favorite problem or performance issue. We will fix it now! Wednesday April 5th 08:30-09:00 Coffee and Networking 09:00-10:00 Transparent Cloud Tiering - Introduction and Use Cases (Meet the Devs) - Nikhil Khandelwal or Rob Basham Life Sciences and Next Generation Sequencing with Spectrum Scale (Solutions) 10:00-11:00 AFM - Introduction and Use Cases (Meet the Devs) - Scott Fadden Spectrum Protect on Spectrum Scale (Solutions) - Jason Basler 11:00-12:00 Meet the Devs - IBM Development Hadoop Integration and Support for HortonWorks HDP (Solutions) - Ted Hoover 12:00-13:00 Lunch and Networking 13:00-13:20 Customer/Partner Talk 13:20-13:40 Customer/Partner Talk 13:40-14:00 Customer/Partner Talk 14:00-14:30 Break 14:30-15:00 Challenges in Tracing - Vasily Tarasov 15:00-15:30 Application Integration with Containers - Dean Hildebrand 15:30-16:00 Wrap up - Bob Oesterlin, Kristy Kallback, Doris Conti Bob Oesterlin Sr Principal Storage Engineer, Nuance Spectrum Scale UG-USA Co-Principal -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.g.richmond at leeds.ac.uk Thu Mar 2 15:41:40 2017 From: a.g.richmond at leeds.ac.uk (Aidan Richmond) Date: Thu, 2 Mar 2017 15:41:40 +0000 Subject: [gpfsug-discuss] Issues getting SMB shares working. In-Reply-To: References: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> <901c6472-1c9c-3551-555e-22bc849fe2b1@leeds.ac.uk> <2ac0911f-65f7-d500-1d01-7ba4b5929a79@leeds.ac.uk> Message-ID: <97b56c55-58d4-3400-f8cf-5f6745583fc9@leeds.ac.uk> mmnfs export list -n /absl/SCRATCH Path Delegations Clients Access_Type Protocols Transports Squash Anonymous_uid Anonymous_gid SecType PrivilegedPort DefaultDelegations Manage_Gids NFS_Commit ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- /absl/SCRATCH none * RW 4 TCP ROOT_SQUASH -2 -2 KRB5I FALSE none FALSE FALSE /absl/SCRATCH none fbscpcu097 RW 3 TCP ROOT_SQUASH -2 -2 sys FALSE none FALSE FALSE Ignore the fbscpcu097 entry, I think my departed colleague was just using that for testing, the NFS clients all access it though nfsv4 which looks to be using kerberos from this. On 01/03/17 20:21, Jan-Frode Myklebust wrote: > This looks to me like a quite plain SYS authorized NFS, maybe also verify > that the individual NFS shares has sec=sys with "mmnfs export list -n > /absl/SCRATCH". > > > If you check "man mmuserauth" there are quite clear examples for how to > connect it to AD populated with unix id and gid. I don't think this will > affect your NFS service, since there doesn't seem to be any kerberos > involved. > > But, please beware that mmuserauth will overwrite any customized sssd.conf, > krb5.conf, winbind, etc.. As it configures the authentication for the whole > host, not just samba/nfs-services. > > > -jf > > ons. 1. mar. 2017 kl. 16.22 skrev Aidan Richmond : > >> mmnfs export list >> Path Delegations Clients >> ------------------------------------- >> /absl/SCRATCH none * >> /absl/SCRATCH none fbscpcu097 >> >> mmnfs config list >> >> NFS Ganesha Configuration: >> ========================== >> NFS_PROTOCOLS: 3,4 >> NFS_PORT: 2049 >> MNT_PORT: 0 >> NLM_PORT: 0 >> RQUOTA_PORT: 0 >> SHORT_FILE_HANDLE: FALSE >> LEASE_LIFETIME: 60 >> DOMAINNAME: LEEDS.AC.UK >> DELEGATIONS: Disabled >> ========================== >> >> STATD Configuration >> ========================== >> STATD_PORT: 0 >> ========================== >> >> CacheInode Configuration >> ========================== >> ENTRIES_HWMARK: 1500000 >> ========================== >> >> Export Defaults >> ========================== >> ACCESS_TYPE: NONE >> PROTOCOLS: 3,4 >> TRANSPORTS: TCP >> ANONYMOUS_UID: -2 >> ANONYMOUS_GID: -2 >> SECTYPE: SYS >> PRIVILEGEDPORT: FALSE >> MANAGE_GIDS: FALSE >> SQUASH: ROOT_SQUASH >> NFS_COMMIT: FALSE >> ========================== >> >> Log Configuration >> ========================== >> LOG_LEVEL: EVENT >> ========================== >> >> Idmapd Configuration >> ========================== >> DOMAIN: DS.LEEDS.AC.UK >> ========================== >> -- Aidan Richmond Apple/Unix Support Officer, IT Garstang 10.137 Faculty of Biological Sciences University of Leeds Clarendon Way LS2 9JT Tel:0113 3434252 a.g.richmond at leeds.ac.uk From p.childs at qmul.ac.uk Thu Mar 2 16:34:09 2017 From: p.childs at qmul.ac.uk (Peter Childs) Date: Thu, 2 Mar 2017 16:34:09 +0000 Subject: [gpfsug-discuss] mmbackup logging issue In-Reply-To: References: Message-ID: We had that issue. we had to export MMBACKUP_PROGRESS_CONTENT=5 export MMBACKUP_PROGRESS_INTERVAL=300 before we run it to get it back. Lets just say IBM changed the behaviour, We ended up opening a PRM to get that answer ;) We also set -L 1 you can change how often the messages are displayed by changing MMBACKUP_PROGRESS_INTERVAL flexable but the default is different;) Peter Childs ITS Research Infrastructure Queen Mary, University of London ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Ashish Thandavan Sent: Tuesday, February 28, 2017 4:10:44 PM To: gpfsug main discussion list Subject: [gpfsug-discuss] mmbackup logging issue Dear all, We have a small GPFS cluster and a separate server running TSM and one of the three NSD servers backs up our GPFS filesystem to the TSM server using mmbackup. After a recent upgrade from v3.5 to 4.1.1, we've noticed that mmbackup no longer logs stuff like it used to : ... Thu Jan 19 05:45:41 2017 mmbackup:Backing up files: 0 backed up, 870532 expired, 2 failed. Thu Jan 19 06:15:41 2017 mmbackup:Backing up files: 0 backed up, 870532 expired, 3 failed. Thu Jan 19 06:45:41 2017 mmbackup:Backing up files: 0 backed up, 870532 expired, 3 failed. ... instead of ... Sat Dec 3 12:01:00 2016 mmbackup:Backing up files: 105030 backed up, 635456 expired, 30 failed. Sat Dec 3 12:31:00 2016 mmbackup:Backing up files: 205934 backed up, 635456 expired, 57 failed. Sat Dec 3 13:01:00 2016 mmbackup:Backing up files: 321702 backed up, 635456 expired, 169 failed. ... like it used to pre-upgrade. I am therefore unable to see how far long it has got, and indeed if it completed successfully, as this is what it logs at the end of a job : ... Tue Jan 17 18:07:31 2017 mmbackup:Completed policy backup run with 0 policy errors, 10012 files failed, 0 severe errors, returning rc=9. Tue Jan 17 18:07:31 2017 mmbackup:Policy for backup returned 9 Highest TSM error 12 mmbackup: TSM Summary Information: Total number of objects inspected: 20617273 Total number of objects backed up: 0 Total number of objects updated: 0 Total number of objects rebound: 0 Total number of objects deleted: 0 Total number of objects expired: 1 Total number of objects failed: 10012 Total number of objects encrypted: 0 Total number of bytes inspected: 3821624716861 Total number of bytes transferred: 3712040943672 Tue Jan 17 18:07:31 2017 mmbackup:Audit files /cs/mmbackup.audit.gpfs* contain 0 failed paths but there were 10012 failures. Cannot reconcile shadow database. Unable to compensate for all TSM errors in new shadow database. Preserving previous shadow database. Run next mmbackup with -q to synchronize shadow database. exit 12 If it helps, the mmbackup job is kicked off with the following options : /usr/lpp/mmfs/bin/mmbackup gpfs -n 8 -t full -B 20000 -L 1 --tsm-servers gpfs_weekly_stanza -N glossop1a | /usr/bin/tee /var/log/mmbackup/gpfs_weekly/backup_log.`date +%Y%m%d_%H_%M` (The excerpts above are from the backup_log. file.) Our NSD servers are running GPFS 4.1.1-11, TSM is at 7.1.1.100 and the File system version is 12.06 (3.4.0.3). Has anyone else seen this behaviour with mmbackup and if so, found a fix? Thanks, Regards, Ash -- ------------------------- Ashish Thandavan UNIX Support Computing Officer Department of Computer Science University of Oxford Wolfson Building Parks Road Oxford OX1 3QD Phone: 01865 610733 Email: ashish.thandavan at cs.ox.ac.uk _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From ashish.thandavan at cs.ox.ac.uk Thu Mar 2 16:50:23 2017 From: ashish.thandavan at cs.ox.ac.uk (Ashish Thandavan) Date: Thu, 2 Mar 2017 16:50:23 +0000 Subject: [gpfsug-discuss] mmbackup logging issue In-Reply-To: References: Message-ID: <6cd84459-ecc3-1ef9-da22-dea843cff154@cs.ox.ac.uk> Dear Peter, On 02/03/17 16:34, Peter Childs wrote: > We had that issue. > > we had to > > export MMBACKUP_PROGRESS_CONTENT=5 > export MMBACKUP_PROGRESS_INTERVAL=300 > > before we run it to get it back. > > Lets just say IBM changed the behaviour, We ended up opening a PRM to get that answer ;) We also set -L 1 > > you can change how often the messages are displayed by changing MMBACKUP_PROGRESS_INTERVAL flexable but the default is different;) > I'll set those variables before kicking off the next mmbackup and hope that fixes it. Thank you!! Regards, Ash -- ------------------------- Ashish Thandavan UNIX Support Computing Officer Department of Computer Science University of Oxford Wolfson Building Parks Road Oxford OX1 3QD Phone: 01865 610733 Email: ashish.thandavan at cs.ox.ac.uk From janfrode at tanso.net Thu Mar 2 20:42:23 2017 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Thu, 02 Mar 2017 20:42:23 +0000 Subject: [gpfsug-discuss] Issues getting SMB shares working. In-Reply-To: <97b56c55-58d4-3400-f8cf-5f6745583fc9@leeds.ac.uk> References: <7f1b340d-93aa-01ab-066b-2442f443a893@leeds.ac.uk> <901c6472-1c9c-3551-555e-22bc849fe2b1@leeds.ac.uk> <2ac0911f-65f7-d500-1d01-7ba4b5929a79@leeds.ac.uk> <97b56c55-58d4-3400-f8cf-5f6745583fc9@leeds.ac.uk> Message-ID: Ouch, yes.. Then the switch to mmuserauth is more difficult. I would recommend setting up a lab cluster (only need a single node), and use mmuserauth to connect it to AD and see that you get both kerberized NFS and SMB working by default there, before doing the same on your production cluster. -jf tor. 2. mar. 2017 kl. 16.42 skrev Aidan Richmond : > mmnfs export list -n /absl/SCRATCH > Path Delegations Clients Access_Type Protocols Transports > Squash Anonymous_uid Anonymous_gid SecType PrivilegedPort > DefaultDelegations Manage_Gids NFS_Commit > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > /absl/SCRATCH none * RW 4 TCP > ROOT_SQUASH -2 -2 KRB5I FALSE none > FALSE FALSE > /absl/SCRATCH none fbscpcu097 RW 3 TCP > ROOT_SQUASH -2 -2 sys FALSE none > FALSE FALSE > > > Ignore the fbscpcu097 entry, I think my departed colleague was just > using that for testing, the NFS clients all access it though nfsv4 which > looks to be using kerberos from this. > > On 01/03/17 20:21, Jan-Frode Myklebust wrote: > > This looks to me like a quite plain SYS authorized NFS, maybe also verify > > that the individual NFS shares has sec=sys with "mmnfs export list -n > > /absl/SCRATCH". > > > > > > If you check "man mmuserauth" there are quite clear examples for how to > > connect it to AD populated with unix id and gid. I don't think this will > > affect your NFS service, since there doesn't seem to be any kerberos > > involved. > > > > But, please beware that mmuserauth will overwrite any customized > sssd.conf, > > krb5.conf, winbind, etc.. As it configures the authentication for the > whole > > host, not just samba/nfs-services. > > > > > > -jf > > > > ons. 1. mar. 2017 kl. 16.22 skrev Aidan Richmond < > a.g.richmond at leeds.ac.uk>: > > > >> mmnfs export list > >> Path Delegations Clients > >> ------------------------------------- > >> /absl/SCRATCH none * > >> /absl/SCRATCH none fbscpcu097 > >> > >> mmnfs config list > >> > >> NFS Ganesha Configuration: > >> ========================== > >> NFS_PROTOCOLS: 3,4 > >> NFS_PORT: 2049 > >> MNT_PORT: 0 > >> NLM_PORT: 0 > >> RQUOTA_PORT: 0 > >> SHORT_FILE_HANDLE: FALSE > >> LEASE_LIFETIME: 60 > >> DOMAINNAME: LEEDS.AC.UK > >> DELEGATIONS: Disabled > >> ========================== > >> > >> STATD Configuration > >> ========================== > >> STATD_PORT: 0 > >> ========================== > >> > >> CacheInode Configuration > >> ========================== > >> ENTRIES_HWMARK: 1500000 > >> ========================== > >> > >> Export Defaults > >> ========================== > >> ACCESS_TYPE: NONE > >> PROTOCOLS: 3,4 > >> TRANSPORTS: TCP > >> ANONYMOUS_UID: -2 > >> ANONYMOUS_GID: -2 > >> SECTYPE: SYS > >> PRIVILEGEDPORT: FALSE > >> MANAGE_GIDS: FALSE > >> SQUASH: ROOT_SQUASH > >> NFS_COMMIT: FALSE > >> ========================== > >> > >> Log Configuration > >> ========================== > >> LOG_LEVEL: EVENT > >> ========================== > >> > >> Idmapd Configuration > >> ========================== > >> DOMAIN: DS.LEEDS.AC.UK > >> ========================== > >> > > -- > Aidan Richmond > Apple/Unix Support Officer, IT > Garstang 10.137 > Faculty of Biological Sciences > University of Leeds > Clarendon Way > LS2 9JT > > Tel:0113 3434252 > a.g.richmond at leeds.ac.uk > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Fri Mar 3 09:20:24 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Fri, 3 Mar 2017 09:20:24 +0000 Subject: [gpfsug-discuss] mmbackup logging issue In-Reply-To: <6cd84459-ecc3-1ef9-da22-dea843cff154@cs.ox.ac.uk> References: <6cd84459-ecc3-1ef9-da22-dea843cff154@cs.ox.ac.uk> Message-ID: HI all We have the same problem (less of a problem, more lack of visibilitiy). Can I just add those lines to the top of our mmbackup.sh script? -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Ashish Thandavan Sent: 02 March 2017 16:50 To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] mmbackup logging issue Dear Peter, On 02/03/17 16:34, Peter Childs wrote: > We had that issue. > > we had to > > export MMBACKUP_PROGRESS_CONTENT=5 > export MMBACKUP_PROGRESS_INTERVAL=300 > > before we run it to get it back. > > Lets just say IBM changed the behaviour, We ended up opening a PRM to > get that answer ;) We also set -L 1 > > you can change how often the messages are displayed by changing > MMBACKUP_PROGRESS_INTERVAL flexable but the default is different;) > I'll set those variables before kicking off the next mmbackup and hope that fixes it. Thank you!! Regards, Ash -- ------------------------- Ashish Thandavan UNIX Support Computing Officer Department of Computer Science University of Oxford Wolfson Building Parks Road Oxford OX1 3QD Phone: 01865 610733 Email: ashish.thandavan at cs.ox.ac.uk _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From p.childs at qmul.ac.uk Fri Mar 3 10:44:03 2017 From: p.childs at qmul.ac.uk (Peter Childs) Date: Fri, 3 Mar 2017 10:44:03 +0000 Subject: [gpfsug-discuss] mmbackup logging issue In-Reply-To: References: <6cd84459-ecc3-1ef9-da22-dea843cff154@cs.ox.ac.uk>, Message-ID: That's basically what we did, They are only environment variables, so if you not using bash to call mmbackup you will need to change the lines accordingly. What they do is in the manual the issue is the default changed between versions..... Peter Childs ITS Research Infrastructure Queen Mary, University of London ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Sobey, Richard A Sent: Friday, March 3, 2017 9:20:24 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] mmbackup logging issue HI all We have the same problem (less of a problem, more lack of visibilitiy). Can I just add those lines to the top of our mmbackup.sh script? -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Ashish Thandavan Sent: 02 March 2017 16:50 To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] mmbackup logging issue Dear Peter, On 02/03/17 16:34, Peter Childs wrote: > We had that issue. > > we had to > > export MMBACKUP_PROGRESS_CONTENT=5 > export MMBACKUP_PROGRESS_INTERVAL=300 > > before we run it to get it back. > > Lets just say IBM changed the behaviour, We ended up opening a PRM to > get that answer ;) We also set -L 1 > > you can change how often the messages are displayed by changing > MMBACKUP_PROGRESS_INTERVAL flexable but the default is different;) > I'll set those variables before kicking off the next mmbackup and hope that fixes it. Thank you!! Regards, Ash -- ------------------------- Ashish Thandavan UNIX Support Computing Officer Department of Computer Science University of Oxford Wolfson Building Parks Road Oxford OX1 3QD Phone: 01865 610733 Email: ashish.thandavan at cs.ox.ac.uk _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From jtucker at pixitmedia.com Fri Mar 3 12:59:48 2017 From: jtucker at pixitmedia.com (Jez Tucker) Date: Fri, 3 Mar 2017 12:59:48 +0000 Subject: [gpfsug-discuss] mmbackup logging issue In-Reply-To: References: Message-ID: <7ca99092-63e6-c1a1-d9a7-e1907bcef7c8@pixitmedia.com> Hi Ash, This is the primary reason for using snapshots for mmbackup ( -S snapname ) and making sure that only mmbackup sends data to backup rather than an oob dsmc: Tue Jan 17 18:07:31 2017 mmbackup:Audit files /cs/mmbackup.audit.gpfs* contain 0 failed paths but there were 10012 failures. Cannot reconcile shadow database. Unable to compensate for all TSM errors in new shadow database. Preserving previous shadow database. Run next mmbackup with -q to synchronize shadow database. exit 12 <------------ ick! Other obvious causes are very, very, odd filenames / paths. Jez On 28/02/17 16:10, Ashish Thandavan wrote: > Dear all, > > We have a small GPFS cluster and a separate server running TSM and one > of the three NSD servers backs up our GPFS filesystem to the TSM > server using mmbackup. After a recent upgrade from v3.5 to 4.1.1, > we've noticed that mmbackup no longer logs stuff like it used to : > > ... > Thu Jan 19 05:45:41 2017 mmbackup:Backing up files: 0 backed up, > 870532 expired, 2 failed. > Thu Jan 19 06:15:41 2017 mmbackup:Backing up files: 0 backed up, > 870532 expired, 3 failed. > Thu Jan 19 06:45:41 2017 mmbackup:Backing up files: 0 backed up, > 870532 expired, 3 failed. > ... > > > instead of > > ... > Sat Dec 3 12:01:00 2016 mmbackup:Backing up files: 105030 backed up, > 635456 expired, 30 failed. > Sat Dec 3 12:31:00 2016 mmbackup:Backing up files: 205934 backed up, > 635456 expired, 57 failed. > Sat Dec 3 13:01:00 2016 mmbackup:Backing up files: 321702 backed up, > 635456 expired, 169 failed. > ... > > like it used to pre-upgrade. > > I am therefore unable to see how far long it has got, and indeed if it > completed successfully, as this is what it logs at the end of a job : > > ... > Tue Jan 17 18:07:31 2017 mmbackup:Completed policy backup run with 0 > policy errors, 10012 files failed, 0 severe errors, returning rc=9. > Tue Jan 17 18:07:31 2017 mmbackup:Policy for backup returned 9 Highest > TSM error 12 > mmbackup: TSM Summary Information: > Total number of objects inspected: 20617273 > Total number of objects backed up: 0 > Total number of objects updated: 0 > Total number of objects rebound: 0 > Total number of objects deleted: 0 > Total number of objects expired: 1 > Total number of objects failed: 10012 > Total number of objects encrypted: 0 > Total number of bytes inspected: 3821624716861 > Total number of bytes transferred: 3712040943672 > Tue Jan 17 18:07:31 2017 mmbackup:Audit files /cs/mmbackup.audit.gpfs* > contain 0 failed paths but there were 10012 failures. > Cannot reconcile shadow database. > Unable to compensate for all TSM errors in new shadow database. > Preserving previous shadow database. > Run next mmbackup with -q to synchronize shadow database. exit 12 > > If it helps, the mmbackup job is kicked off with the following options : > /usr/lpp/mmfs/bin/mmbackup gpfs -n 8 -t full -B 20000 -L 1 > --tsm-servers gpfs_weekly_stanza -N glossop1a | /usr/bin/tee > /var/log/mmbackup/gpfs_weekly/backup_log.`date +%Y%m%d_%H_%M` > > (The excerpts above are from the backup_log. file.) > > Our NSD servers are running GPFS 4.1.1-11, TSM is at 7.1.1.100 and the > File system version is 12.06 (3.4.0.3). Has anyone else seen this > behaviour with mmbackup and if so, found a fix? > > Thanks, > > Regards, > Ash > -- *Jez Tucker* Head of Research and Development, Pixit Media 07764193820 | jtucker at pixitmedia.com www.pixitmedia.com | Tw:@pixitmedia.com -- This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ashish.thandavan at cs.ox.ac.uk Fri Mar 3 13:05:56 2017 From: ashish.thandavan at cs.ox.ac.uk (Ashish Thandavan) Date: Fri, 3 Mar 2017 13:05:56 +0000 Subject: [gpfsug-discuss] mmbackup logging issue In-Reply-To: <7ca99092-63e6-c1a1-d9a7-e1907bcef7c8@pixitmedia.com> References: <7ca99092-63e6-c1a1-d9a7-e1907bcef7c8@pixitmedia.com> Message-ID: <5882c991-4cb0-4658-e25b-b2a1bbc26a48@cs.ox.ac.uk> Hi Jez, > > This is the primary reason for using snapshots for mmbackup ( -S > snapname ) and making sure that only mmbackup sends data to backup > rather than an oob dsmc: > > Tue Jan 17 18:07:31 2017 mmbackup:Audit files /cs/mmbackup.audit.gpfs* > contain 0 failed paths but there were 10012 failures. > Cannot reconcile shadow database. > Unable to compensate for all TSM errors in new shadow database. > Preserving previous shadow database. > Run next mmbackup with -q to synchronize shadow database. exit 12 > <------------ ick! > I was going to send a separate email about this, but as you mention it -- I was wondering if the TSM server requires the shadow files to also be backed up? And that reconciliation of the shadow database will fail if they are not? (Unless of course, you mean above that the shadow database failure is purely on account of the 10012 failures...) Re the use of snapshots for mmbackup, are you saying dsmc does not get involved in sending the files to the TSM server? I haven't used snapshots for mmbackup, but will certainly try! Regards, Ash -- ------------------------- Ashish Thandavan UNIX Support Computing Officer Department of Computer Science University of Oxford Wolfson Building Parks Road Oxford OX1 3QD Phone: 01865 610733 Email: ashish.thandavan at cs.ox.ac.uk From jez at rib-it.org Fri Mar 3 13:19:24 2017 From: jez at rib-it.org (Jez Tucker) Date: Fri, 3 Mar 2017 13:19:24 +0000 Subject: [gpfsug-discuss] mmbackup logging issue In-Reply-To: <5882c991-4cb0-4658-e25b-b2a1bbc26a48@cs.ox.ac.uk> References: <7ca99092-63e6-c1a1-d9a7-e1907bcef7c8@pixitmedia.com> <5882c991-4cb0-4658-e25b-b2a1bbc26a48@cs.ox.ac.uk> Message-ID: <754ae5a7-82e0-17a1-2ab6-b1538bcf4e73@rib-it.org> Comments inline... On 03/03/17 13:05, Ashish Thandavan wrote: > Hi Jez, > >> >> This is the primary reason for using snapshots for mmbackup ( -S >> snapname ) and making sure that only mmbackup sends data to backup >> rather than an oob dsmc: >> >> Tue Jan 17 18:07:31 2017 mmbackup:Audit files >> /cs/mmbackup.audit.gpfs* contain 0 failed paths but there were 10012 >> failures. >> Cannot reconcile shadow database. >> Unable to compensate for all TSM errors in new shadow database. >> Preserving previous shadow database. >> Run next mmbackup with -q to synchronize shadow database. exit 12 >> <------------ ick! >> > > I was going to send a separate email about this, but as you mention it > -- I was wondering if the TSM server requires the shadow files to also > be backed up? And that reconciliation of the shadow database will fail > if they are not? No, that's not a requirement. > (Unless of course, you mean above that the shadow database failure is > purely on account of the 10012 failures...) More than likely. Check the dsmerror.log and the mmbackup logs. There are also other possibilities which could exhibit the similar cases. > > Re the use of snapshots for mmbackup, are you saying dsmc does not get > involved in sending the files to the TSM server? I haven't used > snapshots for mmbackup, but will certainly try! Snapshots provide a consistent dataset to backup. I.E. no errors from 'trying to backup a file which has been deleted since I first knew about it in the scan phase'. But as per previous related thread 'Tracking deleted files', YMMV depending on your workload and environment. > > Regards, > Ash > From chair at spectrumscale.org Fri Mar 3 15:23:00 2017 From: chair at spectrumscale.org (GPFS UG Chair (Simon Thompson)) Date: Fri, 03 Mar 2017 15:23:00 +0000 Subject: [gpfsug-discuss] Save the Date 9th/10th May - UK/European/Global (?) Spectrum Scale user group meeting Message-ID: Hi All, Save the date for the next UK Spectrum Scale user group meeting on 9th/10th May 2017. As last year, this will be a 2 day meeting and having out-grown IBM South Bank client centre who have hosted us over the past few years, we're heading off to Manchester to a bigger venue. Further details on registration and agenda will be available in the next few weeks. Thanks to IBM for supporting the event funding the venue, we are working up our sponsorship package for other companies for an evening function. This year we are able to take up to 200 delegates and we're expecting that it will be expanded out to other European customers looking for an English language event, and we're flattered that Doug listed us a the Global event :-D Thanks Simon (Group Chair) From ewahl at osc.edu Fri Mar 3 18:50:55 2017 From: ewahl at osc.edu (Edward Wahl) Date: Fri, 3 Mar 2017 13:50:55 -0500 Subject: [gpfsug-discuss] mmbackup logging issue In-Reply-To: <7ca99092-63e6-c1a1-d9a7-e1907bcef7c8@pixitmedia.com> References: <7ca99092-63e6-c1a1-d9a7-e1907bcef7c8@pixitmedia.com> Message-ID: <20170303135055.7f046983@osc.edu> Comments inline On Fri, 3 Mar 2017 12:59:48 +0000 Jez Tucker wrote: > Hi Ash, > > This is the primary reason for using snapshots for mmbackup ( -S > snapname ) and making sure that only mmbackup sends data to backup > rather than an oob dsmc: > > Tue Jan 17 18:07:31 2017 mmbackup:Audit files /cs/mmbackup.audit.gpfs* > contain 0 failed paths but there were 10012 failures. > Cannot reconcile shadow database. > Unable to compensate for all TSM errors in new shadow database. > Preserving previous shadow database. > Run next mmbackup with -q to synchronize shadow database. exit 12 > <------------ ick! > > Other obvious causes are very, very, odd filenames / paths. Yes, GPFS allows much more lenient filenames than TSM does. You can attempt to cut this back a bit with 'WILDCARDSARELITERAL yes' and 'QUOTESARELITERAL yes' in your dsm.opt (and I recommend SKIPACLUPDATECHECK yes and UPDATECTIME yes ) But this still won't catch everything. We still tend to find foreign character sets, control sequences that look like people trying to exit emacs, etc. in file names. Valid for the Filesystem, not so much for TSM. Ed > > Jez > > > On 28/02/17 16:10, Ashish Thandavan wrote: > > Dear all, > > > > We have a small GPFS cluster and a separate server running TSM and one > > of the three NSD servers backs up our GPFS filesystem to the TSM > > server using mmbackup. After a recent upgrade from v3.5 to 4.1.1, > > we've noticed that mmbackup no longer logs stuff like it used to : > > > > ... > > Thu Jan 19 05:45:41 2017 mmbackup:Backing up files: 0 backed up, > > 870532 expired, 2 failed. > > Thu Jan 19 06:15:41 2017 mmbackup:Backing up files: 0 backed up, > > 870532 expired, 3 failed. > > Thu Jan 19 06:45:41 2017 mmbackup:Backing up files: 0 backed up, > > 870532 expired, 3 failed. > > ... > > > > > > instead of > > > > ... > > Sat Dec 3 12:01:00 2016 mmbackup:Backing up files: 105030 backed up, > > 635456 expired, 30 failed. > > Sat Dec 3 12:31:00 2016 mmbackup:Backing up files: 205934 backed up, > > 635456 expired, 57 failed. > > Sat Dec 3 13:01:00 2016 mmbackup:Backing up files: 321702 backed up, > > 635456 expired, 169 failed. > > ... > > > > like it used to pre-upgrade. > > > > I am therefore unable to see how far long it has got, and indeed if it > > completed successfully, as this is what it logs at the end of a job : > > > > ... > > Tue Jan 17 18:07:31 2017 mmbackup:Completed policy backup run with 0 > > policy errors, 10012 files failed, 0 severe errors, returning rc=9. > > Tue Jan 17 18:07:31 2017 mmbackup:Policy for backup returned 9 Highest > > TSM error 12 > > mmbackup: TSM Summary Information: > > Total number of objects inspected: 20617273 > > Total number of objects backed up: 0 > > Total number of objects updated: 0 > > Total number of objects rebound: 0 > > Total number of objects deleted: 0 > > Total number of objects expired: 1 > > Total number of objects failed: 10012 > > Total number of objects encrypted: 0 > > Total number of bytes inspected: 3821624716861 > > Total number of bytes transferred: 3712040943672 > > Tue Jan 17 18:07:31 2017 mmbackup:Audit files /cs/mmbackup.audit.gpfs* > > contain 0 failed paths but there were 10012 failures. > > Cannot reconcile shadow database. > > Unable to compensate for all TSM errors in new shadow database. > > Preserving previous shadow database. > > Run next mmbackup with -q to synchronize shadow database. exit 12 > > > > If it helps, the mmbackup job is kicked off with the following options : > > /usr/lpp/mmfs/bin/mmbackup gpfs -n 8 -t full -B 20000 -L 1 > > --tsm-servers gpfs_weekly_stanza -N glossop1a | /usr/bin/tee > > /var/log/mmbackup/gpfs_weekly/backup_log.`date +%Y%m%d_%H_%M` > > > > (The excerpts above are from the backup_log. file.) > > > > Our NSD servers are running GPFS 4.1.1-11, TSM is at 7.1.1.100 and the > > File system version is 12.06 (3.4.0.3). Has anyone else seen this > > behaviour with mmbackup and if so, found a fix? > > > > Thanks, > > > > Regards, > > Ash > > > > -- Ed Wahl Ohio Supercomputer Center 614-292-9302 From joseph_grace at us.ibm.com Fri Mar 3 19:35:48 2017 From: joseph_grace at us.ibm.com (Joseph Grace) Date: Fri, 3 Mar 2017 14:35:48 -0500 Subject: [gpfsug-discuss] shutdown Message-ID: An HTML attachment was scrubbed... URL: From janfrode at tanso.net Fri Mar 3 20:54:23 2017 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Fri, 03 Mar 2017 20:54:23 +0000 Subject: [gpfsug-discuss] shutdown In-Reply-To: References: Message-ID: Unmount filesystems cleanly "mmumount all -a", stop gpfs "mmshutdown -N gss_ppc64" and poweroff "xdsh gss_ppc64 poweroff". -jf fre. 3. mar. 2017 kl. 20.55 skrev Joseph Grace : > Please excuse the newb question but we have a planned power outage coming > and I can't locate a procedure to shutdown an ESSp GL2 system? > Thanks, > > Joe Grace > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jake.carroll at uq.edu.au Sat Mar 4 20:34:51 2017 From: jake.carroll at uq.edu.au (Jake Carroll) Date: Sat, 4 Mar 2017 20:34:51 +0000 Subject: [gpfsug-discuss] Quota issues, eviction, AFM won't stop throwing data to a full location - probably a rookie AFM mistake? Message-ID: <168ADEDD-5712-42E8-9003-CAE63B2F1192@uq.edu.au> Hi all, I think I need some help with GPFS quotas and hard limits vs soft limits + eviction in AFM scenarios. We?ve got a couple of issues: One: ------- We?ve come across a scenario where if a user hits the hard quota while ingesting into cache in an AFM ?home to cache? relationship whilst an eviction loop is being triggered, things seem to go wrong ? and the filesystem runs off into locking up territory. The report I have on the last incident is that a file-set got stuck at 100% (capacity utilisation), the eviction loop either failed or blocked and the IO requests blocked and/or failed (this one I'm a little fuzzy on). Maybe it isn?t a bug and our guess is that someone on here will probably tell us that the likely ?fix? is to right-size our high and low water marks appropriately. We considered a potential bug mechanism or race condition if the eviction loop uses space in the file-set in the .afm directory ? but I then thought better of it and though ?Nah, surely IBM would have thought of that!?. Two: ------- We witness a scenario where AFM doesn't back off if it gets a filesystem full error code when trying to make the cache clean in migrating data to ?home?. If this takes a couple of seconds to raise the error each attempt, gpfs/mmfsd will deplete NFS daemons causing a DoS against the NFS server that is powering the cache/home relationship for the AFM transport. We had a mental model that AFM cache wouldn?t or shouldn?t overload hard and soft quota as the high and low watermarks for cache eviction policies. I guess in our heads, we?d like caches to also enforce and respect quotas based on requests received from home. There are probably lots of reasons this doesn?t make sense programmatically, or to the rest of scale ? but it would (seem to us) that it would clean up this problem or at least some of it. Happy to chat through his further and explain it more if anyone is interested. If there are any AFM users out there, we?d love to hear about how you deal with quotas, hwm/lwm and eviction over-flow scenarios, if they exist in your environment. Thank you as always, list. -jc -------------- next part -------------- An HTML attachment was scrubbed... URL: From mweil at wustl.edu Tue Mar 7 20:21:42 2017 From: mweil at wustl.edu (Matt Weil) Date: Tue, 7 Mar 2017 14:21:42 -0600 Subject: [gpfsug-discuss] numaMemoryInterleave=yes Message-ID: <85f1c29c-c71d-2d43-bda7-65a7278c1c40@wustl.edu> Hello all Is this necessary any more? numastat -p mmfsd seems to spread it out without it. Thanks Matt ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. From Robert.Oesterlin at nuance.com Tue Mar 7 20:32:32 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Tue, 7 Mar 2017 20:32:32 +0000 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? Message-ID: I?m considering enabling trace on all nodes all the time, doing something like this: mmtracectl --set --trace=def --trace-recycle=global --tracedev-write-mode=overwrite --tracedev-overwrite-buffer-size=256M mmtracectl --start My questions are: - What is the performance penalty of leaving this on 100% of the time on a node? - Does anyone have any suggestions on automation on stopping trace when a particular event occurs? - What other issues, if any? Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Tue Mar 7 20:53:44 2017 From: bbanister at jumptrading.com (Bryan Banister) Date: Tue, 7 Mar 2017 20:53:44 +0000 Subject: [gpfsug-discuss] numaMemoryInterleave=yes In-Reply-To: <85f1c29c-c71d-2d43-bda7-65a7278c1c40@wustl.edu> References: <85f1c29c-c71d-2d43-bda7-65a7278c1c40@wustl.edu> Message-ID: I believe the last I saw from Yuri was that this should always be enabled... but don't recall if there are other things this tunes outside of the numactl stuff, -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Tuesday, March 07, 2017 2:22 PM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] numaMemoryInterleave=yes Hello all Is this necessary any more? numastat -p mmfsd seems to spread it out without it. Thanks Matt ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. From bbanister at jumptrading.com Tue Mar 7 20:58:15 2017 From: bbanister at jumptrading.com (Bryan Banister) Date: Tue, 7 Mar 2017 20:58:15 +0000 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? In-Reply-To: References: Message-ID: The performance impact can be quite significant depending on what you are tracing. We even having monitoring that looks for long running traces and the recommended action is to ?kill with impunity!!? I believe IBM recommends never running clusters with continuous tracing. -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Oesterlin, Robert Sent: Tuesday, March 07, 2017 2:33 PM To: gpfsug main discussion list Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? I?m considering enabling trace on all nodes all the time, doing something like this: mmtracectl --set --trace=def --trace-recycle=global --tracedev-write-mode=overwrite --tracedev-overwrite-buffer-size=256M mmtracectl --start My questions are: - What is the performance penalty of leaving this on 100% of the time on a node? - Does anyone have any suggestions on automation on stopping trace when a particular event occurs? - What other issues, if any? Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Tue Mar 7 21:11:33 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Tue, 7 Mar 2017 21:11:33 +0000 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? Message-ID: I?ve been told that V70o0 unified nodes (GPFS under the covers) run with tracing enabled all the time.. but I agree with you Brian on the potential impacts. But when you must catch a trace for a problem that occurs once every few weeks ? how else would I do it? Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 From: on behalf of Bryan Banister Reply-To: gpfsug main discussion list Date: Tuesday, March 7, 2017 at 2:58 PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? I believe IBM recommends never running clusters with continuous tracing. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Tue Mar 7 21:17:35 2017 From: bbanister at jumptrading.com (Bryan Banister) Date: Tue, 7 Mar 2017 21:17:35 +0000 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? In-Reply-To: References: Message-ID: <098c950ba76e454d861d784f7f66cba1@jumptrading.com> Just depends on how your problem is detected? is it in a log? Is it found by running a command (.e.g mm*)? Is it discovered in `ps` output? Is your scheduler failing jobs? We have ongoing monitoring of most of these types of problem detection points and an automated process to capture a trace could be added to trigger upon detection depending on how the problem is detected? -B From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Oesterlin, Robert Sent: Tuesday, March 07, 2017 3:12 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? I?ve been told that V70o0 unified nodes (GPFS under the covers) run with tracing enabled all the time.. but I agree with you Brian on the potential impacts. But when you must catch a trace for a problem that occurs once every few weeks ? how else would I do it? Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 From: > on behalf of Bryan Banister > Reply-To: gpfsug main discussion list > Date: Tuesday, March 7, 2017 at 2:58 PM To: gpfsug main discussion list > Subject: [EXTERNAL] Re: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? I believe IBM recommends never running clusters with continuous tracing. ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Tue Mar 7 21:25:05 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Tue, 7 Mar 2017 21:25:05 +0000 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? Message-ID: mmfsd crash - IBM says, ?we need a trace to debug the issue?. Sigh Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Bryan Banister Reply-To: gpfsug main discussion list Date: Tuesday, March 7, 2017 at 3:17 PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? Just depends on how your problem is detected? is it in a log? Is it found by running a command (.e.g mm*)? Is it discovered in `ps` output? Is your scheduler failing jobs? -------------- next part -------------- An HTML attachment was scrubbed... URL: From valdis.kletnieks at vt.edu Tue Mar 7 21:36:44 2017 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Tue, 07 Mar 2017 16:36:44 -0500 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? In-Reply-To: <098c950ba76e454d861d784f7f66cba1@jumptrading.com> References: <098c950ba76e454d861d784f7f66cba1@jumptrading.com> Message-ID: <19135.1488922604@turing-police.cc.vt.edu> On Tue, 07 Mar 2017 21:17:35 +0000, Bryan Banister said: > Just depends on how your problem is detected??? is it in a log? Is it found by > running a command (.e.g mm*)? Is it discovered in `ps` output? Is your > scheduler failing jobs? I think the problem here is that if you have a sudden cataclysmic event, you want to have been in flight-recorder mode and be able to look at the last 5 or 10 seconds of trace *before* you became aware that your filesystem just went walkies. Sure, you can start tracing when the filesystem dies - but at that point you just get a whole mess of failed I/O requests in the trace, and no hint of where things went south... -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 484 bytes Desc: not available URL: From aaron.s.knister at nasa.gov Tue Mar 7 21:51:23 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 7 Mar 2017 16:51:23 -0500 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? In-Reply-To: References: Message-ID: <6bc4cf00-7904-0ab3-5f93-963beaca1dbc@nasa.gov> Hi Bob, I have the impression the biggest impact is to metadata-type operations rather than throughput but don't quote me on that because I have very little data to back it up. In the process of testing upgrading from GPFS 3.5 to 4.1 we ran fio on 1000 some nodes against an FS in our test environment which sustained about 60-80k iops on the filesystem's metadata LUNs. At one point I couldn't understand why I was struggling to get about 13k iops and realized tracing was turned on on some subset of nsd servers (which are also manager nodes). After turning it off the throughput immediately shot back up to where I was expecting it to be. Also during testing we were tracking down a bug for which I needed to run tracing *everywhere* and then turn it off when one of the manager nodes saw a particular error. I used a script IBM had sent me a while back to help with this that I made some tweaks to. I've attached it in case its helpful. In a nutshell the process looks like: - start tracing everywhere (/usr/lpp/mmfs/bin/mmdsh -Nall /usr/lpp/mmfs/bin/mmtrace start). Doing it this way avoids the need to change the sdrfs file which depending on your cluster size may or may not have some benefits. - run a command to watch for the event in question that when triggered runs /usr/lpp/mmfs/bin/mmdsh -Nall /usr/lpp/mmfs/bin/mmtrace stop If the condition could present itself on multiple nodes within quick succession (as was the case for me) you could wrap the mmdsh for stopping tracing in an flock, using an arbitrary node that stores the lock locally: ssh $stopHost flock -xn /tmp/mmfsTraceStopLock -c "'/usr/lpp/mmfs/bin/mmdsh -N all /usr/lpp/mmfs/bin/mmtrace stop'" Wrapping it in an flock avoids multiple trace format format attempts. -Aaron On 3/7/17 3:32 PM, Oesterlin, Robert wrote: > I?m considering enabling trace on all nodes all the time, doing > something like this: > > > > mmtracectl --set --trace=def --trace-recycle=global > --tracedev-write-mode=overwrite --tracedev-overwrite-buffer-size=256M > mmtracectl --start > > > > My questions are: > > > > - What is the performance penalty of leaving this on 100% of the time on > a node? > > - Does anyone have any suggestions on automation on stopping trace when > a particular event occurs? > > - What other issues, if any? > > > > > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > 507-269-0413 > > > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 -------------- next part -------------- #!/usr/bin/ksh stopHost=loremds20 mmtrace=/usr/lpp/mmfs/bin/mmtrace mmtracectl=/usr/lpp/mmfs/bin/mmtracectl # No automatic start of mmtrace. # Second to sleep between checking. secondsToSleep=2 # Flag to know when tripped or stopped tripped=0 # mmfs log file to monitor logToGrep=/var/log/messages # Path to mmfs bin directory MMFSbin=/usr/lpp/mmfs/bin # Trip file. Will exist if trap is sprung trapHasSprung=/tmp/mmfsTrapHasSprung rm $trapHasSprung 2>/dev/null # Start tracing on this node #${mmtrace} start # Initial count of expelled message in mmfs log baseCount=$(grep "unmounted by the system with return code 301 reason code" $logToGrep | wc -l) # do this loop while the trip file does not exist while [[ ! -f $trapHasSprung ]] do sleep $secondsToSleep # Get current count of expelled to check against the initial. currentCount=$(grep "unmounted by the system with return code 301 reason code" $logToGrep | wc -l) if [[ $currentCount > $baseCount ]] then tripped=1 /usr/lpp/mmfs/bin/mmdsh -N managernodes,quorumnodes touch $trapHasSprung # cluster manager? #stopHost=$(/usr/lpp/mmfs/bin/tslsmgr | grep '^Cluster manager' | awk '{ print $NF }' | sed -e 's/[()]//g') ssh $stopHost flock -xn /tmp/mmfsTraceStopLock -c "'/usr/lpp/mmfs/bin/mmdsh -N all -f128 /usr/lpp/mmfs/bin/mmtrace stop noformat'" fi done From r.sobey at imperial.ac.uk Wed Mar 8 06:53:17 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Wed, 8 Mar 2017 06:53:17 +0000 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? In-Reply-To: <19135.1488922604@turing-police.cc.vt.edu> References: <098c950ba76e454d861d784f7f66cba1@jumptrading.com> <19135.1488922604@turing-police.cc.vt.edu> Message-ID: We're in the same boat...gpfs snap hangs when the cluster / node is unresponsive but they don't know how to give us a root cause without one. Very frustrating. -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of valdis.kletnieks at vt.edu Sent: 07 March 2017 21:37 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? On Tue, 07 Mar 2017 21:17:35 +0000, Bryan Banister said: > Just depends on how your problem is detected??? is it in a log? Is it > found by running a command (.e.g mm*)? Is it discovered in `ps` > output? Is your scheduler failing jobs? I think the problem here is that if you have a sudden cataclysmic event, you want to have been in flight-recorder mode and be able to look at the last 5 or 10 seconds of trace *before* you became aware that your filesystem just went walkies. Sure, you can start tracing when the filesystem dies - but at that point you just get a whole mess of failed I/O requests in the trace, and no hint of where things went south... From vpuvvada at in.ibm.com Wed Mar 8 10:28:35 2017 From: vpuvvada at in.ibm.com (Venkateswara R Puvvada) Date: Wed, 8 Mar 2017 15:58:35 +0530 Subject: [gpfsug-discuss] Quota issues, eviction, AFM won't stop throwing data to a full location - probably a rookie AFM mistake? In-Reply-To: <168ADEDD-5712-42E8-9003-CAE63B2F1192@uq.edu.au> References: <168ADEDD-5712-42E8-9003-CAE63B2F1192@uq.edu.au> Message-ID: 1. What is the version of GPFS ? Eviction should not be blocking the applications. Was partial file caching enabled ? Eviction cannot evict partially cached files in recent releases. Eviction does not use space inside .afm directory, and its logs are stored under /var/mmfs/tmp by default. 2. I did not understand this requirement. a. When IO to home fails with quota exceeded or no space error , the messages are requeued at gateway node and will be retried later (usually 15 minutes). Cache cannot read home quotas today, and in most of the cases this is not valid. b. When soft quota is exceeded on AFM fileset, auto eviction clears data blocks on files based on LRU policy to bring quota below soft limit. These evicted files are uncached and there is no real migration of data to home during eviction. Eviction should get triggered before fileset usage nearing hard quota and applications getting errors. ~Venkat (vpuvvada at in.ibm.com) From: Jake Carroll To: "gpfsug-discuss at spectrumscale.org" Date: 03/05/2017 02:05 AM Subject: [gpfsug-discuss] Quota issues, eviction, AFM won't stop throwing data to a full location - probably a rookie AFM mistake? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi all, I think I need some help with GPFS quotas and hard limits vs soft limits + eviction in AFM scenarios. We?ve got a couple of issues: One: ------- We?ve come across a scenario where if a user hits the hard quota while ingesting into cache in an AFM ?home to cache? relationship whilst an eviction loop is being triggered, things seem to go wrong ? and the filesystem runs off into locking up territory. The report I have on the last incident is that a file-set got stuck at 100% (capacity utilisation), the eviction loop either failed or blocked and the IO requests blocked and/or failed (this one I'm a little fuzzy on). Maybe it isn?t a bug and our guess is that someone on here will probably tell us that the likely ?fix? is to right-size our high and low water marks appropriately. We considered a potential bug mechanism or race condition if the eviction loop uses space in the file-set in the .afm directory ? but I then thought better of it and though ?Nah, surely IBM would have thought of that!?. Two: ------- We witness a scenario where AFM doesn't back off if it gets a filesystem full error code when trying to make the cache clean in migrating data to ?home?. If this takes a couple of seconds to raise the error each attempt, gpfs/mmfsd will deplete NFS daemons causing a DoS against the NFS server that is powering the cache/home relationship for the AFM transport. We had a mental model that AFM cache wouldn?t or shouldn?t overload hard and soft quota as the high and low watermarks for cache eviction policies. I guess in our heads, we?d like caches to also enforce and respect quotas based on requests received from home. There are probably lots of reasons this doesn?t make sense programmatically, or to the rest of scale ? but it would (seem to us) that it would clean up this problem or at least some of it. Happy to chat through his further and explain it more if anyone is interested. If there are any AFM users out there, we?d love to hear about how you deal with quotas, hwm/lwm and eviction over-flow scenarios, if they exist in your environment. Thank you as always, list. -jc _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From duersch at us.ibm.com Wed Mar 8 13:32:37 2017 From: duersch at us.ibm.com (Steve Duersch) Date: Wed, 8 Mar 2017 08:32:37 -0500 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? In-Reply-To: References: Message-ID: You are correct. There is some level of tracing on the v7000U. Also, if the gpfs daemon hits an assert or signal and tracing is running then traces are recycled automatically. There is no need to have another script monitoring for such events. You probably want to have mmtracectl --trace-recycle=global so that an assert on one node grabs traces on all nodes (ie fs manager). As for performance, this one is more difficult and I don't have details. I can ping a colleague to see if he has numbers. Obviously there will be a performance penalty for running tracing. Steve Duersch Spectrum Scale IBM Poughkeepsie, New York > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 7 Mar 2017 21:11:33 +0000 > From: "Oesterlin, Robert" > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Potential problems - leaving trace > enabled in over-write mode? > Message-ID: > Content-Type: text/plain; charset="utf-8" > > I?ve been told that V70o0 unified nodes (GPFS under the covers) run > with tracing enabled all the time.. but I agree with you Brian on > the potential impacts. But when you must catch a trace for a problem > that occurs once every few weeks ? how else would I do it? > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > 507-269-0413 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oehmes at gmail.com Wed Mar 8 13:37:15 2017 From: oehmes at gmail.com (Sven Oehme) Date: Wed, 08 Mar 2017 13:37:15 +0000 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? In-Reply-To: References: Message-ID: given that i love data points, let me put some behind this thread :-) starting in version 3.4 we enhanced the trace code of scale significant. this went on release to release all the way up to 4.2.1. since 4.2.1 we made further improvements, but much smaller changes, more optimization , e.g. reducing of trace levels verbosity, etc . with 4.2.2 we switched from blocking traces to in memory traces as the default trace infrastructure, this infrastructure was designed to be turned on all the time with minimal impact on performance. i just took a 4.2.3 build and did load it on one of my faster NVMe nodes and did a 100% random read test with 64 threads against very fast storage. while this was running i started trace with mmtracectl --start ; sleep 10 ; mmtracectl --off , attached is a screenshot of this run : [image: pasted1] the trace started at 14:19:15 and stopped at 14:20:00 (that includes 10 command processing, format of traces, etc) as one can see on the chart the dip in performance is very small, measured looking at raw data its 8%. the system runs at ~920k 4k random read iops /sec and the workload running on the node pushes the system almost to 100% cpu time even without trace running, reason i tested under this condition is because also on a HPC compute oriented workload you will run without spare cpu cycles, so the 8% is under extreme conditions. said all this, the workload i am running is not causing a lot of metadata i/o as i am running within a small number of files, its all read, so no write, therefore this isn't representative to a lot of real world workloads, but covers one very extreme case , high iops requirements under heavy cpu contention. therefore i repeated the test under a heavy metadata workload , SpecSFS SWBUILD. when i run the test with and without traces turned on i see a ~15% loss on max operations/sec/node. again this is another extreme case as there are only few workloads which have a metadata workload component as heavy as SWBUILD. but given that between the 2 extreme cases we are talking about 8 - 15% overhead its is very likely that all real world workloads suffer less than 15% when traces are turned on. only a test in your environment will really show . i would not recommend to try this with any version prior 4.2.1 as the impact will be significant higher than what i measured. hope this helps make a informed decision. Sven On Tue, Mar 7, 2017 at 9:32 PM Oesterlin, Robert < Robert.Oesterlin at nuance.com> wrote: > I?m considering enabling trace on all nodes all the time, doing something > like this: > > > > mmtracectl --set --trace=def --trace-recycle=global > --tracedev-write-mode=overwrite --tracedev-overwrite-buffer-size=256M > mmtracectl --start > > > > My questions are: > > > > - What is the performance penalty of leaving this on 100% of the time on a > node? > > - Does anyone have any suggestions on automation on stopping trace when a > particular event occurs? > > - What other issues, if any? > > > > > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > 507-269-0413 <(507)%20269-0413> > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pasted1 Type: image/png Size: 216901 bytes Desc: not available URL: From Robert.Oesterlin at nuance.com Wed Mar 8 13:56:32 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Wed, 8 Mar 2017 13:56:32 +0000 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? Message-ID: <90CE7D81-D1F0-4E29-B9F2-02D467911E6C@nuance.com> As always, Sven comes in to back this up with real data :) To net this out, Sven ? I should be able enable trace on my NSD servers running 4.2.2 without much impact, correct? Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Sven Oehme Reply-To: gpfsug main discussion list Date: Wednesday, March 8, 2017 at 7:37 AM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? starting in version 3.4 we enhanced the trace code of scale significant. this went on release to release all the way up to 4.2.1. since 4.2.1 we made further improvements, but much smaller changes, more optimization , e.g. reducing of trace levels verbosity, etc . with 4.2.2 we switched from blocking traces to in memory traces as the default trace infrastructure, this infrastructure was designed to be turned on all the time with minimal impact on performance. -------------- next part -------------- An HTML attachment was scrubbed... URL: From oehmes at gmail.com Wed Mar 8 14:15:55 2017 From: oehmes at gmail.com (Sven Oehme) Date: Wed, 08 Mar 2017 14:15:55 +0000 Subject: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode? In-Reply-To: <90CE7D81-D1F0-4E29-B9F2-02D467911E6C@nuance.com> References: <90CE7D81-D1F0-4E29-B9F2-02D467911E6C@nuance.com> Message-ID: yes , but i would do this in stages given how large your system is. pick one set of nodes (lets say) 100 out of 200 that do similar things and turn tracing on there. this will give you data you can compare between the 2 set of nodes. let it run for a week and if the data with your real workload doesn't show any significant degradation (which is what i expect) turn it on everywhere. the one thing i am not 100% sure about is size of trace buffer as well as the global cut config. what this means is if you apply the settings as mentioned in this first post, if one node asserts in your cluster you will cut a trace on all nodes that will write a 256M buffer into your dump file location. if you have a node thats in an assert loop (asserts, restarts , asserts) this can cause significant load on all nodes. therefore i would probably start without cutting a global trace and reduce the trace size to 64M. i (and i am sure other dev folks) would be very interested in the outcome as we have this debate on a yearly basis if we shouldn't just turn tracing on by default, in the past performance was the biggest hurdle, this is solved now (my claim) . next big questions is how well does that work on larger scale systems with production workloads. as more feedback we will get in this area as better we can make informed decision how and if it could be turned on all the time and work harder on handling cases like i mentioned above to mitigate the risks . sven On Wed, Mar 8, 2017 at 2:56 PM Oesterlin, Robert < Robert.Oesterlin at nuance.com> wrote: > As always, Sven comes in to back this up with real data :) > > > > To net this out, Sven ? I should be able enable trace on my NSD servers > running 4.2.2 without much impact, correct? > > > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > > > > > > *From: * on behalf of Sven > Oehme > *Reply-To: *gpfsug main discussion list > *Date: *Wednesday, March 8, 2017 at 7:37 AM > > > *To: *gpfsug main discussion list > > *Subject: *[EXTERNAL] Re: [gpfsug-discuss] Potential problems - leaving > trace enabled in over-write mode? > > > > starting in version 3.4 we enhanced the trace code of scale significant. > this went on release to release all the way up to 4.2.1. since 4.2.1 we > made further improvements, but much smaller changes, more optimization , > e.g. reducing of trace levels verbosity, etc . > > with 4.2.2 we switched from blocking traces to in memory traces as the > default trace infrastructure, this infrastructure was designed to be turned > on all the time with minimal impact on performance. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Wed Mar 8 15:25:16 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Wed, 8 Mar 2017 15:25:16 +0000 Subject: [gpfsug-discuss] Registration Reminder: GPFSUG-USA Meeting at NERSC April 4-5th Message-ID: Here?s a reminder to register for the upcoming user group meeting! Registration deadline is March 24th! We Still have slots (20 mins each) for customer and partner talks ? please contact me if you want to do a short presentation. Register here: https://www.nersc.gov/research-and-development/data-analytics/spectrum-user-group-meeting/ Bob Oesterlin Sr Principal Storage Engineer, Nuance -------------- next part -------------- An HTML attachment was scrubbed... URL: From jenocram at gmail.com Wed Mar 8 20:15:44 2017 From: jenocram at gmail.com (Jeno Cram) Date: Wed, 8 Mar 2017 15:15:44 -0500 Subject: [gpfsug-discuss] CES, NFSv4 and Kerberos with AD authentication Message-ID: I'm running into an issue and I can't find the answer anywhere in the Spectrum Scale documentation. I'm trying to keberize nfsv4 with AD authentication for file access. I have the instructions for the server (--enable-nfs-kerberos) and also ran "mmnfs configuration change From jenocram at gmail.com Wed Mar 8 20:18:07 2017 From: jenocram at gmail.com (Jeno Cram) Date: Wed, 8 Mar 2017 15:18:07 -0500 Subject: [gpfsug-discuss] CES, NFSv4 and Kerberos with AD authentication In-Reply-To: References: Message-ID: My apologies.. I hit send too soon. I'm running into an issue and I can't find the answer anywhere in the Spectrum Scale documentation. I'm trying to keberize nfsv4 with AD authentication for file access. I have the instructions for the server (--enable-nfs-kerberos) and also ran "mmnfs configuration change idmapd_domain=example.com". Is there anything else that needs to be done on the server side and what needs to be configured on the client to allow this? On Wed, Mar 8, 2017 at 3:15 PM, Jeno Cram wrote: > I'm running into an issue and I can't find the answer anywhere in the > Spectrum Scale documentation. > > I'm trying to keberize nfsv4 with AD authentication for file access. > I have the instructions for the server (--enable-nfs-kerberos) and > also ran "mmnfs configuration change From Robert.Oesterlin at nuance.com Wed Mar 8 21:54:33 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Wed, 8 Mar 2017 21:54:33 +0000 Subject: [gpfsug-discuss] Problem installin CES - 4.2.2-2 Message-ID: <19C1E64C-76E3-4C59-914E-199EA5D37CCF@nuance.com> OK, I?ll admit I?m not protocols install expert. Using the ?spectrumscale? installer command, it?s failing to install the object packages due to some internal dependencies from the IBM supplied repo. Who can help me? Excerpt from the install log: 2017-03-08 16:47:34,319 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com * log[IBM SPECTRUM SCALE: Installing Object packages (SS50).] action write 2017-03-08 16:47:34,319 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,554 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com * yum_package[spectrum-scale-object] action install 2017-03-08 16:47:59,555 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,555 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com ================================================================================ 2017-03-08 16:47:59,556 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error executing action `install` on resource 'yum_package[spectrum-scale-object]' 2017-03-08 16:47:59,556 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com ================================================================================ 2017-03-08 16:47:59,557 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,557 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Chef::Exceptions::Exec 2017-03-08 16:47:59,558 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com ---------------------- 2017-03-08 16:47:59,558 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com yum -d0 -e0 -y install spectrum-scale-object-4.2.2-2 returned 1: 2017-03-08 16:47:59,558 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com STDOUT: Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,560 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,560 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,560 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,563 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,563 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,563 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com You could try using --skip-broken to work around the problem 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com You could try running: rpm -Va --nofiles --nodigest 2017-03-08 16:47:59,565 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,565 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com STDERR: Error: Package: 1:python-novaclient-2.30.3-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,565 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error: Package: 1:python-glanceclient-1.1.2-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error: Package: python-openstackclient-1.7.3-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,569 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,569 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,569 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error: Package: python-neutronclient-3.1.2-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,571 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,571 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,571 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 Bob Oesterlin Sr Principal Storage Engineer, Nuance -------------- next part -------------- An HTML attachment was scrubbed... URL: From stockf at us.ibm.com Wed Mar 8 22:06:18 2017 From: stockf at us.ibm.com (Frederick Stock) Date: Wed, 8 Mar 2017 17:06:18 -0500 Subject: [gpfsug-discuss] Problem installin CES - 4.2.2-2 In-Reply-To: <19C1E64C-76E3-4C59-914E-199EA5D37CCF@nuance.com> References: <19C1E64C-76E3-4C59-914E-199EA5D37CCF@nuance.com> Message-ID: What version of python do you have installed? I think you need at least version 2.7. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 03/08/2017 04:55 PM Subject: [gpfsug-discuss] Problem installin CES - 4.2.2-2 Sent by: gpfsug-discuss-bounces at spectrumscale.org OK, I?ll admit I?m not protocols install expert. Using the ?spectrumscale? installer command, it?s failing to install the object packages due to some internal dependencies from the IBM supplied repo. Who can help me? Excerpt from the install log: 2017-03-08 16:47:34,319 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com * log[IBM SPECTRUM SCALE: Installing Object packages (SS50).] action write 2017-03-08 16:47:34,319 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,554 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com * yum_package[spectrum-scale-object] action install 2017-03-08 16:47:59,555 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,555 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com ================================================================================ 2017-03-08 16:47:59,556 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error executing action `install` on resource 'yum_package[spectrum-scale-object]' 2017-03-08 16:47:59,556 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com ================================================================================ 2017-03-08 16:47:59,557 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,557 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Chef::Exceptions::Exec 2017-03-08 16:47:59,558 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com ---------------------- 2017-03-08 16:47:59,558 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com yum -d0 -e0 -y install spectrum-scale-object-4.2.2-2 returned 1: 2017-03-08 16:47:59,558 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com STDOUT: Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,560 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,560 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,560 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,563 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,563 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,563 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com You could try using --skip-broken to work around the problem 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com You could try running: rpm -Va --nofiles --nodigest 2017-03-08 16:47:59,565 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,565 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com STDERR: Error: Package: 1:python-novaclient-2.30.3-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,565 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error: Package: 1:python-glanceclient-1.1.2-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error: Package: python-openstackclient-1.7.3-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,569 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,569 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,569 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error: Package: python-neutronclient-3.1.2-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,571 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,571 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,571 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 Bob Oesterlin Sr Principal Storage Engineer, Nuance _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mark.Bush at siriuscom.com Wed Mar 8 23:25:07 2017 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Wed, 8 Mar 2017 23:25:07 +0000 Subject: [gpfsug-discuss] Problem installin CES - 4.2.2-2 Message-ID: <4FA8EF66-4E49-4C65-AA6E-5166AF8F45C7@siriuscom.com> What version of RHEL/CENTOS? From: "Oesterlin, Robert" Reply-To: gpfsug main discussion list Date: Wednesday, March 8, 2017 at 3:54 PM To: gpfsug main discussion list Subject: [gpfsug-discuss] Problem installin CES - 4.2.2-2 OK, I?ll admit I?m not protocols install expert. Using the ?spectrumscale? installer command, it?s failing to install the object packages due to some internal dependencies from the IBM supplied repo. Who can help me? Excerpt from the install log: 2017-03-08 16:47:34,319 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com * log[IBM SPECTRUM SCALE: Installing Object packages (SS50).] action write 2017-03-08 16:47:34,319 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,554 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com * yum_package[spectrum-scale-object] action install 2017-03-08 16:47:59,555 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,555 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com ================================================================================ 2017-03-08 16:47:59,556 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error executing action `install` on resource 'yum_package[spectrum-scale-object]' 2017-03-08 16:47:59,556 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com ================================================================================ 2017-03-08 16:47:59,557 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,557 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Chef::Exceptions::Exec 2017-03-08 16:47:59,558 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com ---------------------- 2017-03-08 16:47:59,558 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com yum -d0 -e0 -y install spectrum-scale-object-4.2.2-2 returned 1: 2017-03-08 16:47:59,558 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com STDOUT: Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,559 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,560 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,560 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,560 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,561 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,562 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,563 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,563 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,563 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Package python-keystoneclient is obsoleted by python2-keystoneclient, but obsoleting package does not provide for requirements 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com You could try using --skip-broken to work around the problem 2017-03-08 16:47:59,564 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com You could try running: rpm -Va --nofiles --nodigest 2017-03-08 16:47:59,565 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com 2017-03-08 16:47:59,565 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com STDERR: Error: Package: 1:python-novaclient-2.30.3-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,565 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,566 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error: Package: 1:python-glanceclient-1.1.2-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,567 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error: Package: python-openstackclient-1.7.3-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,568 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,569 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,569 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,569 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Error: Package: python-neutronclient-3.1.2-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Requires: python-keystoneclient < 1:3.0.0 2017-03-08 16:47:59,570 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Available: 1:python-keystoneclient-1.7.5-1.ibm.el7.noarch (ces_object) 2017-03-08 16:47:59,571 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:1.7.5-1.ibm.el7 2017-03-08 16:47:59,571 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com Installing: 1:python2-keystoneclient-3.5.0-1.el7.noarch (tweaks) 2017-03-08 16:47:59,571 [ TRACE ] arch-cnfs02.nrg1.us.grid.nuance.com python-keystoneclient = 1:3.5.0-1.el7 Bob Oesterlin Sr Principal Storage Engineer, Nuance This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Thu Mar 9 00:25:41 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Thu, 9 Mar 2017 00:25:41 +0000 Subject: [gpfsug-discuss] Problem installin CES - 4.2.2-2 In-Reply-To: <4FA8EF66-4E49-4C65-AA6E-5166AF8F45C7@siriuscom.com> References: <4FA8EF66-4E49-4C65-AA6E-5166AF8F45C7@siriuscom.com> Message-ID: 7.2 No other conflicting ?openstack|keystone|swift|glance|nova|neutron? package installed that I can see. Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 From: on behalf of "Mark.Bush at siriuscom.com" Reply-To: gpfsug main discussion list Date: Wednesday, March 8, 2017 at 5:25 PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Problem installin CES - 4.2.2-2 What version of RHEL/CENTOS? -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mark.Bush at siriuscom.com Thu Mar 9 14:12:40 2017 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Thu, 9 Mar 2017 14:12:40 +0000 Subject: [gpfsug-discuss] Problem installin CES - 4.2.2-2 In-Reply-To: References: <4FA8EF66-4E49-4C65-AA6E-5166AF8F45C7@siriuscom.com> Message-ID: I don?t normally use the chef based installer but I have had issues getting OBJ installed on 7.x before and it was all based on the fact that the packages were moved to subdirs /usr/local/mmfs/4.2.2.0/object_rpms/rhel7 where before they were just in the root of object_rpms. Not sure that applies here since I would think the installer was updated to look there instead of the root. It sure seems from your log output that it?s not happy trying to install packages. Worse case you can install manually. Here?s how I do that in my automated script cat </etc/yum.repos.d/IBM-SpecScaleOBJ.repo [spectrum_scale] name=spectrum_scale baseurl=file:///vagrant/4.2.2.0/object_rpms/rhel7/ enabled=1 EOL yum update -y 2>&1 yum install -y /vagrant/4.2.2.0/zimon_rpms/rhel7/gpfs.gss.pm{sensors,collector}*.rpm > /dev/null 2>&1 yum install -y /vagrant/4.2.2.0/gpfs_rpms/gpfs.{base,ext,gskit,gpl,msg,docs,adv,crypto,java,gui}*.rpm > /dev/null 2>&1 yum install -y /vagrant/4.2.2.0/smb_rpms/rhel7/gpfs.*.rpm > /dev/null 2>&1 yum install -y /vagrant/4.2.2.0/ganesha_rpms/rhel7/nfs-*.rpm > /dev/null 2>&1 yum install -y /vagrant/4.2.2.0/object_rpms/rhel7/*.rpm >/dev/null 2>&1 yum install -y spectrum-scale-object 2>&1 From: "Oesterlin, Robert" Reply-To: gpfsug main discussion list Date: Wednesday, March 8, 2017 at 6:25 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Problem installin CES - 4.2.2-2 7.2 No other conflicting ?openstack|keystone|swift|glance|nova|neutron? package installed that I can see. Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 From: on behalf of "Mark.Bush at siriuscom.com" Reply-To: gpfsug main discussion list Date: Wednesday, March 8, 2017 at 5:25 PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Problem installin CES - 4.2.2-2 What version of RHEL/CENTOS? This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Thu Mar 9 14:53:33 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Thu, 9 Mar 2017 14:53:33 +0000 Subject: [gpfsug-discuss] Problem installin CES - 4.2.2-2 Message-ID: Bingo! That worked flawlessly. You are a lifesaver, thanks! Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of "Mark.Bush at siriuscom.com" Reply-To: gpfsug main discussion list Date: Thursday, March 9, 2017 at 8:12 AM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Problem installin CES - 4.2.2-2 I don?t normally use the chef based installer but I have had issues getting OBJ installed on 7.x before and it was all based on the fact that the packages were moved to subdirs /usr/local/mmfs/4.2.2.0/object_rpms/rhel7 where before they were just in the root of object_rpms. Not sure that applies here since I would think the installer was updated to look there instead of the root. It sure seems from your log output that it?s not happy trying to install packages. Worse case you can install manually. Here?s how I do that in my automated script cat </etc/yum.repos.d/IBM-SpecScaleOBJ.repo [spectrum_scale] name=spectrum_scale baseurl=file:///vagrant/4.2.2.0/object_rpms/rhel7/ enabled=1 EOL yum update -y 2>&1 yum install -y /vagrant/4.2.2.0/zimon_rpms/rhel7/gpfs.gss.pm{sensors,collector}*.rpm > /dev/null 2>&1 yum install -y /vagrant/4.2.2.0/gpfs_rpms/gpfs.{base,ext,gskit,gpl,msg,docs,adv,crypto,java,gui}*.rpm > /dev/null 2>&1 yum install -y /vagrant/4.2.2.0/smb_rpms/rhel7/gpfs.*.rpm > /dev/null 2>&1 yum install -y /vagrant/4.2.2.0/ganesha_rpms/rhel7/nfs-*.rpm > /dev/null 2>&1 yum install -y /vagrant/4.2.2.0/object_rpms/rhel7/*.rpm >/dev/null 2>&1 yum install -y spectrum-scale-object 2>&1 From: "Oesterlin, Robert" Reply-To: gpfsug main discussion list Date: Wednesday, March 8, 2017 at 6:25 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Problem installin CES - 4.2.2-2 7.2 No other conflicting ?openstack|keystone|swift|glance|nova|neutron? package installed that I can see. Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 From: on behalf of "Mark.Bush at siriuscom.com" Reply-To: gpfsug main discussion list Date: Wednesday, March 8, 2017 at 5:25 PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Problem installin CES - 4.2.2-2 What version of RHEL/CENTOS? This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Thu Mar 9 16:11:56 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Thu, 9 Mar 2017 16:11:56 +0000 Subject: [gpfsug-discuss] Running gpfs.snap outside of problems Message-ID: Hi all, Is there any best practice as to when to run a gpfs.snap, and the impact it will have on a healthy cluster? Some of my colleagues are keen to gather a snap for example every 2 hours to assist in problem determination. I can elaborate on the "why" part of this if needed but I'm not fully convinced it would help. Said I'd look into it though so I'm asking the question. Cheers Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Thu Mar 9 16:29:52 2017 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Thu, 09 Mar 2017 16:29:52 +0000 Subject: [gpfsug-discuss] Running gpfs.snap outside of problems In-Reply-To: References: Message-ID: There's a manual for it now.. and it points out "The tool impacts performance.." Also it has caused mmfsd crashes for me earlier, so I've learned to be weary of running it.. The manual also says it's collecting using mmfsadm, and the mmfsadm manual warns that it might cause GPFS to fail ("in certain rare cases"). I wouldn't dare running it every few hours. -jf tor. 9. mar. 2017 kl. 17.12 skrev Sobey, Richard A : > Hi all, > > > > Is there any best practice as to when to run a gpfs.snap, and the impact > it will have on a healthy cluster? Some of my colleagues are keen to gather > a snap for example every 2 hours to assist in problem determination. > > > > I can elaborate on the ?why? part of this if needed but I?m not fully > convinced it would help. Said I?d look into it though so I?m asking the > question. > > > > Cheers > > Richard > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mxcheng at uchicago.edu Thu Mar 9 19:23:38 2017 From: mxcheng at uchicago.edu (Mengxing Cheng) Date: Thu, 9 Mar 2017 19:23:38 +0000 Subject: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem Message-ID: Dear all, My name is Mengxing Cheng and I am a HPC system administrator at the University of Chicago. We have a GSS26 running gss2.5.10.3-3b and gpfs-4.2.0.3. Recently, we run mmfsck on a relatively small filesystem with 14TB block and 73863102 inodes but it was unusually slow so as to not be able to finish in 48 hours. In contrast, mmfsck run on a filesystem with the same size and inodes but sitting on a traditional IBM DS3512 storage took only 2 hours to complete. In particular, the mmfsck run in parallel using 3 nodes within the GSS storage cluster, we notice that one gss storage server scans inodes much slower than the other gss storage server and the quorum service node. Has anyone experience the same mmfsck performance issue? Could anyone make recommendation to troubleshoot and improve mmfsck performance? Thank you! Mengxing --- Mengxing Cheng, Ph.D. HPC System Administrator Research Computing Center The University of Chicago 5607 S. Drexel Ave. Chicago, IL 60637 email: mxcheng at uchicago.edu phone: (773) 702-4070 -------------- next part -------------- An HTML attachment was scrubbed... URL: From kevindjo at us.ibm.com Thu Mar 9 19:47:56 2017 From: kevindjo at us.ibm.com (Kevin D Johnson) Date: Thu, 9 Mar 2017 19:47:56 +0000 Subject: [gpfsug-discuss] Running gpfs.snap outside of problems In-Reply-To: References: , Message-ID: An HTML attachment was scrubbed... URL: From esperle at us.ibm.com Thu Mar 9 21:13:08 2017 From: esperle at us.ibm.com (Eric Sperley) Date: Thu, 9 Mar 2017 13:13:08 -0800 Subject: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem In-Reply-To: References: Message-ID: Mengxing, It is nice meeting you. I have seen a situation where the amount of RAM on a node can affect mmfsck times. Do all the nodes have the same amount of RAM, or does the slow running node have less RAM? Best Regards, Eric Eric Sperley SDI Architect "Carpe Diem" IBM Systems esperle at us.ibm.com +15033088721 From: Mengxing Cheng To: "gpfsug-discuss at spectrumscale.org" Date: 03/09/2017 11:24 AM Subject: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem Sent by: gpfsug-discuss-bounces at spectrumscale.org Dear all, My name is Mengxing Cheng and I am a HPC system administrator at the University of Chicago. We have a GSS26 running gss2.5.10.3-3b and gpfs-4.2.0.3. Recently, we run mmfsck on a relatively small filesystem with 14TB block and 73863102 inodes but it was unusually slow so as to not be able to finish in 48 hours. In contrast, mmfsck run on a filesystem with the same size and inodes but sitting on a traditional IBM DS3512 storage took only 2 hours to complete. In particular, the mmfsck run in parallel using 3 nodes within the GSS storage cluster, we notice that one gss storage server scans inodes much slower than the other gss storage server and the quorum service node. Has anyone experience the same mmfsck performance issue? Could anyone make recommendation to troubleshoot and improve mmfsck performance? Thank you! Mengxing --- Mengxing Cheng, Ph.D. HPC System Administrator Research Computing Center The University of Chicago 5607 S. Drexel Ave. Chicago, IL 60637 email: mxcheng at uchicago.edu phone: (773) 702-4070 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 1C803747.gif Type: image/gif Size: 481 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecblank.gif Type: image/gif Size: 45 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 1C166252.gif Type: image/gif Size: 2322 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From mxcheng at uchicago.edu Thu Mar 9 21:24:21 2017 From: mxcheng at uchicago.edu (Mengxing Cheng) Date: Thu, 9 Mar 2017 21:24:21 +0000 Subject: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem In-Reply-To: References: , Message-ID: Eric, thank you very much for replying. Here is the memory configuration and current usage. Note that mmfsck is not running now. The two gss servers have the same 256GB memory and the service node has 128GB. 1. service node: total used free shared buffers cached Mem: 125 58 66 0 0 4 -/+ buffers/cache: 53 71 Swap: 7 0 7 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12990 root 0 -20 71.0g 43g 885m S 7.6 34.4 9306:00 mmfsd 2. gss nodes: ==================================== total used free shared buff/cache available Mem: 251 210 37 0 4 36 Swap: 3 0 3 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 36770 root 0 -20 0.216t 0.192t 2.667g S 48.9 78.1 75684:09 /usr/lpp/mmfs/bin/mmfsd The gss nodes' memory usage is so high because their pagepool is set to 192GB while the service node has 16GB pagepool. Mengxing --- Mengxing Cheng, Ph.D. HPC System Administrator Research Computing Center The University of Chicago 5607 S. Drexel Ave. Chicago, IL 60637 email: mxcheng at uchicago.edu phone: (773) 702-4070 ________________________________ From: Eric Sperley [esperle at us.ibm.com] Sent: Thursday, March 09, 2017 3:13 PM To: Mengxing Cheng Cc: gpfsug main discussion list Subject: Re: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem Mengxing, It is nice meeting you. I have seen a situation where the amount of RAM on a node can affect mmfsck times. Do all the nodes have the same amount of RAM, or does the slow running node have less RAM? Best Regards, Eric [cid:3__=8FBB0A4DDFE7DC3F8f9e8a93df938690918c8FB@] Eric Sperley SDI Architect "Carpe Diem" IBM Systems esperle at us.ibm.com +15033088721 [Inactive hide details for Mengxing Cheng ---03/09/2017 11:24:02 AM---Dear all, My name is Mengxing Cheng and I am a HPC system]Mengxing Cheng ---03/09/2017 11:24:02 AM---Dear all, My name is Mengxing Cheng and I am a HPC system administrator at the University of Chicago From: Mengxing Cheng To: "gpfsug-discuss at spectrumscale.org" Date: 03/09/2017 11:24 AM Subject: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Dear all, My name is Mengxing Cheng and I am a HPC system administrator at the University of Chicago. We have a GSS26 running gss2.5.10.3-3b and gpfs-4.2.0.3. Recently, we run mmfsck on a relatively small filesystem with 14TB block and 73863102 inodes but it was unusually slow so as to not be able to finish in 48 hours. In contrast, mmfsck run on a filesystem with the same size and inodes but sitting on a traditional IBM DS3512 storage took only 2 hours to complete. In particular, the mmfsck run in parallel using 3 nodes within the GSS storage cluster, we notice that one gss storage server scans inodes much slower than the other gss storage server and the quorum service node. Has anyone experience the same mmfsck performance issue? Could anyone make recommendation to troubleshoot and improve mmfsck performance? Thank you! Mengxing --- Mengxing Cheng, Ph.D. HPC System Administrator Research Computing Center The University of Chicago 5607 S. Drexel Ave. Chicago, IL 60637 email: mxcheng at uchicago.edu phone: (773) 702-4070 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 1C803747.gif Type: image/gif Size: 481 bytes Desc: 1C803747.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecblank.gif Type: image/gif Size: 45 bytes Desc: ecblank.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 1C166252.gif Type: image/gif Size: 2322 bytes Desc: 1C166252.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: graycol.gif URL: From r.sobey at imperial.ac.uk Fri Mar 10 09:07:08 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Fri, 10 Mar 2017 09:07:08 +0000 Subject: [gpfsug-discuss] Running gpfs.snap outside of problems In-Reply-To: References: , Message-ID: Thanks Kevin and J-F, I appreciate your thoughts. I?ll go with ?no? then as my answer ? Cheers Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Kevin D Johnson Sent: 09 March 2017 19:48 To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Running gpfs.snap outside of problems I've never heard of running a gpfs.snap every few hours, but it is technically possible. Probably not advisable except in the most limited of circumstances. I wouldn't do it unless Support or another technical resource you trust says to do so. You can limit the snap to run on a few nodes versus a whole cluster and that way limit the effect it has overall. You can also specify a directory outside GPFS to write the snap file. Kevin D. Johnson, MBA, MAFM Spectrum Computing, Senior Managing Consultant IBM Certified Deployment Professional - Spectrum Scale V4.1.1 IBM Certified Deployment Professional - Cloud Object Storage V3.8 IBM Certified Solution Advisor - Spectrum Computing V1 720.349.6199 - kevindjo at us.ibm.com ----- Original message ----- From: Jan-Frode Myklebust Sent by: gpfsug-discuss-bounces at spectrumscale.org To: "gpfsug-discuss at spectrumscale.org" Cc: Subject: Re: [gpfsug-discuss] Running gpfs.snap outside of problems Date: Thu, Mar 9, 2017 11:30 AM There's a manual for it now.. and it points out "The tool impacts performance.." Also it has caused mmfsd crashes for me earlier, so I've learned to be weary of running it.. The manual also says it's collecting using mmfsadm, and the mmfsadm manual warns that it might cause GPFS to fail ("in certain rare cases"). I wouldn't dare running it every few hours. -jf tor. 9. mar. 2017 kl. 17.12 skrev Sobey, Richard A >: Hi all, Is there any best practice as to when to run a gpfs.snap, and the impact it will have on a healthy cluster? Some of my colleagues are keen to gather a snap for example every 2 hours to assist in problem determination. I can elaborate on the ?why? part of this if needed but I?m not fully convinced it would help. Said I?d look into it though so I?m asking the question. Cheers Richard _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From chair at spectrumscale.org Fri Mar 10 10:22:24 2017 From: chair at spectrumscale.org (GPFS UG Chair (Simon Thompson)) Date: Fri, 10 Mar 2017 10:22:24 +0000 Subject: [gpfsug-discuss] UK Spectrum Scale User Group Sponsorship packages Message-ID: Hi All, As we are developing the programme for the UK Spectrum Scale UG on 9th/10th May, we are again looking for commercial sponsors to support the evening event we hope to run. I have already contacted a number of contacts who kindly supported last year's event, however if you are interested in being a sponsor for the event, please get in touch with me directly and I can provide you with more information. Simon From Achim.Rehor at de.ibm.com Fri Mar 10 16:56:30 2017 From: Achim.Rehor at de.ibm.com (Achim Rehor) Date: Fri, 10 Mar 2017 17:56:30 +0100 Subject: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem In-Reply-To: References: , Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 7182 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 481 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 45 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 2322 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 105 bytes Desc: not available URL: From mxcheng at uchicago.edu Fri Mar 10 16:59:43 2017 From: mxcheng at uchicago.edu (Mengxing Cheng) Date: Fri, 10 Mar 2017 16:59:43 +0000 Subject: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem In-Reply-To: References: , , Message-ID: Achim, thank you very much! Mengxing --- Mengxing Cheng, Ph.D. HPC System Administrator Research Computing Center The University of Chicago 5607 S. Drexel Ave. Chicago, IL 60637 email: mxcheng at uchicago.edu phone: (773) 702-4070 ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Achim Rehor [Achim.Rehor at de.ibm.com] Sent: Friday, March 10, 2017 10:56 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem Hi, mmfsck is highly dependent on pagepool, so my best guess would be, the the EMS node with only 16GB of pagepool is sort of a bottleneck for the slow progress. You might want to try to temporarily raise the pagepool on the service node to .. lets say 64GB, or restrict the mmfsck to run only on the storage nodes. note: the fs mgr node will always be part of the mmfsck nodes. so if it is located on the service node, mmchmgr to a storage node first Mit freundlichen Gr??en / Kind regards Achim Rehor ________________________________ Software Technical Support Specialist AIX/ Emea HPC Support [cid:_1_D975D64CD975D0CC005D0FE8C12580DF] IBM Certified Advanced Technical Expert - Power Systems with AIX TSCC Software Service, Dept. 7922 Global Technology Services ________________________________ Phone: +49-7034-274-7862 IBM Deutschland E-Mail: Achim.Rehor at de.ibm.com Am Weiher 24 65451 Kelsterbach Germany ________________________________ IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter Gesch?ftsf?hrung: Martina Koederitz (Vorsitzende), Reinhard Reschke, Dieter Scholz, Gregor Pillen, Ivo Koerner, Christian Noll Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 WEEE-Reg.-Nr. DE 99369940 From: Mengxing Cheng To: Eric Sperley Cc: gpfsug main discussion list Date: 03/09/2017 10:24 PM Subject: Re: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Eric, thank you very much for replying. Here is the memory configuration and current usage. Note that mmfsck is not running now. The two gss servers have the same 256GB memory and the service node has 128GB. 1. service node: total used free shared buffers cached Mem: 125 58 66 0 0 4 -/+ buffers/cache: 53 71 Swap: 7 0 7 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12990 root 0 -20 71.0g 43g 885m S 7.6 34.4 9306:00 mmfsd 2. gss nodes: ==================================== total used free shared buff/cache available Mem: 251 210 37 0 4 36 Swap: 3 0 3 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 36770 root 0 -20 0.216t 0.192t 2.667g S 48.9 78.1 75684:09 /usr/lpp/mmfs/bin/mmfsd The gss nodes' memory usage is so high because their pagepool is set to 192GB while the service node has 16GB pagepool. Mengxing --- Mengxing Cheng, Ph.D. HPC System Administrator Research Computing Center The University of Chicago 5607 S. Drexel Ave. Chicago, IL 60637 email: mxcheng at uchicago.edu phone: (773) 702-4070 ________________________________ From: Eric Sperley [esperle at us.ibm.com] Sent: Thursday, March 09, 2017 3:13 PM To: Mengxing Cheng Cc: gpfsug main discussion list Subject: Re: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem Mengxing, It is nice meeting you. I have seen a situation where the amount of RAM on a node can affect mmfsck times. Do all the nodes have the same amount of RAM, or does the slow running node have less RAM? Best Regards, Eric [cid:_4_0B8EBB580B8EB73C005D0FE8C12580DF] Eric Sperley SDI Architect "Carpe Diem" IBM Systems esperle at us.ibm.com +15033088721 [Inactive hide details for Mengxing Cheng ---03/09/2017 11:24:02 AM---Dear all, My name is Mengxing Cheng and I am a HPC system]Mengxing Cheng ---03/09/2017 11:24:02 AM---Dear all, My name is Mengxing Cheng and I am a HPC system administrator at the University of Chicago From: Mengxing Cheng To: "gpfsug-discuss at spectrumscale.org" Date: 03/09/2017 11:24 AM Subject: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Dear all, My name is Mengxing Cheng and I am a HPC system administrator at the University of Chicago. We have a GSS26 running gss2.5.10.3-3b and gpfs-4.2.0.3. Recently, we run mmfsck on a relatively small filesystem with 14TB block and 73863102 inodes but it was unusually slow so as to not be able to finish in 48 hours. In contrast, mmfsck run on a filesystem with the same size and inodes but sitting on a traditional IBM DS3512 storage took only 2 hours to complete. In particular, the mmfsck run in parallel using 3 nodes within the GSS storage cluster, we notice that one gss storage server scans inodes much slower than the other gss storage server and the quorum service node. Has anyone experience the same mmfsck performance issue? Could anyone make recommendation to troubleshoot and improve mmfsck performance? Thank you! Mengxing --- Mengxing Cheng, Ph.D. HPC System Administrator Research Computing Center The University of Chicago 5607 S. Drexel Ave. Chicago, IL 60637 email: mxcheng at uchicago.edu phone: (773) 702-4070 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00007.gif Type: image/gif Size: 7182 bytes Desc: ATT00007.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00008.png Type: image/png Size: 481 bytes Desc: ATT00008.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00009.gif Type: image/gif Size: 45 bytes Desc: ATT00009.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00010.png Type: image/png Size: 2322 bytes Desc: ATT00010.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00011.gif Type: image/gif Size: 105 bytes Desc: ATT00011.gif URL: From Kevin.Buterbaugh at Vanderbilt.Edu Fri Mar 10 20:43:37 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 10 Mar 2017 20:43:37 +0000 Subject: [gpfsug-discuss] mmcrfs issue Message-ID: <43922F46-548D-4D34-89D4-74D35C6EB447@vanderbilt.edu> Hi All, We are testing out some flash storage. I created a couple of NSDs successfully (?): root at nec:~/gpfs# mmlsnsd -F File system Disk name NSD servers --------------------------------------------------------------------------- (free disk) nsd1 nec (free disk) nsd2 nec root at nec:~/gpfs# So I tried to create a filesystem: root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 GPFS: 6027-441 Unable to open disk 'nsd2' on node nec. GPFS: 6027-441 Unable to open disk 'nsd1' on node nec. No such device No such device GPFS: 6027-538 Error accessing disks. mmcrfs: 6027-1200 tscrfs failed. Cannot create gpfs0 mmcrfs: 6027-1639 Command failed. Examine previous error messages to determine cause. root at nec:~/gpfs# Does this output from readdescraw look normal? root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd16 NSD descriptor in sector 64 of /dev/zd16 NSDid: 0A0023D258C1C02C format version: 1403 Label: Paxos sector: -1931478434 number of sectors: 8192 isPdisk: 0 Comment: NSD descriptor for Thu Mar 9 14:50:52 2017 No Disk descriptor in sector 96 of /dev/zd16 No FS descriptor in sector 2048 of /dev/zd16 root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd32 NSD descriptor in sector 64 of /dev/zd32 NSDid: 0A0023D258C1C02B format version: 1403 Label: Paxos sector: -1880562609 number of sectors: 8192 isPdisk: 0 Comment: NSD descriptor for Thu Mar 9 14:50:51 2017 No Disk descriptor in sector 96 of /dev/zd32 No FS descriptor in sector 2048 of /dev/zd32 root at nec:~/gpfs# Thanks in advance, all? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From valdis.kletnieks at vt.edu Fri Mar 10 20:54:42 2017 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Fri, 10 Mar 2017 15:54:42 -0500 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: <43922F46-548D-4D34-89D4-74D35C6EB447@vanderbilt.edu> References: <43922F46-548D-4D34-89D4-74D35C6EB447@vanderbilt.edu> Message-ID: <16136.1489179282@turing-police.cc.vt.edu> On Fri, 10 Mar 2017 20:43:37 +0000, "Buterbaugh, Kevin L" said: > So I tried to create a filesystem: > > root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 What was in flash.stanza? -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 484 bytes Desc: not available URL: From Kevin.Buterbaugh at Vanderbilt.Edu Fri Mar 10 20:59:11 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 10 Mar 2017 20:59:11 +0000 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: <16136.1489179282@turing-police.cc.vt.edu> References: <43922F46-548D-4D34-89D4-74D35C6EB447@vanderbilt.edu> <16136.1489179282@turing-police.cc.vt.edu> Message-ID: <9FEBE666-48CA-438B-B0AE-9B954491A9FF@vanderbilt.edu> %nsd: nsd=nsd1 usage=metadataOnly failureGroup=1 pool=system servers=nec device=/dev/zd32 %nsd: nsd=nsd2 usage=dataOnly failureGroup=1 pool=gpfsdata servers=nec device=/dev/zd16 %pool: pool=system blockSize=1M usage=metadataOnly layoutMap=scatter allowWriteAffinity=no %pool: pool=gpfsdata blockSize=1M usage=dataOnly layoutMap=scatter allowWriteAffinity=no > On Mar 10, 2017, at 2:54 PM, valdis.kletnieks at vt.edu wrote: > > On Fri, 10 Mar 2017 20:43:37 +0000, "Buterbaugh, Kevin L" said: > >> So I tried to create a filesystem: >> >> root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 > > What was in flash.stanza? > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Paul.Sanchez at deshaw.com Fri Mar 10 21:36:10 2017 From: Paul.Sanchez at deshaw.com (Sanchez, Paul) Date: Fri, 10 Mar 2017 21:36:10 +0000 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: <43922F46-548D-4D34-89D4-74D35C6EB447@vanderbilt.edu> References: <43922F46-548D-4D34-89D4-74D35C6EB447@vanderbilt.edu> Message-ID: <09e39c995afc410c822c6e66665cb98b@mbxtoa1.winmail.deshaw.com> See: https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm GPFS has a limited set of device search specs it uses to find connected NSDs. When using exotic devices, you need to whitelist the devices yourself using the user exit script at /var/mmfs/etc/nsddevices. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Buterbaugh, Kevin L Sent: Friday, March 10, 2017 3:44 PM To: gpfsug main discussion list Subject: [gpfsug-discuss] mmcrfs issue Hi All, We are testing out some flash storage. I created a couple of NSDs successfully (?): root at nec:~/gpfs# mmlsnsd -F File system Disk name NSD servers --------------------------------------------------------------------------- (free disk) nsd1 nec (free disk) nsd2 nec root at nec:~/gpfs# So I tried to create a filesystem: root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 GPFS: 6027-441 Unable to open disk 'nsd2' on node nec. GPFS: 6027-441 Unable to open disk 'nsd1' on node nec. No such device No such device GPFS: 6027-538 Error accessing disks. mmcrfs: 6027-1200 tscrfs failed. Cannot create gpfs0 mmcrfs: 6027-1639 Command failed. Examine previous error messages to determine cause. root at nec:~/gpfs# Does this output from readdescraw look normal? root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd16 NSD descriptor in sector 64 of /dev/zd16 NSDid: 0A0023D258C1C02C format version: 1403 Label: Paxos sector: -1931478434 number of sectors: 8192 isPdisk: 0 Comment: NSD descriptor for Thu Mar 9 14:50:52 2017 No Disk descriptor in sector 96 of /dev/zd16 No FS descriptor in sector 2048 of /dev/zd16 root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd32 NSD descriptor in sector 64 of /dev/zd32 NSDid: 0A0023D258C1C02B format version: 1403 Label: Paxos sector: -1880562609 number of sectors: 8192 isPdisk: 0 Comment: NSD descriptor for Thu Mar 9 14:50:51 2017 No Disk descriptor in sector 96 of /dev/zd32 No FS descriptor in sector 2048 of /dev/zd32 root at nec:~/gpfs# Thanks in advance, all? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Fri Mar 10 21:44:05 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 10 Mar 2017 16:44:05 -0500 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: <09e39c995afc410c822c6e66665cb98b@mbxtoa1.winmail.deshaw.com> References: <43922F46-548D-4D34-89D4-74D35C6EB447@vanderbilt.edu> <09e39c995afc410c822c6e66665cb98b@mbxtoa1.winmail.deshaw.com> Message-ID: <4b1fbfca-95c4-a15a-ef70-fcf3e20d936d@nasa.gov> Those look like zvol's. Out of curiosity have you set sync=always on the filesystem root or zvols themselves? It's my understanding that without that you risk data loss since GPFS won't ever cause a sync to be issued to the zvol for zfs to flush acknowledged but uncommitted writes. -Aaron On 3/10/17 4:36 PM, Sanchez, Paul wrote: > See: > https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm > > > > GPFS has a limited set of device search specs it uses to find connected > NSDs. When using exotic devices, you need to whitelist the devices > yourself using the user exit script at /var/mmfs/etc/nsddevices. > > > > *From:*gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of > *Buterbaugh, Kevin L > *Sent:* Friday, March 10, 2017 3:44 PM > *To:* gpfsug main discussion list > *Subject:* [gpfsug-discuss] mmcrfs issue > > > > Hi All, > > > > We are testing out some flash storage. I created a couple of NSDs > successfully (?): > > > > root at nec:~/gpfs# mmlsnsd -F > > > > File system Disk name NSD servers > > --------------------------------------------------------------------------- > > (free disk) nsd1 nec > > (free disk) nsd2 nec > > > > root at nec:~/gpfs# > > > > So I tried to create a filesystem: > > > > root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j > scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 > > GPFS: 6027-441 Unable to open disk 'nsd2' on node nec. > > GPFS: 6027-441 Unable to open disk 'nsd1' on node nec. > > No such device > > No such device > > GPFS: 6027-538 Error accessing disks. > > mmcrfs: 6027-1200 tscrfs failed. Cannot create gpfs0 > > mmcrfs: 6027-1639 Command failed. Examine previous error messages to > determine cause. > > root at nec:~/gpfs# > > > > Does this output from readdescraw look normal? > > > > root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd16 > > NSD descriptor in sector 64 of /dev/zd16 > > NSDid: 0A0023D258C1C02C format version: 1403 Label: > > Paxos sector: -1931478434 number of sectors: 8192 isPdisk: 0 > > Comment: NSD descriptor for Thu Mar 9 14:50:52 2017 > > No Disk descriptor in sector 96 of /dev/zd16 > > No FS descriptor in sector 2048 of /dev/zd16 > > root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd32 > > NSD descriptor in sector 64 of /dev/zd32 > > NSDid: 0A0023D258C1C02B format version: 1403 Label: > > Paxos sector: -1880562609 number of sectors: 8192 isPdisk: 0 > > Comment: NSD descriptor for Thu Mar 9 14:50:51 2017 > > No Disk descriptor in sector 96 of /dev/zd32 > > No FS descriptor in sector 2048 of /dev/zd32 > > root at nec:~/gpfs# > > > > Thanks in advance, all? > > > > Kevin > > ? > > Kevin Buterbaugh - Senior System Administrator > > Vanderbilt University - Advanced Computing Center for Research and Education > > Kevin.Buterbaugh at vanderbilt.edu > - (615)875-9633 > > > > > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From daniel.kidger at uk.ibm.com Sat Mar 11 10:37:47 2017 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Sat, 11 Mar 2017 10:37:47 +0000 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: <4b1fbfca-95c4-a15a-ef70-fcf3e20d936d@nasa.gov> Message-ID: On the subject of using zvols for software Raid/ replication, can ask as a quick poll, how many people are doing this? And any feedback on stability, tuning and performance? Daniel IBM Technical Presales > On 10 Mar 2017, at 22:44, Aaron Knister wrote: > > Those look like zvol's. Out of curiosity have you set sync=always on the > filesystem root or zvols themselves? It's my understanding that without > that you risk data loss since GPFS won't ever cause a sync to be issued > to the zvol for zfs to flush acknowledged but uncommitted writes. > > -Aaron > >> On 3/10/17 4:36 PM, Sanchez, Paul wrote: >> See: >> https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm >> >> >> >> GPFS has a limited set of device search specs it uses to find connected >> NSDs. When using exotic devices, you need to whitelist the devices >> yourself using the user exit script at /var/mmfs/etc/nsddevices. >> >> >> >> *From:*gpfsug-discuss-bounces at spectrumscale.org >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of >> *Buterbaugh, Kevin L >> *Sent:* Friday, March 10, 2017 3:44 PM >> *To:* gpfsug main discussion list >> *Subject:* [gpfsug-discuss] mmcrfs issue >> >> >> >> Hi All, >> >> >> >> We are testing out some flash storage. I created a couple of NSDs >> successfully (?): >> >> >> >> root at nec:~/gpfs# mmlsnsd -F >> >> >> >> File system Disk name NSD servers >> >> --------------------------------------------------------------------------- >> >> (free disk) nsd1 nec >> >> (free disk) nsd2 nec >> >> >> >> root at nec:~/gpfs# >> >> >> >> So I tried to create a filesystem: >> >> >> >> root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j >> scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 >> >> GPFS: 6027-441 Unable to open disk 'nsd2' on node nec. >> >> GPFS: 6027-441 Unable to open disk 'nsd1' on node nec. >> >> No such device >> >> No such device >> >> GPFS: 6027-538 Error accessing disks. >> >> mmcrfs: 6027-1200 tscrfs failed. Cannot create gpfs0 >> >> mmcrfs: 6027-1639 Command failed. Examine previous error messages to >> determine cause. >> >> root at nec:~/gpfs# >> >> >> >> Does this output from readdescraw look normal? >> >> >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd16 >> >> NSD descriptor in sector 64 of /dev/zd16 >> >> NSDid: 0A0023D258C1C02C format version: 1403 Label: >> >> Paxos sector: -1931478434 number of sectors: 8192 isPdisk: 0 >> >> Comment: NSD descriptor for Thu Mar 9 14:50:52 2017 >> >> No Disk descriptor in sector 96 of /dev/zd16 >> >> No FS descriptor in sector 2048 of /dev/zd16 >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd32 >> >> NSD descriptor in sector 64 of /dev/zd32 >> >> NSDid: 0A0023D258C1C02B format version: 1403 Label: >> >> Paxos sector: -1880562609 number of sectors: 8192 isPdisk: 0 >> >> Comment: NSD descriptor for Thu Mar 9 14:50:51 2017 >> >> No Disk descriptor in sector 96 of /dev/zd32 >> >> No FS descriptor in sector 2048 of /dev/zd32 >> >> root at nec:~/gpfs# >> >> >> >> Thanks in advance, all? >> >> >> >> Kevin >> >> ? >> >> Kevin Buterbaugh - Senior System Administrator >> >> Vanderbilt University - Advanced Computing Center for Research and Education >> >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss >Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Mon Mar 13 13:50:04 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 13 Mar 2017 13:50:04 +0000 Subject: [gpfsug-discuss] Registration Reminder: GPFSUG-USA Meeting at NERSC April 4-5th In-Reply-To: References: Message-ID: Here?s a reminder to register for the upcoming US user group meeting- Registration deadline is March 24th! Register here: https://www.nersc.gov/research-and-development/data-analytics/spectrum-user-group-meeting/ Bob Oesterlin Sr Principal Storage Engineer, Nuance -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Mon Mar 13 20:44:01 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Mon, 13 Mar 2017 20:44:01 +0000 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: References: Message-ID: <357B0FF5-4B7C-4941-8496-BA70D011A898@vanderbilt.edu> Hi All, Two things: 1) Paul?s suggestion to look at the nsddevices script was the answer I needed to fix my mmcrfs issue. Thanks. 2) I am also interested in hearing if anyone is using ZFS to create the equivalent of RAID 8+2P hardware RAID 6 LUNs and presenting that to GPFS to use as disks? Thanks? Kevin On Mar 11, 2017, at 4:37 AM, Daniel Kidger > wrote: On the subject of using zvols for software Raid/ replication, can ask as a quick poll, how many people are doing this? And any feedback on stability, tuning and performance? Daniel IBM Technical Presales > On 10 Mar 2017, at 22:44, Aaron Knister > wrote: > > Those look like zvol's. Out of curiosity have you set sync=always on the > filesystem root or zvols themselves? It's my understanding that without > that you risk data loss since GPFS won't ever cause a sync to be issued > to the zvol for zfs to flush acknowledged but uncommitted writes. > > -Aaron > >> On 3/10/17 4:36 PM, Sanchez, Paul wrote: >> See: >> https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm >> >> >> >> GPFS has a limited set of device search specs it uses to find connected >> NSDs. When using exotic devices, you need to whitelist the devices >> yourself using the user exit script at /var/mmfs/etc/nsddevices. >> >> >> >> *From:*gpfsug-discuss-bounces at spectrumscale.org >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of >> *Buterbaugh, Kevin L >> *Sent:* Friday, March 10, 2017 3:44 PM >> *To:* gpfsug main discussion list >> *Subject:* [gpfsug-discuss] mmcrfs issue >> >> >> >> Hi All, >> >> >> >> We are testing out some flash storage. I created a couple of NSDs >> successfully (?): >> >> >> >> root at nec:~/gpfs# mmlsnsd -F >> >> >> >> File system Disk name NSD servers >> >> --------------------------------------------------------------------------- >> >> (free disk) nsd1 nec >> >> (free disk) nsd2 nec >> >> >> >> root at nec:~/gpfs# >> >> >> >> So I tried to create a filesystem: >> >> >> >> root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j >> scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 >> >> GPFS: 6027-441 Unable to open disk 'nsd2' on node nec. >> >> GPFS: 6027-441 Unable to open disk 'nsd1' on node nec. >> >> No such device >> >> No such device >> >> GPFS: 6027-538 Error accessing disks. >> >> mmcrfs: 6027-1200 tscrfs failed. Cannot create gpfs0 >> >> mmcrfs: 6027-1639 Command failed. Examine previous error messages to >> determine cause. >> >> root at nec:~/gpfs# >> >> >> >> Does this output from readdescraw look normal? >> >> >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd16 >> >> NSD descriptor in sector 64 of /dev/zd16 >> >> NSDid: 0A0023D258C1C02C format version: 1403 Label: >> >> Paxos sector: -1931478434 number of sectors: 8192 isPdisk: 0 >> >> Comment: NSD descriptor for Thu Mar 9 14:50:52 2017 >> >> No Disk descriptor in sector 96 of /dev/zd16 >> >> No FS descriptor in sector 2048 of /dev/zd16 >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd32 >> >> NSD descriptor in sector 64 of /dev/zd32 >> >> NSDid: 0A0023D258C1C02B format version: 1403 Label: >> >> Paxos sector: -1880562609 number of sectors: 8192 isPdisk: 0 >> >> Comment: NSD descriptor for Thu Mar 9 14:50:51 2017 >> >> No Disk descriptor in sector 96 of /dev/zd32 >> >> No FS descriptor in sector 2048 of /dev/zd32 >> >> root at nec:~/gpfs# >> >> >> >> Thanks in advance, all? >> >> >> >> Kevin >> >> ? >> >> Kevin Buterbaugh - Senior System Administrator >> >> Vanderbilt University - Advanced Computing Center for Research and Education >> >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Tue Mar 14 00:06:29 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Mon, 13 Mar 2017 20:06:29 -0400 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: <357B0FF5-4B7C-4941-8496-BA70D011A898@vanderbilt.edu> References: <357B0FF5-4B7C-4941-8496-BA70D011A898@vanderbilt.edu> Message-ID: I was doing this in testing (with fantastic performance too) until I realized the issue with ZFS's behavior with direct io on zvols (e.g. not flushing a write to stable storage after acknowledging it to GPFS). After setting the sync=always parameter to not lose data in the event of a crash or power outage the write performance became unbearably slow (under 100MB/s of writes for an 8+2 RAIDZ2 if I recall correctly). I even tried adding a battery-backed PCIe write cache (http://www.netlist.com/products/vault-memory-storage/expressvault-pcIe-ev3/default.aspx) as a log device to the zpool but the performance was still really slow. I posted to the ZFS mailing list asking about how to optimize for a large block streaming workload but I didn't many bites (http://list.zfsonlinux.org/pipermail/zfs-discuss/2016-February/024851.html). I've got an RFE open with IBM (https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84994) to see if the behavior of GPFS could be changed such that it would issue explicit cache flushes that would allow it to work with ZFS (it might even be beneficial in FPO environments too). -Aaron On 3/13/17 4:44 PM, Buterbaugh, Kevin L wrote: > Hi All, > > Two things: > > 1) Paul?s suggestion to look at the nsddevices script was the answer I > needed to fix my mmcrfs issue. Thanks. > > 2) I am also interested in hearing if anyone is using ZFS to create the > equivalent of RAID 8+2P hardware RAID 6 LUNs and presenting that to GPFS > to use as disks? > > Thanks? > > Kevin > >> On Mar 11, 2017, at 4:37 AM, Daniel Kidger > > wrote: >> >> On the subject of using zvols for software Raid/ replication, can ask >> as a quick poll, how many people are doing this? >> >> And any feedback on stability, tuning and performance? >> >> Daniel >> IBM Technical Presales >> >> > On 10 Mar 2017, at 22:44, Aaron Knister > wrote: >> > >> > Those look like zvol's. Out of curiosity have you set sync=always on the >> > filesystem root or zvols themselves? It's my understanding that without >> > that you risk data loss since GPFS won't ever cause a sync to be issued >> > to the zvol for zfs to flush acknowledged but uncommitted writes. >> > >> > -Aaron >> > >> >> On 3/10/17 4:36 PM, Sanchez, Paul wrote: >> >> See: >> >> https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm >> >> >> >> >> >> >> >> GPFS has a limited set of device search specs it uses to find connected >> >> NSDs. When using exotic devices, you need to whitelist the devices >> >> yourself using the user exit script at /var/mmfs/etc/nsddevices. >> >> >> >> >> >> >> >> *From:*gpfsug-discuss-bounces at spectrumscale.org >> >> >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of >> >> *Buterbaugh, Kevin L >> >> *Sent:* Friday, March 10, 2017 3:44 PM >> >> *To:* gpfsug main discussion list >> >> *Subject:* [gpfsug-discuss] mmcrfs issue >> >> >> >> >> >> >> >> Hi All, >> >> >> >> >> >> >> >> We are testing out some flash storage. I created a couple of NSDs >> >> successfully (?): >> >> >> >> >> >> >> >> root at nec:~/gpfs# mmlsnsd -F >> >> >> >> >> >> >> >> File system Disk name NSD servers >> >> >> >> --------------------------------------------------------------------------- >> >> >> >> (free disk) nsd1 nec >> >> >> >> (free disk) nsd2 nec >> >> >> >> >> >> >> >> root at nec:~/gpfs# >> >> >> >> >> >> >> >> So I tried to create a filesystem: >> >> >> >> >> >> >> >> root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j >> >> scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 >> >> >> >> GPFS: 6027-441 Unable to open disk 'nsd2' on node nec. >> >> >> >> GPFS: 6027-441 Unable to open disk 'nsd1' on node nec. >> >> >> >> No such device >> >> >> >> No such device >> >> >> >> GPFS: 6027-538 Error accessing disks. >> >> >> >> mmcrfs: 6027-1200 tscrfs failed. Cannot create gpfs0 >> >> >> >> mmcrfs: 6027-1639 Command failed. Examine previous error messages to >> >> determine cause. >> >> >> >> root at nec:~/gpfs# >> >> >> >> >> >> >> >> Does this output from readdescraw look normal? >> >> >> >> >> >> >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd16 >> >> >> >> NSD descriptor in sector 64 of /dev/zd16 >> >> >> >> NSDid: 0A0023D258C1C02C format version: 1403 Label: >> >> >> >> Paxos sector: -1931478434 number of sectors: 8192 isPdisk: 0 >> >> >> >> Comment: NSD descriptor for Thu Mar 9 14:50:52 2017 >> >> >> >> No Disk descriptor in sector 96 of /dev/zd16 >> >> >> >> No FS descriptor in sector 2048 of /dev/zd16 >> >> >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd32 >> >> >> >> NSD descriptor in sector 64 of /dev/zd32 >> >> >> >> NSDid: 0A0023D258C1C02B format version: 1403 Label: >> >> >> >> Paxos sector: -1880562609 number of sectors: 8192 isPdisk: 0 >> >> >> >> Comment: NSD descriptor for Thu Mar 9 14:50:51 2017 >> >> >> >> No Disk descriptor in sector 96 of /dev/zd32 >> >> >> >> No FS descriptor in sector 2048 of /dev/zd32 >> >> >> >> root at nec:~/gpfs# >> >> >> >> >> >> >> >> Thanks in advance, all? >> >> >> >> >> >> >> >> Kevin >> >> >> >> ? >> >> >> >> Kevin Buterbaugh - Senior System Administrator >> >> >> >> Vanderbilt University - Advanced Computing Center for Research and Education >> >> >> >> Kevin.Buterbaugh at vanderbilt.edu >> >> - (615)875-9633 >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> gpfsug-discuss mailing list >> >> gpfsug-discuss at spectrumscale.org >> >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> > >> > -- >> > Aaron Knister >> > NASA Center for Climate Simulation (Code 606.2) >> > Goddard Space Flight Center >> > (301) 286-2776 >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > >> Unless stated otherwise above: >> IBM United Kingdom Limited - Registered in England and Wales with >> number 741598. >> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From valdis.kletnieks at vt.edu Tue Mar 14 18:15:17 2017 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Tue, 14 Mar 2017 14:15:17 -0400 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: References: <357B0FF5-4B7C-4941-8496-BA70D011A898@vanderbilt.edu> Message-ID: <16774.1489515317@turing-police.cc.vt.edu> On Mon, 13 Mar 2017 20:06:29 -0400, Aaron Knister said: > After setting the sync=always parameter to not lose data in the event of > a crash or power outage the write performance became unbearably slow > (under 100MB/s of writes for an 8+2 RAIDZ2 if I recall correctly). I Not at all surprising. Forcing a 'sync every write' has been painful for pretty much everything - first time I saw it was on a SunOS 3.2 box doing NFS over 3 decades ago. > I've got an RFE open with IBM > (https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84994) > to see if the behavior of GPFS could be changed such that it would issue > explicit cache flushes that would allow it to work with ZFS (it might > even be beneficial in FPO environments too). Do you have a suggestion for how to do this without it turning into 'sync=always' just done at a different layer in the stack? -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 484 bytes Desc: not available URL: From aaron.s.knister at nasa.gov Tue Mar 14 18:31:55 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 14 Mar 2017 14:31:55 -0400 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: <16774.1489515317@turing-police.cc.vt.edu> References: <357B0FF5-4B7C-4941-8496-BA70D011A898@vanderbilt.edu> <16774.1489515317@turing-police.cc.vt.edu> Message-ID: <411ef927-533e-c2b6-289f-44f0d8845c4f@nasa.gov> On 3/14/17 2:15 PM, valdis.kletnieks at vt.edu wrote: > On Mon, 13 Mar 2017 20:06:29 -0400, Aaron Knister said: >> After setting the sync=always parameter to not lose data in the event of >> a crash or power outage the write performance became unbearably slow >> (under 100MB/s of writes for an 8+2 RAIDZ2 if I recall correctly). I > > Not at all surprising. Forcing a 'sync every write' has been painful for pretty > much everything - first time I saw it was on a SunOS 3.2 box doing NFS over 3 > decades ago. > You know, what's interesting is when you do direct I/O to a linux md RAID device each acknowledged I/O to the md device results in SCSI SYNCHRONIZE CACHE commands being sent to the underlying member devices. I consider that doing a sync of sorts after each write and as long as everything is properly aligned you can get pretty fantastic performance from this. No data checksums, though :( >> I've got an RFE open with IBM >> (https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84994) >> to see if the behavior of GPFS could be changed such that it would issue >> explicit cache flushes that would allow it to work with ZFS (it might >> even be beneficial in FPO environments too). > > Do you have a suggestion for how to do this without it turning into 'sync=always' > just done at a different layer in the stack? > I don't think that GPFS would need to issue a sync after *every* write. I think the key is more that GPFS is aware of the state of acknowledged but uncommitted writes so that if there's something *really* important to be written (i.e. an internal piece of metadata) GPFS can force it to be committed to avoid GPFS itself becoming inconsistent. You don't want to have to mmfsck after a power outage/crash if there were lost writes that GPFS assumed were committed. -Aaron > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: OpenPGP digital signature URL: From luke.raimbach at googlemail.com Tue Mar 14 21:59:50 2017 From: luke.raimbach at googlemail.com (Luke Raimbach) Date: Tue, 14 Mar 2017 21:59:50 +0000 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: References: <357B0FF5-4B7C-4941-8496-BA70D011A898@vanderbilt.edu> Message-ID: Can I ask what the fascination with zvols is? Using a copy-on-write file system to underpin another block based file system seems counterintuitive. Perhaps I've missed something vital, in which case I'd be delighted to have my eyes opened! On Tue, 14 Mar 2017, 00:06 Aaron Knister, wrote: > I was doing this in testing (with fantastic performance too) until I > realized the issue with ZFS's behavior with direct io on zvols (e.g. not > flushing a write to stable storage after acknowledging it to GPFS). > After setting the sync=always parameter to not lose data in the event of > a crash or power outage the write performance became unbearably slow > (under 100MB/s of writes for an 8+2 RAIDZ2 if I recall correctly). I > even tried adding a battery-backed PCIe write cache > ( > http://www.netlist.com/products/vault-memory-storage/expressvault-pcIe-ev3/default.aspx > ) > as a log device to the zpool but the performance was still really slow. > I posted to the ZFS mailing list asking about how to optimize for a > large block streaming workload but I didn't many bites > ( > http://list.zfsonlinux.org/pipermail/zfs-discuss/2016-February/024851.html > ). > > I've got an RFE open with IBM > ( > https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84994 > ) > to see if the behavior of GPFS could be changed such that it would issue > explicit cache flushes that would allow it to work with ZFS (it might > even be beneficial in FPO environments too). > > -Aaron > > On 3/13/17 4:44 PM, Buterbaugh, Kevin L wrote: > > Hi All, > > > > Two things: > > > > 1) Paul?s suggestion to look at the nsddevices script was the answer I > > needed to fix my mmcrfs issue. Thanks. > > > > 2) I am also interested in hearing if anyone is using ZFS to create the > > equivalent of RAID 8+2P hardware RAID 6 LUNs and presenting that to GPFS > > to use as disks? > > > > Thanks? > > > > Kevin > > > >> On Mar 11, 2017, at 4:37 AM, Daniel Kidger >> > wrote: > >> > >> On the subject of using zvols for software Raid/ replication, can ask > >> as a quick poll, how many people are doing this? > >> > >> And any feedback on stability, tuning and performance? > >> > >> Daniel > >> IBM Technical Presales > >> > >> > On 10 Mar 2017, at 22:44, Aaron Knister > wrote: > >> > > >> > Those look like zvol's. Out of curiosity have you set sync=always on > the > >> > filesystem root or zvols themselves? It's my understanding that > without > >> > that you risk data loss since GPFS won't ever cause a sync to be > issued > >> > to the zvol for zfs to flush acknowledged but uncommitted writes. > >> > > >> > -Aaron > >> > > >> >> On 3/10/17 4:36 PM, Sanchez, Paul wrote: > >> >> See: > >> >> > https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm > >> >> > >> >> > >> >> > >> >> GPFS has a limited set of device search specs it uses to find > connected > >> >> NSDs. When using exotic devices, you need to whitelist the devices > >> >> yourself using the user exit script at /var/mmfs/etc/nsddevices. > >> >> > >> >> > >> >> > >> >> *From:*gpfsug-discuss-bounces at spectrumscale.org > >> > >> >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of > >> >> *Buterbaugh, Kevin L > >> >> *Sent:* Friday, March 10, 2017 3:44 PM > >> >> *To:* gpfsug main discussion list > >> >> *Subject:* [gpfsug-discuss] mmcrfs issue > >> >> > >> >> > >> >> > >> >> Hi All, > >> >> > >> >> > >> >> > >> >> We are testing out some flash storage. I created a couple of NSDs > >> >> successfully (?): > >> >> > >> >> > >> >> > >> >> root at nec:~/gpfs# mmlsnsd -F > >> >> > >> >> > >> >> > >> >> File system Disk name NSD servers > >> >> > >> >> > --------------------------------------------------------------------------- > >> >> > >> >> (free disk) nsd1 nec > >> >> > >> >> (free disk) nsd2 nec > >> >> > >> >> > >> >> > >> >> root at nec:~/gpfs# > >> >> > >> >> > >> >> > >> >> So I tried to create a filesystem: > >> >> > >> >> > >> >> > >> >> root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j > >> >> scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 > >> >> > >> >> GPFS: 6027-441 Unable to open disk 'nsd2' on node nec. > >> >> > >> >> GPFS: 6027-441 Unable to open disk 'nsd1' on node nec. > >> >> > >> >> No such device > >> >> > >> >> No such device > >> >> > >> >> GPFS: 6027-538 Error accessing disks. > >> >> > >> >> mmcrfs: 6027-1200 tscrfs failed. Cannot create gpfs0 > >> >> > >> >> mmcrfs: 6027-1639 Command failed. Examine previous error messages to > >> >> determine cause. > >> >> > >> >> root at nec:~/gpfs# > >> >> > >> >> > >> >> > >> >> Does this output from readdescraw look normal? > >> >> > >> >> > >> >> > >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd16 > >> >> > >> >> NSD descriptor in sector 64 of /dev/zd16 > >> >> > >> >> NSDid: 0A0023D258C1C02C format version: 1403 Label: > >> >> > >> >> Paxos sector: -1931478434 number of sectors: 8192 isPdisk: 0 > >> >> > >> >> Comment: NSD descriptor for Thu Mar 9 14:50:52 2017 > >> >> > >> >> No Disk descriptor in sector 96 of /dev/zd16 > >> >> > >> >> No FS descriptor in sector 2048 of /dev/zd16 > >> >> > >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd32 > >> >> > >> >> NSD descriptor in sector 64 of /dev/zd32 > >> >> > >> >> NSDid: 0A0023D258C1C02B format version: 1403 Label: > >> >> > >> >> Paxos sector: -1880562609 number of sectors: 8192 isPdisk: 0 > >> >> > >> >> Comment: NSD descriptor for Thu Mar 9 14:50:51 2017 > >> >> > >> >> No Disk descriptor in sector 96 of /dev/zd32 > >> >> > >> >> No FS descriptor in sector 2048 of /dev/zd32 > >> >> > >> >> root at nec:~/gpfs# > >> >> > >> >> > >> >> > >> >> Thanks in advance, all? > >> >> > >> >> > >> >> > >> >> Kevin > >> >> > >> >> ? > >> >> > >> >> Kevin Buterbaugh - Senior System Administrator > >> >> > >> >> Vanderbilt University - Advanced Computing Center for Research and > Education > >> >> > >> >> Kevin.Buterbaugh at vanderbilt.edu Kevin.Buterbaugh at vanderbilt.edu> > >> >> - (615)875-9633 > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> _______________________________________________ > >> >> gpfsug-discuss mailing list > >> >> gpfsug-discuss at spectrumscale.org > >> >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> >> > >> > > >> > -- > >> > Aaron Knister > >> > NASA Center for Climate Simulation (Code 606.2) > >> > Goddard Space Flight Center > >> > (301) 286-2776 > >> > _______________________________________________ > >> > gpfsug-discuss mailing list > >> > gpfsug-discuss at spectrumscale.org > >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > >> Unless stated otherwise above: > >> IBM United Kingdom Limited - Registered in England and Wales with > >> number 741598. > >> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 > 3AU > >> > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Tue Mar 14 22:16:43 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 14 Mar 2017 18:16:43 -0400 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: References: <357B0FF5-4B7C-4941-8496-BA70D011A898@vanderbilt.edu> Message-ID: <2d4fa452-3137-0a44-fd18-bed4b9923279@nasa.gov> For me it's the protection against bitrot and added protection against silent data corruption and in theory the write caching offered by adding log devices that could help with small random writes (although there are other problems with ZFS + synchronous workloads that stop this from actually materializing). -Aaron On 3/14/17 5:59 PM, Luke Raimbach wrote: > Can I ask what the fascination with zvols is? Using a copy-on-write file > system to underpin another block based file system seems > counterintuitive. Perhaps I've missed something vital, in which case I'd > be delighted to have my eyes opened! > > On Tue, 14 Mar 2017, 00:06 Aaron Knister, > wrote: > > I was doing this in testing (with fantastic performance too) until I > realized the issue with ZFS's behavior with direct io on zvols (e.g. not > flushing a write to stable storage after acknowledging it to GPFS). > After setting the sync=always parameter to not lose data in the event of > a crash or power outage the write performance became unbearably slow > (under 100MB/s of writes for an 8+2 RAIDZ2 if I recall correctly). I > even tried adding a battery-backed PCIe write cache > (http://www.netlist.com/products/vault-memory-storage/expressvault-pcIe-ev3/default.aspx) > as a log device to the zpool but the performance was still really slow. > I posted to the ZFS mailing list asking about how to optimize for a > large block streaming workload but I didn't many bites > (http://list.zfsonlinux.org/pipermail/zfs-discuss/2016-February/024851.html). > > I've got an RFE open with IBM > (https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84994) > to see if the behavior of GPFS could be changed such that it would issue > explicit cache flushes that would allow it to work with ZFS (it might > even be beneficial in FPO environments too). > > -Aaron > > On 3/13/17 4:44 PM, Buterbaugh, Kevin L wrote: > > Hi All, > > > > Two things: > > > > 1) Paul?s suggestion to look at the nsddevices script was the answer I > > needed to fix my mmcrfs issue. Thanks. > > > > 2) I am also interested in hearing if anyone is using ZFS to create the > > equivalent of RAID 8+2P hardware RAID 6 LUNs and presenting that to GPFS > > to use as disks? > > > > Thanks? > > > > Kevin > > > >> On Mar 11, 2017, at 4:37 AM, Daniel Kidger > >> >> wrote: > >> > >> On the subject of using zvols for software Raid/ replication, can ask > >> as a quick poll, how many people are doing this? > >> > >> And any feedback on stability, tuning and performance? > >> > >> Daniel > >> IBM Technical Presales > >> > >> > On 10 Mar 2017, at 22:44, Aaron Knister > >> > wrote: > >> > > >> > Those look like zvol's. Out of curiosity have you set sync=always on the > >> > filesystem root or zvols themselves? It's my understanding that without > >> > that you risk data loss since GPFS won't ever cause a sync to be issued > >> > to the zvol for zfs to flush acknowledged but uncommitted writes. > >> > > >> > -Aaron > >> > > >> >> On 3/10/17 4:36 PM, Sanchez, Paul wrote: > >> >> See: > >> >> https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm > >> >> > >> >> > >> >> > >> >> GPFS has a limited set of device search specs it uses to find connected > >> >> NSDs. When using exotic devices, you need to whitelist the devices > >> >> yourself using the user exit script at /var/mmfs/etc/nsddevices. > >> >> > >> >> > >> >> > >> >> *From:*gpfsug-discuss-bounces at spectrumscale.org > > >> > > >> >> [mailto:gpfsug-discuss-bounces at spectrumscale.org > ] *On Behalf Of > >> >> *Buterbaugh, Kevin L > >> >> *Sent:* Friday, March 10, 2017 3:44 PM > >> >> *To:* gpfsug main discussion list > >> >> *Subject:* [gpfsug-discuss] mmcrfs issue > >> >> > >> >> > >> >> > >> >> Hi All, > >> >> > >> >> > >> >> > >> >> We are testing out some flash storage. I created a couple of NSDs > >> >> successfully (?): > >> >> > >> >> > >> >> > >> >> root at nec:~/gpfs# mmlsnsd -F > >> >> > >> >> > >> >> > >> >> File system Disk name NSD servers > >> >> > >> >> --------------------------------------------------------------------------- > >> >> > >> >> (free disk) nsd1 nec > >> >> > >> >> (free disk) nsd2 nec > >> >> > >> >> > >> >> > >> >> root at nec:~/gpfs# > >> >> > >> >> > >> >> > >> >> So I tried to create a filesystem: > >> >> > >> >> > >> >> > >> >> root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j > >> >> scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 > >> >> > >> >> GPFS: 6027-441 Unable to open disk 'nsd2' on node nec. > >> >> > >> >> GPFS: 6027-441 Unable to open disk 'nsd1' on node nec. > >> >> > >> >> No such device > >> >> > >> >> No such device > >> >> > >> >> GPFS: 6027-538 Error accessing disks. > >> >> > >> >> mmcrfs: 6027-1200 tscrfs failed. Cannot create gpfs0 > >> >> > >> >> mmcrfs: 6027-1639 Command failed. Examine previous error messages to > >> >> determine cause. > >> >> > >> >> root at nec:~/gpfs# > >> >> > >> >> > >> >> > >> >> Does this output from readdescraw look normal? > >> >> > >> >> > >> >> > >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd16 > >> >> > >> >> NSD descriptor in sector 64 of /dev/zd16 > >> >> > >> >> NSDid: 0A0023D258C1C02C format version: 1403 Label: > >> >> > >> >> Paxos sector: -1931478434 number of sectors: 8192 isPdisk: 0 > >> >> > >> >> Comment: NSD descriptor for Thu Mar 9 14:50:52 2017 > >> >> > >> >> No Disk descriptor in sector 96 of /dev/zd16 > >> >> > >> >> No FS descriptor in sector 2048 of /dev/zd16 > >> >> > >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd32 > >> >> > >> >> NSD descriptor in sector 64 of /dev/zd32 > >> >> > >> >> NSDid: 0A0023D258C1C02B format version: 1403 Label: > >> >> > >> >> Paxos sector: -1880562609 number of sectors: 8192 isPdisk: 0 > >> >> > >> >> Comment: NSD descriptor for Thu Mar 9 14:50:51 2017 > >> >> > >> >> No Disk descriptor in sector 96 of /dev/zd32 > >> >> > >> >> No FS descriptor in sector 2048 of /dev/zd32 > >> >> > >> >> root at nec:~/gpfs# > >> >> > >> >> > >> >> > >> >> Thanks in advance, all? > >> >> > >> >> > >> >> > >> >> Kevin > >> >> > >> >> ? > >> >> > >> >> Kevin Buterbaugh - Senior System Administrator > >> >> > >> >> Vanderbilt University - Advanced Computing Center for Research and Education > >> >> > >> >> Kevin.Buterbaugh at vanderbilt.edu > > > > >> >> > - (615)875-9633 > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> _______________________________________________ > >> >> gpfsug-discuss mailing list > >> >> gpfsug-discuss at spectrumscale.org > >> >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> >> > >> > > >> > -- > >> > Aaron Knister > >> > NASA Center for Climate Simulation (Code 606.2) > >> > Goddard Space Flight Center > >> > (301) 286-2776 > >> > _______________________________________________ > >> > gpfsug-discuss mailing list > >> > gpfsug-discuss at spectrumscale.org > >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > >> Unless stated otherwise above: > >> IBM United Kingdom Limited - Registered in England and Wales with > >> number 741598. > >> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > >> > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From xhejtman at ics.muni.cz Wed Mar 15 09:50:31 2017 From: xhejtman at ics.muni.cz (Lukas Hejtmanek) Date: Wed, 15 Mar 2017 10:50:31 +0100 Subject: [gpfsug-discuss] default inode size Message-ID: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> Hello, does anyone know why with GPFS 4.x, there is inode size 4096B by default. Is there any problem with, e.g., only 1024B inode size? -- Luk?? Hejtm?nek From ulmer at ulmer.org Wed Mar 15 12:22:22 2017 From: ulmer at ulmer.org (Stephen Ulmer) Date: Wed, 15 Mar 2017 08:22:22 -0400 Subject: [gpfsug-discuss] default inode size In-Reply-To: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> References: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> Message-ID: <0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> You need 4K nodes to store encryption keys. You can also put other useful things in there, like extended attributes and (possibly) the entire file. Are you worried about wasting space? -- Stephen > On Mar 15, 2017, at 5:50 AM, Lukas Hejtmanek > wrote: > > Hello, > > does anyone know why with GPFS 4.x, there is inode size 4096B by default. > > Is there any problem with, e.g., only 1024B inode size? > > -- > Luk?? Hejtm?nek > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From xhejtman at ics.muni.cz Wed Mar 15 12:37:00 2017 From: xhejtman at ics.muni.cz (Lukas Hejtmanek) Date: Wed, 15 Mar 2017 13:37:00 +0100 Subject: [gpfsug-discuss] default inode size In-Reply-To: <0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> References: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> <0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> Message-ID: <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> On Wed, Mar 15, 2017 at 08:22:22AM -0400, Stephen Ulmer wrote: > You need 4K nodes to store encryption keys. You can also put other useful things in there, like extended attributes and (possibly) the entire file. > > Are you worried about wasting space? well, I have 1PB of capacity for data disks and 3TB of capacity for metadata (SSD). With 512B inodes, this ration seemed to be quite ok, while with 4k inodes, I run out of free space on SSD pretty fast. So I'm thinking about re-creating file system with smaller inode size. I don't think I will ever need encryption keys. -- Luk?? Hejtm?nek From luis.bolinches at fi.ibm.com Wed Mar 15 13:09:07 2017 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Wed, 15 Mar 2017 13:09:07 +0000 Subject: [gpfsug-discuss] default inode size In-Reply-To: <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> References: <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz>, <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz><0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> Message-ID: An HTML attachment was scrubbed... URL: From dieter.gorecki at atos.net Wed Mar 15 13:12:44 2017 From: dieter.gorecki at atos.net (GORECKI, DIETER) Date: Wed, 15 Mar 2017 13:12:44 +0000 Subject: [gpfsug-discuss] default inode size In-Reply-To: <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> References: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> <0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> Message-ID: Lukas, One other thing to consider is the storage of data inside the inode itself for very small files. GPFS has the ability to use the remaining [kilo]bytes of the inode to store the data of the file whenever the file is small enough to fit in. Anyone correct me if I am wrong, but with 4k inodes, you can store up to (4096-128 header) 3968 bytes of data. (without ILM) So regarding the size of the files you intend to store into your filesystem, it might be very interesting to take advantage of the performance of your SSD's to store small files. Regards, Dieter -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Lukas Hejtmanek Sent: Wednesday, March 15, 2017 1:37 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] default inode size On Wed, Mar 15, 2017 at 08:22:22AM -0400, Stephen Ulmer wrote: > You need 4K nodes to store encryption keys. You can also put other useful things in there, like extended attributes and (possibly) the entire file. > > Are you worried about wasting space? well, I have 1PB of capacity for data disks and 3TB of capacity for metadata (SSD). With 512B inodes, this ration seemed to be quite ok, while with 4k inodes, I run out of free space on SSD pretty fast. So I'm thinking about re-creating file system with smaller inode size. I don't think I will ever need encryption keys. -- Luk?? Hejtm?nek _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From xhejtman at ics.muni.cz Wed Mar 15 13:21:40 2017 From: xhejtman at ics.muni.cz (Lukas Hejtmanek) Date: Wed, 15 Mar 2017 14:21:40 +0100 Subject: [gpfsug-discuss] default inode size In-Reply-To: References: <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> <0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> Message-ID: <20170315132140.v5rfagqy4hqzqsfh@ics.muni.cz> Hello, On Wed, Mar 15, 2017 at 01:09:07PM +0000, Luis Bolinches wrote: > My 2 cents. Before even thinking too much on that path I would check the > following > ? > - What the is physical size on those SSD if they are already 4K you won't > "save" anything size of what? > - Do you use small (>3.5Kb) files? If so I would still keep 4K inodes > - Check if 512 can hold NSDv2 format, from top of my head it cannot but > pls check. If so, do you want to use a format that anyway is going to fade > away and lose the goodies that format brings? > ? > I am sure few other pros (and cons) can be put on but I would start with > those 3 I was thinking about 1024B inodes. Is this still not enough for NSDv2 format? -- Luk?? Hejtm?nek From luis.bolinches at fi.ibm.com Wed Mar 15 13:24:11 2017 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Wed, 15 Mar 2017 13:24:11 +0000 Subject: [gpfsug-discuss] default inode size In-Reply-To: <20170315132140.v5rfagqy4hqzqsfh@ics.muni.cz> References: <20170315132140.v5rfagqy4hqzqsfh@ics.muni.cz>, <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz><20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz><0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> Message-ID: An HTML attachment was scrubbed... URL: From xhejtman at ics.muni.cz Wed Mar 15 13:26:09 2017 From: xhejtman at ics.muni.cz (Lukas Hejtmanek) Date: Wed, 15 Mar 2017 14:26:09 +0100 Subject: [gpfsug-discuss] default inode size In-Reply-To: References: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> <0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> Message-ID: <20170315132609.wjplicnvhizlwc3q@ics.muni.cz> On Wed, Mar 15, 2017 at 01:12:44PM +0000, GORECKI, DIETER wrote: > One other thing to consider is the storage of data inside the inode itself for very small files. GPFS has the ability to use the remaining [kilo]bytes of the inode to store the data of the file whenever the file is small enough to fit in. > > Anyone correct me if I am wrong, but with 4k inodes, you can store up to (4096-128 header) 3968 bytes of data. (without ILM) > > So regarding the size of the files you intend to store into your filesystem, it might be very interesting to take advantage of the performance of your SSD's to store small files. I agree it would, however, I have 30 % free capacity on SSDs only right now (with some snapshots and ~70M files). So I'm afraid I have to either change the rotational disks to hold metadata as well or do not create more files/snapshots or decrease the inode size. I cannot add more SSDs as I do not have free disk slots in HW. -- Luk?? Hejtm?nek From ulmer at ulmer.org Wed Mar 15 13:36:24 2017 From: ulmer at ulmer.org (Stephen Ulmer) Date: Wed, 15 Mar 2017 09:36:24 -0400 Subject: [gpfsug-discuss] default inode size In-Reply-To: <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> References: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> <0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> Message-ID: > On Mar 15, 2017, at 8:37 AM, Lukas Hejtmanek wrote: > > On Wed, Mar 15, 2017 at 08:22:22AM -0400, Stephen Ulmer wrote: >> You need 4K nodes to store encryption keys. You can also put other useful things in there, like extended attributes and (possibly) the entire file. >> >> Are you worried about wasting space? > > well, I have 1PB of capacity for data disks and 3TB of capacity for metadata > (SSD). > > With 512B inodes, this ration seemed to be quite ok, while with 4k inodes, > I run out of free space on SSD pretty fast. So I'm thinking about re-creating > file system with smaller inode size. > > I don't think I will ever need encryption keys. What?s the average file size? Every file that is less than *about* 3.5K will wind up in the (4k) inode itself. Also check the HW sector size of the SSDs (many are 4K now). If that?s the case, you?ll actually be much more efficient from an access point of view with 4K inodes. If you don? t have 1PB of data yet (or won?t in the next year or so), you can always expand the system pool with more SSDs in the future. It?s not scary to pick another size ? it works just fine, but you?re about to create a filesystem that will be a pain in the rear to change. It will take a long time to move/backfill even a few hundred TB of data if you change you mind later. You?ll definitely give up (possibly future) features by choosing not 4K, but you?ve got the flexibility to do so. :) Liberty, -- Stephen From xhejtman at ics.muni.cz Wed Mar 15 13:46:54 2017 From: xhejtman at ics.muni.cz (Lukas Hejtmanek) Date: Wed, 15 Mar 2017 14:46:54 +0100 Subject: [gpfsug-discuss] default inode size In-Reply-To: References: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> <0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> Message-ID: <20170315134654.335p4rsrnaxgrjrq@ics.muni.cz> On Wed, Mar 15, 2017 at 09:36:24AM -0400, Stephen Ulmer wrote: > What?s the average file size? Every file that is less than *about* 3.5K will wind up in the (4k) inode itself. average file size is about 4MB. > Also check the HW sector size of the SSDs (many are 4K now). If that?s the case, you?ll actually be much more efficient from an access point of view with 4K inodes. HW sector size reported by disk array is 512B. SSD itself uses 4k though, however, Linux thinks it is only 512B. > If you don? t have 1PB of data yet (or won?t in the next year or so), you can always expand the system pool with more SSDs in the future. I cannot. I don't have free HW slots. > It?s not scary to pick another size ? it works just fine, but you?re about to create a filesystem that will be a pain in the rear to change. It will take a long time to move/backfill even a few hundred TB of data if you change you mind later. You?ll definitely give up (possibly future) features by choosing not 4K, but you?ve got the flexibility to do so. :) But at least I will have space to store metadata :) -- Luk?? Hejtm?nek From aaron.s.knister at nasa.gov Wed Mar 15 13:59:52 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Wed, 15 Mar 2017 09:59:52 -0400 Subject: [gpfsug-discuss] default inode size In-Reply-To: <20170315134654.335p4rsrnaxgrjrq@ics.muni.cz> References: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> <0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> <20170315134654.335p4rsrnaxgrjrq@ics.muni.cz> Message-ID: <4bb5ec26-bf98-76b4-5399-0d315b502458@nasa.gov> I'm not sure because I've not tried it but if you chose an inode size less than 4K does the FS still show as being 4K aligned? If it doesn't this could prevent you from expanding the metadata capacity of the FS in the future because in a non 4k aligned fs GPFS won't let you add 4K sector size LUNS that contain metadata. -Aaron On 3/15/17 9:46 AM, Lukas Hejtmanek wrote: > On Wed, Mar 15, 2017 at 09:36:24AM -0400, Stephen Ulmer wrote: >> What?s the average file size? Every file that is less than *about* 3.5K will wind up in the (4k) inode itself. > > average file size is about 4MB. > >> Also check the HW sector size of the SSDs (many are 4K now). If that?s the case, you?ll actually be much more efficient from an access point of view with 4K inodes. > > HW sector size reported by disk array is 512B. SSD itself uses 4k though, > however, Linux thinks it is only 512B. > >> If you don? t have 1PB of data yet (or won?t in the next year or so), you can always expand the system pool with more SSDs in the future. > > I cannot. I don't have free HW slots. > >> It?s not scary to pick another size ? it works just fine, but you?re about to create a filesystem that will be a pain in the rear to change. It will take a long time to move/backfill even a few hundred TB of data if you change you mind later. You?ll definitely give up (possibly future) features by choosing not 4K, but you?ve got the flexibility to do so. :) > > But at least I will have space to store metadata :) > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From janfrode at tanso.net Wed Mar 15 14:22:36 2017 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Wed, 15 Mar 2017 15:22:36 +0100 Subject: [gpfsug-discuss] default inode size In-Reply-To: <20170315132609.wjplicnvhizlwc3q@ics.muni.cz> References: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz> <0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org> <20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> <20170315132609.wjplicnvhizlwc3q@ics.muni.cz> Message-ID: Another thing to consider is how many disk block pointers you have room for in the inode, and when you'll need to add additional indirect blocks. Ref: http://files.gpfsug.org/presentations/2016/south-bank/ D2_P2_A_spectrum_scale_metadata_dark_V2a.pdf If I understand that presentation correctly.. a block pointer is 12 bytes, so a 1 KB inode can hold approximately 70 block pointers -- ie. any file larger than 70 blocks will need additional indirect blocks to hold the addresses, requiring minimum one additional sub-block. With a 1 MB blocksize filesystem (data and metadata), files between 75 MB and 3 GB will need 1x inode and 1x indirect block = 1 KB + 32 KB metadata space. If your files are smaller than 75 MB, the address pointers fit in the 1K inode. If you have 4 KB inode, files smaller than 340 MB will fit all pointers in the inode. -jf On Wed, Mar 15, 2017 at 2:26 PM, Lukas Hejtmanek wrote: > On Wed, Mar 15, 2017 at 01:12:44PM +0000, GORECKI, DIETER wrote: > > One other thing to consider is the storage of data inside the inode > itself for very small files. GPFS has the ability to use the remaining > [kilo]bytes of the inode to store the data of the file whenever the file is > small enough to fit in. > > > > Anyone correct me if I am wrong, but with 4k inodes, you can store up to > (4096-128 header) 3968 bytes of data. (without ILM) > > > > So regarding the size of the files you intend to store into your > filesystem, it might be very interesting to take advantage of the > performance of your SSD's to store small files. > > I agree it would, however, I have 30 % free capacity on SSDs only right > now (with > some snapshots and ~70M files). So I'm afraid I have to either change the > rotational disks to hold metadata as well or do not create more > files/snapshots or decrease the inode size. I cannot add more SSDs as I do > not > have free disk slots in HW. > > -- > Luk?? Hejtm?nek > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Wed Mar 15 14:25:41 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Wed, 15 Mar 2017 14:25:41 +0000 Subject: [gpfsug-discuss] mmcrfs issue In-Reply-To: <2d4fa452-3137-0a44-fd18-bed4b9923279@nasa.gov> References: <357B0FF5-4B7C-4941-8496-BA70D011A898@vanderbilt.edu> <2d4fa452-3137-0a44-fd18-bed4b9923279@nasa.gov> Message-ID: Hi All, Since I started this thread I guess I should chime in, too ? for us it was simply that we were testing a device that did not have hardware RAID controllers and we were wanting to implement something roughly equivalent to RAID 6 LUNs. Kevin > On Mar 14, 2017, at 5:16 PM, Aaron Knister wrote: > > For me it's the protection against bitrot and added protection against silent data corruption and in theory the write caching offered by adding log devices that could help with small random writes (although there are other problems with ZFS + synchronous workloads that stop this from actually materializing). > > -Aaron > > On 3/14/17 5:59 PM, Luke Raimbach wrote: >> Can I ask what the fascination with zvols is? Using a copy-on-write file >> system to underpin another block based file system seems >> counterintuitive. Perhaps I've missed something vital, in which case I'd >> be delighted to have my eyes opened! >> >> On Tue, 14 Mar 2017, 00:06 Aaron Knister, > > wrote: >> >> I was doing this in testing (with fantastic performance too) until I >> realized the issue with ZFS's behavior with direct io on zvols (e.g. not >> flushing a write to stable storage after acknowledging it to GPFS). >> After setting the sync=always parameter to not lose data in the event of >> a crash or power outage the write performance became unbearably slow >> (under 100MB/s of writes for an 8+2 RAIDZ2 if I recall correctly). I >> even tried adding a battery-backed PCIe write cache >> (http://www.netlist.com/products/vault-memory-storage/expressvault-pcIe-ev3/default.aspx) >> as a log device to the zpool but the performance was still really slow. >> I posted to the ZFS mailing list asking about how to optimize for a >> large block streaming workload but I didn't many bites >> (http://list.zfsonlinux.org/pipermail/zfs-discuss/2016-February/024851.html). >> >> I've got an RFE open with IBM >> (https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84994) >> to see if the behavior of GPFS could be changed such that it would issue >> explicit cache flushes that would allow it to work with ZFS (it might >> even be beneficial in FPO environments too). >> >> -Aaron >> >> On 3/13/17 4:44 PM, Buterbaugh, Kevin L wrote: >> > Hi All, >> > >> > Two things: >> > >> > 1) Paul?s suggestion to look at the nsddevices script was the answer I >> > needed to fix my mmcrfs issue. Thanks. >> > >> > 2) I am also interested in hearing if anyone is using ZFS to create the >> > equivalent of RAID 8+2P hardware RAID 6 LUNs and presenting that to GPFS >> > to use as disks? >> > >> > Thanks? >> > >> > Kevin >> > >> >> On Mar 11, 2017, at 4:37 AM, Daniel Kidger >> >> >> wrote: >> >> >> >> On the subject of using zvols for software Raid/ replication, can ask >> >> as a quick poll, how many people are doing this? >> >> >> >> And any feedback on stability, tuning and performance? >> >> >> >> Daniel >> >> IBM Technical Presales >> >> >> >> > On 10 Mar 2017, at 22:44, Aaron Knister >> >> >> wrote: >> >> > >> >> > Those look like zvol's. Out of curiosity have you set sync=always on the >> >> > filesystem root or zvols themselves? It's my understanding that without >> >> > that you risk data loss since GPFS won't ever cause a sync to be issued >> >> > to the zvol for zfs to flush acknowledged but uncommitted writes. >> >> > >> >> > -Aaron >> >> > >> >> >> On 3/10/17 4:36 PM, Sanchez, Paul wrote: >> >> >> See: >> >> >> https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm >> >> >> >> >> >> >> >> >> >> >> >> GPFS has a limited set of device search specs it uses to find connected >> >> >> NSDs. When using exotic devices, you need to whitelist the devices >> >> >> yourself using the user exit script at /var/mmfs/etc/nsddevices. >> >> >> >> >> >> >> >> >> >> >> >> *From:*gpfsug-discuss-bounces at spectrumscale.org >> >> >> > > >> >> >> [mailto:gpfsug-discuss-bounces at spectrumscale.org >> ] *On Behalf Of >> >> >> *Buterbaugh, Kevin L >> >> >> *Sent:* Friday, March 10, 2017 3:44 PM >> >> >> *To:* gpfsug main discussion list >> >> >> *Subject:* [gpfsug-discuss] mmcrfs issue >> >> >> >> >> >> >> >> >> >> >> >> Hi All, >> >> >> >> >> >> >> >> >> >> >> >> We are testing out some flash storage. I created a couple of NSDs >> >> >> successfully (?): >> >> >> >> >> >> >> >> >> >> >> >> root at nec:~/gpfs# mmlsnsd -F >> >> >> >> >> >> >> >> >> >> >> >> File system Disk name NSD servers >> >> >> >> >> >> --------------------------------------------------------------------------- >> >> >> >> >> >> (free disk) nsd1 nec >> >> >> >> >> >> (free disk) nsd2 nec >> >> >> >> >> >> >> >> >> >> >> >> root at nec:~/gpfs# >> >> >> >> >> >> >> >> >> >> >> >> So I tried to create a filesystem: >> >> >> >> >> >> >> >> >> >> >> >> root at nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j >> >> >> scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0 >> >> >> >> >> >> GPFS: 6027-441 Unable to open disk 'nsd2' on node nec. >> >> >> >> >> >> GPFS: 6027-441 Unable to open disk 'nsd1' on node nec. >> >> >> >> >> >> No such device >> >> >> >> >> >> No such device >> >> >> >> >> >> GPFS: 6027-538 Error accessing disks. >> >> >> >> >> >> mmcrfs: 6027-1200 tscrfs failed. Cannot create gpfs0 >> >> >> >> >> >> mmcrfs: 6027-1639 Command failed. Examine previous error messages to >> >> >> determine cause. >> >> >> >> >> >> root at nec:~/gpfs# >> >> >> >> >> >> >> >> >> >> >> >> Does this output from readdescraw look normal? >> >> >> >> >> >> >> >> >> >> >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd16 >> >> >> >> >> >> NSD descriptor in sector 64 of /dev/zd16 >> >> >> >> >> >> NSDid: 0A0023D258C1C02C format version: 1403 Label: >> >> >> >> >> >> Paxos sector: -1931478434 number of sectors: 8192 isPdisk: 0 >> >> >> >> >> >> Comment: NSD descriptor for Thu Mar 9 14:50:52 2017 >> >> >> >> >> >> No Disk descriptor in sector 96 of /dev/zd16 >> >> >> >> >> >> No FS descriptor in sector 2048 of /dev/zd16 >> >> >> >> >> >> root at nec:~/gpfs# mmfsadm test readdescraw /dev/zd32 >> >> >> >> >> >> NSD descriptor in sector 64 of /dev/zd32 >> >> >> >> >> >> NSDid: 0A0023D258C1C02B format version: 1403 Label: >> >> >> >> >> >> Paxos sector: -1880562609 number of sectors: 8192 isPdisk: 0 >> >> >> >> >> >> Comment: NSD descriptor for Thu Mar 9 14:50:51 2017 >> >> >> >> >> >> No Disk descriptor in sector 96 of /dev/zd32 >> >> >> >> >> >> No FS descriptor in sector 2048 of /dev/zd32 >> >> >> >> >> >> root at nec:~/gpfs# >> >> >> >> >> >> >> >> >> >> >> >> Thanks in advance, all? >> >> >> >> >> >> >> >> >> >> >> >> Kevin >> >> >> >> >> >> ? >> >> >> >> >> >> Kevin Buterbaugh - Senior System Administrator >> >> >> >> >> >> Vanderbilt University - Advanced Computing Center for Research and Education >> >> >> >> >> >> Kevin.Buterbaugh at vanderbilt.edu >> >> > > >> >> >> > > - (615)875-9633 >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> gpfsug-discuss mailing list >> >> >> gpfsug-discuss at spectrumscale.org >> >> >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> >> >> > >> >> > -- >> >> > Aaron Knister >> >> > NASA Center for Climate Simulation (Code 606.2) >> >> > Goddard Space Flight Center >> >> > (301) 286-2776 >> >> > _______________________________________________ >> >> > gpfsug-discuss mailing list >> >> > gpfsug-discuss at spectrumscale.org >> >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> > >> >> Unless stated otherwise above: >> >> IBM United Kingdom Limited - Registered in England and Wales with >> >> number 741598. >> >> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >> >> >> >> _______________________________________________ >> >> gpfsug-discuss mailing list >> >> gpfsug-discuss at spectrumscale.org >> >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > >> > >> > >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > >> >> -- >> Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) >> Goddard Space Flight Center >> (301) 286-2776 >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Robert.Oesterlin at nuance.com Wed Mar 15 14:27:20 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Wed, 15 Mar 2017 14:27:20 +0000 Subject: [gpfsug-discuss] Running multiple mmrestripefs in a single cluster? Message-ID: I?m looking at migrating multiple file systems from one set of NSDs to another. Assuming I put aside any potential IO bottlenecks, has anyone tried running multiple ?mmrestripefs? commands in a single cluster? Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Wed Mar 15 14:53:44 2017 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Wed, 15 Mar 2017 09:53:44 -0500 Subject: [gpfsug-discuss] default inode size - remarks In-Reply-To: <20170315132609.wjplicnvhizlwc3q@ics.muni.cz> References: <20170315095031.r5rzoqrrwqs6djwl@ics.muni.cz><0B03B92B-DF24-4389-9301-D3501074145C@ulmer.org><20170315123700.s6fn7j6nz7lu2png@ics.muni.cz> <20170315132609.wjplicnvhizlwc3q@ics.muni.cz> Message-ID: You can set inode size to any one of 512, 1024, 2048 or 4096 when you mmcrfs. Try it on a test system. You want all the metadata of a file to fit into the inode. Also small files, and small directories, can be stored in their inodes. On a test file system you can examine how much of an inode is occupied with which data and metadata using the inode subcommand of the tsdbfs command. (Root only, and usual cautions -- use only to look, if you make a mistake patching - I told you not to do that!) inode number of any path is easily discovered with the standard ls -ild pathname command -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Wed Mar 15 20:03:35 2017 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Wed, 15 Mar 2017 21:03:35 +0100 Subject: [gpfsug-discuss] Running multiple mmrestripefs in a single cluster? In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Wed Mar 15 20:33:19 2017 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Wed, 15 Mar 2017 15:33:19 -0500 Subject: [gpfsug-discuss] Running multiple mmrestripefs in a singlecluster? In-Reply-To: References: Message-ID: You can control the load of mmrestripefs (and all maintenance commands) on your system using mmchqos ... From: "Olaf Weiser" To: gpfsug main discussion list Date: 03/15/2017 04:04 PM Subject: Re: [gpfsug-discuss] Running multiple mmrestripefs in a single cluster? Sent by: gpfsug-discuss-bounces at spectrumscale.org yes.. and please be carefully about the number of nodes , doing the job because of multiple PIT worker hammering against your data if you limit the restripe to 2 nodes (-N ......) of adjust the PITworker down to 8 or even 4 ... you can run multiple restripes.. without hurting the application workload to much ... but the final duration of your restripe then will be affected cheers From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 03/15/2017 03:27 PM Subject: [gpfsug-discuss] Running multiple mmrestripefs in a single cluster? Sent by: gpfsug-discuss-bounces at spectrumscale.org I?m looking at migrating multiple file systems from one set of NSDs to another. Assuming I put aside any potential IO bottlenecks, has anyone tried running multiple ?mmrestripefs? commands in a single cluster? Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From duersch at us.ibm.com Thu Mar 16 00:26:28 2017 From: duersch at us.ibm.com (Steve Duersch) Date: Wed, 15 Mar 2017 19:26:28 -0500 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 62, Issue 33 In-Reply-To: References: Message-ID: >>For me it's the protection against bitrot and added protection against silent data corruption GNR has this functionality. Right now that is available through ESS though. Not yet as software only. Steve Duersch Spectrum Scale 845-433-7902 IBM Poughkeepsie, New York gpfsug-discuss-bounces at spectrumscale.org wrote on 03/15/2017 10:25:59 AM: > > Message: 6 > Date: Wed, 15 Mar 2017 14:25:41 +0000 > From: "Buterbaugh, Kevin L" > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] mmcrfs issue > Message-ID: > Content-Type: text/plain; charset="utf-8" > > Hi All, > > Since I started this thread I guess I should chime in, too ? for us > it was simply that we were testing a device that did not have > hardware RAID controllers and we were wanting to implement something > roughly equivalent to RAID 6 LUNs. > > Kevin > > > On Mar 14, 2017, at 5:16 PM, Aaron Knister wrote: > > > > For me it's the protection against bitrot and added protection > against silent data corruption and in theory the write caching > offered by adding log devices that could help with small random > writes (although there are other problems with ZFS + synchronous > workloads that stop this from actually materializing). > > > > -Aaron > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.knister at gmail.com Thu Mar 16 00:52:58 2017 From: aaron.knister at gmail.com (Aaron Knister) Date: Wed, 15 Mar 2017 20:52:58 -0400 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 62, Issue 33 In-Reply-To: References: Message-ID: *drags out soapbox* Sorry in advance for the rant, this is one of my huge pet peeves :) There are some serious blockers for GNR adoption in my environment. It drives me up a wall that the only way to get end to end checksums in GPFS is with vendor hardware lock-in. I find it infuriating. Lustre can do this for free with ZFS. Historically it has also offered various other features too like eating your data so I guess it's a tradeoff ;-) I believe that either GNR should be available for any hardware that passes a validation suite or GPFS should support checksums on non-GNR NSDs either by leveraging T10-PI information or by checksumming blocks/subblocks and storing that somewhere. I opened an RFE for this and it was rejected and I was effectively told to go use GNR/ESS, but well... can't do GNR. But lets say I could run GNR on any hardware of my choosing after perhaps paying some modest licensing fee and passing a hardware validation test there's another blocker for me. Because GPFS doesn't support anything like an LNet router I'm fairly limited on the number of high speed verbs rdma fabrics I can connect GNR to. Furthermore even if I had enough PCIe slots the configuration may not be supported (e.g. a site with an OPA and an IB fabric that would like to use rdma verbs on both). There could even be a situation where a vendor of an HPC solution requires a specific OFED version for support purposes that's not the version running on the GNR nodes. If an NSD protocol router were available I could perhaps use ethernet as a common medium to work around this. I'd really like IBM to *do* something about this situation but I've not gotten any traction on it so far. -Aaron On Wed, Mar 15, 2017 at 8:26 PM, Steve Duersch wrote: > >>For me it's the protection against bitrot and added protection against > silent data corruption > GNR has this functionality. Right now that is available through ESS > though. Not yet as software only. > > Steve Duersch > Spectrum Scale > 845-433-7902 <(845)%20433-7902> > IBM Poughkeepsie, New York > > > > > gpfsug-discuss-bounces at spectrumscale.org wrote on 03/15/2017 10:25:59 AM: > > > > > > Message: 6 > > Date: Wed, 15 Mar 2017 14:25:41 +0000 > > From: "Buterbaugh, Kevin L" > > To: gpfsug main discussion list > > Subject: Re: [gpfsug-discuss] mmcrfs issue > > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > > > Hi All, > > > > Since I started this thread I guess I should chime in, too ? for us > > it was simply that we were testing a device that did not have > > hardware RAID controllers and we were wanting to implement something > > roughly equivalent to RAID 6 LUNs. > > > > Kevin > > > > > On Mar 14, 2017, at 5:16 PM, Aaron Knister > wrote: > > > > > > For me it's the protection against bitrot and added protection > > against silent data corruption and in theory the write caching > > offered by adding log devices that could help with small random > > writes (although there are other problems with ZFS + synchronous > > workloads that stop this from actually materializing). > > > > > > -Aaron > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan at buzzard.me.uk Thu Mar 16 11:19:30 2017 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Thu, 16 Mar 2017 11:19:30 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 62, Issue 33 In-Reply-To: References: Message-ID: <1489663170.17464.39.camel@buzzard.me.uk> On Wed, 2017-03-15 at 20:52 -0400, Aaron Knister wrote: > *drags out soapbox* > > > Sorry in advance for the rant, this is one of my huge pet peeves :) > > > > There are some serious blockers for GNR adoption in my environment. It > drives me up a wall that the only way to get end to end checksums in > GPFS is with vendor hardware lock-in. As I understand it that is a factually inaccurate statement. You are free to use a DIF/DIX solution which is part of the T10 SCSI specification and as such an open(ish) standard. That is if you pay a suitable amount you will get a copy of the standard and are free to implement it. It is arguably more open than ZFS that has licensing issues. Further as I understand it the DIF/DIX solution is actually better than the ZFS solution on a number of counts, that revolve around it being baked into the hardware. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From janfrode at tanso.net Thu Mar 16 13:50:48 2017 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Thu, 16 Mar 2017 14:50:48 +0100 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 62, Issue 33 In-Reply-To: References: Message-ID: Why would you need a NSD protocol router when the NSD servers can have a mix of infiniband and ethernet adapters? F.ex. 4x EDR + 2x 100GbE per io-node in an ESS should give you lots of bandwidth for your common ethernet medium. -jf On Thu, Mar 16, 2017 at 1:52 AM, Aaron Knister wrote: > *drags out soapbox* > > Sorry in advance for the rant, this is one of my huge pet peeves :) > > There are some serious blockers for GNR adoption in my environment. It > drives me up a wall that the only way to get end to end checksums in GPFS > is with vendor hardware lock-in. I find it infuriating. Lustre can do this > for free with ZFS. Historically it has also offered various other features > too like eating your data so I guess it's a tradeoff ;-) I believe that > either GNR should be available for any hardware that passes a validation > suite or GPFS should support checksums on non-GNR NSDs either by leveraging > T10-PI information or by checksumming blocks/subblocks and storing that > somewhere. I opened an RFE for this and it was rejected and I was > effectively told to go use GNR/ESS, but well... can't do GNR. > > But lets say I could run GNR on any hardware of my choosing after perhaps > paying some modest licensing fee and passing a hardware validation test > there's another blocker for me. Because GPFS doesn't support anything like > an LNet router I'm fairly limited on the number of high speed verbs rdma > fabrics I can connect GNR to. Furthermore even if I had enough PCIe slots > the configuration may not be supported (e.g. a site with an OPA and an IB > fabric that would like to use rdma verbs on both). There could even be a > situation where a vendor of an HPC solution requires a specific OFED > version for support purposes that's not the version running on the GNR > nodes. If an NSD protocol router were available I could perhaps use > ethernet as a common medium to work around this. > > I'd really like IBM to *do* something about this situation but I've not > gotten any traction on it so far. > > -Aaron > > > > On Wed, Mar 15, 2017 at 8:26 PM, Steve Duersch wrote: > >> >>For me it's the protection against bitrot and added protection against >> silent data corruption >> GNR has this functionality. Right now that is available through ESS >> though. Not yet as software only. >> >> Steve Duersch >> Spectrum Scale >> 845-433-7902 <(845)%20433-7902> >> IBM Poughkeepsie, New York >> >> >> >> >> gpfsug-discuss-bounces at spectrumscale.org wrote on 03/15/2017 10:25:59 AM: >> >> >> > >> > Message: 6 >> > Date: Wed, 15 Mar 2017 14:25:41 +0000 >> > From: "Buterbaugh, Kevin L" >> > To: gpfsug main discussion list >> > Subject: Re: [gpfsug-discuss] mmcrfs issue >> > Message-ID: >> > Content-Type: text/plain; charset="utf-8" >> > >> > Hi All, >> > >> > Since I started this thread I guess I should chime in, too ? for us >> > it was simply that we were testing a device that did not have >> > hardware RAID controllers and we were wanting to implement something >> > roughly equivalent to RAID 6 LUNs. >> > >> > Kevin >> > >> > > On Mar 14, 2017, at 5:16 PM, Aaron Knister >> wrote: >> > > >> > > For me it's the protection against bitrot and added protection >> > against silent data corruption and in theory the write caching >> > offered by adding log devices that could help with small random >> > writes (although there are other problems with ZFS + synchronous >> > workloads that stop this from actually materializing). >> > > >> > > -Aaron >> > > >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.knister at gmail.com Thu Mar 16 14:39:13 2017 From: aaron.knister at gmail.com (Aaron Knister) Date: Thu, 16 Mar 2017 10:39:13 -0400 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 62, Issue 33 In-Reply-To: <1489663170.17464.39.camel@buzzard.me.uk> References: <1489663170.17464.39.camel@buzzard.me.uk> Message-ID: In all seriousness, I'd love to be wrong about this. I'm not sure which part(s) of what I said was inaccurate-- the vendor lock in and/or that GNR is the only way to get end to end checksums. When I say end to end checksums I mean that from the moment an FS write is submitted to mmfsd a checksum is calculated that is passed through every layer (memory, network, sas, fibre channel etc.) down to individual disk drives (understanding that the RAID controller may need to derive the checksum based on whatever RAID algorithm it's using). It's my understanding that the only way to achieve this with GPFS today is with GNR which is only available via purchasing a GSS or ESS solution from IBM/Lenovo. One is of course free to by hardware from any vendor but you don't get GNR. I should really have said "if you want GNR you have to buy hardware from IBM or Lenovo" which to me is being locked in to a vendor as long as you'd like end to end checksums. If there's another way to get end-to-end checksums, could you help me understand how? Regarding DIF/DIX, it's my understanding that I can/could use T10-DIF today (with the correct hardware) without purchasing any standards which would checksum data from the HBA to the RAID controller (and in theory disks). However, in an environment with NSD servers the origin of a read/write could in theory be quite a bit further away from the HBA in terms of hops. T10-DIF would be completely separate from GPFS as I understand it. I'm not aware of any integration (T10-DIF + DIX). What I'm really looking for, I think, is T10-DIF + DIX where the application (GPFS) generates protection information that's then passed along to the layers below it. -Aaron On Thu, Mar 16, 2017 at 7:19 AM, Jonathan Buzzard wrote: > On Wed, 2017-03-15 at 20:52 -0400, Aaron Knister wrote: > > *drags out soapbox* > > > > > > Sorry in advance for the rant, this is one of my huge pet peeves :) > > > > > > > > There are some serious blockers for GNR adoption in my environment. It > > drives me up a wall that the only way to get end to end checksums in > > GPFS is with vendor hardware lock-in. > > As I understand it that is a factually inaccurate statement. > > You are free to use a DIF/DIX solution which is part of the T10 SCSI > specification and as such an open(ish) standard. That is if you pay a > suitable amount you will get a copy of the standard and are free to > implement it. It is arguably more open than ZFS that has licensing > issues. > > Further as I understand it the DIF/DIX solution is actually better than > the ZFS solution on a number of counts, that revolve around it being > baked into the hardware. > > JAB. > > -- > Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk > Fife, United Kingdom. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Thu Mar 16 14:43:47 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Thu, 16 Mar 2017 10:43:47 -0400 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 62, Issue 33 In-Reply-To: References: Message-ID: <26bc0188-b5de-a128-9122-e6ed145aba69@nasa.gov> Perhaps an environment where one has OPA and IB fabrics. Taken from here (https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html): RDMA is not supported on a node when both Mellanox HCAs and Intel Omni-Path HFIs are enabled for RDMA. The alternative being a situation where multiple IB fabrics exist that require different OFED versions from each other (and most likely from ESS) for support reasons (speaking from experience). That is to say if $VENDOR supports OFED version X on an IB fabric, and ESS/GSS ships with version Y and there's a problem on the IB fabric $VENDOR may point at the different OFED version on the ESS/GSS and say they don't support it and then one is in a bad spot. -Aaron On 3/16/17 9:50 AM, Jan-Frode Myklebust wrote: > Why would you need a NSD protocol router when the NSD servers can have a > mix of infiniband and ethernet adapters? F.ex. 4x EDR + 2x 100GbE per > io-node in an ESS should give you lots of bandwidth for your common > ethernet medium. > > > -jf > > On Thu, Mar 16, 2017 at 1:52 AM, Aaron Knister > wrote: > > *drags out soapbox* > > Sorry in advance for the rant, this is one of my huge pet peeves :) > > There are some serious blockers for GNR adoption in my environment. > It drives me up a wall that the only way to get end to end checksums > in GPFS is with vendor hardware lock-in. I find it infuriating. > Lustre can do this for free with ZFS. Historically it has also > offered various other features too like eating your data so I guess > it's a tradeoff ;-) I believe that either GNR should be available > for any hardware that passes a validation suite or GPFS should > support checksums on non-GNR NSDs either by leveraging T10-PI > information or by checksumming blocks/subblocks and storing that > somewhere. I opened an RFE for this and it was rejected and I was > effectively told to go use GNR/ESS, but well... can't do GNR. > > But lets say I could run GNR on any hardware of my choosing after > perhaps paying some modest licensing fee and passing a hardware > validation test there's another blocker for me. Because GPFS doesn't > support anything like an LNet router I'm fairly limited on the > number of high speed verbs rdma fabrics I can connect GNR to. > Furthermore even if I had enough PCIe slots the configuration may > not be supported (e.g. a site with an OPA and an IB fabric that > would like to use rdma verbs on both). There could even be a > situation where a vendor of an HPC solution requires a specific OFED > version for support purposes that's not the version running on the > GNR nodes. If an NSD protocol router were available I could perhaps > use ethernet as a common medium to work around this. > > I'd really like IBM to *do* something about this situation but I've > not gotten any traction on it so far. > > -Aaron > > > > On Wed, Mar 15, 2017 at 8:26 PM, Steve Duersch > wrote: > > >>For me it's the protection against bitrot and added protection > against silent data corruption > GNR has this functionality. Right now that is available through > ESS though. Not yet as software only. > > Steve Duersch > Spectrum Scale > 845-433-7902 > IBM Poughkeepsie, New York > > > > > gpfsug-discuss-bounces at spectrumscale.org > wrote on > 03/15/2017 10:25:59 AM: > > > > > > Message: 6 > > Date: Wed, 15 Mar 2017 14:25:41 +0000 > > From: "Buterbaugh, Kevin L" > > To: gpfsug main discussion list > > > Subject: Re: [gpfsug-discuss] mmcrfs issue > > Message-ID: > > > Content-Type: text/plain; charset="utf-8" > > > > Hi All, > > > > Since I started this thread I guess I should chime in, too ? for us > > it was simply that we were testing a device that did not have > > hardware RAID controllers and we were wanting to implement something > > roughly equivalent to RAID 6 LUNs. > > > > Kevin > > > > > On Mar 14, 2017, at 5:16 PM, Aaron Knister > wrote: > > > > > > For me it's the protection against bitrot and added protection > > against silent data corruption and in theory the write caching > > offered by adding log devices that could help with small random > > writes (although there are other problems with ZFS + synchronous > > workloads that stop this from actually materializing). > > > > > > -Aaron > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From jonathan at buzzard.me.uk Thu Mar 16 16:54:32 2017 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Thu, 16 Mar 2017 16:54:32 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 62, Issue 33 In-Reply-To: References: <1489663170.17464.39.camel@buzzard.me.uk> Message-ID: <1489683272.17464.67.camel@buzzard.me.uk> On Thu, 2017-03-16 at 10:39 -0400, Aaron Knister wrote: > In all seriousness, I'd love to be wrong about this. I'm not sure > which part(s) of what I said was inaccurate-- the vendor lock in > and/or that GNR is the only way to get end to end checksums. The end to end checksums, or at least the ability to protect against silent corruption by having the data checksummed at all stages. > > When I say end to end checksums I mean that from the moment an FS > write is submitted to mmfsd a checksum is calculated that is passed > through every layer (memory, network, sas, fibre channel etc.) down to > individual disk drives (understanding that the RAID controller may > need to derive the checksum based on whatever RAID algorithm it's > using). It's my understanding that the only way to achieve this with > GPFS today is with GNR which is only available via purchasing a GSS or > ESS solution from IBM/Lenovo. One is of course free to by hardware > from any vendor but you don't get GNR. I should really have said "if > you want GNR you have to buy hardware from IBM or Lenovo" which to me > is being locked in to a vendor as long as you'd like end to end > checksums. > > If there's another way to get end-to-end checksums, could you help me > understand how? > Bear in mind the purpose of the checksums is to ensure the data is not corrupted. Noting you don't get true end to end because the checksums are never exposed to the application even in ZFS, at least as I understand it; you just get a read failure. > Regarding DIF/DIX, it's my understanding that I can/could use T10-DIF > today (with the correct hardware) without purchasing any standards > which would checksum data from the HBA to the RAID controller (and in > theory disks). However, in an environment with NSD servers the origin > of a read/write could in theory be quite a bit further away from the > HBA in terms of hops. T10-DIF would be completely separate from GPFS > as I understand it. I'm not aware of any integration (T10-DIF + DIX). > What I'm really looking for, I think, is T10-DIF + DIX where the > application (GPFS) generates protection information that's then passed > along to the layers below it. > Well yes an end user does not need to purchase a copy of the standard. However some people take the view that as you need to spend money to get the standard it is not truly open, so I was just making that clear. Right, so bearing in mind that the purpose of the checksums is to ensure data is not corrupted if the NSD servers are using DIF/DIX, then the options for silent data corruption are very limited if you are using ECC memory and you are using standard TCP/IP to communicate between the nodes. That is the TCP/IP checksums will protect your data between the clients and the NSD nodes, and once at the NSD node the DIF/DIX will protect your data as it makes it way to the disk. In between all that the ECC memory in your machines will protect everything once in RAM, and the PCIe bus has a 32bit CRC protecting everything that moves over the PCIe bus. The only "hole" in this is as far as I am aware is that the CPU itself, because at least x86 ones (as far as I am aware) do not "checksum" themselves as they do calculations, but you have the same problem with ZFS for exactly the same reason. So the point is that what you should be interested in as an admin; ensuring your data is protected against silent corruption, is achievable without hardware vendor lock in using open standards. The advantage of DIF/DIX over ZFS is as I understand it unless you do an immediate read on written blocks with ZFS and take a large performance penalty on writes you have no idea if your data has been corrupted until you try and read it back which could be months if not years down the line. Where DIF/DIX should try and fix it immediately by a retry and if that does not work pass the failure back up the line immediately. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From jonathan at buzzard.me.uk Thu Mar 16 17:18:22 2017 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Thu, 16 Mar 2017 17:18:22 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 62, Issue 33 In-Reply-To: <26bc0188-b5de-a128-9122-e6ed145aba69@nasa.gov> References: <26bc0188-b5de-a128-9122-e6ed145aba69@nasa.gov> Message-ID: <1489684702.17464.84.camel@buzzard.me.uk> On Thu, 2017-03-16 at 10:43 -0400, Aaron Knister wrote: > Perhaps an environment where one has OPA and IB fabrics. Taken from here > (https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html): > > RDMA is not supported on a node when both Mellanox HCAs and Intel > Omni-Path HFIs are enabled for RDMA. > > The alternative being a situation where multiple IB fabrics exist that > require different OFED versions from each other (and most likely from > ESS) for support reasons (speaking from experience). That is to say if > $VENDOR supports OFED version X on an IB fabric, and ESS/GSS ships with > version Y and there's a problem on the IB fabric $VENDOR may point at > the different OFED version on the ESS/GSS and say they don't support it > and then one is in a bad spot. > Or just use Ethernet for the GPFS traffic everywhere. It's 2017, 10GbE to compute nodes is cheap enough to be the norm, and you can use 40GbE and 100GbE everywhere else as required, and note unless you are a pure SSD system (which must be vanishingly rare at the moment) the latency is all in the disks anyway. I can't imagine why you would use IB and/or Omni-Path on anything other than compute clusters, so using Ethernet for your storage has advantages too especially if you are using IB. One of the features of Omni-Path over IB is that it will prioritize MPI traffic over storage, but given current core counts in CPU's separating storage out onto Ethernet is not a bad thing anyway as it keeps the MPI bandwidth per core up. My feeling is that in a compute cluster 10Gb for storage is more than enough for a node because much more and it would only take a handful of nodes to denial of service your storage, which is not good. A 500 node compute cluster with 10GbE for storage can in theory try and hit your storage for 600GB/s which very very few storage systems will be able to keep up with. On a storage GPFS cluster then you can up to 40/100GbE and put in multiple links for redundancy which is not possible with IB/Omni-path as far as I am aware which makes it a better solution. Even in a compute cluster with Ethernet I can stick in multiple connections for my NSD nodes for redundancy which is an improvement over the IB/Omni-path options. Especially given in my experience IB links go wonky orders of magnitude more often than Ethernet ones do. Loose a link to a compute node I loose one job on the cluster. Loose a link to the NSD server and I face loosing all the jobs on the cluster... JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From Robert.Oesterlin at nuance.com Thu Mar 16 18:43:59 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Thu, 16 Mar 2017 18:43:59 +0000 Subject: [gpfsug-discuss] Registration Reminder: GPFSUG-USA Meeting at NERSC April 4-5th Message-ID: Here?s a reminder to register for the upcoming user group meeting! Registration deadline extended to March 29th! Registration is filling up ? don?t wait. We Still have slots (20 mins each) for customer and partner talks ? please contact me if you want to do a short presentation. Register here: https://www.nersc.gov/research-and-development/data-analytics/spectrum-user-group-meeting/ Bob Oesterlin Sr Principal Storage Engineer, Nuance -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Fri Mar 17 18:50:11 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Fri, 17 Mar 2017 18:50:11 +0000 Subject: [gpfsug-discuss] mmnetverify Message-ID: Hi all, Just wondering if anyone has used the mmnetverify tool at all? Having made some changes to our internal L3 routing this week, I was interested to see what it claimed. As a side-note, it picked up some DNS resolution issues, though I'm not clear on some of those why it was claiming this as doing a "dig" on the node, it resolved fine (but adding the NSD servers to the hosts files cleared the error). Its actually the bandwidth tests that I'm interested in hearing other people's experience with as the numbers that some out from it are very different (lower) than if we use iperf to test performance between two nodes. Anyone any thoughts at all on this? Thanks Simon From Paul.Sanchez at deshaw.com Fri Mar 17 19:43:20 2017 From: Paul.Sanchez at deshaw.com (Sanchez, Paul) Date: Fri, 17 Mar 2017 19:43:20 +0000 Subject: [gpfsug-discuss] mmnetverify In-Reply-To: References: Message-ID: <18729d8a86fc430fac09c1c3a22159d2@mbxtoa1.winmail.deshaw.com> Sven will tell you: "RPC isn't streaming" and that may account for the discrepancy. If the tests are doing any "fan-in" where multiple nodes are sending to single node, then it's also possible that you are exhausting switch buffer memory in a way that a 1:1 iperf wouldn't. For our internal benchmarking we've used /usr/lpp/mmfs/samples/net/nsdperf to more closely estimate the real performance. I haven't played with mmnetverify yet though. -Paul -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (Research Computing - IT Services) Sent: Friday, March 17, 2017 2:50 PM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] mmnetverify Hi all, Just wondering if anyone has used the mmnetverify tool at all? Having made some changes to our internal L3 routing this week, I was interested to see what it claimed. As a side-note, it picked up some DNS resolution issues, though I'm not clear on some of those why it was claiming this as doing a "dig" on the node, it resolved fine (but adding the NSD servers to the hosts files cleared the error). Its actually the bandwidth tests that I'm interested in hearing other people's experience with as the numbers that some out from it are very different (lower) than if we use iperf to test performance between two nodes. Anyone any thoughts at all on this? Thanks Simon _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From S.J.Thompson at bham.ac.uk Fri Mar 17 20:13:22 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Fri, 17 Mar 2017 20:13:22 +0000 Subject: [gpfsug-discuss] mmnetverify In-Reply-To: <18729d8a86fc430fac09c1c3a22159d2@mbxtoa1.winmail.deshaw.com> References: , <18729d8a86fc430fac09c1c3a22159d2@mbxtoa1.winmail.deshaw.com> Message-ID: It looks to run sequential tests to each node one at a time and isn't using NSD protocol but echo server. We found some weird looking numbers that i don't quite understand and not in the places we might expect. For example between hosts on the same switch, traffic flowing to another switch and traffic flowing to nodes in another data centre where it's several switch hops. Some nodes over there were significantly faster than switch local nodes. I think it was only added in 4.2.2 and is listed as "not yet a replacement for nsdperf". I get that is different as it's using NSD protocol, but was struggling a bit with what mmnetverify might be doing. Simon From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Sanchez, Paul [Paul.Sanchez at deshaw.com] Sent: 17 March 2017 19:43 To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] mmnetverify Sven will tell you: "RPC isn't streaming" and that may account for the discrepancy. If the tests are doing any "fan-in" where multiple nodes are sending to single node, then it's also possible that you are exhausting switch buffer memory in a way that a 1:1 iperf wouldn't. For our internal benchmarking we've used /usr/lpp/mmfs/samples/net/nsdperf to more closely estimate the real performance. I haven't played with mmnetverify yet though. -Paul -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (Research Computing - IT Services) Sent: Friday, March 17, 2017 2:50 PM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] mmnetverify Hi all, Just wondering if anyone has used the mmnetverify tool at all? Having made some changes to our internal L3 routing this week, I was interested to see what it claimed. As a side-note, it picked up some DNS resolution issues, though I'm not clear on some of those why it was claiming this as doing a "dig" on the node, it resolved fine (but adding the NSD servers to the hosts files cleared the error). Its actually the bandwidth tests that I'm interested in hearing other people's experience with as the numbers that some out from it are very different (lower) than if we use iperf to test performance between two nodes. Anyone any thoughts at all on this? Thanks Simon _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Robert.Oesterlin at nuance.com Mon Mar 20 13:07:43 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 20 Mar 2017 13:07:43 +0000 Subject: [gpfsug-discuss] SSUG-USA: Registration Reminder and Agenda Update Message-ID: Only two weeks until the user group meeting! Registration is filling up fast ? Deadline is March 29th. We Still have 2 slots (20 mins each) for customer and partner talks ? please contact me if you want to do a short presentation. Register here: https://www.nersc.gov/research-and-development/data-analytics/spectrum-user-group-meeting/ Duration Start End Title Speaker Tues April 4th 60 9:00 10:00 Registration and Coffee n/a 30 10:00 10:30 Welcome Bob Oesterlin, Kristy Kallback, Doris Conti 60 10:30 11:30 Spectrum Scale & ESS Update Ulf Troppens / Scott Fadden 60 11:30 12:30 Problem Avoidance - Best Practices in the Field Brian Yaeger 60 12:30 13:30 Lunch and Networking n/a 120 13:30 15:30 Problem Determination - Deep Dive Yuri Volobuev 30 15:30 16:00 Coffee and Networking n/a 60 16:00 17:00 Problem Determination - What's New? Matthias Dietz 60 17:00 18:00 Bring your favorite problem or performance issue. We will fix it now! All Wed April 5th 30 8:30 9:00 Coffee and Networking n/a 60 9:00 10:00 1) AFM - Introduction and Use Cases (Meet the Devs) Scott Fadden 2) Life Sciences and Next Generation Sequencing with Spectrum Scale (Solutions) Madhav Ponamgi 3) Cloud and Virtualization Discussion (BOF) John Lewars 60 10:00 11:00 1) Transparent Cloud Tiering - Introduction and Use Cases (Meet the Devs) Rob Basham 2) Spectrum Protect on Spectrum Scale (Solutions) Jason Basler 3) Open User BOF TBD 60 11:00 12:00 1) Spectrum Scale Security (Meet the Devs) Felipe Knop 2) Hadoop Integration and Support for HortonWorks HDP (Solutions) Ted Hoover 3) Open User BOF TBD 60 12:00 13:00 Lunch and Networking n/a 20 13:00 13:20 System Analytics/IO Profiling TBD Customer 20 13:20 13:40 Placeholder - Customer or Partner talk TBD Customer 20 13:40 14:00 Placeholder - Customer or Partner talk TBD Customer 20 14:00 14:20 REST API Ulf Troppens 30 14:20 14:50 Coffee and Networking n/a 30 14:50 15:20 Application Integration with Containers Dean Hildebrand 30 15:20 15:50 Challenges in Tracing Vasily Tarasov 10 15:50 16:00 Wrap-up Bob Oesterlin, Kristy Kallback, Doris Conti Bob Oesterlin Sr Principal Storage Engineer, Nuance -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.wonderley at vt.edu Mon Mar 20 14:02:56 2017 From: eric.wonderley at vt.edu (J. Eric Wonderley) Date: Mon, 20 Mar 2017 10:02:56 -0400 Subject: [gpfsug-discuss] snapshots & tiering in a busy filesystem Message-ID: I found this link and it didn't give me much hope for doing snapshots & backup in a home(busy) filesystem: http://www.spectrumscale.org/pipermail/gpfsug-discuss/2013- February/000200.html I realize this is dated and I wondered if qos etc have made is a tolerable thing to do now. Gpfs I think was barely above v3.5 in mid 2013. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Mon Mar 20 14:49:07 2017 From: bbanister at jumptrading.com (Bryan Banister) Date: Mon, 20 Mar 2017 14:49:07 +0000 Subject: [gpfsug-discuss] snapshots & tiering in a busy filesystem In-Reply-To: References: Message-ID: We run snapshots every 4 hours on a busy home file system without issues for many years now, -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of J. Eric Wonderley Sent: Monday, March 20, 2017 9:03 AM To: gpfsug main discussion list Subject: [gpfsug-discuss] snapshots & tiering in a busy filesystem I found this link and it didn't give me much hope for doing snapshots & backup in a home(busy) filesystem: http://www.spectrumscale.org/pipermail/gpfsug-discuss/2013-February/000200.html I realize this is dated and I wondered if qos etc have made is a tolerable thing to do now. Gpfs I think was barely above v3.5 in mid 2013. ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Mon Mar 20 14:53:51 2017 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 20 Mar 2017 09:53:51 -0500 Subject: [gpfsug-discuss] snapshots & tiering in a busy filesystem In-Reply-To: References: Message-ID: Yes. QOS (see mmchqos) is now the way (4.2.x) to control maintenance commands like snapshots, backup, restripe. From: "J. Eric Wonderley" To: gpfsug main discussion list Date: 03/20/2017 10:03 AM Subject: [gpfsug-discuss] snapshots & tiering in a busy filesystem Sent by: gpfsug-discuss-bounces at spectrumscale.org I found this link and it didn't give me much hope for doing snapshots & backup in a home(busy) filesystem: http://www.spectrumscale.org/pipermail/gpfsug-discuss/2013-February/000200.html I realize this is dated and I wondered if qos etc have made is a tolerable thing to do now. Gpfs I think was barely above v3.5 in mid 2013. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.g.richmond at leeds.ac.uk Tue Mar 21 10:56:41 2017 From: a.g.richmond at leeds.ac.uk (Aidan Richmond) Date: Tue, 21 Mar 2017 10:56:41 +0000 Subject: [gpfsug-discuss] Configuration spectrum scale to use SFU schema Message-ID: <8d1622e5-99dc-dad4-998b-b0d5a88c8f70@leeds.ac.uk> Hello Is it possible to configure authentication services to bind to AD using the older SFU schema for ID mappings? We have standard samba and nfs servers that use this ID mapping scheme with "idmap config DS:schema_mode = sfu" in the samba configuration files, but if its possible using the mmuserauth command it doesn't seem to be documented. Thanks. -- Aidan Richmond Apple/Unix Support Officer, IT Garstang 10.137 Faculty of Biological Sciences University of Leeds Clarendon Way LS2 9JT Tel:0113 3434252 a.g.richmond at leeds.ac.uk From chair at spectrumscale.org Tue Mar 21 14:00:50 2017 From: chair at spectrumscale.org (Spectrum Scale UG Chair (Simon Thompson)) Date: Tue, 21 Mar 2017 14:00:50 +0000 Subject: [gpfsug-discuss] Sydney April Spectrum Scale UG Meeting In-Reply-To: References: Message-ID: Hi All, If you are located in the Australia region, you may be interested in the Spectrum Scale User Group meeting being hosted in Sydney, Australia: >We have a good part of the day for a Meet the Developers team with Kedar >Karmarkar form India and his team. >Plus Mellanox, Pawsey Supercomputing Centre, and a lessons learn from DDN >on IB networking at the Australian Bureau of Meteorology (subject to >disclosure agreement). Agenda and registration are at: >https://www.eventbrite.com/e/spectrum-scale-user-group-australia-gpfsugaus >-april-2017-tickets-29136246297 There's also a slack channel for discussion at: >https://ibmau-spectrumscale.slack.com/messages/C3C10AC9H/details/ This is being organised by Chris Schlipalius from Pawsey Supercomputing Centre, please contact Chris directly for more details (see link on Eventbrite page). Simon UK Group Chair From christof.schmitt at us.ibm.com Tue Mar 21 17:24:03 2017 From: christof.schmitt at us.ibm.com (Christof Schmitt) Date: Tue, 21 Mar 2017 10:24:03 -0700 Subject: [gpfsug-discuss] Configuration spectrum scale to use SFU schema In-Reply-To: <8d1622e5-99dc-dad4-998b-b0d5a88c8f70@leeds.ac.uk> References: <8d1622e5-99dc-dad4-998b-b0d5a88c8f70@leeds.ac.uk> Message-ID: gpfsug-discuss-bounces at spectrumscale.org wrote on 03/21/2017 03:56:41 AM: > Is it possible to configure authentication services to bind to AD using > the older SFU schema for ID mappings? We have standard samba and nfs > servers that use this ID mapping scheme with "idmap config > DS:schema_mode = sfu" in the samba configuration files, but if its > possible using the mmuserauth command it doesn't seem to be documented. This falls into the category that we use Samba to provide these services, but cannot test and support every possible configuration. The feature to read the old MS style id attributes has not been tested with Spectrum Scale and is not supported. That said, it can still be set in the internal config: /usr/lpp/mmfs/bin/net conf setparm global 'idmap config DS:schema_mode' sfu Activating this change probably would require a restart of the process using this config: /usr/lpp/mmfs/bin/mmdsh -N CesNodes systemctl restart gpfs-winbind Opening a PMR on this will likely result in the answer that this is unsupported. The best way to request official support would be opening a RFE, let others vote and then product management can decide on the priority of the request. Regards, Christof Schmitt || IBM || Spectrum Scale Development || Tucson, AZ christof.schmitt at us.ibm.com || +1-520-799-2469 (T/L: 321-2469) From TROPPENS at de.ibm.com Tue Mar 21 17:51:09 2017 From: TROPPENS at de.ibm.com (Ulf Troppens) Date: Tue, 21 Mar 2017 18:51:09 +0100 Subject: [gpfsug-discuss] Charts of German User Meeting now online Message-ID: Thanks a lot to all speaker. Event report will follow later. http://www.spectrumscale.org/presentations/ -- IBM Spectrum Scale Development - Client Engagements & Solutions Delivery Consulting IT Specialist Author "Storage Networks Explained" IBM Deutschland Research & Development GmbH Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.g.richmond at leeds.ac.uk Wed Mar 22 08:01:33 2017 From: a.g.richmond at leeds.ac.uk (Aidan Richmond) Date: Wed, 22 Mar 2017 08:01:33 +0000 Subject: [gpfsug-discuss] Configuration spectrum scale to use SFU schema In-Reply-To: References: <8d1622e5-99dc-dad4-998b-b0d5a88c8f70@leeds.ac.uk> Message-ID: <75a2acf9-e598-89c2-9546-6cedd80186c1@leeds.ac.uk> On 21/03/17 17:24, Christof Schmitt wrote: > gpfsug-discuss-bounces at spectrumscale.org wrote on 03/21/2017 03:56:41 AM: > >> Is it possible to configure authentication services to bind to AD using >> the older SFU schema for ID mappings? We have standard samba and nfs >> servers that use this ID mapping scheme with "idmap config >> DS:schema_mode = sfu" in the samba configuration files, but if its >> possible using the mmuserauth command it doesn't seem to be documented. > > This falls into the category that we use Samba to provide these services, > but cannot test and support > every possible configuration. The feature to read the old MS style id > attributes has not been tested > with Spectrum Scale and is not supported. That said, it can still be set > in the internal config: > > /usr/lpp/mmfs/bin/net conf setparm global 'idmap config DS:schema_mode' > sfu > > Activating this change probably would require a restart of the process > using this config: > > /usr/lpp/mmfs/bin/mmdsh -N CesNodes systemctl restart gpfs-winbind > > Opening a PMR on this will likely result in the answer that this is > unsupported. The best way > to request official support would be opening a RFE, let others vote and > then product > management can decide on the priority of the request. > > Regards, > > Christof Schmitt || IBM || Spectrum Scale Development || Tucson, AZ > christof.schmitt at us.ibm.com || +1-520-799-2469 (T/L: 321-2469) > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > Thanks, would I need to re-run mmuserauth as per https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.1/com.ibm.spectrum.scale.v4r21.doc/bl1adm_adwithrfc2307.htm or would the schema change be needed after that when configuring the authentication services? -- Aidan Richmond Apple/Unix Support Officer, IT Garstang 10.137 Faculty of Biological Sciences University of Leeds Clarendon Way LS2 9JT Tel:0113 3434252 a.g.richmond at leeds.ac.uk From r.sobey at imperial.ac.uk Wed Mar 22 10:47:37 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Wed, 22 Mar 2017 10:47:37 +0000 Subject: [gpfsug-discuss] snapshots & tiering in a busy filesystem In-Reply-To: References: Message-ID: We?re also snapshotting 4 times a day. Filesystem isn?t tremendously busy at all but we?re creating snaps for each fileset. [root at cesnode tmp]# mmlssnapshot gpfs | wc -l 6916 From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of J. Eric Wonderley Sent: 20 March 2017 14:03 To: gpfsug main discussion list Subject: [gpfsug-discuss] snapshots & tiering in a busy filesystem I found this link and it didn't give me much hope for doing snapshots & backup in a home(busy) filesystem: http://www.spectrumscale.org/pipermail/gpfsug-discuss/2013-February/000200.html I realize this is dated and I wondered if qos etc have made is a tolerable thing to do now. Gpfs I think was barely above v3.5 in mid 2013. -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.wonderley at vt.edu Wed Mar 22 13:41:26 2017 From: eric.wonderley at vt.edu (J. Eric Wonderley) Date: Wed, 22 Mar 2017 09:41:26 -0400 Subject: [gpfsug-discuss] snapshots & tiering in a busy filesystem In-Reply-To: References: Message-ID: The filesystem I'm working with has about 100M files and 80Tb of data. What kind of metadata latency do you observe? I did a mmdiag --iohist and filtered out all of the md devices and averaged over reads and writes. I'm seeing ~.28ms on a one off dump. The pure array which we have is 10G iscsi connected and is reporting average .25ms. On Wed, Mar 22, 2017 at 6:47 AM, Sobey, Richard A wrote: > We?re also snapshotting 4 times a day. Filesystem isn?t tremendously busy > at all but we?re creating snaps for each fileset. > > > > [root at cesnode tmp]# mmlssnapshot gpfs | wc -l > > 6916 > > > > *From:* gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss- > bounces at spectrumscale.org] *On Behalf Of *J. Eric Wonderley > *Sent:* 20 March 2017 14:03 > *To:* gpfsug main discussion list > *Subject:* [gpfsug-discuss] snapshots & tiering in a busy filesystem > > > > I found this link and it didn't give me much hope for doing snapshots & > backup in a home(busy) filesystem: > > http://www.spectrumscale.org/pipermail/gpfsug-discuss/2013- > February/000200.html > > I realize this is dated and I wondered if qos etc have made is a tolerable > thing to do now. Gpfs I think was barely above v3.5 in mid 2013. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mweil at wustl.edu Wed Mar 22 16:42:35 2017 From: mweil at wustl.edu (Matt Weil) Date: Wed, 22 Mar 2017 11:42:35 -0500 Subject: [gpfsug-discuss] CES node slow to respond Message-ID: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu> All, We had an indecent yesterday where one of our CES nodes slowed to a crawl. GPFS waiters showed pre fetch threads going after inodes. iohist also showed lots of inode fetching. Then we noticed that the CES host had 5.4 million files open. The change I made was to set maxStatCache=DEFAULT because it is linux. And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. Is there something else we should change as well. Thanks Matt ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. From bbanister at jumptrading.com Wed Mar 22 17:01:26 2017 From: bbanister at jumptrading.com (Bryan Banister) Date: Wed, 22 Mar 2017 17:01:26 +0000 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu> References: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu> Message-ID: <47e18bb0062544f4b510ce0e87049fd3@jumptrading.com> We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. -Bryan -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Wednesday, March 22, 2017 11:43 AM To: gpfsug main discussion list Subject: [gpfsug-discuss] CES node slow to respond All, We had an indecent yesterday where one of our CES nodes slowed to a crawl. GPFS waiters showed pre fetch threads going after inodes. iohist also showed lots of inode fetching. Then we noticed that the CES host had 5.4 million files open. The change I made was to set maxStatCache=DEFAULT because it is linux. And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. Is there something else we should change as well. Thanks Matt ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. From r.sobey at imperial.ac.uk Wed Mar 22 17:44:56 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Wed, 22 Mar 2017 17:44:56 +0000 Subject: [gpfsug-discuss] snapshots & tiering in a busy filesystem In-Reply-To: References: Message-ID: Embarrassingly I don?t know how to do that scientifically. I can tell you latency of the storage system which at a very simplistic snapshot is averaging around 2ms. We are not using SSD for metadata but 10K SAS. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of J. Eric Wonderley Sent: 22 March 2017 13:41 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] snapshots & tiering in a busy filesystem The filesystem I'm working with has about 100M files and 80Tb of data. What kind of metadata latency do you observe? I did a mmdiag --iohist and filtered out all of the md devices and averaged over reads and writes. I'm seeing ~.28ms on a one off dump. The pure array which we have is 10G iscsi connected and is reporting average .25ms. On Wed, Mar 22, 2017 at 6:47 AM, Sobey, Richard A > wrote: We?re also snapshotting 4 times a day. Filesystem isn?t tremendously busy at all but we?re creating snaps for each fileset. [root at cesnode tmp]# mmlssnapshot gpfs | wc -l 6916 From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of J. Eric Wonderley Sent: 20 March 2017 14:03 To: gpfsug main discussion list > Subject: [gpfsug-discuss] snapshots & tiering in a busy filesystem I found this link and it didn't give me much hope for doing snapshots & backup in a home(busy) filesystem: http://www.spectrumscale.org/pipermail/gpfsug-discuss/2013-February/000200.html I realize this is dated and I wondered if qos etc have made is a tolerable thing to do now. Gpfs I think was barely above v3.5 in mid 2013. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From valdis.kletnieks at vt.edu Wed Mar 22 21:21:42 2017 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Wed, 22 Mar 2017 17:21:42 -0400 Subject: [gpfsug-discuss] LTFS/EE release question.. Message-ID: <37386.1490217702@turing-police.cc.vt.edu> We're currently on ltfs/ee 1.2.1.1. IBM has recommended we upgrade to gpfs 4.2.2.3 to fix a performance issue. I can't seem to find a support matrix that says what ltfs release we need for 4.2.2. Anybody have info on that? -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 484 bytes Desc: not available URL: From christof.schmitt at us.ibm.com Wed Mar 22 22:41:04 2017 From: christof.schmitt at us.ibm.com (Christof Schmitt) Date: Wed, 22 Mar 2017 15:41:04 -0700 Subject: [gpfsug-discuss] Configuration spectrum scale to use SFU schema In-Reply-To: <75a2acf9-e598-89c2-9546-6cedd80186c1@leeds.ac.uk> References: <8d1622e5-99dc-dad4-998b-b0d5a88c8f70@leeds.ac.uk> <75a2acf9-e598-89c2-9546-6cedd80186c1@leeds.ac.uk> Message-ID: > Thanks, would I need to re-run mmuserauth as per > https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.1/ > com.ibm.spectrum.scale.v4r21.doc/bl1adm_adwithrfc2307.htm > or would the schema change be needed after that when configuring the > authentication services? You would still need the domain join and the basic configuration done by mmuserauth. The easiest way would be running mmuserauth with --unix-domains as usually. The only change on top is then the switch to the old attribute format, which requires a winbindd restart to pick-up the change: mmuserauth service create ??data?access?method file --type ad ... /usr/lpp/mmfs/bin/net conf setparm global 'idmap config DS:schema_mode' sfu /usr/lpp/mmfs/bin/mmdsh -N CesNodes systemctl restart gpfs-winbind This also illustrates the support statement: The config through mmuserauth is supported, changing the Samba config under the hood is outside the normal support. Regards, Christof Schmitt || IBM || Spectrum Scale Development || Tucson, AZ christof.schmitt at us.ibm.com || +1-520-799-2469 (T/L: 321-2469) From kevindjo at us.ibm.com Thu Mar 23 13:35:24 2017 From: kevindjo at us.ibm.com (Kevin D Johnson) Date: Thu, 23 Mar 2017 13:35:24 +0000 Subject: [gpfsug-discuss] LTFS/EE release question.. In-Reply-To: <37386.1490217702@turing-police.cc.vt.edu> References: <37386.1490217702@turing-police.cc.vt.edu> Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: att66cs2.dat Type: application/octet-stream Size: 495 bytes Desc: not available URL: From olaf.weiser at de.ibm.com Thu Mar 23 14:24:38 2017 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Thu, 23 Mar 2017 15:24:38 +0100 Subject: [gpfsug-discuss] CES doesn't assign addresses to nodes In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From jonathon.anderson at colorado.edu Thu Mar 23 15:02:10 2017 From: jonathon.anderson at colorado.edu (Jonathon A Anderson) Date: Thu, 23 Mar 2017 15:02:10 +0000 Subject: [gpfsug-discuss] CES doesn't assign addresses to nodes In-Reply-To: References: Message-ID: Thanks! I?m looking forward to upgrading our CES nodes and resuming work on the project. ~jonathon On 3/23/17, 8:24 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Olaf Weiser" wrote: the issue is fixed, an APAR will be released soon - IV93100 From: Olaf Weiser/Germany/IBM at IBMDE To: "gpfsug main discussion list" Cc: "gpfsug main discussion list" Date: 01/31/2017 11:47 PM Subject: Re: [gpfsug-discuss] CES doesn't assign addresses to nodes Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________________ Yeah... depending on the #nodes you 're affected or not. ..... So if your remote ces cluster is small enough in terms of the #nodes ... you'll neuer hit into this issue Gesendet von IBM Verse Simon Thompson (Research Computing - IT Services) --- Re: [gpfsug-discuss] CES doesn't assign addresses to nodes --- Von:"Simon Thompson (Research Computing - IT Services)" An:"gpfsug main discussion list" Datum:Di. 31.01.2017 21:07Betreff:Re: [gpfsug-discuss] CES doesn't assign addresses to nodes ________________________________________ We use multicluster for our environment, storage systems in a separate cluster to hpc nodes on a separate cluster from protocol nodes. According to the docs, this isn't supported, but we haven't seen any issues. Note unsupported as opposed to broken. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Jonathon A Anderson [jonathon.anderson at colorado.edu] Sent: 31 January 2017 17:47 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] CES doesn't assign addresses to nodes Yeah, I searched around for places where ` tsctl shownodes up` appears in the GPFS code I have access to (i.e., the ksh and python stuff); but it?s only in CES. I suspect there just haven?t been that many people exporting CES out of an HPC cluster environment. ~jonathon From: on behalf of Olaf Weiser Reply-To: gpfsug main discussion list Date: Tuesday, January 31, 2017 at 10:45 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] CES doesn't assign addresses to nodes I ll open a pmr here for my env ... the issue may hurt you in a ces env. only... but needs to be fixed in core gpfs.base i thi k Gesendet von IBM Verse Jonathon A Anderson --- Re: [gpfsug-discuss] CES doesn't assign addresses to nodes --- Von: "Jonathon A Anderson" An: "gpfsug main discussion list" Datum: Di. 31.01.2017 17:32 Betreff: Re: [gpfsug-discuss] CES doesn't assign addresses to nodes ________________________________ No, I?m having trouble getting this through DDN support because, while we have a GPFS server license and GRIDScaler support, apparently we don?t have ?protocol node? support, so they?ve pushed back on supporting this as an overall CES-rooted effort. I do have a DDN case open, though: 78804. If you are (as I suspect) a GPFS developer, do you mind if I cite your info from here in my DDN case to get them to open a PMR? Thanks. ~jonathon From: on behalf of Olaf Weiser Reply-To: gpfsug main discussion list Date: Tuesday, January 31, 2017 at 8:42 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] CES doesn't assign addresses to nodes ok.. so obviously ... it seems , that we have several issues.. the 3983 characters is obviously a defect have you already raised a PMR , if so , can you send me the number ? From: Jonathon A Anderson To: gpfsug main discussion list Date: 01/31/2017 04:14 PM Subject: Re: [gpfsug-discuss] CES doesn't assign addresses to nodes Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ The tail isn?t the issue; that? my addition, so that I didn?t have to paste the hundred or so line nodelist into the thread. The actual command is tsctl shownodes up | $tr ',' '\n' | $sort -o $upnodefile But you can see in my tailed output that the last hostname listed is cut-off halfway through the hostname. Less obvious in the example, but true, is the fact that it?s only showing the first 120 hosts, when we have 403 nodes in our gpfs cluster. [root at sgate2 ~]# tsctl shownodes up | tr ',' '\n' | wc -l 120 [root at sgate2 ~]# mmlscluster | grep '\-opa' | wc -l 403 Perhaps more explicitly, it looks like `tsctl shownodes up` can only transmit 3983 characters. [root at sgate2 ~]# tsctl shownodes up | wc -c 3983 Again, I?m convinced this is a bug not only because the command doesn?t actually produce a list of all of the up nodes in our cluster; but because the last name listed is incomplete. [root at sgate2 ~]# tsctl shownodes up | tr ',' '\n' | tail -n 1 shas0260-opa.rc.int.col[root at sgate2 ~]# I?d continue my investigation within tsctl itself but, alas, it?s a binary with no source code available to me. :) I?m trying to get this opened as a bug / PMR; but I?m still working through the DDN support infrastructure. Thanks for reporting it, though. For the record: [root at sgate2 ~]# rpm -qa | grep -i gpfs gpfs.base-4.2.1-2.x86_64 gpfs.msg.en_US-4.2.1-2.noarch gpfs.gplbin-3.10.0-327.el7.x86_64-4.2.1-0.x86_64 gpfs.gskit-8.0.50-57.x86_64 gpfs.gpl-4.2.1-2.noarch nfs-ganesha-gpfs-2.3.2-0.ibm24.el7.x86_64 gpfs.ext-4.2.1-2.x86_64 gpfs.gplbin-3.10.0-327.36.3.el7.x86_64-4.2.1-2.x86_64 gpfs.docs-4.2.1-2.noarch ~jonathon From: on behalf of Olaf Weiser Reply-To: gpfsug main discussion list Date: Tuesday, January 31, 2017 at 1:30 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] CES doesn't assign addresses to nodes Hi ...same thing here.. everything after 10 nodes will be truncated.. though I don't have an issue with it ... I 'll open a PMR .. and I recommend you to do the same thing.. ;-) the reason seems simple.. it is the "| tail" .at the end of the command.. .. which truncates the output to the last 10 items... should be easy to fix.. cheers olaf From: Jonathon A Anderson To: "gpfsug-discuss at spectrumscale.org" Date: 01/30/2017 11:11 PM Subject: Re: [gpfsug-discuss] CES doesn't assign addresses to nodes Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ In trying to figure this out on my own, I?m relatively certain I?ve found a bug in GPFS related to the truncation of output from `tsctl shownodes up`. Any chance someone in development can confirm? Here are the details of my investigation: ## GPFS is up on sgate2 [root at sgate2 ~]# mmgetstate Node number Node name GPFS state ------------------------------------------ 414 sgate2-opa active ## but if I tell ces to explicitly put one of our ces addresses on that node, it says that GPFS is down [root at sgate2 ~]# mmces address move --ces-ip 10.225.71.102 --ces-node sgate2-opa mmces address move: GPFS is down on this node. mmces address move: Command failed. Examine previous error messages to determine cause. ## the ?GPFS is down on this node? message is defined as code 109 in mmglobfuncs [root at sgate2 ~]# grep --before-context=1 "GPFS is down on this node." /usr/lpp/mmfs/bin/mmglobfuncs 109 ) msgTxt=\ "%s: GPFS is down on this node." ## and is generated by printErrorMsg in mmcesnetmvaddress when it detects that the current node is identified as ?down? by getDownCesNodeList [root at sgate2 ~]# grep --before-context=5 'printErrorMsg 109' /usr/lpp/mmfs/bin/mmcesnetmvaddress downNodeList=$(getDownCesNodeList) for downNode in $downNodeList do if [[ $toNodeName == $downNode ]] then printErrorMsg 109 "$mmcmd" ## getDownCesNodeList is the intersection of all ces nodes with GPFS cluster nodes listed in `tsctl shownodes up` [root at sgate2 ~]# grep --after-context=16 '^function getDownCesNodeList' /usr/lpp/mmfs/bin/mmcesfuncs function getDownCesNodeList { typeset sourceFile="mmcesfuncs.sh" [[ -n $DEBUG || -n $DEBUGgetDownCesNodeList ]] &&set -x $mmTRACE_ENTER "$*" typeset upnodefile=${cmdTmpDir}upnodefile typeset downNodeList # get all CES nodes $sort -o $nodefile $mmfsCesNodes.dae $tsctl shownodes up | $tr ',' '\n' | $sort -o $upnodefile downNodeList=$($comm -23 $nodefile $upnodefile) print -- $downNodeList } #----- end of function getDownCesNodeList -------------------- ## but not only are the sgate nodes not listed by `tsctl shownodes up`; its output is obviously and erroneously truncated [root at sgate2 ~]# tsctl shownodes up | tr ',' '\n' | tail shas0251-opa.rc.int.colorado.edu shas0252-opa.rc.int.colorado.edu shas0253-opa.rc.int.colorado.edu shas0254-opa.rc.int.colorado.edu shas0255-opa.rc.int.colorado.edu shas0256-opa.rc.int.colorado.edu shas0257-opa.rc.int.colorado.edu shas0258-opa.rc.int.colorado.edu shas0259-opa.rc.int.colorado.edu shas0260-opa.rc.int.col[root at sgate2 ~]# ## I expect that this is a bug in GPFS, likely related to a maximum output buffer for `tsctl shownodes up`. On 1/24/17, 12:48 PM, "Jonathon A Anderson" wrote: I think I'm having the same issue described here: http://www.spectrumscale.org/pipermail/gpfsug-discuss/2016-October/002288.html Any advice or further troubleshooting steps would be much appreciated. Full disclosure: I also have a DDN case open. (78804) We've got a four-node (snsd{1..4}) DDN gridscaler system. I'm trying to add two CES protocol nodes (sgate{1,2}) to serve NFS. Here's the steps I took: --- mmcrnodeclass protocol -N sgate1-opa,sgate2-opa mmcrnodeclass nfs -N sgate1-opa,sgate2-opa mmchconfig cesSharedRoot=/gpfs/summit/ces mmchcluster --ccr-enable mmchnode --ces-enable -N protocol mmces service enable NFS mmces service start NFS -N nfs mmces address add --ces-ip 10.225.71.104,10.225.71.105 mmces address policy even-coverage mmces address move --rebalance --- This worked the very first time I ran it, but the CES addresses weren't re-distributed after restarting GPFS or a node reboot. Things I've tried: * disabling ces on the sgate nodes and re-running the above procedure * moving the cluster and filesystem managers to different snsd nodes * deleting and re-creating the cesSharedRoot directory Meanwhile, the following log entry appears in mmfs.log.latest every ~30s: --- Mon Jan 23 20:31:20 MST 2017: mmcesnetworkmonitor: Found unassigned address 10.225.71.104 Mon Jan 23 20:31:20 MST 2017: mmcesnetworkmonitor: Found unassigned address 10.225.71.105 Mon Jan 23 20:31:20 MST 2017: mmcesnetworkmonitor: handleNetworkProblem with lock held: assignIP 10.225.71.104_0-_+,10.225.71.105_0-_+ 1 Mon Jan 23 20:31:20 MST 2017: mmcesnetworkmonitor: Assigning addresses: 10.225.71.104_0-_+,10.225.71.105_0-_+ Mon Jan 23 20:31:20 MST 2017: mmcesnetworkmonitor: moveCesIPs: 10.225.71.104_0-_+,10.225.71.105_0-_+ --- Also notable, whenever I add or remove addresses now, I see this in mmsysmonitor.log (among a lot of other entries): --- 2017-01-23T20:40:56.363 sgate1 D ET_cesnetwork Entity state without requireUnique: ces_network_ips_down WARNING No CES relevant NICs detected - Service.calculateAndUpdateState:275 2017-01-23T20:40:11.364 sgate1 D ET_cesnetwork Update multiple entities at once {'p2p2': 1, 'bond0': 1, 'p2p1': 1} - Service.setLocalState:333 --- For the record, here's the interface I expect to get the address on sgate1: --- 11: bond0: mtu 9000 qdisc noqueue state UP link/ether 3c:fd:fe:08:a7:c0 brd ff:ff:ff:ff:ff:ff inet 10.225.71.107/20 brd 10.225.79.255 scope global bond0 valid_lft forever preferred_lft forever inet6 fe80::3efd:feff:fe08:a7c0/64 scope link valid_lft forever preferred_lft forever --- which is a bond of p2p1 and p2p2. --- 6: p2p1: mtu 9000 qdisc mq master bond0 state UP qlen 1000 link/ether 3c:fd:fe:08:a7:c0 brd ff:ff:ff:ff:ff:ff 7: p2p2: mtu 9000 qdisc mq master bond0 state UP qlen 1000 link/ether 3c:fd:fe:08:a7:c0 brd ff:ff:ff:ff:ff:ff --- A similar bond0 exists on sgate2. I crawled around in /usr/lpp/mmfs/lib/mmsysmon/CESNetworkService.py for a while trying to figure it out, but have been unsuccessful so far. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From mweil at wustl.edu Thu Mar 23 15:23:56 2017 From: mweil at wustl.edu (Matt Weil) Date: Thu, 23 Mar 2017 10:23:56 -0500 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: <47e18bb0062544f4b510ce0e87049fd3@jumptrading.com> References: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu> <47e18bb0062544f4b510ce0e87049fd3@jumptrading.com> Message-ID: <643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> FYI all We also ran into this after bumping maxFilesToCache. Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: unexpected error: Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many open files in system fix sysctl -w fs.file-max=1000000 On 3/22/17 12:01 PM, Bryan Banister wrote: > We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. > > Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil > Sent: Wednesday, March 22, 2017 11:43 AM > To: gpfsug main discussion list > Subject: [gpfsug-discuss] CES node slow to respond > > All, > > We had an indecent yesterday where one of our CES nodes slowed to a > crawl. GPFS waiters showed pre fetch threads going after inodes. > iohist also showed lots of inode fetching. Then we noticed that the CES > host had 5.4 million files open. > > The change I made was to set maxStatCache=DEFAULT because it is linux. > And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. > > Is there something else we should change as well. > > Thanks > > Matt > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. From bbanister at jumptrading.com Thu Mar 23 15:26:57 2017 From: bbanister at jumptrading.com (Bryan Banister) Date: Thu, 23 Mar 2017 15:26:57 +0000 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: <643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> References: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu> <47e18bb0062544f4b510ce0e87049fd3@jumptrading.com> <643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> Message-ID: Anybody from IBM willing/able to give us some explanation of why ganesha is holding open so many files? Is this expected/needed/etc? Or do we have to open a PMR to get some kind of explanation? -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Thursday, March 23, 2017 10:24 AM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] CES node slow to respond FYI all We also ran into this after bumping maxFilesToCache. Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: unexpected error: Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many open files in system fix sysctl -w fs.file-max=1000000 On 3/22/17 12:01 PM, Bryan Banister wrote: > We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. > > Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil > Sent: Wednesday, March 22, 2017 11:43 AM > To: gpfsug main discussion list > Subject: [gpfsug-discuss] CES node slow to respond > > All, > > We had an indecent yesterday where one of our CES nodes slowed to a > crawl. GPFS waiters showed pre fetch threads going after inodes. > iohist also showed lots of inode fetching. Then we noticed that the CES > host had 5.4 million files open. > > The change I made was to set maxStatCache=DEFAULT because it is linux. > And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. > > Is there something else we should change as well. > > Thanks > > Matt > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. From mweil at wustl.edu Thu Mar 23 15:35:30 2017 From: mweil at wustl.edu (Matt Weil) Date: Thu, 23 Mar 2017 10:35:30 -0500 Subject: [gpfsug-discuss] Dual homing CES nodes Message-ID: Hello all, Are there any issues with connecting CES nodes to multiple networks? Thanks Matt ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. From kevindjo at us.ibm.com Thu Mar 23 15:39:48 2017 From: kevindjo at us.ibm.com (Kevin D Johnson) Date: Thu, 23 Mar 2017 15:39:48 +0000 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: References: , <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu><47e18bb0062544f4b510ce0e87049fd3@jumptrading.com><643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> Message-ID: An HTML attachment was scrubbed... URL: From jonathon.anderson at colorado.edu Thu Mar 23 15:41:47 2017 From: jonathon.anderson at colorado.edu (Jonathon A Anderson) Date: Thu, 23 Mar 2017 15:41:47 +0000 Subject: [gpfsug-discuss] Dual homing CES nodes In-Reply-To: References: Message-ID: <0FE05202-B444-4EBB-994C-AB2CC229F1DA@colorado.edu> We?ve run CES with addresses on a different network than the GPFS admin or daemon interface; but I haven?t tried running CES with addresses on multiple networks itself. ~jonathon On 3/23/17, 9:35 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Matt Weil" wrote: Hello all, Are there any issues with connecting CES nodes to multiple networks? Thanks Matt ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From skylar2 at u.washington.edu Thu Mar 23 15:42:18 2017 From: skylar2 at u.washington.edu (Skylar Thompson) Date: Thu, 23 Mar 2017 15:42:18 +0000 Subject: [gpfsug-discuss] Dual homing CES nodes In-Reply-To: References: Message-ID: <20170323154218.wqjhjaszffinajy5@utumno.gs.washington.edu> The only thing I can think of is you should be careful that you have distinct CES groups so that addresses will failover to the right networks. On Thu, Mar 23, 2017 at 10:35:30AM -0500, Matt Weil wrote: > Hello all, > > Are there any issues with connecting CES nodes to multiple networks? -- -- Skylar Thompson (skylar2 at u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine From mweil at wustl.edu Thu Mar 23 15:44:47 2017 From: mweil at wustl.edu (Matt Weil) Date: Thu, 23 Mar 2017 10:44:47 -0500 Subject: [gpfsug-discuss] Dual homing CES nodes In-Reply-To: <20170323154218.wqjhjaszffinajy5@utumno.gs.washington.edu> References: <20170323154218.wqjhjaszffinajy5@utumno.gs.washington.edu> Message-ID: <43fe47d2-f3a7-2b52-4a21-b135c7141e9f@wustl.edu> yeah it seems to know which interface to use. here is my quick test.. no failure groups. > em2: flags=4099 mtu 1500 > inet 192.168.1.2 netmask 255.255.255.0 broadcast 192.168.1.255 > ether 84:2b:2b:47:70:35 txqueuelen 1000 (Ethernet) > RX packets 0 bytes 0 (0.0 B) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 0 bytes 0 (0.0 B) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > em2:0: flags=4099 mtu 1500 > inet 192.168.1.5 netmask 255.255.255.0 broadcast 192.168.1.255 > ether 84:2b:2b:47:70:35 txqueuelen 1000 (Ethernet) On 3/23/17 10:42 AM, Skylar Thompson wrote: > The only thing I can think of is you should be careful that you have > distinct CES groups so that addresses will failover to the right networks. > > On Thu, Mar 23, 2017 at 10:35:30AM -0500, Matt Weil wrote: >> Hello all, >> >> Are there any issues with connecting CES nodes to multiple networks? ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. From S.J.Thompson at bham.ac.uk Thu Mar 23 15:46:38 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Thu, 23 Mar 2017 15:46:38 +0000 Subject: [gpfsug-discuss] Dual homing CES nodes In-Reply-To: References: Message-ID: We've done this in an unsupported manner when we were testing connectivity to private VM networks - manually attach VXLAN interface to host, then enable CES on the segment. Worked fine for us in this, so don't see why it should;t work fine for real/VLAN interfaces. Just be mindful of asymmetric routing of course. Simon >Hello all, > >Are there any issues with connecting CES nodes to multiple networks? > >Thanks >Matt > >________________________________ >The materials in this message are private and may contain Protected >Healthcare Information or other information of a sensitive nature. If you >are not the intended recipient, be advised that any unauthorized use, >disclosure, copying or the taking of any action in reliance on the >contents of this information is strictly prohibited. If you have received >this email in error, please immediately notify the sender via telephone >or return mail. >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss From ulmer at ulmer.org Thu Mar 23 18:04:17 2017 From: ulmer at ulmer.org (Stephen Ulmer) Date: Thu, 23 Mar 2017 14:04:17 -0400 Subject: [gpfsug-discuss] GPFS on PowerVM Shared Storage Pools Message-ID: <91792F99-B4EB-4B96-9F89-01B30F49108A@ulmer.org> I have a client who is very enamored with PowerVM Shared Storage Pools (because they work great for them). Has anyone here implemented GPFS on such? I think that technically there?s no reason that it wouldn?t work, but I?ve never actually "owned" an installation with GPFS on SSPs. Any thoughts? NB: There is an entry on the SSP FAQ that lists "What disk types are NOT supported for Shared Storage Pool use?", among which GPFS is listed. I take that to mean that you can?t deploy an SSP on GPFS storage (not the other way around). This entry also lists iSCSI, for example, which is definitely not supported for SSP use. Liberty, -- Stephen From aaron.s.knister at nasa.gov Fri Mar 24 16:43:02 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 24 Mar 2017 12:43:02 -0400 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock Message-ID: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Since yesterday morning we've noticed some deadlocks on one of our filesystems that seem to be triggered by writing to it. The waiters on the clients look like this: 0x19450B0 ( 6730) waiting 2063.294589599 seconds, SyncHandlerThread: on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) (InodeFlushCondVar), reason 'waiting for the flush flag to commit metadata' 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 (0x7FFFDAC7FE28) (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRelinquishRegion on node 10.1.52.33 0x197EE70 ( 6776) waiting 0.000198354 seconds, FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRequestRegion on node 10.1.52.33 (10.1.52.33/c0n3271 is the fs manager for the filesystem in question) there's a single process running on this node writing to the filesystem in question (well, trying to write, it's been blocked doing nothing for half an hour now). There are ~10 other client nodes in this situation right now. We had many more last night before the problem seemed to disappear in the early hours of the morning and now its back. Waiters on the fs manager look like this. While the individual waiter is short it's a near constant stream: 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg handler allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg handler allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg handler allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) I don't know if this data point is useful but both yesterday and today the metadata NSDs for this filesystem have had a constant aggregate stream of 25MB/s 4kop/s reads during each episode (very low latency though so I don't believe the storage is a bottleneck here). Writes are only a few hundred ops and didn't strike me as odd. I have a PMR open for this but I'm curious if folks have seen this in the wild and what it might mean. -Aaron -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From jfosburg at mdanderson.org Fri Mar 24 16:50:03 2017 From: jfosburg at mdanderson.org (Fosburgh,Jonathan) Date: Fri, 24 Mar 2017 16:50:03 +0000 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: <1490374200.1901.0.camel@mdanderson.org> This is one of the more annoying long waiter problems. We've seen it several times and I'm not sure they all had the same cause. What version of GPFS? Do you have anything like Tivoli HSM or LTFSEE? -- Jonathan Fosburgh Principal Application Systems Analyst Storage Team IT Operations jfosburg at mdanderson.org (713) 745-9346 -----Original Message----- Date: Fri, 24 Mar 2017 12:43:02 -0400 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock To: gpfsug main discussion list > Reply-to: gpfsug main discussion list From: Aaron Knister > Since yesterday morning we've noticed some deadlocks on one of our filesystems that seem to be triggered by writing to it. The waiters on the clients look like this: 0x19450B0 ( 6730) waiting 2063.294589599 seconds, SyncHandlerThread: on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) (InodeFlushCondVar), reason 'waiting for the flush flag to commit metadata' 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 (0x7FFFDAC7FE28) (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRelinquishRegion on node 10.1.52.33 0x197EE70 ( 6776) waiting 0.000198354 seconds, FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRequestRegion on node 10.1.52.33 (10.1.52.33/c0n3271 is the fs manager for the filesystem in question) there's a single process running on this node writing to the filesystem in question (well, trying to write, it's been blocked doing nothing for half an hour now). There are ~10 other client nodes in this situation right now. We had many more last night before the problem seemed to disappear in the early hours of the morning and now its back. Waiters on the fs manager look like this. While the individual waiter is short it's a near constant stream: 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg handler allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg handler allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg handler allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) I don't know if this data point is useful but both yesterday and today the metadata NSDs for this filesystem have had a constant aggregate stream of 25MB/s 4kop/s reads during each episode (very low latency though so I don't believe the storage is a bottleneck here). Writes are only a few hundred ops and didn't strike me as odd. I have a PMR open for this but I'm curious if folks have seen this in the wild and what it might mean. -Aaron The information contained in this e-mail message may be privileged, confidential, and/or protected from disclosure. This e-mail message may contain protected health information (PHI); dissemination of PHI should comply with applicable federal and state laws. If you are not the intended recipient, or an authorized representative of the intended recipient, any further review, disclosure, use, dissemination, distribution, or copying of this message or any attachment (or the information contained therein) is strictly prohibited. If you think that you have received this e-mail message in error, please notify the sender by return e-mail and delete all references to it and its contents from your systems. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Fri Mar 24 16:50:47 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Fri, 24 Mar 2017 16:50:47 +0000 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock Message-ID: <29B113F2-BDE9-456A-BBEB-9B1C203CB5F4@nuance.com> Hi Aaron Yes, I have seen this several times over the last 6 months. I opened at least one PMR on it and they never could track it down. I did some snap dumps but without some traces, they did not have enough. I ended up getting out of it by selectively rebooting some of my NSD servers. My suspicion is that one of them had a thread deadlocked and was holding up IO to the rest of the filesystem. I haven?t seen it since I updated to 4.2.2-2, but I?m not convinced (yet) that it?s not lurking in the background. Bob Oesterlin Sr Principal Storage Engineer, Nuance On 3/24/17, 11:43 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Aaron Knister" wrote: Since yesterday morning we've noticed some deadlocks on one of our filesystems that seem to be triggered by writing to it. The waiters on the clients look like this: 0x19450B0 ( 6730) waiting 2063.294589599 seconds, SyncHandlerThread: on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) (InodeFlushCondVar), reason 'waiting for the flush flag to commit metadata' 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 (0x7FFFDAC7FE28) (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRelinquishRegion on node 10.1.52.33 0x197EE70 ( 6776) waiting 0.000198354 seconds, FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRequestRegion on node 10.1.52.33 (10.1.52.33/c0n3271 is the fs manager for the filesystem in question) there's a single process running on this node writing to the filesystem in question (well, trying to write, it's been blocked doing nothing for half an hour now). There are ~10 other client nodes in this situation right now. We had many more last night before the problem seemed to disappear in the early hours of the morning and now its back. Waiters on the fs manager look like this. While the individual waiter is short it's a near constant stream: 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg handler allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg handler allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg handler allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg handler allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) (AllocManagerMutex) I don't know if this data point is useful but both yesterday and today the metadata NSDs for this filesystem have had a constant aggregate stream of 25MB/s 4kop/s reads during each episode (very low latency though so I don't believe the storage is a bottleneck here). Writes are only a few hundred ops and didn't strike me as odd. I have a PMR open for this but I'm curious if folks have seen this in the wild and what it might mean. -Aaron -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=fUSq1Wdz1p_yJQVOgOoqe7fu7nsPXewvz8BUeKJyiRg&s=9sXk_xPuEtEyJgVhsZ7FCgM-rfytQGDDC2EyqwYgLhQ&e= From oehmes at gmail.com Fri Mar 24 17:04:02 2017 From: oehmes at gmail.com (Sven Oehme) Date: Fri, 24 Mar 2017 17:04:02 +0000 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: while this is happening run top and see if there is very high cpu utilization at this time on the NSD Server. if there is , run perf top (you might need to install perf command) and see if the top cpu contender is a spinlock . if so send a screenshot of perf top as i may know what that is and how to fix. sven On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister wrote: > Since yesterday morning we've noticed some deadlocks on one of our > filesystems that seem to be triggered by writing to it. The waiters on > the clients look like this: > > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, SyncHandlerThread: > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) (InodeFlushCondVar), reason > 'waiting for the flush flag to commit metadata' > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 (0x7FFFDAC7FE28) > (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRelinquishRegion > on node 10.1.52.33 > 0x197EE70 ( 6776) waiting 0.000198354 seconds, > FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for > allocMsgTypeRequestRegion on node 10.1.52.33 > > (10.1.52.33/c0n3271 is the fs manager for the filesystem in question) > > there's a single process running on this node writing to the filesystem > in question (well, trying to write, it's been blocked doing nothing for > half an hour now). There are ~10 other client nodes in this situation > right now. We had many more last night before the problem seemed to > disappear in the early hours of the morning and now its back. > > Waiters on the fs manager look like this. While the individual waiter is > short it's a near constant stream: > > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > I don't know if this data point is useful but both yesterday and today > the metadata NSDs for this filesystem have had a constant aggregate > stream of 25MB/s 4kop/s reads during each episode (very low latency > though so I don't believe the storage is a bottleneck here). Writes are > only a few hundred ops and didn't strike me as odd. > > I have a PMR open for this but I'm curious if folks have seen this in > the wild and what it might mean. > > -Aaron > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Fri Mar 24 17:07:58 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 24 Mar 2017 13:07:58 -0400 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: Hi Sven, Which NSD server should I run top on, the fs manager? If so the CPU load is about 155%. I'm working on perf top but not off to a great start... # perf top PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% [1000Hz cycles], (all, 28 CPUs) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Segmentation fault -Aaron On 3/24/17 1:04 PM, Sven Oehme wrote: > while this is happening run top and see if there is very high cpu > utilization at this time on the NSD Server. > > if there is , run perf top (you might need to install perf command) and > see if the top cpu contender is a spinlock . if so send a screenshot of > perf top as i may know what that is and how to fix. > > sven > > > On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister > wrote: > > Since yesterday morning we've noticed some deadlocks on one of our > filesystems that seem to be triggered by writing to it. The waiters on > the clients look like this: > > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, SyncHandlerThread: > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) (InodeFlushCondVar), reason > 'waiting for the flush flag to commit metadata' > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 (0x7FFFDAC7FE28) > (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRelinquishRegion > on node 10.1.52.33 > 0x197EE70 ( 6776) waiting 0.000198354 seconds, > FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for > allocMsgTypeRequestRegion on node 10.1.52.33 > > (10.1.52.33/c0n3271 is the fs manager > for the filesystem in question) > > there's a single process running on this node writing to the filesystem > in question (well, trying to write, it's been blocked doing nothing for > half an hour now). There are ~10 other client nodes in this situation > right now. We had many more last night before the problem seemed to > disappear in the early hours of the morning and now its back. > > Waiters on the fs manager look like this. While the individual waiter is > short it's a near constant stream: > > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > I don't know if this data point is useful but both yesterday and today > the metadata NSDs for this filesystem have had a constant aggregate > stream of 25MB/s 4kop/s reads during each episode (very low latency > though so I don't believe the storage is a bottleneck here). Writes are > only a few hundred ops and didn't strike me as odd. > > I have a PMR open for this but I'm curious if folks have seen this in > the wild and what it might mean. > > -Aaron > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From scale at us.ibm.com Fri Mar 24 17:17:33 2017 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Fri, 24 Mar 2017 09:17:33 -0800 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: References: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu><47e18bb0062544f4b510ce0e87049fd3@jumptrading.com><643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> Message-ID: Hi Bryan, Making sure Malahal's reply was received by the user group. >> Then we noticed that the CES host had 5.4 million files open This is technically not possible with ganesha alone. A process can only open 1 million files on RHEL distro. Either we have leaks in kernel or some other processes contributing to this. Ganesha does keep NFSv3 files open and keep open for performance. It doesn't have a good logic to close them after some inactivity. It does close them if the number is close to max open files which is configurable. PS: kNFS does open, read/write, and then close. No caching in older versions. They did have a feature in a recent code to cache open files in NFSv3 as well. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Bryan Banister To: gpfsug main discussion list Date: 03/23/2017 08:27 AM Subject: Re: [gpfsug-discuss] CES node slow to respond Sent by: gpfsug-discuss-bounces at spectrumscale.org Anybody from IBM willing/able to give us some explanation of why ganesha is holding open so many files? Is this expected/needed/etc? Or do we have to open a PMR to get some kind of explanation? -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Thursday, March 23, 2017 10:24 AM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] CES node slow to respond FYI all We also ran into this after bumping maxFilesToCache. Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: unexpected error: Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many open files in system fix sysctl -w fs.file-max=1000000 On 3/22/17 12:01 PM, Bryan Banister wrote: > We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. > > Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil > Sent: Wednesday, March 22, 2017 11:43 AM > To: gpfsug main discussion list > Subject: [gpfsug-discuss] CES node slow to respond > > All, > > We had an indecent yesterday where one of our CES nodes slowed to a > crawl. GPFS waiters showed pre fetch threads going after inodes. > iohist also showed lots of inode fetching. Then we noticed that the CES > host had 5.4 million files open. > > The change I made was to set maxStatCache=DEFAULT because it is linux. > And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. > > Is there something else we should change as well. > > Thanks > > Matt > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From oehmes at gmail.com Fri Mar 24 17:22:12 2017 From: oehmes at gmail.com (Sven Oehme) Date: Fri, 24 Mar 2017 17:22:12 +0000 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: you must be on sles as this segfaults only on sles to my knowledge :-) i am looking for a NSD or manager node in your cluster that runs at 100% cpu usage. do you have zimon deployed to look at cpu utilization across your nodes ? sven On Fri, Mar 24, 2017 at 10:08 AM Aaron Knister wrote: Hi Sven, Which NSD server should I run top on, the fs manager? If so the CPU load is about 155%. I'm working on perf top but not off to a great start... # perf top PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% [1000Hz cycles], (all, 28 CPUs) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Segmentation fault -Aaron On 3/24/17 1:04 PM, Sven Oehme wrote: > while this is happening run top and see if there is very high cpu > utilization at this time on the NSD Server. > > if there is , run perf top (you might need to install perf command) and > see if the top cpu contender is a spinlock . if so send a screenshot of > perf top as i may know what that is and how to fix. > > sven > > > On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister > wrote: > > Since yesterday morning we've noticed some deadlocks on one of our > filesystems that seem to be triggered by writing to it. The waiters on > the clients look like this: > > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, SyncHandlerThread: > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) (InodeFlushCondVar), reason > 'waiting for the flush flag to commit metadata' > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 (0x7FFFDAC7FE28) > (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRelinquishRegion > on node 10.1.52.33 > 0x197EE70 ( 6776) waiting 0.000198354 seconds, > FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for > allocMsgTypeRequestRegion on node 10.1.52.33 > > (10.1.52.33/c0n3271 is the fs manager > for the filesystem in question) > > there's a single process running on this node writing to the filesystem > in question (well, trying to write, it's been blocked doing nothing for > half an hour now). There are ~10 other client nodes in this situation > right now. We had many more last night before the problem seemed to > disappear in the early hours of the morning and now its back. > > Waiters on the fs manager look like this. While the individual waiter is > short it's a near constant stream: > > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > I don't know if this data point is useful but both yesterday and today > the metadata NSDs for this filesystem have had a constant aggregate > stream of 25MB/s 4kop/s reads during each episode (very low latency > though so I don't believe the storage is a bottleneck here). Writes are > only a few hundred ops and didn't strike me as odd. > > I have a PMR open for this but I'm curious if folks have seen this in > the wild and what it might mean. > > -Aaron > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Fri Mar 24 17:32:26 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 24 Mar 2017 13:32:26 -0400 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: heh, yep we're on sles :) here's a screenshot of the fs manager from the deadlocked filesystem. I don't think there's an nsd server or manager node that's running full throttle across all cpus. There is one that's got relatively high CPU utilization though (300-400%). I'll send a screenshot of it in a sec. no zimon yet but we do have other tools to see cpu utilization. -Aaron On 3/24/17 1:22 PM, Sven Oehme wrote: > you must be on sles as this segfaults only on sles to my knowledge :-) > > i am looking for a NSD or manager node in your cluster that runs at 100% > cpu usage. > > do you have zimon deployed to look at cpu utilization across your nodes ? > > sven > > > > On Fri, Mar 24, 2017 at 10:08 AM Aaron Knister > wrote: > > Hi Sven, > > Which NSD server should I run top on, the fs manager? If so the CPU load > is about 155%. I'm working on perf top but not off to a great start... > > # perf top > PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% [1000Hz > cycles], (all, 28 CPUs) > --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Segmentation fault > > -Aaron > > On 3/24/17 1:04 PM, Sven Oehme wrote: > > while this is happening run top and see if there is very high cpu > > utilization at this time on the NSD Server. > > > > if there is , run perf top (you might need to install perf command) and > > see if the top cpu contender is a spinlock . if so send a screenshot of > > perf top as i may know what that is and how to fix. > > > > sven > > > > > > On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister > > >> wrote: > > > > Since yesterday morning we've noticed some deadlocks on one of our > > filesystems that seem to be triggered by writing to it. The waiters on > > the clients look like this: > > > > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, SyncHandlerThread: > > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) (InodeFlushCondVar), reason > > 'waiting for the flush flag to commit metadata' > > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, > > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 (0x7FFFDAC7FE28) > > (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRelinquishRegion > > on node 10.1.52.33 > > 0x197EE70 ( 6776) waiting 0.000198354 seconds, > > FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 > > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for > > allocMsgTypeRequestRegion on node 10.1.52.33 > > > > (10.1.52.33/c0n3271 > is the fs manager > > for the filesystem in question) > > > > there's a single process running on this node writing to the filesystem > > in question (well, trying to write, it's been blocked doing nothing for > > half an hour now). There are ~10 other client nodes in this situation > > right now. We had many more last night before the problem seemed to > > disappear in the early hours of the morning and now its back. > > > > Waiters on the fs manager look like this. While the individual waiter is > > short it's a near constant stream: > > > > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg handler > > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > > (AllocManagerMutex) > > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg handler > > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg handler > > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > > (AllocManagerMutex) > > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg handler > > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > > (AllocManagerMutex) > > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg handler > > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg handler > > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > > (AllocManagerMutex) > > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg handler > > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > > (AllocManagerMutex) > > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg handler > > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > > > I don't know if this data point is useful but both yesterday and today > > the metadata NSDs for this filesystem have had a constant aggregate > > stream of 25MB/s 4kop/s reads during each episode (very low latency > > though so I don't believe the storage is a bottleneck here). Writes are > > only a few hundred ops and didn't strike me as odd. > > > > I have a PMR open for this but I'm curious if folks have seen this in > > the wild and what it might mean. > > > > -Aaron > > > > -- > > Aaron Knister > > NASA Center for Climate Simulation (Code 606.2) > > Goddard Space Flight Center > > (301) 286-2776 > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 -------------- next part -------------- A non-text attachment was scrubbed... Name: perf_top.png Type: image/png Size: 218901 bytes Desc: not available URL: From aaron.s.knister at nasa.gov Fri Mar 24 17:34:30 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 24 Mar 2017 13:34:30 -0400 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: Here's the screenshot from the other node with the high cpu utilization. On 3/24/17 1:32 PM, Aaron Knister wrote: > heh, yep we're on sles :) > > here's a screenshot of the fs manager from the deadlocked filesystem. I > don't think there's an nsd server or manager node that's running full > throttle across all cpus. There is one that's got relatively high CPU > utilization though (300-400%). I'll send a screenshot of it in a sec. > > no zimon yet but we do have other tools to see cpu utilization. > > -Aaron > > On 3/24/17 1:22 PM, Sven Oehme wrote: >> you must be on sles as this segfaults only on sles to my knowledge :-) >> >> i am looking for a NSD or manager node in your cluster that runs at 100% >> cpu usage. >> >> do you have zimon deployed to look at cpu utilization across your nodes ? >> >> sven >> >> >> >> On Fri, Mar 24, 2017 at 10:08 AM Aaron Knister > > wrote: >> >> Hi Sven, >> >> Which NSD server should I run top on, the fs manager? If so the >> CPU load >> is about 155%. I'm working on perf top but not off to a great >> start... >> >> # perf top >> PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% [1000Hz >> cycles], (all, 28 CPUs) >> >> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >> >> Segmentation fault >> >> -Aaron >> >> On 3/24/17 1:04 PM, Sven Oehme wrote: >> > while this is happening run top and see if there is very high cpu >> > utilization at this time on the NSD Server. >> > >> > if there is , run perf top (you might need to install perf >> command) and >> > see if the top cpu contender is a spinlock . if so send a >> screenshot of >> > perf top as i may know what that is and how to fix. >> > >> > sven >> > >> > >> > On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister >> >> > > >> wrote: >> > >> > Since yesterday morning we've noticed some deadlocks on one >> of our >> > filesystems that seem to be triggered by writing to it. The >> waiters on >> > the clients look like this: >> > >> > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, >> SyncHandlerThread: >> > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) >> (InodeFlushCondVar), reason >> > 'waiting for the flush flag to commit metadata' >> > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, >> > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 >> (0x7FFFDAC7FE28) >> > (MsgRecordCondvar), reason 'RPC wait' for >> allocMsgTypeRelinquishRegion >> > on node 10.1.52.33 >> > 0x197EE70 ( 6776) waiting 0.000198354 seconds, >> > FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 >> > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for >> > allocMsgTypeRequestRegion on node 10.1.52.33 >> > >> > (10.1.52.33/c0n3271 >> is the fs manager >> > for the filesystem in question) >> > >> > there's a single process running on this node writing to the >> filesystem >> > in question (well, trying to write, it's been blocked doing >> nothing for >> > half an hour now). There are ~10 other client nodes in this >> situation >> > right now. We had many more last night before the problem >> seemed to >> > disappear in the early hours of the morning and now its back. >> > >> > Waiters on the fs manager look like this. While the >> individual waiter is >> > short it's a near constant stream: >> > >> > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg >> handler >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) >> > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg >> handler >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) >> > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg >> handler >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) >> > >> > I don't know if this data point is useful but both yesterday >> and today >> > the metadata NSDs for this filesystem have had a constant >> aggregate >> > stream of 25MB/s 4kop/s reads during each episode (very low >> latency >> > though so I don't believe the storage is a bottleneck here). >> Writes are >> > only a few hundred ops and didn't strike me as odd. >> > >> > I have a PMR open for this but I'm curious if folks have >> seen this in >> > the wild and what it might mean. >> > >> > -Aaron >> > >> > -- >> > Aaron Knister >> > NASA Center for Climate Simulation (Code 606.2) >> > Goddard Space Flight Center >> > (301) 286-2776 >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > >> > >> > >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > >> >> -- >> Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) >> Goddard Space Flight Center >> (301) 286-2776 >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 -------------- next part -------------- A non-text attachment was scrubbed... Name: nsd35.png Type: image/png Size: 243009 bytes Desc: not available URL: From oehmes at gmail.com Fri Mar 24 17:41:10 2017 From: oehmes at gmail.com (Sven Oehme) Date: Fri, 24 Mar 2017 17:41:10 +0000 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: ok, that seems a different problem then i was thinking. can you send output of mmlscluster, mmlsconfig, mmlsfs all ? also are you getting close to fill grade on inodes or capacity on any of the filesystems ? sven On Fri, Mar 24, 2017 at 10:34 AM Aaron Knister wrote: > Here's the screenshot from the other node with the high cpu utilization. > > On 3/24/17 1:32 PM, Aaron Knister wrote: > > heh, yep we're on sles :) > > > > here's a screenshot of the fs manager from the deadlocked filesystem. I > > don't think there's an nsd server or manager node that's running full > > throttle across all cpus. There is one that's got relatively high CPU > > utilization though (300-400%). I'll send a screenshot of it in a sec. > > > > no zimon yet but we do have other tools to see cpu utilization. > > > > -Aaron > > > > On 3/24/17 1:22 PM, Sven Oehme wrote: > >> you must be on sles as this segfaults only on sles to my knowledge :-) > >> > >> i am looking for a NSD or manager node in your cluster that runs at 100% > >> cpu usage. > >> > >> do you have zimon deployed to look at cpu utilization across your nodes > ? > >> > >> sven > >> > >> > >> > >> On Fri, Mar 24, 2017 at 10:08 AM Aaron Knister < > aaron.s.knister at nasa.gov > >> > wrote: > >> > >> Hi Sven, > >> > >> Which NSD server should I run top on, the fs manager? If so the > >> CPU load > >> is about 155%. I'm working on perf top but not off to a great > >> start... > >> > >> # perf top > >> PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% [1000Hz > >> cycles], (all, 28 CPUs) > >> > >> > --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > >> > >> Segmentation fault > >> > >> -Aaron > >> > >> On 3/24/17 1:04 PM, Sven Oehme wrote: > >> > while this is happening run top and see if there is very high cpu > >> > utilization at this time on the NSD Server. > >> > > >> > if there is , run perf top (you might need to install perf > >> command) and > >> > see if the top cpu contender is a spinlock . if so send a > >> screenshot of > >> > perf top as i may know what that is and how to fix. > >> > > >> > sven > >> > > >> > > >> > On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister > >> > >> > >> >> wrote: > >> > > >> > Since yesterday morning we've noticed some deadlocks on one > >> of our > >> > filesystems that seem to be triggered by writing to it. The > >> waiters on > >> > the clients look like this: > >> > > >> > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, > >> SyncHandlerThread: > >> > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) > >> (InodeFlushCondVar), reason > >> > 'waiting for the flush flag to commit metadata' > >> > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, > >> > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 > >> (0x7FFFDAC7FE28) > >> > (MsgRecordCondvar), reason 'RPC wait' for > >> allocMsgTypeRelinquishRegion > >> > on node 10.1.52.33 > >> > 0x197EE70 ( 6776) waiting 0.000198354 seconds, > >> > FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 > >> > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for > >> > allocMsgTypeRequestRegion on node 10.1.52.33 > >> > > >> > (10.1.52.33/c0n3271 > >> is the fs manager > >> > for the filesystem in question) > >> > > >> > there's a single process running on this node writing to the > >> filesystem > >> > in question (well, trying to write, it's been blocked doing > >> nothing for > >> > half an hour now). There are ~10 other client nodes in this > >> situation > >> > right now. We had many more last night before the problem > >> seemed to > >> > disappear in the early hours of the morning and now its back. > >> > > >> > Waiters on the fs manager look like this. While the > >> individual waiter is > >> > short it's a near constant stream: > >> > > >> > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg > >> handler > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > >> (0xFFFFC9002163A2E0) > >> > (AllocManagerMutex) > >> > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg > >> handler > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > >> > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg > >> handler > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > >> (0xFFFFC9002163A2E0) > >> > (AllocManagerMutex) > >> > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg > >> handler > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > >> (0xFFFFC9002163A2E0) > >> > (AllocManagerMutex) > >> > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg > >> handler > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > >> > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg > >> handler > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > >> (0xFFFFC9002163A2E0) > >> > (AllocManagerMutex) > >> > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg > >> handler > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > >> (0xFFFFC9002163A2E0) > >> > (AllocManagerMutex) > >> > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg > >> handler > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > >> > > >> > I don't know if this data point is useful but both yesterday > >> and today > >> > the metadata NSDs for this filesystem have had a constant > >> aggregate > >> > stream of 25MB/s 4kop/s reads during each episode (very low > >> latency > >> > though so I don't believe the storage is a bottleneck here). > >> Writes are > >> > only a few hundred ops and didn't strike me as odd. > >> > > >> > I have a PMR open for this but I'm curious if folks have > >> seen this in > >> > the wild and what it might mean. > >> > > >> > -Aaron > >> > > >> > -- > >> > Aaron Knister > >> > NASA Center for Climate Simulation (Code 606.2) > >> > Goddard Space Flight Center > >> > (301) 286-2776 > >> > _______________________________________________ > >> > gpfsug-discuss mailing list > >> > gpfsug-discuss at spectrumscale.org > >> > >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > >> > > >> > > >> > _______________________________________________ > >> > gpfsug-discuss mailing list > >> > gpfsug-discuss at spectrumscale.org > >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > >> > >> -- > >> Aaron Knister > >> NASA Center for Climate Simulation (Code 606.2) > >> Goddard Space Flight Center > >> (301) 286-2776 > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > >> > >> > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Fri Mar 24 17:53:02 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 24 Mar 2017 13:53:02 -0400 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: <29B113F2-BDE9-456A-BBEB-9B1C203CB5F4@nuance.com> References: <29B113F2-BDE9-456A-BBEB-9B1C203CB5F4@nuance.com> Message-ID: Thanks Bob, Jonathan. We're running GPFS 4.1.1.10 and no HSM/LTFSEE. I'm currently gathering, as requested, a snap from all nodes (with traces). With 3500 nodes this ought to be entertaining. -Aaron On 3/24/17 12:50 PM, Oesterlin, Robert wrote: > Hi Aaron > > Yes, I have seen this several times over the last 6 months. I opened at least one PMR on it and they never could track it down. I did some snap dumps but without some traces, they did not have enough. I ended up getting out of it by selectively rebooting some of my NSD servers. My suspicion is that one of them had a thread deadlocked and was holding up IO to the rest of the filesystem. > > I haven?t seen it since I updated to 4.2.2-2, but I?m not convinced (yet) that it?s not lurking in the background. > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > > On 3/24/17, 11:43 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Aaron Knister" wrote: > > Since yesterday morning we've noticed some deadlocks on one of our > filesystems that seem to be triggered by writing to it. The waiters on > the clients look like this: > > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, SyncHandlerThread: > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) (InodeFlushCondVar), reason > 'waiting for the flush flag to commit metadata' > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 (0x7FFFDAC7FE28) > (MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRelinquishRegion > on node 10.1.52.33 > 0x197EE70 ( 6776) waiting 0.000198354 seconds, > FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for > allocMsgTypeRequestRegion on node 10.1.52.33 > > (10.1.52.33/c0n3271 is the fs manager for the filesystem in question) > > there's a single process running on this node writing to the filesystem > in question (well, trying to write, it's been blocked doing nothing for > half an hour now). There are ~10 other client nodes in this situation > right now. We had many more last night before the problem seemed to > disappear in the early hours of the morning and now its back. > > Waiters on the fs manager look like this. While the individual waiter is > short it's a near constant stream: > > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg handler > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0) > (AllocManagerMutex) > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg handler > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > I don't know if this data point is useful but both yesterday and today > the metadata NSDs for this filesystem have had a constant aggregate > stream of 25MB/s 4kop/s reads during each episode (very low latency > though so I don't believe the storage is a bottleneck here). Writes are > only a few hundred ops and didn't strike me as odd. > > I have a PMR open for this but I'm curious if folks have seen this in > the wild and what it might mean. > > -Aaron > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=fUSq1Wdz1p_yJQVOgOoqe7fu7nsPXewvz8BUeKJyiRg&s=9sXk_xPuEtEyJgVhsZ7FCgM-rfytQGDDC2EyqwYgLhQ&e= > > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From aaron.s.knister at nasa.gov Fri Mar 24 17:58:18 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 24 Mar 2017 13:58:18 -0400 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: I feel a little awkward about posting wlists of IP's and hostnames on the mailing list (even though they're all internal) but I'm happy to send to you directly. I've attached both an lsfs and an mmdf output of the fs in question here since that may be useful for others to see. Just a note about disk d23_02_021-- it's been evacuated for several weeks now due to a hardware issue in the disk enclosure. The fs is rather full percentage wise (93%) but in terms of capacity there's a good amount free. 93% full of a 7PB filesystem still leaves 551T. Metadata, as you'll see, is 31% free (roughly 800GB). The fs has 40M inodes allocated and 12M free. -Aaron On 3/24/17 1:41 PM, Sven Oehme wrote: > ok, that seems a different problem then i was thinking. > can you send output of mmlscluster, mmlsconfig, mmlsfs all ? > also are you getting close to fill grade on inodes or capacity on any of > the filesystems ? > > sven > > > On Fri, Mar 24, 2017 at 10:34 AM Aaron Knister > wrote: > > Here's the screenshot from the other node with the high cpu utilization. > > On 3/24/17 1:32 PM, Aaron Knister wrote: > > heh, yep we're on sles :) > > > > here's a screenshot of the fs manager from the deadlocked filesystem. I > > don't think there's an nsd server or manager node that's running full > > throttle across all cpus. There is one that's got relatively high CPU > > utilization though (300-400%). I'll send a screenshot of it in a sec. > > > > no zimon yet but we do have other tools to see cpu utilization. > > > > -Aaron > > > > On 3/24/17 1:22 PM, Sven Oehme wrote: > >> you must be on sles as this segfaults only on sles to my knowledge :-) > >> > >> i am looking for a NSD or manager node in your cluster that runs at 100% > >> cpu usage. > >> > >> do you have zimon deployed to look at cpu utilization across your nodes ? > >> > >> sven > >> > >> > >> > >> On Fri, Mar 24, 2017 at 10:08 AM Aaron Knister > >> >> wrote: > >> > >> Hi Sven, > >> > >> Which NSD server should I run top on, the fs manager? If so the > >> CPU load > >> is about 155%. I'm working on perf top but not off to a great > >> start... > >> > >> # perf top > >> PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% [1000Hz > >> cycles], (all, 28 CPUs) > >> > >> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > >> > >> Segmentation fault > >> > >> -Aaron > >> > >> On 3/24/17 1:04 PM, Sven Oehme wrote: > >> > while this is happening run top and see if there is very high cpu > >> > utilization at this time on the NSD Server. > >> > > >> > if there is , run perf top (you might need to install perf > >> command) and > >> > see if the top cpu contender is a spinlock . if so send a > >> screenshot of > >> > perf top as i may know what that is and how to fix. > >> > > >> > sven > >> > > >> > > >> > On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister > >> > > > >> > > >> >>> wrote: > >> > > >> > Since yesterday morning we've noticed some deadlocks on one > >> of our > >> > filesystems that seem to be triggered by writing to it. The > >> waiters on > >> > the clients look like this: > >> > > >> > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, > >> SyncHandlerThread: > >> > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) > >> (InodeFlushCondVar), reason > >> > 'waiting for the flush flag to commit metadata' > >> > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, > >> > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 > >> (0x7FFFDAC7FE28) > >> > (MsgRecordCondvar), reason 'RPC wait' for > >> allocMsgTypeRelinquishRegion > >> > on node 10.1.52.33 > >> > 0x197EE70 ( 6776) waiting 0.000198354 seconds, > >> > FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 > >> > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for > >> > allocMsgTypeRequestRegion on node 10.1.52.33 > >> > > >> > (10.1.52.33/c0n3271 > > >> is the fs manager > >> > for the filesystem in question) > >> > > >> > there's a single process running on this node writing to the > >> filesystem > >> > in question (well, trying to write, it's been blocked doing > >> nothing for > >> > half an hour now). There are ~10 other client nodes in this > >> situation > >> > right now. We had many more last night before the problem > >> seemed to > >> > disappear in the early hours of the morning and now its back. > >> > > >> > Waiters on the fs manager look like this. While the > >> individual waiter is > >> > short it's a near constant stream: > >> > > >> > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg > >> handler > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > >> (0xFFFFC9002163A2E0) > >> > (AllocManagerMutex) > >> > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg > >> handler > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > >> > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg > >> handler > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > >> (0xFFFFC9002163A2E0) > >> > (AllocManagerMutex) > >> > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg > >> handler > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > >> (0xFFFFC9002163A2E0) > >> > (AllocManagerMutex) > >> > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg > >> handler > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > >> > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg > >> handler > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > >> (0xFFFFC9002163A2E0) > >> > (AllocManagerMutex) > >> > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg > >> handler > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > >> (0xFFFFC9002163A2E0) > >> > (AllocManagerMutex) > >> > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg > >> handler > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > >> > > >> > I don't know if this data point is useful but both yesterday > >> and today > >> > the metadata NSDs for this filesystem have had a constant > >> aggregate > >> > stream of 25MB/s 4kop/s reads during each episode (very low > >> latency > >> > though so I don't believe the storage is a bottleneck here). > >> Writes are > >> > only a few hundred ops and didn't strike me as odd. > >> > > >> > I have a PMR open for this but I'm curious if folks have > >> seen this in > >> > the wild and what it might mean. > >> > > >> > -Aaron > >> > > >> > -- > >> > Aaron Knister > >> > NASA Center for Climate Simulation (Code 606.2) > >> > Goddard Space Flight Center > >> > (301) 286-2776 > > >> > _______________________________________________ > >> > gpfsug-discuss mailing list > >> > gpfsug-discuss at spectrumscale.org > >> > >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > >> > > >> > > >> > _______________________________________________ > >> > gpfsug-discuss mailing list > >> > gpfsug-discuss at spectrumscale.org > >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > >> > >> -- > >> Aaron Knister > >> NASA Center for Climate Simulation (Code 606.2) > >> Goddard Space Flight Center > >> (301) 286-2776 > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > >> > >> > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 -------------- next part -------------- disk disk size failure holds holds free KB free KB name in KB group metadata data in full blocks in fragments --------------- ------------- -------- -------- ----- -------------------- ------------------- Disks in storage pool: system (Maximum disk size allowed is 8.4 TB) m01_12_061 52428800 30 yes no 40660992 ( 78%) 689168 ( 1%) m01_12_062 52428800 30 yes no 40662528 ( 78%) 687168 ( 1%) m01_12_063 52428800 30 yes no 40657920 ( 78%) 694256 ( 1%) m01_12_064 52428800 30 yes no 40654080 ( 78%) 703464 ( 1%) m01_12_096 52428800 30 yes no 15453184 ( 29%) 1138392 ( 2%) m01_12_095 52428800 30 yes no 15514112 ( 30%) 1139072 ( 2%) m01_12_094 52428800 30 yes no 15425536 ( 29%) 1111776 ( 2%) m01_12_093 52428800 30 yes no 15340544 ( 29%) 1126584 ( 2%) m01_12_068 52428800 30 yes no 40716544 ( 78%) 688752 ( 1%) m01_12_067 52428800 30 yes no 40690944 ( 78%) 692200 ( 1%) m01_12_066 52428800 30 yes no 40687104 ( 78%) 692976 ( 1%) m01_12_065 52428800 30 yes no 40674304 ( 78%) 690848 ( 1%) m01_12_081 52428800 30 yes no 137728 ( 0%) 487760 ( 1%) m01_12_082 52428800 30 yes no 60416 ( 0%) 512632 ( 1%) m01_12_083 52428800 30 yes no 102144 ( 0%) 674152 ( 1%) m01_12_084 52428800 30 yes no 126208 ( 0%) 684296 ( 1%) m01_12_085 52428800 30 yes no 117504 ( 0%) 705952 ( 1%) m01_12_086 52428800 30 yes no 119296 ( 0%) 691056 ( 1%) m01_12_087 52428800 30 yes no 57344 ( 0%) 493992 ( 1%) m01_12_088 52428800 30 yes no 60672 ( 0%) 547360 ( 1%) m01_12_089 52428800 30 yes no 1455616 ( 3%) 888688 ( 2%) m01_12_090 52428800 30 yes no 1467392 ( 3%) 919312 ( 2%) m01_12_091 52428800 30 yes no 190464 ( 0%) 745456 ( 1%) m01_12_092 52428800 30 yes no 1367296 ( 3%) 771400 ( 1%) m02_22_081 52428800 40 yes no 1245696 ( 2%) 855992 ( 2%) m02_22_082 52428800 40 yes no 1261056 ( 2%) 869336 ( 2%) m02_22_083 52428800 40 yes no 1254912 ( 2%) 865656 ( 2%) m02_22_084 52428800 40 yes no 62464 ( 0%) 698480 ( 1%) m02_22_085 52428800 40 yes no 64256 ( 0%) 703016 ( 1%) m02_22_086 52428800 40 yes no 62208 ( 0%) 690032 ( 1%) m02_22_087 52428800 40 yes no 62464 ( 0%) 687584 ( 1%) m02_22_088 52428800 40 yes no 68608 ( 0%) 699848 ( 1%) m02_22_089 52428800 40 yes no 84480 ( 0%) 698144 ( 1%) m02_22_090 52428800 40 yes no 85248 ( 0%) 720216 ( 1%) m02_22_091 52428800 40 yes no 98816 ( 0%) 711824 ( 1%) m02_22_092 52428800 40 yes no 104448 ( 0%) 732808 ( 1%) m02_22_068 52428800 40 yes no 40727552 ( 78%) 702472 ( 1%) m02_22_067 52428800 40 yes no 40713728 ( 78%) 688576 ( 1%) m02_22_066 52428800 40 yes no 40694272 ( 78%) 700960 ( 1%) m02_22_065 52428800 40 yes no 40694016 ( 78%) 689936 ( 1%) m02_22_064 52428800 40 yes no 40683264 ( 78%) 695144 ( 1%) m02_22_063 52428800 40 yes no 40676864 ( 78%) 701288 ( 1%) m02_22_062 52428800 40 yes no 40670976 ( 78%) 692984 ( 1%) m02_22_061 52428800 40 yes no 40672512 ( 78%) 690024 ( 1%) m02_22_096 52428800 40 yes no 15327232 ( 29%) 1149064 ( 2%) m02_22_095 52428800 40 yes no 15363584 ( 29%) 1146384 ( 2%) m02_22_094 52428800 40 yes no 15397376 ( 29%) 1172856 ( 2%) m02_22_093 52428800 40 yes no 15374336 ( 29%) 1163832 ( 2%) ------------- -------------------- ------------------- (pool total) 2516582400 783850240 ( 31%) 37303168 ( 1%) Disks in storage pool: sp_1620 (Maximum disk size allowed is 5.4 PB) d23_01_001 46028292096 1620 no yes 3541176320 ( 8%) 37724768 ( 0%) d23_01_002 46028292096 1620 no yes 3542331392 ( 8%) 37545024 ( 0%) d23_01_003 46028292096 1620 no yes 3541968896 ( 8%) 37765344 ( 0%) d23_01_004 46028292096 1620 no yes 3544687616 ( 8%) 37720576 ( 0%) d23_01_005 46028292096 1620 no yes 3543368704 ( 8%) 37647456 ( 0%) d23_01_006 46028292096 1620 no yes 3542778880 ( 8%) 37695232 ( 0%) d23_01_007 46028292096 1620 no yes 3543220224 ( 8%) 37539712 ( 0%) d23_01_008 46028292096 1620 no yes 3540293632 ( 8%) 37548928 ( 0%) d23_01_009 46028292096 1620 no yes 3544590336 ( 8%) 37547424 ( 0%) d23_01_010 46028292096 1620 no yes 3542993920 ( 8%) 37865728 ( 0%) d23_01_011 46028292096 1620 no yes 3542859776 ( 8%) 37889408 ( 0%) d23_01_012 46028292096 1620 no yes 3542452224 ( 8%) 37721440 ( 0%) d23_01_013 46028292096 1620 no yes 3542286336 ( 8%) 37797824 ( 0%) d23_01_014 46028292096 1620 no yes 3543352320 ( 8%) 37647456 ( 0%) d23_01_015 46028292096 1620 no yes 3542906880 ( 8%) 37776960 ( 0%) d23_01_016 46028292096 1620 no yes 3540386816 ( 8%) 37521632 ( 0%) d23_01_017 46028292096 1620 no yes 3543212032 ( 8%) 37568480 ( 0%) d23_01_018 46028292096 1620 no yes 3542416384 ( 8%) 37467648 ( 0%) d23_01_019 46028292096 1620 no yes 3542659072 ( 8%) 37865344 ( 0%) d23_01_020 46028292096 1620 no yes 3542518784 ( 8%) 37623840 ( 0%) d23_01_021 46028292096 1620 no yes 3543202816 ( 8%) 37630848 ( 0%) d23_01_022 46028292096 1620 no yes 3544535040 ( 8%) 37723968 ( 0%) d23_01_023 46028292096 1620 no yes 3543248896 ( 8%) 37542656 ( 0%) d23_01_024 46028292096 1620 no yes 3541811200 ( 8%) 37775360 ( 0%) d23_01_025 46028292096 1620 no yes 3544839168 ( 8%) 37887744 ( 0%) d23_01_026 46028292096 1620 no yes 3542474752 ( 8%) 37820672 ( 0%) d23_01_027 46028292096 1620 no yes 3542050816 ( 8%) 37847296 ( 0%) d23_01_028 46028292096 1620 no yes 3540822016 ( 8%) 37578400 ( 0%) d23_01_029 46028292096 1620 no yes 3542011904 ( 8%) 37423328 ( 0%) d23_01_030 46028292096 1620 no yes 3542572032 ( 8%) 37751840 ( 0%) d23_01_031 46028292096 1620 no yes 3541582848 ( 8%) 37648896 ( 0%) d23_01_032 46028292096 1620 no yes 3542650880 ( 8%) 37715840 ( 0%) d23_01_033 46028292096 1620 no yes 3542251520 ( 8%) 37598432 ( 0%) d23_01_034 46028292096 1620 no yes 3542195200 ( 8%) 37582944 ( 0%) d23_01_035 46028292096 1620 no yes 3541298176 ( 8%) 37694848 ( 0%) d23_01_036 46028292096 1620 no yes 3541215232 ( 8%) 37869568 ( 0%) d23_01_037 46028292096 1620 no yes 3542111232 ( 8%) 37601088 ( 0%) d23_01_038 46028292096 1620 no yes 3541210112 ( 8%) 37474400 ( 0%) d23_01_039 46028292096 1620 no yes 3540457472 ( 8%) 37654656 ( 0%) d23_01_040 46028292096 1620 no yes 3541776384 ( 8%) 37645760 ( 0%) d23_01_041 46028292096 1620 no yes 3542624256 ( 8%) 37798880 ( 0%) d23_01_042 46028292096 1620 no yes 3541653504 ( 8%) 37595936 ( 0%) d23_01_043 46028292096 1620 no yes 3540583424 ( 8%) 37751936 ( 0%) d23_01_044 46028292096 1620 no yes 3542136832 ( 8%) 37793536 ( 0%) d23_01_045 46028292096 1620 no yes 3543443456 ( 8%) 37683872 ( 0%) d23_01_046 46028292096 1620 no yes 3540705280 ( 8%) 37896096 ( 0%) d23_01_047 46028292096 1620 no yes 3541550080 ( 8%) 37577760 ( 0%) d23_01_048 46028292096 1620 no yes 3542068224 ( 8%) 37724960 ( 0%) d23_01_049 46028292096 1620 no yes 3544568832 ( 8%) 37687264 ( 0%) d23_01_050 46028292096 1620 no yes 3543891968 ( 8%) 37737824 ( 0%) d23_01_051 46028292096 1620 no yes 3541944320 ( 8%) 37787904 ( 0%) d23_01_052 46028292096 1620 no yes 3542128640 ( 8%) 37960704 ( 0%) d23_01_053 46028292096 1620 no yes 3542494208 ( 8%) 37823104 ( 0%) d23_01_054 46028292096 1620 no yes 3541776384 ( 8%) 37652064 ( 0%) d23_01_055 46028292096 1620 no yes 3543655424 ( 8%) 37802656 ( 0%) d23_01_056 46028292096 1620 no yes 3541664768 ( 8%) 37694272 ( 0%) d23_01_057 46028292096 1620 no yes 3542197248 ( 8%) 37798272 ( 0%) d23_01_058 46028292096 1620 no yes 3543078912 ( 8%) 37740448 ( 0%) d23_01_059 46028292096 1620 no yes 3544783872 ( 8%) 37741248 ( 0%) d23_01_060 46028292096 1620 no yes 3542276096 ( 8%) 37818304 ( 0%) d23_01_061 46028292096 1620 no yes 3543452672 ( 8%) 37727104 ( 0%) d23_01_062 46028292096 1620 no yes 3543225344 ( 8%) 37754720 ( 0%) d23_01_063 46028292096 1620 no yes 3543173120 ( 8%) 37685280 ( 0%) d23_01_064 46028292096 1620 no yes 3541703680 ( 8%) 37711424 ( 0%) d23_01_065 46028292096 1620 no yes 3541797888 ( 8%) 37836992 ( 0%) d23_01_066 46028292096 1620 no yes 3542709248 ( 8%) 37780864 ( 0%) d23_01_067 46028292096 1620 no yes 3542996992 ( 8%) 37798976 ( 0%) d23_01_068 46028292096 1620 no yes 3542989824 ( 8%) 37672352 ( 0%) d23_01_069 46028292096 1620 no yes 3542004736 ( 8%) 37688608 ( 0%) d23_01_070 46028292096 1620 no yes 3541458944 ( 8%) 37648320 ( 0%) d23_01_071 46028292096 1620 no yes 3542049792 ( 8%) 37874368 ( 0%) d23_01_072 46028292096 1620 no yes 3541520384 ( 8%) 37650368 ( 0%) d23_01_073 46028292096 1620 no yes 3542274048 ( 8%) 37759776 ( 0%) d23_01_074 46028292096 1620 no yes 3541511168 ( 8%) 37569472 ( 0%) d23_01_075 46028292096 1620 no yes 3544001536 ( 8%) 37685952 ( 0%) d23_01_076 46028292096 1620 no yes 3543203840 ( 8%) 37690880 ( 0%) d23_01_077 46028292096 1620 no yes 3541925888 ( 8%) 37710848 ( 0%) d23_01_078 46028292096 1620 no yes 3543930880 ( 8%) 37588672 ( 0%) d23_01_079 46028292096 1620 no yes 3541520384 ( 8%) 37626432 ( 0%) d23_01_080 46028292096 1620 no yes 3541615616 ( 8%) 37796576 ( 0%) d23_01_081 46028292096 1620 no yes 3542212608 ( 8%) 37773056 ( 0%) d23_01_082 46028292096 1620 no yes 3541496832 ( 8%) 37863200 ( 0%) d23_01_083 46028292096 1620 no yes 3541881856 ( 8%) 37822016 ( 0%) d23_01_084 46028292096 1620 no yes 3543436288 ( 8%) 37838144 ( 0%) d23_02_001 46028292096 1620 no yes 3543580672 ( 8%) 37784480 ( 0%) d23_02_002 46028292096 1620 no yes 3541958656 ( 8%) 38029312 ( 0%) d23_02_003 46028292096 1620 no yes 3542037504 ( 8%) 37781888 ( 0%) d23_02_004 46028292096 1620 no yes 3541141504 ( 8%) 37535936 ( 0%) d23_02_005 46028292096 1620 no yes 3541710848 ( 8%) 37585504 ( 0%) d23_02_006 46028292096 1620 no yes 3542758400 ( 8%) 37699968 ( 0%) d23_02_007 46028292096 1620 no yes 3541051392 ( 8%) 37609824 ( 0%) d23_02_008 46028292096 1620 no yes 3541925888 ( 8%) 37791872 ( 0%) d23_02_009 46028292096 1620 no yes 3542461440 ( 8%) 37854464 ( 0%) d23_02_010 46028292096 1620 no yes 3544000512 ( 8%) 37642048 ( 0%) d23_02_011 46028292096 1620 no yes 3542000640 ( 8%) 37811840 ( 0%) d23_02_012 46028292096 1620 no yes 3543025664 ( 8%) 37802784 ( 0%) d23_02_013 46028292096 1620 no yes 3541744640 ( 8%) 37776608 ( 0%) d23_02_014 46028292096 1620 no yes 3542261760 ( 8%) 37699648 ( 0%) d23_02_015 46028292096 1620 no yes 3542729728 ( 8%) 37690944 ( 0%) d23_02_016 46028292096 1620 no yes 3543721984 ( 8%) 37657472 ( 0%) d23_02_017 46028292096 1620 no yes 3540802560 ( 8%) 37531328 ( 0%) d23_02_018 46028292096 1620 no yes 3542657024 ( 8%) 37860768 ( 0%) d23_02_019 46028292096 1620 no yes 3543438336 ( 8%) 37573760 ( 0%) d23_02_020 46028292096 1620 no yes 3543243776 ( 8%) 37662976 ( 0%) d23_02_021 46028292096 1620 no yes 46028197888 (100%) 32576 ( 0%) * d23_02_022 46028292096 1620 no yes 3543544832 ( 8%) 37620160 ( 0%) d23_02_023 46028292096 1620 no yes 3540716544 ( 8%) 37649536 ( 0%) d23_02_024 46028292096 1620 no yes 3542181888 ( 8%) 37553760 ( 0%) d23_02_025 46028292096 1620 no yes 3540486144 ( 8%) 37529376 ( 0%) d23_02_026 46028292096 1620 no yes 3541583872 ( 8%) 37833792 ( 0%) d23_02_027 46028292096 1620 no yes 3542169600 ( 8%) 37778912 ( 0%) d23_02_028 46028292096 1620 no yes 3541048320 ( 8%) 37696864 ( 0%) d23_02_029 46028292096 1620 no yes 3542336512 ( 8%) 37859264 ( 0%) d23_02_030 46028292096 1620 no yes 3542102016 ( 8%) 38059168 ( 0%) d23_02_031 46028292096 1620 no yes 3541311488 ( 8%) 37784480 ( 0%) d23_02_032 46028292096 1620 no yes 3542036480 ( 8%) 37783456 ( 0%) d23_02_033 46028292096 1620 no yes 3541478400 ( 8%) 37753792 ( 0%) d23_02_034 46028292096 1620 no yes 3540772864 ( 8%) 37725312 ( 0%) d23_02_035 46028292096 1620 no yes 3541840896 ( 8%) 37709664 ( 0%) d23_02_036 46028292096 1620 no yes 3542415360 ( 8%) 37580448 ( 0%) d23_02_037 46028292096 1620 no yes 3542515712 ( 8%) 37587808 ( 0%) d23_02_038 46028292096 1620 no yes 3541250048 ( 8%) 37550976 ( 0%) d23_02_039 46028292096 1620 no yes 3542627328 ( 8%) 37389952 ( 0%) d23_02_040 46028292096 1620 no yes 3541750784 ( 8%) 37709216 ( 0%) d23_02_041 46028292096 1620 no yes 3542558720 ( 8%) 37760288 ( 0%) d23_02_042 46028292096 1620 no yes 3540994048 ( 8%) 37491680 ( 0%) d23_02_043 46028292096 1620 no yes 3542491136 ( 8%) 37768576 ( 0%) d23_02_044 46028292096 1620 no yes 3542805504 ( 8%) 37638144 ( 0%) d23_02_045 46028292096 1620 no yes 3540658176 ( 8%) 37613120 ( 0%) d23_02_046 46028292096 1620 no yes 3543098368 ( 8%) 37920320 ( 0%) d23_02_047 46028292096 1620 no yes 3543087104 ( 8%) 37590432 ( 0%) d23_02_048 46028292096 1620 no yes 3541494784 ( 8%) 37468224 ( 0%) d23_02_049 46028292096 1620 no yes 3541920768 ( 8%) 37675968 ( 0%) d23_02_050 46028292096 1620 no yes 3542463488 ( 8%) 37670816 ( 0%) d23_02_051 46028292096 1620 no yes 3542427648 ( 8%) 37678176 ( 0%) d23_02_052 46028292096 1620 no yes 3539824640 ( 8%) 37590176 ( 0%) d23_02_053 46028292096 1620 no yes 3542251520 ( 8%) 37835200 ( 0%) d23_02_054 46028292096 1620 no yes 3541064704 ( 8%) 37636224 ( 0%) d23_02_055 46028292096 1620 no yes 3540130816 ( 8%) 37703360 ( 0%) d23_02_056 46028292096 1620 no yes 3545320448 ( 8%) 37767712 ( 0%) d23_02_057 46028292096 1620 no yes 3543144448 ( 8%) 37658208 ( 0%) d23_02_058 46028292096 1620 no yes 3541233664 ( 8%) 37720640 ( 0%) d23_02_059 46028292096 1620 no yes 3541435392 ( 8%) 37680896 ( 0%) d23_02_060 46028292096 1620 no yes 3542780928 ( 8%) 37567360 ( 0%) d23_02_061 46028292096 1620 no yes 3542073344 ( 8%) 37420992 ( 0%) d23_02_062 46028292096 1620 no yes 3543192576 ( 8%) 37575232 ( 0%) d23_02_063 46028292096 1620 no yes 3542048768 ( 8%) 37696192 ( 0%) d23_02_064 46028292096 1620 no yes 3541509120 ( 8%) 37536224 ( 0%) d23_02_065 46028292096 1620 no yes 3542299648 ( 8%) 37648736 ( 0%) d23_02_066 46028292096 1620 no yes 3541864448 ( 8%) 37710976 ( 0%) d23_02_067 46028292096 1620 no yes 3538870272 ( 8%) 37532288 ( 0%) d23_02_068 46028292096 1620 no yes 3541297152 ( 8%) 37829184 ( 0%) d23_02_069 46028292096 1620 no yes 3542179840 ( 8%) 37583424 ( 0%) d23_02_070 46028292096 1620 no yes 3541616640 ( 8%) 37742080 ( 0%) d23_02_071 46028292096 1620 no yes 3541706752 ( 8%) 37687328 ( 0%) d23_02_072 46028292096 1620 no yes 3542240256 ( 8%) 37582112 ( 0%) d23_02_073 46028292096 1620 no yes 3542102016 ( 8%) 37752736 ( 0%) d23_02_074 46028292096 1620 no yes 3542754304 ( 8%) 37687456 ( 0%) d23_02_075 46028292096 1620 no yes 3542194176 ( 8%) 37729344 ( 0%) d23_02_076 46028292096 1620 no yes 3542798336 ( 8%) 37672096 ( 0%) d23_02_077 46028292096 1620 no yes 3543918592 ( 8%) 37709024 ( 0%) d23_02_078 46028292096 1620 no yes 3540992000 ( 8%) 37540992 ( 0%) d23_02_079 46028292096 1620 no yes 3543330816 ( 8%) 37573888 ( 0%) d23_02_080 46028292096 1620 no yes 3543735296 ( 8%) 37818208 ( 0%) d23_02_081 46028292096 1620 no yes 3542054912 ( 8%) 37698016 ( 0%) d23_02_082 46028292096 1620 no yes 3540747264 ( 8%) 37643776 ( 0%) d23_02_083 46028292096 1620 no yes 3542846464 ( 8%) 37715680 ( 0%) d23_02_084 46028292096 1620 no yes 3542061056 ( 8%) 37703904 ( 0%) ------------- -------------------- ------------------- (pool total) 7686724780032 591558139904 ( 8%) 6295334976 ( 0%) ============= ==================== =================== (data) 7686724780032 591558139904 ( 8%) 6295334976 ( 0%) (metadata) 2516582400 783850240 ( 31%) 37303168 ( 1%) ============= ==================== =================== (total) 7689241362432 592341990144 ( 8%) 6332638144 ( 0%) Inode Information ----------------- Number of used inodes: 30247288 Number of free inodes: 11695752 Number of allocated inodes: 41943040 Maximum number of inodes: 41943040 -------------- next part -------------- flag value description ------------------- ------------------------ ----------------------------------- -f 8192 Minimum fragment size in bytes (system pool) 32768 Minimum fragment size in bytes (other pools) -i 512 Inode size in bytes -I 32768 Indirect block size in bytes -m 2 Default number of metadata replicas -M 2 Maximum number of metadata replicas -r 1 Default number of data replicas -R 2 Maximum number of data replicas -j cluster Block allocation type -D nfs4 File locking semantics in effect -k posix ACL semantics in effect -n 5000 Estimated number of nodes that will mount file system -B 262144 Block size (system pool) 1048576 Block size (other pools) -Q user;group;fileset Quotas accounting enabled user;group;fileset Quotas enforced none Default quotas enabled --perfileset-quota no Per-fileset quota enforcement --filesetdf no Fileset df enabled? -V 13.23 (3.5.0.7) File system version --create-time Fri Dec 6 15:23:28 2013 File system creation time -z no Is DMAPI enabled? -L 8388608 Logfile size -E yes Exact mtime mount option -S no Suppress atime mount option -K whenpossible Strict replica allocation option --fastea yes Fast external attributes enabled? --encryption no Encryption enabled? --inode-limit 41943040 Maximum number of inodes --log-replicas 0 Number of log replicas --is4KAligned no is4KAligned? --rapid-repair no rapidRepair enabled? --write-cache-threshold 0 HAWC Threshold (max 65536) -P system;sp_1620 Disk storage pools in file system -d m01_12_093;m01_12_094;m01_12_095;m01_12_096;m02_22_093;m02_22_094;m02_22_095;m02_22_096;m01_12_061;m01_12_062;m01_12_063;m01_12_064;m01_12_065;m01_12_066;m01_12_067; -d m01_12_068;m02_22_061;m02_22_062;m02_22_063;m02_22_064;m02_22_065;m02_22_066;m02_22_067;m02_22_068;m01_12_081;m01_12_082;m01_12_083;m01_12_084;m01_12_085;m01_12_086; -d m01_12_087;m01_12_088;m01_12_089;m01_12_090;m01_12_091;m01_12_092;m02_22_081;m02_22_082;m02_22_083;m02_22_084;m02_22_085;m02_22_086;m02_22_087;m02_22_088;m02_22_089; -d m02_22_090;m02_22_091;m02_22_092;d23_01_001;d23_01_002;d23_01_003;d23_01_004;d23_01_005;d23_01_006;d23_01_007;d23_01_008;d23_01_009;d23_01_010;d23_01_011;d23_01_012; -d d23_01_013;d23_01_014;d23_01_015;d23_01_016;d23_01_017;d23_01_018;d23_01_019;d23_01_020;d23_01_021;d23_01_022;d23_01_023;d23_01_024;d23_01_025;d23_01_026;d23_01_027; -d d23_01_028;d23_01_029;d23_01_030;d23_01_031;d23_01_032;d23_01_033;d23_01_034;d23_01_035;d23_01_036;d23_01_037;d23_01_038;d23_01_039;d23_01_040;d23_01_041;d23_01_042; -d d23_01_043;d23_01_044;d23_01_045;d23_01_046;d23_01_047;d23_01_048;d23_01_049;d23_01_050;d23_01_051;d23_01_052;d23_01_053;d23_01_054;d23_01_055;d23_01_056;d23_01_057; -d d23_01_058;d23_01_059;d23_01_060;d23_01_061;d23_01_062;d23_01_063;d23_01_064;d23_01_065;d23_01_066;d23_01_067;d23_01_068;d23_01_069;d23_01_070;d23_01_071;d23_01_072; -d d23_01_073;d23_01_074;d23_01_075;d23_01_076;d23_01_077;d23_01_078;d23_01_079;d23_01_080;d23_01_081;d23_01_082;d23_01_083;d23_01_084;d23_02_001;d23_02_002;d23_02_003; -d d23_02_004;d23_02_005;d23_02_006;d23_02_007;d23_02_008;d23_02_009;d23_02_010;d23_02_011;d23_02_012;d23_02_013;d23_02_014;d23_02_015;d23_02_016;d23_02_017;d23_02_018; -d d23_02_019;d23_02_020;d23_02_021;d23_02_022;d23_02_023;d23_02_024;d23_02_025;d23_02_026;d23_02_027;d23_02_028;d23_02_029;d23_02_030;d23_02_031;d23_02_032;d23_02_033; -d d23_02_034;d23_02_035;d23_02_036;d23_02_037;d23_02_038;d23_02_039;d23_02_040;d23_02_041;d23_02_042;d23_02_043;d23_02_044;d23_02_045;d23_02_046;d23_02_047;d23_02_048; -d d23_02_049;d23_02_050;d23_02_051;d23_02_052;d23_02_053;d23_02_054;d23_02_055;d23_02_056;d23_02_057;d23_02_058;d23_02_059;d23_02_060;d23_02_061;d23_02_062;d23_02_063; -d d23_02_064;d23_02_065;d23_02_066;d23_02_067;d23_02_068;d23_02_069;d23_02_070;d23_02_071;d23_02_072;d23_02_073;d23_02_074;d23_02_075;d23_02_076;d23_02_077;d23_02_078; -d d23_02_079;d23_02_080;d23_02_081;d23_02_082;d23_02_083;d23_02_084 Disks in file system -A no Automatic mount option -o nodev,nosuid Additional mount options -T /gpfsm/dnb03 Default mount point --mount-priority 0 Mount priority From jfosburg at mdanderson.org Fri Mar 24 18:03:22 2017 From: jfosburg at mdanderson.org (Fosburgh,Jonathan) Date: Fri, 24 Mar 2017 18:03:22 +0000 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: <1490378600.1901.4.camel@mdanderson.org> 7PB filesystem and only 28 million inodes in use? What is your average file size? Our large filesystem is 7.5P (currently 71% used) with over 1 billion inodes in use. -- Jonathan Fosburgh Principal Application Systems Analyst Storage Team IT Operations jfosburg at mdanderson.org (713) 745-9346 -----Original Message----- Date: Fri, 24 Mar 2017 13:58:18 -0400 Subject: Re: [gpfsug-discuss] strange waiters + filesystem deadlock To: gpfsug-discuss at spectrumscale.org Reply-to: gpfsug main discussion list From: Aaron Knister > I feel a little awkward about posting wlists of IP's and hostnames on the mailing list (even though they're all internal) but I'm happy to send to you directly. I've attached both an lsfs and an mmdf output of the fs in question here since that may be useful for others to see. Just a note about disk d23_02_021-- it's been evacuated for several weeks now due to a hardware issue in the disk enclosure. The fs is rather full percentage wise (93%) but in terms of capacity there's a good amount free. 93% full of a 7PB filesystem still leaves 551T. Metadata, as you'll see, is 31% free (roughly 800GB). The fs has 40M inodes allocated and 12M free. -Aaron On 3/24/17 1:41 PM, Sven Oehme wrote: ok, that seems a different problem then i was thinking. can you send output of mmlscluster, mmlsconfig, mmlsfs all ? also are you getting close to fill grade on inodes or capacity on any of the filesystems ? sven On Fri, Mar 24, 2017 at 10:34 AM Aaron Knister > wrote: Here's the screenshot from the other node with the high cpu utilization. On 3/24/17 1:32 PM, Aaron Knister wrote: > heh, yep we're on sles :) > > here's a screenshot of the fs manager from the deadlocked filesystem. I > don't think there's an nsd server or manager node that's running full > throttle across all cpus. There is one that's got relatively high CPU > utilization though (300-400%). I'll send a screenshot of it in a sec. > > no zimon yet but we do have other tools to see cpu utilization. > > -Aaron > > On 3/24/17 1:22 PM, Sven Oehme wrote: >> you must be on sles as this segfaults only on sles to my knowledge :-) >> >> i am looking for a NSD or manager node in your cluster that runs at 100% >> cpu usage. >> >> do you have zimon deployed to look at cpu utilization across your nodes ? >> >> sven >> >> >> >> On Fri, Mar 24, 2017 at 10:08 AM Aaron Knister >> >> wrote: >> >> Hi Sven, >> >> Which NSD server should I run top on, the fs manager? If so the >> CPU load >> is about 155%. I'm working on perf top but not off to a great >> start... >> >> # perf top >> PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% [1000Hz >> cycles], (all, 28 CPUs) >> >> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >> >> Segmentation fault >> >> -Aaron >> >> On 3/24/17 1:04 PM, Sven Oehme wrote: >> > while this is happening run top and see if there is very high cpu >> > utilization at this time on the NSD Server. >> > >> > if there is , run perf top (you might need to install perf >> command) and >> > see if the top cpu contender is a spinlock . if so send a >> screenshot of >> > perf top as i may know what that is and how to fix. >> > >> > sven >> > >> > >> > On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister >> > >> > >> >>> wrote: >> > >> > Since yesterday morning we've noticed some deadlocks on one >> of our >> > filesystems that seem to be triggered by writing to it. The >> waiters on >> > the clients look like this: >> > >> > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, >> SyncHandlerThread: >> > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) >> (InodeFlushCondVar), reason >> > 'waiting for the flush flag to commit metadata' >> > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, >> > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 >> (0x7FFFDAC7FE28) >> > (MsgRecordCondvar), reason 'RPC wait' for >> allocMsgTypeRelinquishRegion >> > on node 10.1.52.33 >> > 0x197EE70 ( 6776) waiting 0.000198354 seconds, >> > FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 >> > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for >> > allocMsgTypeRequestRegion on node 10.1.52.33 >> > >> > (10.1.52.33/c0n3271 >> is the fs manager >> > for the filesystem in question) >> > >> > there's a single process running on this node writing to the >> filesystem >> > in question (well, trying to write, it's been blocked doing >> nothing for >> > half an hour now). There are ~10 other client nodes in this >> situation >> > right now. We had many more last night before the problem >> seemed to >> > disappear in the early hours of the morning and now its back. >> > >> > Waiters on the fs manager look like this. While the >> individual waiter is >> > short it's a near constant stream: >> > >> > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg >> handler >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) >> > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg >> handler >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) >> > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg >> handler >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) >> > >> > I don't know if this data point is useful but both yesterday >> and today >> > the metadata NSDs for this filesystem have had a constant >> aggregate >> > stream of 25MB/s 4kop/s reads during each episode (very low >> latency >> > though so I don't believe the storage is a bottleneck here). >> Writes are >> > only a few hundred ops and didn't strike me as odd. >> > >> > I have a PMR open for this but I'm curious if folks have >> seen this in >> > the wild and what it might mean. >> > >> > -Aaron >> > >> > -- >> > Aaron Knister >> > NASA Center for Climate Simulation (Code 606.2) >> > Goddard Space Flight Center >> > (301) 286-2776 >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > >> > >> > >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > >> >> -- >> Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) >> Goddard Space Flight Center >> (301) 286-2776 >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss The information contained in this e-mail message may be privileged, confidential, and/or protected from disclosure. This e-mail message may contain protected health information (PHI); dissemination of PHI should comply with applicable federal and state laws. If you are not the intended recipient, or an authorized representative of the intended recipient, any further review, disclosure, use, dissemination, distribution, or copying of this message or any attachment (or the information contained therein) is strictly prohibited. If you think that you have received this e-mail message in error, please notify the sender by return e-mail and delete all references to it and its contents from your systems. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Fri Mar 24 18:05:26 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 24 Mar 2017 14:05:26 -0400 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: <1490378600.1901.4.camel@mdanderson.org> References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> <1490378600.1901.4.camel@mdanderson.org> Message-ID: <5e630e9a-5f6f-5def-7e92-5bc7c6c01a4f@nasa.gov> It's large, I do know that much. I'll defer to one of our other storage admins. Jordan, do you have that number handy? -Aaron On 3/24/17 2:03 PM, Fosburgh,Jonathan wrote: > 7PB filesystem and only 28 million inodes in use? What is your average > file size? Our large filesystem is 7.5P (currently 71% used) with over 1 > billion inodes in use. > > -- > > Jonathan Fosburgh > Principal Application Systems Analyst > Storage Team > IT Operations > jfosburg at mdanderson.org > (713) 745-9346 > > -----Original Message----- > > *Date*: Fri, 24 Mar 2017 13:58:18 -0400 > *Subject*: Re: [gpfsug-discuss] strange waiters + filesystem deadlock > *To*: gpfsug-discuss at spectrumscale.org > > Reply-to: gpfsug main discussion list > *From*: Aaron Knister > > > I feel a little awkward about posting wlists of IP's and hostnames on > the mailing list (even though they're all internal) but I'm happy to > send to you directly. I've attached both an lsfs and an mmdf output of > the fs in question here since that may be useful for others to see. Just > a note about disk d23_02_021-- it's been evacuated for several weeks now > due to a hardware issue in the disk enclosure. > > The fs is rather full percentage wise (93%) but in terms of capacity > there's a good amount free. 93% full of a 7PB filesystem still leaves > 551T. Metadata, as you'll see, is 31% free (roughly 800GB). > > The fs has 40M inodes allocated and 12M free. > > -Aaron > > On 3/24/17 1:41 PM, Sven Oehme wrote: >> ok, that seems a different problem then i was thinking. can you send >> output of mmlscluster, mmlsconfig, mmlsfs all ? also are you getting >> close to fill grade on inodes or capacity on any of the filesystems ? >> sven On Fri, Mar 24, 2017 at 10:34 AM Aaron Knister >> >> > wrote: Here's the screenshot from >> the other node with the high cpu utilization. On 3/24/17 1:32 PM, >> Aaron Knister wrote: > heh, yep we're on sles :) > > here's a >> screenshot of the fs manager from the deadlocked filesystem. I > don't >> think there's an nsd server or manager node that's running full > >> throttle across all cpus. There is one that's got relatively high CPU >> > utilization though (300-400%). I'll send a screenshot of it in a >> sec. > > no zimon yet but we do have other tools to see cpu >> utilization. > > -Aaron > > On 3/24/17 1:22 PM, Sven Oehme wrote: >> >> you must be on sles as this segfaults only on sles to my knowledge :-) >> >> >> i am looking for a NSD or manager node in your cluster that runs >> at 100% >> cpu usage. >> >> do you have zimon deployed to look at cpu >> utilization across your nodes ? >> >> sven >> >> >> >> On Fri, Mar 24, >> 2017 at 10:08 AM Aaron Knister > >> >> >> >> wrote: >> >> Hi Sven, >> >> Which NSD server should I run top on, the >> fs manager? If so the >> CPU load >> is about 155%. I'm working on >> perf top but not off to a great >> start... >> >> # perf top >> >> PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% [1000Hz >> cycles], >> (all, 28 CPUs) >> >> >> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >> >> >> Segmentation fault >> >> -Aaron >> >> On 3/24/17 1:04 PM, Sven >> Oehme wrote: >> > while this is happening run top and see if there is >> very high cpu >> > utilization at this time on the NSD Server. >> > >> >> > if there is , run perf top (you might need to install perf >> >> command) and >> > see if the top cpu contender is a spinlock . if so >> send a >> screenshot of >> > perf top as i may know what that is and >> how to fix. >> > >> > sven >> > >> > >> > On Fri, Mar 24, 2017 at 9:43 >> AM Aaron Knister >> > >> > >> >> > >> >> > >>> wrote: >> > >> > Since yesterday >> morning we've noticed some deadlocks on one >> of our >> > filesystems >> that seem to be triggered by writing to it. The >> waiters on >> > the >> clients look like this: >> > >> > 0x19450B0 ( 6730) waiting >> 2063.294589599 seconds, >> SyncHandlerThread: >> > on ThCond >> 0x1802585CB10 (0xFFFFC9002585CB10) >> (InodeFlushCondVar), reason >> > >> 'waiting for the flush flag to commit metadata' >> > 0x7FFFDA65E200 ( >> 22850) waiting 0.000246257 seconds, >> > AllocReduceHelperThread: on >> ThCond 0x7FFFDAC7FE28 >> (0x7FFFDAC7FE28) >> > (MsgRecordCondvar), >> reason 'RPC wait' for >> allocMsgTypeRelinquishRegion >> > on node >> 10.1.52.33 >> > 0x197EE70 ( 6776) waiting 0.000198354 >> seconds, >> > FileBlockWriteFetchHandlerThread: on ThCond >> 0x7FFFF00CD598 >> > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC >> wait' for >> > allocMsgTypeRequestRegion on node 10.1.52.33 >> >> > >> > (10.1.52.33/c0n3271 >> >> is the fs >> manager >> > for the filesystem in question) >> > >> > there's a >> single process running on this node writing to the >> filesystem >> > >> in question (well, trying to write, it's been blocked doing >> nothing >> for >> > half an hour now). There are ~10 other client nodes in this >> >> situation >> > right now. We had many more last night before the >> problem >> seemed to >> > disappear in the early hours of the morning >> and now its back. >> > >> > Waiters on the fs manager look like this. >> While the >> individual waiter is >> > short it's a near constant >> stream: >> > >> > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, >> Msg >> handler >> > allocMsgTypeRequestRegion: on ThMutex >> 0x1802163A2E0 >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > >> 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg >> handler >> >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 >> > >> (0xFFFFC9002163A2E0) (AllocManagerMutex) >> > 0x7FFF91C10080 ( 14723) >> waiting 0.000959649 seconds, Msg >> handler >> > >> allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFFB03C2910 ( >> 12636) waiting 0.000769611 seconds, Msg >> handler >> > >> allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF8C092850 ( >> 18215) waiting 0.000682275 seconds, Msg >> handler >> > >> allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 >> > >> (0xFFFFC9002163A2E0) (AllocManagerMutex) >> > 0x7FFF9423F730 ( 12652) >> waiting 0.000641915 seconds, Msg >> handler >> > >> allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF9422D770 ( >> 12625) waiting 0.000494256 seconds, Msg >> handler >> > >> allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 >> >> (0xFFFFC9002163A2E0) >> > (AllocManagerMutex) >> > 0x7FFF9423E310 ( >> 12651) waiting 0.000437760 seconds, Msg >> handler >> > >> allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 >> > >> (0xFFFFC9002163A2E0) (AllocManagerMutex) >> > >> > I don't know if >> this data point is useful but both yesterday >> and today >> > the >> metadata NSDs for this filesystem have had a constant >> aggregate >> >> > stream of 25MB/s 4kop/s reads during each episode (very low >> >> latency >> > though so I don't believe the storage is a bottleneck >> here). >> Writes are >> > only a few hundred ops and didn't strike me >> as odd. >> > >> > I have a PMR open for this but I'm curious if folks >> have >> seen this in >> > the wild and what it might mean. >> > >> > >> -Aaron >> > >> > -- >> > Aaron Knister >> > NASA Center for Climate >> Simulation (Code 606.2) >> > Goddard Space Flight Center >> > (301) >> 286-2776 >> >> > >> _______________________________________________ >> > gpfsug-discuss >> mailing list >> > gpfsug-discuss at spectrumscale.org >> >> >> >> > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > >> > >> > >> > >> _______________________________________________ >> > gpfsug-discuss >> mailing list >> > gpfsug-discuss at spectrumscale.org >> >> > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > >> >> -- >> >> Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) >> >> Goddard Space Flight Center >> (301) 286-2776 >> >> >> _______________________________________________ >> gpfsug-discuss >> mailing list >> gpfsug-discuss at spectrumscale.org >> >> >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> >> >> _______________________________________________ >> gpfsug-discuss >> mailing list >> gpfsug-discuss at spectrumscale.org >> >> >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > > > >> _______________________________________________ > gpfsug-discuss >> mailing list > gpfsug-discuss at spectrumscale.org >> > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight >> Center (301) 286-2776 >> _______________________________________________ gpfsug-discuss mailing >> list gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> _______________________________________________ gpfsug-discuss mailing >> list gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > The information contained in this e-mail message may be privileged, > confidential, and/or protected from disclosure. This e-mail message may > contain protected health information (PHI); dissemination of PHI should > comply with applicable federal and state laws. If you are not the > intended recipient, or an authorized representative of the intended > recipient, any further review, disclosure, use, dissemination, > distribution, or copying of this message or any attachment (or the > information contained therein) is strictly prohibited. If you think that > you have received this e-mail message in error, please notify the sender > by return e-mail and delete all references to it and its contents from > your systems. > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From oehmes at gmail.com Fri Mar 24 18:05:58 2017 From: oehmes at gmail.com (Sven Oehme) Date: Fri, 24 Mar 2017 18:05:58 +0000 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: was this filesystem creates with -n 5000 ? or was that changed later with mmchfs ? please send the mmlsconfig/mmlscluster output to me at oehmes at us.ibm.com On Fri, Mar 24, 2017 at 10:58 AM Aaron Knister wrote: > I feel a little awkward about posting wlists of IP's and hostnames on > the mailing list (even though they're all internal) but I'm happy to > send to you directly. I've attached both an lsfs and an mmdf output of > the fs in question here since that may be useful for others to see. Just > a note about disk d23_02_021-- it's been evacuated for several weeks now > due to a hardware issue in the disk enclosure. > > The fs is rather full percentage wise (93%) but in terms of capacity > there's a good amount free. 93% full of a 7PB filesystem still leaves > 551T. Metadata, as you'll see, is 31% free (roughly 800GB). > > The fs has 40M inodes allocated and 12M free. > > -Aaron > > On 3/24/17 1:41 PM, Sven Oehme wrote: > > ok, that seems a different problem then i was thinking. > > can you send output of mmlscluster, mmlsconfig, mmlsfs all ? > > also are you getting close to fill grade on inodes or capacity on any of > > the filesystems ? > > > > sven > > > > > > On Fri, Mar 24, 2017 at 10:34 AM Aaron Knister > > wrote: > > > > Here's the screenshot from the other node with the high cpu > utilization. > > > > On 3/24/17 1:32 PM, Aaron Knister wrote: > > > heh, yep we're on sles :) > > > > > > here's a screenshot of the fs manager from the deadlocked > filesystem. I > > > don't think there's an nsd server or manager node that's running > full > > > throttle across all cpus. There is one that's got relatively high > CPU > > > utilization though (300-400%). I'll send a screenshot of it in a > sec. > > > > > > no zimon yet but we do have other tools to see cpu utilization. > > > > > > -Aaron > > > > > > On 3/24/17 1:22 PM, Sven Oehme wrote: > > >> you must be on sles as this segfaults only on sles to my > knowledge :-) > > >> > > >> i am looking for a NSD or manager node in your cluster that runs > at 100% > > >> cpu usage. > > >> > > >> do you have zimon deployed to look at cpu utilization across your > nodes ? > > >> > > >> sven > > >> > > >> > > >> > > >> On Fri, Mar 24, 2017 at 10:08 AM Aaron Knister < > aaron.s.knister at nasa.gov > > >> >> > wrote: > > >> > > >> Hi Sven, > > >> > > >> Which NSD server should I run top on, the fs manager? If so > the > > >> CPU load > > >> is about 155%. I'm working on perf top but not off to a great > > >> start... > > >> > > >> # perf top > > >> PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% > [1000Hz > > >> cycles], (all, 28 CPUs) > > >> > > >> > --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > >> > > >> Segmentation fault > > >> > > >> -Aaron > > >> > > >> On 3/24/17 1:04 PM, Sven Oehme wrote: > > >> > while this is happening run top and see if there is very > high cpu > > >> > utilization at this time on the NSD Server. > > >> > > > >> > if there is , run perf top (you might need to install perf > > >> command) and > > >> > see if the top cpu contender is a spinlock . if so send a > > >> screenshot of > > >> > perf top as i may know what that is and how to fix. > > >> > > > >> > sven > > >> > > > >> > > > >> > On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister > > >> > > > > > >> > aaron.s.knister at nasa.gov> > > >> >>> > wrote: > > >> > > > >> > Since yesterday morning we've noticed some deadlocks on > one > > >> of our > > >> > filesystems that seem to be triggered by writing to it. > The > > >> waiters on > > >> > the clients look like this: > > >> > > > >> > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, > > >> SyncHandlerThread: > > >> > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) > > >> (InodeFlushCondVar), reason > > >> > 'waiting for the flush flag to commit metadata' > > >> > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, > > >> > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 > > >> (0x7FFFDAC7FE28) > > >> > (MsgRecordCondvar), reason 'RPC wait' for > > >> allocMsgTypeRelinquishRegion > > >> > on node 10.1.52.33 > > >> > 0x197EE70 ( 6776) waiting 0.000198354 seconds, > > >> > FileBlockWriteFetchHandlerThread: on ThCond > 0x7FFFF00CD598 > > >> > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' > for > > >> > allocMsgTypeRequestRegion on node 10.1.52.33 > > >> > > > >> > (10.1.52.33/c0n3271 > > > > >> is the fs manager > > >> > for the filesystem in question) > > >> > > > >> > there's a single process running on this node writing > to the > > >> filesystem > > >> > in question (well, trying to write, it's been blocked > doing > > >> nothing for > > >> > half an hour now). There are ~10 other client nodes in > this > > >> situation > > >> > right now. We had many more last night before the > problem > > >> seemed to > > >> > disappear in the early hours of the morning and now its > back. > > >> > > > >> > Waiters on the fs manager look like this. While the > > >> individual waiter is > > >> > short it's a near constant stream: > > >> > > > >> > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, > Msg > > >> handler > > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > > >> (0xFFFFC9002163A2E0) > > >> > (AllocManagerMutex) > > >> > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, > Msg > > >> handler > > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > >> > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, > Msg > > >> handler > > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > > >> (0xFFFFC9002163A2E0) > > >> > (AllocManagerMutex) > > >> > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, > Msg > > >> handler > > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > > >> (0xFFFFC9002163A2E0) > > >> > (AllocManagerMutex) > > >> > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, > Msg > > >> handler > > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > >> > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, > Msg > > >> handler > > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > > >> (0xFFFFC9002163A2E0) > > >> > (AllocManagerMutex) > > >> > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, > Msg > > >> handler > > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > > >> (0xFFFFC9002163A2E0) > > >> > (AllocManagerMutex) > > >> > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, > Msg > > >> handler > > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > >> > > > >> > I don't know if this data point is useful but both > yesterday > > >> and today > > >> > the metadata NSDs for this filesystem have had a > constant > > >> aggregate > > >> > stream of 25MB/s 4kop/s reads during each episode (very > low > > >> latency > > >> > though so I don't believe the storage is a bottleneck > here). > > >> Writes are > > >> > only a few hundred ops and didn't strike me as odd. > > >> > > > >> > I have a PMR open for this but I'm curious if folks have > > >> seen this in > > >> > the wild and what it might mean. > > >> > > > >> > -Aaron > > >> > > > >> > -- > > >> > Aaron Knister > > >> > NASA Center for Climate Simulation (Code 606.2) > > >> > Goddard Space Flight Center > > >> > (301) 286-2776 > > > > > >> > _______________________________________________ > > >> > gpfsug-discuss mailing list > > >> > gpfsug-discuss at spectrumscale.org < > http://spectrumscale.org> > > >> > > >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > >> > > > >> > > > >> > > > >> > _______________________________________________ > > >> > gpfsug-discuss mailing list > > >> > gpfsug-discuss at spectrumscale.org < > http://spectrumscale.org> > > >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > >> > > > >> > > >> -- > > >> Aaron Knister > > >> NASA Center for Climate Simulation (Code 606.2) > > >> Goddard Space Flight Center > > >> (301) 286-2776 > > >> _______________________________________________ > > >> gpfsug-discuss mailing list > > >> gpfsug-discuss at spectrumscale.org > > > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > >> > > >> > > >> > > >> _______________________________________________ > > >> gpfsug-discuss mailing list > > >> gpfsug-discuss at spectrumscale.org > > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > >> > > > > > > > > > > > > _______________________________________________ > > > gpfsug-discuss mailing list > > > gpfsug-discuss at spectrumscale.org > > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > -- > > Aaron Knister > > NASA Center for Climate Simulation (Code 606.2) > > Goddard Space Flight Center > > (301) 286-2776 > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Fri Mar 24 18:08:44 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 24 Mar 2017 14:08:44 -0400 Subject: [gpfsug-discuss] strange waiters + filesystem deadlock In-Reply-To: References: <817e2298-7946-11dd-396f-55a675475917@nasa.gov> Message-ID: <47206c08-671f-efab-d094-aa1200627b22@nasa.gov> I believe it was created with -n 5000. Here's the exact command that was used: /usr/lpp/mmfs/bin/mmcrfs dnb03 -F ./disc_mmcrnsd_dnb03.lst -T /gpfsm/dnb03 -j cluster -B 1M -n 5000 -N 20M -r1 -R2 -m2 -M2 -A no -Q yes -v yes -i 512 --metadata-block-size=256K -L 8388608 -Aaron On 3/24/17 2:05 PM, Sven Oehme wrote: > was this filesystem creates with -n 5000 ? or was that changed later > with mmchfs ? > please send the mmlsconfig/mmlscluster output to me at oehmes at us.ibm.com > > > > > On Fri, Mar 24, 2017 at 10:58 AM Aaron Knister > wrote: > > I feel a little awkward about posting wlists of IP's and hostnames on > the mailing list (even though they're all internal) but I'm happy to > send to you directly. I've attached both an lsfs and an mmdf output of > the fs in question here since that may be useful for others to see. Just > a note about disk d23_02_021-- it's been evacuated for several weeks now > due to a hardware issue in the disk enclosure. > > The fs is rather full percentage wise (93%) but in terms of capacity > there's a good amount free. 93% full of a 7PB filesystem still leaves > 551T. Metadata, as you'll see, is 31% free (roughly 800GB). > > The fs has 40M inodes allocated and 12M free. > > -Aaron > > On 3/24/17 1:41 PM, Sven Oehme wrote: > > ok, that seems a different problem then i was thinking. > > can you send output of mmlscluster, mmlsconfig, mmlsfs all ? > > also are you getting close to fill grade on inodes or capacity on any of > > the filesystems ? > > > > sven > > > > > > On Fri, Mar 24, 2017 at 10:34 AM Aaron Knister > > >> wrote: > > > > Here's the screenshot from the other node with the high cpu utilization. > > > > On 3/24/17 1:32 PM, Aaron Knister wrote: > > > heh, yep we're on sles :) > > > > > > here's a screenshot of the fs manager from the deadlocked filesystem. I > > > don't think there's an nsd server or manager node that's running full > > > throttle across all cpus. There is one that's got relatively high CPU > > > utilization though (300-400%). I'll send a screenshot of it in a sec. > > > > > > no zimon yet but we do have other tools to see cpu utilization. > > > > > > -Aaron > > > > > > On 3/24/17 1:22 PM, Sven Oehme wrote: > > >> you must be on sles as this segfaults only on sles to my knowledge :-) > > >> > > >> i am looking for a NSD or manager node in your cluster that runs at 100% > > >> cpu usage. > > >> > > >> do you have zimon deployed to look at cpu utilization across your nodes ? > > >> > > >> sven > > >> > > >> > > >> > > >> On Fri, Mar 24, 2017 at 10:08 AM Aaron Knister > > > > >> > >>> wrote: > > >> > > >> Hi Sven, > > >> > > >> Which NSD server should I run top on, the fs manager? If so the > > >> CPU load > > >> is about 155%. I'm working on perf top but not off to a great > > >> start... > > >> > > >> # perf top > > >> PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% [1000Hz > > >> cycles], (all, 28 CPUs) > > >> > > >> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > >> > > >> Segmentation fault > > >> > > >> -Aaron > > >> > > >> On 3/24/17 1:04 PM, Sven Oehme wrote: > > >> > while this is happening run top and see if there is very high cpu > > >> > utilization at this time on the NSD Server. > > >> > > > >> > if there is , run perf top (you might need to install perf > > >> command) and > > >> > see if the top cpu contender is a spinlock . if so send a > > >> screenshot of > > >> > perf top as i may know what that is and how to fix. > > >> > > > >> > sven > > >> > > > >> > > > >> > On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister > > >> > > > > > >> > > >> > > > > > >> > >>>> wrote: > > >> > > > >> > Since yesterday morning we've noticed some deadlocks on one > > >> of our > > >> > filesystems that seem to be triggered by writing to it. The > > >> waiters on > > >> > the clients look like this: > > >> > > > >> > 0x19450B0 ( 6730) waiting 2063.294589599 seconds, > > >> SyncHandlerThread: > > >> > on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) > > >> (InodeFlushCondVar), reason > > >> > 'waiting for the flush flag to commit metadata' > > >> > 0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds, > > >> > AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 > > >> (0x7FFFDAC7FE28) > > >> > (MsgRecordCondvar), reason 'RPC wait' for > > >> allocMsgTypeRelinquishRegion > > >> > on node 10.1.52.33 > > >> > 0x197EE70 ( 6776) waiting 0.000198354 seconds, > > >> > FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598 > > >> > (0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for > > >> > allocMsgTypeRequestRegion on node 10.1.52.33 > > >> > > > >> > (10.1.52.33/c0n3271 > > > > > >> is the fs manager > > >> > for the filesystem in question) > > >> > > > >> > there's a single process running on this node writing to the > > >> filesystem > > >> > in question (well, trying to write, it's been blocked doing > > >> nothing for > > >> > half an hour now). There are ~10 other client nodes in this > > >> situation > > >> > right now. We had many more last night before the problem > > >> seemed to > > >> > disappear in the early hours of the morning and now its back. > > >> > > > >> > Waiters on the fs manager look like this. While the > > >> individual waiter is > > >> > short it's a near constant stream: > > >> > > > >> > 0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg > > >> handler > > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > > >> (0xFFFFC9002163A2E0) > > >> > (AllocManagerMutex) > > >> > 0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg > > >> handler > > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > >> > 0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg > > >> handler > > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > > >> (0xFFFFC9002163A2E0) > > >> > (AllocManagerMutex) > > >> > 0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg > > >> handler > > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > > >> (0xFFFFC9002163A2E0) > > >> > (AllocManagerMutex) > > >> > 0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg > > >> handler > > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > >> > 0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg > > >> handler > > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > > >> (0xFFFFC9002163A2E0) > > >> > (AllocManagerMutex) > > >> > 0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg > > >> handler > > >> > allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 > > >> (0xFFFFC9002163A2E0) > > >> > (AllocManagerMutex) > > >> > 0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg > > >> handler > > >> > allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0 > > >> > (0xFFFFC9002163A2E0) (AllocManagerMutex) > > >> > > > >> > I don't know if this data point is useful but both yesterday > > >> and today > > >> > the metadata NSDs for this filesystem have had a constant > > >> aggregate > > >> > stream of 25MB/s 4kop/s reads during each episode (very low > > >> latency > > >> > though so I don't believe the storage is a bottleneck here). > > >> Writes are > > >> > only a few hundred ops and didn't strike me as odd. > > >> > > > >> > I have a PMR open for this but I'm curious if folks have > > >> seen this in > > >> > the wild and what it might mean. > > >> > > > >> > -Aaron > > >> > > > >> > -- > > >> > Aaron Knister > > >> > NASA Center for Climate Simulation (Code 606.2) > > >> > Goddard Space Flight Center > > >> > (301) 286-2776 > > > > > >> > _______________________________________________ > > >> > gpfsug-discuss mailing list > > >> > gpfsug-discuss at spectrumscale.org > > >> > > >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > >> > > > >> > > > >> > > > >> > _______________________________________________ > > >> > gpfsug-discuss mailing list > > >> > gpfsug-discuss at spectrumscale.org > > > >> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > >> > > > >> > > >> -- > > >> Aaron Knister > > >> NASA Center for Climate Simulation (Code 606.2) > > >> Goddard Space Flight Center > > >> (301) 286-2776 > > > >> _______________________________________________ > > >> gpfsug-discuss mailing list > > >> gpfsug-discuss at spectrumscale.org > > > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > >> > > >> > > >> > > >> _______________________________________________ > > >> gpfsug-discuss mailing list > > >> gpfsug-discuss at spectrumscale.org > > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > >> > > > > > > > > > > > > _______________________________________________ > > > gpfsug-discuss mailing list > > > gpfsug-discuss at spectrumscale.org > > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > -- > > Aaron Knister > > NASA Center for Climate Simulation (Code 606.2) > > Goddard Space Flight Center > > (301) 286-2776 > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From bbanister at jumptrading.com Fri Mar 24 18:13:33 2017 From: bbanister at jumptrading.com (Bryan Banister) Date: Fri, 24 Mar 2017 18:13:33 +0000 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: References: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu><47e18bb0062544f4b510ce0e87049fd3@jumptrading.com><643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> Message-ID: Hi Vipul, Hmm... interesting. We have dedicated systems running CES and nothing else, so the only thing opening files on GPFS is ganesha. IBM Support recommended we massively increase the maxFilesToCache to fix the performance issues we were having. I could try to reproduce the problem to investigate further, but the impact to users would not be appreciated. Setting such a high value for maxFilesToCache has direct impact on the token manager so fixing this would be very good. Perhaps IBM could take a closer look at this without our involvement? -Bryan From: Vipul Paul [mailto:vpaul at us.ibm.com] On Behalf Of IBM Spectrum Scale Sent: Friday, March 24, 2017 12:18 PM To: gpfsug main discussion list ; Bryan Banister Subject: Re: CES node slow to respond Hi Bryan, Making sure Malahal's reply was received by the user group. >> Then we noticed that the CES host had 5.4 million files open This is technically not possible with ganesha alone. A process can only open 1 million files on RHEL distro. Either we have leaks in kernel or some other processes contributing to this. Ganesha does keep NFSv3 files open and keep open for performance. It doesn't have a good logic to close them after some inactivity. It does close them if the number is close to max open files which is configurable. PS: kNFS does open, read/write, and then close. No caching in older versions. They did have a feature in a recent code to cache open files in NFSv3 as well. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Bryan Banister > To: gpfsug main discussion list > Date: 03/23/2017 08:27 AM Subject: Re: [gpfsug-discuss] CES node slow to respond Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Anybody from IBM willing/able to give us some explanation of why ganesha is holding open so many files? Is this expected/needed/etc? Or do we have to open a PMR to get some kind of explanation? -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Thursday, March 23, 2017 10:24 AM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] CES node slow to respond FYI all We also ran into this after bumping maxFilesToCache. Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: unexpected error: Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many open files in system fix sysctl -w fs.file-max=1000000 On 3/22/17 12:01 PM, Bryan Banister wrote: > We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. > > Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil > Sent: Wednesday, March 22, 2017 11:43 AM > To: gpfsug main discussion list > > Subject: [gpfsug-discuss] CES node slow to respond > > All, > > We had an indecent yesterday where one of our CES nodes slowed to a > crawl. GPFS waiters showed pre fetch threads going after inodes. > iohist also showed lots of inode fetching. Then we noticed that the CES > host had 5.4 million files open. > > The change I made was to set maxStatCache=DEFAULT because it is linux. > And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. > > Is there something else we should change as well. > > Thanks > > Matt > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From oehmes at gmail.com Fri Mar 24 18:19:52 2017 From: oehmes at gmail.com (Sven Oehme) Date: Fri, 24 Mar 2017 18:19:52 +0000 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: References: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu> <47e18bb0062544f4b510ce0e87049fd3@jumptrading.com> <643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> Message-ID: changes in ganesha management code were made in April 2016 to reduce the need for high maxfilestocache value, the ganesha daemon adjusts it allowed file cache by reading the maxfilestocache value and then reducing its allowed NOFILE value . the code shipped with 4.2.2 release. you want a high maxfilestocache value on CES nodes for various reasons. just FYI we have some customers who have MFTC set to 15 Million :-) sven On Fri, Mar 24, 2017 at 11:13 AM Bryan Banister wrote: > Hi Vipul, > > > > Hmm? interesting. We have dedicated systems running CES and nothing else, > so the only thing opening files on GPFS is ganesha. IBM Support > recommended we massively increase the maxFilesToCache to fix the > performance issues we were having. I could try to reproduce the problem to > investigate further, but the impact to users would not be appreciated. > > > > Setting such a high value for maxFilesToCache has direct impact on the > token manager so fixing this would be very good. > > > > Perhaps IBM could take a closer look at this without our involvement? > > -Bryan > > > > *From:* Vipul Paul [mailto:vpaul at us.ibm.com] *On Behalf Of *IBM Spectrum > Scale > *Sent:* Friday, March 24, 2017 12:18 PM > *To:* gpfsug main discussion list ; > Bryan Banister > *Subject:* Re: CES node slow to respond > > > > Hi Bryan, > > Making sure Malahal's reply was received by the user group. > > >> Then we noticed that the CES host had 5.4 million files open > > This is technically not possible with ganesha alone. A process can only > open 1 million files on RHEL distro. Either we have leaks in kernel or some > other processes contributing to this. > > Ganesha does keep NFSv3 files open and keep open for performance. It > doesn't have a good logic to close them after some inactivity. It does > close them if the number is close to max open files which is configurable. > > PS: kNFS does open, read/write, and then close. No caching in older > versions. They did have a feature in a recent code to cache open files in > NFSv3 as well. > > Regards, The Spectrum Scale (GPFS) team > > > ------------------------------------------------------------------------------------------------------------------ > If you feel that your question can benefit other users of Spectrum Scale > (GPFS), then please post it to the public IBM developerWroks Forum at > https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. > > > If your query concerns a potential software error in Spectrum Scale (GPFS) > and you have an IBM software maintenance contract please contact > 1-800-237-5511 <(800)%20237-5511> in the United States or your local IBM > Service Center in other countries. > > The forum is informally monitored as time permits and should not be used > for priority messages to the Spectrum Scale (GPFS) team. > > > > From: Bryan Banister > To: gpfsug main discussion list > Date: 03/23/2017 08:27 AM > Subject: Re: [gpfsug-discuss] CES node slow to respond > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------ > > > > > Anybody from IBM willing/able to give us some explanation of why ganesha > is holding open so many files? Is this expected/needed/etc? > > Or do we have to open a PMR to get some kind of explanation? > -B > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [ > mailto:gpfsug-discuss-bounces at spectrumscale.org > ] On Behalf Of Matt Weil > Sent: Thursday, March 23, 2017 10:24 AM > To: gpfsug-discuss at spectrumscale.org > Subject: Re: [gpfsug-discuss] CES node slow to respond > > FYI all > > We also ran into this after bumping maxFilesToCache. > > Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: > unexpected error: > Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many > open files in system > > fix > > sysctl -w fs.file-max=1000000 > > > On 3/22/17 12:01 PM, Bryan Banister wrote: > > We had a similar issue and also were instructed by IBM Support to > increase the maxFilesToCache to an insane value... Basically when the file > cache gets full then the host will spend all of its cycles looking for a > file to evict every time a new file is opened... baaaah. > > > > Not sure why Ganesha has to keep so many files open... I can't believe > our NFS clients actually keep that many open. cNFS never needed this. > > -Bryan > > > > -----Original Message----- > > From: gpfsug-discuss-bounces at spectrumscale.org [ > mailto:gpfsug-discuss-bounces at spectrumscale.org > ] On Behalf Of Matt Weil > > Sent: Wednesday, March 22, 2017 11:43 AM > > To: gpfsug main discussion list > > Subject: [gpfsug-discuss] CES node slow to respond > > > > All, > > > > We had an indecent yesterday where one of our CES nodes slowed to a > > crawl. GPFS waiters showed pre fetch threads going after inodes. > > iohist also showed lots of inode fetching. Then we noticed that the CES > > host had 5.4 million files open. > > > > The change I made was to set maxStatCache=DEFAULT because it is linux. > > And set maxFilesToCache=10000000 it was set to 500000. Then restarted > GPFS. > > > > Is there something else we should change as well. > > > > Thanks > > > > Matt > > > > > > ________________________________ > > The materials in this message are private and may contain Protected > Healthcare Information or other information of a sensitive nature. If you > are not the intended recipient, be advised that any unauthorized use, > disclosure, copying or the taking of any action in reliance on the contents > of this information is strictly prohibited. If you have received this email > in error, please immediately notify the sender via telephone or return mail. > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > ________________________________ > > > > Note: This email is for the confidential use of the named addressee(s) > only and may contain proprietary, confidential or privileged information. > If you are not the intended recipient, you are hereby notified that any > review, dissemination or copying of this email is strictly prohibited, and > to please notify the sender immediately and destroy this email and any > attachments. Email transmission cannot be guaranteed to be secure or > error-free. The Company, therefore, does not make any guarantees as to the > completeness or accuracy of this email or any attachments. This email is > for informational purposes only and does not constitute a recommendation, > offer, request or solicitation of any kind to buy, sell, subscribe, redeem > or perform any type of transaction of a financial product. > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > ________________________________ > The materials in this message are private and may contain Protected > Healthcare Information or other information of a sensitive nature. If you > are not the intended recipient, be advised that any unauthorized use, > disclosure, copying or the taking of any action in reliance on the contents > of this information is strictly prohibited. If you have received this email > in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) > only and may contain proprietary, confidential or privileged information. > If you are not the intended recipient, you are hereby notified that any > review, dissemination or copying of this email is strictly prohibited, and > to please notify the sender immediately and destroy this email and any > attachments. Email transmission cannot be guaranteed to be secure or > error-free. The Company, therefore, does not make any guarantees as to the > completeness or accuracy of this email or any attachments. This email is > for informational purposes only and does not constitute a recommendation, > offer, request or solicitation of any kind to buy, sell, subscribe, redeem > or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > ------------------------------ > > Note: This email is for the confidential use of the named addressee(s) > only and may contain proprietary, confidential or privileged information. > If you are not the intended recipient, you are hereby notified that any > review, dissemination or copying of this email is strictly prohibited, and > to please notify the sender immediately and destroy this email and any > attachments. Email transmission cannot be guaranteed to be secure or > error-free. The Company, therefore, does not make any guarantees as to the > completeness or accuracy of this email or any attachments. This email is > for informational purposes only and does not constitute a recommendation, > offer, request or solicitation of any kind to buy, sell, subscribe, redeem > or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Fri Mar 24 18:22:50 2017 From: bbanister at jumptrading.com (Bryan Banister) Date: Fri, 24 Mar 2017 18:22:50 +0000 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: References: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu> <47e18bb0062544f4b510ce0e87049fd3@jumptrading.com> <643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> Message-ID: <47c157d6cb5747698d3ccf11bd7a27ab@jumptrading.com> Thanks Sven! We recently upgrade to 4.2.2 and will see about lowering the maxFilesToCache to something more appropriate. We?re not offering NFS access as a performance solution? but it can?t come to a crawl either! Your help is greatly appreciated as always, -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sven Oehme Sent: Friday, March 24, 2017 1:20 PM To: gpfsug main discussion list ; IBM Spectrum Scale Subject: Re: [gpfsug-discuss] CES node slow to respond changes in ganesha management code were made in April 2016 to reduce the need for high maxfilestocache value, the ganesha daemon adjusts it allowed file cache by reading the maxfilestocache value and then reducing its allowed NOFILE value . the code shipped with 4.2.2 release. you want a high maxfilestocache value on CES nodes for various reasons. just FYI we have some customers who have MFTC set to 15 Million :-) sven On Fri, Mar 24, 2017 at 11:13 AM Bryan Banister > wrote: Hi Vipul, Hmm? interesting. We have dedicated systems running CES and nothing else, so the only thing opening files on GPFS is ganesha. IBM Support recommended we massively increase the maxFilesToCache to fix the performance issues we were having. I could try to reproduce the problem to investigate further, but the impact to users would not be appreciated. Setting such a high value for maxFilesToCache has direct impact on the token manager so fixing this would be very good. Perhaps IBM could take a closer look at this without our involvement? -Bryan From: Vipul Paul [mailto:vpaul at us.ibm.com] On Behalf Of IBM Spectrum Scale Sent: Friday, March 24, 2017 12:18 PM To: gpfsug main discussion list >; Bryan Banister > Subject: Re: CES node slow to respond Hi Bryan, Making sure Malahal's reply was received by the user group. >> Then we noticed that the CES host had 5.4 million files open This is technically not possible with ganesha alone. A process can only open 1 million files on RHEL distro. Either we have leaks in kernel or some other processes contributing to this. Ganesha does keep NFSv3 files open and keep open for performance. It doesn't have a good logic to close them after some inactivity. It does close them if the number is close to max open files which is configurable. PS: kNFS does open, read/write, and then close. No caching in older versions. They did have a feature in a recent code to cache open files in NFSv3 as well. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Bryan Banister > To: gpfsug main discussion list > Date: 03/23/2017 08:27 AM Subject: Re: [gpfsug-discuss] CES node slow to respond Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Anybody from IBM willing/able to give us some explanation of why ganesha is holding open so many files? Is this expected/needed/etc? Or do we have to open a PMR to get some kind of explanation? -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Thursday, March 23, 2017 10:24 AM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] CES node slow to respond FYI all We also ran into this after bumping maxFilesToCache. Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: unexpected error: Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many open files in system fix sysctl -w fs.file-max=1000000 On 3/22/17 12:01 PM, Bryan Banister wrote: > We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. > > Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil > Sent: Wednesday, March 22, 2017 11:43 AM > To: gpfsug main discussion list > > Subject: [gpfsug-discuss] CES node slow to respond > > All, > > We had an indecent yesterday where one of our CES nodes slowed to a > crawl. GPFS waiters showed pre fetch threads going after inodes. > iohist also showed lots of inode fetching. Then we noticed that the CES > host had 5.4 million files open. > > The change I made was to set maxStatCache=DEFAULT because it is linux. > And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. > > Is there something else we should change as well. > > Thanks > > Matt > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mweil at wustl.edu Fri Mar 24 19:57:42 2017 From: mweil at wustl.edu (Matt Weil) Date: Fri, 24 Mar 2017 14:57:42 -0500 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: References: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu> <47e18bb0062544f4b510ce0e87049fd3@jumptrading.com> <643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> Message-ID: <5a3aeddb-3e09-4003-8e57-299e999fe62d@wustl.edu> On 3/24/17 1:13 PM, Bryan Banister wrote: Hi Vipul, Hmm? interesting. We have dedicated systems running CES and nothing else, so the only thing opening files on GPFS is ganesha. IBM Support recommended we massively increase the maxFilesToCache to fix the performance issues we were having. I could try to reproduce the problem to investigate further, but the impact to users would not be appreciated. Same with us the system only runs CES and NFS. All open files 5.4 million where tied to it. This excluded anything the system had open. Setting such a high value for maxFilesToCache has direct impact on the token manager so fixing this would be very good. Perhaps IBM could take a closer look at this without our involvement? -Bryan From: Vipul Paul [mailto:vpaul at us.ibm.com] On Behalf Of IBM Spectrum Scale Sent: Friday, March 24, 2017 12:18 PM To: gpfsug main discussion list ; Bryan Banister Subject: Re: CES node slow to respond Hi Bryan, Making sure Malahal's reply was received by the user group. >> Then we noticed that the CES host had 5.4 million files open This is technically not possible with ganesha alone. A process can only open 1 million files on RHEL distro. Either we have leaks in kernel or some other processes contributing to this. Ganesha does keep NFSv3 files open and keep open for performance. It doesn't have a good logic to close them after some inactivity. It does close them if the number is close to max open files which is configurable. PS: kNFS does open, read/write, and then close. No caching in older versions. They did have a feature in a recent code to cache open files in NFSv3 as well. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Bryan Banister > To: gpfsug main discussion list > Date: 03/23/2017 08:27 AM Subject: Re: [gpfsug-discuss] CES node slow to respond Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Anybody from IBM willing/able to give us some explanation of why ganesha is holding open so many files? Is this expected/needed/etc? Or do we have to open a PMR to get some kind of explanation? -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Thursday, March 23, 2017 10:24 AM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] CES node slow to respond FYI all We also ran into this after bumping maxFilesToCache. Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: unexpected error: Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many open files in system fix sysctl -w fs.file-max=1000000 On 3/22/17 12:01 PM, Bryan Banister wrote: > We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. > > Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil > Sent: Wednesday, March 22, 2017 11:43 AM > To: gpfsug main discussion list > > Subject: [gpfsug-discuss] CES node slow to respond > > All, > > We had an indecent yesterday where one of our CES nodes slowed to a > crawl. GPFS waiters showed pre fetch threads going after inodes. > iohist also showed lots of inode fetching. Then we noticed that the CES > host had 5.4 million files open. > > The change I made was to set maxStatCache=DEFAULT because it is linux. > And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. > > Is there something else we should change as well. > > Thanks > > Matt > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mweil at wustl.edu Fri Mar 24 20:03:22 2017 From: mweil at wustl.edu (Matt Weil) Date: Fri, 24 Mar 2017 15:03:22 -0500 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: <5a3aeddb-3e09-4003-8e57-299e999fe62d@wustl.edu> References: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu> <47e18bb0062544f4b510ce0e87049fd3@jumptrading.com> <643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> <5a3aeddb-3e09-4003-8e57-299e999fe62d@wustl.edu> Message-ID: also running version 4.2.2.2. On 3/24/17 2:57 PM, Matt Weil wrote: On 3/24/17 1:13 PM, Bryan Banister wrote: Hi Vipul, Hmm? interesting. We have dedicated systems running CES and nothing else, so the only thing opening files on GPFS is ganesha. IBM Support recommended we massively increase the maxFilesToCache to fix the performance issues we were having. I could try to reproduce the problem to investigate further, but the impact to users would not be appreciated. Same with us the system only runs CES and NFS. All open files 5.4 million where tied to it. This excluded anything the system had open. Setting such a high value for maxFilesToCache has direct impact on the token manager so fixing this would be very good. Perhaps IBM could take a closer look at this without our involvement? -Bryan From: Vipul Paul [mailto:vpaul at us.ibm.com] On Behalf Of IBM Spectrum Scale Sent: Friday, March 24, 2017 12:18 PM To: gpfsug main discussion list ; Bryan Banister Subject: Re: CES node slow to respond Hi Bryan, Making sure Malahal's reply was received by the user group. >> Then we noticed that the CES host had 5.4 million files open This is technically not possible with ganesha alone. A process can only open 1 million files on RHEL distro. Either we have leaks in kernel or some other processes contributing to this. Ganesha does keep NFSv3 files open and keep open for performance. It doesn't have a good logic to close them after some inactivity. It does close them if the number is close to max open files which is configurable. PS: kNFS does open, read/write, and then close. No caching in older versions. They did have a feature in a recent code to cache open files in NFSv3 as well. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Bryan Banister > To: gpfsug main discussion list > Date: 03/23/2017 08:27 AM Subject: Re: [gpfsug-discuss] CES node slow to respond Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Anybody from IBM willing/able to give us some explanation of why ganesha is holding open so many files? Is this expected/needed/etc? Or do we have to open a PMR to get some kind of explanation? -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Thursday, March 23, 2017 10:24 AM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] CES node slow to respond FYI all We also ran into this after bumping maxFilesToCache. Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: unexpected error: Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many open files in system fix sysctl -w fs.file-max=1000000 On 3/22/17 12:01 PM, Bryan Banister wrote: > We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. > > Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil > Sent: Wednesday, March 22, 2017 11:43 AM > To: gpfsug main discussion list > > Subject: [gpfsug-discuss] CES node slow to respond > > All, > > We had an indecent yesterday where one of our CES nodes slowed to a > crawl. GPFS waiters showed pre fetch threads going after inodes. > iohist also showed lots of inode fetching. Then we noticed that the CES > host had 5.4 million files open. > > The change I made was to set maxStatCache=DEFAULT because it is linux. > And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. > > Is there something else we should change as well. > > Thanks > > Matt > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mweil at wustl.edu Fri Mar 24 20:20:18 2017 From: mweil at wustl.edu (Matt Weil) Date: Fri, 24 Mar 2017 15:20:18 -0500 Subject: [gpfsug-discuss] CES node slow to respond In-Reply-To: References: <379a0484-3571-8cf8-6de3-2bbc90d12346@wustl.edu> <47e18bb0062544f4b510ce0e87049fd3@jumptrading.com> <643614dc-92c9-43a0-2010-5f326b5109bd@wustl.edu> Message-ID: On 3/24/17 12:17 PM, IBM Spectrum Scale wrote: Hi Bryan, Making sure Malahal's reply was received by the user group. >> Then we noticed that the CES host had 5.4 million files open This is technically not possible with ganesha alone. A process can only open 1 million files on RHEL distro. Either we have leaks in kernel or some other processes contributing to this. Ganesha does keep NFSv3 files open and keep open for performance. It doesn't have a good logic to close them after some inactivity. It does close them if the number is close to max open files which is configurable. how do you set the max open files for Ganesha? PS: kNFS does open, read/write, and then close. No caching in older versions. They did have a feature in a recent code to cache open files in NFSv3 as well. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Bryan Banister To: gpfsug main discussion list Date: 03/23/2017 08:27 AM Subject: Re: [gpfsug-discuss] CES node slow to respond Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Anybody from IBM willing/able to give us some explanation of why ganesha is holding open so many files? Is this expected/needed/etc? Or do we have to open a PMR to get some kind of explanation? -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Thursday, March 23, 2017 10:24 AM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] CES node slow to respond FYI all We also ran into this after bumping maxFilesToCache. Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: unexpected error: Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many open files in system fix sysctl -w fs.file-max=1000000 On 3/22/17 12:01 PM, Bryan Banister wrote: > We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. > > Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil > Sent: Wednesday, March 22, 2017 11:43 AM > To: gpfsug main discussion list > Subject: [gpfsug-discuss] CES node slow to respond > > All, > > We had an indecent yesterday where one of our CES nodes slowed to a > crawl. GPFS waiters showed pre fetch threads going after inodes. > iohist also showed lots of inode fetching. Then we noticed that the CES > host had 5.4 million files open. > > The change I made was to set maxStatCache=DEFAULT because it is linux. > And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. > > Is there something else we should change as well. > > Thanks > > Matt > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Sat Mar 25 05:46:34 2017 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Fri, 24 Mar 2017 21:46:34 -0800 Subject: [gpfsug-discuss] CES node slow to respond Message-ID: Forwarding Malahal's reply. Max open files of the ganesha process is set internally (and written to /etc/sysconfig/ganesha file as NOFILE parameter) based on MFTC (max files to cache) gpfs parameter. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. >> From: Matt Weil To: Date: 03/24/2017 01:20 PM Subject: Re: [gpfsug-discuss] CES node slow to respond Sent by: gpfsug-discuss-bounces at spectrumscale.org On 3/24/17 12:17 PM, IBM Spectrum Scale wrote: Hi Bryan, Making sure Malahal's reply was received by the user group. >> Then we noticed that the CES host had 5.4 million files open This is technically not possible with ganesha alone. A process can only open 1 million files on RHEL distro. Either we have leaks in kernel or some other processes contributing to this. Ganesha does keep NFSv3 files open and keep open for performance. It doesn't have a good logic to close them after some inactivity. It does close them if the number is close to max open files which is configurable. how do you set the max open files for Ganesha? PS: kNFS does open, read/write, and then close. No caching in older versions. They did have a feature in a recent code to cache open files in NFSv3 as well. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Bryan Banister To: gpfsug main discussion list Date: 03/23/2017 08:27 AM Subject: Re: [gpfsug-discuss] CES node slow to respond Sent by: gpfsug-discuss-bounces at spectrumscale.org Anybody from IBM willing/able to give us some explanation of why ganesha is holding open so many files? Is this expected/needed/etc? Or do we have to open a PMR to get some kind of explanation? -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Thursday, March 23, 2017 10:24 AM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] CES node slow to respond FYI all We also ran into this after bumping maxFilesToCache. Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: unexpected error: Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many open files in system fix sysctl -w fs.file-max=1000000 On 3/22/17 12:01 PM, Bryan Banister wrote: > We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. > > Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil > Sent: Wednesday, March 22, 2017 11:43 AM > To: gpfsug main discussion list > Subject: [gpfsug-discuss] CES node slow to respond > > All, > > We had an indecent yesterday where one of our CES nodes slowed to a > crawl. GPFS waiters showed pre fetch threads going after inodes. > iohist also showed lots of inode fetching. Then we noticed that the CES > host had 5.4 million files open. > > The change I made was to set maxStatCache=DEFAULT because it is linux. > And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. > > Is there something else we should change as well. > > Thanks > > Matt > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail._______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Sat Mar 25 05:55:57 2017 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Fri, 24 Mar 2017 21:55:57 -0800 Subject: [gpfsug-discuss] Fw: CES node slow to respond Message-ID: Caching of file descriptors can be disabled with "Cache_FDs = FALSE;" in cacheinode{} block. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. >> From: Bryan Banister To: IBM Spectrum Scale/Poughkeepsie/IBM at IBMUS, gpfsug main discussion list Date: 03/24/2017 11:13 AM Subject: RE: CES node slow to respond Hi Vipul, Hmm? interesting. We have dedicated systems running CES and nothing else, so the only thing opening files on GPFS is ganesha. IBM Support recommended we massively increase the maxFilesToCache to fix the performance issues we were having. I could try to reproduce the problem to investigate further, but the impact to users would not be appreciated. Setting such a high value for maxFilesToCache has direct impact on the token manager so fixing this would be very good. Perhaps IBM could take a closer look at this without our involvement? -Bryan From: Vipul Paul [mailto:vpaul at us.ibm.com] On Behalf Of IBM Spectrum Scale Sent: Friday, March 24, 2017 12:18 PM To: gpfsug main discussion list ; Bryan Banister Subject: Re: CES node slow to respond Hi Bryan, Making sure Malahal's reply was received by the user group. >> Then we noticed that the CES host had 5.4 million files open This is technically not possible with ganesha alone. A process can only open 1 million files on RHEL distro. Either we have leaks in kernel or some other processes contributing to this. Ganesha does keep NFSv3 files open and keep open for performance. It doesn't have a good logic to close them after some inactivity. It does close them if the number is close to max open files which is configurable. PS: kNFS does open, read/write, and then close. No caching in older versions. They did have a feature in a recent code to cache open files in NFSv3 as well. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Bryan Banister To: gpfsug main discussion list Date: 03/23/2017 08:27 AM Subject: Re: [gpfsug-discuss] CES node slow to respond Sent by: gpfsug-discuss-bounces at spectrumscale.org Anybody from IBM willing/able to give us some explanation of why ganesha is holding open so many files? Is this expected/needed/etc? Or do we have to open a PMR to get some kind of explanation? -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil Sent: Thursday, March 23, 2017 10:24 AM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] CES node slow to respond FYI all We also ran into this after bumping maxFilesToCache. Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348: unexpected error: Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many open files in system fix sysctl -w fs.file-max=1000000 On 3/22/17 12:01 PM, Bryan Banister wrote: > We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah. > > Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this. > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Matt Weil > Sent: Wednesday, March 22, 2017 11:43 AM > To: gpfsug main discussion list > Subject: [gpfsug-discuss] CES node slow to respond > > All, > > We had an indecent yesterday where one of our CES nodes slowed to a > crawl. GPFS waiters showed pre fetch threads going after inodes. > iohist also showed lots of inode fetching. Then we noticed that the CES > host had 5.4 million files open. > > The change I made was to set maxStatCache=DEFAULT because it is linux. > And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS. > > Is there something else we should change as well. > > Thanks > > Matt > > > ________________________________ > The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chair at spectrumscale.org Mon Mar 27 08:55:04 2017 From: chair at spectrumscale.org (Spectrum Scale UG Chair (Simon Thompson)) Date: Mon, 27 Mar 2017 08:55:04 +0100 Subject: [gpfsug-discuss] UK User Group Registration NOW OPEN! [May 9th / 10th 2017] Message-ID: Registration for the UK Spectrum Scale (GPFS) user group (May 9th/10th) has opened this morning. Registration is via EventBrite at: https://www.eventbrite.com/e/spectrum-scalegpfs-user-group-spring-2017-regi stration-32113696932 This year we have moved to a larger capacity venue (Manchester Metropole Hotel), with thanks to IBM for sponsoring and supporting the event. We would also like to thank our other sponsors ArcaStream, DDN Storage, Ellexus, Mellanox, OCF and Seagate for supporting the event. Their sponsorship goes to help ensure the event remains free for the community and supports the evening social event as well as providing them with a technical (non sales) slot on the agenda. This year, the evening social event will be taking place at Manchester Museum of Science and Industry, and we're very grateful that are sponsors are able to support us to ensure this happens. The agenda is still a work in progress and may be subject to change as we confirm speakers and timings. As usual, we have reserved some slots on the agenda for user speakers, so if you have an interesting use case of Spectrum Scale, please let me know. As sectors we don't really get to hear from, it would be great if one of the members from the finance or sport industry were able to speak! Of course the event isn't limited to people from the UK, and we welcome registrants from across the world! Please note that to ensure attendance from a wide range of organisations, we are currently limiting attendance to 3 people per organisation. The venue is situated just a few minutes walk from Manchester Piccadilly station in Central Manchester with many hotels near by. The venue also has a limited number of special rate rooms, details of how you can book direct with the hotel are on the registration page. Thanks Simon UK Group Chair From p.childs at qmul.ac.uk Mon Mar 27 19:14:34 2017 From: p.childs at qmul.ac.uk (Peter Childs) Date: Mon, 27 Mar 2017 18:14:34 +0000 Subject: [gpfsug-discuss] UK User Group Registration NOW OPEN! [May 9th / 10th 2017] In-Reply-To: References: Message-ID: <0d1hhnpr5cn1dc5n2h7jpcec.1490638472562@email.android.com> Can I verify that this is at the Manchester McDonald hotel as eventbright, and website says, not the Manchester metropole hotel as the mailing list messages says, and I'm having difficulty finding. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ---- Spectrum Scale UG Chair (Simon Thompson) wrote ---- Registration for the UK Spectrum Scale (GPFS) user group (May 9th/10th) has opened this morning. Registration is via EventBrite at: https://www.eventbrite.com/e/spectrum-scalegpfs-user-group-spring-2017-regi stration-32113696932 This year we have moved to a larger capacity venue (Manchester Metropole Hotel), with thanks to IBM for sponsoring and supporting the event. We would also like to thank our other sponsors ArcaStream, DDN Storage, Ellexus, Mellanox, OCF and Seagate for supporting the event. Their sponsorship goes to help ensure the event remains free for the community and supports the evening social event as well as providing them with a technical (non sales) slot on the agenda. This year, the evening social event will be taking place at Manchester Museum of Science and Industry, and we're very grateful that are sponsors are able to support us to ensure this happens. The agenda is still a work in progress and may be subject to change as we confirm speakers and timings. As usual, we have reserved some slots on the agenda for user speakers, so if you have an interesting use case of Spectrum Scale, please let me know. As sectors we don't really get to hear from, it would be great if one of the members from the finance or sport industry were able to speak! Of course the event isn't limited to people from the UK, and we welcome registrants from across the world! Please note that to ensure attendance from a wide range of organisations, we are currently limiting attendance to 3 people per organisation. The venue is situated just a few minutes walk from Manchester Piccadilly station in Central Manchester with many hotels near by. The venue also has a limited number of special rate rooms, details of how you can book direct with the hotel are on the registration page. Thanks Simon UK Group Chair _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Mon Mar 27 19:25:53 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Mon, 27 Mar 2017 18:25:53 +0000 Subject: [gpfsug-discuss] UK User Group Registration NOW OPEN! [May 9th / 10th 2017] In-Reply-To: <0d1hhnpr5cn1dc5n2h7jpcec.1490638472562@email.android.com> References: <0d1hhnpr5cn1dc5n2h7jpcec.1490638472562@email.android.com> Message-ID: Hi Peter, Sorry, my mistake, the eventbrite page is correct - Manchester McDonald hotel. (we looked at a couple of different options in different areas and I obviously got confused!) Simon From: > on behalf of Peter Childs > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Monday, 27 March 2017 at 19:14 To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] UK User Group Registration NOW OPEN! [May 9th / 10th 2017] Can I verify that this is at the Manchester McDonald hotel as eventbright, and website says, not the Manchester metropole hotel as the mailing list messages says, and I'm having difficulty finding. Peter Childs Research Storage ITS Research and Teaching Support Queen Mary, University of London ---- Spectrum Scale UG Chair (Simon Thompson) wrote ---- Registration for the UK Spectrum Scale (GPFS) user group (May 9th/10th) has opened this morning. Registration is via EventBrite at: https://www.eventbrite.com/e/spectrum-scalegpfs-user-group-spring-2017-regi stration-32113696932 This year we have moved to a larger capacity venue (Manchester Metropole Hotel), with thanks to IBM for sponsoring and supporting the event. We would also like to thank our other sponsors ArcaStream, DDN Storage, Ellexus, Mellanox, OCF and Seagate for supporting the event. Their sponsorship goes to help ensure the event remains free for the community and supports the evening social event as well as providing them with a technical (non sales) slot on the agenda. This year, the evening social event will be taking place at Manchester Museum of Science and Industry, and we're very grateful that are sponsors are able to support us to ensure this happens. The agenda is still a work in progress and may be subject to change as we confirm speakers and timings. As usual, we have reserved some slots on the agenda for user speakers, so if you have an interesting use case of Spectrum Scale, please let me know. As sectors we don't really get to hear from, it would be great if one of the members from the finance or sport industry were able to speak! Of course the event isn't limited to people from the UK, and we welcome registrants from across the world! Please note that to ensure attendance from a wide range of organisations, we are currently limiting attendance to 3 people per organisation. The venue is situated just a few minutes walk from Manchester Piccadilly station in Central Manchester with many hotels near by. The venue also has a limited number of special rate rooms, details of how you can book direct with the hotel are on the registration page. Thanks Simon UK Group Chair _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From pinto at scinet.utoronto.ca Tue Mar 28 15:47:44 2017 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Tue, 28 Mar 2017 10:47:44 -0400 Subject: [gpfsug-discuss] fix mmrepquota report format during grace periods Message-ID: <20170328104744.138874ih550oncls@support.scinet.utoronto.ca> Any chance you guys in the GPFS devel team could patch the mmrepquota code so that during grace periods the report column for "none" would still be replaced with >>>*ONE*<<< word? By that I mean, instead of "2 days" for example, just print "2-days" or "2days" or "2_days", and so on. I have a number of scripts that fail for users when they are over their quotas under grace periods, because the report shifts the remaining information for that user 1 column to the right. Obviously it would cost me absolutely nothing to patch my scripts to deal with this, however the principle here is that the reports generated by GPFS should be the ones keeping consistence. Thanks Jaime ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. From Robert.Oesterlin at nuance.com Tue Mar 28 15:54:42 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Tue, 28 Mar 2017 14:54:42 +0000 Subject: [gpfsug-discuss] fix mmrepquota report format during grace periods Message-ID: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> Try running it with the ?-Y? option, it returns an easily to read output: mmrepquota -Y dns mmrepquota::HEADER:version:reserved:reserved:filesystemName:quotaType:id:name:blockUsage:blockQuota:blockLimit:blockInDoubt:blockGrace:filesUsage:filesQuota:filesLimit:filesInDoubt:filesGrace:remarks:quota:defQuota:fid:filesetname: mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:root: mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:users: mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:root: mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:users: mmrepquota::0:1:::dns:FILESET:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off::: mmrepquota::0:1:::dns:FILESET:1:users:0:4294967296:4294967296:0:none:1:0:0:0:none:e:on:off::: Bob Oesterlin Sr Principal Storage Engineer, Nuance On 3/28/17, 9:47 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Jaime Pinto" wrote: Any chance you guys in the GPFS devel team could patch the mmrepquota code so that during grace periods the report column for "none" would still be replaced with >>>*ONE*<<< word? By that I mean, instead of "2 days" for example, just print "2-days" or "2days" or "2_days", and so on. I have a number of scripts that fail for users when they are over their quotas under grace periods, because the report shifts the remaining information for that user 1 column to the right. Obviously it would cost me absolutely nothing to patch my scripts to deal with this, however the principle here is that the reports generated by GPFS should be the ones keeping consistence. Thanks Jaime ************************************ TELL US ABOUT YOUR SUCCESS STORIES https://urldefense.proofpoint.com/v2/url?u=http-3A__www.scinethpc.ca_testimonials&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzNsKbADKWCeLhA&s=TVGnqMwSWqNI1Vu1BlCcwXiVGsLUO9ZnbqlasVmT2HU&e= ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzNsKbADKWCeLhA&s=AXRwDMAVkYdwEaSFzejLQzNnS-KXoKj9GauzeEuu2H8&e= From Kevin.Buterbaugh at Vanderbilt.Edu Tue Mar 28 16:00:26 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 28 Mar 2017 15:00:26 +0000 Subject: [gpfsug-discuss] fix mmrepquota report format during grace periods In-Reply-To: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> Message-ID: <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> Hi Bob, Jaime, and GPFS team, That?s great for mmrepquota, but mmlsquota does not have a similar option AFAICT. That has really caused me grief ? for example, I?ve got a Perl script that takes mmlsquota output for a user and does two things: 1) converts it into something easier for them to parse, and 2) doesn?t display anything for the several dozen filesets they don?t have access to. That Perl script is ~300 lines and probably about a third of that is dealing with the grace period spacing issue? Kevin > On Mar 28, 2017, at 9:54 AM, Oesterlin, Robert wrote: > > Try running it with the ?-Y? option, it returns an easily to read output: > mmrepquota -Y dns > mmrepquota::HEADER:version:reserved:reserved:filesystemName:quotaType:id:name:blockUsage:blockQuota:blockLimit:blockInDoubt:blockGrace:filesUsage:filesQuota:filesLimit:filesInDoubt:filesGrace:remarks:quota:defQuota:fid:filesetname: > mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:root: > mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:users: > mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:root: > mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:users: > mmrepquota::0:1:::dns:FILESET:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off::: > mmrepquota::0:1:::dns:FILESET:1:users:0:4294967296:4294967296:0:none:1:0:0:0:none:e:on:off::: > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > > On 3/28/17, 9:47 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Jaime Pinto" wrote: > > Any chance you guys in the GPFS devel team could patch the mmrepquota > code so that during grace periods the report column for "none" would > still be replaced with >>>*ONE*<<< word? By that I mean, instead of "2 > days" for example, just print "2-days" or "2days" or "2_days", and so > on. > > I have a number of scripts that fail for users when they are over > their quotas under grace periods, because the report shifts the > remaining information for that user 1 column to the right. > > Obviously it would cost me absolutely nothing to patch my scripts to > deal with this, however the principle here is that the reports > generated by GPFS should be the ones keeping consistence. > > Thanks > Jaime > > > > > ************************************ > TELL US ABOUT YOUR SUCCESS STORIES > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.scinethpc.ca_testimonials&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzNsKbADKWCeLhA&s=TVGnqMwSWqNI1Vu1BlCcwXiVGsLUO9ZnbqlasVmT2HU&e= > ************************************ > --- > Jaime Pinto > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.ca > University of Toronto > 661 University Ave. (MaRS), Suite 1140 > Toronto, ON, M5G1M1 > P: 416-978-2755 > C: 416-505-1477 > > ---------------------------------------------------------------- > This message was sent using IMP at SciNet Consortium, University of Toronto. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzNsKbADKWCeLhA&s=AXRwDMAVkYdwEaSFzejLQzNnS-KXoKj9GauzeEuu2H8&e= > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Kevin.Buterbaugh at Vanderbilt.Edu Tue Mar 28 16:04:58 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 28 Mar 2017 15:04:58 +0000 Subject: [gpfsug-discuss] fix mmrepquota report format during grace periods In-Reply-To: <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> Message-ID: <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> Ugh! Of course, I?m wrong ? mmlsquota does support the ?-Y? option ? it?s just not documented. Why not???? Kevin > On Mar 28, 2017, at 10:00 AM, Buterbaugh, Kevin L wrote: > > Hi Bob, Jaime, and GPFS team, > > That?s great for mmrepquota, but mmlsquota does not have a similar option AFAICT. > > That has really caused me grief ? for example, I?ve got a Perl script that takes mmlsquota output for a user and does two things: 1) converts it into something easier for them to parse, and 2) doesn?t display anything for the several dozen filesets they don?t have access to. That Perl script is ~300 lines and probably about a third of that is dealing with the grace period spacing issue? > > Kevin > >> On Mar 28, 2017, at 9:54 AM, Oesterlin, Robert wrote: >> >> Try running it with the ?-Y? option, it returns an easily to read output: >> mmrepquota -Y dns >> mmrepquota::HEADER:version:reserved:reserved:filesystemName:quotaType:id:name:blockUsage:blockQuota:blockLimit:blockInDoubt:blockGrace:filesUsage:filesQuota:filesLimit:filesInDoubt:filesGrace:remarks:quota:defQuota:fid:filesetname: >> mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:root: >> mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:users: >> mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:root: >> mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:users: >> mmrepquota::0:1:::dns:FILESET:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off::: >> mmrepquota::0:1:::dns:FILESET:1:users:0:4294967296:4294967296:0:none:1:0:0:0:none:e:on:off::: >> >> Bob Oesterlin >> Sr Principal Storage Engineer, Nuance >> >> >> >> On 3/28/17, 9:47 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Jaime Pinto" wrote: >> >> Any chance you guys in the GPFS devel team could patch the mmrepquota >> code so that during grace periods the report column for "none" would >> still be replaced with >>>*ONE*<<< word? By that I mean, instead of "2 >> days" for example, just print "2-days" or "2days" or "2_days", and so >> on. >> >> I have a number of scripts that fail for users when they are over >> their quotas under grace periods, because the report shifts the >> remaining information for that user 1 column to the right. >> >> Obviously it would cost me absolutely nothing to patch my scripts to >> deal with this, however the principle here is that the reports >> generated by GPFS should be the ones keeping consistence. >> >> Thanks >> Jaime >> >> >> >> >> ************************************ >> TELL US ABOUT YOUR SUCCESS STORIES >> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.scinethpc.ca_testimonials&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzNsKbADKWCeLhA&s=TVGnqMwSWqNI1Vu1BlCcwXiVGsLUO9ZnbqlasVmT2HU&e= >> ************************************ >> --- >> Jaime Pinto >> SciNet HPC Consortium - Compute/Calcul Canada >> www.scinet.utoronto.ca - www.computecanada.ca >> University of Toronto >> 661 University Ave. (MaRS), Suite 1140 >> Toronto, ON, M5G1M1 >> P: 416-978-2755 >> C: 416-505-1477 >> >> ---------------------------------------------------------------- >> This message was sent using IMP at SciNet Consortium, University of Toronto. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzNsKbADKWCeLhA&s=AXRwDMAVkYdwEaSFzejLQzNnS-KXoKj9GauzeEuu2H8&e= >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From pinto at scinet.utoronto.ca Tue Mar 28 16:09:45 2017 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Tue, 28 Mar 2017 11:09:45 -0400 Subject: [gpfsug-discuss] fix mmrepquota report format during grace periods In-Reply-To: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> Message-ID: <20170328110945.18511am6iq6gf1dl@support.scinet.utoronto.ca> Aah! Another one of those options not so well documented or exposed: Usage: mmrepquota [-u] [-g] [-e] [-q] [-n] [-v] [-t] [--block-size {BlockSize | auto}] {-a | Device[:Fileset] ...} or mmrepquota -j [-e] [-q] [-n] [-v] [-t] [--block-size {BlockSize | auto}] {-a | Device ...} I agree, in this way it would be easier for a script to deal with fields that have spaces, using ':' as a field separator. However it mangles all the information together, making it very difficult for human eyes of a sysadmin to deal with in its original format. I'll take it under consideration for the scripts version (many of them to be revised), however the best is for the original and plain reports to have consistence. Thanks Jaime Quoting "Oesterlin, Robert" : > Try running it with the ?-Y? option, it returns an easily to read output: > mmrepquota -Y dns > mmrepquota::HEADER:version:reserved:reserved:filesystemName:quotaType:id:name:blockUsage:blockQuota:blockLimit:blockInDoubt:blockGrace:filesUsage:filesQuota:filesLimit:filesInDoubt:filesGrace:remarks:quota:defQuota:fid:filesetname: > mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:root: > mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:users: > mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:root: > mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:users: > mmrepquota::0:1:::dns:FILESET:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off::: > mmrepquota::0:1:::dns:FILESET:1:users:0:4294967296:4294967296:0:none:1:0:0:0:none:e:on:off::: > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > > On 3/28/17, 9:47 AM, "gpfsug-discuss-bounces at spectrumscale.org on > behalf of Jaime Pinto" behalf of pinto at scinet.utoronto.ca> wrote: > > Any chance you guys in the GPFS devel team could patch the mmrepquota > code so that during grace periods the report column for "none" would > still be replaced with >>>*ONE*<<< word? By that I mean, instead of "2 > days" for example, just print "2-days" or "2days" or "2_days", and so > on. > > I have a number of scripts that fail for users when they are over > their quotas under grace periods, because the report shifts the > remaining information for that user 1 column to the right. > > Obviously it would cost me absolutely nothing to patch my scripts to > deal with this, however the principle here is that the reports > generated by GPFS should be the ones keeping consistence. > > Thanks > Jaime > > > > > ************************************ > TELL US ABOUT YOUR SUCCESS STORIES > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.scinethpc.ca_testimonials&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzNsKbADKWCeLhA&s=TVGnqMwSWqNI1Vu1BlCcwXiVGsLUO9ZnbqlasVmT2HU&e= > ************************************ > --- > Jaime Pinto > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.ca > University of Toronto > 661 University Ave. (MaRS), Suite 1140 > Toronto, ON, M5G1M1 > P: 416-978-2755 > C: 416-505-1477 > > ---------------------------------------------------------------- > This message was sent using IMP at SciNet Consortium, University > of Toronto. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzNsKbADKWCeLhA&s=AXRwDMAVkYdwEaSFzejLQzNnS-KXoKj9GauzeEuu2H8&e= > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. From S.J.Thompson at bham.ac.uk Tue Mar 28 16:11:58 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Tue, 28 Mar 2017 15:11:58 +0000 Subject: [gpfsug-discuss] fix mmrepquota report format during grace periods In-Reply-To: <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> Message-ID: I thought this was because the -Y flag was going into new commands and being added to older commands during later releases. So it might be that it was added, but the docs not updated. Simon On 28/03/2017, 16:04, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Buterbaugh, Kevin L" wrote: >Ugh! Of course, I?m wrong ? mmlsquota does support the ?-Y? option ? >it?s just not documented. Why not???? > >Kevin > >> On Mar 28, 2017, at 10:00 AM, Buterbaugh, Kevin L >> wrote: >> >> Hi Bob, Jaime, and GPFS team, >> >> That?s great for mmrepquota, but mmlsquota does not have a similar >>option AFAICT. >> >> That has really caused me grief ? for example, I?ve got a Perl script >>that takes mmlsquota output for a user and does two things: 1) converts >>it into something easier for them to parse, and 2) doesn?t display >>anything for the several dozen filesets they don?t have access to. That >>Perl script is ~300 lines and probably about a third of that is dealing >>with the grace period spacing issue? >> >> Kevin >> >>> On Mar 28, 2017, at 9:54 AM, Oesterlin, Robert >>> wrote: >>> >>> Try running it with the ?-Y? option, it returns an easily to read >>>output: >>> mmrepquota -Y dns >>> >>>mmrepquota::HEADER:version:reserved:reserved:filesystemName:quotaType:id >>>:name:blockUsage:blockQuota:blockLimit:blockInDoubt:blockGrace:filesUsag >>>e:filesQuota:filesLimit:filesInDoubt:filesGrace:remarks:quota:defQuota:f >>>id:filesetname: >>> >>>mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:ro >>>ot: >>> >>>mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:us >>>ers: >>> >>>mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:ro >>>ot: >>> >>>mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:us >>>ers: >>> >>>mmrepquota::0:1:::dns:FILESET:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off: >>>:: >>> >>>mmrepquota::0:1:::dns:FILESET:1:users:0:4294967296:4294967296:0:none:1:0 >>>:0:0:none:e:on:off::: >>> >>> Bob Oesterlin >>> Sr Principal Storage Engineer, Nuance >>> >>> >>> >>> On 3/28/17, 9:47 AM, "gpfsug-discuss-bounces at spectrumscale.org on >>>behalf of Jaime Pinto" >>behalf of pinto at scinet.utoronto.ca> wrote: >>> >>> Any chance you guys in the GPFS devel team could patch the >>>mmrepquota >>> code so that during grace periods the report column for "none" would >>> >>> still be replaced with >>>*ONE*<<< word? By that I mean, instead of >>>"2 >>> days" for example, just print "2-days" or "2days" or "2_days", and >>>so >>> on. >>> >>> I have a number of scripts that fail for users when they are over >>> their quotas under grace periods, because the report shifts the >>> remaining information for that user 1 column to the right. >>> >>> Obviously it would cost me absolutely nothing to patch my scripts to >>> >>> deal with this, however the principle here is that the reports >>> generated by GPFS should be the ones keeping consistence. >>> >>> Thanks >>> Jaime >>> >>> >>> >>> >>> ************************************ >>> TELL US ABOUT YOUR SUCCESS STORIES >>> >>>https://urldefense.proofpoint.com/v2/url?u=http-3A__www.scinethpc.ca_tes >>>timonials&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDew >>>t1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzN >>>sKbADKWCeLhA&s=TVGnqMwSWqNI1Vu1BlCcwXiVGsLUO9ZnbqlasVmT2HU&e= >>> ************************************ >>> --- >>> Jaime Pinto >>> SciNet HPC Consortium - Compute/Calcul Canada >>> www.scinet.utoronto.ca - www.computecanada.ca >>> University of Toronto >>> 661 University Ave. (MaRS), Suite 1140 >>> Toronto, ON, M5G1M1 >>> P: 416-978-2755 >>> C: 416-505-1477 >>> >>> ---------------------------------------------------------------- >>> This message was sent using IMP at SciNet Consortium, University of >>>Toronto. >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> >>>https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_l >>>istinfo_gpfsug-2Ddiscuss&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0 >>>rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCI >>>ZvUgTr2CN-RqtzNsKbADKWCeLhA&s=AXRwDMAVkYdwEaSFzejLQzNnS-KXoKj9GauzeEuu2H >>>8&e= >>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Kevin.Buterbaugh at Vanderbilt.Edu Tue Mar 28 16:34:33 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 28 Mar 2017 15:34:33 +0000 Subject: [gpfsug-discuss] fix mmrepquota report format during grace periods In-Reply-To: References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> Message-ID: <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> All, Could someone(s) from the GPFS team please: 1) see that the appropriate documentation gets updated (manuals, ?-h? option to commands, the man pages for the commands, etc.)? 2) let us know what version of GPFS introduced the undocumented ?-Y? option for mmrepquota and mmlsquota? I?ve got numerous quota related scripts and I?m just curious to go back and figure out how much time I?ve wasted because I didn?t know about it. These ?unknown unknowns? bite us again? ;-) Kevin > On Mar 28, 2017, at 10:11 AM, Simon Thompson (Research Computing - IT Services) wrote: > > I thought this was because the -Y flag was going into new commands and > being added to older commands during later releases. > > So it might be that it was added, but the docs not updated. > > Simon > > On 28/03/2017, 16:04, "gpfsug-discuss-bounces at spectrumscale.org on behalf > of Buterbaugh, Kevin L" behalf of Kevin.Buterbaugh at Vanderbilt.Edu> wrote: > >> Ugh! Of course, I?m wrong ? mmlsquota does support the ?-Y? option ? >> it?s just not documented. Why not???? >> >> Kevin >> >>> On Mar 28, 2017, at 10:00 AM, Buterbaugh, Kevin L >>> wrote: >>> >>> Hi Bob, Jaime, and GPFS team, >>> >>> That?s great for mmrepquota, but mmlsquota does not have a similar >>> option AFAICT. >>> >>> That has really caused me grief ? for example, I?ve got a Perl script >>> that takes mmlsquota output for a user and does two things: 1) converts >>> it into something easier for them to parse, and 2) doesn?t display >>> anything for the several dozen filesets they don?t have access to. That >>> Perl script is ~300 lines and probably about a third of that is dealing >>> with the grace period spacing issue? >>> >>> Kevin >>> >>>> On Mar 28, 2017, at 9:54 AM, Oesterlin, Robert >>>> wrote: >>>> >>>> Try running it with the ?-Y? option, it returns an easily to read >>>> output: >>>> mmrepquota -Y dns >>>> >>>> mmrepquota::HEADER:version:reserved:reserved:filesystemName:quotaType:id >>>> :name:blockUsage:blockQuota:blockLimit:blockInDoubt:blockGrace:filesUsag >>>> e:filesQuota:filesLimit:filesInDoubt:filesGrace:remarks:quota:defQuota:f >>>> id:filesetname: >>>> >>>> mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:ro >>>> ot: >>>> >>>> mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:us >>>> ers: >>>> >>>> mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:ro >>>> ot: >>>> >>>> mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:us >>>> ers: >>>> >>>> mmrepquota::0:1:::dns:FILESET:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off: >>>> :: >>>> >>>> mmrepquota::0:1:::dns:FILESET:1:users:0:4294967296:4294967296:0:none:1:0 >>>> :0:0:none:e:on:off::: >>>> >>>> Bob Oesterlin >>>> Sr Principal Storage Engineer, Nuance >>>> >>>> >>>> >>>> On 3/28/17, 9:47 AM, "gpfsug-discuss-bounces at spectrumscale.org on >>>> behalf of Jaime Pinto" >>> behalf of pinto at scinet.utoronto.ca> wrote: >>>> >>>> Any chance you guys in the GPFS devel team could patch the >>>> mmrepquota >>>> code so that during grace periods the report column for "none" would >>>> >>>> still be replaced with >>>*ONE*<<< word? By that I mean, instead of >>>> "2 >>>> days" for example, just print "2-days" or "2days" or "2_days", and >>>> so >>>> on. >>>> >>>> I have a number of scripts that fail for users when they are over >>>> their quotas under grace periods, because the report shifts the >>>> remaining information for that user 1 column to the right. >>>> >>>> Obviously it would cost me absolutely nothing to patch my scripts to >>>> >>>> deal with this, however the principle here is that the reports >>>> generated by GPFS should be the ones keeping consistence. >>>> >>>> Thanks >>>> Jaime >>>> >>>> >>>> >>>> >>>> ************************************ >>>> TELL US ABOUT YOUR SUCCESS STORIES >>>> >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.scinethpc.ca_tes >>>> timonials&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDew >>>> t1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzN >>>> sKbADKWCeLhA&s=TVGnqMwSWqNI1Vu1BlCcwXiVGsLUO9ZnbqlasVmT2HU&e= >>>> ************************************ >>>> --- >>>> Jaime Pinto >>>> SciNet HPC Consortium - Compute/Calcul Canada >>>> www.scinet.utoronto.ca - www.computecanada.ca >>>> University of Toronto >>>> 661 University Ave. (MaRS), Suite 1140 >>>> Toronto, ON, M5G1M1 >>>> P: 416-978-2755 >>>> C: 416-505-1477 >>>> >>>> ---------------------------------------------------------------- >>>> This message was sent using IMP at SciNet Consortium, University of >>>> Toronto. >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_l >>>> istinfo_gpfsug-2Ddiscuss&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0 >>>> rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCI >>>> ZvUgTr2CN-RqtzNsKbADKWCeLhA&s=AXRwDMAVkYdwEaSFzejLQzNnS-KXoKj9GauzeEuu2H >>>> 8&e= >>>> >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From makaplan at us.ibm.com Tue Mar 28 17:14:18 2017 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 28 Mar 2017 11:14:18 -0500 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com><4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu><4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> Message-ID: Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and Programming) Reference I only found a few commands that officially are documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. But as many of you have discovered, -Y is accepted and yields "interesting" output for many of the mmXXXX commands. Moreover the output *seems to* have easily discernible patterns that can be parsed by simple programs. I believe there is no guarantee that the exact command output formats will not change from release to release, so, as a practical matter, if you're going to parse command output, you're probably better off parsing the -Y output, even if that is not officially supported. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathon.anderson at colorado.edu Tue Mar 28 17:17:29 2017 From: jonathon.anderson at colorado.edu (Jonathon A Anderson) Date: Tue, 28 Mar 2017 16:17:29 +0000 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> Message-ID: I had a bit of a run-in with a gpfs developer at a conference once when I complained about how command output changes frequently, breaking customer scripts. He was confused why we weren?t using `-Y`, and didn?t believe me that it?s simply not documented anywhere! `-Y` output is essential for interfacing programmatically with GPFS, and I don?t understand why it?s not mentioned in any of the guides or manpages. (Though apparently, according to this thead, it?s since appeared in some newer manpages.) ~jonathon On 3/28/17, 10:14 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Marc A Kaplan" wrote: Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and Programming) Reference I only found a few commands that officially are documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. But as many of you have discovered, -Y is accepted and yields "interesting" output for many of the mmXXXX commands. Moreover the output *seems to* have easily discernible patterns that can be parsed by simple programs. I believe there is no guarantee that the exact command output formats will not change from release to release, so, as a practical matter, if you're going to parse command output, you're probably better off parsing the -Y output, even if that is not officially supported. From stockf at us.ibm.com Tue Mar 28 17:21:57 2017 From: stockf at us.ibm.com (Frederick Stock) Date: Tue, 28 Mar 2017 11:21:57 -0500 Subject: [gpfsug-discuss] fix mmrepquota report format during grace periods In-Reply-To: <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com><4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu><4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> Message-ID: My understanding is that with the upcoming 4.2.3 release the -Y option will be documented for many commands, but perhaps not all. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 03/28/2017 11:35 AM Subject: Re: [gpfsug-discuss] fix mmrepquota report format during grace periods Sent by: gpfsug-discuss-bounces at spectrumscale.org All, Could someone(s) from the GPFS team please: 1) see that the appropriate documentation gets updated (manuals, ?-h? option to commands, the man pages for the commands, etc.)? 2) let us know what version of GPFS introduced the undocumented ?-Y? option for mmrepquota and mmlsquota? I?ve got numerous quota related scripts and I?m just curious to go back and figure out how much time I?ve wasted because I didn?t know about it. These ?unknown unknowns? bite us again? ;-) Kevin > On Mar 28, 2017, at 10:11 AM, Simon Thompson (Research Computing - IT Services) wrote: > > I thought this was because the -Y flag was going into new commands and > being added to older commands during later releases. > > So it might be that it was added, but the docs not updated. > > Simon > > On 28/03/2017, 16:04, "gpfsug-discuss-bounces at spectrumscale.org on behalf > of Buterbaugh, Kevin L" behalf of Kevin.Buterbaugh at Vanderbilt.Edu> wrote: > >> Ugh! Of course, I?m wrong ? mmlsquota does support the ?-Y? option ? >> it?s just not documented. Why not???? >> >> Kevin >> >>> On Mar 28, 2017, at 10:00 AM, Buterbaugh, Kevin L >>> wrote: >>> >>> Hi Bob, Jaime, and GPFS team, >>> >>> That?s great for mmrepquota, but mmlsquota does not have a similar >>> option AFAICT. >>> >>> That has really caused me grief ? for example, I?ve got a Perl script >>> that takes mmlsquota output for a user and does two things: 1) converts >>> it into something easier for them to parse, and 2) doesn?t display >>> anything for the several dozen filesets they don?t have access to. That >>> Perl script is ~300 lines and probably about a third of that is dealing >>> with the grace period spacing issue? >>> >>> Kevin >>> >>>> On Mar 28, 2017, at 9:54 AM, Oesterlin, Robert >>>> wrote: >>>> >>>> Try running it with the ?-Y? option, it returns an easily to read >>>> output: >>>> mmrepquota -Y dns >>>> >>>> mmrepquota::HEADER:version:reserved:reserved:filesystemName:quotaType:id >>>> :name:blockUsage:blockQuota:blockLimit:blockInDoubt:blockGrace:filesUsag >>>> e:filesQuota:filesLimit:filesInDoubt:filesGrace:remarks:quota:defQuota:f >>>> id:filesetname: >>>> >>>> mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:ro >>>> ot: >>>> >>>> mmrepquota::0:1:::dns:USR:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:us >>>> ers: >>>> >>>> mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:0:ro >>>> ot: >>>> >>>> mmrepquota::0:1:::dns:GRP:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off:1:us >>>> ers: >>>> >>>> mmrepquota::0:1:::dns:FILESET:0:root:0:0:0:0:none:1:0:0:0:none:i:on:off: >>>> :: >>>> >>>> mmrepquota::0:1:::dns:FILESET:1:users:0:4294967296:4294967296:0:none:1:0 >>>> :0:0:none:e:on:off::: >>>> >>>> Bob Oesterlin >>>> Sr Principal Storage Engineer, Nuance >>>> >>>> >>>> >>>> On 3/28/17, 9:47 AM, "gpfsug-discuss-bounces at spectrumscale.org on >>>> behalf of Jaime Pinto" >>> behalf of pinto at scinet.utoronto.ca> wrote: >>>> >>>> Any chance you guys in the GPFS devel team could patch the >>>> mmrepquota >>>> code so that during grace periods the report column for "none" would >>>> >>>> still be replaced with >>>*ONE*<<< word? By that I mean, instead of >>>> "2 >>>> days" for example, just print "2-days" or "2days" or "2_days", and >>>> so >>>> on. >>>> >>>> I have a number of scripts that fail for users when they are over >>>> their quotas under grace periods, because the report shifts the >>>> remaining information for that user 1 column to the right. >>>> >>>> Obviously it would cost me absolutely nothing to patch my scripts to >>>> >>>> deal with this, however the principle here is that the reports >>>> generated by GPFS should be the ones keeping consistence. >>>> >>>> Thanks >>>> Jaime >>>> >>>> >>>> >>>> >>>> ************************************ >>>> TELL US ABOUT YOUR SUCCESS STORIES >>>> >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.scinethpc.ca_tes >>>> timonials&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDew >>>> t1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCIZvUgTr2CN-RqtzN >>>> sKbADKWCeLhA&s=TVGnqMwSWqNI1Vu1BlCcwXiVGsLUO9ZnbqlasVmT2HU&e= >>>> ************************************ >>>> --- >>>> Jaime Pinto >>>> SciNet HPC Consortium - Compute/Calcul Canada >>>> www.scinet.utoronto.ca - www.computecanada.ca >>>> University of Toronto >>>> 661 University Ave. (MaRS), Suite 1140 >>>> Toronto, ON, M5G1M1 >>>> P: 416-978-2755 >>>> C: 416-505-1477 >>>> >>>> ---------------------------------------------------------------- >>>> This message was sent using IMP at SciNet Consortium, University of >>>> Toronto. >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_l >>>> istinfo_gpfsug-2Ddiscuss&d=DwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0 >>>> rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=PnZlzkqTEICwnHCI >>>> ZvUgTr2CN-RqtzNsKbADKWCeLhA&s=AXRwDMAVkYdwEaSFzejLQzNnS-KXoKj9GauzeEuu2H >>>> 8&e= >>>> >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From luis.bolinches at fi.ibm.com Tue Mar 28 17:22:04 2017 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Tue, 28 Mar 2017 16:22:04 +0000 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: References: , <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com><4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu><4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu><5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> Message-ID: An HTML attachment was scrubbed... URL: From jonathon.anderson at colorado.edu Tue Mar 28 17:24:19 2017 From: jonathon.anderson at colorado.edu (Jonathon A Anderson) Date: Tue, 28 Mar 2017 16:24:19 +0000 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> Message-ID: <99B1AC2E-91B4-4A7F-B270-8FDDC49918EA@colorado.edu> Looks great. I look forward to its maturity. ~jonathon On 3/28/17, 10:22 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Luis Bolinches" wrote: Hi While I understand the frustration of tiem that could be used otherwise. depending of what you are plannig with script wrapping I would recommend you seriously take a look to the REST API http://files.gpfsug.org/presentations/2017/Ehningen/23_-_SSUG17DE_-_Alexander_Wolf-Reber_-_Spectrum_Scale_ReST_API.pdf -- Yst?v?llisin terveisin / Kind regards / Saludos cordiales / Salutations Luis Bolinches Lab Services http://www-03.ibm.com/systems/services/labservices/ IBM Laajalahdentie 23 (main Entrance) Helsinki, 00330 Finland Phone: +358 503112585 "If you continually give you will continually have." Anonymous ----- Original message ----- From: Jonathon A Anderson Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list Cc: Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Date: Tue, Mar 28, 2017 7:17 PM I had a bit of a run-in with a gpfs developer at a conference once when I complained about how command output changes frequently, breaking customer scripts. He was confused why we weren?t using `-Y`, and didn?t believe me that it?s simply not documented anywhere! `-Y` output is essential for interfacing programmatically with GPFS, and I don?t understand why it?s not mentioned in any of the guides or manpages. (Though apparently, according to this thead, it?s since appeared in some newer manpages.) ~jonathon On 3/28/17, 10:14 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Marc A Kaplan" wrote: Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and Programming) Reference I only found a few commands that officially are documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. But as many of you have discovered, -Y is accepted and yields "interesting" output for many of the mmXXXX commands. Moreover the output *seems to* have easily discernible patterns that can be parsed by simple programs. I believe there is no guarantee that the exact command output formats will not change from release to release, so, as a practical matter, if you're going to parse command output, you're probably better off parsing the -Y output, even if that is not officially supported. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland From S.J.Thompson at bham.ac.uk Tue Mar 28 17:37:12 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Tue, 28 Mar 2017 16:37:12 +0000 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> Message-ID: I thought the header rows defined what the format of the output was. Its a bit weird as there can be multiple header rows for different content rows ... Simon From: > on behalf of Marc A Kaplan > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Tuesday, 28 March 2017 at 17:14 To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and Programming) Reference I only found a few commands that officially are documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. But as many of you have discovered, -Y is accepted and yields "interesting" output for many of the mmXXXX commands. Moreover the output *seems to* have easily discernible patterns that can be parsed by simple programs. I believe there is no guarantee that the exact command output formats will not change from release to release, so, as a practical matter, if you're going to parse command output, you're probably better off parsing the -Y output, even if that is not officially supported. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathon.anderson at colorado.edu Tue Mar 28 17:37:47 2017 From: jonathon.anderson at colorado.edu (Jonathon A Anderson) Date: Tue, 28 Mar 2017 16:37:47 +0000 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> Message-ID: Sure; but that only helps if you know the flag even exists. ~jonathon On 3/28/17, 10:37 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Simon Thompson (Research Computing - IT Services)" wrote: I thought the header rows defined what the format of the output was. Its a bit weird as there can be multiple header rows for different content rows ... Simon From: on behalf of Marc A Kaplan Reply-To: "gpfsug-discuss at spectrumscale.org" Date: Tuesday, 28 March 2017 at 17:14 To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and Programming) Reference I only found a few commands that officially are documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. But as many of you have discovered, -Y is accepted and yields "interesting" output for many of the mmXXXX commands. Moreover the output *seems to* have easily discernible patterns that can be parsed by simple programs. I believe there is no guarantee that the exact command output formats will not change from release to release, so, as a practical matter, if you're going to parse command output, you're probably better off parsing the -Y output, even if that is not officially supported. From Kevin.Buterbaugh at Vanderbilt.Edu Tue Mar 28 17:47:14 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 28 Mar 2017 16:47:14 +0000 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> Message-ID: <53E9C309-3498-40FB-8018-8CABD9C6C316@vanderbilt.edu> Agreed. From what has been said on this thread about ?-Y? being unsupported and the command output could change from release to release ? well, that?s a ?known unknown? that can be dealt with. But the fact that ?-Y? was completely undocumented (at least as far as mmrepquota / mmlsquota are concerned) makes ?-Y? one of those ?unknown unknowns?? ;-) Kevin > On Mar 28, 2017, at 11:37 AM, Jonathon A Anderson wrote: > > Sure; but that only helps if you know the flag even exists. > > ~jonathon > > > On 3/28/17, 10:37 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Simon Thompson (Research Computing - IT Services)" wrote: > > I thought the header rows defined what the format of the output was. Its a bit weird as there can be multiple header rows for different content rows ... > > > Simon > > > From: on behalf of Marc A Kaplan > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Tuesday, 28 March 2017 at 17:14 > To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! > > > > Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and Programming) Reference I only found a few commands that officially are documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. > > But as many of you have discovered, -Y is accepted and yields "interesting" output for many of the mmXXXX commands. > Moreover the output *seems to* have easily discernible patterns that can be parsed by simple programs. > > I believe there is no guarantee that the exact command output formats will not change from release to release, so, as a practical matter, if you're going to parse command output, you're probably better off parsing the -Y output, > even if that is not officially supported. > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From S.J.Thompson at bham.ac.uk Tue Mar 28 17:51:12 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Tue, 28 Mar 2017 16:51:12 +0000 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> Message-ID: Come to one of the user group events going on around the world in the next few months. NERSC, Berkeley, USA, 4th/5th April: https://www.nersc.gov/research-and-development/data-analytics/spectrum-user -group-meeting/ *** REGISTRATION CLOSES TOMORROW 29th *** Sydney, Australia, 27th April: https://www.eventbrite.com/e/spectrum-scale-user-group-australia-gpfsugaus- april-2017-tickets-29136246297 Manchester, UK, 9th/10th May: https://www.eventbrite.com/e/spectrum-scalegpfs-user-group-spring-2017-regi stration-32113696932 I know the presence of the -Y flag has been discussed there, so its a great place to pick up tips! Simon On 28/03/2017, 17:37, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Jonathon A Anderson" wrote: >Sure; but that only helps if you know the flag even exists. > >~jonathon > > >On 3/28/17, 10:37 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf >of Simon Thompson (Research Computing - IT Services)" >S.J.Thompson at bham.ac.uk> wrote: > > I thought the header rows defined what the format of the output was. >Its a bit weird as there can be multiple header rows for different >content rows ... > > > Simon > > > From: on behalf of Marc A >Kaplan > Reply-To: "gpfsug-discuss at spectrumscale.org" > > Date: Tuesday, 28 March 2017 at 17:14 > To: "gpfsug-discuss at spectrumscale.org" > > Subject: Re: [gpfsug-discuss] -Y option for many commands, precious >few officially! > > > > Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and >Programming) Reference I only found a few commands that officially are >documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. > > But as many of you have discovered, -Y is accepted and yields >"interesting" output for many of the mmXXXX commands. > Moreover the output *seems to* have easily discernible patterns that >can be parsed by simple programs. > > I believe there is no guarantee that the exact command output formats >will not change from release to release, so, as a practical matter, if >you're going to parse command output, you're probably better off parsing >the -Y output, > even if that is not officially supported. > > > > > >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss From carlz at us.ibm.com Tue Mar 28 18:02:32 2017 From: carlz at us.ibm.com (Carl Zetie) Date: Tue, 28 Mar 2017 17:02:32 +0000 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From jonathon.anderson at colorado.edu Tue Mar 28 18:08:20 2017 From: jonathon.anderson at colorado.edu (Jonathon A Anderson) Date: Tue, 28 Mar 2017 17:08:20 +0000 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: References: <20B7A4A2-6663-42B4-B3E9-AA062F427A40@nuance.com> <4C1ABD3C-6CD2-4DBC-A17A-CD42162DF994@vanderbilt.edu> <4354CC8F-E886-48B3-917D-A6BCB1AF570A@vanderbilt.edu> <5564D0CA-D7A3-4159-B667-7344FC65C84F@vanderbilt.edu> Message-ID: <67E4C386-8884-4E92-B262-089FDFA32C94@colorado.edu> > I know the presence of the -Y flag has been discussed there, so it?s a great place to pick up tips! Sure, and I ?picked up this tip? at a community event; but really, it shouldn?t have to be picked up as arcana. It should be documented. Glad to hear that that?s the plan. ~jonathon On 3/28/17, 10:51 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Simon Thompson (Research Computing - IT Services)" wrote: Come to one of the user group events going on around the world in the next few months. NERSC, Berkeley, USA, 4th/5th April: https://www.nersc.gov/research-and-development/data-analytics/spectrum-user -group-meeting/ *** REGISTRATION CLOSES TOMORROW 29th *** Sydney, Australia, 27th April: https://www.eventbrite.com/e/spectrum-scale-user-group-australia-gpfsugaus- april-2017-tickets-29136246297 Manchester, UK, 9th/10th May: https://www.eventbrite.com/e/spectrum-scalegpfs-user-group-spring-2017-regi stration-32113696932 I know the presence of the -Y flag has been discussed there, so its a great place to pick up tips! Simon On 28/03/2017, 17:37, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Jonathon A Anderson" wrote: >Sure; but that only helps if you know the flag even exists. > >~jonathon > > >On 3/28/17, 10:37 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf >of Simon Thompson (Research Computing - IT Services)" >S.J.Thompson at bham.ac.uk> wrote: > > I thought the header rows defined what the format of the output was. >Its a bit weird as there can be multiple header rows for different >content rows ... > > > Simon > > > From: on behalf of Marc A >Kaplan > Reply-To: "gpfsug-discuss at spectrumscale.org" > > Date: Tuesday, 28 March 2017 at 17:14 > To: "gpfsug-discuss at spectrumscale.org" > > Subject: Re: [gpfsug-discuss] -Y option for many commands, precious >few officially! > > > > Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and >Programming) Reference I only found a few commands that officially are >documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. > > But as many of you have discovered, -Y is accepted and yields >"interesting" output for many of the mmXXXX commands. > Moreover the output *seems to* have easily discernible patterns that >can be parsed by simple programs. > > I believe there is no guarantee that the exact command output formats >will not change from release to release, so, as a practical matter, if >you're going to parse command output, you're probably better off parsing >the -Y output, > even if that is not officially supported. > > > > > >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From aleciarm at us.ibm.com Tue Mar 28 18:17:11 2017 From: aleciarm at us.ibm.com (Alecia A Ramsay) Date: Tue, 28 Mar 2017 12:17:11 -0500 Subject: [gpfsug-discuss] -Y option for many commands, precious few officially! In-Reply-To: References: Message-ID: For the benefit of those on digest, here's Carl's list again in plain text (I hope). I have also notified our documentation team that this is of great interest to clients, so they are aware. In the 4.2.3 draft documentation, the -y option information has been added to a number of commands: -Y option Added the -Y option to the following commands: mmblock mmhealth mmlsfileset mmlsnodeclass mmnetverify mmcloudgateway mmkeyserv mmlsfs mmlsnsd mmnfs mmdf mmlscluster mmlslicense mmlspolicy mmrepquota mmdiag mmlsconfig mmlsmgr mmlsquota mmsmb mmgetstate mmlsdisk mmlsmount mmlssnapshot mmuserauth Alecia A. Ramsay, PMP? Program Manager, New Technology Introduction IBM Systems - Storage aleciarm at us.ibm.com work: 919-435-6494; mobile: 651-260-4928 http://ibm.biz/NewTechnologyIntroduction From: Carl Zetie/Fairfax/IBM at IBMUS To: gpfsug-discuss at spectrumscale.org Date: 03/28/2017 01:03 PM Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Sent by: gpfsug-discuss-bounces at spectrumscale.org I checked the draft documentation for 4.2.3, and here is the list of what is planned. As usual -- please remember this is an Intention not a commitment, and subject to change between now and the release. regards, Carl -Y option Added the -Y option to the following commands: mmblock mmhealth mmlsfileset mmlsnodeclass mmnetverify mmcloudgateway mmkeyserv mmlsfs mmlsnsd mmnfs mmdf mmlscluster mmlslicense mmlspolicy mmrepquota mmdiag mmlsconfig mmlsmgr mmlsquota mmsmb mmgetstate mmlsdisk mmlsmount mmlssnapshot mmuserauth Carl Zetie Offering Manager for Spectrum Scale, IBM carlz at us.ibm.com ----- Original message ----- From: gpfsug-discuss-request at spectrumscale.org Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Cc: Subject: gpfsug-discuss Digest, Vol 62, Issue 70 Date: Tue, Mar 28, 2017 12:38 PM Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: -Y option for many commands, precious few officially! (Luis Bolinches) 2. Re: -Y option for many commands, precious few officially! (Jonathon A Anderson) 3. Re: -Y option for many commands, precious few officially! (Simon Thompson (Research Computing - IT Services)) 4. Re: -Y option for many commands, precious few officially! (Jonathon A Anderson) ---------------------------------------------------------------------- Message: 1 Date: Tue, 28 Mar 2017 16:22:04 +0000 From: "Luis Bolinches" To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Message-ID: Content-Type: text/plain; charset="us-ascii" An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20170328/694119bd/attachment-0001.html > ------------------------------ Message: 2 Date: Tue, 28 Mar 2017 16:24:19 +0000 From: Jonathon A Anderson To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Message-ID: <99B1AC2E-91B4-4A7F-B270-8FDDC49918EA at colorado.edu> Content-Type: text/plain; charset="utf-8" Looks great. I look forward to its maturity. ~jonathon On 3/28/17, 10:22 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Luis Bolinches" wrote: Hi While I understand the frustration of tiem that could be used otherwise. depending of what you are plannig with script wrapping I would recommend you seriously take a look to the REST API http://files.gpfsug.org/presentations/2017/Ehningen/23_-_SSUG17DE_-_Alexander_Wolf-Reber_-_Spectrum_Scale_ReST_API.pdf -- Yst?v?llisin terveisin / Kind regards / Saludos cordiales / Salutations Luis Bolinches Lab Services http://www-03.ibm.com/systems/services/labservices/ IBM Laajalahdentie 23 (main Entrance) Helsinki, 00330 Finland Phone: +358 503112585 "If you continually give you will continually have." Anonymous ----- Original message ----- From: Jonathon A Anderson Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list Cc: Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Date: Tue, Mar 28, 2017 7:17 PM I had a bit of a run-in with a gpfs developer at a conference once when I complained about how command output changes frequently, breaking customer scripts. He was confused why we weren?t using `-Y`, and didn?t believe me that it?s simply not documented anywhere! `-Y` output is essential for interfacing programmatically with GPFS, and I don?t understand why it?s not mentioned in any of the guides or manpages. (Though apparently, according to this thead, it?s since appeared in some newer manpages.) ~jonathon On 3/28/17, 10:14 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Marc A Kaplan" wrote: Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and Programming) Reference I only found a few commands that officially are documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. But as many of you have discovered, -Y is accepted and yields "interesting" output for many of the mmXXXX commands. Moreover the output *seems to* have easily discernible patterns that can be parsed by simple programs. I believe there is no guarantee that the exact command output formats will not change from release to release, so, as a practical matter, if you're going to parse command output, you're probably better off parsing the -Y output, even if that is not officially supported. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland ------------------------------ Message: 3 Date: Tue, 28 Mar 2017 16:37:12 +0000 From: "Simon Thompson (Research Computing - IT Services)" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Message-ID: Content-Type: text/plain; charset="us-ascii" I thought the header rows defined what the format of the output was. Its a bit weird as there can be multiple header rows for different content rows ... Simon From: > on behalf of Marc A Kaplan > Reply-To: "gpfsug-discuss at spectrumscale.org< mailto:gpfsug-discuss at spectrumscale.org>" > Date: Tuesday, 28 March 2017 at 17:14 To: "gpfsug-discuss at spectrumscale.org< mailto:gpfsug-discuss at spectrumscale.org>" > Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and Programming) Reference I only found a few commands that officially are documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. But as many of you have discovered, -Y is accepted and yields "interesting" output for many of the mmXXXX commands. Moreover the output *seems to* have easily discernible patterns that can be parsed by simple programs. I believe there is no guarantee that the exact command output formats will not change from release to release, so, as a practical matter, if you're going to parse command output, you're probably better off parsing the -Y output, even if that is not officially supported. -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20170328/2c551c41/attachment-0001.html > ------------------------------ Message: 4 Date: Tue, 28 Mar 2017 16:37:47 +0000 From: Jonathon A Anderson To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Message-ID: Content-Type: text/plain; charset="utf-8" Sure; but that only helps if you know the flag even exists. ~jonathon On 3/28/17, 10:37 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Simon Thompson (Research Computing - IT Services)" wrote: I thought the header rows defined what the format of the output was. Its a bit weird as there can be multiple header rows for different content rows ... Simon From: on behalf of Marc A Kaplan Reply-To: "gpfsug-discuss at spectrumscale.org" Date: Tuesday, 28 March 2017 at 17:14 To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] -Y option for many commands, precious few officially! Just looking/scanning the latest (4.2.2) Spectrum Scale Command (and Programming) Reference I only found a few commands that officially are documented as supporting a -Y option: mmnfs, mmsmb, mmuserauth. But as many of you have discovered, -Y is accepted and yields "interesting" output for many of the mmXXXX commands. Moreover the output *seems to* have easily discernible patterns that can be parsed by simple programs. I believe there is no guarantee that the exact command output formats will not change from release to release, so, as a practical matter, if you're going to parse command output, you're probably better off parsing the -Y output, even if that is not officially supported. ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 62, Issue 70 ********************************************** _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From ckrafft at de.ibm.com Wed Mar 29 12:34:29 2017 From: ckrafft at de.ibm.com (Christoph Krafft) Date: Wed, 29 Mar 2017 12:34:29 +0100 Subject: [gpfsug-discuss] GPFS / Spectrum Scale support with x86 SUSE SLES 12 SP2 to be expected when? Message-ID: Folks, does anyone have any tentative information when the above SUSE OS Level will be supported? Thank you in advance for hints and tips! Mit freundlichen Gr??en / Sincerely Christoph Krafft Client Technical Specialist - Power Systems, IBM Systems Certified IT Specialist @ The Open Group Phone: +49 (0) 7034 643 2171 IBM Deutschland GmbH Mobile: +49 (0) 160 97 81 86 12 Am Weiher 24 Email: ckrafft at de.ibm.com 65451 Kelsterbach Germany IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter Gesch?ftsf?hrung: Martina Koederitz (Vorsitzende), Nicole Reimer, Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Stefan Lutz Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecblank.gif Type: image/gif Size: 45 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 16373446.gif Type: image/gif Size: 1851 bytes Desc: not available URL: From aaron.knister at gmail.com Wed Mar 29 14:18:25 2017 From: aaron.knister at gmail.com (Aaron Knister) Date: Wed, 29 Mar 2017 09:18:25 -0400 Subject: [gpfsug-discuss] GPFS / Spectrum Scale support with x86 SUSE SLES 12 SP2 to be expected when? In-Reply-To: References: Message-ID: <57030974-8C80-4808-87F1-3540E1271D6E@gmail.com> I could be wrong but I believe it is currently supported (as in it works, at least that's my recollection). Did you try with a particular version that wouldn't work? Sent from my iPhone > On Mar 29, 2017, at 07:34, Christoph Krafft wrote: > > Folks, > > does anyone have any tentative information when the above SUSE OS Level will be supported? > > Thank you in advance for hints and tips! > > > Mit freundlichen Gr??en / Sincerely > > Christoph Krafft > > Client Technical Specialist - Power Systems, IBM Systems > Certified IT Specialist @ The Open Group > > > Phone: +49 (0) 7034 643 2171 IBM Deutschland GmbH > <16373446.gif> > Mobile: +49 (0) 160 97 81 86 12 Am Weiher 24 > Email: ckrafft at de.ibm.com 65451 Kelsterbach > Germany > IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter > Gesch?ftsf?hrung: Martina Koederitz (Vorsitzende), Nicole Reimer, Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Stefan Lutz > Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Wed Mar 29 14:55:17 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Wed, 29 Mar 2017 09:55:17 -0400 Subject: [gpfsug-discuss] GPFS / Spectrum Scale support with x86 SUSE SLES 12 SP2 to be expected when? In-Reply-To: <57030974-8C80-4808-87F1-3540E1271D6E@gmail.com> References: <57030974-8C80-4808-87F1-3540E1271D6E@gmail.com> Message-ID: Just to make sure I wasn't mis-remembering I just installed 4.1.1.10 on a SLES12 SP1 machine (it has a SLES12 SP2 kernel though), built and loaded the kernel modules. There could be something significantly different in user space between SLES12 SP1 and SP2 that prevents RPM installation but I'd be surprised. -Aaron On 3/29/17 9:18 AM, Aaron Knister wrote: > I could be wrong but I believe it is currently supported (as in it > works, at least that's my recollection). Did you try with a particular > version that wouldn't work? > > Sent from my iPhone > > On Mar 29, 2017, at 07:34, Christoph Krafft > wrote: > >> Folks, >> >> does anyone have any tentative information when the above SUSE OS >> Level will be supported? >> >> Thank you in advance for hints and tips! >> >> >> Mit freundlichen Gr??en / Sincerely >> >> *Christoph Krafft* >> >> Client Technical Specialist - Power Systems, IBM Systems >> Certified IT Specialist @ The Open Group >> >> ------------------------------------------------------------------------ >> >> Phone: +49 (0) 7034 643 2171 IBM Deutschland GmbH >> <16373446.gif> >> Mobile: +49 (0) 160 97 81 86 12 Am Weiher 24 >> Email: ckrafft at de.ibm.com 65451 Kelsterbach >> Germany >> ------------------------------------------------------------------------ >> IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter >> Gesch?ftsf?hrung: Martina Koederitz (Vorsitzende), Nicole Reimer, >> Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Stefan Lutz >> Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht >> Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From S.J.Thompson at bham.ac.uk Wed Mar 29 15:01:33 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Wed, 29 Mar 2017 14:01:33 +0000 Subject: [gpfsug-discuss] GPFS / Spectrum Scale support with x86 SUSE SLES 12 SP2 to be expected when? In-Reply-To: References: <57030974-8C80-4808-87F1-3540E1271D6E@gmail.com>, Message-ID: I think the question was supported, not works though. We've just been round something similar with TSM support, works fine on el7.3, but not supported, the docs actually said 7.1 or higher. We still got support and IBM now officially support 7.3, but I can see people may not want to deploy on something unsupported. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Aaron Knister [aaron.s.knister at nasa.gov] Sent: 29 March 2017 14:55 To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] GPFS / Spectrum Scale support with x86 SUSE SLES 12 SP2 to be expected when? Just to make sure I wasn't mis-remembering I just installed 4.1.1.10 on a SLES12 SP1 machine (it has a SLES12 SP2 kernel though), built and loaded the kernel modules. There could be something significantly different in user space between SLES12 SP1 and SP2 that prevents RPM installation but I'd be surprised. -Aaron On 3/29/17 9:18 AM, Aaron Knister wrote: > I could be wrong but I believe it is currently supported (as in it > works, at least that's my recollection). Did you try with a particular > version that wouldn't work? > > Sent from my iPhone > > On Mar 29, 2017, at 07:34, Christoph Krafft > wrote: > >> Folks, >> >> does anyone have any tentative information when the above SUSE OS >> Level will be supported? >> >> Thank you in advance for hints and tips! >> >> >> Mit freundlichen Gr??en / Sincerely >> >> *Christoph Krafft* >> >> Client Technical Specialist - Power Systems, IBM Systems >> Certified IT Specialist @ The Open Group >> >> ------------------------------------------------------------------------ >> >> Phone: +49 (0) 7034 643 2171 IBM Deutschland GmbH >> <16373446.gif> >> Mobile: +49 (0) 160 97 81 86 12 Am Weiher 24 >> Email: ckrafft at de.ibm.com 65451 Kelsterbach >> Germany >> ------------------------------------------------------------------------ >> IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter >> Gesch?ftsf?hrung: Martina Koederitz (Vorsitzende), Nicole Reimer, >> Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Stefan Lutz >> Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht >> Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From aaron.s.knister at nasa.gov Wed Mar 29 15:17:19 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Wed, 29 Mar 2017 10:17:19 -0400 Subject: [gpfsug-discuss] GPFS / Spectrum Scale support with x86 SUSE SLES 12 SP2 to be expected when? In-Reply-To: References: <57030974-8C80-4808-87F1-3540E1271D6E@gmail.com> Message-ID: Ah, good point. Support means different things to different people, I suppose. The FAQ does say this which I'd not seen before: "IBM Spectrum Scale is not supported on SLES 12 SP2" That's actually a problem for me so I second Christoph's question, then. -Aaron On 3/29/17 10:01 AM, Simon Thompson (Research Computing - IT Services) wrote: > I think the question was supported, not works though. > > We've just been round something similar with TSM support, works fine on el7.3, but not supported, the docs actually said 7.1 or higher. We still got support and IBM now officially support 7.3, but I can see people may not want to deploy on something unsupported. > > Simon > ________________________________________ > From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Aaron Knister [aaron.s.knister at nasa.gov] > Sent: 29 March 2017 14:55 > To: gpfsug-discuss at spectrumscale.org > Subject: Re: [gpfsug-discuss] GPFS / Spectrum Scale support with x86 SUSE SLES 12 SP2 to be expected when? > > Just to make sure I wasn't mis-remembering I just installed 4.1.1.10 on > a SLES12 SP1 machine (it has a SLES12 SP2 kernel though), built and > loaded the kernel modules. There could be something significantly > different in user space between SLES12 SP1 and SP2 that prevents RPM > installation but I'd be surprised. > > -Aaron > > On 3/29/17 9:18 AM, Aaron Knister wrote: >> I could be wrong but I believe it is currently supported (as in it >> works, at least that's my recollection). Did you try with a particular >> version that wouldn't work? >> >> Sent from my iPhone >> >> On Mar 29, 2017, at 07:34, Christoph Krafft > > wrote: >> >>> Folks, >>> >>> does anyone have any tentative information when the above SUSE OS >>> Level will be supported? >>> >>> Thank you in advance for hints and tips! >>> >>> >>> Mit freundlichen Gr??en / Sincerely >>> >>> *Christoph Krafft* >>> >>> Client Technical Specialist - Power Systems, IBM Systems >>> Certified IT Specialist @ The Open Group >>> >>> ------------------------------------------------------------------------ >>> >>> Phone: +49 (0) 7034 643 2171 IBM Deutschland GmbH >>> <16373446.gif> >>> Mobile: +49 (0) 160 97 81 86 12 Am Weiher 24 >>> Email: ckrafft at de.ibm.com 65451 Kelsterbach >>> Germany >>> ------------------------------------------------------------------------ >>> IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter >>> Gesch?ftsf?hrung: Martina Koederitz (Vorsitzende), Nicole Reimer, >>> Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Stefan Lutz >>> Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht >>> Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 >>> >>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From abeattie at au1.ibm.com Wed Mar 29 22:53:06 2017 From: abeattie at au1.ibm.com (Andrew Beattie) Date: Wed, 29 Mar 2017 21:53:06 +0000 Subject: [gpfsug-discuss] GPFS / Spectrum Scale support with x86 SUSE SLES 12 SP2 to be expected when? In-Reply-To: References: , <57030974-8C80-4808-87F1-3540E1271D6E@gmail.com> Message-ID: An HTML attachment was scrubbed... URL: From Greg.Lehmann at csiro.au Thu Mar 30 00:12:35 2017 From: Greg.Lehmann at csiro.au (Greg.Lehmann at csiro.au) Date: Wed, 29 Mar 2017 23:12:35 +0000 Subject: [gpfsug-discuss] question on viewing block distribution across NSDs Message-ID: <2c0ba2b9466b42b4b3e5b3202ab26825@exch1-cdc.nexus.csiro.au> Hi All, I added some NSDs to an existing filesystem and ran mmrestripefs. I was sort of curious to see what the distribution looked like before and after the restripe. Is there any way of looking at it? Cheers, Greg. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Thu Mar 30 00:45:23 2017 From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]) Date: Wed, 29 Mar 2017 23:45:23 +0000 Subject: [gpfsug-discuss] question on viewing block distribution across NSDs References: question on viewing block distribution across NSDs Message-ID: <5F910253243E6A47B81A9A2EB424BBA101F5998D@NDMSMBX404.ndc.nasa.gov> Hi Greg, You could run an mmdf which will show you how full each NSD is. I'm not sure how to look back in time though to see the fs before the restripe. Do you perhaps have a gpfs.snap you took somewhat recently before the restripe? Maybe an internaldump in /tmp/mmfs somewhere? From: greg.lehmann at csiro.au Sent: 3/29/17, 19:21 To: gpfsug main discussion list Subject: [gpfsug-discuss] question on viewing block distribution across NSDs Hi All, I added some NSDs to an existing filesystem and ran mmrestripefs. I was sort of curious to see what the distribution looked like before and after the restripe. Is there any way of looking at it? Cheers, Greg. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Greg.Lehmann at csiro.au Thu Mar 30 00:51:37 2017 From: Greg.Lehmann at csiro.au (Greg.Lehmann at csiro.au) Date: Wed, 29 Mar 2017 23:51:37 +0000 Subject: [gpfsug-discuss] question on viewing block distribution across NSDs In-Reply-To: <5F910253243E6A47B81A9A2EB424BBA101F5998D@NDMSMBX404.ndc.nasa.gov> References: question on viewing block distribution across NSDs <5F910253243E6A47B81A9A2EB424BBA101F5998D@NDMSMBX404.ndc.nasa.gov> Message-ID: <3ec38bd65cad452f974c1d6c0d762853@exch1-cdc.nexus.csiro.au> Thanks. I don't have a snap. I'll keep that in mind for next time I do this. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] Sent: Thursday, 30 March 2017 9:45 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] question on viewing block distribution across NSDs Hi Greg, You could run an mmdf which will show you how full each NSD is. I'm not sure how to look back in time though to see the fs before the restripe. Do you perhaps have a gpfs.snap you took somewhat recently before the restripe? Maybe an internaldump in /tmp/mmfs somewhere? From: greg.lehmann at csiro.au Sent: 3/29/17, 19:21 To: gpfsug main discussion list Subject: [gpfsug-discuss] question on viewing block distribution across NSDs Hi All, I added some NSDs to an existing filesystem and ran mmrestripefs. I was sort of curious to see what the distribution looked like before and after the restripe. Is there any way of looking at it? Cheers, Greg. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Thu Mar 30 00:55:02 2017 From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]) Date: Wed, 29 Mar 2017 23:55:02 +0000 Subject: [gpfsug-discuss] question on viewing block distribution across NSDs References: [gpfsug-discuss] question on viewing block distribution across NSDs Message-ID: <5F910253243E6A47B81A9A2EB424BBA101F59A0A@NDMSMBX404.ndc.nasa.gov> I don't necessarily think you need to run a snap prior, just the output of mmdf should be enough. Something to keep in mind that I should have said before-- an mmdf can be stressful on your system particularly if you have spinning disk for your metadata. We're fortunate enough to have all flash for our metadata and I tend to take it for granted some times :) From: greg.lehmann at csiro.au Sent: 3/29/17, 19:52 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] question on viewing block distribution across NSDs Thanks. I don?t have a snap. I?ll keep that in mind for next time I do this. From: mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] Sent: Thursday, 30 March 2017 9:45 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] question on viewing block distribution across NSDs Hi Greg, You could run an mmdf which will show you how full each NSD is. I'm not sure how to look back in time though to see the fs before the restripe. Do you perhaps have a gpfs.snap you took somewhat recently before the restripe? Maybe an internaldump in /tmp/mmfs somewhere? From: greg.lehmann at csiro.au Sent: 3/29/17, 19:21 To: gpfsug main discussion list Subject: [gpfsug-discuss] question on viewing block distribution across NSDs Hi All, I added some NSDs to an existing filesystem and ran mmrestripefs. I was sort of curious to see what the distribution looked like before and after the restripe. Is there any way of looking at it? Cheers, Greg. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Greg.Lehmann at csiro.au Thu Mar 30 00:59:41 2017 From: Greg.Lehmann at csiro.au (Greg.Lehmann at csiro.au) Date: Wed, 29 Mar 2017 23:59:41 +0000 Subject: [gpfsug-discuss] question on viewing block distribution across NSDs In-Reply-To: <5F910253243E6A47B81A9A2EB424BBA101F59A0A@NDMSMBX404.ndc.nasa.gov> References: [gpfsug-discuss] question on viewing block distribution across NSDs <5F910253243E6A47B81A9A2EB424BBA101F59A0A@NDMSMBX404.ndc.nasa.gov> Message-ID: I was going to keep mmdf in mind, not gpfs.snap. I will now also keep in mind that mmdf can have an impact as at present we have spinning disk for metadata. The system I am playing around on is not production yet, so I am safe for the moment. Thanks again. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] Sent: Thursday, 30 March 2017 9:55 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] question on viewing block distribution across NSDs I don't necessarily think you need to run a snap prior, just the output of mmdf should be enough. Something to keep in mind that I should have said before-- an mmdf can be stressful on your system particularly if you have spinning disk for your metadata. We're fortunate enough to have all flash for our metadata and I tend to take it for granted some times :) From: greg.lehmann at csiro.au Sent: 3/29/17, 19:52 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] question on viewing block distribution across NSDs Thanks. I don?t have a snap. I?ll keep that in mind for next time I do this. From: mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] Sent: Thursday, 30 March 2017 9:45 AM To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] question on viewing block distribution across NSDs Hi Greg, You could run an mmdf which will show you how full each NSD is. I'm not sure how to look back in time though to see the fs before the restripe. Do you perhaps have a gpfs.snap you took somewhat recently before the restripe? Maybe an internaldump in /tmp/mmfs somewhere? From: greg.lehmann at csiro.au Sent: 3/29/17, 19:21 To: gpfsug main discussion list Subject: [gpfsug-discuss] question on viewing block distribution across NSDs Hi All, I added some NSDs to an existing filesystem and ran mmrestripefs. I was sort of curious to see what the distribution looked like before and after the restripe. Is there any way of looking at it? Cheers, Greg. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kums at us.ibm.com Thu Mar 30 14:04:09 2017 From: kums at us.ibm.com (Kumaran Rajaram) Date: Thu, 30 Mar 2017 08:04:09 -0500 Subject: [gpfsug-discuss] question on viewing block distribution across NSDs In-Reply-To: References: [gpfsug-discuss] question on viewing block distribution acrossNSDs <5F910253243E6A47B81A9A2EB424BBA101F59A0A@NDMSMBX404.ndc.nasa.gov> Message-ID: Hi, Yes, you could use "mmdf" to obtain file-system "usage" across the NSDs (comprising the file-system). If you want to obtain "data block distribution corresponding to a file across the NSDs", then there is a utility "mmgetlocation" in /usr/lpp/mmfs/samples/fpo that can be used to get file-data-blocks to NSD mapping. Example: # File-system comprises of single storage pool, all NSDs configured as dataAndMetadata, -m 1 -r 1, FS block-size=2MiB # mmlsfs gpfs1b | grep 'Block size' -B 2097152 Block size # The file-system is comprised of 10 x dataAndMetadata NSDs # mmlsdisk gpfs1b | grep DMD | wc -l 10 # Create a sample file that is 40MiB (20 data blocks) /mnt/sw/benchmarks/gpfsperf/gpfsperf create seq -r 2m -n 40m /mnt/gpfs1b/temp_dir/lf.s.1 # File size is 40 MiB (09:52:49) c25m3n07:~ # ls -lh /mnt/gpfs1b/temp_dir/lf.s.1 -rw-r--r-- 1 root root 40M Mar 17 09:52 /mnt/gpfs1b/temp_dir/lf.s.1 (09:52:54) c25m3n07:~ # du -sh /mnt/gpfs1b/temp_dir/lf.s.1 40M /mnt/gpfs1b/temp_dir/lf.s.1 # Verified through mmgetlocation that the file data blocks is uniformly striped across all the dataAndMetadata NSDs, with each NSD containing 2 file data blocks # In the output below, "DMD_NSDX" is name of the NSDs. (09:53:00) c25m3n07:~ # /usr/lpp/mmfs/samples/fpo/mmgetlocation -f /mnt/gpfs1b/temp_dir/lf.s.1 [FILE INFO] ------------------------------------------------------------------------ blockSize 2 MB blockGroupFactor 1 metadataBlockSize 2M writeAffinityDepth 0 flags: data replication: 1 max 2 storage pool name: system metadata replication: 1 max 2 Chunk 0 (offset 0) is located at disks: [ DMD_NSD09 c25m3n07-ib,c25m3n08-ib ] Chunk 1 (offset 2097152) is located at disks: [ DMD_NSD10 c25m3n08-ib,c25m3n07-ib ] Chunk 2 (offset 4194304) is located at disks: [ DMD_NSD01 c25m3n07-ib,c25m3n08-ib ] Chunk 3 (offset 6291456) is located at disks: [ DMD_NSD02 c25m3n08-ib,c25m3n07-ib ] Chunk 4 (offset 8388608) is located at disks: [ DMD_NSD03 c25m3n07-ib,c25m3n08-ib ] Chunk 5 (offset 10485760) is located at disks: [ DMD_NSD04 c25m3n08-ib,c25m3n07-ib ] Chunk 6 (offset 12582912) is located at disks: [ DMD_NSD05 c25m3n07-ib,c25m3n08-ib ] Chunk 7 (offset 14680064) is located at disks: [ DMD_NSD06 c25m3n08-ib,c25m3n07-ib ] Chunk 8 (offset 16777216) is located at disks: [ DMD_NSD07 c25m3n07-ib,c25m3n08-ib ] Chunk 9 (offset 18874368) is located at disks: [ DMD_NSD08 c25m3n08-ib,c25m3n07-ib ] Chunk 10 (offset 20971520) is located at disks: [ DMD_NSD09 c25m3n07-ib,c25m3n08-ib ] Chunk 11 (offset 23068672) is located at disks: [ DMD_NSD10 c25m3n08-ib,c25m3n07-ib ] Chunk 12 (offset 25165824) is located at disks: [ DMD_NSD01 c25m3n07-ib,c25m3n08-ib ] Chunk 13 (offset 27262976) is located at disks: [ DMD_NSD02 c25m3n08-ib,c25m3n07-ib ] Chunk 14 (offset 29360128) is located at disks: [ DMD_NSD03 c25m3n07-ib,c25m3n08-ib ] Chunk 15 (offset 31457280) is located at disks: [ DMD_NSD04 c25m3n08-ib,c25m3n07-ib ] Chunk 16 (offset 33554432) is located at disks: [ DMD_NSD05 c25m3n07-ib,c25m3n08-ib ] Chunk 17 (offset 35651584) is located at disks: [ DMD_NSD06 c25m3n08-ib,c25m3n07-ib ] Chunk 18 (offset 37748736) is located at disks: [ DMD_NSD07 c25m3n07-ib,c25m3n08-ib ] Chunk 19 (offset 39845888) is located at disks: [ DMD_NSD08 c25m3n08-ib,c25m3n07-ib ] [SUMMARY INFO] ---------------------------------------------------------------------------------------------------------- Replica num Nodename TotalChunkst Replica 1 : c25m3n07-ib,c25m3n08-ib: Total : 10 Replica 1 : c25m3n08-ib,c25m3n07-ib: Total : 10 Best Regards, -Kums From: To: Date: 03/29/2017 08:00 PM Subject: Re: [gpfsug-discuss] question on viewing block distribution across NSDs Sent by: gpfsug-discuss-bounces at spectrumscale.org I was going to keep mmdf in mind, not gpfs.snap. I will now also keep in mind that mmdf can have an impact as at present we have spinning disk for metadata. The system I am playing around on is not production yet, so I am safe for the moment. Thanks again. From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] Sent: Thursday, 30 March 2017 9:55 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] question on viewing block distribution across NSDs I don't necessarily think you need to run a snap prior, just the output of mmdf should be enough. Something to keep in mind that I should have said before-- an mmdf can be stressful on your system particularly if you have spinning disk for your metadata. We're fortunate enough to have all flash for our metadata and I tend to take it for granted some times :) From: greg.lehmann at csiro.au Sent: 3/29/17, 19:52 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] question on viewing block distribution across NSDs Thanks. I don?t have a snap. I?ll keep that in mind for next time I do this. From: mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] Sent: Thursday, 30 March 2017 9:45 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] question on viewing block distribution across NSDs Hi Greg, You could run an mmdf which will show you how full each NSD is. I'm not sure how to look back in time though to see the fs before the restripe. Do you perhaps have a gpfs.snap you took somewhat recently before the restripe? Maybe an internaldump in /tmp/mmfs somewhere? From: greg.lehmann at csiro.au Sent: 3/29/17, 19:21 To: gpfsug main discussion list Subject: [gpfsug-discuss] question on viewing block distribution across NSDs Hi All, I added some NSDs to an existing filesystem and ran mmrestripefs. I was sort of curious to see what the distribution looked like before and after the restripe. Is there any way of looking at it? Cheers, Greg._______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From willi.engeli at id.ethz.ch Thu Mar 30 14:29:26 2017 From: willi.engeli at id.ethz.ch (Engeli Willi (ID SD)) Date: Thu, 30 Mar 2017 13:29:26 +0000 Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Message-ID: Hi everybody, In our organization, the management of AD is strictly separated from management of storage. Since we install spectrum scale with protocol SMB and NFS support, we need to join the systems to AD, and have at least the joining user added as well to the local administrators group. Any idea of how to achieve this? Asking our Domain Admin is not the correct method to add other groups, this needs to be in our hands. Regards Willi -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5461 bytes Desc: not available URL: From r.sobey at imperial.ac.uk Thu Mar 30 14:53:15 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Thu, 30 Mar 2017 13:53:15 +0000 Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group In-Reply-To: References: Message-ID: Last time I checked simply adding a normal computer object to the domain didn't add the account of the adding user to the local administrators group and CES is no exception. Is it a political reason why you cannot ask your Domain Admin team to add you to the admin group for your CES cluster object? From there you can manage it yourself. Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Engeli Willi (ID SD) Sent: 30 March 2017 14:29 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Hi everybody, In our organization, the management of AD is strictly separated from management of storage. Since we install spectrum scale with protocol SMB and NFS support, we need to join the systems to AD, and have at least the joining user added as well to the local administrators group. Any idea of how to achieve this? Asking our Domain Admin is not the correct method to add other groups, this needs to be in our hands. Regards Willi -------------- next part -------------- An HTML attachment was scrubbed... URL: From laurence at qsplace.co.uk Thu Mar 30 15:02:19 2017 From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow) Date: Thu, 30 Mar 2017 15:02:19 +0100 Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group In-Reply-To: References: Message-ID: <2329870e-00f8-258c-187d-feec9589df93@qsplace.co.uk> Hi Willi, Could you just expand on your issue? Are you requiring CES to bind to AD to allow authenticated users to access your NFS/SMB shares. However you require the ability to add additional groups to these users on the CES system? Or are you trying to use your own account that can join the domain as a local admin on a CES node? -- Lauz On 30/03/2017 14:53, Sobey, Richard A wrote: > > Last time I checked simply adding a normal computer object to the > domain didn?t add the account of the adding user to the local > administrators group and CES is no exception. > > Is it a political reason why you cannot ask your Domain Admin team to > add you to the admin group for your CES cluster object? From there you > can manage it yourself. > > Richard > > *From:*gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of > *Engeli Willi (ID SD) > *Sent:* 30 March 2017 14:29 > *To:* gpfsug-discuss at spectrumscale.org > *Subject:* [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin > to local Administrators group > > Hi everybody, > > In our organization, the management of AD is strictly separated from > management of storage. Since we install spectrum scale with protocol > SMB and NFS support, we need to join the systems to AD, and have at > least the joining user added as well to the local administrators group. > > Any idea of how to achieve this? Asking our Domain Admin is not the > correct method to add other groups, this needs to be in our hands. > > Regards Willi > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From jgwood at sandia.gov Thu Mar 30 15:00:50 2017 From: jgwood at sandia.gov (Wood, Justin Gregory) Date: Thu, 30 Mar 2017 14:00:50 +0000 Subject: [gpfsug-discuss] [EXTERNAL] Spectrum Scale CES adds only Domain Admin to local Administrators group In-Reply-To: References: Message-ID: <069E1DC2-8711-4F4E-BD65-250C524ED341@sandia.gov> I recently went through the process of installing some protocol nodes and our organization also has a separate AD/Windows group. All I had to do was to get the AD admins to add a ?stub? computer object, then they made me an administrator of that object. That allowed me to run mmuserauth to do the setup. It was fairly simple and allowed me to do lots of work/testing without interaction with the AD group. -Justin. From: on behalf of "Engeli Willi (ID SD)" Reply-To: gpfsug main discussion list Date: Thursday, March 30, 2017 at 7:29 AM To: "gpfsug-discuss at spectrumscale.org" Subject: [EXTERNAL] [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Hi everybody, In our organization, the management of AD is strictly separated from management of storage. Since we install spectrum scale with protocol SMB and NFS support, we need to join the systems to AD, and have at least the joining user added as well to the local administrators group. Any idea of how to achieve this? Asking our Domain Admin is not the correct method to add other groups, this needs to be in our hands. Regards Willi -------------- next part -------------- An HTML attachment was scrubbed... URL: From willi.engeli at id.ethz.ch Thu Mar 30 15:23:40 2017 From: willi.engeli at id.ethz.ch (Engeli Willi (ID SD)) Date: Thu, 30 Mar 2017 14:23:40 +0000 Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Message-ID: >-Last time I checked simply adding a normal computer object to the domain didn't add the account of the adding user to the local administrators group and CES is no exception. We have been using before a competitor Product as a NAS system. With that system, we were able to define virtual NAS Servers, each one joined as an independent object to AD. When joined, we found the 'Domain Admin' group and the joining user as member of local administrators group of that virtual server. Since out AD is quite big, it is structured into many OU. We as the Storage OU have OU admin rights, but we are not member of "Domain Admin" group. Looking Back, we were able by ourselves to add the required groups as needed to the local Administrators group of the NAS server. Why is this important? Since we have quit a mix of OS accessing our shares, some of the create exclusive access rights at the time they create profiles etc. At the end of the lifecycle, one needs to delete those files via the SMB / NFSV4 protocol, which is difficult if not having access rights. On the other hand, we have seen situations, where one OS corrupted the ACL and could not access anymore. Also this needs to be handled by us, giving us a hard time not being member of the administrators group. I.e. the MS tool subinacl does check the privileges before trying to modify ACLs, and if not being member of the Administrators group, not all required privileges are granted. >-Is it a political reason why you cannot ask your Domain Admin team to add you to the admin group for your CES cluster object? From there you can manage it yourself. Yes and no. We have a clear boundary, where we need to be able to manage the AD Objects, and for security reason it seems to make sense to not use Domain Admin Accounts for such kind of work (statement of our AD Group). So much for the Situation, did I missed something? Willi -----Urspr?ngliche Nachricht----- Von: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] Im Auftrag von gpfsug-discuss-request at spectrumscale.org Gesendet: Donnerstag, 30. M?rz 2017 16:02 An: gpfsug-discuss at spectrumscale.org Betreff: gpfsug-discuss Digest, Vol 62, Issue 77 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Spectrum Scale CES adds only Domain Admin to local Administrators group (Engeli Willi (ID SD)) 2. Re: Spectrum Scale CES adds only Domain Admin to local Administrators group (Sobey, Richard A) 3. Re: Spectrum Scale CES adds only Domain Admin to local Administrators group (Laurence Horrocks-Barlow) ---------------------------------------------------------------------- Message: 1 Date: Thu, 30 Mar 2017 13:29:26 +0000 From: "Engeli Willi (ID SD)" To: "gpfsug-discuss at spectrumscale.org" Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Message-ID: Content-Type: text/plain; charset="us-ascii" Hi everybody, In our organization, the management of AD is strictly separated from management of storage. Since we install spectrum scale with protocol SMB and NFS support, we need to join the systems to AD, and have at least the joining user added as well to the local administrators group. Any idea of how to achieve this? Asking our Domain Admin is not the correct method to add other groups, this needs to be in our hands. Regards Willi -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5461 bytes Desc: not available URL: ------------------------------ Message: 2 Date: Thu, 30 Mar 2017 13:53:15 +0000 From: "Sobey, Richard A" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Message-ID: Content-Type: text/plain; charset="us-ascii" Last time I checked simply adding a normal computer object to the domain didn't add the account of the adding user to the local administrators group and CES is no exception. Is it a political reason why you cannot ask your Domain Admin team to add you to the admin group for your CES cluster object? From there you can manage it yourself. Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Engeli Willi (ID SD) Sent: 30 March 2017 14:29 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Hi everybody, In our organization, the management of AD is strictly separated from management of storage. Since we install spectrum scale with protocol SMB and NFS support, we need to join the systems to AD, and have at least the joining user added as well to the local administrators group. Any idea of how to achieve this? Asking our Domain Admin is not the correct method to add other groups, this needs to be in our hands. Regards Willi -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 3 Date: Thu, 30 Mar 2017 15:02:19 +0100 From: Laurence Horrocks-Barlow To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Message-ID: <2329870e-00f8-258c-187d-feec9589df93 at qsplace.co.uk> Content-Type: text/plain; charset="windows-1252"; Format="flowed" Hi Willi, Could you just expand on your issue? Are you requiring CES to bind to AD to allow authenticated users to access your NFS/SMB shares. However you require the ability to add additional groups to these users on the CES system? Or are you trying to use your own account that can join the domain as a local admin on a CES node? -- Lauz On 30/03/2017 14:53, Sobey, Richard A wrote: > > Last time I checked simply adding a normal computer object to the > domain didn?t add the account of the adding user to the local > administrators group and CES is no exception. > > Is it a political reason why you cannot ask your Domain Admin team to > add you to the admin group for your CES cluster object? From there you > can manage it yourself. > > Richard > > *From:*gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of > *Engeli Willi (ID SD) > *Sent:* 30 March 2017 14:29 > *To:* gpfsug-discuss at spectrumscale.org > *Subject:* [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin > to local Administrators group > > Hi everybody, > > In our organization, the management of AD is strictly separated from > management of storage. Since we install spectrum scale with protocol > SMB and NFS support, we need to join the systems to AD, and have at > least the joining user added as well to the local administrators group. > > Any idea of how to achieve this? Asking our Domain Admin is not the > correct method to add other groups, this needs to be in our hands. > > Regards Willi > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 62, Issue 77 ********************************************** -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5461 bytes Desc: not available URL: From gmcpheeters at anl.gov Thu Mar 30 15:39:45 2017 From: gmcpheeters at anl.gov (McPheeters, Gordon) Date: Thu, 30 Mar 2017 14:39:45 +0000 Subject: [gpfsug-discuss] question on viewing block distribution across NSDs In-Reply-To: <2c0ba2b9466b42b4b3e5b3202ab26825@exch1-cdc.nexus.csiro.au> References: <2c0ba2b9466b42b4b3e5b3202ab26825@exch1-cdc.nexus.csiro.au> Message-ID: <2A55B023-7DE1-417E-BD30-88FD44DF0ADA@anl.gov> mmdf will show you the distribution across NSDs. Gordon On Mar 29, 2017, at 6:12 PM, Greg.Lehmann at csiro.au wrote: Hi All, I added some NSDs to an existing filesystem and ran mmrestripefs. I was sort of curious to see what the distribution looked like before and after the restripe. Is there any way of looking at it? Cheers, Greg. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Thu Mar 30 15:50:13 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Thu, 30 Mar 2017 14:50:13 +0000 Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group In-Reply-To: References: Message-ID: Did your AD team perchance define a group policy on the OU such that any object placed into that OU inherited a specific set of local administrators? That's the only way I can think that your NAS ended up with the calling user in the local admin group. I understand where you're coming from - we do not manage AD ourselves but we do not want Domain Admins to have administrator control of our CES nodes. So once it was joined to AD (with their help) I simply removed Domain Admins and added the storage team DL in its place. But back to the original question, I'm afraid I do not know how to make CES add a specific user to its local administrator group. Richard -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Engeli Willi (ID SD) Sent: 30 March 2017 15:24 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group >-Last time I checked simply adding a normal computer object to the >domain didn't add the account of the adding user to the local administrators group and CES is no exception. We have been using before a competitor Product as a NAS system. With that system, we were able to define virtual NAS Servers, each one joined as an independent object to AD. When joined, we found the 'Domain Admin' group and the joining user as member of local administrators group of that virtual server. Since out AD is quite big, it is structured into many OU. We as the Storage OU have OU admin rights, but we are not member of "Domain Admin" group. Looking Back, we were able by ourselves to add the required groups as needed to the local Administrators group of the NAS server. Why is this important? Since we have quit a mix of OS accessing our shares, some of the create exclusive access rights at the time they create profiles etc. At the end of the lifecycle, one needs to delete those files via the SMB / NFSV4 protocol, which is difficult if not having access rights. On the other hand, we have seen situations, where one OS corrupted the ACL and could not access anymore. Also this needs to be handled by us, giving us a hard time not being member of the administrators group. I.e. the MS tool subinacl does check the privileges before trying to modify ACLs, and if not being member of the Administrators group, not all required privileges are granted. >-Is it a political reason why you cannot ask your Domain Admin team to >add you to the admin group for your CES cluster object? From there you can manage it yourself. Yes and no. We have a clear boundary, where we need to be able to manage the AD Objects, and for security reason it seems to make sense to not use Domain Admin Accounts for such kind of work (statement of our AD Group). So much for the Situation, did I missed something? Willi -----Urspr?ngliche Nachricht----- Von: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] Im Auftrag von gpfsug-discuss-request at spectrumscale.org Gesendet: Donnerstag, 30. M?rz 2017 16:02 An: gpfsug-discuss at spectrumscale.org Betreff: gpfsug-discuss Digest, Vol 62, Issue 77 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Spectrum Scale CES adds only Domain Admin to local Administrators group (Engeli Willi (ID SD)) 2. Re: Spectrum Scale CES adds only Domain Admin to local Administrators group (Sobey, Richard A) 3. Re: Spectrum Scale CES adds only Domain Admin to local Administrators group (Laurence Horrocks-Barlow) ---------------------------------------------------------------------- Message: 1 Date: Thu, 30 Mar 2017 13:29:26 +0000 From: "Engeli Willi (ID SD)" To: "gpfsug-discuss at spectrumscale.org" Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Message-ID: Content-Type: text/plain; charset="us-ascii" Hi everybody, In our organization, the management of AD is strictly separated from management of storage. Since we install spectrum scale with protocol SMB and NFS support, we need to join the systems to AD, and have at least the joining user added as well to the local administrators group. Any idea of how to achieve this? Asking our Domain Admin is not the correct method to add other groups, this needs to be in our hands. Regards Willi -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5461 bytes Desc: not available URL: ------------------------------ Message: 2 Date: Thu, 30 Mar 2017 13:53:15 +0000 From: "Sobey, Richard A" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Message-ID: Content-Type: text/plain; charset="us-ascii" Last time I checked simply adding a normal computer object to the domain didn't add the account of the adding user to the local administrators group and CES is no exception. Is it a political reason why you cannot ask your Domain Admin team to add you to the admin group for your CES cluster object? From there you can manage it yourself. Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Engeli Willi (ID SD) Sent: 30 March 2017 14:29 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Hi everybody, In our organization, the management of AD is strictly separated from management of storage. Since we install spectrum scale with protocol SMB and NFS support, we need to join the systems to AD, and have at least the joining user added as well to the local administrators group. Any idea of how to achieve this? Asking our Domain Admin is not the correct method to add other groups, this needs to be in our hands. Regards Willi -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 3 Date: Thu, 30 Mar 2017 15:02:19 +0100 From: Laurence Horrocks-Barlow To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Message-ID: <2329870e-00f8-258c-187d-feec9589df93 at qsplace.co.uk> Content-Type: text/plain; charset="windows-1252"; Format="flowed" Hi Willi, Could you just expand on your issue? Are you requiring CES to bind to AD to allow authenticated users to access your NFS/SMB shares. However you require the ability to add additional groups to these users on the CES system? Or are you trying to use your own account that can join the domain as a local admin on a CES node? -- Lauz On 30/03/2017 14:53, Sobey, Richard A wrote: > > Last time I checked simply adding a normal computer object to the > domain didn?t add the account of the adding user to the local > administrators group and CES is no exception. > > Is it a political reason why you cannot ask your Domain Admin team to > add you to the admin group for your CES cluster object? From there you > can manage it yourself. > > Richard > > *From:*gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of > *Engeli Willi (ID SD) > *Sent:* 30 March 2017 14:29 > *To:* gpfsug-discuss at spectrumscale.org > *Subject:* [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin > to local Administrators group > > Hi everybody, > > In our organization, the management of AD is strictly separated from > management of storage. Since we install spectrum scale with protocol > SMB and NFS support, we need to join the systems to AD, and have at > least the joining user added as well to the local administrators group. > > Any idea of how to achieve this? Asking our Domain Admin is not the > correct method to add other groups, this needs to be in our hands. > > Regards Willi > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 62, Issue 77 ********************************************** From Mark.Bush at siriuscom.com Thu Mar 30 17:09:15 2017 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Thu, 30 Mar 2017 16:09:15 +0000 Subject: [gpfsug-discuss] AFM vs mmremotefs Message-ID: <5A22811B-7FBD-4091-B6FD-E5E74053DDF2@siriuscom.com> When does it make sense to use mmremotefs vs AFM? With mmremotefs the actual data lives on the remote cluster right so if you need data closer then AFM make more sense to use? If the clusters get disconnected with mmremotefs I have not access to the data with AFM I do. Do I have these concepts right in my head? [id:image001.png at 01D2709D.6EF65720] Mark R. Bush| Storage Architect Mobile: 210-237-8415 Twitter: @bushmr | LinkedIn: /markreedbush 10100 Reunion Place, Suite 500, San Antonio, TX 78216 www.siriuscom.com |mark.bush at siriuscom.com This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 8745 bytes Desc: image001.png URL: From S.J.Thompson at bham.ac.uk Thu Mar 30 17:19:07 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Thu, 30 Mar 2017 16:19:07 +0000 Subject: [gpfsug-discuss] AFM vs mmremotefs Message-ID: We are just planning to deploy AFM as a cache tier for our HPC cluster and bulk data storage. So whilst they are locally connected etc, I can use a different type of storage for performance for HPC work. Bulk data has two copies over two data centres, so the AFM cache would ACK the write to the HPC client and then "push it as soon as possible" to the home (bulk data storage and do the two copies). Similarly for ingest systems, for example streaming data off instruments, you might want to do really really fast (to a flash system) and then have the AFM write to home at some point. (of course your cache capacity needs to be sufficiently large that you are able to push writes/expel). On the flip side, I don't think AFM has remote identity mapping (someone might correct me on that of course) So I don't think its just about data location/connectivity. Simon From: > on behalf of "Mark.Bush at siriuscom.com" > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Thursday, 30 March 2017 at 17:09 To: "gpfsug-discuss at spectrumscale.org" > Subject: [gpfsug-discuss] AFM vs mmremotefs When does it make sense to use mmremotefs vs AFM? With mmremotefs the actual data lives on the remote cluster right so if you need data closer then AFM make more sense to use? If the clusters get disconnected with mmremotefs I have not access to the data with AFM I do. Do I have these concepts right in my head? [id:image001.png at 01D2709D.6EF65720] Mark R. Bush| Storage Architect Mobile: 210-237-8415 Twitter: @bushmr | LinkedIn: /markreedbush 10100 Reunion Place, Suite 500, San Antonio, TX 78216 www.siriuscom.com |mark.bush at siriuscom.com This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 8745 bytes Desc: image001.png URL: From makaplan at us.ibm.com Thu Mar 30 17:26:34 2017 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Thu, 30 Mar 2017 11:26:34 -0500 Subject: [gpfsug-discuss] AFM vs mmremotefs In-Reply-To: <5A22811B-7FBD-4091-B6FD-E5E74053DDF2@siriuscom.com> References: <5A22811B-7FBD-4091-B6FD-E5E74053DDF2@siriuscom.com> Message-ID: I think you "got it" - despite the name "remote" - mmremotefs is an access method and a way of organizing your data and machines into several clusters - which can work great if your network(s) are fast enough. For example you can have one or more client clusters with no GPFS(Spectrum Scale) file systems stored "locally". And one or several clusters that are servers to those client clusters. AFM is about caching and replicating files... You might do that because of "remote-ness" - or for other reasons... --marc From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Date: 03/30/2017 12:09 PM Subject: [gpfsug-discuss] AFM vs mmremotefs Sent by: gpfsug-discuss-bounces at spectrumscale.org When does it make sense to use mmremotefs vs AFM? With mmremotefs the actual data lives on the remote cluster right so if you need data closer then AFM make more sense to use? If the clusters get disconnected with mmremotefs I have not access to the data with AFM I do. Do I have these concepts right in my head? Mark R. Bush| Storage Architect Mobile: 210-237-8415 Twitter: @bushmr | LinkedIn: /markreedbush 10100 Reunion Place, Suite 500, San Antonio, TX 78216 www.siriuscom.com |mark.bush at siriuscom.com This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 8745 bytes Desc: not available URL: From gcorneau at us.ibm.com Thu Mar 30 17:42:38 2017 From: gcorneau at us.ibm.com (Glen Corneau) Date: Thu, 30 Mar 2017 10:42:38 -0600 Subject: [gpfsug-discuss] AFM vs mmremotefs In-Reply-To: References: <5A22811B-7FBD-4091-B6FD-E5E74053DDF2@siriuscom.com> Message-ID: Just a extra note: Multi-cluster (mmremotefs) does not always mean "network" for file system data transfer. If both cluster(s) see the storage over the SAN then the data happens over that path. This is common in a localized scenario for authorization reasons (i.e. nodes 1,2,3 can access file systems A & B, but not C; but nodes 4,5,6 in a separate cluster can access all 3 file systems) ------------------ Glen Corneau Power Systems Washington Systems Center gcorneau at us.ibm.com From: Marc A Kaplan/Watson/IBM at IBMUS To: gpfsug main discussion list Date: 03/30/2017 11:27 AM Subject: Re: [gpfsug-discuss] AFM vs mmremotefs Sent by: gpfsug-discuss-bounces at spectrumscale.org I think you "got it" - despite the name "remote" - mmremotefs is an access method and a way of organizing your data and machines into several clusters - which can work great if your network(s) are fast enough. For example you can have one or more client clusters with no GPFS(Spectrum Scale) file systems stored "locally". And one or several clusters that are servers to those client clusters. AFM is about caching and replicating files... You might do that because of "remote-ness" - or for other reasons... --marc From: "Mark.Bush at siriuscom.com" To: gpfsug main discussion list Date: 03/30/2017 12:09 PM Subject: [gpfsug-discuss] AFM vs mmremotefs Sent by: gpfsug-discuss-bounces at spectrumscale.org When does it make sense to use mmremotefs vs AFM? With mmremotefs the actual data lives on the remote cluster right so if you need data closer then AFM make more sense to use? If the clusters get disconnected with mmremotefs I have not access to the data with AFM I do. Do I have these concepts right in my head? Mark R. Bush| Storage Architect Mobile: 210-237-8415 Twitter: @bushmr| LinkedIn: /markreedbush 10100 Reunion Place, Suite 500, San Antonio, TX 78216 www.siriuscom.com|mark.bush at siriuscom.com This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 26117 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 8745 bytes Desc: not available URL: From christof.schmitt at us.ibm.com Thu Mar 30 21:18:21 2017 From: christof.schmitt at us.ibm.com (Christof Schmitt) Date: Thu, 30 Mar 2017 13:18:21 -0700 Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group In-Reply-To: References: Message-ID: willi.engeli at id.ethz.ch wrote on 03/30/2017 07:23:40 AM: > >-Last time I checked simply adding a normal computer object to the domain > didn't add the account of the adding user to the local administrators group > and CES is no exception. > > We have been using before a competitor Product as a NAS system. With that > system, we were able to define virtual NAS Servers, each one joined as an > independent object to AD. When joined, we found the 'Domain Admin' group and > the joining user as member of local administrators group of that virtual > server. > Since out AD is quite big, it is structured into many OU. We as the Storage > OU have OU admin rights, but we are not member of "Domain Admin" group. > Looking Back, we were able by ourselves to add the required groups as needed > to the local Administrators group of the NAS server. > Why is this important? Since we have quit a mix of OS accessing our shares, > some of the create exclusive access rights at the time they create profiles > etc. At the end of the lifecycle, one needs to delete those files via the > SMB / NFSV4 protocol, which is difficult if not having access rights. On the > other hand, we have seen situations, where one OS corrupted the ACL and > could not access anymore. Also this needs to be handled by us, giving us a > hard time not being member of the administrators group. I.e. the MS tool > subinacl does check the privileges before trying to modify ACLs, and if not > being member of the Administrators group, not all required privileges are > granted. There is two parts to that in Spectrum Scale: 1) There is an option to declare a user as 'admin users'. The notion there is that this user is mapped to root on access, thus this user can always access files and fix access issues. The user defined here should not be used for normal usage, this is only recommended for data migrations and to fix access issues. 2) When joining Spectrum Scale to an Active Directory domain, the Domain Admins groups is added to the internal Administrators group (sometimes referred to as BUILTIN\Administrators). One way to change the membership in that group would be through the MMC on a Windows client. Initially only Domain Admins are allowed, a member of this group would be required to add other users or groups. Alternatively, the "net sam" interface can be used to modify the group from root access on the protocol nodes: /usr/lpp/mmfs/bin/net sam listmem Administrators to list the members of the Administrators groups. /usr/lpp/mmfs/bin/net sam addmem Administrators DOMAIN\user to add a member. /usr/lpp/mmfs/bin/net sam delmem Administrators DOMAIN\user to remove a member This is currently an untested feature and not exposed through the CLI. If there is a need to have this exposed through the CLI or GUI, that should be requested through a RFE so that it can feed into the planning and prioritization for future releases. Regards, Christof Schmitt || IBM || Spectrum Scale Development || Tucson, AZ christof.schmitt at us.ibm.com || +1-520-799-2469 (T/L: 321-2469) From willi.engeli at id.ethz.ch Fri Mar 31 12:46:02 2017 From: willi.engeli at id.ethz.ch (Engeli Willi (ID SD)) Date: Fri, 31 Mar 2017 11:46:02 +0000 Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Message-ID: Hi Christoph, This solved my issues in most areas. Now I will probably add our Storage Management Group to local Administrators group, this way we are able to use all strong utilities like subinacl etc, and will be able to migrate to Spectrum Scale using robocopy with /ZB option working properly. For our Share-responsible Administrator we probably will add their Management user to the 'admin Users' option of the specific share allowing them to do wat ever they need to do, knowing that some tools may work with limitations. Do you know if we may also add a builtin group named BackupOperators? Regards Willi -----Urspr?ngliche Nachricht----- Von: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] Im Auftrag von gpfsug-discuss-request at spectrumscale.org Gesendet: Freitag, 31. M?rz 2017 13:00 An: gpfsug-discuss at spectrumscale.org Betreff: gpfsug-discuss Digest, Vol 62, Issue 82 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Spectrum Scale CES adds only Domain Admin to local Administrators group (Christof Schmitt) ---------------------------------------------------------------------- Message: 1 Date: Thu, 30 Mar 2017 13:18:21 -0700 From: "Christof Schmitt" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin to local Administrators group Message-ID: Content-Type: text/plain; charset="US-ASCII" willi.engeli at id.ethz.ch wrote on 03/30/2017 07:23:40 AM: > >-Last time I checked simply adding a normal computer object to the domain > didn't add the account of the adding user to the local administrators group > and CES is no exception. > > We have been using before a competitor Product as a NAS system. With that > system, we were able to define virtual NAS Servers, each one joined as an > independent object to AD. When joined, we found the 'Domain Admin' > group and > the joining user as member of local administrators group of that > virtual server. > Since out AD is quite big, it is structured into many OU. We as the Storage > OU have OU admin rights, but we are not member of "Domain Admin" group. > Looking Back, we were able by ourselves to add the required groups as needed > to the local Administrators group of the NAS server. > Why is this important? Since we have quit a mix of OS accessing our shares, > some of the create exclusive access rights at the time they create profiles > etc. At the end of the lifecycle, one needs to delete those files via the > SMB / NFSV4 protocol, which is difficult if not having access rights. > On the > other hand, we have seen situations, where one OS corrupted the ACL > and could not access anymore. Also this needs to be handled by us, > giving us a > hard time not being member of the administrators group. I.e. the MS > tool subinacl does check the privileges before trying to modify ACLs, > and if not > being member of the Administrators group, not all required privileges are > granted. There is two parts to that in Spectrum Scale: 1) There is an option to declare a user as 'admin users'. The notion there is that this user is mapped to root on access, thus this user can always access files and fix access issues. The user defined here should not be used for normal usage, this is only recommended for data migrations and to fix access issues. 2) When joining Spectrum Scale to an Active Directory domain, the Domain Admins groups is added to the internal Administrators group (sometimes referred to as BUILTIN\Administrators). One way to change the membership in that group would be through the MMC on a Windows client. Initially only Domain Admins are allowed, a member of this group would be required to add other users or groups. Alternatively, the "net sam" interface can be used to modify the group from root access on the protocol nodes: /usr/lpp/mmfs/bin/net sam listmem Administrators to list the members of the Administrators groups. /usr/lpp/mmfs/bin/net sam addmem Administrators DOMAIN\user to add a member. /usr/lpp/mmfs/bin/net sam delmem Administrators DOMAIN\user to remove a member This is currently an untested feature and not exposed through the CLI. If there is a need to have this exposed through the CLI or GUI, that should be requested through a RFE so that it can feed into the planning and prioritization for future releases. Regards, Christof Schmitt || IBM || Spectrum Scale Development || Tucson, AZ christof.schmitt at us.ibm.com || +1-520-799-2469 (T/L: 321-2469) ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 62, Issue 82 ********************************************** -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5461 bytes Desc: not available URL: From pinto at scinet.utoronto.ca Fri Mar 31 15:18:34 2017 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Fri, 31 Mar 2017 10:18:34 -0400 Subject: [gpfsug-discuss] BIG LAG since 3.5 on quota accounting reconciliation Message-ID: <20170331101834.162212izrlj4swu2@support.scinet.utoronto.ca> In the old days of DDN 9900 and gpfs 3.4 I only had to run mmcheckquota once a month, usually after the massive monthly purge. I noticed that starting with the GSS and ESS appliances under 3.5 that I needed to run mmcheckquota more often, at least once a week, or as often as daily, to clear the slippage errors in the accounting information, otherwise users complained that they were hitting their quotas, even throughout they deleted a lot of stuff. More recently we adopted a G200 appliance (1.8PB), with v4.1, and now things have gotten worst, and I have to run it twice daily, just in case. So, what I am missing? Is their a parameter since 3.5 and through 4.1 that we can set, so that GPFS will reconcile the quota accounting internally more often and on its own? Thanks Jaime ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. From billowen at us.ibm.com Fri Mar 31 19:23:35 2017 From: billowen at us.ibm.com (Bill Owen) Date: Fri, 31 Mar 2017 11:23:35 -0700 Subject: [gpfsug-discuss] mmnetverify In-Reply-To: References: , <18729d8a86fc430fac09c1c3a22159d2@mbxtoa1.winmail.deshaw.com> Message-ID: mmnetverify is a new tool that aims to make it easier to identify network problems. Regarding bandwidth commands, in 4.2.2, there are two options: mmnetverify bandwidth-node - [1 to 1] this will communicate from local node (or one or more nodes specified with -N option) to one or more target nodes. The bandwidth tests are executed serially from nodes in node list to target, iterating through each target node, one by one. The serially calculated bandwidth with each node is reported. mmnetverify bandwidth-cluster - [1 to many] this is a measure of parallel communication from the local node (or one or more nodes specified with -N option) to all of the other nodes in the cluster. The concurrent bandwidth with each target node in the cluster is reported. In both of these tests, we establish a socket connection, and pass a fixed number of bytes over the connection and calculate bandwidth based on how long that transmission took. For 4.2.3, there is a new bandwidth test called gnr-bandwidth. It is similar to the bandwidth-cluster [1 to many] except that it uses the following steps: 1. establish connection from node to all other target nodes in the cluster 2. start sending data to target for some ramp up period 3. after ramp up period, continue sending data for test period 4. calculate bandwidth based on bytes transmitted during test period The bandwidth to each node is summed to return a total bandwidth from the command node to the other nodes in the cluster. In future releases, we may modify bandwidth-node & bandwidth-cluster tests to use the gnr-bandwidth methodology (and deprecate gnr-bandwidth). Your feedback on how to improve mmnetverify is appreciated. Regarding: > We found some weird looking numbers that i don't quite understand and not in the places we might expect. > For example between hosts on the same switch, traffic flowing to another switch and traffic flowing to > nodes in another data centre where it's several switch hops. Some nodes over there were significantly > faster than switch local nodes. Note that system load can impact the test results. Is it possible that the slow nodes on the local switch were heavily loaded? Or is it possible they are using an interface that is lower bandwidth? (sorry, i had to ask that one to be sure...) Regards, Bill Owen billowen at us.ibm.com Spectrum Scale Development 520-799-4829 From: "Simon Thompson (Research Computing - IT Services)" To: gpfsug main discussion list Date: 03/17/2017 01:13 PM Subject: Re: [gpfsug-discuss] mmnetverify Sent by: gpfsug-discuss-bounces at spectrumscale.org It looks to run sequential tests to each node one at a time and isn't using NSD protocol but echo server. We found some weird looking numbers that i don't quite understand and not in the places we might expect. For example between hosts on the same switch, traffic flowing to another switch and traffic flowing to nodes in another data centre where it's several switch hops. Some nodes over there were significantly faster than switch local nodes. I think it was only added in 4.2.2 and is listed as "not yet a replacement for nsdperf". I get that is different as it's using NSD protocol, but was struggling a bit with what mmnetverify might be doing. Simon From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Sanchez, Paul [Paul.Sanchez at deshaw.com] Sent: 17 March 2017 19:43 To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] mmnetverify Sven will tell you: "RPC isn't streaming" and that may account for the discrepancy. If the tests are doing any "fan-in" where multiple nodes are sending to single node, then it's also possible that you are exhausting switch buffer memory in a way that a 1:1 iperf wouldn't. For our internal benchmarking we've used /usr/lpp/mmfs/samples/net/nsdperf to more closely estimate the real performance. I haven't played with mmnetverify yet though. -Paul -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (Research Computing - IT Services) Sent: Friday, March 17, 2017 2:50 PM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] mmnetverify Hi all, Just wondering if anyone has used the mmnetverify tool at all? Having made some changes to our internal L3 routing this week, I was interested to see what it claimed. As a side-note, it picked up some DNS resolution issues, though I'm not clear on some of those why it was claiming this as doing a "dig" on the node, it resolved fine (but adding the NSD servers to the hosts files cleared the error). Its actually the bandwidth tests that I'm interested in hearing other people's experience with as the numbers that some out from it are very different (lower) than if we use iperf to test performance between two nodes. Anyone any thoughts at all on this? Thanks Simon _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From christof.schmitt at us.ibm.com Fri Mar 31 21:22:36 2017 From: christof.schmitt at us.ibm.com (Christof Schmitt) Date: Fri, 31 Mar 2017 13:22:36 -0700 Subject: [gpfsug-discuss] Spectrum Scale CES adds only Domain Admin tolocal In-Reply-To: References: Message-ID: willi.engeli at id.ethz.ch wrote on 03/31/2017 04:46:02 AM: > Hi Christoph, > This solved my issues in most areas. > Now I will probably add our Storage Management Group to local > Administrators group, this way we are able to use all strong > utilities like subinacl etc, and will be able to migrate to Spectrum > Scale using robocopy with /ZB option working properly. > For our Share-responsible Administrator we probably will add their > Management user to the 'admin Users' option of the specific share > allowing them to do wat ever they need to do, knowing that some > tools may work with limitations. > > Do you know if we may also add a builtin group named BackupOperators? Privileges are not supported in Spectrum Scale: https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.2/com.ibm.spectrum.scale.v4r22.doc/bl1adm_fileauthlimitations.htm "Access privileges defined in Windows are not honored. Those privileges are tied to administrator groups and allow access, where the ACL alone does not grant it." You can create a group and assign the BackupOperator privilege: /usr/lpp/mmfs/bin/net sam createbuiltingroup 'Backup Operators' /usr/lpp/mmfs/bin/net sam rights grant 'Backup Operators' SeBackupPrivilege Without looking at all the details, i would suspect that this does not work. Access for a member of this group would overwrite the internal access check in Samba, but i would expect that it would fail as the file system also enforces the permissions defined in the ACL, and these are not overwritten by the additional privilege. The current workaround is the 'admin users' option. You might want to raise a RFE to request better support of the Backup privilege and the "Backup Operators" group. Regards, Christof Schmitt || IBM || Spectrum Scale Development || Tucson, AZ christof.schmitt at us.ibm.com || +1-520-799-2469 (T/L: 321-2469)