From adam.huffman at crick.ac.uk Mon Mar 2 09:24:59 2015 From: adam.huffman at crick.ac.uk (Adam Huffman) Date: Mon, 2 Mar 2015 09:24:59 +0000 Subject: [gpfsug-discuss] Introduction Message-ID: Hello A couple of weeks ago I joined Bruno Silva?s HPC team at the Francis Crick Institute, with special responsibility for HPC, OpenStack and virtualization. I?m very much a GPFS novice so I?m hoping to be able to draw on the knowledge in this group, while hopefully being able to help others with OpenStack. As Bruno stated in his message, we?re particularly interested in how to present GPFS to instances securely. I?ve read the discussion from November on this list, which didn?t seem to come to any firm conclusions. Has anyone involved then made substantial progress since? Cheers, Adam ? Adam Huffman Senior HPC & Virtualization Systems Engineer The Francis Crick Institute Gibbs Building 215 Euston Road London NW1 2BE T: E: adam.huffman at crick.ac.uk W: www.crick.ac.uk [Description: Description: ZA102637861][Description: Description: Twitter] The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no.06885462, with its registered office at 215 Euston Road, London NW1 2BE -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 158254D2-CDA8-43E5-96C1-A0AAEB10314B.png Type: image/png Size: 1151 bytes Desc: 158254D2-CDA8-43E5-96C1-A0AAEB10314B.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: B09B8AB7-5CA2-4DE7-BFE7-9D8F1B2A5166.png Type: image/png Size: 1211 bytes Desc: B09B8AB7-5CA2-4DE7-BFE7-9D8F1B2A5166.png URL: From viccornell at gmail.com Mon Mar 2 09:32:57 2015 From: viccornell at gmail.com (Vic Cornell) Date: Mon, 2 Mar 2015 09:32:57 +0000 Subject: [gpfsug-discuss] Introduction In-Reply-To: References: Message-ID: Hi Adam, I guess that one of the things that would help push it forward is a definition of what "secure" means to you. Regards, Vic On 2 Mar 2015, at 09:24, Adam Huffman wrote: > Hello > > A couple of weeks ago I joined Bruno Silva?s HPC team at the Francis Crick Institute, with special responsibility for HPC, OpenStack and virtualization. I?m very much a GPFS novice so I?m hoping to be able to draw on the knowledge in this group, while hopefully being able to help others with OpenStack. > > As Bruno stated in his message, we?re particularly interested in how to present GPFS to instances securely. I?ve read the discussion from November on this list, which didn?t seem to come to any firm conclusions. Has anyone involved then made substantial progress since? > > Cheers, > Adam > > ? > > Adam Huffman > Senior HPC & Virtualization Systems Engineer > The Francis Crick Institute > Gibbs Building > 215 Euston Road > London NW1 2BE > > T: > E: adam.huffman at crick.ac.uk > W: www.crick.ac.uk > <158254D2-CDA8-43E5-96C1-A0AAEB10314B.png> > > The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no.06885462, with its registered office at 215 Euston Road, London NW1 2BE > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam.huffman at crick.ac.uk Mon Mar 2 09:39:36 2015 From: adam.huffman at crick.ac.uk (Adam Huffman) Date: Mon, 2 Mar 2015 09:39:36 +0000 Subject: [gpfsug-discuss] Introduction In-Reply-To: References: Message-ID: Hi Vic, Re-emphasising that I?m still very much learning about GPFS, one of the approaches being discussed is running the GPFS client inside the instances. The concern here is over the case where users have root privileges inside their instances (a pretty common assumption for those used to AWS, for example) and the implications this may have for GPFS. Does it mean there would be a risk of spoofing? Cheers, Adam From: Vic Cornell Reply-To: gpfsug main discussion list Date: Monday, 2 March 2015 09:32 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Introduction Hi Adam, I guess that one of the things that would help push it forward is a definition of what "secure" means to you. Regards, Vic On 2 Mar 2015, at 09:24, Adam Huffman > wrote: Hello A couple of weeks ago I joined Bruno Silva?s HPC team at the Francis Crick Institute, with special responsibility for HPC, OpenStack and virtualization. I?m very much a GPFS novice so I?m hoping to be able to draw on the knowledge in this group, while hopefully being able to help others with OpenStack. As Bruno stated in his message, we?re particularly interested in how to present GPFS to instances securely. I?ve read the discussion from November on this list, which didn?t seem to come to any firm conclusions. Has anyone involved then made substantial progress since? Cheers, Adam ? Adam Huffman Senior HPC & Virtualization Systems Engineer The Francis Crick Institute Gibbs Building 215 Euston Road London NW1 2BE T: E: adam.huffman at crick.ac.uk W: www.crick.ac.uk <158254D2-CDA8-43E5-96C1-A0AAEB10314B.png> The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no.06885462, with its registered office at 215 Euston Road, London NW1 2BE _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From luke.raimbach at oerc.ox.ac.uk Mon Mar 2 19:21:25 2015 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Mon, 2 Mar 2015 19:21:25 +0000 Subject: [gpfsug-discuss] Introduction In-Reply-To: References: Message-ID: HI Adam, We run virtualised GPFS client nodes in a VMware cluster here at Oxford e-Research Centre. We had a requirement where one research group wanted root access to their VMs but also wanted fast direct access to their data on our GPFS cluster. The technical setup was (relatively) simple. We span up a small three-node virtual GPFS cluster (with no file system of its own). We then used the multi-cluster feature of GPFS to allow this small virtual cluster to join our main file system cluster which now gives that group very good IO performance. However, the problem of spoofing you mention is relevant ? we installed the virtual cluster nodes for the research group and put our Active Directory client on them. We also used the root-squash configuration option of the multi-cluster setup to prevent remote-cluster root access in to our file system. We also agreed with the research group that they would nominate one Administrator to have root access in their cluster and that they would maintain the AAA framework we put in place. We have to trust the group?s Administrator not to fiddle his UID or to let others escalate their privileges. If you were letting untrusted root users spin up Stacks, then you could still run GPFS clients in the OpenStack instance nodes to give them fast access to their data. Here are some musings on a recipe (others please feel free to pull these ideas to pieces): 1. Start with Cluster A ? your main production GPFS file system. It has GPFS device name /gpfs. 2. Pretend you have lots of money for extra disk to go under your OpenStack cluster (say you buy something like a DDN SFA7700 with a couple of expansion shelves and fill it up with 4TB drives ? 180 drives). 3. Drive this disk array with two, preferably four (or however many you want, really) decent NSD servers. Configure quorum nodes, etc. appropriately. Call this Cluster B. 4. Carve up the disk array in to something like 30 x RAID6 (4 + 2) LUNs and configure them as GPFS NSDs; but don?t create a file system (line up the stripe sizes etc. and choose a nice block size, etc. etc.)? 5. Put the GPFS metadata on some SSD NSDs somewhere. I like putting it on SSDs in the NSD server nodes and replicating it. Other people like putting it in their disk arrays. 6. As part of someone spinning up a Stack, get some scripts to do the following ?magic?: a. Connect to Cluster A and find out how big their target data-set is. b. Connect to Cluster B and create a new GPFS file system with a reasonable (dependent on the above result) number of NSD disks. Call this new GPFS device something unique other that /gpfs e.g. /gpfs0001. You could slice bits off your SSDs for the metadata NSDs in each file system you create in this manner (if you haven?t got many SSDs). c. As part of a new Stack, provide a few (three, say) GPFS quorum nodes that you?ve configured. Call this Cluster C. Add the rest of the stack instances to Cluster C. No File System. d. Pop back over to Cluster A. Export their target data-set from Cluster A using AFM (over GPFS or NFS ? pick your favourite: GPFS is probably better performant but means you need Cluster A to stay online). e. Now return to Cluster B. Import the target data to a local AFM cache on Cluster B?s new file system. Name the AFM file-set whatever you like, but link it in to the Cluster B /gpfs0001 namespace at the same level as it is in Cluster A. For example Cluster A: /gpfs/projects/dataset01 imports to an AFM fileset in Cluster B named userdataset01. Link this under /gpfs0001/projects/dataset01. f. Configure multi-cluster support on Cluster B to export GPFS device /gpfs0001 to Cluster C. Encrypt traffic if you want a headache. g. Configure multi-cluster support on Cluster C to remote mount Cluster B:/gpfs0001 as local device /gpfs. 7. You now have fast GPFS access to this user dataset *only* using GPFS clients inside the OpenStack instance nodes. You have also preserved the file system namespace in Cluster C?s instance nodes. If you only want to run over the data in the stack instances, you could pre-fetch the entire data-set using AFM Control from Cluster A in to the Cluster B file-set (if it?s big enough). 8. Now your users are finished and want to destroy the stack ? you need some more script ?magic?: a. Dismount the file system /gpfs in Cluster C. b. Connect to Cluster B and use AFM control to flush all the data back home to Cluster A. c. Unlink the file-set in Cluster B and force delete it; then delete the file system to free the NSDs back to the pool available to Cluster B. d. Connect back to Cluster A and unexport the original data-set directory structure. e. Throw away the VMs in the stack Things to worry about: ? Inode space will be different if users happen to be working on the data in Cluster A and Cluster C and want to know about inodes. GPFS XATTRS are preserved. ? If you use AFM over NFS because Cluster A and B are far away from each other and laggy, then there?s no locking with your AFM cache running as an Independent Writer. Writes at home (Cluster A) and in cache (Cluster B from Cluster C) will be nondeterministic. Your users will need to know this to avoid disappointment. ? If you use AFM over GPFS because Cluster A and B are near to each other and have a fast network, then there might still not be locking, but if Cluster A goes offline, it will put the AFM cache in to a ?dismounted? state. ? If your users want access to other parts of the Cluster A /gpfs namespace within their stack instances (because you have tools they use or they want to see other stuff), you can export them as read-only to a read-only AFM cache in Cluster B and they will be able to see things in Cluster C provided you link the AFM caches in the right places. Remember they have root access here. ? AFM updates are sent from cache to home as root so users can potentially overflow their quota at the Cluster A home site (root doesn?t care about quotas at home). ? Other frightening things might happen that I?ve not thought about. Hope this helps! Luke -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Adam Huffman Sent: 02 March 2015 09:40 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Introduction Hi Vic, Re-emphasising that I?m still very much learning about GPFS, one of the approaches being discussed is running the GPFS client inside the instances. The concern here is over the case where users have root privileges inside their instances (a pretty common assumption for those used to AWS, for example) and the implications this may have for GPFS. Does it mean there would be a risk of spoofing? Cheers, Adam From: Vic Cornell Reply-To: gpfsug main discussion list Date: Monday, 2 March 2015 09:32 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Introduction Hi Adam, I guess that one of the things that would help push it forward is a definition of what "secure" means to you. Regards, Vic On 2 Mar 2015, at 09:24, Adam Huffman > wrote: Hello A couple of weeks ago I joined Bruno Silva?s HPC team at the Francis Crick Institute, with special responsibility for HPC, OpenStack and virtualization. I?m very much a GPFS novice so I?m hoping to be able to draw on the knowledge in this group, while hopefully being able to help others with OpenStack. As Bruno stated in his message, we?re particularly interested in how to present GPFS to instances securely. I?ve read the discussion from November on this list, which didn?t seem to come to any firm conclusions. Has anyone involved then made substantial progress since? Cheers, Adam ? Adam Huffman Senior HPC & Virtualization Systems Engineer The Francis Crick Institute Gibbs Building 215 Euston Road London NW1 2BE T: E: adam.huffman at crick.ac.uk W: www.crick.ac.uk <158254D2-CDA8-43E5-96C1-A0AAEB10314B.png> The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no.06885462, with its registered office at 215 Euston Road, London NW1 2BE _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger_eriksson at se.ibm.com Thu Mar 12 08:25:22 2015 From: roger_eriksson at se.ibm.com (Roger Eriksson) Date: Thu, 12 Mar 2015 09:25:22 +0100 Subject: [gpfsug-discuss] GPFS UG 2015 date potential alteration In-Reply-To: <54EDD5E8.7090905@gpfsug.org> References: <54EDD5E8.7090905@gpfsug.org> Message-ID: Hi, has the GPFS User Group Meeting date been settled yet ? //Roger _________________________________________________________________ H?lsningar / Regards __________________________________________________________________ Roger Eriksson IBM Systems Lab Services Senior Accredited Product Services Professional XIV / Flashsystems / SVC / Storwize / SONAS / SAN EMEA Storage Competence Center (ESCC), Mainz, Germany IBM Svenska AB Isafjordsgatan 1 SE-164 92 Stockholm, Sweden Mobile +46(0)707933518 Email: roger_eriksson at se.ibm.com Notes: Roger Eriksson/Sweden/IBM at IBMSE Need help from EMEA Storage Competence Center ? IBMer ESCC @ Intranet --> http://escc.mainz.de.ibm.com/escc4u BPs ESCC @ Internet --> http://www.ibm.com/partnerworld/page/pw_com_bp_ecampus From: "Jez Tucker (Chair)" To: gpfsug main discussion list Date: 25/02/2015 15:02 Subject: [gpfsug-discuss] GPFS UG 2015 date potential alteration Sent by: gpfsug-discuss-bounces at gpfsug.org Alteration. Due to a scheduling issue, we may have to potentially move the date from 1st May. We will confirm this as soon as possible. Apologies. Jez _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss S?vida annat inte anges ovan: / Unless stated otherwise above: IBM Svenska AB Organisationsnummer: 556026-6883 Adress: 164 92 Stockholm -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 7526 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 7414 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 8944 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 9733 bytes Desc: not available URL: From chair at gpfsug.org Thu Mar 12 16:24:52 2015 From: chair at gpfsug.org (Jez Tucker (Chair)) Date: Thu, 12 Mar 2015 16:24:52 +0000 Subject: [gpfsug-discuss] GPFS UG 2015 date potential alteration In-Reply-To: References: <54EDD5E8.7090905@gpfsug.org> Message-ID: <5501BDD4.5060506@gpfsug.org> Hello all A fair question. We are working to a date of Wednesday 20th May. So please pencil this. The last pieces of paper work are being put in place, so please don't anybody book flights / expenses until this has been done. I would like to be in a position to have a definitive update on Monday. Jez On 12/03/15 08:25, Roger Eriksson wrote: > Hi, > > has the GPFS User Group Meeting date been settled yet ? > > //Roger > */_________________________________________________________________/* > */H?lsningar / Regards/* > */__________________________________________________________________/* > */Roger Eriksson/* > IBM Systems Lab Services > Senior Accredited Product Services Professional > XIV / Flashsystems / SVC / Storwize / SONAS / SAN > EMEA Storage Competence Center (ESCC), Mainz, Germany > IBM Svenska AB > Isafjordsgatan 1 > SE-164 92 Stockholm, Sweden > Mobile +46(0)707933518 > Email: _roger_eriksson at se.ibm.com_ > Notes: Roger Eriksson/Sweden/IBM at IBMSE > > > *Need help from EMEA Storage Competence Center ? * > *IBMer ESCC @ Intranet --> **http://escc.mainz.de.ibm.com/escc4u* > *BPs ESCC @ Internet --> > **http://www.ibm.com/partnerworld/page/pw_com_bp_ecampus* > > > > From: "Jez Tucker (Chair)" > To: gpfsug main discussion list > Date: 25/02/2015 15:02 > Subject: [gpfsug-discuss] GPFS UG 2015 date potential alteration > Sent by: gpfsug-discuss-bounces at gpfsug.org > ------------------------------------------------------------------------ > > > > Alteration. > > Due to a scheduling issue, we may have to potentially move the date > from 1st May. > > We will confirm this as soon as possible. > > Apologies. > > Jez > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > S?vida annat inte anges ovan: / Unless stated otherwise above: > IBM Svenska AB > Organisationsnummer: 556026-6883 > Adress: 164 92 Stockholm > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 7526 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 7414 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 8944 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 9733 bytes Desc: not available URL: From luke.raimbach at oerc.ox.ac.uk Fri Mar 13 14:44:33 2015 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Fri, 13 Mar 2015 14:44:33 +0000 Subject: [gpfsug-discuss] GPFS Journal Logs Message-ID: Hi, I'm interested in knowing more about the internal log journals GPFS uses to track file system activity from client nodes. My understanding is that when a file is opened for writing on a node, that (first) node becomes the 'metanode' for the file and is responsible for making updates to the file's metadata. Does the matanode also have responsibility for tracking changes made by other nodes that open the same file in its own journal, or are these recorded in each nodes own journal? I'm also interested in what happens when a node fails and its logs are replayed by the FS manager... what happens if that node is acting as metanode for a lot of open files which are also open elsewhere for writing on other nodes? If in the above scenario the metanode is handling metadata updates and tracking other changes in its journal, do all other nodes need to wait for the metanode to be ejected and have its journal replayed before they can continue? What is the process for selecting a successor metanode for the open files? Another related question I have is about initial sizing when creating a file system. Presumably the estimate for the total number of nodes mounting the file system affects the structure / allocation of internal journal logs? How are these arranged and is there any advice to overestimate this number if you intend to add more nodes at a future time? In the same vein, do multicluster nodes that join another remote stripegroup get allocated internal journal space as if they were new nodes in the remote cluster? Are these nodes allowed to be metanodes for files in a remote cluster? What happens when these nodes fail? Any insights greatly appreciated! Cheers, Luke. -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 From richard.lefebvre at calculquebec.ca Fri Mar 13 18:20:14 2015 From: richard.lefebvre at calculquebec.ca (Richard Lefebvre) Date: Fri, 13 Mar 2015 14:20:14 -0400 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. Message-ID: Hi, I have a GPFS cluster (v3.5). Last month, one of the filesystem had a problem with one of it's NSD. With the help tech support, the problem was fix and everything is OK data wise. The thing is when the problem occurred the filesystem was offline and once the filesystem was back to being OK, we were able to remount it. The problem is that now, every time a node is restarted, the filesystem is not mounted automatically as before. The other filesystems mount automatically. I did a mmlsfs on the file system and -A is at yes. Can someone tell my what I'm missing here. I'm sure it is something simple, but I'm drawing a blank right now. Richard -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) -------------- next part -------------- An HTML attachment was scrubbed... URL: From chair at gpfsug.org Mon Mar 16 14:02:09 2015 From: chair at gpfsug.org (Jez Tucker (Chair)) Date: Mon, 16 Mar 2015 14:02:09 +0000 Subject: [gpfsug-discuss] GPFS UG Meeting Agenda Message-ID: <5506E261.8000403@gpfsug.org> Hello all We have an internal agenda for the forthcoming UG. Before we publish this, I thought it prudent to solicit requests from our user base so that we might not miss anything important. So; - What's your burning issue, what's the one thing you really want to see? Regards, Jez From janfrode at tanso.net Mon Mar 16 15:11:13 2015 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Mon, 16 Mar 2015 16:11:13 +0100 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: Message-ID: Not much details to go by here, but I know I've had problems with SElinux blocking for getting filesystems mounted automatically at boot, while manually mounting afterwards worked fine. I fixed that by disabling automatic startup of gpfs from standard initscripts, and used this in rc.local instead: # make sure GPFS is not confined by selinux policies: runcon -t unconfined_t /usr/lpp/mmfs/bin/mmstartup -jf On Fri, Mar 13, 2015 at 7:20 PM, Richard Lefebvre < richard.lefebvre at calculquebec.ca> wrote: > Hi, > > I have a GPFS cluster (v3.5). Last month, one of the filesystem had a > problem with one of it's NSD. With the help tech support, the problem was > fix and everything is OK data wise. The thing is when the problem occurred > the filesystem was offline and once the filesystem was back to being OK, we > were able to remount it. The problem is that now, every time a node is > restarted, the filesystem is not mounted automatically as before. The other > filesystems mount automatically. I did a mmlsfs on the file system and -A > is at yes. Can someone tell my what I'm missing here. I'm sure it is > something simple, but I'm drawing a blank right now. > > Richard > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From YARD at il.ibm.com Mon Mar 16 15:14:25 2015 From: YARD at il.ibm.com (Yaron Daniel) Date: Mon, 16 Mar 2015 17:14:25 +0200 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: Message-ID: Hi In case of Linux - Check also : #chkconfig --list |grep gpfs Regards Yaron Daniel 94 Em Ha'Moshavot Rd Server, Storage and Data Services - Team Leader Petach Tiqva, 49527 Global Technology Services Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com IBM Israel From: Jan-Frode Myklebust To: gpfsug main discussion list Date: 03/16/2015 05:11 PM Subject: Re: [gpfsug-discuss] GPFS file system won't mount automatically anymore. Sent by: gpfsug-discuss-bounces at gpfsug.org Not much details to go by here, but I know I've had problems with SElinux blocking for getting filesystems mounted automatically at boot, while manually mounting afterwards worked fine. I fixed that by disabling automatic startup of gpfs from standard initscripts, and used this in rc.local instead: # make sure GPFS is not confined by selinux policies: runcon -t unconfined_t /usr/lpp/mmfs/bin/mmstartup -jf On Fri, Mar 13, 2015 at 7:20 PM, Richard Lefebvre < richard.lefebvre at calculquebec.ca> wrote: Hi, I have a GPFS cluster (v3.5). Last month, one of the filesystem had a problem with one of it's NSD. With the help tech support, the problem was fix and everything is OK data wise. The thing is when the problem occurred the filesystem was offline and once the filesystem was back to being OK, we were able to remount it. The problem is that now, every time a node is restarted, the filesystem is not mounted automatically as before. The other filesystems mount automatically. I did a mmlsfs on the file system and -A is at yes. Can someone tell my what I'm missing here. I'm sure it is something simple, but I'm drawing a blank right now. Richard -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1851 bytes Desc: not available URL: From luke.raimbach at oerc.ox.ac.uk Tue Mar 17 12:52:25 2015 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Tue, 17 Mar 2015 12:52:25 +0000 Subject: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans Message-ID: Hi Experts, I'm worrying about creating independent filesets which start to get in the way of one another: Let's pretend I create independent "fileset1" with a new inode space and preallocate say 1 million inodes. I start pushing files in to fileset1. I also start pushing files in to the root fileset. When fileset1 fills up, and because I've not specified a maximum number of inodes at creation time, presumably the file system manager gets on the case and preallocates more inodes as needed for fileset1. But while fileset1 has been filling up, I've also been filling up the root fileset (pretend I put 5 million files in there by the time fileset1 filled up). Now what happens? I'm guessing the FS manager starts chopping out sections of the root fileset beyond the 6 million region and marking them as independent and belonging to fileset1, in much the same way as it extends the allocated inode space when the file system starts running low. I hope this happens anyway. Let's say it does. So, now what about performance when running policy engine scans? I'm wanting to speed things up by restricting the policy scan to fileset1 (because I know that's where I put my special files). So I write the rules to select files and give them to mmapplypolicy. Does performance go down the drain as my once neatly arranged inode space is fragmented by not having enough inodes preallocated to begin with? Imagine fileset1 and the root fileset filling continuously at a similar rate and constantly "overtaking" one another in the inode space... does the performance advantage of independence start to go away rapidly? Cheers, Luke. -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 From makaplan at us.ibm.com Tue Mar 17 14:22:58 2015 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 17 Mar 2015 10:22:58 -0400 Subject: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans In-Reply-To: References: Message-ID: Luke, Thanks for your question. Independent filesets and their inodes are not implemented the way you might be imagining or guessing. Suppose you have two independent filesets "root" and "fset2" in the same GPFS file system. It is true that all the inode records (typically 512 bytes each - see mmcrfs) go into the same special file. BUT if you look at any given metadata allocation block --metadata-block-size (defaults to 256K) you'll only see inodes for either "root" or "fset2" not both in the same block. Moreover we try to "pre"-allocate several blocks of inodes for the same independent fileset contiguously - so that typically GPFS can do one seek and then read several blocks of inodes from the same independent fileset. So there can be performance advantages to using independent filesets and restricting your mmapplypolicy scans to just the fileset you need. To gain maximal advantage, use the following form of the command: mmapplypolicy /gpfs/path-to-the-directory-tree-I-want-to-scan --scope {fileset | inodespace } ... An inodespace is the set of all inodes in an independent fileset. An inodespace may contain several "dependent" filesets. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 21994 bytes Desc: not available URL: From zgiles at gmail.com Tue Mar 17 15:49:06 2015 From: zgiles at gmail.com (Zachary Giles) Date: Tue, 17 Mar 2015 11:49:06 -0400 Subject: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans In-Reply-To: References: Message-ID: I've always wondered also what the decision point is to make new inode tables per fileset. Sounds like the main benefit is scope of mmapplypolicy ( and other commands maybe ) and subblocks being grouped together for performance reasons. Is that right? Are there other benefits like portability or grouping a fileset's inode table into specific failure groups (in the system pool) etc? Any downsides? ( for example: now you have more overall metadata reads.. or only a limited number of inode tables.. or total number of inodes in the system is higher due to overallocation on N tables ? ) -Zach On Tue, Mar 17, 2015 at 10:22 AM, Marc A Kaplan wrote: > Luke, > > Thanks for your question. Independent filesets and their inodes are not > implemented the way you might be imagining or guessing. > > Suppose you have two independent filesets "root" and "fset2" in the same > GPFS file system. > It is true that all the inode records (typically 512 bytes each - see > mmcrfs) go into the same special file. BUT if you look at any given > metadata allocation block --metadata-block-size (defaults to 256K) you'll > only see inodes for either "root" or "fset2" not both in the same block. > > Moreover we try to "pre"-allocate several blocks of inodes for the same > independent fileset contiguously - so that typically GPFS can do one seek > and then read several blocks of inodes from the same independent fileset. > > So there can be performance advantages to using independent filesets and > restricting your mmapplypolicy scans to just the fileset you need. > To gain maximal advantage, use the following form of the command: > > mmapplypolicy /gpfs/path-to-the-directory-tree-I-want-to-scan --scope > {fileset | inodespace } ... > > > An inodespace is the set of all inodes in an independent fileset. An > inodespace may contain several "dependent" filesets. > > > > > [image: Marc A Kaplan] > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- Zach Giles zgiles at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 21994 bytes Desc: not available URL: From Paul.Sanchez at deshaw.com Tue Mar 17 16:01:21 2015 From: Paul.Sanchez at deshaw.com (Sanchez, Paul) Date: Tue, 17 Mar 2015 16:01:21 +0000 Subject: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans In-Reply-To: References: Message-ID: <201D6001C896B846A9CFC2E841986AC145360AA6@mailnycmb2a.winmail.deshaw.com> It?s also required for per-fileset snapshots, which are useful if you have differing snapshot requirements or want the ability to flush high-churn snapshots that are consuming an unexpected amount of space, but with better resolution than a whole filesystem. -Paul Sanchez From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Zachary Giles Sent: Tuesday, March 17, 2015 11:49 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans I've always wondered also what the decision point is to make new inode tables per fileset. Sounds like the main benefit is scope of mmapplypolicy ( and other commands maybe ) and subblocks being grouped together for performance reasons. Is that right? Are there other benefits like portability or grouping a fileset's inode table into specific failure groups (in the system pool) etc? Any downsides? ( for example: now you have more overall metadata reads.. or only a limited number of inode tables.. or total number of inodes in the system is higher due to overallocation on N tables ? ) -Zach On Tue, Mar 17, 2015 at 10:22 AM, Marc A Kaplan > wrote: Luke, Thanks for your question. Independent filesets and their inodes are not implemented the way you might be imagining or guessing. Suppose you have two independent filesets "root" and "fset2" in the same GPFS file system. It is true that all the inode records (typically 512 bytes each - see mmcrfs) go into the same special file. BUT if you look at any given metadata allocation block --metadata-block-size (defaults to 256K) you'll only see inodes for either "root" or "fset2" not both in the same block. Moreover we try to "pre"-allocate several blocks of inodes for the same independent fileset contiguously - so that typically GPFS can do one seek and then read several blocks of inodes from the same independent fileset. So there can be performance advantages to using independent filesets and restricting your mmapplypolicy scans to just the fileset you need. To gain maximal advantage, use the following form of the command: mmapplypolicy /gpfs/path-to-the-directory-tree-I-want-to-scan --scope {fileset | inodespace } ... An inodespace is the set of all inodes in an independent fileset. An inodespace may contain several "dependent" filesets. [Marc A Kaplan] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Zach Giles zgiles at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 21994 bytes Desc: image001.gif URL: From zgiles at gmail.com Tue Mar 17 16:04:25 2015 From: zgiles at gmail.com (Zachary Giles) Date: Tue, 17 Mar 2015 12:04:25 -0400 Subject: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans In-Reply-To: <201D6001C896B846A9CFC2E841986AC145360AA6@mailnycmb2a.winmail.deshaw.com> References: <201D6001C896B846A9CFC2E841986AC145360AA6@mailnycmb2a.winmail.deshaw.com> Message-ID: Ah. that's a good point about the snapshots. On Tue, Mar 17, 2015 at 12:01 PM, Sanchez, Paul wrote: > It?s also required for per-fileset snapshots, which are useful if you > have differing snapshot requirements or want the ability to flush > high-churn snapshots that are consuming an unexpected amount of space, but > with better resolution than a whole filesystem. > > > > -Paul Sanchez > > > > *From:* gpfsug-discuss-bounces at gpfsug.org [mailto: > gpfsug-discuss-bounces at gpfsug.org] *On Behalf Of *Zachary Giles > *Sent:* Tuesday, March 17, 2015 11:49 AM > *To:* gpfsug main discussion list > *Subject:* Re: [gpfsug-discuss] Fragmented Inode Space and Performance of > Policy Scans > > > > I've always wondered also what the decision point is to make new inode > tables per fileset. > > > > Sounds like the main benefit is scope of mmapplypolicy ( and other > commands maybe ) and subblocks being grouped together for performance > reasons. Is that right? > > Are there other benefits like portability or grouping a fileset's inode > table into specific failure groups (in the system pool) etc? > > Any downsides? ( for example: now you have more overall metadata reads.. > or only a limited number of inode tables.. or total number of inodes in the > system is higher due to overallocation on N tables ? ) > > > > -Zach > > > > > > On Tue, Mar 17, 2015 at 10:22 AM, Marc A Kaplan > wrote: > > Luke, > > Thanks for your question. Independent filesets and their inodes are not > implemented the way you might be imagining or guessing. > > Suppose you have two independent filesets "root" and "fset2" in the same > GPFS file system. > It is true that all the inode records (typically 512 bytes each - see > mmcrfs) go into the same special file. BUT if you look at any given > metadata allocation block --metadata-block-size (defaults to 256K) you'll > only see inodes for either "root" or "fset2" not both in the same block. > > Moreover we try to "pre"-allocate several blocks of inodes for the same > independent fileset contiguously - so that typically GPFS can do one seek > and then read several blocks of inodes from the same independent fileset. > > So there can be performance advantages to using independent filesets and > restricting your mmapplypolicy scans to just the fileset you need. > To gain maximal advantage, use the following form of the command: > > mmapplypolicy /gpfs/path-to-the-directory-tree-I-want-to-scan --scope > {fileset | inodespace } ... > > > An inodespace is the set of all inodes in an independent fileset. An > inodespace may contain several "dependent" filesets. > > > > > [image: Marc A Kaplan] > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > -- > > Zach Giles > zgiles at gmail.com > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- Zach Giles zgiles at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 21994 bytes Desc: not available URL: From chair at gpfsug.org Tue Mar 17 16:47:43 2015 From: chair at gpfsug.org (Jez Tucker (Chair)) Date: Tue, 17 Mar 2015 16:47:43 +0000 Subject: [gpfsug-discuss] GPFS User Group 2015 Meeting Date confirmed - RSVP Message-ID: <55085AAF.8050108@gpfsug.org> Hello all I can now confirm that 2015 GPFS User Group Meeting will be on May 20th, in tandem with SPXXL. For full details please see: http://www.gpfsug.org/gpfs-user-group-2015-invite/ RSVP to secretary at gpfsug.org See you there. Jez From richard.lefebvre at calculquebec.ca Thu Mar 19 22:32:58 2015 From: richard.lefebvre at calculquebec.ca (Richard Lefebvre) Date: Thu, 19 Mar 2015 18:32:58 -0400 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: Message-ID: Thanks for those who replied. One thing I didn't mention correctly is that I have several gpfs mount say: /gpfs1 /gpfs2 /gpfs3 Say it was gpfs2 that had the NSD problem. So now when I reboot a node, /gpfs1 and /gpfs3 mount, but /gpfs2 doesn't, I have to manually type mmmount gpfs2 on the node to get it mounted after the boot. So I don't think that it is a problem with mounting within linux boot sequence. But a flag than I'm missing in the gpfs setup. Or status of the filesystem that is not set OK that I cannot figure out. The output of mmlsfs of a system that mounts compared the the one that doesn't are the same except for mount point, NSDs and creation time. Is there another "mm..." command with which to check the status of a filesystem? Richard 2015-03-13 14:20 GMT-04:00 Richard Lefebvre < richard.lefebvre at calculquebec.ca>: > Hi, > > I have a GPFS cluster (v3.5). Last month, one of the filesystem had a > problem with one of it's NSD. With the help tech support, the problem was > fix and everything is OK data wise. The thing is when the problem occurred > the filesystem was offline and once the filesystem was back to being OK, we > were able to remount it. The problem is that now, every time a node is > restarted, the filesystem is not mounted automatically as before. The other > filesystems mount automatically. I did a mmlsfs on the file system and -A > is at yes. Can someone tell my what I'm missing here. I'm sure it is > something simple, but I'm drawing a blank right now. > > Richard > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Thu Mar 19 22:35:38 2015 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Thu, 19 Mar 2015 22:35:38 +0000 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: Message-ID: Check "mmlsdisk". Are all disks for the filesystem ok? -jf On tor. 19. mar. 2015 at 23.33 Richard Lefebvre < richard.lefebvre at calculquebec.ca> wrote: > Thanks for those who replied. One thing I didn't mention correctly is that > I have several gpfs mount say: > /gpfs1 > /gpfs2 > /gpfs3 > > Say it was gpfs2 that had the NSD problem. So now when I reboot a node, > /gpfs1 and /gpfs3 mount, but /gpfs2 doesn't, I have to manually type > mmmount gpfs2 on the node to get it mounted after the boot. > > So I don't think that it is a problem with mounting within linux boot > sequence. But a flag than I'm missing in the gpfs setup. Or status of the > filesystem that is not set OK that I cannot figure out. The output of > mmlsfs of a system that mounts compared the the one that doesn't are the > same except for mount point, NSDs and creation time. Is there another > "mm..." command with which to check the status of a filesystem? > > Richard > > > 2015-03-13 14:20 GMT-04:00 Richard Lefebvre < > richard.lefebvre at calculquebec.ca>: > > Hi, >> >> I have a GPFS cluster (v3.5). Last month, one of the filesystem had a >> problem with one of it's NSD. With the help tech support, the problem was >> fix and everything is OK data wise. The thing is when the problem occurred >> the filesystem was offline and once the filesystem was back to being OK, we >> were able to remount it. The problem is that now, every time a node is >> restarted, the filesystem is not mounted automatically as before. The other >> filesystems mount automatically. I did a mmlsfs on the file system and -A >> is at yes. Can someone tell my what I'm missing here. I'm sure it is >> something simple, but I'm drawing a blank right now. >> >> Richard >> >> -- >> Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" >> Richard.Lefebvre at calculquebec.ca -- THGTTG >> Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) >> > > > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.lefebvre at calculquebec.ca Thu Mar 19 22:38:43 2015 From: richard.lefebvre at calculquebec.ca (Richard Lefebvre) Date: Thu, 19 Mar 2015 18:38:43 -0400 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: Message-ID: Hi JF, There are all up and ready. Otherwise I wouldn't be able to mmmount it. Richard 2015-03-19 18:35 GMT-04:00 Jan-Frode Myklebust : > Check "mmlsdisk". Are all disks for the filesystem ok? > > > -jf > On tor. 19. mar. 2015 at 23.33 Richard Lefebvre < > richard.lefebvre at calculquebec.ca> wrote: > >> Thanks for those who replied. One thing I didn't mention correctly is >> that I have several gpfs mount say: >> /gpfs1 >> /gpfs2 >> /gpfs3 >> >> Say it was gpfs2 that had the NSD problem. So now when I reboot a node, >> /gpfs1 and /gpfs3 mount, but /gpfs2 doesn't, I have to manually type >> mmmount gpfs2 on the node to get it mounted after the boot. >> >> So I don't think that it is a problem with mounting within linux boot >> sequence. But a flag than I'm missing in the gpfs setup. Or status of the >> filesystem that is not set OK that I cannot figure out. The output of >> mmlsfs of a system that mounts compared the the one that doesn't are the >> same except for mount point, NSDs and creation time. Is there another >> "mm..." command with which to check the status of a filesystem? >> >> Richard >> >> >> 2015-03-13 14:20 GMT-04:00 Richard Lefebvre < >> richard.lefebvre at calculquebec.ca>: >> >> Hi, >>> >>> I have a GPFS cluster (v3.5). Last month, one of the filesystem had a >>> problem with one of it's NSD. With the help tech support, the problem was >>> fix and everything is OK data wise. The thing is when the problem occurred >>> the filesystem was offline and once the filesystem was back to being OK, we >>> were able to remount it. The problem is that now, every time a node is >>> restarted, the filesystem is not mounted automatically as before. The other >>> filesystems mount automatically. I did a mmlsfs on the file system and -A >>> is at yes. Can someone tell my what I'm missing here. I'm sure it is >>> something simple, but I'm drawing a blank right now. >>> >>> Richard >>> >>> -- >>> Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" >>> Richard.Lefebvre at calculquebec.ca -- THGTTG >>> Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) >>> >> >> >> >> -- >> Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" >> Richard.Lefebvre at calculquebec.ca -- THGTTG >> Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at gpfsug.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) -------------- next part -------------- An HTML attachment was scrubbed... URL: From red_steel at au1.ibm.com Fri Mar 20 01:48:52 2015 From: red_steel at au1.ibm.com (Tony Steel) Date: Fri, 20 Mar 2015 12:48:52 +1100 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: Message-ID: Hi You don't have a file (assuming gpfs2 is the problem) /var/mmfs/etc/ignoreStartupMount.gpfs2 ?? Cheers, - Sense Amid Madness, Wit Amidst Folly - ---------- Antony (Red) Steel, Advanced Technical Support -------------------- Senior IT Specialist Ph: +61 293978414 Mb: +61 409 971 083 Home Page: http://sydgsa.ibm.com/projects/t/team_web/index.html ----------------------------- red_steel at au.ibm.com ----------------------------------- From ckerner at ncsa.uiuc.edu Fri Mar 20 02:56:55 2015 From: ckerner at ncsa.uiuc.edu (Chad Kerner) Date: Thu, 19 Mar 2015 21:56:55 -0500 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: Message-ID: <20150320025655.GA2140@logos.ncsa.illinois.edu> Richard, what does "mmlsfs gpfs2 -A" return? Is it set to automount? --Chad Chad Kerner, Systems Engineer Storage Enabling Technologies National Center for Supercomputing Applications University of Illinois, Urbana-Champaign On Thu, Mar 19, 2015 at 06:38:43PM -0400, Richard Lefebvre wrote: > Hi JF, > > There are all up and ready. Otherwise I wouldn't be able to mmmount it. > > Richard > > > > 2015-03-19 18:35 GMT-04:00 Jan-Frode Myklebust : > > Check "mmlsdisk". Are all disks for the filesystem ok? > > > -jf > On tor. 19. mar. 2015 at 23.33 Richard Lefebvre < > richard.lefebvre at calculquebec.ca> wrote: > > Thanks for those who replied. One thing I didn't mention correctly is > that I have several gpfs mount say: > /gpfs1 > /gpfs2 > /gpfs3 > > Say it was gpfs2 that had the NSD problem. So now when I reboot a node, > /gpfs1 and /gpfs3 mount, but /gpfs2 doesn't, I have to manually type > mmmount gpfs2 on the node to get it mounted after the boot. > > So I don't think that it is a problem with mounting within linux boot > sequence. But a flag than I'm missing in the gpfs setup. Or status of > the filesystem that is not set OK that I cannot figure out. The output > of mmlsfs of a system that mounts compared the the one that doesn't are > the same except for mount point, NSDs and creation time. Is there > another "mm..." command with which to check the status of a filesystem? > > Richard > > > 2015-03-13 14:20 GMT-04:00 Richard Lefebvre < > richard.lefebvre at calculquebec.ca>: > > > Hi, > > I have a GPFS cluster (v3.5). Last month, one of the filesystem had > a problem with one of it's NSD. With the help tech support, the > problem was fix and everything is OK data wise. The thing is when > the problem occurred the filesystem was offline and once the > filesystem was back to being OK, we were able to remount it. The > problem is that now, every time a node is restarted, the filesystem > is not mounted automatically as before. The other filesystems mount > automatically. I did a mmlsfs on the file system and -A is at yes. > Can someone tell my what I'm missing here. I'm sure it is something > simple, but I'm drawing a blank right now. > > Richard > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't > Panic" > Richard.Lefebvre at calculquebec.ca -- > THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada ( > computecanada.ca) > > > > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't > Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From luke.raimbach at oerc.ox.ac.uk Fri Mar 20 08:37:02 2015 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Fri, 20 Mar 2015 08:37:02 +0000 Subject: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans Message-ID: Hi Marc, Thanks for this information. One thing I still can't extract from the manual (though I guess I could just do a small experiment to find out) is whether the number of allocated inodes is automatically extended when running low, up to the maximum for an independent fileset? Cheers, Luke. From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Marc A Kaplan Sent: 17 March 2015 14:23 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans Luke, Thanks for your question. Independent filesets and their inodes are not implemented the way you might be imagining or guessing. Suppose you have two independent filesets "root" and "fset2" in the same GPFS file system. It is true that all the inode records (typically 512 bytes each - see mmcrfs) go into the same special file. BUT if you look at any given metadata allocation block --metadata-block-size (defaults to 256K) you'll only see inodes for either "root" or "fset2" not both in the same block. Moreover we try to "pre"-allocate several blocks of inodes for the same independent fileset contiguously - so that typically GPFS can do one seek and then read several blocks of inodes from the same independent fileset. So there can be performance advantages to using independent filesets and restricting your mmapplypolicy scans to just the fileset you need. To gain maximal advantage, use the following form of the command: mmapplypolicy /gpfs/path-to-the-directory-tree-I-want-to-scan --scope {fileset | inodespace } ... An inodespace is the set of all inodes in an independent fileset. An inodespace may contain several "dependent" filesets. [Marc A Kaplan] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 21994 bytes Desc: image001.gif URL: From makaplan at us.ibm.com Fri Mar 20 12:10:34 2015 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 20 Mar 2015 08:10:34 -0400 Subject: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans In-Reply-To: References: Message-ID: Seems to me that's why this option has two parameters: mmchfileset ... [--inode-limit MaxNumInodes[:NumInodesToPreallocate]] Specifies the new inode limit for the inode space owned by the specified fileset. The FilesetName or JunctionPath must refer to an independent fileset. The NumInodesToPreallocate specifies an optional number of additional inodes to pre-allocate for the inode space. Use the mmchfs command to change inode limits for the root fileset. See also mmchfs! And RTFM! ( ReadTheFantasicManual ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.lefebvre at calculquebec.ca Fri Mar 20 17:22:43 2015 From: richard.lefebvre at calculquebec.ca (Richard Lefebvre) Date: Fri, 20 Mar 2015 13:22:43 -0400 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: <20150320025655.GA2140@logos.ncsa.illinois.edu> References: <20150320025655.GA2140@logos.ncsa.illinois.edu> Message-ID: Hi Tony, Chad, There is nothing in /var/mmfs/etc directory of the nodes. "-A" on the nodes return "yes" Richard 2015-03-19 22:56 GMT-04:00 Chad Kerner : > Richard, what does "mmlsfs gpfs2 -A" return? Is it set to automount? > > --Chad > Chad Kerner, Systems Engineer > Storage Enabling Technologies > National Center for Supercomputing Applications > University of Illinois, Urbana-Champaign > > On Thu, Mar 19, 2015 at 06:38:43PM -0400, Richard Lefebvre wrote: > > Hi JF, > > > > There are all up and ready. Otherwise I wouldn't be able to mmmount it. > > > > Richard > > > > > > > > 2015-03-19 18:35 GMT-04:00 Jan-Frode Myklebust : > > > > Check "mmlsdisk". Are all disks for the filesystem ok? > > > > > > -jf > > On tor. 19. mar. 2015 at 23.33 Richard Lefebvre < > > richard.lefebvre at calculquebec.ca> wrote: > > > > Thanks for those who replied. One thing I didn't mention > correctly is > > that I have several gpfs mount say: > > /gpfs1 > > /gpfs2 > > /gpfs3 > > > > Say it was gpfs2 that had the NSD problem. So now when I reboot > a node, > > /gpfs1 and /gpfs3 mount, but /gpfs2 doesn't, I have to manually > type > > mmmount gpfs2 on the node to get it mounted after the boot. > > > > So I don't think that it is a problem with mounting within linux > boot > > sequence. But a flag than I'm missing in the gpfs setup. Or > status of > > the filesystem that is not set OK that I cannot figure out. The > output > > of mmlsfs of a system that mounts compared the the one that > doesn't are > > the same except for mount point, NSDs and creation time. Is there > > another "mm..." command with which to check the status of a > filesystem? > > > > Richard > > > > > > 2015-03-13 14:20 GMT-04:00 Richard Lefebvre < > > richard.lefebvre at calculquebec.ca>: > > > > > > Hi, > > > > I have a GPFS cluster (v3.5). Last month, one of the > filesystem had > > a problem with one of it's NSD. With the help tech support, > the > > problem was fix and everything is OK data wise. The thing is > when > > the problem occurred the filesystem was offline and once the > > filesystem was back to being OK, we were able to remount it. > The > > problem is that now, every time a node is restarted, the > filesystem > > is not mounted automatically as before. The other > filesystems mount > > automatically. I did a mmlsfs on the file system and -A is > at yes. > > Can someone tell my what I'm missing here. I'm sure it is > something > > simple, but I'm drawing a blank right now. > > > > Richard > > > > -- > > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 > "Don't > > Panic" > > Richard.Lefebvre at calculquebec.ca > -- > > THGTTG > > Calcul Quebec (calculquebec.ca) ------ Calcul Canada ( > > computecanada.ca) > > > > > > > > > > -- > > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 > "Don't > > Panic" > > Richard.Lefebvre at calculquebec.ca > -- THGTTG > > Calcul Quebec (calculquebec.ca) ------ Calcul Canada ( > computecanada.ca) > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at gpfsug.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at gpfsug.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > > > > > > -- > > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" > > Richard.Lefebvre at calculquebec.ca -- THGTTG > > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at gpfsug.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) -------------- next part -------------- An HTML attachment was scrubbed... URL: From YARD at il.ibm.com Fri Mar 20 17:50:33 2015 From: YARD at il.ibm.com (Yaron Daniel) Date: Fri, 20 Mar 2015 19:50:33 +0200 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: <20150320025655.GA2140@logos.ncsa.illinois.edu> Message-ID: Hi Can u send the output of: #mmlsdisk gpfs2 #mmlsnsd |grep gpfs2 #getenforce #iptables -L #cat /etc/fstab Regards Yaron Daniel 94 Em Ha'Moshavot Rd Server, Storage and Data Services - Team Leader Petach Tiqva, 49527 Global Technology Services Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com IBM Israel From: Richard Lefebvre To: gpfsug main discussion list Date: 03/20/2015 07:22 PM Subject: Re: [gpfsug-discuss] GPFS file system won't mount automatically anymore. Sent by: gpfsug-discuss-bounces at gpfsug.org Hi Tony, Chad, There is nothing in /var/mmfs/etc directory of the nodes. "-A" on the nodes return "yes" Richard 2015-03-19 22:56 GMT-04:00 Chad Kerner : Richard, what does "mmlsfs gpfs2 -A" return? Is it set to automount? --Chad Chad Kerner, Systems Engineer Storage Enabling Technologies National Center for Supercomputing Applications University of Illinois, Urbana-Champaign On Thu, Mar 19, 2015 at 06:38:43PM -0400, Richard Lefebvre wrote: > Hi JF, > > There are all up and ready. Otherwise I wouldn't be able to mmmount it. > > Richard > > > > 2015-03-19 18:35 GMT-04:00 Jan-Frode Myklebust : > > Check "mmlsdisk". Are all disks for the filesystem ok? > > > -jf > On tor. 19. mar. 2015 at 23.33 Richard Lefebvre < > richard.lefebvre at calculquebec.ca> wrote: > > Thanks for those who replied. One thing I didn't mention correctly is > that I have several gpfs mount say: > /gpfs1 > /gpfs2 > /gpfs3 > > Say it was gpfs2 that had the NSD problem. So now when I reboot a node, > /gpfs1 and /gpfs3 mount, but /gpfs2 doesn't, I have to manually type > mmmount gpfs2 on the node to get it mounted after the boot. > > So I don't think that it is a problem with mounting within linux boot > sequence. But a flag than I'm missing in the gpfs setup. Or status of > the filesystem that is not set OK that I cannot figure out. The output > of mmlsfs of a system that mounts compared the the one that doesn't are > the same except for mount point, NSDs and creation time. Is there > another "mm..." command with which to check the status of a filesystem? > > Richard > > > 2015-03-13 14:20 GMT-04:00 Richard Lefebvre < > richard.lefebvre at calculquebec.ca>: > > > Hi, > > I have a GPFS cluster (v3.5). Last month, one of the filesystem had > a problem with one of it's NSD. With the help tech support, the > problem was fix and everything is OK data wise. The thing is when > the problem occurred the filesystem was offline and once the > filesystem was back to being OK, we were able to remount it. The > problem is that now, every time a node is restarted, the filesystem > is not mounted automatically as before. The other filesystems mount > automatically. I did a mmlsfs on the file system and -A is at yes. > Can someone tell my what I'm missing here. I'm sure it is something > simple, but I'm drawing a blank right now. > > Richard > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't > Panic" > Richard.Lefebvre at calculquebec.ca -- > THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada ( > computecanada.ca) > > > > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't > Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada ( computecanada.ca) > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1851 bytes Desc: not available URL: From makaplan at us.ibm.com Fri Mar 20 19:24:25 2015 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 20 Mar 2015 15:24:25 -0400 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: <20150320025655.GA2140@logos.ncsa.illinois.edu> Message-ID: Look in /var/adm/ras/mmfs.log.* -- on a node that should be mounting the filesystem -- find a file that covers the time when GPFS is starting up and should be mounting the filesystem. Look for clues and/or error messages. If you are not familiar, you may also want to look at what messages are issued when everything is working just right. -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.lefebvre at calculquebec.ca Fri Mar 20 20:32:46 2015 From: richard.lefebvre at calculquebec.ca (Richard Lefebvre) Date: Fri, 20 Mar 2015 16:32:46 -0400 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: <20150320025655.GA2140@logos.ncsa.illinois.edu> Message-ID: Hi Mark, Thank you for your answer. You are right, I have the following: Mon Mar 16 11:02:24 EDT 2015: mounting /dev/gpfs2 Mon Mar 16 11:02:24.296 2015: Command: mount gpfs2 Mon Mar 16 11:02:24.723 2015: Disk failure. Volume gpfs2. rc = 5. Physical volume gpfs2nsd04. Mon Mar 16 11:02:24.724 2015: Disk failure. Volume gpfs2. rc = 5. Physical volume gpfs2nsd12. Mon Mar 16 11:02:24.723 2015: File System gpfs2 unmounted by the system with return code 5 reason code 0 Mon Mar 16 11:02:24.724 2015: Input/output error Mon Mar 16 11:02:24.725 2015: Failed to open gpfs2. Mon Mar 16 11:02:24.724 2015: Input/output error Mon Mar 16 11:02:24.725 2015: Command: err 666: mount gpfs2 Mon Mar 16 11:02:24.724 2015: Input/output error mount: Stale NFS file handle Mon Mar 16 11:02:24 EDT 2015: finished mounting /dev/gpfs2 But it is not mounted. This is after issuing the "mmmount gpfs2" command Mon Mar 16 11:03:52.010 2015: Command: mount gpfs2 Mon Mar 16 11:03:52.234 2015: VERBS RDMA connecting to 172.25.101.2 (heca2-ib0) on mlx4_0 port 1 id 4 Mon Mar 16 11:03:52.235 2015: VERBS RDMA connected to 172.25.101.2 (heca2-ib0) on mlx4_0 port 1 sl 0 Mon Mar 16 11:03:52.381 2015: Command: err 0: mount gpfs2 Now, my next step it to figure out how the clear the failure since it is not failed anymore. The mmlsdisk reports all the nsd are up and ready. One command I'm also looking for is a command to list which nsd server is used for a particular nsd. I have a redundant config for which 2 servers can access the same nsd. But I think that for some of them, the default server is not used and the load of the servers is not distributed as wanted. Richard 2015-03-20 15:24 GMT-04:00 Marc A Kaplan : > Look in /var/adm/ras/mmfs.log.* -- on a node that should be mounting the > filesystem -- find a file that covers the time when GPFS is starting up > and should be mounting the filesystem. > > Look for clues and/or error messages. If you are not familiar, you may > also want to look at what messages are issued when everything is working > just right. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) -------------- next part -------------- An HTML attachment was scrubbed... URL: From Dan.Foster at bristol.ac.uk Mon Mar 23 08:43:33 2015 From: Dan.Foster at bristol.ac.uk (Dan Foster) Date: Mon, 23 Mar 2015 08:43:33 +0000 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: <20150320025655.GA2140@logos.ncsa.illinois.edu> Message-ID: Hi Richard, On 20 March 2015 at 20:32, Richard Lefebvre wrote: > One command I'm also looking for is a command to list which nsd server is > used for a particular nsd. I have a redundant config for which 2 servers can > access the same nsd. But I think that for some of them, the default server > is not used and the load of the servers is not distributed as wanted. Does the '-M' option to mmlsdisk provide the information you're looking for? i.e. from the man page: """ Displays whether I/O requests to the disk are satisfied on the local node, or using an NSD server. If the I/O is done using an NSD server, shows the NSD server name and the underlying disk name on that server node. """ It should display something similar to the following: Disk name IO performed on node Device Availability ------------ ----------------------- ----------------- ------------ nsd1 localhost /dev/disk1 up nsd2 node01.example.com /dev/disk2 up nsd3 localhost /dev/disk3 up HTH, Dan. > > 2015-03-20 15:24 GMT-04:00 Marc A Kaplan : >> >> Look in /var/adm/ras/mmfs.log.* -- on a node that should be mounting the >> filesystem -- find a file that covers the time when GPFS is starting up >> and should be mounting the filesystem. >> >> Look for clues and/or error messages. If you are not familiar, you may >> also want to look at what messages are issued when everything is working >> just right. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at gpfsug.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Dan Foster | Senior Storage Systems Administrator | IT Services e: dan.foster at bristol.ac.uk | t: 0117 3941170 [x41170] m: Advanced Computing Research Centre, University of Bristol, 8-10 Berkeley Square, Bristol BS8 1HH From PEREZJOS at uk.ibm.com Mon Mar 23 14:13:08 2015 From: PEREZJOS at uk.ibm.com (Jose Perez) Date: Mon, 23 Mar 2015 14:13:08 +0000 Subject: [gpfsug-discuss] Jose Perez is OoO. Message-ID: I will be out of the office starting 23/03/2015 and will not return until 24/03/2015. I will respond to your message when I return. For urgent issues please contact Ross Keeping, Thanks. Thanks, From chair at gpfsug.org Mon Mar 23 14:34:20 2015 From: chair at gpfsug.org (Jez Tucker (Chair)) Date: Mon, 23 Mar 2015 14:34:20 +0000 Subject: [gpfsug-discuss] Jose Perez is OoO. In-Reply-To: References: Message-ID: <5510246C.6090009@gpfsug.org> Apols for that one which slipped through, we'll add 'OoO' to the list blocker. Jez On 23/03/15 14:13, Jose Perez wrote: > I will be out of the office starting 23/03/2015 and will not return until > 24/03/2015. > > I will respond to your message when I return. For urgent issues please > contact Ross Keeping, Thanks. > Thanks, > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From makaplan at us.ibm.com Mon Mar 23 14:33:29 2015 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 23 Mar 2015 10:33:29 -0400 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: <20150320025655.GA2140@logos.ncsa.illinois.edu> Message-ID: mmlsnsd has several options... RTFM... but this one will show you which nodes can directly access and/or serve each NSD: mmlsnsd -M -v -M Maps the NSD names to its disk device name on all nodes. This is a slow operation and its usage is suggested for problem determination only. -v Specifies that the output should contain error information, where available. -------------- next part -------------- An HTML attachment was scrubbed... URL: From secretary at gpfsug.org Wed Mar 25 14:42:52 2015 From: secretary at gpfsug.org (Secretary GPFS UG) Date: Wed, 25 Mar 2015 14:42:52 +0000 Subject: [gpfsug-discuss] 2nd 'Meet the Devs' Event 28/4/15 Message-ID: <6715b1493f52d222ab24b49195ab8fc2@webmail.gpfsug.org> Dear members, Our second Meet the GPFS Developers 'coffee shop' meet is going to be taking place on Tuesday 28th April. The day will give you the opportunity to meet the developers and get hands on beta exploration and test of unreleased GPFS functions. Date: 28th April 2015 Time: 10:30 - 15:30 Location: IBM, 3rd Floor, Maybrook House, 40 Blackfriars Street, Manchester M3 2EG Agenda: 10:30 Introduction discussion 11:30 Beta session 1 12:30 Break for pizza (social chats with wider GPFS dev team in Manchester) 13:30 Beta session 2 14:45 Group discussion and feedback 15:30 End Lunch and refreshments will be provided. To attend, please register with Claire Robson, GPFS UG Secretary secretary at gpfsug.org. 10 places are available. Full directions and instructions will be sent to confirmed attendees. We look forward to seeing you in April. Thanks, Claire -- GPFS User Group Secretary From ross.keeping at uk.ibm.com Thu Mar 26 14:59:31 2015 From: ross.keeping at uk.ibm.com (Ross Keeping3) Date: Thu, 26 Mar 2015 14:59:31 +0000 Subject: [gpfsug-discuss] Potential talks for GPFS UG May 20th Message-ID: Hi There are a couple of talks we could include in the one day GPFS UG on May 20th. We'd appreciate some feedback on whether there is any interest or objection to including these. These are: 1) Tivoli Storage Productivity Center Technical Demo: A tool for managing heterogeneous storage infrastructures and specifically GPFS. Can read more here http://www-03.ibm.com/software/products/en/tivostorprodcent 2) GPFS for life sciences: A member of NERSC would like to talk about their efforts to improve GPFS file systems for a life sciences workload - specifically genomic sequencing and analysis that ensues. Details about life science workflow and requirements will be provided so the listener may understand the aspects of GPFS that are important. Cheers, Ross Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From stijn.deweirdt at ugent.be Thu Mar 26 15:07:29 2015 From: stijn.deweirdt at ugent.be (Stijn De Weirdt) Date: Thu, 26 Mar 2015 16:07:29 +0100 Subject: [gpfsug-discuss] Potential talks for GPFS UG May 20th In-Reply-To: References: Message-ID: <551420B1.3080002@ugent.be> no objections for any of them, but i'd prefer talk 2 (if a choice has to be made) stijn On 03/26/2015 03:59 PM, Ross Keeping3 wrote: > Hi > > There are a couple of talks we could include in the one day GPFS UG on May > 20th. > > We'd appreciate some feedback on whether there is any interest or > objection to including these. > > These are: > 1) Tivoli Storage Productivity Center Technical Demo: A tool for managing > heterogeneous storage infrastructures and specifically GPFS. Can read more > here http://www-03.ibm.com/software/products/en/tivostorprodcent > > 2) GPFS for life sciences: A member of NERSC would like to talk about > their efforts to improve GPFS file systems for a life sciences workload - > specifically genomic sequencing and analysis that ensues. Details about > life science workflow and requirements will be provided so the listener > may understand the aspects of GPFS that are important. > > Cheers, > Ross > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From robert at strubi.ox.ac.uk Thu Mar 26 15:13:44 2015 From: robert at strubi.ox.ac.uk (Robert Esnouf) Date: Thu, 26 Mar 2015 15:13:44 +0000 (GMT) Subject: [gpfsug-discuss] Potential talks for GPFS UG May 20th In-Reply-To: References: Message-ID: <201503261513.062218@mail.strubi.ox.ac.uk> Yes, unsurprisingly I'd be interested in the second talk! Thanks, Robert -- Dr. Robert Esnouf, University Research Lecturer, Head of Research Computing Core, NDM Research Computing Strategy Officer Room 10/028, Wellcome Trust Centre for Human Genetics, Old Road Campus, Roosevelt Drive, Oxford OX3 7BN, UK Email: robert at strubi.ox.ac.uk / robert at well.ox.ac.uk Tel: (+44) - 1865 - 287783 ---- Original message ---- >Date: Thu, 26 Mar 2015 14:59:31 +0000 >From: gpfsug-discuss-bounces at gpfsug.org (on behalf of Ross Keeping3 ) >Subject: [gpfsug-discuss] Potential talks for GPFS UG May 20th >To: gpfsug-discuss at gpfsug.org > > Hi > > There are a couple of talks we could include in the > one day GPFS UG on May 20th. > > We'd appreciate some feedback on whether there is > any interest or objection to including these. > > These are: > 1) Tivoli Storage Productivity Center Technical > Demo: A tool for managing heterogeneous storage > infrastructures and specifically GPFS. Can read more > here > http://www-03.ibm.com/software/products/en/tivostorprodcent > > 2) GPFS for life sciences: A member of NERSC would > like to talk about their efforts to improve GPFS > file systems for a life sciences workload - > specifically genomic sequencing and analysis that > ensues. Details about life science workflow and > requirements will be provided so the listener may > understand the aspects of GPFS that are important. > > Cheers, > Ross > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England > and Wales with number 741598. > Registered office: PO Box 41, North Harbour, > Portsmouth, Hampshire PO6 3AU >________________ >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at gpfsug.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Mark.Roberts at awe.co.uk Thu Mar 26 15:24:17 2015 From: Mark.Roberts at awe.co.uk (Mark.Roberts at awe.co.uk) Date: Thu, 26 Mar 2015 15:24:17 +0000 Subject: [gpfsug-discuss] EXTERNAL: Potential talks for GPFS UG May 20th In-Reply-To: References: Message-ID: <201503261524.t2QFOQrO020615@msw1.awe.co.uk> As well as supporting the efforts of our friends at NERSC I think should be a good interesting talk. Mark Roberts ________________________________ From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Ross Keeping3 Sent: 26 March 2015 15:00 To: gpfsug-discuss at gpfsug.org Subject: EXTERNAL: [gpfsug-discuss] Potential talks for GPFS UG May 20th Hi There are a couple of talks we could include in the one day GPFS UG on May 20th. We'd appreciate some feedback on whether there is any interest or objection to including these. These are: 1) Tivoli Storage Productivity Center Technical Demo: A tool for managing heterogeneous storage infrastructures and specifically GPFS. Can read more here http://www-03.ibm.com/software/products/en/tivostorprodcent 2) GPFS for life sciences: A member of NERSC would like to talk about their efforts to improve GPFS file systems for a life sciences workload - specifically genomic sequencing and analysis that ensues. Details about life science workflow and requirements will be provided so the listener may understand the aspects of GPFS that are important. Cheers, Ross Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU ___________________________________________________ ____________________________ The information in this email and in any attachment(s) is commercial in confidence. If you are not the named addressee(s) or if you receive this email in error then any distribution, copying or use of this communication or the information in it is strictly prohibited. Please notify us immediately by email at admin.internet(at)awe.co.uk, and then delete this message from your computer. While attachments are virus checked, AWE plc does not accept any liability in respect of any virus which is not detected. AWE Plc Registered in England and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR -------------- next part -------------- An HTML attachment was scrubbed... URL: From Sandra.McLaughlin at astrazeneca.com Thu Mar 26 15:56:16 2015 From: Sandra.McLaughlin at astrazeneca.com (McLaughlin, Sandra M) Date: Thu, 26 Mar 2015 15:56:16 +0000 Subject: [gpfsug-discuss] EXTERNAL: Potential talks for GPFS UG May 20th In-Reply-To: <201503261524.t2QFOQrO020615@msw1.awe.co.uk> References: <201503261524.t2QFOQrO020615@msw1.awe.co.uk> Message-ID: Life Sciences talk would be spot on for me too, please. Sandra McLaughlin From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Mark.Roberts at awe.co.uk Sent: 26 March 2015 15:24 To: gpfsug-discuss at gpfsug.org Subject: Re: [gpfsug-discuss] EXTERNAL: Potential talks for GPFS UG May 20th As well as supporting the efforts of our friends at NERSC I think should be a good interesting talk. Mark Roberts ________________________________ From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Ross Keeping3 Sent: 26 March 2015 15:00 To: gpfsug-discuss at gpfsug.org Subject: EXTERNAL: [gpfsug-discuss] Potential talks for GPFS UG May 20th Hi There are a couple of talks we could include in the one day GPFS UG on May 20th. We'd appreciate some feedback on whether there is any interest or objection to including these. These are: 1) Tivoli Storage Productivity Center Technical Demo: A tool for managing heterogeneous storage infrastructures and specifically GPFS. Can read more here http://www-03.ibm.com/software/products/en/tivostorprodcent 2) GPFS for life sciences: A member of NERSC would like to talk about their efforts to improve GPFS file systems for a life sciences workload - specifically genomic sequencing and analysis that ensues. Details about life science workflow and requirements will be provided so the listener may understand the aspects of GPFS that are important. Cheers, Ross Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU ___________________________________________________ ____________________________ The information in this email and in any attachment(s) is commercial in confidence. If you are not the named addressee(s) or if you receive this email in error then any distribution, copying or use of this communication or the information in it is strictly prohibited. Please notify us immediately by email at admin.internet(at)awe.co.uk, and then delete this message from your computer. While attachments are virus checked, AWE plc does not accept any liability in respect of any virus which is not detected. AWE Plc Registered in England and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR ________________________________ AstraZeneca UK Limited is a company incorporated in England and Wales with registered number: 03674842 and a registered office at 2 Kingdom Street, London, W2 6BD. Confidentiality Notice: This message is private and may contain confidential, proprietary and legally privileged information. If you have received this message in error, please notify us and remove it from your system and note that you must not copy, distribute or take any action in reliance on it. Any unauthorised use or disclosure of the contents of this message is not permitted and may be unlawful. Disclaimer: Email messages may be subject to delays, interception, non-delivery and unauthorised alterations. Therefore, information expressed in this message is not given or endorsed by AstraZeneca UK Limited unless otherwise notified by an authorised representative independent of this message. No contractual relationship is created by this message by any person unless specifically indicated by agreement in writing other than email. Monitoring: AstraZeneca UK Limited may monitor email traffic data and content for the purposes of the prevention and detection of crime, ensuring the security of our computer systems and checking compliance with our Code of Conduct and policies. -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.kidger at uk.ibm.com Thu Mar 26 16:06:24 2015 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Thu, 26 Mar 2015 16:06:24 +0000 Subject: [gpfsug-discuss] Potential talks for GPFS UG May 20th In-Reply-To: <201503261513.062218@mail.strubi.ox.ac.uk> References: <201503261513.062218@mail.strubi.ox.ac.uk> Message-ID: I too would want to upvote the second. I am seeing lots of interest in the Life Sciences particularly for things like large Genomic datasets and NGS. Daniel Dr.Daniel Kidger No. 1 The Square, Technical Specialist SDI (formerly Platform Computing) Temple Quay, Bristol BS1 6DG Mobile: +44-07818 522 266 United Kingdom Landline: +44-02392 564 121 (Internal ITN 3726 9250) e-mail: daniel.kidger at uk.ibm.com From: Robert Esnouf To: "gpfsug main discussion list" Date: 26/03/2015 15:13 Subject: Re: [gpfsug-discuss] Potential talks for GPFS UG May 20th Sent by: gpfsug-discuss-bounces at gpfsug.org Yes, unsurprisingly I'd be interested in the second talk! Thanks, Robert -- Dr. Robert Esnouf, University Research Lecturer, Head of Research Computing Core, NDM Research Computing Strategy Officer Room 10/028, Wellcome Trust Centre for Human Genetics, Old Road Campus, Roosevelt Drive, Oxford OX3 7BN, UK Email: robert at strubi.ox.ac.uk / robert at well.ox.ac.uk Tel: (+44) - 1865 - 287783 ---- Original message ---- >Date: Thu, 26 Mar 2015 14:59:31 +0000 >From: gpfsug-discuss-bounces at gpfsug.org (on behalf of Ross Keeping3 ) >Subject: [gpfsug-discuss] Potential talks for GPFS UG May 20th >To: gpfsug-discuss at gpfsug.org > > Hi > > There are a couple of talks we could include in the > one day GPFS UG on May 20th. > > We'd appreciate some feedback on whether there is > any interest or objection to including these. > > These are: > 1) Tivoli Storage Productivity Center Technical > Demo: A tool for managing heterogeneous storage > infrastructures and specifically GPFS. Can read more > here > http://www-03.ibm.com/software/products/en/tivostorprodcent > > 2) GPFS for life sciences: A member of NERSC would > like to talk about their efforts to improve GPFS > file systems for a life sciences workload - > specifically genomic sequencing and analysis that > ensues. Details about life science workflow and > requirements will be provided so the listener may > understand the aspects of GPFS that are important. > > Cheers, > Ross > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England > and Wales with number 741598. > Registered office: PO Box 41, North Harbour, > Portsmouth, Hampshire PO6 3AU >________________ >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at gpfsug.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 360 bytes Desc: not available URL: From zgiles at gmail.com Thu Mar 26 16:14:36 2015 From: zgiles at gmail.com (Zachary Giles) Date: Thu, 26 Mar 2015 12:14:36 -0400 Subject: [gpfsug-discuss] EXTERNAL: Potential talks for GPFS UG May 20th In-Reply-To: References: <201503261524.t2QFOQrO020615@msw1.awe.co.uk> Message-ID: It's at the GPFSUG in the UK. Looks like the UK GPFSUG is much more active than the US one's. They've had a few meetups actually. I'd fly out there and attend them, but I figured probably it wouldn't count since it's not a real conf. This on is in May 20th. On Thu, Mar 26, 2015 at 11:56 AM, McLaughlin, Sandra M < Sandra.McLaughlin at astrazeneca.com> wrote: > Life Sciences talk would be spot on for me too, please. > > > > Sandra McLaughlin > > > > *From:* gpfsug-discuss-bounces at gpfsug.org [mailto: > gpfsug-discuss-bounces at gpfsug.org] *On Behalf Of *Mark.Roberts at awe.co.uk > *Sent:* 26 March 2015 15:24 > *To:* gpfsug-discuss at gpfsug.org > *Subject:* Re: [gpfsug-discuss] EXTERNAL: Potential talks for GPFS UG May > 20th > > > > As well as supporting the efforts of our friends at NERSC I think should > be a good interesting talk. > > > > Mark Roberts > > > ------------------------------ > > *From:* gpfsug-discuss-bounces at gpfsug.org [ > mailto:gpfsug-discuss-bounces at gpfsug.org > ] *On Behalf Of *Ross Keeping3 > *Sent:* 26 March 2015 15:00 > *To:* gpfsug-discuss at gpfsug.org > *Subject:* EXTERNAL: [gpfsug-discuss] Potential talks for GPFS UG May 20th > > > > Hi > > There are a couple of talks we could include in the one day GPFS UG on May > 20th. > > We'd appreciate some feedback on whether there is any interest or > objection to including these. > > These are: > 1) Tivoli Storage Productivity Center Technical Demo: A tool for managing > heterogeneous storage infrastructures and specifically GPFS. Can read more > here http://www-03.ibm.com/software/products/en/tivostorprodcent > > 2) GPFS for life sciences: A member of NERSC would like to talk about > their efforts to improve GPFS file systems for a life sciences workload - > specifically genomic sequencing and analysis that ensues. Details about > life science workflow and requirements will be provided so the listener may > understand the aspects of GPFS that are important. > > Cheers, > Ross > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > ___________________________________________________ > ____________________________ The information in this email and in any > attachment(s) is commercial in confidence. If you are not the named > addressee(s) or if you receive this email in error then any distribution, > copying or use of this communication or the information in it is strictly > prohibited. Please notify us immediately by email at admin.internet(at) > awe.co.uk, and then delete this message from your computer. While > attachments are virus checked, AWE plc does not accept any liability in > respect of any virus which is not detected. AWE Plc Registered in England > and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR > ------------------------------ > > AstraZeneca UK Limited is a company incorporated in England and Wales with > registered number: 03674842 and a registered office at 2 Kingdom Street, > London, W2 6BD. > > *Confidentiality Notice: *This message is private and may contain > confidential, proprietary and legally privileged information. If you have > received this message in error, please notify us and remove it from your > system and note that you must not copy, distribute or take any action in > reliance on it. Any unauthorised use or disclosure of the contents of this > message is not permitted and may be unlawful. > > *Disclaimer:* Email messages may be subject to delays, interception, > non-delivery and unauthorised alterations. Therefore, information expressed > in this message is not given or endorsed by AstraZeneca UK Limited unless > otherwise notified by an authorised representative independent of this > message. No contractual relationship is created by this message by any > person unless specifically indicated by agreement in writing other than > email. > > *Monitoring: *AstraZeneca UK Limited may monitor email traffic data and > content for the purposes of the prevention and detection of crime, ensuring > the security of our computer systems and checking compliance with our Code > of Conduct and policies. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- Zach Giles zgiles at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From luke.raimbach at oerc.ox.ac.uk Fri Mar 27 12:16:16 2015 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Fri, 27 Mar 2015 12:16:16 +0000 Subject: [gpfsug-discuss] mmunlinkfileset problems Message-ID: Hi All, I'm having a problem where unlinking a fileset is taking 7 or 8 minutes to complete. We are running 3.5.0.22 and the cluster is small (9 nodes: 8 Linux plus 1 Windows node), one file system remote mounted by a 3-node Linux cluster, and generally not very busy. I wouldn't expect mmunlinkfileset to require the file system to quiesce (and even if it did 8 minutes would seem like an awfully long time). Previous unlink commands (on version 3.5.0.19) have returned within milliseconds. This very long running command appeared to pause IO for a sufficiently long time that cNFS clients timed out too and various services elsewhere crashed. Is there any reason why unlinking a fileset would take this long? Also, I can't find anywhere a list of commands which might require the FS to quiesce - is this information available somewhere? Cheers, Luke. -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 From jonathan at buzzard.me.uk Fri Mar 27 12:49:38 2015 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Fri, 27 Mar 2015 12:49:38 +0000 Subject: [gpfsug-discuss] mmunlinkfileset problems In-Reply-To: References: Message-ID: <1427460578.3553.153.camel@buzzard.phy.strath.ac.uk> On Fri, 2015-03-27 at 12:16 +0000, Luke Raimbach wrote: > Hi All, > > I'm having a problem where unlinking a fileset is taking 7 or 8 minutes > to complete. We are running 3.5.0.22 and the cluster is small (9 nodes: > 8 Linux plus 1 Windows node), one file system remote mounted by a > 3-node Linux cluster, and generally not very busy. > > I wouldn't expect mmunlinkfileset to require the file system to quiesce > (and even if it did 8 minutes would seem like an awfully long time). > Previous unlink commands (on version 3.5.0.19) have returned within > milliseconds. If there is a waiter around mmunlinkfileset can block even with an empty fileset and turn your file system unresponsive. > > This very long running command appeared to pause IO for a sufficiently > long time that cNFS clients timed out too and various services > elsewhere crashed. > Yep, lesson for the day only unlink filesets when the file system is quiet, aka a maintenance window. > Is there any reason why unlinking a fileset would take this long? Also, > I can't find anywhere a list of commands which might require the FS to > quiesce - is this information available somewhere? > Deleting snapshots, at least in the past. It might be that guidance from IBM has changed but once you have been stung by this you kind of thing you don't like to try it again any time soon. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From luke.raimbach at oerc.ox.ac.uk Fri Mar 27 13:27:20 2015 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Fri, 27 Mar 2015 13:27:20 +0000 Subject: [gpfsug-discuss] mmunlinkfileset problems In-Reply-To: <1427460578.3553.153.camel@buzzard.phy.strath.ac.uk> References: <1427460578.3553.153.camel@buzzard.phy.strath.ac.uk> Message-ID: > > Is there any reason why unlinking a fileset would take this long? > > Also, I can't find anywhere a list of commands which might require the > > FS to quiesce - is this information available somewhere? > > > > Deleting snapshots, at least in the past. It might be that guidance from IBM > has changed but once you have been stung by this you kind of thing you > don't like to try it again any time soon. Precisely why I'm after a list of commands that might require quiescence! I know that snapshot operations (generally) and deleting filesets require a consistent view so will pause IO, but unlinking (in my mind) wouldn't need this... I'd probably want to generate lists of 'cleanup' commands that might interrupt IO and run these in maintenance windows as you say. Gather up all the filesets that want deleting and do them once a week / whenever. Cheers, Luke. From jonathan at buzzard.me.uk Fri Mar 27 13:39:19 2015 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Fri, 27 Mar 2015 13:39:19 +0000 Subject: [gpfsug-discuss] mmunlinkfileset problems In-Reply-To: References: <1427460578.3553.153.camel@buzzard.phy.strath.ac.uk> Message-ID: <1427463559.3553.156.camel@buzzard.phy.strath.ac.uk> On Fri, 2015-03-27 at 13:27 +0000, Luke Raimbach wrote: > > > Is there any reason why unlinking a fileset would take this long? > > > Also, I can't find anywhere a list of commands which might require the > > > FS to quiesce - is this information available somewhere? > > > > > > > Deleting snapshots, at least in the past. It might be that guidance from IBM > > has changed but once you have been stung by this you kind of thing you > > don't like to try it again any time soon. > > Precisely why I'm after a list of commands that might require > quiescence! I know that snapshot operations (generally) and deleting > filesets require a consistent view so will pause IO, but unlinking (in > my mind) wouldn't need this... > mmdeldisk is another. From what I can make out anything that takes away from the file system is a potential problem. Anything that adds to the file system is just peachy fine to be run at any time. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From adam.huffman at crick.ac.uk Mon Mar 2 09:24:59 2015 From: adam.huffman at crick.ac.uk (Adam Huffman) Date: Mon, 2 Mar 2015 09:24:59 +0000 Subject: [gpfsug-discuss] Introduction Message-ID: Hello A couple of weeks ago I joined Bruno Silva?s HPC team at the Francis Crick Institute, with special responsibility for HPC, OpenStack and virtualization. I?m very much a GPFS novice so I?m hoping to be able to draw on the knowledge in this group, while hopefully being able to help others with OpenStack. As Bruno stated in his message, we?re particularly interested in how to present GPFS to instances securely. I?ve read the discussion from November on this list, which didn?t seem to come to any firm conclusions. Has anyone involved then made substantial progress since? Cheers, Adam ? Adam Huffman Senior HPC & Virtualization Systems Engineer The Francis Crick Institute Gibbs Building 215 Euston Road London NW1 2BE T: E: adam.huffman at crick.ac.uk W: www.crick.ac.uk [Description: Description: ZA102637861][Description: Description: Twitter] The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no.06885462, with its registered office at 215 Euston Road, London NW1 2BE -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 158254D2-CDA8-43E5-96C1-A0AAEB10314B.png Type: image/png Size: 1151 bytes Desc: 158254D2-CDA8-43E5-96C1-A0AAEB10314B.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: B09B8AB7-5CA2-4DE7-BFE7-9D8F1B2A5166.png Type: image/png Size: 1211 bytes Desc: B09B8AB7-5CA2-4DE7-BFE7-9D8F1B2A5166.png URL: From viccornell at gmail.com Mon Mar 2 09:32:57 2015 From: viccornell at gmail.com (Vic Cornell) Date: Mon, 2 Mar 2015 09:32:57 +0000 Subject: [gpfsug-discuss] Introduction In-Reply-To: References: Message-ID: Hi Adam, I guess that one of the things that would help push it forward is a definition of what "secure" means to you. Regards, Vic On 2 Mar 2015, at 09:24, Adam Huffman wrote: > Hello > > A couple of weeks ago I joined Bruno Silva?s HPC team at the Francis Crick Institute, with special responsibility for HPC, OpenStack and virtualization. I?m very much a GPFS novice so I?m hoping to be able to draw on the knowledge in this group, while hopefully being able to help others with OpenStack. > > As Bruno stated in his message, we?re particularly interested in how to present GPFS to instances securely. I?ve read the discussion from November on this list, which didn?t seem to come to any firm conclusions. Has anyone involved then made substantial progress since? > > Cheers, > Adam > > ? > > Adam Huffman > Senior HPC & Virtualization Systems Engineer > The Francis Crick Institute > Gibbs Building > 215 Euston Road > London NW1 2BE > > T: > E: adam.huffman at crick.ac.uk > W: www.crick.ac.uk > <158254D2-CDA8-43E5-96C1-A0AAEB10314B.png> > > The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no.06885462, with its registered office at 215 Euston Road, London NW1 2BE > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam.huffman at crick.ac.uk Mon Mar 2 09:39:36 2015 From: adam.huffman at crick.ac.uk (Adam Huffman) Date: Mon, 2 Mar 2015 09:39:36 +0000 Subject: [gpfsug-discuss] Introduction In-Reply-To: References: Message-ID: Hi Vic, Re-emphasising that I?m still very much learning about GPFS, one of the approaches being discussed is running the GPFS client inside the instances. The concern here is over the case where users have root privileges inside their instances (a pretty common assumption for those used to AWS, for example) and the implications this may have for GPFS. Does it mean there would be a risk of spoofing? Cheers, Adam From: Vic Cornell Reply-To: gpfsug main discussion list Date: Monday, 2 March 2015 09:32 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Introduction Hi Adam, I guess that one of the things that would help push it forward is a definition of what "secure" means to you. Regards, Vic On 2 Mar 2015, at 09:24, Adam Huffman > wrote: Hello A couple of weeks ago I joined Bruno Silva?s HPC team at the Francis Crick Institute, with special responsibility for HPC, OpenStack and virtualization. I?m very much a GPFS novice so I?m hoping to be able to draw on the knowledge in this group, while hopefully being able to help others with OpenStack. As Bruno stated in his message, we?re particularly interested in how to present GPFS to instances securely. I?ve read the discussion from November on this list, which didn?t seem to come to any firm conclusions. Has anyone involved then made substantial progress since? Cheers, Adam ? Adam Huffman Senior HPC & Virtualization Systems Engineer The Francis Crick Institute Gibbs Building 215 Euston Road London NW1 2BE T: E: adam.huffman at crick.ac.uk W: www.crick.ac.uk <158254D2-CDA8-43E5-96C1-A0AAEB10314B.png> The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no.06885462, with its registered office at 215 Euston Road, London NW1 2BE _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From luke.raimbach at oerc.ox.ac.uk Mon Mar 2 19:21:25 2015 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Mon, 2 Mar 2015 19:21:25 +0000 Subject: [gpfsug-discuss] Introduction In-Reply-To: References: Message-ID: HI Adam, We run virtualised GPFS client nodes in a VMware cluster here at Oxford e-Research Centre. We had a requirement where one research group wanted root access to their VMs but also wanted fast direct access to their data on our GPFS cluster. The technical setup was (relatively) simple. We span up a small three-node virtual GPFS cluster (with no file system of its own). We then used the multi-cluster feature of GPFS to allow this small virtual cluster to join our main file system cluster which now gives that group very good IO performance. However, the problem of spoofing you mention is relevant ? we installed the virtual cluster nodes for the research group and put our Active Directory client on them. We also used the root-squash configuration option of the multi-cluster setup to prevent remote-cluster root access in to our file system. We also agreed with the research group that they would nominate one Administrator to have root access in their cluster and that they would maintain the AAA framework we put in place. We have to trust the group?s Administrator not to fiddle his UID or to let others escalate their privileges. If you were letting untrusted root users spin up Stacks, then you could still run GPFS clients in the OpenStack instance nodes to give them fast access to their data. Here are some musings on a recipe (others please feel free to pull these ideas to pieces): 1. Start with Cluster A ? your main production GPFS file system. It has GPFS device name /gpfs. 2. Pretend you have lots of money for extra disk to go under your OpenStack cluster (say you buy something like a DDN SFA7700 with a couple of expansion shelves and fill it up with 4TB drives ? 180 drives). 3. Drive this disk array with two, preferably four (or however many you want, really) decent NSD servers. Configure quorum nodes, etc. appropriately. Call this Cluster B. 4. Carve up the disk array in to something like 30 x RAID6 (4 + 2) LUNs and configure them as GPFS NSDs; but don?t create a file system (line up the stripe sizes etc. and choose a nice block size, etc. etc.)? 5. Put the GPFS metadata on some SSD NSDs somewhere. I like putting it on SSDs in the NSD server nodes and replicating it. Other people like putting it in their disk arrays. 6. As part of someone spinning up a Stack, get some scripts to do the following ?magic?: a. Connect to Cluster A and find out how big their target data-set is. b. Connect to Cluster B and create a new GPFS file system with a reasonable (dependent on the above result) number of NSD disks. Call this new GPFS device something unique other that /gpfs e.g. /gpfs0001. You could slice bits off your SSDs for the metadata NSDs in each file system you create in this manner (if you haven?t got many SSDs). c. As part of a new Stack, provide a few (three, say) GPFS quorum nodes that you?ve configured. Call this Cluster C. Add the rest of the stack instances to Cluster C. No File System. d. Pop back over to Cluster A. Export their target data-set from Cluster A using AFM (over GPFS or NFS ? pick your favourite: GPFS is probably better performant but means you need Cluster A to stay online). e. Now return to Cluster B. Import the target data to a local AFM cache on Cluster B?s new file system. Name the AFM file-set whatever you like, but link it in to the Cluster B /gpfs0001 namespace at the same level as it is in Cluster A. For example Cluster A: /gpfs/projects/dataset01 imports to an AFM fileset in Cluster B named userdataset01. Link this under /gpfs0001/projects/dataset01. f. Configure multi-cluster support on Cluster B to export GPFS device /gpfs0001 to Cluster C. Encrypt traffic if you want a headache. g. Configure multi-cluster support on Cluster C to remote mount Cluster B:/gpfs0001 as local device /gpfs. 7. You now have fast GPFS access to this user dataset *only* using GPFS clients inside the OpenStack instance nodes. You have also preserved the file system namespace in Cluster C?s instance nodes. If you only want to run over the data in the stack instances, you could pre-fetch the entire data-set using AFM Control from Cluster A in to the Cluster B file-set (if it?s big enough). 8. Now your users are finished and want to destroy the stack ? you need some more script ?magic?: a. Dismount the file system /gpfs in Cluster C. b. Connect to Cluster B and use AFM control to flush all the data back home to Cluster A. c. Unlink the file-set in Cluster B and force delete it; then delete the file system to free the NSDs back to the pool available to Cluster B. d. Connect back to Cluster A and unexport the original data-set directory structure. e. Throw away the VMs in the stack Things to worry about: ? Inode space will be different if users happen to be working on the data in Cluster A and Cluster C and want to know about inodes. GPFS XATTRS are preserved. ? If you use AFM over NFS because Cluster A and B are far away from each other and laggy, then there?s no locking with your AFM cache running as an Independent Writer. Writes at home (Cluster A) and in cache (Cluster B from Cluster C) will be nondeterministic. Your users will need to know this to avoid disappointment. ? If you use AFM over GPFS because Cluster A and B are near to each other and have a fast network, then there might still not be locking, but if Cluster A goes offline, it will put the AFM cache in to a ?dismounted? state. ? If your users want access to other parts of the Cluster A /gpfs namespace within their stack instances (because you have tools they use or they want to see other stuff), you can export them as read-only to a read-only AFM cache in Cluster B and they will be able to see things in Cluster C provided you link the AFM caches in the right places. Remember they have root access here. ? AFM updates are sent from cache to home as root so users can potentially overflow their quota at the Cluster A home site (root doesn?t care about quotas at home). ? Other frightening things might happen that I?ve not thought about. Hope this helps! Luke -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Adam Huffman Sent: 02 March 2015 09:40 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Introduction Hi Vic, Re-emphasising that I?m still very much learning about GPFS, one of the approaches being discussed is running the GPFS client inside the instances. The concern here is over the case where users have root privileges inside their instances (a pretty common assumption for those used to AWS, for example) and the implications this may have for GPFS. Does it mean there would be a risk of spoofing? Cheers, Adam From: Vic Cornell Reply-To: gpfsug main discussion list Date: Monday, 2 March 2015 09:32 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Introduction Hi Adam, I guess that one of the things that would help push it forward is a definition of what "secure" means to you. Regards, Vic On 2 Mar 2015, at 09:24, Adam Huffman > wrote: Hello A couple of weeks ago I joined Bruno Silva?s HPC team at the Francis Crick Institute, with special responsibility for HPC, OpenStack and virtualization. I?m very much a GPFS novice so I?m hoping to be able to draw on the knowledge in this group, while hopefully being able to help others with OpenStack. As Bruno stated in his message, we?re particularly interested in how to present GPFS to instances securely. I?ve read the discussion from November on this list, which didn?t seem to come to any firm conclusions. Has anyone involved then made substantial progress since? Cheers, Adam ? Adam Huffman Senior HPC & Virtualization Systems Engineer The Francis Crick Institute Gibbs Building 215 Euston Road London NW1 2BE T: E: adam.huffman at crick.ac.uk W: www.crick.ac.uk <158254D2-CDA8-43E5-96C1-A0AAEB10314B.png> The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no.06885462, with its registered office at 215 Euston Road, London NW1 2BE _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger_eriksson at se.ibm.com Thu Mar 12 08:25:22 2015 From: roger_eriksson at se.ibm.com (Roger Eriksson) Date: Thu, 12 Mar 2015 09:25:22 +0100 Subject: [gpfsug-discuss] GPFS UG 2015 date potential alteration In-Reply-To: <54EDD5E8.7090905@gpfsug.org> References: <54EDD5E8.7090905@gpfsug.org> Message-ID: Hi, has the GPFS User Group Meeting date been settled yet ? //Roger _________________________________________________________________ H?lsningar / Regards __________________________________________________________________ Roger Eriksson IBM Systems Lab Services Senior Accredited Product Services Professional XIV / Flashsystems / SVC / Storwize / SONAS / SAN EMEA Storage Competence Center (ESCC), Mainz, Germany IBM Svenska AB Isafjordsgatan 1 SE-164 92 Stockholm, Sweden Mobile +46(0)707933518 Email: roger_eriksson at se.ibm.com Notes: Roger Eriksson/Sweden/IBM at IBMSE Need help from EMEA Storage Competence Center ? IBMer ESCC @ Intranet --> http://escc.mainz.de.ibm.com/escc4u BPs ESCC @ Internet --> http://www.ibm.com/partnerworld/page/pw_com_bp_ecampus From: "Jez Tucker (Chair)" To: gpfsug main discussion list Date: 25/02/2015 15:02 Subject: [gpfsug-discuss] GPFS UG 2015 date potential alteration Sent by: gpfsug-discuss-bounces at gpfsug.org Alteration. Due to a scheduling issue, we may have to potentially move the date from 1st May. We will confirm this as soon as possible. Apologies. Jez _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss S?vida annat inte anges ovan: / Unless stated otherwise above: IBM Svenska AB Organisationsnummer: 556026-6883 Adress: 164 92 Stockholm -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 7526 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 7414 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 8944 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 9733 bytes Desc: not available URL: From chair at gpfsug.org Thu Mar 12 16:24:52 2015 From: chair at gpfsug.org (Jez Tucker (Chair)) Date: Thu, 12 Mar 2015 16:24:52 +0000 Subject: [gpfsug-discuss] GPFS UG 2015 date potential alteration In-Reply-To: References: <54EDD5E8.7090905@gpfsug.org> Message-ID: <5501BDD4.5060506@gpfsug.org> Hello all A fair question. We are working to a date of Wednesday 20th May. So please pencil this. The last pieces of paper work are being put in place, so please don't anybody book flights / expenses until this has been done. I would like to be in a position to have a definitive update on Monday. Jez On 12/03/15 08:25, Roger Eriksson wrote: > Hi, > > has the GPFS User Group Meeting date been settled yet ? > > //Roger > */_________________________________________________________________/* > */H?lsningar / Regards/* > */__________________________________________________________________/* > */Roger Eriksson/* > IBM Systems Lab Services > Senior Accredited Product Services Professional > XIV / Flashsystems / SVC / Storwize / SONAS / SAN > EMEA Storage Competence Center (ESCC), Mainz, Germany > IBM Svenska AB > Isafjordsgatan 1 > SE-164 92 Stockholm, Sweden > Mobile +46(0)707933518 > Email: _roger_eriksson at se.ibm.com_ > Notes: Roger Eriksson/Sweden/IBM at IBMSE > > > *Need help from EMEA Storage Competence Center ? * > *IBMer ESCC @ Intranet --> **http://escc.mainz.de.ibm.com/escc4u* > *BPs ESCC @ Internet --> > **http://www.ibm.com/partnerworld/page/pw_com_bp_ecampus* > > > > From: "Jez Tucker (Chair)" > To: gpfsug main discussion list > Date: 25/02/2015 15:02 > Subject: [gpfsug-discuss] GPFS UG 2015 date potential alteration > Sent by: gpfsug-discuss-bounces at gpfsug.org > ------------------------------------------------------------------------ > > > > Alteration. > > Due to a scheduling issue, we may have to potentially move the date > from 1st May. > > We will confirm this as soon as possible. > > Apologies. > > Jez > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > S?vida annat inte anges ovan: / Unless stated otherwise above: > IBM Svenska AB > Organisationsnummer: 556026-6883 > Adress: 164 92 Stockholm > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 7526 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 7414 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 8944 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 9733 bytes Desc: not available URL: From luke.raimbach at oerc.ox.ac.uk Fri Mar 13 14:44:33 2015 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Fri, 13 Mar 2015 14:44:33 +0000 Subject: [gpfsug-discuss] GPFS Journal Logs Message-ID: Hi, I'm interested in knowing more about the internal log journals GPFS uses to track file system activity from client nodes. My understanding is that when a file is opened for writing on a node, that (first) node becomes the 'metanode' for the file and is responsible for making updates to the file's metadata. Does the matanode also have responsibility for tracking changes made by other nodes that open the same file in its own journal, or are these recorded in each nodes own journal? I'm also interested in what happens when a node fails and its logs are replayed by the FS manager... what happens if that node is acting as metanode for a lot of open files which are also open elsewhere for writing on other nodes? If in the above scenario the metanode is handling metadata updates and tracking other changes in its journal, do all other nodes need to wait for the metanode to be ejected and have its journal replayed before they can continue? What is the process for selecting a successor metanode for the open files? Another related question I have is about initial sizing when creating a file system. Presumably the estimate for the total number of nodes mounting the file system affects the structure / allocation of internal journal logs? How are these arranged and is there any advice to overestimate this number if you intend to add more nodes at a future time? In the same vein, do multicluster nodes that join another remote stripegroup get allocated internal journal space as if they were new nodes in the remote cluster? Are these nodes allowed to be metanodes for files in a remote cluster? What happens when these nodes fail? Any insights greatly appreciated! Cheers, Luke. -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 From richard.lefebvre at calculquebec.ca Fri Mar 13 18:20:14 2015 From: richard.lefebvre at calculquebec.ca (Richard Lefebvre) Date: Fri, 13 Mar 2015 14:20:14 -0400 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. Message-ID: Hi, I have a GPFS cluster (v3.5). Last month, one of the filesystem had a problem with one of it's NSD. With the help tech support, the problem was fix and everything is OK data wise. The thing is when the problem occurred the filesystem was offline and once the filesystem was back to being OK, we were able to remount it. The problem is that now, every time a node is restarted, the filesystem is not mounted automatically as before. The other filesystems mount automatically. I did a mmlsfs on the file system and -A is at yes. Can someone tell my what I'm missing here. I'm sure it is something simple, but I'm drawing a blank right now. Richard -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) -------------- next part -------------- An HTML attachment was scrubbed... URL: From chair at gpfsug.org Mon Mar 16 14:02:09 2015 From: chair at gpfsug.org (Jez Tucker (Chair)) Date: Mon, 16 Mar 2015 14:02:09 +0000 Subject: [gpfsug-discuss] GPFS UG Meeting Agenda Message-ID: <5506E261.8000403@gpfsug.org> Hello all We have an internal agenda for the forthcoming UG. Before we publish this, I thought it prudent to solicit requests from our user base so that we might not miss anything important. So; - What's your burning issue, what's the one thing you really want to see? Regards, Jez From janfrode at tanso.net Mon Mar 16 15:11:13 2015 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Mon, 16 Mar 2015 16:11:13 +0100 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: Message-ID: Not much details to go by here, but I know I've had problems with SElinux blocking for getting filesystems mounted automatically at boot, while manually mounting afterwards worked fine. I fixed that by disabling automatic startup of gpfs from standard initscripts, and used this in rc.local instead: # make sure GPFS is not confined by selinux policies: runcon -t unconfined_t /usr/lpp/mmfs/bin/mmstartup -jf On Fri, Mar 13, 2015 at 7:20 PM, Richard Lefebvre < richard.lefebvre at calculquebec.ca> wrote: > Hi, > > I have a GPFS cluster (v3.5). Last month, one of the filesystem had a > problem with one of it's NSD. With the help tech support, the problem was > fix and everything is OK data wise. The thing is when the problem occurred > the filesystem was offline and once the filesystem was back to being OK, we > were able to remount it. The problem is that now, every time a node is > restarted, the filesystem is not mounted automatically as before. The other > filesystems mount automatically. I did a mmlsfs on the file system and -A > is at yes. Can someone tell my what I'm missing here. I'm sure it is > something simple, but I'm drawing a blank right now. > > Richard > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From YARD at il.ibm.com Mon Mar 16 15:14:25 2015 From: YARD at il.ibm.com (Yaron Daniel) Date: Mon, 16 Mar 2015 17:14:25 +0200 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: Message-ID: Hi In case of Linux - Check also : #chkconfig --list |grep gpfs Regards Yaron Daniel 94 Em Ha'Moshavot Rd Server, Storage and Data Services - Team Leader Petach Tiqva, 49527 Global Technology Services Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com IBM Israel From: Jan-Frode Myklebust To: gpfsug main discussion list Date: 03/16/2015 05:11 PM Subject: Re: [gpfsug-discuss] GPFS file system won't mount automatically anymore. Sent by: gpfsug-discuss-bounces at gpfsug.org Not much details to go by here, but I know I've had problems with SElinux blocking for getting filesystems mounted automatically at boot, while manually mounting afterwards worked fine. I fixed that by disabling automatic startup of gpfs from standard initscripts, and used this in rc.local instead: # make sure GPFS is not confined by selinux policies: runcon -t unconfined_t /usr/lpp/mmfs/bin/mmstartup -jf On Fri, Mar 13, 2015 at 7:20 PM, Richard Lefebvre < richard.lefebvre at calculquebec.ca> wrote: Hi, I have a GPFS cluster (v3.5). Last month, one of the filesystem had a problem with one of it's NSD. With the help tech support, the problem was fix and everything is OK data wise. The thing is when the problem occurred the filesystem was offline and once the filesystem was back to being OK, we were able to remount it. The problem is that now, every time a node is restarted, the filesystem is not mounted automatically as before. The other filesystems mount automatically. I did a mmlsfs on the file system and -A is at yes. Can someone tell my what I'm missing here. I'm sure it is something simple, but I'm drawing a blank right now. Richard -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1851 bytes Desc: not available URL: From luke.raimbach at oerc.ox.ac.uk Tue Mar 17 12:52:25 2015 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Tue, 17 Mar 2015 12:52:25 +0000 Subject: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans Message-ID: Hi Experts, I'm worrying about creating independent filesets which start to get in the way of one another: Let's pretend I create independent "fileset1" with a new inode space and preallocate say 1 million inodes. I start pushing files in to fileset1. I also start pushing files in to the root fileset. When fileset1 fills up, and because I've not specified a maximum number of inodes at creation time, presumably the file system manager gets on the case and preallocates more inodes as needed for fileset1. But while fileset1 has been filling up, I've also been filling up the root fileset (pretend I put 5 million files in there by the time fileset1 filled up). Now what happens? I'm guessing the FS manager starts chopping out sections of the root fileset beyond the 6 million region and marking them as independent and belonging to fileset1, in much the same way as it extends the allocated inode space when the file system starts running low. I hope this happens anyway. Let's say it does. So, now what about performance when running policy engine scans? I'm wanting to speed things up by restricting the policy scan to fileset1 (because I know that's where I put my special files). So I write the rules to select files and give them to mmapplypolicy. Does performance go down the drain as my once neatly arranged inode space is fragmented by not having enough inodes preallocated to begin with? Imagine fileset1 and the root fileset filling continuously at a similar rate and constantly "overtaking" one another in the inode space... does the performance advantage of independence start to go away rapidly? Cheers, Luke. -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 From makaplan at us.ibm.com Tue Mar 17 14:22:58 2015 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 17 Mar 2015 10:22:58 -0400 Subject: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans In-Reply-To: References: Message-ID: Luke, Thanks for your question. Independent filesets and their inodes are not implemented the way you might be imagining or guessing. Suppose you have two independent filesets "root" and "fset2" in the same GPFS file system. It is true that all the inode records (typically 512 bytes each - see mmcrfs) go into the same special file. BUT if you look at any given metadata allocation block --metadata-block-size (defaults to 256K) you'll only see inodes for either "root" or "fset2" not both in the same block. Moreover we try to "pre"-allocate several blocks of inodes for the same independent fileset contiguously - so that typically GPFS can do one seek and then read several blocks of inodes from the same independent fileset. So there can be performance advantages to using independent filesets and restricting your mmapplypolicy scans to just the fileset you need. To gain maximal advantage, use the following form of the command: mmapplypolicy /gpfs/path-to-the-directory-tree-I-want-to-scan --scope {fileset | inodespace } ... An inodespace is the set of all inodes in an independent fileset. An inodespace may contain several "dependent" filesets. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 21994 bytes Desc: not available URL: From zgiles at gmail.com Tue Mar 17 15:49:06 2015 From: zgiles at gmail.com (Zachary Giles) Date: Tue, 17 Mar 2015 11:49:06 -0400 Subject: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans In-Reply-To: References: Message-ID: I've always wondered also what the decision point is to make new inode tables per fileset. Sounds like the main benefit is scope of mmapplypolicy ( and other commands maybe ) and subblocks being grouped together for performance reasons. Is that right? Are there other benefits like portability or grouping a fileset's inode table into specific failure groups (in the system pool) etc? Any downsides? ( for example: now you have more overall metadata reads.. or only a limited number of inode tables.. or total number of inodes in the system is higher due to overallocation on N tables ? ) -Zach On Tue, Mar 17, 2015 at 10:22 AM, Marc A Kaplan wrote: > Luke, > > Thanks for your question. Independent filesets and their inodes are not > implemented the way you might be imagining or guessing. > > Suppose you have two independent filesets "root" and "fset2" in the same > GPFS file system. > It is true that all the inode records (typically 512 bytes each - see > mmcrfs) go into the same special file. BUT if you look at any given > metadata allocation block --metadata-block-size (defaults to 256K) you'll > only see inodes for either "root" or "fset2" not both in the same block. > > Moreover we try to "pre"-allocate several blocks of inodes for the same > independent fileset contiguously - so that typically GPFS can do one seek > and then read several blocks of inodes from the same independent fileset. > > So there can be performance advantages to using independent filesets and > restricting your mmapplypolicy scans to just the fileset you need. > To gain maximal advantage, use the following form of the command: > > mmapplypolicy /gpfs/path-to-the-directory-tree-I-want-to-scan --scope > {fileset | inodespace } ... > > > An inodespace is the set of all inodes in an independent fileset. An > inodespace may contain several "dependent" filesets. > > > > > [image: Marc A Kaplan] > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- Zach Giles zgiles at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 21994 bytes Desc: not available URL: From Paul.Sanchez at deshaw.com Tue Mar 17 16:01:21 2015 From: Paul.Sanchez at deshaw.com (Sanchez, Paul) Date: Tue, 17 Mar 2015 16:01:21 +0000 Subject: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans In-Reply-To: References: Message-ID: <201D6001C896B846A9CFC2E841986AC145360AA6@mailnycmb2a.winmail.deshaw.com> It?s also required for per-fileset snapshots, which are useful if you have differing snapshot requirements or want the ability to flush high-churn snapshots that are consuming an unexpected amount of space, but with better resolution than a whole filesystem. -Paul Sanchez From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Zachary Giles Sent: Tuesday, March 17, 2015 11:49 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans I've always wondered also what the decision point is to make new inode tables per fileset. Sounds like the main benefit is scope of mmapplypolicy ( and other commands maybe ) and subblocks being grouped together for performance reasons. Is that right? Are there other benefits like portability or grouping a fileset's inode table into specific failure groups (in the system pool) etc? Any downsides? ( for example: now you have more overall metadata reads.. or only a limited number of inode tables.. or total number of inodes in the system is higher due to overallocation on N tables ? ) -Zach On Tue, Mar 17, 2015 at 10:22 AM, Marc A Kaplan > wrote: Luke, Thanks for your question. Independent filesets and their inodes are not implemented the way you might be imagining or guessing. Suppose you have two independent filesets "root" and "fset2" in the same GPFS file system. It is true that all the inode records (typically 512 bytes each - see mmcrfs) go into the same special file. BUT if you look at any given metadata allocation block --metadata-block-size (defaults to 256K) you'll only see inodes for either "root" or "fset2" not both in the same block. Moreover we try to "pre"-allocate several blocks of inodes for the same independent fileset contiguously - so that typically GPFS can do one seek and then read several blocks of inodes from the same independent fileset. So there can be performance advantages to using independent filesets and restricting your mmapplypolicy scans to just the fileset you need. To gain maximal advantage, use the following form of the command: mmapplypolicy /gpfs/path-to-the-directory-tree-I-want-to-scan --scope {fileset | inodespace } ... An inodespace is the set of all inodes in an independent fileset. An inodespace may contain several "dependent" filesets. [Marc A Kaplan] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Zach Giles zgiles at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 21994 bytes Desc: image001.gif URL: From zgiles at gmail.com Tue Mar 17 16:04:25 2015 From: zgiles at gmail.com (Zachary Giles) Date: Tue, 17 Mar 2015 12:04:25 -0400 Subject: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans In-Reply-To: <201D6001C896B846A9CFC2E841986AC145360AA6@mailnycmb2a.winmail.deshaw.com> References: <201D6001C896B846A9CFC2E841986AC145360AA6@mailnycmb2a.winmail.deshaw.com> Message-ID: Ah. that's a good point about the snapshots. On Tue, Mar 17, 2015 at 12:01 PM, Sanchez, Paul wrote: > It?s also required for per-fileset snapshots, which are useful if you > have differing snapshot requirements or want the ability to flush > high-churn snapshots that are consuming an unexpected amount of space, but > with better resolution than a whole filesystem. > > > > -Paul Sanchez > > > > *From:* gpfsug-discuss-bounces at gpfsug.org [mailto: > gpfsug-discuss-bounces at gpfsug.org] *On Behalf Of *Zachary Giles > *Sent:* Tuesday, March 17, 2015 11:49 AM > *To:* gpfsug main discussion list > *Subject:* Re: [gpfsug-discuss] Fragmented Inode Space and Performance of > Policy Scans > > > > I've always wondered also what the decision point is to make new inode > tables per fileset. > > > > Sounds like the main benefit is scope of mmapplypolicy ( and other > commands maybe ) and subblocks being grouped together for performance > reasons. Is that right? > > Are there other benefits like portability or grouping a fileset's inode > table into specific failure groups (in the system pool) etc? > > Any downsides? ( for example: now you have more overall metadata reads.. > or only a limited number of inode tables.. or total number of inodes in the > system is higher due to overallocation on N tables ? ) > > > > -Zach > > > > > > On Tue, Mar 17, 2015 at 10:22 AM, Marc A Kaplan > wrote: > > Luke, > > Thanks for your question. Independent filesets and their inodes are not > implemented the way you might be imagining or guessing. > > Suppose you have two independent filesets "root" and "fset2" in the same > GPFS file system. > It is true that all the inode records (typically 512 bytes each - see > mmcrfs) go into the same special file. BUT if you look at any given > metadata allocation block --metadata-block-size (defaults to 256K) you'll > only see inodes for either "root" or "fset2" not both in the same block. > > Moreover we try to "pre"-allocate several blocks of inodes for the same > independent fileset contiguously - so that typically GPFS can do one seek > and then read several blocks of inodes from the same independent fileset. > > So there can be performance advantages to using independent filesets and > restricting your mmapplypolicy scans to just the fileset you need. > To gain maximal advantage, use the following form of the command: > > mmapplypolicy /gpfs/path-to-the-directory-tree-I-want-to-scan --scope > {fileset | inodespace } ... > > > An inodespace is the set of all inodes in an independent fileset. An > inodespace may contain several "dependent" filesets. > > > > > [image: Marc A Kaplan] > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > -- > > Zach Giles > zgiles at gmail.com > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- Zach Giles zgiles at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 21994 bytes Desc: not available URL: From chair at gpfsug.org Tue Mar 17 16:47:43 2015 From: chair at gpfsug.org (Jez Tucker (Chair)) Date: Tue, 17 Mar 2015 16:47:43 +0000 Subject: [gpfsug-discuss] GPFS User Group 2015 Meeting Date confirmed - RSVP Message-ID: <55085AAF.8050108@gpfsug.org> Hello all I can now confirm that 2015 GPFS User Group Meeting will be on May 20th, in tandem with SPXXL. For full details please see: http://www.gpfsug.org/gpfs-user-group-2015-invite/ RSVP to secretary at gpfsug.org See you there. Jez From richard.lefebvre at calculquebec.ca Thu Mar 19 22:32:58 2015 From: richard.lefebvre at calculquebec.ca (Richard Lefebvre) Date: Thu, 19 Mar 2015 18:32:58 -0400 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: Message-ID: Thanks for those who replied. One thing I didn't mention correctly is that I have several gpfs mount say: /gpfs1 /gpfs2 /gpfs3 Say it was gpfs2 that had the NSD problem. So now when I reboot a node, /gpfs1 and /gpfs3 mount, but /gpfs2 doesn't, I have to manually type mmmount gpfs2 on the node to get it mounted after the boot. So I don't think that it is a problem with mounting within linux boot sequence. But a flag than I'm missing in the gpfs setup. Or status of the filesystem that is not set OK that I cannot figure out. The output of mmlsfs of a system that mounts compared the the one that doesn't are the same except for mount point, NSDs and creation time. Is there another "mm..." command with which to check the status of a filesystem? Richard 2015-03-13 14:20 GMT-04:00 Richard Lefebvre < richard.lefebvre at calculquebec.ca>: > Hi, > > I have a GPFS cluster (v3.5). Last month, one of the filesystem had a > problem with one of it's NSD. With the help tech support, the problem was > fix and everything is OK data wise. The thing is when the problem occurred > the filesystem was offline and once the filesystem was back to being OK, we > were able to remount it. The problem is that now, every time a node is > restarted, the filesystem is not mounted automatically as before. The other > filesystems mount automatically. I did a mmlsfs on the file system and -A > is at yes. Can someone tell my what I'm missing here. I'm sure it is > something simple, but I'm drawing a blank right now. > > Richard > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Thu Mar 19 22:35:38 2015 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Thu, 19 Mar 2015 22:35:38 +0000 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: Message-ID: Check "mmlsdisk". Are all disks for the filesystem ok? -jf On tor. 19. mar. 2015 at 23.33 Richard Lefebvre < richard.lefebvre at calculquebec.ca> wrote: > Thanks for those who replied. One thing I didn't mention correctly is that > I have several gpfs mount say: > /gpfs1 > /gpfs2 > /gpfs3 > > Say it was gpfs2 that had the NSD problem. So now when I reboot a node, > /gpfs1 and /gpfs3 mount, but /gpfs2 doesn't, I have to manually type > mmmount gpfs2 on the node to get it mounted after the boot. > > So I don't think that it is a problem with mounting within linux boot > sequence. But a flag than I'm missing in the gpfs setup. Or status of the > filesystem that is not set OK that I cannot figure out. The output of > mmlsfs of a system that mounts compared the the one that doesn't are the > same except for mount point, NSDs and creation time. Is there another > "mm..." command with which to check the status of a filesystem? > > Richard > > > 2015-03-13 14:20 GMT-04:00 Richard Lefebvre < > richard.lefebvre at calculquebec.ca>: > > Hi, >> >> I have a GPFS cluster (v3.5). Last month, one of the filesystem had a >> problem with one of it's NSD. With the help tech support, the problem was >> fix and everything is OK data wise. The thing is when the problem occurred >> the filesystem was offline and once the filesystem was back to being OK, we >> were able to remount it. The problem is that now, every time a node is >> restarted, the filesystem is not mounted automatically as before. The other >> filesystems mount automatically. I did a mmlsfs on the file system and -A >> is at yes. Can someone tell my what I'm missing here. I'm sure it is >> something simple, but I'm drawing a blank right now. >> >> Richard >> >> -- >> Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" >> Richard.Lefebvre at calculquebec.ca -- THGTTG >> Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) >> > > > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.lefebvre at calculquebec.ca Thu Mar 19 22:38:43 2015 From: richard.lefebvre at calculquebec.ca (Richard Lefebvre) Date: Thu, 19 Mar 2015 18:38:43 -0400 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: Message-ID: Hi JF, There are all up and ready. Otherwise I wouldn't be able to mmmount it. Richard 2015-03-19 18:35 GMT-04:00 Jan-Frode Myklebust : > Check "mmlsdisk". Are all disks for the filesystem ok? > > > -jf > On tor. 19. mar. 2015 at 23.33 Richard Lefebvre < > richard.lefebvre at calculquebec.ca> wrote: > >> Thanks for those who replied. One thing I didn't mention correctly is >> that I have several gpfs mount say: >> /gpfs1 >> /gpfs2 >> /gpfs3 >> >> Say it was gpfs2 that had the NSD problem. So now when I reboot a node, >> /gpfs1 and /gpfs3 mount, but /gpfs2 doesn't, I have to manually type >> mmmount gpfs2 on the node to get it mounted after the boot. >> >> So I don't think that it is a problem with mounting within linux boot >> sequence. But a flag than I'm missing in the gpfs setup. Or status of the >> filesystem that is not set OK that I cannot figure out. The output of >> mmlsfs of a system that mounts compared the the one that doesn't are the >> same except for mount point, NSDs and creation time. Is there another >> "mm..." command with which to check the status of a filesystem? >> >> Richard >> >> >> 2015-03-13 14:20 GMT-04:00 Richard Lefebvre < >> richard.lefebvre at calculquebec.ca>: >> >> Hi, >>> >>> I have a GPFS cluster (v3.5). Last month, one of the filesystem had a >>> problem with one of it's NSD. With the help tech support, the problem was >>> fix and everything is OK data wise. The thing is when the problem occurred >>> the filesystem was offline and once the filesystem was back to being OK, we >>> were able to remount it. The problem is that now, every time a node is >>> restarted, the filesystem is not mounted automatically as before. The other >>> filesystems mount automatically. I did a mmlsfs on the file system and -A >>> is at yes. Can someone tell my what I'm missing here. I'm sure it is >>> something simple, but I'm drawing a blank right now. >>> >>> Richard >>> >>> -- >>> Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" >>> Richard.Lefebvre at calculquebec.ca -- THGTTG >>> Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) >>> >> >> >> >> -- >> Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" >> Richard.Lefebvre at calculquebec.ca -- THGTTG >> Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at gpfsug.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) -------------- next part -------------- An HTML attachment was scrubbed... URL: From red_steel at au1.ibm.com Fri Mar 20 01:48:52 2015 From: red_steel at au1.ibm.com (Tony Steel) Date: Fri, 20 Mar 2015 12:48:52 +1100 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: Message-ID: Hi You don't have a file (assuming gpfs2 is the problem) /var/mmfs/etc/ignoreStartupMount.gpfs2 ?? Cheers, - Sense Amid Madness, Wit Amidst Folly - ---------- Antony (Red) Steel, Advanced Technical Support -------------------- Senior IT Specialist Ph: +61 293978414 Mb: +61 409 971 083 Home Page: http://sydgsa.ibm.com/projects/t/team_web/index.html ----------------------------- red_steel at au.ibm.com ----------------------------------- From ckerner at ncsa.uiuc.edu Fri Mar 20 02:56:55 2015 From: ckerner at ncsa.uiuc.edu (Chad Kerner) Date: Thu, 19 Mar 2015 21:56:55 -0500 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: Message-ID: <20150320025655.GA2140@logos.ncsa.illinois.edu> Richard, what does "mmlsfs gpfs2 -A" return? Is it set to automount? --Chad Chad Kerner, Systems Engineer Storage Enabling Technologies National Center for Supercomputing Applications University of Illinois, Urbana-Champaign On Thu, Mar 19, 2015 at 06:38:43PM -0400, Richard Lefebvre wrote: > Hi JF, > > There are all up and ready. Otherwise I wouldn't be able to mmmount it. > > Richard > > > > 2015-03-19 18:35 GMT-04:00 Jan-Frode Myklebust : > > Check "mmlsdisk". Are all disks for the filesystem ok? > > > -jf > On tor. 19. mar. 2015 at 23.33 Richard Lefebvre < > richard.lefebvre at calculquebec.ca> wrote: > > Thanks for those who replied. One thing I didn't mention correctly is > that I have several gpfs mount say: > /gpfs1 > /gpfs2 > /gpfs3 > > Say it was gpfs2 that had the NSD problem. So now when I reboot a node, > /gpfs1 and /gpfs3 mount, but /gpfs2 doesn't, I have to manually type > mmmount gpfs2 on the node to get it mounted after the boot. > > So I don't think that it is a problem with mounting within linux boot > sequence. But a flag than I'm missing in the gpfs setup. Or status of > the filesystem that is not set OK that I cannot figure out. The output > of mmlsfs of a system that mounts compared the the one that doesn't are > the same except for mount point, NSDs and creation time. Is there > another "mm..." command with which to check the status of a filesystem? > > Richard > > > 2015-03-13 14:20 GMT-04:00 Richard Lefebvre < > richard.lefebvre at calculquebec.ca>: > > > Hi, > > I have a GPFS cluster (v3.5). Last month, one of the filesystem had > a problem with one of it's NSD. With the help tech support, the > problem was fix and everything is OK data wise. The thing is when > the problem occurred the filesystem was offline and once the > filesystem was back to being OK, we were able to remount it. The > problem is that now, every time a node is restarted, the filesystem > is not mounted automatically as before. The other filesystems mount > automatically. I did a mmlsfs on the file system and -A is at yes. > Can someone tell my what I'm missing here. I'm sure it is something > simple, but I'm drawing a blank right now. > > Richard > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't > Panic" > Richard.Lefebvre at calculquebec.ca -- > THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada ( > computecanada.ca) > > > > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't > Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From luke.raimbach at oerc.ox.ac.uk Fri Mar 20 08:37:02 2015 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Fri, 20 Mar 2015 08:37:02 +0000 Subject: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans Message-ID: Hi Marc, Thanks for this information. One thing I still can't extract from the manual (though I guess I could just do a small experiment to find out) is whether the number of allocated inodes is automatically extended when running low, up to the maximum for an independent fileset? Cheers, Luke. From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Marc A Kaplan Sent: 17 March 2015 14:23 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans Luke, Thanks for your question. Independent filesets and their inodes are not implemented the way you might be imagining or guessing. Suppose you have two independent filesets "root" and "fset2" in the same GPFS file system. It is true that all the inode records (typically 512 bytes each - see mmcrfs) go into the same special file. BUT if you look at any given metadata allocation block --metadata-block-size (defaults to 256K) you'll only see inodes for either "root" or "fset2" not both in the same block. Moreover we try to "pre"-allocate several blocks of inodes for the same independent fileset contiguously - so that typically GPFS can do one seek and then read several blocks of inodes from the same independent fileset. So there can be performance advantages to using independent filesets and restricting your mmapplypolicy scans to just the fileset you need. To gain maximal advantage, use the following form of the command: mmapplypolicy /gpfs/path-to-the-directory-tree-I-want-to-scan --scope {fileset | inodespace } ... An inodespace is the set of all inodes in an independent fileset. An inodespace may contain several "dependent" filesets. [Marc A Kaplan] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 21994 bytes Desc: image001.gif URL: From makaplan at us.ibm.com Fri Mar 20 12:10:34 2015 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 20 Mar 2015 08:10:34 -0400 Subject: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans In-Reply-To: References: Message-ID: Seems to me that's why this option has two parameters: mmchfileset ... [--inode-limit MaxNumInodes[:NumInodesToPreallocate]] Specifies the new inode limit for the inode space owned by the specified fileset. The FilesetName or JunctionPath must refer to an independent fileset. The NumInodesToPreallocate specifies an optional number of additional inodes to pre-allocate for the inode space. Use the mmchfs command to change inode limits for the root fileset. See also mmchfs! And RTFM! ( ReadTheFantasicManual ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.lefebvre at calculquebec.ca Fri Mar 20 17:22:43 2015 From: richard.lefebvre at calculquebec.ca (Richard Lefebvre) Date: Fri, 20 Mar 2015 13:22:43 -0400 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: <20150320025655.GA2140@logos.ncsa.illinois.edu> References: <20150320025655.GA2140@logos.ncsa.illinois.edu> Message-ID: Hi Tony, Chad, There is nothing in /var/mmfs/etc directory of the nodes. "-A" on the nodes return "yes" Richard 2015-03-19 22:56 GMT-04:00 Chad Kerner : > Richard, what does "mmlsfs gpfs2 -A" return? Is it set to automount? > > --Chad > Chad Kerner, Systems Engineer > Storage Enabling Technologies > National Center for Supercomputing Applications > University of Illinois, Urbana-Champaign > > On Thu, Mar 19, 2015 at 06:38:43PM -0400, Richard Lefebvre wrote: > > Hi JF, > > > > There are all up and ready. Otherwise I wouldn't be able to mmmount it. > > > > Richard > > > > > > > > 2015-03-19 18:35 GMT-04:00 Jan-Frode Myklebust : > > > > Check "mmlsdisk". Are all disks for the filesystem ok? > > > > > > -jf > > On tor. 19. mar. 2015 at 23.33 Richard Lefebvre < > > richard.lefebvre at calculquebec.ca> wrote: > > > > Thanks for those who replied. One thing I didn't mention > correctly is > > that I have several gpfs mount say: > > /gpfs1 > > /gpfs2 > > /gpfs3 > > > > Say it was gpfs2 that had the NSD problem. So now when I reboot > a node, > > /gpfs1 and /gpfs3 mount, but /gpfs2 doesn't, I have to manually > type > > mmmount gpfs2 on the node to get it mounted after the boot. > > > > So I don't think that it is a problem with mounting within linux > boot > > sequence. But a flag than I'm missing in the gpfs setup. Or > status of > > the filesystem that is not set OK that I cannot figure out. The > output > > of mmlsfs of a system that mounts compared the the one that > doesn't are > > the same except for mount point, NSDs and creation time. Is there > > another "mm..." command with which to check the status of a > filesystem? > > > > Richard > > > > > > 2015-03-13 14:20 GMT-04:00 Richard Lefebvre < > > richard.lefebvre at calculquebec.ca>: > > > > > > Hi, > > > > I have a GPFS cluster (v3.5). Last month, one of the > filesystem had > > a problem with one of it's NSD. With the help tech support, > the > > problem was fix and everything is OK data wise. The thing is > when > > the problem occurred the filesystem was offline and once the > > filesystem was back to being OK, we were able to remount it. > The > > problem is that now, every time a node is restarted, the > filesystem > > is not mounted automatically as before. The other > filesystems mount > > automatically. I did a mmlsfs on the file system and -A is > at yes. > > Can someone tell my what I'm missing here. I'm sure it is > something > > simple, but I'm drawing a blank right now. > > > > Richard > > > > -- > > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 > "Don't > > Panic" > > Richard.Lefebvre at calculquebec.ca > -- > > THGTTG > > Calcul Quebec (calculquebec.ca) ------ Calcul Canada ( > > computecanada.ca) > > > > > > > > > > -- > > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 > "Don't > > Panic" > > Richard.Lefebvre at calculquebec.ca > -- THGTTG > > Calcul Quebec (calculquebec.ca) ------ Calcul Canada ( > computecanada.ca) > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at gpfsug.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at gpfsug.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > > > > > > -- > > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" > > Richard.Lefebvre at calculquebec.ca -- THGTTG > > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at gpfsug.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) -------------- next part -------------- An HTML attachment was scrubbed... URL: From YARD at il.ibm.com Fri Mar 20 17:50:33 2015 From: YARD at il.ibm.com (Yaron Daniel) Date: Fri, 20 Mar 2015 19:50:33 +0200 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: <20150320025655.GA2140@logos.ncsa.illinois.edu> Message-ID: Hi Can u send the output of: #mmlsdisk gpfs2 #mmlsnsd |grep gpfs2 #getenforce #iptables -L #cat /etc/fstab Regards Yaron Daniel 94 Em Ha'Moshavot Rd Server, Storage and Data Services - Team Leader Petach Tiqva, 49527 Global Technology Services Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com IBM Israel From: Richard Lefebvre To: gpfsug main discussion list Date: 03/20/2015 07:22 PM Subject: Re: [gpfsug-discuss] GPFS file system won't mount automatically anymore. Sent by: gpfsug-discuss-bounces at gpfsug.org Hi Tony, Chad, There is nothing in /var/mmfs/etc directory of the nodes. "-A" on the nodes return "yes" Richard 2015-03-19 22:56 GMT-04:00 Chad Kerner : Richard, what does "mmlsfs gpfs2 -A" return? Is it set to automount? --Chad Chad Kerner, Systems Engineer Storage Enabling Technologies National Center for Supercomputing Applications University of Illinois, Urbana-Champaign On Thu, Mar 19, 2015 at 06:38:43PM -0400, Richard Lefebvre wrote: > Hi JF, > > There are all up and ready. Otherwise I wouldn't be able to mmmount it. > > Richard > > > > 2015-03-19 18:35 GMT-04:00 Jan-Frode Myklebust : > > Check "mmlsdisk". Are all disks for the filesystem ok? > > > -jf > On tor. 19. mar. 2015 at 23.33 Richard Lefebvre < > richard.lefebvre at calculquebec.ca> wrote: > > Thanks for those who replied. One thing I didn't mention correctly is > that I have several gpfs mount say: > /gpfs1 > /gpfs2 > /gpfs3 > > Say it was gpfs2 that had the NSD problem. So now when I reboot a node, > /gpfs1 and /gpfs3 mount, but /gpfs2 doesn't, I have to manually type > mmmount gpfs2 on the node to get it mounted after the boot. > > So I don't think that it is a problem with mounting within linux boot > sequence. But a flag than I'm missing in the gpfs setup. Or status of > the filesystem that is not set OK that I cannot figure out. The output > of mmlsfs of a system that mounts compared the the one that doesn't are > the same except for mount point, NSDs and creation time. Is there > another "mm..." command with which to check the status of a filesystem? > > Richard > > > 2015-03-13 14:20 GMT-04:00 Richard Lefebvre < > richard.lefebvre at calculquebec.ca>: > > > Hi, > > I have a GPFS cluster (v3.5). Last month, one of the filesystem had > a problem with one of it's NSD. With the help tech support, the > problem was fix and everything is OK data wise. The thing is when > the problem occurred the filesystem was offline and once the > filesystem was back to being OK, we were able to remount it. The > problem is that now, every time a node is restarted, the filesystem > is not mounted automatically as before. The other filesystems mount > automatically. I did a mmlsfs on the file system and -A is at yes. > Can someone tell my what I'm missing here. I'm sure it is something > simple, but I'm drawing a blank right now. > > Richard > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't > Panic" > Richard.Lefebvre at calculquebec.ca -- > THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada ( > computecanada.ca) > > > > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't > Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada ( computecanada.ca) > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1851 bytes Desc: not available URL: From makaplan at us.ibm.com Fri Mar 20 19:24:25 2015 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 20 Mar 2015 15:24:25 -0400 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: <20150320025655.GA2140@logos.ncsa.illinois.edu> Message-ID: Look in /var/adm/ras/mmfs.log.* -- on a node that should be mounting the filesystem -- find a file that covers the time when GPFS is starting up and should be mounting the filesystem. Look for clues and/or error messages. If you are not familiar, you may also want to look at what messages are issued when everything is working just right. -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.lefebvre at calculquebec.ca Fri Mar 20 20:32:46 2015 From: richard.lefebvre at calculquebec.ca (Richard Lefebvre) Date: Fri, 20 Mar 2015 16:32:46 -0400 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: <20150320025655.GA2140@logos.ncsa.illinois.edu> Message-ID: Hi Mark, Thank you for your answer. You are right, I have the following: Mon Mar 16 11:02:24 EDT 2015: mounting /dev/gpfs2 Mon Mar 16 11:02:24.296 2015: Command: mount gpfs2 Mon Mar 16 11:02:24.723 2015: Disk failure. Volume gpfs2. rc = 5. Physical volume gpfs2nsd04. Mon Mar 16 11:02:24.724 2015: Disk failure. Volume gpfs2. rc = 5. Physical volume gpfs2nsd12. Mon Mar 16 11:02:24.723 2015: File System gpfs2 unmounted by the system with return code 5 reason code 0 Mon Mar 16 11:02:24.724 2015: Input/output error Mon Mar 16 11:02:24.725 2015: Failed to open gpfs2. Mon Mar 16 11:02:24.724 2015: Input/output error Mon Mar 16 11:02:24.725 2015: Command: err 666: mount gpfs2 Mon Mar 16 11:02:24.724 2015: Input/output error mount: Stale NFS file handle Mon Mar 16 11:02:24 EDT 2015: finished mounting /dev/gpfs2 But it is not mounted. This is after issuing the "mmmount gpfs2" command Mon Mar 16 11:03:52.010 2015: Command: mount gpfs2 Mon Mar 16 11:03:52.234 2015: VERBS RDMA connecting to 172.25.101.2 (heca2-ib0) on mlx4_0 port 1 id 4 Mon Mar 16 11:03:52.235 2015: VERBS RDMA connected to 172.25.101.2 (heca2-ib0) on mlx4_0 port 1 sl 0 Mon Mar 16 11:03:52.381 2015: Command: err 0: mount gpfs2 Now, my next step it to figure out how the clear the failure since it is not failed anymore. The mmlsdisk reports all the nsd are up and ready. One command I'm also looking for is a command to list which nsd server is used for a particular nsd. I have a redundant config for which 2 servers can access the same nsd. But I think that for some of them, the default server is not used and the load of the servers is not distributed as wanted. Richard 2015-03-20 15:24 GMT-04:00 Marc A Kaplan : > Look in /var/adm/ras/mmfs.log.* -- on a node that should be mounting the > filesystem -- find a file that covers the time when GPFS is starting up > and should be mounting the filesystem. > > Look for clues and/or error messages. If you are not familiar, you may > also want to look at what messages are issued when everything is working > just right. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) -------------- next part -------------- An HTML attachment was scrubbed... URL: From Dan.Foster at bristol.ac.uk Mon Mar 23 08:43:33 2015 From: Dan.Foster at bristol.ac.uk (Dan Foster) Date: Mon, 23 Mar 2015 08:43:33 +0000 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: <20150320025655.GA2140@logos.ncsa.illinois.edu> Message-ID: Hi Richard, On 20 March 2015 at 20:32, Richard Lefebvre wrote: > One command I'm also looking for is a command to list which nsd server is > used for a particular nsd. I have a redundant config for which 2 servers can > access the same nsd. But I think that for some of them, the default server > is not used and the load of the servers is not distributed as wanted. Does the '-M' option to mmlsdisk provide the information you're looking for? i.e. from the man page: """ Displays whether I/O requests to the disk are satisfied on the local node, or using an NSD server. If the I/O is done using an NSD server, shows the NSD server name and the underlying disk name on that server node. """ It should display something similar to the following: Disk name IO performed on node Device Availability ------------ ----------------------- ----------------- ------------ nsd1 localhost /dev/disk1 up nsd2 node01.example.com /dev/disk2 up nsd3 localhost /dev/disk3 up HTH, Dan. > > 2015-03-20 15:24 GMT-04:00 Marc A Kaplan : >> >> Look in /var/adm/ras/mmfs.log.* -- on a node that should be mounting the >> filesystem -- find a file that covers the time when GPFS is starting up >> and should be mounting the filesystem. >> >> Look for clues and/or error messages. If you are not familiar, you may >> also want to look at what messages are issued when everything is working >> just right. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at gpfsug.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Dan Foster | Senior Storage Systems Administrator | IT Services e: dan.foster at bristol.ac.uk | t: 0117 3941170 [x41170] m: Advanced Computing Research Centre, University of Bristol, 8-10 Berkeley Square, Bristol BS8 1HH From PEREZJOS at uk.ibm.com Mon Mar 23 14:13:08 2015 From: PEREZJOS at uk.ibm.com (Jose Perez) Date: Mon, 23 Mar 2015 14:13:08 +0000 Subject: [gpfsug-discuss] Jose Perez is OoO. Message-ID: I will be out of the office starting 23/03/2015 and will not return until 24/03/2015. I will respond to your message when I return. For urgent issues please contact Ross Keeping, Thanks. Thanks, From chair at gpfsug.org Mon Mar 23 14:34:20 2015 From: chair at gpfsug.org (Jez Tucker (Chair)) Date: Mon, 23 Mar 2015 14:34:20 +0000 Subject: [gpfsug-discuss] Jose Perez is OoO. In-Reply-To: References: Message-ID: <5510246C.6090009@gpfsug.org> Apols for that one which slipped through, we'll add 'OoO' to the list blocker. Jez On 23/03/15 14:13, Jose Perez wrote: > I will be out of the office starting 23/03/2015 and will not return until > 24/03/2015. > > I will respond to your message when I return. For urgent issues please > contact Ross Keeping, Thanks. > Thanks, > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From makaplan at us.ibm.com Mon Mar 23 14:33:29 2015 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 23 Mar 2015 10:33:29 -0400 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: <20150320025655.GA2140@logos.ncsa.illinois.edu> Message-ID: mmlsnsd has several options... RTFM... but this one will show you which nodes can directly access and/or serve each NSD: mmlsnsd -M -v -M Maps the NSD names to its disk device name on all nodes. This is a slow operation and its usage is suggested for problem determination only. -v Specifies that the output should contain error information, where available. -------------- next part -------------- An HTML attachment was scrubbed... URL: From secretary at gpfsug.org Wed Mar 25 14:42:52 2015 From: secretary at gpfsug.org (Secretary GPFS UG) Date: Wed, 25 Mar 2015 14:42:52 +0000 Subject: [gpfsug-discuss] 2nd 'Meet the Devs' Event 28/4/15 Message-ID: <6715b1493f52d222ab24b49195ab8fc2@webmail.gpfsug.org> Dear members, Our second Meet the GPFS Developers 'coffee shop' meet is going to be taking place on Tuesday 28th April. The day will give you the opportunity to meet the developers and get hands on beta exploration and test of unreleased GPFS functions. Date: 28th April 2015 Time: 10:30 - 15:30 Location: IBM, 3rd Floor, Maybrook House, 40 Blackfriars Street, Manchester M3 2EG Agenda: 10:30 Introduction discussion 11:30 Beta session 1 12:30 Break for pizza (social chats with wider GPFS dev team in Manchester) 13:30 Beta session 2 14:45 Group discussion and feedback 15:30 End Lunch and refreshments will be provided. To attend, please register with Claire Robson, GPFS UG Secretary secretary at gpfsug.org. 10 places are available. Full directions and instructions will be sent to confirmed attendees. We look forward to seeing you in April. Thanks, Claire -- GPFS User Group Secretary From ross.keeping at uk.ibm.com Thu Mar 26 14:59:31 2015 From: ross.keeping at uk.ibm.com (Ross Keeping3) Date: Thu, 26 Mar 2015 14:59:31 +0000 Subject: [gpfsug-discuss] Potential talks for GPFS UG May 20th Message-ID: Hi There are a couple of talks we could include in the one day GPFS UG on May 20th. We'd appreciate some feedback on whether there is any interest or objection to including these. These are: 1) Tivoli Storage Productivity Center Technical Demo: A tool for managing heterogeneous storage infrastructures and specifically GPFS. Can read more here http://www-03.ibm.com/software/products/en/tivostorprodcent 2) GPFS for life sciences: A member of NERSC would like to talk about their efforts to improve GPFS file systems for a life sciences workload - specifically genomic sequencing and analysis that ensues. Details about life science workflow and requirements will be provided so the listener may understand the aspects of GPFS that are important. Cheers, Ross Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From stijn.deweirdt at ugent.be Thu Mar 26 15:07:29 2015 From: stijn.deweirdt at ugent.be (Stijn De Weirdt) Date: Thu, 26 Mar 2015 16:07:29 +0100 Subject: [gpfsug-discuss] Potential talks for GPFS UG May 20th In-Reply-To: References: Message-ID: <551420B1.3080002@ugent.be> no objections for any of them, but i'd prefer talk 2 (if a choice has to be made) stijn On 03/26/2015 03:59 PM, Ross Keeping3 wrote: > Hi > > There are a couple of talks we could include in the one day GPFS UG on May > 20th. > > We'd appreciate some feedback on whether there is any interest or > objection to including these. > > These are: > 1) Tivoli Storage Productivity Center Technical Demo: A tool for managing > heterogeneous storage infrastructures and specifically GPFS. Can read more > here http://www-03.ibm.com/software/products/en/tivostorprodcent > > 2) GPFS for life sciences: A member of NERSC would like to talk about > their efforts to improve GPFS file systems for a life sciences workload - > specifically genomic sequencing and analysis that ensues. Details about > life science workflow and requirements will be provided so the listener > may understand the aspects of GPFS that are important. > > Cheers, > Ross > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From robert at strubi.ox.ac.uk Thu Mar 26 15:13:44 2015 From: robert at strubi.ox.ac.uk (Robert Esnouf) Date: Thu, 26 Mar 2015 15:13:44 +0000 (GMT) Subject: [gpfsug-discuss] Potential talks for GPFS UG May 20th In-Reply-To: References: Message-ID: <201503261513.062218@mail.strubi.ox.ac.uk> Yes, unsurprisingly I'd be interested in the second talk! Thanks, Robert -- Dr. Robert Esnouf, University Research Lecturer, Head of Research Computing Core, NDM Research Computing Strategy Officer Room 10/028, Wellcome Trust Centre for Human Genetics, Old Road Campus, Roosevelt Drive, Oxford OX3 7BN, UK Email: robert at strubi.ox.ac.uk / robert at well.ox.ac.uk Tel: (+44) - 1865 - 287783 ---- Original message ---- >Date: Thu, 26 Mar 2015 14:59:31 +0000 >From: gpfsug-discuss-bounces at gpfsug.org (on behalf of Ross Keeping3 ) >Subject: [gpfsug-discuss] Potential talks for GPFS UG May 20th >To: gpfsug-discuss at gpfsug.org > > Hi > > There are a couple of talks we could include in the > one day GPFS UG on May 20th. > > We'd appreciate some feedback on whether there is > any interest or objection to including these. > > These are: > 1) Tivoli Storage Productivity Center Technical > Demo: A tool for managing heterogeneous storage > infrastructures and specifically GPFS. Can read more > here > http://www-03.ibm.com/software/products/en/tivostorprodcent > > 2) GPFS for life sciences: A member of NERSC would > like to talk about their efforts to improve GPFS > file systems for a life sciences workload - > specifically genomic sequencing and analysis that > ensues. Details about life science workflow and > requirements will be provided so the listener may > understand the aspects of GPFS that are important. > > Cheers, > Ross > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England > and Wales with number 741598. > Registered office: PO Box 41, North Harbour, > Portsmouth, Hampshire PO6 3AU >________________ >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at gpfsug.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Mark.Roberts at awe.co.uk Thu Mar 26 15:24:17 2015 From: Mark.Roberts at awe.co.uk (Mark.Roberts at awe.co.uk) Date: Thu, 26 Mar 2015 15:24:17 +0000 Subject: [gpfsug-discuss] EXTERNAL: Potential talks for GPFS UG May 20th In-Reply-To: References: Message-ID: <201503261524.t2QFOQrO020615@msw1.awe.co.uk> As well as supporting the efforts of our friends at NERSC I think should be a good interesting talk. Mark Roberts ________________________________ From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Ross Keeping3 Sent: 26 March 2015 15:00 To: gpfsug-discuss at gpfsug.org Subject: EXTERNAL: [gpfsug-discuss] Potential talks for GPFS UG May 20th Hi There are a couple of talks we could include in the one day GPFS UG on May 20th. We'd appreciate some feedback on whether there is any interest or objection to including these. These are: 1) Tivoli Storage Productivity Center Technical Demo: A tool for managing heterogeneous storage infrastructures and specifically GPFS. Can read more here http://www-03.ibm.com/software/products/en/tivostorprodcent 2) GPFS for life sciences: A member of NERSC would like to talk about their efforts to improve GPFS file systems for a life sciences workload - specifically genomic sequencing and analysis that ensues. Details about life science workflow and requirements will be provided so the listener may understand the aspects of GPFS that are important. Cheers, Ross Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU ___________________________________________________ ____________________________ The information in this email and in any attachment(s) is commercial in confidence. If you are not the named addressee(s) or if you receive this email in error then any distribution, copying or use of this communication or the information in it is strictly prohibited. Please notify us immediately by email at admin.internet(at)awe.co.uk, and then delete this message from your computer. While attachments are virus checked, AWE plc does not accept any liability in respect of any virus which is not detected. AWE Plc Registered in England and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR -------------- next part -------------- An HTML attachment was scrubbed... URL: From Sandra.McLaughlin at astrazeneca.com Thu Mar 26 15:56:16 2015 From: Sandra.McLaughlin at astrazeneca.com (McLaughlin, Sandra M) Date: Thu, 26 Mar 2015 15:56:16 +0000 Subject: [gpfsug-discuss] EXTERNAL: Potential talks for GPFS UG May 20th In-Reply-To: <201503261524.t2QFOQrO020615@msw1.awe.co.uk> References: <201503261524.t2QFOQrO020615@msw1.awe.co.uk> Message-ID: Life Sciences talk would be spot on for me too, please. Sandra McLaughlin From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Mark.Roberts at awe.co.uk Sent: 26 March 2015 15:24 To: gpfsug-discuss at gpfsug.org Subject: Re: [gpfsug-discuss] EXTERNAL: Potential talks for GPFS UG May 20th As well as supporting the efforts of our friends at NERSC I think should be a good interesting talk. Mark Roberts ________________________________ From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Ross Keeping3 Sent: 26 March 2015 15:00 To: gpfsug-discuss at gpfsug.org Subject: EXTERNAL: [gpfsug-discuss] Potential talks for GPFS UG May 20th Hi There are a couple of talks we could include in the one day GPFS UG on May 20th. We'd appreciate some feedback on whether there is any interest or objection to including these. These are: 1) Tivoli Storage Productivity Center Technical Demo: A tool for managing heterogeneous storage infrastructures and specifically GPFS. Can read more here http://www-03.ibm.com/software/products/en/tivostorprodcent 2) GPFS for life sciences: A member of NERSC would like to talk about their efforts to improve GPFS file systems for a life sciences workload - specifically genomic sequencing and analysis that ensues. Details about life science workflow and requirements will be provided so the listener may understand the aspects of GPFS that are important. Cheers, Ross Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU ___________________________________________________ ____________________________ The information in this email and in any attachment(s) is commercial in confidence. If you are not the named addressee(s) or if you receive this email in error then any distribution, copying or use of this communication or the information in it is strictly prohibited. Please notify us immediately by email at admin.internet(at)awe.co.uk, and then delete this message from your computer. While attachments are virus checked, AWE plc does not accept any liability in respect of any virus which is not detected. AWE Plc Registered in England and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR ________________________________ AstraZeneca UK Limited is a company incorporated in England and Wales with registered number: 03674842 and a registered office at 2 Kingdom Street, London, W2 6BD. Confidentiality Notice: This message is private and may contain confidential, proprietary and legally privileged information. If you have received this message in error, please notify us and remove it from your system and note that you must not copy, distribute or take any action in reliance on it. Any unauthorised use or disclosure of the contents of this message is not permitted and may be unlawful. Disclaimer: Email messages may be subject to delays, interception, non-delivery and unauthorised alterations. Therefore, information expressed in this message is not given or endorsed by AstraZeneca UK Limited unless otherwise notified by an authorised representative independent of this message. No contractual relationship is created by this message by any person unless specifically indicated by agreement in writing other than email. Monitoring: AstraZeneca UK Limited may monitor email traffic data and content for the purposes of the prevention and detection of crime, ensuring the security of our computer systems and checking compliance with our Code of Conduct and policies. -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.kidger at uk.ibm.com Thu Mar 26 16:06:24 2015 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Thu, 26 Mar 2015 16:06:24 +0000 Subject: [gpfsug-discuss] Potential talks for GPFS UG May 20th In-Reply-To: <201503261513.062218@mail.strubi.ox.ac.uk> References: <201503261513.062218@mail.strubi.ox.ac.uk> Message-ID: I too would want to upvote the second. I am seeing lots of interest in the Life Sciences particularly for things like large Genomic datasets and NGS. Daniel Dr.Daniel Kidger No. 1 The Square, Technical Specialist SDI (formerly Platform Computing) Temple Quay, Bristol BS1 6DG Mobile: +44-07818 522 266 United Kingdom Landline: +44-02392 564 121 (Internal ITN 3726 9250) e-mail: daniel.kidger at uk.ibm.com From: Robert Esnouf To: "gpfsug main discussion list" Date: 26/03/2015 15:13 Subject: Re: [gpfsug-discuss] Potential talks for GPFS UG May 20th Sent by: gpfsug-discuss-bounces at gpfsug.org Yes, unsurprisingly I'd be interested in the second talk! Thanks, Robert -- Dr. Robert Esnouf, University Research Lecturer, Head of Research Computing Core, NDM Research Computing Strategy Officer Room 10/028, Wellcome Trust Centre for Human Genetics, Old Road Campus, Roosevelt Drive, Oxford OX3 7BN, UK Email: robert at strubi.ox.ac.uk / robert at well.ox.ac.uk Tel: (+44) - 1865 - 287783 ---- Original message ---- >Date: Thu, 26 Mar 2015 14:59:31 +0000 >From: gpfsug-discuss-bounces at gpfsug.org (on behalf of Ross Keeping3 ) >Subject: [gpfsug-discuss] Potential talks for GPFS UG May 20th >To: gpfsug-discuss at gpfsug.org > > Hi > > There are a couple of talks we could include in the > one day GPFS UG on May 20th. > > We'd appreciate some feedback on whether there is > any interest or objection to including these. > > These are: > 1) Tivoli Storage Productivity Center Technical > Demo: A tool for managing heterogeneous storage > infrastructures and specifically GPFS. Can read more > here > http://www-03.ibm.com/software/products/en/tivostorprodcent > > 2) GPFS for life sciences: A member of NERSC would > like to talk about their efforts to improve GPFS > file systems for a life sciences workload - > specifically genomic sequencing and analysis that > ensues. Details about life science workflow and > requirements will be provided so the listener may > understand the aspects of GPFS that are important. > > Cheers, > Ross > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England > and Wales with number 741598. > Registered office: PO Box 41, North Harbour, > Portsmouth, Hampshire PO6 3AU >________________ >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at gpfsug.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 360 bytes Desc: not available URL: From zgiles at gmail.com Thu Mar 26 16:14:36 2015 From: zgiles at gmail.com (Zachary Giles) Date: Thu, 26 Mar 2015 12:14:36 -0400 Subject: [gpfsug-discuss] EXTERNAL: Potential talks for GPFS UG May 20th In-Reply-To: References: <201503261524.t2QFOQrO020615@msw1.awe.co.uk> Message-ID: It's at the GPFSUG in the UK. Looks like the UK GPFSUG is much more active than the US one's. They've had a few meetups actually. I'd fly out there and attend them, but I figured probably it wouldn't count since it's not a real conf. This on is in May 20th. On Thu, Mar 26, 2015 at 11:56 AM, McLaughlin, Sandra M < Sandra.McLaughlin at astrazeneca.com> wrote: > Life Sciences talk would be spot on for me too, please. > > > > Sandra McLaughlin > > > > *From:* gpfsug-discuss-bounces at gpfsug.org [mailto: > gpfsug-discuss-bounces at gpfsug.org] *On Behalf Of *Mark.Roberts at awe.co.uk > *Sent:* 26 March 2015 15:24 > *To:* gpfsug-discuss at gpfsug.org > *Subject:* Re: [gpfsug-discuss] EXTERNAL: Potential talks for GPFS UG May > 20th > > > > As well as supporting the efforts of our friends at NERSC I think should > be a good interesting talk. > > > > Mark Roberts > > > ------------------------------ > > *From:* gpfsug-discuss-bounces at gpfsug.org [ > mailto:gpfsug-discuss-bounces at gpfsug.org > ] *On Behalf Of *Ross Keeping3 > *Sent:* 26 March 2015 15:00 > *To:* gpfsug-discuss at gpfsug.org > *Subject:* EXTERNAL: [gpfsug-discuss] Potential talks for GPFS UG May 20th > > > > Hi > > There are a couple of talks we could include in the one day GPFS UG on May > 20th. > > We'd appreciate some feedback on whether there is any interest or > objection to including these. > > These are: > 1) Tivoli Storage Productivity Center Technical Demo: A tool for managing > heterogeneous storage infrastructures and specifically GPFS. Can read more > here http://www-03.ibm.com/software/products/en/tivostorprodcent > > 2) GPFS for life sciences: A member of NERSC would like to talk about > their efforts to improve GPFS file systems for a life sciences workload - > specifically genomic sequencing and analysis that ensues. Details about > life science workflow and requirements will be provided so the listener may > understand the aspects of GPFS that are important. > > Cheers, > Ross > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > ___________________________________________________ > ____________________________ The information in this email and in any > attachment(s) is commercial in confidence. If you are not the named > addressee(s) or if you receive this email in error then any distribution, > copying or use of this communication or the information in it is strictly > prohibited. Please notify us immediately by email at admin.internet(at) > awe.co.uk, and then delete this message from your computer. While > attachments are virus checked, AWE plc does not accept any liability in > respect of any virus which is not detected. AWE Plc Registered in England > and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR > ------------------------------ > > AstraZeneca UK Limited is a company incorporated in England and Wales with > registered number: 03674842 and a registered office at 2 Kingdom Street, > London, W2 6BD. > > *Confidentiality Notice: *This message is private and may contain > confidential, proprietary and legally privileged information. If you have > received this message in error, please notify us and remove it from your > system and note that you must not copy, distribute or take any action in > reliance on it. Any unauthorised use or disclosure of the contents of this > message is not permitted and may be unlawful. > > *Disclaimer:* Email messages may be subject to delays, interception, > non-delivery and unauthorised alterations. Therefore, information expressed > in this message is not given or endorsed by AstraZeneca UK Limited unless > otherwise notified by an authorised representative independent of this > message. No contractual relationship is created by this message by any > person unless specifically indicated by agreement in writing other than > email. > > *Monitoring: *AstraZeneca UK Limited may monitor email traffic data and > content for the purposes of the prevention and detection of crime, ensuring > the security of our computer systems and checking compliance with our Code > of Conduct and policies. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- Zach Giles zgiles at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From luke.raimbach at oerc.ox.ac.uk Fri Mar 27 12:16:16 2015 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Fri, 27 Mar 2015 12:16:16 +0000 Subject: [gpfsug-discuss] mmunlinkfileset problems Message-ID: Hi All, I'm having a problem where unlinking a fileset is taking 7 or 8 minutes to complete. We are running 3.5.0.22 and the cluster is small (9 nodes: 8 Linux plus 1 Windows node), one file system remote mounted by a 3-node Linux cluster, and generally not very busy. I wouldn't expect mmunlinkfileset to require the file system to quiesce (and even if it did 8 minutes would seem like an awfully long time). Previous unlink commands (on version 3.5.0.19) have returned within milliseconds. This very long running command appeared to pause IO for a sufficiently long time that cNFS clients timed out too and various services elsewhere crashed. Is there any reason why unlinking a fileset would take this long? Also, I can't find anywhere a list of commands which might require the FS to quiesce - is this information available somewhere? Cheers, Luke. -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 From jonathan at buzzard.me.uk Fri Mar 27 12:49:38 2015 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Fri, 27 Mar 2015 12:49:38 +0000 Subject: [gpfsug-discuss] mmunlinkfileset problems In-Reply-To: References: Message-ID: <1427460578.3553.153.camel@buzzard.phy.strath.ac.uk> On Fri, 2015-03-27 at 12:16 +0000, Luke Raimbach wrote: > Hi All, > > I'm having a problem where unlinking a fileset is taking 7 or 8 minutes > to complete. We are running 3.5.0.22 and the cluster is small (9 nodes: > 8 Linux plus 1 Windows node), one file system remote mounted by a > 3-node Linux cluster, and generally not very busy. > > I wouldn't expect mmunlinkfileset to require the file system to quiesce > (and even if it did 8 minutes would seem like an awfully long time). > Previous unlink commands (on version 3.5.0.19) have returned within > milliseconds. If there is a waiter around mmunlinkfileset can block even with an empty fileset and turn your file system unresponsive. > > This very long running command appeared to pause IO for a sufficiently > long time that cNFS clients timed out too and various services > elsewhere crashed. > Yep, lesson for the day only unlink filesets when the file system is quiet, aka a maintenance window. > Is there any reason why unlinking a fileset would take this long? Also, > I can't find anywhere a list of commands which might require the FS to > quiesce - is this information available somewhere? > Deleting snapshots, at least in the past. It might be that guidance from IBM has changed but once you have been stung by this you kind of thing you don't like to try it again any time soon. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From luke.raimbach at oerc.ox.ac.uk Fri Mar 27 13:27:20 2015 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Fri, 27 Mar 2015 13:27:20 +0000 Subject: [gpfsug-discuss] mmunlinkfileset problems In-Reply-To: <1427460578.3553.153.camel@buzzard.phy.strath.ac.uk> References: <1427460578.3553.153.camel@buzzard.phy.strath.ac.uk> Message-ID: > > Is there any reason why unlinking a fileset would take this long? > > Also, I can't find anywhere a list of commands which might require the > > FS to quiesce - is this information available somewhere? > > > > Deleting snapshots, at least in the past. It might be that guidance from IBM > has changed but once you have been stung by this you kind of thing you > don't like to try it again any time soon. Precisely why I'm after a list of commands that might require quiescence! I know that snapshot operations (generally) and deleting filesets require a consistent view so will pause IO, but unlinking (in my mind) wouldn't need this... I'd probably want to generate lists of 'cleanup' commands that might interrupt IO and run these in maintenance windows as you say. Gather up all the filesets that want deleting and do them once a week / whenever. Cheers, Luke. From jonathan at buzzard.me.uk Fri Mar 27 13:39:19 2015 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Fri, 27 Mar 2015 13:39:19 +0000 Subject: [gpfsug-discuss] mmunlinkfileset problems In-Reply-To: References: <1427460578.3553.153.camel@buzzard.phy.strath.ac.uk> Message-ID: <1427463559.3553.156.camel@buzzard.phy.strath.ac.uk> On Fri, 2015-03-27 at 13:27 +0000, Luke Raimbach wrote: > > > Is there any reason why unlinking a fileset would take this long? > > > Also, I can't find anywhere a list of commands which might require the > > > FS to quiesce - is this information available somewhere? > > > > > > > Deleting snapshots, at least in the past. It might be that guidance from IBM > > has changed but once you have been stung by this you kind of thing you > > don't like to try it again any time soon. > > Precisely why I'm after a list of commands that might require > quiescence! I know that snapshot operations (generally) and deleting > filesets require a consistent view so will pause IO, but unlinking (in > my mind) wouldn't need this... > mmdeldisk is another. From what I can make out anything that takes away from the file system is a potential problem. Anything that adds to the file system is just peachy fine to be run at any time. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From adam.huffman at crick.ac.uk Mon Mar 2 09:24:59 2015 From: adam.huffman at crick.ac.uk (Adam Huffman) Date: Mon, 2 Mar 2015 09:24:59 +0000 Subject: [gpfsug-discuss] Introduction Message-ID: Hello A couple of weeks ago I joined Bruno Silva?s HPC team at the Francis Crick Institute, with special responsibility for HPC, OpenStack and virtualization. I?m very much a GPFS novice so I?m hoping to be able to draw on the knowledge in this group, while hopefully being able to help others with OpenStack. As Bruno stated in his message, we?re particularly interested in how to present GPFS to instances securely. I?ve read the discussion from November on this list, which didn?t seem to come to any firm conclusions. Has anyone involved then made substantial progress since? Cheers, Adam ? Adam Huffman Senior HPC & Virtualization Systems Engineer The Francis Crick Institute Gibbs Building 215 Euston Road London NW1 2BE T: E: adam.huffman at crick.ac.uk W: www.crick.ac.uk [Description: Description: ZA102637861][Description: Description: Twitter] The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no.06885462, with its registered office at 215 Euston Road, London NW1 2BE -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 158254D2-CDA8-43E5-96C1-A0AAEB10314B.png Type: image/png Size: 1151 bytes Desc: 158254D2-CDA8-43E5-96C1-A0AAEB10314B.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: B09B8AB7-5CA2-4DE7-BFE7-9D8F1B2A5166.png Type: image/png Size: 1211 bytes Desc: B09B8AB7-5CA2-4DE7-BFE7-9D8F1B2A5166.png URL: From viccornell at gmail.com Mon Mar 2 09:32:57 2015 From: viccornell at gmail.com (Vic Cornell) Date: Mon, 2 Mar 2015 09:32:57 +0000 Subject: [gpfsug-discuss] Introduction In-Reply-To: References: Message-ID: Hi Adam, I guess that one of the things that would help push it forward is a definition of what "secure" means to you. Regards, Vic On 2 Mar 2015, at 09:24, Adam Huffman wrote: > Hello > > A couple of weeks ago I joined Bruno Silva?s HPC team at the Francis Crick Institute, with special responsibility for HPC, OpenStack and virtualization. I?m very much a GPFS novice so I?m hoping to be able to draw on the knowledge in this group, while hopefully being able to help others with OpenStack. > > As Bruno stated in his message, we?re particularly interested in how to present GPFS to instances securely. I?ve read the discussion from November on this list, which didn?t seem to come to any firm conclusions. Has anyone involved then made substantial progress since? > > Cheers, > Adam > > ? > > Adam Huffman > Senior HPC & Virtualization Systems Engineer > The Francis Crick Institute > Gibbs Building > 215 Euston Road > London NW1 2BE > > T: > E: adam.huffman at crick.ac.uk > W: www.crick.ac.uk > <158254D2-CDA8-43E5-96C1-A0AAEB10314B.png> > > The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no.06885462, with its registered office at 215 Euston Road, London NW1 2BE > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam.huffman at crick.ac.uk Mon Mar 2 09:39:36 2015 From: adam.huffman at crick.ac.uk (Adam Huffman) Date: Mon, 2 Mar 2015 09:39:36 +0000 Subject: [gpfsug-discuss] Introduction In-Reply-To: References: Message-ID: Hi Vic, Re-emphasising that I?m still very much learning about GPFS, one of the approaches being discussed is running the GPFS client inside the instances. The concern here is over the case where users have root privileges inside their instances (a pretty common assumption for those used to AWS, for example) and the implications this may have for GPFS. Does it mean there would be a risk of spoofing? Cheers, Adam From: Vic Cornell Reply-To: gpfsug main discussion list Date: Monday, 2 March 2015 09:32 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Introduction Hi Adam, I guess that one of the things that would help push it forward is a definition of what "secure" means to you. Regards, Vic On 2 Mar 2015, at 09:24, Adam Huffman > wrote: Hello A couple of weeks ago I joined Bruno Silva?s HPC team at the Francis Crick Institute, with special responsibility for HPC, OpenStack and virtualization. I?m very much a GPFS novice so I?m hoping to be able to draw on the knowledge in this group, while hopefully being able to help others with OpenStack. As Bruno stated in his message, we?re particularly interested in how to present GPFS to instances securely. I?ve read the discussion from November on this list, which didn?t seem to come to any firm conclusions. Has anyone involved then made substantial progress since? Cheers, Adam ? Adam Huffman Senior HPC & Virtualization Systems Engineer The Francis Crick Institute Gibbs Building 215 Euston Road London NW1 2BE T: E: adam.huffman at crick.ac.uk W: www.crick.ac.uk <158254D2-CDA8-43E5-96C1-A0AAEB10314B.png> The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no.06885462, with its registered office at 215 Euston Road, London NW1 2BE _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From luke.raimbach at oerc.ox.ac.uk Mon Mar 2 19:21:25 2015 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Mon, 2 Mar 2015 19:21:25 +0000 Subject: [gpfsug-discuss] Introduction In-Reply-To: References: Message-ID: HI Adam, We run virtualised GPFS client nodes in a VMware cluster here at Oxford e-Research Centre. We had a requirement where one research group wanted root access to their VMs but also wanted fast direct access to their data on our GPFS cluster. The technical setup was (relatively) simple. We span up a small three-node virtual GPFS cluster (with no file system of its own). We then used the multi-cluster feature of GPFS to allow this small virtual cluster to join our main file system cluster which now gives that group very good IO performance. However, the problem of spoofing you mention is relevant ? we installed the virtual cluster nodes for the research group and put our Active Directory client on them. We also used the root-squash configuration option of the multi-cluster setup to prevent remote-cluster root access in to our file system. We also agreed with the research group that they would nominate one Administrator to have root access in their cluster and that they would maintain the AAA framework we put in place. We have to trust the group?s Administrator not to fiddle his UID or to let others escalate their privileges. If you were letting untrusted root users spin up Stacks, then you could still run GPFS clients in the OpenStack instance nodes to give them fast access to their data. Here are some musings on a recipe (others please feel free to pull these ideas to pieces): 1. Start with Cluster A ? your main production GPFS file system. It has GPFS device name /gpfs. 2. Pretend you have lots of money for extra disk to go under your OpenStack cluster (say you buy something like a DDN SFA7700 with a couple of expansion shelves and fill it up with 4TB drives ? 180 drives). 3. Drive this disk array with two, preferably four (or however many you want, really) decent NSD servers. Configure quorum nodes, etc. appropriately. Call this Cluster B. 4. Carve up the disk array in to something like 30 x RAID6 (4 + 2) LUNs and configure them as GPFS NSDs; but don?t create a file system (line up the stripe sizes etc. and choose a nice block size, etc. etc.)? 5. Put the GPFS metadata on some SSD NSDs somewhere. I like putting it on SSDs in the NSD server nodes and replicating it. Other people like putting it in their disk arrays. 6. As part of someone spinning up a Stack, get some scripts to do the following ?magic?: a. Connect to Cluster A and find out how big their target data-set is. b. Connect to Cluster B and create a new GPFS file system with a reasonable (dependent on the above result) number of NSD disks. Call this new GPFS device something unique other that /gpfs e.g. /gpfs0001. You could slice bits off your SSDs for the metadata NSDs in each file system you create in this manner (if you haven?t got many SSDs). c. As part of a new Stack, provide a few (three, say) GPFS quorum nodes that you?ve configured. Call this Cluster C. Add the rest of the stack instances to Cluster C. No File System. d. Pop back over to Cluster A. Export their target data-set from Cluster A using AFM (over GPFS or NFS ? pick your favourite: GPFS is probably better performant but means you need Cluster A to stay online). e. Now return to Cluster B. Import the target data to a local AFM cache on Cluster B?s new file system. Name the AFM file-set whatever you like, but link it in to the Cluster B /gpfs0001 namespace at the same level as it is in Cluster A. For example Cluster A: /gpfs/projects/dataset01 imports to an AFM fileset in Cluster B named userdataset01. Link this under /gpfs0001/projects/dataset01. f. Configure multi-cluster support on Cluster B to export GPFS device /gpfs0001 to Cluster C. Encrypt traffic if you want a headache. g. Configure multi-cluster support on Cluster C to remote mount Cluster B:/gpfs0001 as local device /gpfs. 7. You now have fast GPFS access to this user dataset *only* using GPFS clients inside the OpenStack instance nodes. You have also preserved the file system namespace in Cluster C?s instance nodes. If you only want to run over the data in the stack instances, you could pre-fetch the entire data-set using AFM Control from Cluster A in to the Cluster B file-set (if it?s big enough). 8. Now your users are finished and want to destroy the stack ? you need some more script ?magic?: a. Dismount the file system /gpfs in Cluster C. b. Connect to Cluster B and use AFM control to flush all the data back home to Cluster A. c. Unlink the file-set in Cluster B and force delete it; then delete the file system to free the NSDs back to the pool available to Cluster B. d. Connect back to Cluster A and unexport the original data-set directory structure. e. Throw away the VMs in the stack Things to worry about: ? Inode space will be different if users happen to be working on the data in Cluster A and Cluster C and want to know about inodes. GPFS XATTRS are preserved. ? If you use AFM over NFS because Cluster A and B are far away from each other and laggy, then there?s no locking with your AFM cache running as an Independent Writer. Writes at home (Cluster A) and in cache (Cluster B from Cluster C) will be nondeterministic. Your users will need to know this to avoid disappointment. ? If you use AFM over GPFS because Cluster A and B are near to each other and have a fast network, then there might still not be locking, but if Cluster A goes offline, it will put the AFM cache in to a ?dismounted? state. ? If your users want access to other parts of the Cluster A /gpfs namespace within their stack instances (because you have tools they use or they want to see other stuff), you can export them as read-only to a read-only AFM cache in Cluster B and they will be able to see things in Cluster C provided you link the AFM caches in the right places. Remember they have root access here. ? AFM updates are sent from cache to home as root so users can potentially overflow their quota at the Cluster A home site (root doesn?t care about quotas at home). ? Other frightening things might happen that I?ve not thought about. Hope this helps! Luke -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Adam Huffman Sent: 02 March 2015 09:40 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Introduction Hi Vic, Re-emphasising that I?m still very much learning about GPFS, one of the approaches being discussed is running the GPFS client inside the instances. The concern here is over the case where users have root privileges inside their instances (a pretty common assumption for those used to AWS, for example) and the implications this may have for GPFS. Does it mean there would be a risk of spoofing? Cheers, Adam From: Vic Cornell Reply-To: gpfsug main discussion list Date: Monday, 2 March 2015 09:32 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Introduction Hi Adam, I guess that one of the things that would help push it forward is a definition of what "secure" means to you. Regards, Vic On 2 Mar 2015, at 09:24, Adam Huffman > wrote: Hello A couple of weeks ago I joined Bruno Silva?s HPC team at the Francis Crick Institute, with special responsibility for HPC, OpenStack and virtualization. I?m very much a GPFS novice so I?m hoping to be able to draw on the knowledge in this group, while hopefully being able to help others with OpenStack. As Bruno stated in his message, we?re particularly interested in how to present GPFS to instances securely. I?ve read the discussion from November on this list, which didn?t seem to come to any firm conclusions. Has anyone involved then made substantial progress since? Cheers, Adam ? Adam Huffman Senior HPC & Virtualization Systems Engineer The Francis Crick Institute Gibbs Building 215 Euston Road London NW1 2BE T: E: adam.huffman at crick.ac.uk W: www.crick.ac.uk <158254D2-CDA8-43E5-96C1-A0AAEB10314B.png> The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no.06885462, with its registered office at 215 Euston Road, London NW1 2BE _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger_eriksson at se.ibm.com Thu Mar 12 08:25:22 2015 From: roger_eriksson at se.ibm.com (Roger Eriksson) Date: Thu, 12 Mar 2015 09:25:22 +0100 Subject: [gpfsug-discuss] GPFS UG 2015 date potential alteration In-Reply-To: <54EDD5E8.7090905@gpfsug.org> References: <54EDD5E8.7090905@gpfsug.org> Message-ID: Hi, has the GPFS User Group Meeting date been settled yet ? //Roger _________________________________________________________________ H?lsningar / Regards __________________________________________________________________ Roger Eriksson IBM Systems Lab Services Senior Accredited Product Services Professional XIV / Flashsystems / SVC / Storwize / SONAS / SAN EMEA Storage Competence Center (ESCC), Mainz, Germany IBM Svenska AB Isafjordsgatan 1 SE-164 92 Stockholm, Sweden Mobile +46(0)707933518 Email: roger_eriksson at se.ibm.com Notes: Roger Eriksson/Sweden/IBM at IBMSE Need help from EMEA Storage Competence Center ? IBMer ESCC @ Intranet --> http://escc.mainz.de.ibm.com/escc4u BPs ESCC @ Internet --> http://www.ibm.com/partnerworld/page/pw_com_bp_ecampus From: "Jez Tucker (Chair)" To: gpfsug main discussion list Date: 25/02/2015 15:02 Subject: [gpfsug-discuss] GPFS UG 2015 date potential alteration Sent by: gpfsug-discuss-bounces at gpfsug.org Alteration. Due to a scheduling issue, we may have to potentially move the date from 1st May. We will confirm this as soon as possible. Apologies. Jez _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss S?vida annat inte anges ovan: / Unless stated otherwise above: IBM Svenska AB Organisationsnummer: 556026-6883 Adress: 164 92 Stockholm -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 7526 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 7414 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 8944 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 9733 bytes Desc: not available URL: From chair at gpfsug.org Thu Mar 12 16:24:52 2015 From: chair at gpfsug.org (Jez Tucker (Chair)) Date: Thu, 12 Mar 2015 16:24:52 +0000 Subject: [gpfsug-discuss] GPFS UG 2015 date potential alteration In-Reply-To: References: <54EDD5E8.7090905@gpfsug.org> Message-ID: <5501BDD4.5060506@gpfsug.org> Hello all A fair question. We are working to a date of Wednesday 20th May. So please pencil this. The last pieces of paper work are being put in place, so please don't anybody book flights / expenses until this has been done. I would like to be in a position to have a definitive update on Monday. Jez On 12/03/15 08:25, Roger Eriksson wrote: > Hi, > > has the GPFS User Group Meeting date been settled yet ? > > //Roger > */_________________________________________________________________/* > */H?lsningar / Regards/* > */__________________________________________________________________/* > */Roger Eriksson/* > IBM Systems Lab Services > Senior Accredited Product Services Professional > XIV / Flashsystems / SVC / Storwize / SONAS / SAN > EMEA Storage Competence Center (ESCC), Mainz, Germany > IBM Svenska AB > Isafjordsgatan 1 > SE-164 92 Stockholm, Sweden > Mobile +46(0)707933518 > Email: _roger_eriksson at se.ibm.com_ > Notes: Roger Eriksson/Sweden/IBM at IBMSE > > > *Need help from EMEA Storage Competence Center ? * > *IBMer ESCC @ Intranet --> **http://escc.mainz.de.ibm.com/escc4u* > *BPs ESCC @ Internet --> > **http://www.ibm.com/partnerworld/page/pw_com_bp_ecampus* > > > > From: "Jez Tucker (Chair)" > To: gpfsug main discussion list > Date: 25/02/2015 15:02 > Subject: [gpfsug-discuss] GPFS UG 2015 date potential alteration > Sent by: gpfsug-discuss-bounces at gpfsug.org > ------------------------------------------------------------------------ > > > > Alteration. > > Due to a scheduling issue, we may have to potentially move the date > from 1st May. > > We will confirm this as soon as possible. > > Apologies. > > Jez > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > S?vida annat inte anges ovan: / Unless stated otherwise above: > IBM Svenska AB > Organisationsnummer: 556026-6883 > Adress: 164 92 Stockholm > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 7526 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 7414 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 8944 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 9733 bytes Desc: not available URL: From luke.raimbach at oerc.ox.ac.uk Fri Mar 13 14:44:33 2015 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Fri, 13 Mar 2015 14:44:33 +0000 Subject: [gpfsug-discuss] GPFS Journal Logs Message-ID: Hi, I'm interested in knowing more about the internal log journals GPFS uses to track file system activity from client nodes. My understanding is that when a file is opened for writing on a node, that (first) node becomes the 'metanode' for the file and is responsible for making updates to the file's metadata. Does the matanode also have responsibility for tracking changes made by other nodes that open the same file in its own journal, or are these recorded in each nodes own journal? I'm also interested in what happens when a node fails and its logs are replayed by the FS manager... what happens if that node is acting as metanode for a lot of open files which are also open elsewhere for writing on other nodes? If in the above scenario the metanode is handling metadata updates and tracking other changes in its journal, do all other nodes need to wait for the metanode to be ejected and have its journal replayed before they can continue? What is the process for selecting a successor metanode for the open files? Another related question I have is about initial sizing when creating a file system. Presumably the estimate for the total number of nodes mounting the file system affects the structure / allocation of internal journal logs? How are these arranged and is there any advice to overestimate this number if you intend to add more nodes at a future time? In the same vein, do multicluster nodes that join another remote stripegroup get allocated internal journal space as if they were new nodes in the remote cluster? Are these nodes allowed to be metanodes for files in a remote cluster? What happens when these nodes fail? Any insights greatly appreciated! Cheers, Luke. -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 From richard.lefebvre at calculquebec.ca Fri Mar 13 18:20:14 2015 From: richard.lefebvre at calculquebec.ca (Richard Lefebvre) Date: Fri, 13 Mar 2015 14:20:14 -0400 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. Message-ID: Hi, I have a GPFS cluster (v3.5). Last month, one of the filesystem had a problem with one of it's NSD. With the help tech support, the problem was fix and everything is OK data wise. The thing is when the problem occurred the filesystem was offline and once the filesystem was back to being OK, we were able to remount it. The problem is that now, every time a node is restarted, the filesystem is not mounted automatically as before. The other filesystems mount automatically. I did a mmlsfs on the file system and -A is at yes. Can someone tell my what I'm missing here. I'm sure it is something simple, but I'm drawing a blank right now. Richard -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) -------------- next part -------------- An HTML attachment was scrubbed... URL: From chair at gpfsug.org Mon Mar 16 14:02:09 2015 From: chair at gpfsug.org (Jez Tucker (Chair)) Date: Mon, 16 Mar 2015 14:02:09 +0000 Subject: [gpfsug-discuss] GPFS UG Meeting Agenda Message-ID: <5506E261.8000403@gpfsug.org> Hello all We have an internal agenda for the forthcoming UG. Before we publish this, I thought it prudent to solicit requests from our user base so that we might not miss anything important. So; - What's your burning issue, what's the one thing you really want to see? Regards, Jez From janfrode at tanso.net Mon Mar 16 15:11:13 2015 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Mon, 16 Mar 2015 16:11:13 +0100 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: Message-ID: Not much details to go by here, but I know I've had problems with SElinux blocking for getting filesystems mounted automatically at boot, while manually mounting afterwards worked fine. I fixed that by disabling automatic startup of gpfs from standard initscripts, and used this in rc.local instead: # make sure GPFS is not confined by selinux policies: runcon -t unconfined_t /usr/lpp/mmfs/bin/mmstartup -jf On Fri, Mar 13, 2015 at 7:20 PM, Richard Lefebvre < richard.lefebvre at calculquebec.ca> wrote: > Hi, > > I have a GPFS cluster (v3.5). Last month, one of the filesystem had a > problem with one of it's NSD. With the help tech support, the problem was > fix and everything is OK data wise. The thing is when the problem occurred > the filesystem was offline and once the filesystem was back to being OK, we > were able to remount it. The problem is that now, every time a node is > restarted, the filesystem is not mounted automatically as before. The other > filesystems mount automatically. I did a mmlsfs on the file system and -A > is at yes. Can someone tell my what I'm missing here. I'm sure it is > something simple, but I'm drawing a blank right now. > > Richard > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From YARD at il.ibm.com Mon Mar 16 15:14:25 2015 From: YARD at il.ibm.com (Yaron Daniel) Date: Mon, 16 Mar 2015 17:14:25 +0200 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: Message-ID: Hi In case of Linux - Check also : #chkconfig --list |grep gpfs Regards Yaron Daniel 94 Em Ha'Moshavot Rd Server, Storage and Data Services - Team Leader Petach Tiqva, 49527 Global Technology Services Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com IBM Israel From: Jan-Frode Myklebust To: gpfsug main discussion list Date: 03/16/2015 05:11 PM Subject: Re: [gpfsug-discuss] GPFS file system won't mount automatically anymore. Sent by: gpfsug-discuss-bounces at gpfsug.org Not much details to go by here, but I know I've had problems with SElinux blocking for getting filesystems mounted automatically at boot, while manually mounting afterwards worked fine. I fixed that by disabling automatic startup of gpfs from standard initscripts, and used this in rc.local instead: # make sure GPFS is not confined by selinux policies: runcon -t unconfined_t /usr/lpp/mmfs/bin/mmstartup -jf On Fri, Mar 13, 2015 at 7:20 PM, Richard Lefebvre < richard.lefebvre at calculquebec.ca> wrote: Hi, I have a GPFS cluster (v3.5). Last month, one of the filesystem had a problem with one of it's NSD. With the help tech support, the problem was fix and everything is OK data wise. The thing is when the problem occurred the filesystem was offline and once the filesystem was back to being OK, we were able to remount it. The problem is that now, every time a node is restarted, the filesystem is not mounted automatically as before. The other filesystems mount automatically. I did a mmlsfs on the file system and -A is at yes. Can someone tell my what I'm missing here. I'm sure it is something simple, but I'm drawing a blank right now. Richard -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1851 bytes Desc: not available URL: From luke.raimbach at oerc.ox.ac.uk Tue Mar 17 12:52:25 2015 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Tue, 17 Mar 2015 12:52:25 +0000 Subject: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans Message-ID: Hi Experts, I'm worrying about creating independent filesets which start to get in the way of one another: Let's pretend I create independent "fileset1" with a new inode space and preallocate say 1 million inodes. I start pushing files in to fileset1. I also start pushing files in to the root fileset. When fileset1 fills up, and because I've not specified a maximum number of inodes at creation time, presumably the file system manager gets on the case and preallocates more inodes as needed for fileset1. But while fileset1 has been filling up, I've also been filling up the root fileset (pretend I put 5 million files in there by the time fileset1 filled up). Now what happens? I'm guessing the FS manager starts chopping out sections of the root fileset beyond the 6 million region and marking them as independent and belonging to fileset1, in much the same way as it extends the allocated inode space when the file system starts running low. I hope this happens anyway. Let's say it does. So, now what about performance when running policy engine scans? I'm wanting to speed things up by restricting the policy scan to fileset1 (because I know that's where I put my special files). So I write the rules to select files and give them to mmapplypolicy. Does performance go down the drain as my once neatly arranged inode space is fragmented by not having enough inodes preallocated to begin with? Imagine fileset1 and the root fileset filling continuously at a similar rate and constantly "overtaking" one another in the inode space... does the performance advantage of independence start to go away rapidly? Cheers, Luke. -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 From makaplan at us.ibm.com Tue Mar 17 14:22:58 2015 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 17 Mar 2015 10:22:58 -0400 Subject: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans In-Reply-To: References: Message-ID: Luke, Thanks for your question. Independent filesets and their inodes are not implemented the way you might be imagining or guessing. Suppose you have two independent filesets "root" and "fset2" in the same GPFS file system. It is true that all the inode records (typically 512 bytes each - see mmcrfs) go into the same special file. BUT if you look at any given metadata allocation block --metadata-block-size (defaults to 256K) you'll only see inodes for either "root" or "fset2" not both in the same block. Moreover we try to "pre"-allocate several blocks of inodes for the same independent fileset contiguously - so that typically GPFS can do one seek and then read several blocks of inodes from the same independent fileset. So there can be performance advantages to using independent filesets and restricting your mmapplypolicy scans to just the fileset you need. To gain maximal advantage, use the following form of the command: mmapplypolicy /gpfs/path-to-the-directory-tree-I-want-to-scan --scope {fileset | inodespace } ... An inodespace is the set of all inodes in an independent fileset. An inodespace may contain several "dependent" filesets. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 21994 bytes Desc: not available URL: From zgiles at gmail.com Tue Mar 17 15:49:06 2015 From: zgiles at gmail.com (Zachary Giles) Date: Tue, 17 Mar 2015 11:49:06 -0400 Subject: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans In-Reply-To: References: Message-ID: I've always wondered also what the decision point is to make new inode tables per fileset. Sounds like the main benefit is scope of mmapplypolicy ( and other commands maybe ) and subblocks being grouped together for performance reasons. Is that right? Are there other benefits like portability or grouping a fileset's inode table into specific failure groups (in the system pool) etc? Any downsides? ( for example: now you have more overall metadata reads.. or only a limited number of inode tables.. or total number of inodes in the system is higher due to overallocation on N tables ? ) -Zach On Tue, Mar 17, 2015 at 10:22 AM, Marc A Kaplan wrote: > Luke, > > Thanks for your question. Independent filesets and their inodes are not > implemented the way you might be imagining or guessing. > > Suppose you have two independent filesets "root" and "fset2" in the same > GPFS file system. > It is true that all the inode records (typically 512 bytes each - see > mmcrfs) go into the same special file. BUT if you look at any given > metadata allocation block --metadata-block-size (defaults to 256K) you'll > only see inodes for either "root" or "fset2" not both in the same block. > > Moreover we try to "pre"-allocate several blocks of inodes for the same > independent fileset contiguously - so that typically GPFS can do one seek > and then read several blocks of inodes from the same independent fileset. > > So there can be performance advantages to using independent filesets and > restricting your mmapplypolicy scans to just the fileset you need. > To gain maximal advantage, use the following form of the command: > > mmapplypolicy /gpfs/path-to-the-directory-tree-I-want-to-scan --scope > {fileset | inodespace } ... > > > An inodespace is the set of all inodes in an independent fileset. An > inodespace may contain several "dependent" filesets. > > > > > [image: Marc A Kaplan] > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- Zach Giles zgiles at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 21994 bytes Desc: not available URL: From Paul.Sanchez at deshaw.com Tue Mar 17 16:01:21 2015 From: Paul.Sanchez at deshaw.com (Sanchez, Paul) Date: Tue, 17 Mar 2015 16:01:21 +0000 Subject: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans In-Reply-To: References: Message-ID: <201D6001C896B846A9CFC2E841986AC145360AA6@mailnycmb2a.winmail.deshaw.com> It?s also required for per-fileset snapshots, which are useful if you have differing snapshot requirements or want the ability to flush high-churn snapshots that are consuming an unexpected amount of space, but with better resolution than a whole filesystem. -Paul Sanchez From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Zachary Giles Sent: Tuesday, March 17, 2015 11:49 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans I've always wondered also what the decision point is to make new inode tables per fileset. Sounds like the main benefit is scope of mmapplypolicy ( and other commands maybe ) and subblocks being grouped together for performance reasons. Is that right? Are there other benefits like portability or grouping a fileset's inode table into specific failure groups (in the system pool) etc? Any downsides? ( for example: now you have more overall metadata reads.. or only a limited number of inode tables.. or total number of inodes in the system is higher due to overallocation on N tables ? ) -Zach On Tue, Mar 17, 2015 at 10:22 AM, Marc A Kaplan > wrote: Luke, Thanks for your question. Independent filesets and their inodes are not implemented the way you might be imagining or guessing. Suppose you have two independent filesets "root" and "fset2" in the same GPFS file system. It is true that all the inode records (typically 512 bytes each - see mmcrfs) go into the same special file. BUT if you look at any given metadata allocation block --metadata-block-size (defaults to 256K) you'll only see inodes for either "root" or "fset2" not both in the same block. Moreover we try to "pre"-allocate several blocks of inodes for the same independent fileset contiguously - so that typically GPFS can do one seek and then read several blocks of inodes from the same independent fileset. So there can be performance advantages to using independent filesets and restricting your mmapplypolicy scans to just the fileset you need. To gain maximal advantage, use the following form of the command: mmapplypolicy /gpfs/path-to-the-directory-tree-I-want-to-scan --scope {fileset | inodespace } ... An inodespace is the set of all inodes in an independent fileset. An inodespace may contain several "dependent" filesets. [Marc A Kaplan] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Zach Giles zgiles at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 21994 bytes Desc: image001.gif URL: From zgiles at gmail.com Tue Mar 17 16:04:25 2015 From: zgiles at gmail.com (Zachary Giles) Date: Tue, 17 Mar 2015 12:04:25 -0400 Subject: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans In-Reply-To: <201D6001C896B846A9CFC2E841986AC145360AA6@mailnycmb2a.winmail.deshaw.com> References: <201D6001C896B846A9CFC2E841986AC145360AA6@mailnycmb2a.winmail.deshaw.com> Message-ID: Ah. that's a good point about the snapshots. On Tue, Mar 17, 2015 at 12:01 PM, Sanchez, Paul wrote: > It?s also required for per-fileset snapshots, which are useful if you > have differing snapshot requirements or want the ability to flush > high-churn snapshots that are consuming an unexpected amount of space, but > with better resolution than a whole filesystem. > > > > -Paul Sanchez > > > > *From:* gpfsug-discuss-bounces at gpfsug.org [mailto: > gpfsug-discuss-bounces at gpfsug.org] *On Behalf Of *Zachary Giles > *Sent:* Tuesday, March 17, 2015 11:49 AM > *To:* gpfsug main discussion list > *Subject:* Re: [gpfsug-discuss] Fragmented Inode Space and Performance of > Policy Scans > > > > I've always wondered also what the decision point is to make new inode > tables per fileset. > > > > Sounds like the main benefit is scope of mmapplypolicy ( and other > commands maybe ) and subblocks being grouped together for performance > reasons. Is that right? > > Are there other benefits like portability or grouping a fileset's inode > table into specific failure groups (in the system pool) etc? > > Any downsides? ( for example: now you have more overall metadata reads.. > or only a limited number of inode tables.. or total number of inodes in the > system is higher due to overallocation on N tables ? ) > > > > -Zach > > > > > > On Tue, Mar 17, 2015 at 10:22 AM, Marc A Kaplan > wrote: > > Luke, > > Thanks for your question. Independent filesets and their inodes are not > implemented the way you might be imagining or guessing. > > Suppose you have two independent filesets "root" and "fset2" in the same > GPFS file system. > It is true that all the inode records (typically 512 bytes each - see > mmcrfs) go into the same special file. BUT if you look at any given > metadata allocation block --metadata-block-size (defaults to 256K) you'll > only see inodes for either "root" or "fset2" not both in the same block. > > Moreover we try to "pre"-allocate several blocks of inodes for the same > independent fileset contiguously - so that typically GPFS can do one seek > and then read several blocks of inodes from the same independent fileset. > > So there can be performance advantages to using independent filesets and > restricting your mmapplypolicy scans to just the fileset you need. > To gain maximal advantage, use the following form of the command: > > mmapplypolicy /gpfs/path-to-the-directory-tree-I-want-to-scan --scope > {fileset | inodespace } ... > > > An inodespace is the set of all inodes in an independent fileset. An > inodespace may contain several "dependent" filesets. > > > > > [image: Marc A Kaplan] > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > -- > > Zach Giles > zgiles at gmail.com > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- Zach Giles zgiles at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 21994 bytes Desc: not available URL: From chair at gpfsug.org Tue Mar 17 16:47:43 2015 From: chair at gpfsug.org (Jez Tucker (Chair)) Date: Tue, 17 Mar 2015 16:47:43 +0000 Subject: [gpfsug-discuss] GPFS User Group 2015 Meeting Date confirmed - RSVP Message-ID: <55085AAF.8050108@gpfsug.org> Hello all I can now confirm that 2015 GPFS User Group Meeting will be on May 20th, in tandem with SPXXL. For full details please see: http://www.gpfsug.org/gpfs-user-group-2015-invite/ RSVP to secretary at gpfsug.org See you there. Jez From richard.lefebvre at calculquebec.ca Thu Mar 19 22:32:58 2015 From: richard.lefebvre at calculquebec.ca (Richard Lefebvre) Date: Thu, 19 Mar 2015 18:32:58 -0400 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: Message-ID: Thanks for those who replied. One thing I didn't mention correctly is that I have several gpfs mount say: /gpfs1 /gpfs2 /gpfs3 Say it was gpfs2 that had the NSD problem. So now when I reboot a node, /gpfs1 and /gpfs3 mount, but /gpfs2 doesn't, I have to manually type mmmount gpfs2 on the node to get it mounted after the boot. So I don't think that it is a problem with mounting within linux boot sequence. But a flag than I'm missing in the gpfs setup. Or status of the filesystem that is not set OK that I cannot figure out. The output of mmlsfs of a system that mounts compared the the one that doesn't are the same except for mount point, NSDs and creation time. Is there another "mm..." command with which to check the status of a filesystem? Richard 2015-03-13 14:20 GMT-04:00 Richard Lefebvre < richard.lefebvre at calculquebec.ca>: > Hi, > > I have a GPFS cluster (v3.5). Last month, one of the filesystem had a > problem with one of it's NSD. With the help tech support, the problem was > fix and everything is OK data wise. The thing is when the problem occurred > the filesystem was offline and once the filesystem was back to being OK, we > were able to remount it. The problem is that now, every time a node is > restarted, the filesystem is not mounted automatically as before. The other > filesystems mount automatically. I did a mmlsfs on the file system and -A > is at yes. Can someone tell my what I'm missing here. I'm sure it is > something simple, but I'm drawing a blank right now. > > Richard > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Thu Mar 19 22:35:38 2015 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Thu, 19 Mar 2015 22:35:38 +0000 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: Message-ID: Check "mmlsdisk". Are all disks for the filesystem ok? -jf On tor. 19. mar. 2015 at 23.33 Richard Lefebvre < richard.lefebvre at calculquebec.ca> wrote: > Thanks for those who replied. One thing I didn't mention correctly is that > I have several gpfs mount say: > /gpfs1 > /gpfs2 > /gpfs3 > > Say it was gpfs2 that had the NSD problem. So now when I reboot a node, > /gpfs1 and /gpfs3 mount, but /gpfs2 doesn't, I have to manually type > mmmount gpfs2 on the node to get it mounted after the boot. > > So I don't think that it is a problem with mounting within linux boot > sequence. But a flag than I'm missing in the gpfs setup. Or status of the > filesystem that is not set OK that I cannot figure out. The output of > mmlsfs of a system that mounts compared the the one that doesn't are the > same except for mount point, NSDs and creation time. Is there another > "mm..." command with which to check the status of a filesystem? > > Richard > > > 2015-03-13 14:20 GMT-04:00 Richard Lefebvre < > richard.lefebvre at calculquebec.ca>: > > Hi, >> >> I have a GPFS cluster (v3.5). Last month, one of the filesystem had a >> problem with one of it's NSD. With the help tech support, the problem was >> fix and everything is OK data wise. The thing is when the problem occurred >> the filesystem was offline and once the filesystem was back to being OK, we >> were able to remount it. The problem is that now, every time a node is >> restarted, the filesystem is not mounted automatically as before. The other >> filesystems mount automatically. I did a mmlsfs on the file system and -A >> is at yes. Can someone tell my what I'm missing here. I'm sure it is >> something simple, but I'm drawing a blank right now. >> >> Richard >> >> -- >> Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" >> Richard.Lefebvre at calculquebec.ca -- THGTTG >> Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) >> > > > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.lefebvre at calculquebec.ca Thu Mar 19 22:38:43 2015 From: richard.lefebvre at calculquebec.ca (Richard Lefebvre) Date: Thu, 19 Mar 2015 18:38:43 -0400 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: Message-ID: Hi JF, There are all up and ready. Otherwise I wouldn't be able to mmmount it. Richard 2015-03-19 18:35 GMT-04:00 Jan-Frode Myklebust : > Check "mmlsdisk". Are all disks for the filesystem ok? > > > -jf > On tor. 19. mar. 2015 at 23.33 Richard Lefebvre < > richard.lefebvre at calculquebec.ca> wrote: > >> Thanks for those who replied. One thing I didn't mention correctly is >> that I have several gpfs mount say: >> /gpfs1 >> /gpfs2 >> /gpfs3 >> >> Say it was gpfs2 that had the NSD problem. So now when I reboot a node, >> /gpfs1 and /gpfs3 mount, but /gpfs2 doesn't, I have to manually type >> mmmount gpfs2 on the node to get it mounted after the boot. >> >> So I don't think that it is a problem with mounting within linux boot >> sequence. But a flag than I'm missing in the gpfs setup. Or status of the >> filesystem that is not set OK that I cannot figure out. The output of >> mmlsfs of a system that mounts compared the the one that doesn't are the >> same except for mount point, NSDs and creation time. Is there another >> "mm..." command with which to check the status of a filesystem? >> >> Richard >> >> >> 2015-03-13 14:20 GMT-04:00 Richard Lefebvre < >> richard.lefebvre at calculquebec.ca>: >> >> Hi, >>> >>> I have a GPFS cluster (v3.5). Last month, one of the filesystem had a >>> problem with one of it's NSD. With the help tech support, the problem was >>> fix and everything is OK data wise. The thing is when the problem occurred >>> the filesystem was offline and once the filesystem was back to being OK, we >>> were able to remount it. The problem is that now, every time a node is >>> restarted, the filesystem is not mounted automatically as before. The other >>> filesystems mount automatically. I did a mmlsfs on the file system and -A >>> is at yes. Can someone tell my what I'm missing here. I'm sure it is >>> something simple, but I'm drawing a blank right now. >>> >>> Richard >>> >>> -- >>> Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" >>> Richard.Lefebvre at calculquebec.ca -- THGTTG >>> Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) >>> >> >> >> >> -- >> Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" >> Richard.Lefebvre at calculquebec.ca -- THGTTG >> Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at gpfsug.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) -------------- next part -------------- An HTML attachment was scrubbed... URL: From red_steel at au1.ibm.com Fri Mar 20 01:48:52 2015 From: red_steel at au1.ibm.com (Tony Steel) Date: Fri, 20 Mar 2015 12:48:52 +1100 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: Message-ID: Hi You don't have a file (assuming gpfs2 is the problem) /var/mmfs/etc/ignoreStartupMount.gpfs2 ?? Cheers, - Sense Amid Madness, Wit Amidst Folly - ---------- Antony (Red) Steel, Advanced Technical Support -------------------- Senior IT Specialist Ph: +61 293978414 Mb: +61 409 971 083 Home Page: http://sydgsa.ibm.com/projects/t/team_web/index.html ----------------------------- red_steel at au.ibm.com ----------------------------------- From ckerner at ncsa.uiuc.edu Fri Mar 20 02:56:55 2015 From: ckerner at ncsa.uiuc.edu (Chad Kerner) Date: Thu, 19 Mar 2015 21:56:55 -0500 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: Message-ID: <20150320025655.GA2140@logos.ncsa.illinois.edu> Richard, what does "mmlsfs gpfs2 -A" return? Is it set to automount? --Chad Chad Kerner, Systems Engineer Storage Enabling Technologies National Center for Supercomputing Applications University of Illinois, Urbana-Champaign On Thu, Mar 19, 2015 at 06:38:43PM -0400, Richard Lefebvre wrote: > Hi JF, > > There are all up and ready. Otherwise I wouldn't be able to mmmount it. > > Richard > > > > 2015-03-19 18:35 GMT-04:00 Jan-Frode Myklebust : > > Check "mmlsdisk". Are all disks for the filesystem ok? > > > -jf > On tor. 19. mar. 2015 at 23.33 Richard Lefebvre < > richard.lefebvre at calculquebec.ca> wrote: > > Thanks for those who replied. One thing I didn't mention correctly is > that I have several gpfs mount say: > /gpfs1 > /gpfs2 > /gpfs3 > > Say it was gpfs2 that had the NSD problem. So now when I reboot a node, > /gpfs1 and /gpfs3 mount, but /gpfs2 doesn't, I have to manually type > mmmount gpfs2 on the node to get it mounted after the boot. > > So I don't think that it is a problem with mounting within linux boot > sequence. But a flag than I'm missing in the gpfs setup. Or status of > the filesystem that is not set OK that I cannot figure out. The output > of mmlsfs of a system that mounts compared the the one that doesn't are > the same except for mount point, NSDs and creation time. Is there > another "mm..." command with which to check the status of a filesystem? > > Richard > > > 2015-03-13 14:20 GMT-04:00 Richard Lefebvre < > richard.lefebvre at calculquebec.ca>: > > > Hi, > > I have a GPFS cluster (v3.5). Last month, one of the filesystem had > a problem with one of it's NSD. With the help tech support, the > problem was fix and everything is OK data wise. The thing is when > the problem occurred the filesystem was offline and once the > filesystem was back to being OK, we were able to remount it. The > problem is that now, every time a node is restarted, the filesystem > is not mounted automatically as before. The other filesystems mount > automatically. I did a mmlsfs on the file system and -A is at yes. > Can someone tell my what I'm missing here. I'm sure it is something > simple, but I'm drawing a blank right now. > > Richard > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't > Panic" > Richard.Lefebvre at calculquebec.ca -- > THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada ( > computecanada.ca) > > > > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't > Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From luke.raimbach at oerc.ox.ac.uk Fri Mar 20 08:37:02 2015 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Fri, 20 Mar 2015 08:37:02 +0000 Subject: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans Message-ID: Hi Marc, Thanks for this information. One thing I still can't extract from the manual (though I guess I could just do a small experiment to find out) is whether the number of allocated inodes is automatically extended when running low, up to the maximum for an independent fileset? Cheers, Luke. From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Marc A Kaplan Sent: 17 March 2015 14:23 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans Luke, Thanks for your question. Independent filesets and their inodes are not implemented the way you might be imagining or guessing. Suppose you have two independent filesets "root" and "fset2" in the same GPFS file system. It is true that all the inode records (typically 512 bytes each - see mmcrfs) go into the same special file. BUT if you look at any given metadata allocation block --metadata-block-size (defaults to 256K) you'll only see inodes for either "root" or "fset2" not both in the same block. Moreover we try to "pre"-allocate several blocks of inodes for the same independent fileset contiguously - so that typically GPFS can do one seek and then read several blocks of inodes from the same independent fileset. So there can be performance advantages to using independent filesets and restricting your mmapplypolicy scans to just the fileset you need. To gain maximal advantage, use the following form of the command: mmapplypolicy /gpfs/path-to-the-directory-tree-I-want-to-scan --scope {fileset | inodespace } ... An inodespace is the set of all inodes in an independent fileset. An inodespace may contain several "dependent" filesets. [Marc A Kaplan] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 21994 bytes Desc: image001.gif URL: From makaplan at us.ibm.com Fri Mar 20 12:10:34 2015 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 20 Mar 2015 08:10:34 -0400 Subject: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans In-Reply-To: References: Message-ID: Seems to me that's why this option has two parameters: mmchfileset ... [--inode-limit MaxNumInodes[:NumInodesToPreallocate]] Specifies the new inode limit for the inode space owned by the specified fileset. The FilesetName or JunctionPath must refer to an independent fileset. The NumInodesToPreallocate specifies an optional number of additional inodes to pre-allocate for the inode space. Use the mmchfs command to change inode limits for the root fileset. See also mmchfs! And RTFM! ( ReadTheFantasicManual ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.lefebvre at calculquebec.ca Fri Mar 20 17:22:43 2015 From: richard.lefebvre at calculquebec.ca (Richard Lefebvre) Date: Fri, 20 Mar 2015 13:22:43 -0400 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: <20150320025655.GA2140@logos.ncsa.illinois.edu> References: <20150320025655.GA2140@logos.ncsa.illinois.edu> Message-ID: Hi Tony, Chad, There is nothing in /var/mmfs/etc directory of the nodes. "-A" on the nodes return "yes" Richard 2015-03-19 22:56 GMT-04:00 Chad Kerner : > Richard, what does "mmlsfs gpfs2 -A" return? Is it set to automount? > > --Chad > Chad Kerner, Systems Engineer > Storage Enabling Technologies > National Center for Supercomputing Applications > University of Illinois, Urbana-Champaign > > On Thu, Mar 19, 2015 at 06:38:43PM -0400, Richard Lefebvre wrote: > > Hi JF, > > > > There are all up and ready. Otherwise I wouldn't be able to mmmount it. > > > > Richard > > > > > > > > 2015-03-19 18:35 GMT-04:00 Jan-Frode Myklebust : > > > > Check "mmlsdisk". Are all disks for the filesystem ok? > > > > > > -jf > > On tor. 19. mar. 2015 at 23.33 Richard Lefebvre < > > richard.lefebvre at calculquebec.ca> wrote: > > > > Thanks for those who replied. One thing I didn't mention > correctly is > > that I have several gpfs mount say: > > /gpfs1 > > /gpfs2 > > /gpfs3 > > > > Say it was gpfs2 that had the NSD problem. So now when I reboot > a node, > > /gpfs1 and /gpfs3 mount, but /gpfs2 doesn't, I have to manually > type > > mmmount gpfs2 on the node to get it mounted after the boot. > > > > So I don't think that it is a problem with mounting within linux > boot > > sequence. But a flag than I'm missing in the gpfs setup. Or > status of > > the filesystem that is not set OK that I cannot figure out. The > output > > of mmlsfs of a system that mounts compared the the one that > doesn't are > > the same except for mount point, NSDs and creation time. Is there > > another "mm..." command with which to check the status of a > filesystem? > > > > Richard > > > > > > 2015-03-13 14:20 GMT-04:00 Richard Lefebvre < > > richard.lefebvre at calculquebec.ca>: > > > > > > Hi, > > > > I have a GPFS cluster (v3.5). Last month, one of the > filesystem had > > a problem with one of it's NSD. With the help tech support, > the > > problem was fix and everything is OK data wise. The thing is > when > > the problem occurred the filesystem was offline and once the > > filesystem was back to being OK, we were able to remount it. > The > > problem is that now, every time a node is restarted, the > filesystem > > is not mounted automatically as before. The other > filesystems mount > > automatically. I did a mmlsfs on the file system and -A is > at yes. > > Can someone tell my what I'm missing here. I'm sure it is > something > > simple, but I'm drawing a blank right now. > > > > Richard > > > > -- > > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 > "Don't > > Panic" > > Richard.Lefebvre at calculquebec.ca > -- > > THGTTG > > Calcul Quebec (calculquebec.ca) ------ Calcul Canada ( > > computecanada.ca) > > > > > > > > > > -- > > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 > "Don't > > Panic" > > Richard.Lefebvre at calculquebec.ca > -- THGTTG > > Calcul Quebec (calculquebec.ca) ------ Calcul Canada ( > computecanada.ca) > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at gpfsug.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at gpfsug.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > > > > > > -- > > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" > > Richard.Lefebvre at calculquebec.ca -- THGTTG > > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at gpfsug.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) -------------- next part -------------- An HTML attachment was scrubbed... URL: From YARD at il.ibm.com Fri Mar 20 17:50:33 2015 From: YARD at il.ibm.com (Yaron Daniel) Date: Fri, 20 Mar 2015 19:50:33 +0200 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: <20150320025655.GA2140@logos.ncsa.illinois.edu> Message-ID: Hi Can u send the output of: #mmlsdisk gpfs2 #mmlsnsd |grep gpfs2 #getenforce #iptables -L #cat /etc/fstab Regards Yaron Daniel 94 Em Ha'Moshavot Rd Server, Storage and Data Services - Team Leader Petach Tiqva, 49527 Global Technology Services Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com IBM Israel From: Richard Lefebvre To: gpfsug main discussion list Date: 03/20/2015 07:22 PM Subject: Re: [gpfsug-discuss] GPFS file system won't mount automatically anymore. Sent by: gpfsug-discuss-bounces at gpfsug.org Hi Tony, Chad, There is nothing in /var/mmfs/etc directory of the nodes. "-A" on the nodes return "yes" Richard 2015-03-19 22:56 GMT-04:00 Chad Kerner : Richard, what does "mmlsfs gpfs2 -A" return? Is it set to automount? --Chad Chad Kerner, Systems Engineer Storage Enabling Technologies National Center for Supercomputing Applications University of Illinois, Urbana-Champaign On Thu, Mar 19, 2015 at 06:38:43PM -0400, Richard Lefebvre wrote: > Hi JF, > > There are all up and ready. Otherwise I wouldn't be able to mmmount it. > > Richard > > > > 2015-03-19 18:35 GMT-04:00 Jan-Frode Myklebust : > > Check "mmlsdisk". Are all disks for the filesystem ok? > > > -jf > On tor. 19. mar. 2015 at 23.33 Richard Lefebvre < > richard.lefebvre at calculquebec.ca> wrote: > > Thanks for those who replied. One thing I didn't mention correctly is > that I have several gpfs mount say: > /gpfs1 > /gpfs2 > /gpfs3 > > Say it was gpfs2 that had the NSD problem. So now when I reboot a node, > /gpfs1 and /gpfs3 mount, but /gpfs2 doesn't, I have to manually type > mmmount gpfs2 on the node to get it mounted after the boot. > > So I don't think that it is a problem with mounting within linux boot > sequence. But a flag than I'm missing in the gpfs setup. Or status of > the filesystem that is not set OK that I cannot figure out. The output > of mmlsfs of a system that mounts compared the the one that doesn't are > the same except for mount point, NSDs and creation time. Is there > another "mm..." command with which to check the status of a filesystem? > > Richard > > > 2015-03-13 14:20 GMT-04:00 Richard Lefebvre < > richard.lefebvre at calculquebec.ca>: > > > Hi, > > I have a GPFS cluster (v3.5). Last month, one of the filesystem had > a problem with one of it's NSD. With the help tech support, the > problem was fix and everything is OK data wise. The thing is when > the problem occurred the filesystem was offline and once the > filesystem was back to being OK, we were able to remount it. The > problem is that now, every time a node is restarted, the filesystem > is not mounted automatically as before. The other filesystems mount > automatically. I did a mmlsfs on the file system and -A is at yes. > Can someone tell my what I'm missing here. I'm sure it is something > simple, but I'm drawing a blank right now. > > Richard > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't > Panic" > Richard.Lefebvre at calculquebec.ca -- > THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada ( > computecanada.ca) > > > > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't > Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada ( computecanada.ca) > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1851 bytes Desc: not available URL: From makaplan at us.ibm.com Fri Mar 20 19:24:25 2015 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 20 Mar 2015 15:24:25 -0400 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: <20150320025655.GA2140@logos.ncsa.illinois.edu> Message-ID: Look in /var/adm/ras/mmfs.log.* -- on a node that should be mounting the filesystem -- find a file that covers the time when GPFS is starting up and should be mounting the filesystem. Look for clues and/or error messages. If you are not familiar, you may also want to look at what messages are issued when everything is working just right. -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.lefebvre at calculquebec.ca Fri Mar 20 20:32:46 2015 From: richard.lefebvre at calculquebec.ca (Richard Lefebvre) Date: Fri, 20 Mar 2015 16:32:46 -0400 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: <20150320025655.GA2140@logos.ncsa.illinois.edu> Message-ID: Hi Mark, Thank you for your answer. You are right, I have the following: Mon Mar 16 11:02:24 EDT 2015: mounting /dev/gpfs2 Mon Mar 16 11:02:24.296 2015: Command: mount gpfs2 Mon Mar 16 11:02:24.723 2015: Disk failure. Volume gpfs2. rc = 5. Physical volume gpfs2nsd04. Mon Mar 16 11:02:24.724 2015: Disk failure. Volume gpfs2. rc = 5. Physical volume gpfs2nsd12. Mon Mar 16 11:02:24.723 2015: File System gpfs2 unmounted by the system with return code 5 reason code 0 Mon Mar 16 11:02:24.724 2015: Input/output error Mon Mar 16 11:02:24.725 2015: Failed to open gpfs2. Mon Mar 16 11:02:24.724 2015: Input/output error Mon Mar 16 11:02:24.725 2015: Command: err 666: mount gpfs2 Mon Mar 16 11:02:24.724 2015: Input/output error mount: Stale NFS file handle Mon Mar 16 11:02:24 EDT 2015: finished mounting /dev/gpfs2 But it is not mounted. This is after issuing the "mmmount gpfs2" command Mon Mar 16 11:03:52.010 2015: Command: mount gpfs2 Mon Mar 16 11:03:52.234 2015: VERBS RDMA connecting to 172.25.101.2 (heca2-ib0) on mlx4_0 port 1 id 4 Mon Mar 16 11:03:52.235 2015: VERBS RDMA connected to 172.25.101.2 (heca2-ib0) on mlx4_0 port 1 sl 0 Mon Mar 16 11:03:52.381 2015: Command: err 0: mount gpfs2 Now, my next step it to figure out how the clear the failure since it is not failed anymore. The mmlsdisk reports all the nsd are up and ready. One command I'm also looking for is a command to list which nsd server is used for a particular nsd. I have a redundant config for which 2 servers can access the same nsd. But I think that for some of them, the default server is not used and the load of the servers is not distributed as wanted. Richard 2015-03-20 15:24 GMT-04:00 Marc A Kaplan : > Look in /var/adm/ras/mmfs.log.* -- on a node that should be mounting the > filesystem -- find a file that covers the time when GPFS is starting up > and should be mounting the filesystem. > > Look for clues and/or error messages. If you are not familiar, you may > also want to look at what messages are issued when everything is working > just right. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) -------------- next part -------------- An HTML attachment was scrubbed... URL: From Dan.Foster at bristol.ac.uk Mon Mar 23 08:43:33 2015 From: Dan.Foster at bristol.ac.uk (Dan Foster) Date: Mon, 23 Mar 2015 08:43:33 +0000 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: <20150320025655.GA2140@logos.ncsa.illinois.edu> Message-ID: Hi Richard, On 20 March 2015 at 20:32, Richard Lefebvre wrote: > One command I'm also looking for is a command to list which nsd server is > used for a particular nsd. I have a redundant config for which 2 servers can > access the same nsd. But I think that for some of them, the default server > is not used and the load of the servers is not distributed as wanted. Does the '-M' option to mmlsdisk provide the information you're looking for? i.e. from the man page: """ Displays whether I/O requests to the disk are satisfied on the local node, or using an NSD server. If the I/O is done using an NSD server, shows the NSD server name and the underlying disk name on that server node. """ It should display something similar to the following: Disk name IO performed on node Device Availability ------------ ----------------------- ----------------- ------------ nsd1 localhost /dev/disk1 up nsd2 node01.example.com /dev/disk2 up nsd3 localhost /dev/disk3 up HTH, Dan. > > 2015-03-20 15:24 GMT-04:00 Marc A Kaplan : >> >> Look in /var/adm/ras/mmfs.log.* -- on a node that should be mounting the >> filesystem -- find a file that covers the time when GPFS is starting up >> and should be mounting the filesystem. >> >> Look for clues and/or error messages. If you are not familiar, you may >> also want to look at what messages are issued when everything is working >> just right. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at gpfsug.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Dan Foster | Senior Storage Systems Administrator | IT Services e: dan.foster at bristol.ac.uk | t: 0117 3941170 [x41170] m: Advanced Computing Research Centre, University of Bristol, 8-10 Berkeley Square, Bristol BS8 1HH From PEREZJOS at uk.ibm.com Mon Mar 23 14:13:08 2015 From: PEREZJOS at uk.ibm.com (Jose Perez) Date: Mon, 23 Mar 2015 14:13:08 +0000 Subject: [gpfsug-discuss] Jose Perez is OoO. Message-ID: I will be out of the office starting 23/03/2015 and will not return until 24/03/2015. I will respond to your message when I return. For urgent issues please contact Ross Keeping, Thanks. Thanks, From chair at gpfsug.org Mon Mar 23 14:34:20 2015 From: chair at gpfsug.org (Jez Tucker (Chair)) Date: Mon, 23 Mar 2015 14:34:20 +0000 Subject: [gpfsug-discuss] Jose Perez is OoO. In-Reply-To: References: Message-ID: <5510246C.6090009@gpfsug.org> Apols for that one which slipped through, we'll add 'OoO' to the list blocker. Jez On 23/03/15 14:13, Jose Perez wrote: > I will be out of the office starting 23/03/2015 and will not return until > 24/03/2015. > > I will respond to your message when I return. For urgent issues please > contact Ross Keeping, Thanks. > Thanks, > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From makaplan at us.ibm.com Mon Mar 23 14:33:29 2015 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 23 Mar 2015 10:33:29 -0400 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: <20150320025655.GA2140@logos.ncsa.illinois.edu> Message-ID: mmlsnsd has several options... RTFM... but this one will show you which nodes can directly access and/or serve each NSD: mmlsnsd -M -v -M Maps the NSD names to its disk device name on all nodes. This is a slow operation and its usage is suggested for problem determination only. -v Specifies that the output should contain error information, where available. -------------- next part -------------- An HTML attachment was scrubbed... URL: From secretary at gpfsug.org Wed Mar 25 14:42:52 2015 From: secretary at gpfsug.org (Secretary GPFS UG) Date: Wed, 25 Mar 2015 14:42:52 +0000 Subject: [gpfsug-discuss] 2nd 'Meet the Devs' Event 28/4/15 Message-ID: <6715b1493f52d222ab24b49195ab8fc2@webmail.gpfsug.org> Dear members, Our second Meet the GPFS Developers 'coffee shop' meet is going to be taking place on Tuesday 28th April. The day will give you the opportunity to meet the developers and get hands on beta exploration and test of unreleased GPFS functions. Date: 28th April 2015 Time: 10:30 - 15:30 Location: IBM, 3rd Floor, Maybrook House, 40 Blackfriars Street, Manchester M3 2EG Agenda: 10:30 Introduction discussion 11:30 Beta session 1 12:30 Break for pizza (social chats with wider GPFS dev team in Manchester) 13:30 Beta session 2 14:45 Group discussion and feedback 15:30 End Lunch and refreshments will be provided. To attend, please register with Claire Robson, GPFS UG Secretary secretary at gpfsug.org. 10 places are available. Full directions and instructions will be sent to confirmed attendees. We look forward to seeing you in April. Thanks, Claire -- GPFS User Group Secretary From ross.keeping at uk.ibm.com Thu Mar 26 14:59:31 2015 From: ross.keeping at uk.ibm.com (Ross Keeping3) Date: Thu, 26 Mar 2015 14:59:31 +0000 Subject: [gpfsug-discuss] Potential talks for GPFS UG May 20th Message-ID: Hi There are a couple of talks we could include in the one day GPFS UG on May 20th. We'd appreciate some feedback on whether there is any interest or objection to including these. These are: 1) Tivoli Storage Productivity Center Technical Demo: A tool for managing heterogeneous storage infrastructures and specifically GPFS. Can read more here http://www-03.ibm.com/software/products/en/tivostorprodcent 2) GPFS for life sciences: A member of NERSC would like to talk about their efforts to improve GPFS file systems for a life sciences workload - specifically genomic sequencing and analysis that ensues. Details about life science workflow and requirements will be provided so the listener may understand the aspects of GPFS that are important. Cheers, Ross Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From stijn.deweirdt at ugent.be Thu Mar 26 15:07:29 2015 From: stijn.deweirdt at ugent.be (Stijn De Weirdt) Date: Thu, 26 Mar 2015 16:07:29 +0100 Subject: [gpfsug-discuss] Potential talks for GPFS UG May 20th In-Reply-To: References: Message-ID: <551420B1.3080002@ugent.be> no objections for any of them, but i'd prefer talk 2 (if a choice has to be made) stijn On 03/26/2015 03:59 PM, Ross Keeping3 wrote: > Hi > > There are a couple of talks we could include in the one day GPFS UG on May > 20th. > > We'd appreciate some feedback on whether there is any interest or > objection to including these. > > These are: > 1) Tivoli Storage Productivity Center Technical Demo: A tool for managing > heterogeneous storage infrastructures and specifically GPFS. Can read more > here http://www-03.ibm.com/software/products/en/tivostorprodcent > > 2) GPFS for life sciences: A member of NERSC would like to talk about > their efforts to improve GPFS file systems for a life sciences workload - > specifically genomic sequencing and analysis that ensues. Details about > life science workflow and requirements will be provided so the listener > may understand the aspects of GPFS that are important. > > Cheers, > Ross > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From robert at strubi.ox.ac.uk Thu Mar 26 15:13:44 2015 From: robert at strubi.ox.ac.uk (Robert Esnouf) Date: Thu, 26 Mar 2015 15:13:44 +0000 (GMT) Subject: [gpfsug-discuss] Potential talks for GPFS UG May 20th In-Reply-To: References: Message-ID: <201503261513.062218@mail.strubi.ox.ac.uk> Yes, unsurprisingly I'd be interested in the second talk! Thanks, Robert -- Dr. Robert Esnouf, University Research Lecturer, Head of Research Computing Core, NDM Research Computing Strategy Officer Room 10/028, Wellcome Trust Centre for Human Genetics, Old Road Campus, Roosevelt Drive, Oxford OX3 7BN, UK Email: robert at strubi.ox.ac.uk / robert at well.ox.ac.uk Tel: (+44) - 1865 - 287783 ---- Original message ---- >Date: Thu, 26 Mar 2015 14:59:31 +0000 >From: gpfsug-discuss-bounces at gpfsug.org (on behalf of Ross Keeping3 ) >Subject: [gpfsug-discuss] Potential talks for GPFS UG May 20th >To: gpfsug-discuss at gpfsug.org > > Hi > > There are a couple of talks we could include in the > one day GPFS UG on May 20th. > > We'd appreciate some feedback on whether there is > any interest or objection to including these. > > These are: > 1) Tivoli Storage Productivity Center Technical > Demo: A tool for managing heterogeneous storage > infrastructures and specifically GPFS. Can read more > here > http://www-03.ibm.com/software/products/en/tivostorprodcent > > 2) GPFS for life sciences: A member of NERSC would > like to talk about their efforts to improve GPFS > file systems for a life sciences workload - > specifically genomic sequencing and analysis that > ensues. Details about life science workflow and > requirements will be provided so the listener may > understand the aspects of GPFS that are important. > > Cheers, > Ross > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England > and Wales with number 741598. > Registered office: PO Box 41, North Harbour, > Portsmouth, Hampshire PO6 3AU >________________ >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at gpfsug.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Mark.Roberts at awe.co.uk Thu Mar 26 15:24:17 2015 From: Mark.Roberts at awe.co.uk (Mark.Roberts at awe.co.uk) Date: Thu, 26 Mar 2015 15:24:17 +0000 Subject: [gpfsug-discuss] EXTERNAL: Potential talks for GPFS UG May 20th In-Reply-To: References: Message-ID: <201503261524.t2QFOQrO020615@msw1.awe.co.uk> As well as supporting the efforts of our friends at NERSC I think should be a good interesting talk. Mark Roberts ________________________________ From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Ross Keeping3 Sent: 26 March 2015 15:00 To: gpfsug-discuss at gpfsug.org Subject: EXTERNAL: [gpfsug-discuss] Potential talks for GPFS UG May 20th Hi There are a couple of talks we could include in the one day GPFS UG on May 20th. We'd appreciate some feedback on whether there is any interest or objection to including these. These are: 1) Tivoli Storage Productivity Center Technical Demo: A tool for managing heterogeneous storage infrastructures and specifically GPFS. Can read more here http://www-03.ibm.com/software/products/en/tivostorprodcent 2) GPFS for life sciences: A member of NERSC would like to talk about their efforts to improve GPFS file systems for a life sciences workload - specifically genomic sequencing and analysis that ensues. Details about life science workflow and requirements will be provided so the listener may understand the aspects of GPFS that are important. Cheers, Ross Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU ___________________________________________________ ____________________________ The information in this email and in any attachment(s) is commercial in confidence. If you are not the named addressee(s) or if you receive this email in error then any distribution, copying or use of this communication or the information in it is strictly prohibited. Please notify us immediately by email at admin.internet(at)awe.co.uk, and then delete this message from your computer. While attachments are virus checked, AWE plc does not accept any liability in respect of any virus which is not detected. AWE Plc Registered in England and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR -------------- next part -------------- An HTML attachment was scrubbed... URL: From Sandra.McLaughlin at astrazeneca.com Thu Mar 26 15:56:16 2015 From: Sandra.McLaughlin at astrazeneca.com (McLaughlin, Sandra M) Date: Thu, 26 Mar 2015 15:56:16 +0000 Subject: [gpfsug-discuss] EXTERNAL: Potential talks for GPFS UG May 20th In-Reply-To: <201503261524.t2QFOQrO020615@msw1.awe.co.uk> References: <201503261524.t2QFOQrO020615@msw1.awe.co.uk> Message-ID: Life Sciences talk would be spot on for me too, please. Sandra McLaughlin From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Mark.Roberts at awe.co.uk Sent: 26 March 2015 15:24 To: gpfsug-discuss at gpfsug.org Subject: Re: [gpfsug-discuss] EXTERNAL: Potential talks for GPFS UG May 20th As well as supporting the efforts of our friends at NERSC I think should be a good interesting talk. Mark Roberts ________________________________ From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Ross Keeping3 Sent: 26 March 2015 15:00 To: gpfsug-discuss at gpfsug.org Subject: EXTERNAL: [gpfsug-discuss] Potential talks for GPFS UG May 20th Hi There are a couple of talks we could include in the one day GPFS UG on May 20th. We'd appreciate some feedback on whether there is any interest or objection to including these. These are: 1) Tivoli Storage Productivity Center Technical Demo: A tool for managing heterogeneous storage infrastructures and specifically GPFS. Can read more here http://www-03.ibm.com/software/products/en/tivostorprodcent 2) GPFS for life sciences: A member of NERSC would like to talk about their efforts to improve GPFS file systems for a life sciences workload - specifically genomic sequencing and analysis that ensues. Details about life science workflow and requirements will be provided so the listener may understand the aspects of GPFS that are important. Cheers, Ross Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU ___________________________________________________ ____________________________ The information in this email and in any attachment(s) is commercial in confidence. If you are not the named addressee(s) or if you receive this email in error then any distribution, copying or use of this communication or the information in it is strictly prohibited. Please notify us immediately by email at admin.internet(at)awe.co.uk, and then delete this message from your computer. While attachments are virus checked, AWE plc does not accept any liability in respect of any virus which is not detected. AWE Plc Registered in England and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR ________________________________ AstraZeneca UK Limited is a company incorporated in England and Wales with registered number: 03674842 and a registered office at 2 Kingdom Street, London, W2 6BD. Confidentiality Notice: This message is private and may contain confidential, proprietary and legally privileged information. If you have received this message in error, please notify us and remove it from your system and note that you must not copy, distribute or take any action in reliance on it. Any unauthorised use or disclosure of the contents of this message is not permitted and may be unlawful. Disclaimer: Email messages may be subject to delays, interception, non-delivery and unauthorised alterations. Therefore, information expressed in this message is not given or endorsed by AstraZeneca UK Limited unless otherwise notified by an authorised representative independent of this message. No contractual relationship is created by this message by any person unless specifically indicated by agreement in writing other than email. Monitoring: AstraZeneca UK Limited may monitor email traffic data and content for the purposes of the prevention and detection of crime, ensuring the security of our computer systems and checking compliance with our Code of Conduct and policies. -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.kidger at uk.ibm.com Thu Mar 26 16:06:24 2015 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Thu, 26 Mar 2015 16:06:24 +0000 Subject: [gpfsug-discuss] Potential talks for GPFS UG May 20th In-Reply-To: <201503261513.062218@mail.strubi.ox.ac.uk> References: <201503261513.062218@mail.strubi.ox.ac.uk> Message-ID: I too would want to upvote the second. I am seeing lots of interest in the Life Sciences particularly for things like large Genomic datasets and NGS. Daniel Dr.Daniel Kidger No. 1 The Square, Technical Specialist SDI (formerly Platform Computing) Temple Quay, Bristol BS1 6DG Mobile: +44-07818 522 266 United Kingdom Landline: +44-02392 564 121 (Internal ITN 3726 9250) e-mail: daniel.kidger at uk.ibm.com From: Robert Esnouf To: "gpfsug main discussion list" Date: 26/03/2015 15:13 Subject: Re: [gpfsug-discuss] Potential talks for GPFS UG May 20th Sent by: gpfsug-discuss-bounces at gpfsug.org Yes, unsurprisingly I'd be interested in the second talk! Thanks, Robert -- Dr. Robert Esnouf, University Research Lecturer, Head of Research Computing Core, NDM Research Computing Strategy Officer Room 10/028, Wellcome Trust Centre for Human Genetics, Old Road Campus, Roosevelt Drive, Oxford OX3 7BN, UK Email: robert at strubi.ox.ac.uk / robert at well.ox.ac.uk Tel: (+44) - 1865 - 287783 ---- Original message ---- >Date: Thu, 26 Mar 2015 14:59:31 +0000 >From: gpfsug-discuss-bounces at gpfsug.org (on behalf of Ross Keeping3 ) >Subject: [gpfsug-discuss] Potential talks for GPFS UG May 20th >To: gpfsug-discuss at gpfsug.org > > Hi > > There are a couple of talks we could include in the > one day GPFS UG on May 20th. > > We'd appreciate some feedback on whether there is > any interest or objection to including these. > > These are: > 1) Tivoli Storage Productivity Center Technical > Demo: A tool for managing heterogeneous storage > infrastructures and specifically GPFS. Can read more > here > http://www-03.ibm.com/software/products/en/tivostorprodcent > > 2) GPFS for life sciences: A member of NERSC would > like to talk about their efforts to improve GPFS > file systems for a life sciences workload - > specifically genomic sequencing and analysis that > ensues. Details about life science workflow and > requirements will be provided so the listener may > understand the aspects of GPFS that are important. > > Cheers, > Ross > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England > and Wales with number 741598. > Registered office: PO Box 41, North Harbour, > Portsmouth, Hampshire PO6 3AU >________________ >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at gpfsug.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 360 bytes Desc: not available URL: From zgiles at gmail.com Thu Mar 26 16:14:36 2015 From: zgiles at gmail.com (Zachary Giles) Date: Thu, 26 Mar 2015 12:14:36 -0400 Subject: [gpfsug-discuss] EXTERNAL: Potential talks for GPFS UG May 20th In-Reply-To: References: <201503261524.t2QFOQrO020615@msw1.awe.co.uk> Message-ID: It's at the GPFSUG in the UK. Looks like the UK GPFSUG is much more active than the US one's. They've had a few meetups actually. I'd fly out there and attend them, but I figured probably it wouldn't count since it's not a real conf. This on is in May 20th. On Thu, Mar 26, 2015 at 11:56 AM, McLaughlin, Sandra M < Sandra.McLaughlin at astrazeneca.com> wrote: > Life Sciences talk would be spot on for me too, please. > > > > Sandra McLaughlin > > > > *From:* gpfsug-discuss-bounces at gpfsug.org [mailto: > gpfsug-discuss-bounces at gpfsug.org] *On Behalf Of *Mark.Roberts at awe.co.uk > *Sent:* 26 March 2015 15:24 > *To:* gpfsug-discuss at gpfsug.org > *Subject:* Re: [gpfsug-discuss] EXTERNAL: Potential talks for GPFS UG May > 20th > > > > As well as supporting the efforts of our friends at NERSC I think should > be a good interesting talk. > > > > Mark Roberts > > > ------------------------------ > > *From:* gpfsug-discuss-bounces at gpfsug.org [ > mailto:gpfsug-discuss-bounces at gpfsug.org > ] *On Behalf Of *Ross Keeping3 > *Sent:* 26 March 2015 15:00 > *To:* gpfsug-discuss at gpfsug.org > *Subject:* EXTERNAL: [gpfsug-discuss] Potential talks for GPFS UG May 20th > > > > Hi > > There are a couple of talks we could include in the one day GPFS UG on May > 20th. > > We'd appreciate some feedback on whether there is any interest or > objection to including these. > > These are: > 1) Tivoli Storage Productivity Center Technical Demo: A tool for managing > heterogeneous storage infrastructures and specifically GPFS. Can read more > here http://www-03.ibm.com/software/products/en/tivostorprodcent > > 2) GPFS for life sciences: A member of NERSC would like to talk about > their efforts to improve GPFS file systems for a life sciences workload - > specifically genomic sequencing and analysis that ensues. Details about > life science workflow and requirements will be provided so the listener may > understand the aspects of GPFS that are important. > > Cheers, > Ross > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > ___________________________________________________ > ____________________________ The information in this email and in any > attachment(s) is commercial in confidence. If you are not the named > addressee(s) or if you receive this email in error then any distribution, > copying or use of this communication or the information in it is strictly > prohibited. Please notify us immediately by email at admin.internet(at) > awe.co.uk, and then delete this message from your computer. While > attachments are virus checked, AWE plc does not accept any liability in > respect of any virus which is not detected. AWE Plc Registered in England > and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR > ------------------------------ > > AstraZeneca UK Limited is a company incorporated in England and Wales with > registered number: 03674842 and a registered office at 2 Kingdom Street, > London, W2 6BD. > > *Confidentiality Notice: *This message is private and may contain > confidential, proprietary and legally privileged information. If you have > received this message in error, please notify us and remove it from your > system and note that you must not copy, distribute or take any action in > reliance on it. Any unauthorised use or disclosure of the contents of this > message is not permitted and may be unlawful. > > *Disclaimer:* Email messages may be subject to delays, interception, > non-delivery and unauthorised alterations. Therefore, information expressed > in this message is not given or endorsed by AstraZeneca UK Limited unless > otherwise notified by an authorised representative independent of this > message. No contractual relationship is created by this message by any > person unless specifically indicated by agreement in writing other than > email. > > *Monitoring: *AstraZeneca UK Limited may monitor email traffic data and > content for the purposes of the prevention and detection of crime, ensuring > the security of our computer systems and checking compliance with our Code > of Conduct and policies. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- Zach Giles zgiles at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From luke.raimbach at oerc.ox.ac.uk Fri Mar 27 12:16:16 2015 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Fri, 27 Mar 2015 12:16:16 +0000 Subject: [gpfsug-discuss] mmunlinkfileset problems Message-ID: Hi All, I'm having a problem where unlinking a fileset is taking 7 or 8 minutes to complete. We are running 3.5.0.22 and the cluster is small (9 nodes: 8 Linux plus 1 Windows node), one file system remote mounted by a 3-node Linux cluster, and generally not very busy. I wouldn't expect mmunlinkfileset to require the file system to quiesce (and even if it did 8 minutes would seem like an awfully long time). Previous unlink commands (on version 3.5.0.19) have returned within milliseconds. This very long running command appeared to pause IO for a sufficiently long time that cNFS clients timed out too and various services elsewhere crashed. Is there any reason why unlinking a fileset would take this long? Also, I can't find anywhere a list of commands which might require the FS to quiesce - is this information available somewhere? Cheers, Luke. -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 From jonathan at buzzard.me.uk Fri Mar 27 12:49:38 2015 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Fri, 27 Mar 2015 12:49:38 +0000 Subject: [gpfsug-discuss] mmunlinkfileset problems In-Reply-To: References: Message-ID: <1427460578.3553.153.camel@buzzard.phy.strath.ac.uk> On Fri, 2015-03-27 at 12:16 +0000, Luke Raimbach wrote: > Hi All, > > I'm having a problem where unlinking a fileset is taking 7 or 8 minutes > to complete. We are running 3.5.0.22 and the cluster is small (9 nodes: > 8 Linux plus 1 Windows node), one file system remote mounted by a > 3-node Linux cluster, and generally not very busy. > > I wouldn't expect mmunlinkfileset to require the file system to quiesce > (and even if it did 8 minutes would seem like an awfully long time). > Previous unlink commands (on version 3.5.0.19) have returned within > milliseconds. If there is a waiter around mmunlinkfileset can block even with an empty fileset and turn your file system unresponsive. > > This very long running command appeared to pause IO for a sufficiently > long time that cNFS clients timed out too and various services > elsewhere crashed. > Yep, lesson for the day only unlink filesets when the file system is quiet, aka a maintenance window. > Is there any reason why unlinking a fileset would take this long? Also, > I can't find anywhere a list of commands which might require the FS to > quiesce - is this information available somewhere? > Deleting snapshots, at least in the past. It might be that guidance from IBM has changed but once you have been stung by this you kind of thing you don't like to try it again any time soon. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From luke.raimbach at oerc.ox.ac.uk Fri Mar 27 13:27:20 2015 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Fri, 27 Mar 2015 13:27:20 +0000 Subject: [gpfsug-discuss] mmunlinkfileset problems In-Reply-To: <1427460578.3553.153.camel@buzzard.phy.strath.ac.uk> References: <1427460578.3553.153.camel@buzzard.phy.strath.ac.uk> Message-ID: > > Is there any reason why unlinking a fileset would take this long? > > Also, I can't find anywhere a list of commands which might require the > > FS to quiesce - is this information available somewhere? > > > > Deleting snapshots, at least in the past. It might be that guidance from IBM > has changed but once you have been stung by this you kind of thing you > don't like to try it again any time soon. Precisely why I'm after a list of commands that might require quiescence! I know that snapshot operations (generally) and deleting filesets require a consistent view so will pause IO, but unlinking (in my mind) wouldn't need this... I'd probably want to generate lists of 'cleanup' commands that might interrupt IO and run these in maintenance windows as you say. Gather up all the filesets that want deleting and do them once a week / whenever. Cheers, Luke. From jonathan at buzzard.me.uk Fri Mar 27 13:39:19 2015 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Fri, 27 Mar 2015 13:39:19 +0000 Subject: [gpfsug-discuss] mmunlinkfileset problems In-Reply-To: References: <1427460578.3553.153.camel@buzzard.phy.strath.ac.uk> Message-ID: <1427463559.3553.156.camel@buzzard.phy.strath.ac.uk> On Fri, 2015-03-27 at 13:27 +0000, Luke Raimbach wrote: > > > Is there any reason why unlinking a fileset would take this long? > > > Also, I can't find anywhere a list of commands which might require the > > > FS to quiesce - is this information available somewhere? > > > > > > > Deleting snapshots, at least in the past. It might be that guidance from IBM > > has changed but once you have been stung by this you kind of thing you > > don't like to try it again any time soon. > > Precisely why I'm after a list of commands that might require > quiescence! I know that snapshot operations (generally) and deleting > filesets require a consistent view so will pause IO, but unlinking (in > my mind) wouldn't need this... > mmdeldisk is another. From what I can make out anything that takes away from the file system is a potential problem. Anything that adds to the file system is just peachy fine to be run at any time. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From adam.huffman at crick.ac.uk Mon Mar 2 09:24:59 2015 From: adam.huffman at crick.ac.uk (Adam Huffman) Date: Mon, 2 Mar 2015 09:24:59 +0000 Subject: [gpfsug-discuss] Introduction Message-ID: Hello A couple of weeks ago I joined Bruno Silva?s HPC team at the Francis Crick Institute, with special responsibility for HPC, OpenStack and virtualization. I?m very much a GPFS novice so I?m hoping to be able to draw on the knowledge in this group, while hopefully being able to help others with OpenStack. As Bruno stated in his message, we?re particularly interested in how to present GPFS to instances securely. I?ve read the discussion from November on this list, which didn?t seem to come to any firm conclusions. Has anyone involved then made substantial progress since? Cheers, Adam ? Adam Huffman Senior HPC & Virtualization Systems Engineer The Francis Crick Institute Gibbs Building 215 Euston Road London NW1 2BE T: E: adam.huffman at crick.ac.uk W: www.crick.ac.uk [Description: Description: ZA102637861][Description: Description: Twitter] The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no.06885462, with its registered office at 215 Euston Road, London NW1 2BE -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 158254D2-CDA8-43E5-96C1-A0AAEB10314B.png Type: image/png Size: 1151 bytes Desc: 158254D2-CDA8-43E5-96C1-A0AAEB10314B.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: B09B8AB7-5CA2-4DE7-BFE7-9D8F1B2A5166.png Type: image/png Size: 1211 bytes Desc: B09B8AB7-5CA2-4DE7-BFE7-9D8F1B2A5166.png URL: From viccornell at gmail.com Mon Mar 2 09:32:57 2015 From: viccornell at gmail.com (Vic Cornell) Date: Mon, 2 Mar 2015 09:32:57 +0000 Subject: [gpfsug-discuss] Introduction In-Reply-To: References: Message-ID: Hi Adam, I guess that one of the things that would help push it forward is a definition of what "secure" means to you. Regards, Vic On 2 Mar 2015, at 09:24, Adam Huffman wrote: > Hello > > A couple of weeks ago I joined Bruno Silva?s HPC team at the Francis Crick Institute, with special responsibility for HPC, OpenStack and virtualization. I?m very much a GPFS novice so I?m hoping to be able to draw on the knowledge in this group, while hopefully being able to help others with OpenStack. > > As Bruno stated in his message, we?re particularly interested in how to present GPFS to instances securely. I?ve read the discussion from November on this list, which didn?t seem to come to any firm conclusions. Has anyone involved then made substantial progress since? > > Cheers, > Adam > > ? > > Adam Huffman > Senior HPC & Virtualization Systems Engineer > The Francis Crick Institute > Gibbs Building > 215 Euston Road > London NW1 2BE > > T: > E: adam.huffman at crick.ac.uk > W: www.crick.ac.uk > <158254D2-CDA8-43E5-96C1-A0AAEB10314B.png> > > The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no.06885462, with its registered office at 215 Euston Road, London NW1 2BE > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam.huffman at crick.ac.uk Mon Mar 2 09:39:36 2015 From: adam.huffman at crick.ac.uk (Adam Huffman) Date: Mon, 2 Mar 2015 09:39:36 +0000 Subject: [gpfsug-discuss] Introduction In-Reply-To: References: Message-ID: Hi Vic, Re-emphasising that I?m still very much learning about GPFS, one of the approaches being discussed is running the GPFS client inside the instances. The concern here is over the case where users have root privileges inside their instances (a pretty common assumption for those used to AWS, for example) and the implications this may have for GPFS. Does it mean there would be a risk of spoofing? Cheers, Adam From: Vic Cornell Reply-To: gpfsug main discussion list Date: Monday, 2 March 2015 09:32 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Introduction Hi Adam, I guess that one of the things that would help push it forward is a definition of what "secure" means to you. Regards, Vic On 2 Mar 2015, at 09:24, Adam Huffman > wrote: Hello A couple of weeks ago I joined Bruno Silva?s HPC team at the Francis Crick Institute, with special responsibility for HPC, OpenStack and virtualization. I?m very much a GPFS novice so I?m hoping to be able to draw on the knowledge in this group, while hopefully being able to help others with OpenStack. As Bruno stated in his message, we?re particularly interested in how to present GPFS to instances securely. I?ve read the discussion from November on this list, which didn?t seem to come to any firm conclusions. Has anyone involved then made substantial progress since? Cheers, Adam ? Adam Huffman Senior HPC & Virtualization Systems Engineer The Francis Crick Institute Gibbs Building 215 Euston Road London NW1 2BE T: E: adam.huffman at crick.ac.uk W: www.crick.ac.uk <158254D2-CDA8-43E5-96C1-A0AAEB10314B.png> The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no.06885462, with its registered office at 215 Euston Road, London NW1 2BE _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From luke.raimbach at oerc.ox.ac.uk Mon Mar 2 19:21:25 2015 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Mon, 2 Mar 2015 19:21:25 +0000 Subject: [gpfsug-discuss] Introduction In-Reply-To: References: Message-ID: HI Adam, We run virtualised GPFS client nodes in a VMware cluster here at Oxford e-Research Centre. We had a requirement where one research group wanted root access to their VMs but also wanted fast direct access to their data on our GPFS cluster. The technical setup was (relatively) simple. We span up a small three-node virtual GPFS cluster (with no file system of its own). We then used the multi-cluster feature of GPFS to allow this small virtual cluster to join our main file system cluster which now gives that group very good IO performance. However, the problem of spoofing you mention is relevant ? we installed the virtual cluster nodes for the research group and put our Active Directory client on them. We also used the root-squash configuration option of the multi-cluster setup to prevent remote-cluster root access in to our file system. We also agreed with the research group that they would nominate one Administrator to have root access in their cluster and that they would maintain the AAA framework we put in place. We have to trust the group?s Administrator not to fiddle his UID or to let others escalate their privileges. If you were letting untrusted root users spin up Stacks, then you could still run GPFS clients in the OpenStack instance nodes to give them fast access to their data. Here are some musings on a recipe (others please feel free to pull these ideas to pieces): 1. Start with Cluster A ? your main production GPFS file system. It has GPFS device name /gpfs. 2. Pretend you have lots of money for extra disk to go under your OpenStack cluster (say you buy something like a DDN SFA7700 with a couple of expansion shelves and fill it up with 4TB drives ? 180 drives). 3. Drive this disk array with two, preferably four (or however many you want, really) decent NSD servers. Configure quorum nodes, etc. appropriately. Call this Cluster B. 4. Carve up the disk array in to something like 30 x RAID6 (4 + 2) LUNs and configure them as GPFS NSDs; but don?t create a file system (line up the stripe sizes etc. and choose a nice block size, etc. etc.)? 5. Put the GPFS metadata on some SSD NSDs somewhere. I like putting it on SSDs in the NSD server nodes and replicating it. Other people like putting it in their disk arrays. 6. As part of someone spinning up a Stack, get some scripts to do the following ?magic?: a. Connect to Cluster A and find out how big their target data-set is. b. Connect to Cluster B and create a new GPFS file system with a reasonable (dependent on the above result) number of NSD disks. Call this new GPFS device something unique other that /gpfs e.g. /gpfs0001. You could slice bits off your SSDs for the metadata NSDs in each file system you create in this manner (if you haven?t got many SSDs). c. As part of a new Stack, provide a few (three, say) GPFS quorum nodes that you?ve configured. Call this Cluster C. Add the rest of the stack instances to Cluster C. No File System. d. Pop back over to Cluster A. Export their target data-set from Cluster A using AFM (over GPFS or NFS ? pick your favourite: GPFS is probably better performant but means you need Cluster A to stay online). e. Now return to Cluster B. Import the target data to a local AFM cache on Cluster B?s new file system. Name the AFM file-set whatever you like, but link it in to the Cluster B /gpfs0001 namespace at the same level as it is in Cluster A. For example Cluster A: /gpfs/projects/dataset01 imports to an AFM fileset in Cluster B named userdataset01. Link this under /gpfs0001/projects/dataset01. f. Configure multi-cluster support on Cluster B to export GPFS device /gpfs0001 to Cluster C. Encrypt traffic if you want a headache. g. Configure multi-cluster support on Cluster C to remote mount Cluster B:/gpfs0001 as local device /gpfs. 7. You now have fast GPFS access to this user dataset *only* using GPFS clients inside the OpenStack instance nodes. You have also preserved the file system namespace in Cluster C?s instance nodes. If you only want to run over the data in the stack instances, you could pre-fetch the entire data-set using AFM Control from Cluster A in to the Cluster B file-set (if it?s big enough). 8. Now your users are finished and want to destroy the stack ? you need some more script ?magic?: a. Dismount the file system /gpfs in Cluster C. b. Connect to Cluster B and use AFM control to flush all the data back home to Cluster A. c. Unlink the file-set in Cluster B and force delete it; then delete the file system to free the NSDs back to the pool available to Cluster B. d. Connect back to Cluster A and unexport the original data-set directory structure. e. Throw away the VMs in the stack Things to worry about: ? Inode space will be different if users happen to be working on the data in Cluster A and Cluster C and want to know about inodes. GPFS XATTRS are preserved. ? If you use AFM over NFS because Cluster A and B are far away from each other and laggy, then there?s no locking with your AFM cache running as an Independent Writer. Writes at home (Cluster A) and in cache (Cluster B from Cluster C) will be nondeterministic. Your users will need to know this to avoid disappointment. ? If you use AFM over GPFS because Cluster A and B are near to each other and have a fast network, then there might still not be locking, but if Cluster A goes offline, it will put the AFM cache in to a ?dismounted? state. ? If your users want access to other parts of the Cluster A /gpfs namespace within their stack instances (because you have tools they use or they want to see other stuff), you can export them as read-only to a read-only AFM cache in Cluster B and they will be able to see things in Cluster C provided you link the AFM caches in the right places. Remember they have root access here. ? AFM updates are sent from cache to home as root so users can potentially overflow their quota at the Cluster A home site (root doesn?t care about quotas at home). ? Other frightening things might happen that I?ve not thought about. Hope this helps! Luke -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Adam Huffman Sent: 02 March 2015 09:40 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Introduction Hi Vic, Re-emphasising that I?m still very much learning about GPFS, one of the approaches being discussed is running the GPFS client inside the instances. The concern here is over the case where users have root privileges inside their instances (a pretty common assumption for those used to AWS, for example) and the implications this may have for GPFS. Does it mean there would be a risk of spoofing? Cheers, Adam From: Vic Cornell Reply-To: gpfsug main discussion list Date: Monday, 2 March 2015 09:32 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Introduction Hi Adam, I guess that one of the things that would help push it forward is a definition of what "secure" means to you. Regards, Vic On 2 Mar 2015, at 09:24, Adam Huffman > wrote: Hello A couple of weeks ago I joined Bruno Silva?s HPC team at the Francis Crick Institute, with special responsibility for HPC, OpenStack and virtualization. I?m very much a GPFS novice so I?m hoping to be able to draw on the knowledge in this group, while hopefully being able to help others with OpenStack. As Bruno stated in his message, we?re particularly interested in how to present GPFS to instances securely. I?ve read the discussion from November on this list, which didn?t seem to come to any firm conclusions. Has anyone involved then made substantial progress since? Cheers, Adam ? Adam Huffman Senior HPC & Virtualization Systems Engineer The Francis Crick Institute Gibbs Building 215 Euston Road London NW1 2BE T: E: adam.huffman at crick.ac.uk W: www.crick.ac.uk <158254D2-CDA8-43E5-96C1-A0AAEB10314B.png> The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no.06885462, with its registered office at 215 Euston Road, London NW1 2BE _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger_eriksson at se.ibm.com Thu Mar 12 08:25:22 2015 From: roger_eriksson at se.ibm.com (Roger Eriksson) Date: Thu, 12 Mar 2015 09:25:22 +0100 Subject: [gpfsug-discuss] GPFS UG 2015 date potential alteration In-Reply-To: <54EDD5E8.7090905@gpfsug.org> References: <54EDD5E8.7090905@gpfsug.org> Message-ID: Hi, has the GPFS User Group Meeting date been settled yet ? //Roger _________________________________________________________________ H?lsningar / Regards __________________________________________________________________ Roger Eriksson IBM Systems Lab Services Senior Accredited Product Services Professional XIV / Flashsystems / SVC / Storwize / SONAS / SAN EMEA Storage Competence Center (ESCC), Mainz, Germany IBM Svenska AB Isafjordsgatan 1 SE-164 92 Stockholm, Sweden Mobile +46(0)707933518 Email: roger_eriksson at se.ibm.com Notes: Roger Eriksson/Sweden/IBM at IBMSE Need help from EMEA Storage Competence Center ? IBMer ESCC @ Intranet --> http://escc.mainz.de.ibm.com/escc4u BPs ESCC @ Internet --> http://www.ibm.com/partnerworld/page/pw_com_bp_ecampus From: "Jez Tucker (Chair)" To: gpfsug main discussion list Date: 25/02/2015 15:02 Subject: [gpfsug-discuss] GPFS UG 2015 date potential alteration Sent by: gpfsug-discuss-bounces at gpfsug.org Alteration. Due to a scheduling issue, we may have to potentially move the date from 1st May. We will confirm this as soon as possible. Apologies. Jez _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss S?vida annat inte anges ovan: / Unless stated otherwise above: IBM Svenska AB Organisationsnummer: 556026-6883 Adress: 164 92 Stockholm -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 7526 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 7414 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 8944 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 9733 bytes Desc: not available URL: From chair at gpfsug.org Thu Mar 12 16:24:52 2015 From: chair at gpfsug.org (Jez Tucker (Chair)) Date: Thu, 12 Mar 2015 16:24:52 +0000 Subject: [gpfsug-discuss] GPFS UG 2015 date potential alteration In-Reply-To: References: <54EDD5E8.7090905@gpfsug.org> Message-ID: <5501BDD4.5060506@gpfsug.org> Hello all A fair question. We are working to a date of Wednesday 20th May. So please pencil this. The last pieces of paper work are being put in place, so please don't anybody book flights / expenses until this has been done. I would like to be in a position to have a definitive update on Monday. Jez On 12/03/15 08:25, Roger Eriksson wrote: > Hi, > > has the GPFS User Group Meeting date been settled yet ? > > //Roger > */_________________________________________________________________/* > */H?lsningar / Regards/* > */__________________________________________________________________/* > */Roger Eriksson/* > IBM Systems Lab Services > Senior Accredited Product Services Professional > XIV / Flashsystems / SVC / Storwize / SONAS / SAN > EMEA Storage Competence Center (ESCC), Mainz, Germany > IBM Svenska AB > Isafjordsgatan 1 > SE-164 92 Stockholm, Sweden > Mobile +46(0)707933518 > Email: _roger_eriksson at se.ibm.com_ > Notes: Roger Eriksson/Sweden/IBM at IBMSE > > > *Need help from EMEA Storage Competence Center ? * > *IBMer ESCC @ Intranet --> **http://escc.mainz.de.ibm.com/escc4u* > *BPs ESCC @ Internet --> > **http://www.ibm.com/partnerworld/page/pw_com_bp_ecampus* > > > > From: "Jez Tucker (Chair)" > To: gpfsug main discussion list > Date: 25/02/2015 15:02 > Subject: [gpfsug-discuss] GPFS UG 2015 date potential alteration > Sent by: gpfsug-discuss-bounces at gpfsug.org > ------------------------------------------------------------------------ > > > > Alteration. > > Due to a scheduling issue, we may have to potentially move the date > from 1st May. > > We will confirm this as soon as possible. > > Apologies. > > Jez > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > S?vida annat inte anges ovan: / Unless stated otherwise above: > IBM Svenska AB > Organisationsnummer: 556026-6883 > Adress: 164 92 Stockholm > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 7526 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 7414 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 8944 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 9733 bytes Desc: not available URL: From luke.raimbach at oerc.ox.ac.uk Fri Mar 13 14:44:33 2015 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Fri, 13 Mar 2015 14:44:33 +0000 Subject: [gpfsug-discuss] GPFS Journal Logs Message-ID: Hi, I'm interested in knowing more about the internal log journals GPFS uses to track file system activity from client nodes. My understanding is that when a file is opened for writing on a node, that (first) node becomes the 'metanode' for the file and is responsible for making updates to the file's metadata. Does the matanode also have responsibility for tracking changes made by other nodes that open the same file in its own journal, or are these recorded in each nodes own journal? I'm also interested in what happens when a node fails and its logs are replayed by the FS manager... what happens if that node is acting as metanode for a lot of open files which are also open elsewhere for writing on other nodes? If in the above scenario the metanode is handling metadata updates and tracking other changes in its journal, do all other nodes need to wait for the metanode to be ejected and have its journal replayed before they can continue? What is the process for selecting a successor metanode for the open files? Another related question I have is about initial sizing when creating a file system. Presumably the estimate for the total number of nodes mounting the file system affects the structure / allocation of internal journal logs? How are these arranged and is there any advice to overestimate this number if you intend to add more nodes at a future time? In the same vein, do multicluster nodes that join another remote stripegroup get allocated internal journal space as if they were new nodes in the remote cluster? Are these nodes allowed to be metanodes for files in a remote cluster? What happens when these nodes fail? Any insights greatly appreciated! Cheers, Luke. -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 From richard.lefebvre at calculquebec.ca Fri Mar 13 18:20:14 2015 From: richard.lefebvre at calculquebec.ca (Richard Lefebvre) Date: Fri, 13 Mar 2015 14:20:14 -0400 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. Message-ID: Hi, I have a GPFS cluster (v3.5). Last month, one of the filesystem had a problem with one of it's NSD. With the help tech support, the problem was fix and everything is OK data wise. The thing is when the problem occurred the filesystem was offline and once the filesystem was back to being OK, we were able to remount it. The problem is that now, every time a node is restarted, the filesystem is not mounted automatically as before. The other filesystems mount automatically. I did a mmlsfs on the file system and -A is at yes. Can someone tell my what I'm missing here. I'm sure it is something simple, but I'm drawing a blank right now. Richard -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) -------------- next part -------------- An HTML attachment was scrubbed... URL: From chair at gpfsug.org Mon Mar 16 14:02:09 2015 From: chair at gpfsug.org (Jez Tucker (Chair)) Date: Mon, 16 Mar 2015 14:02:09 +0000 Subject: [gpfsug-discuss] GPFS UG Meeting Agenda Message-ID: <5506E261.8000403@gpfsug.org> Hello all We have an internal agenda for the forthcoming UG. Before we publish this, I thought it prudent to solicit requests from our user base so that we might not miss anything important. So; - What's your burning issue, what's the one thing you really want to see? Regards, Jez From janfrode at tanso.net Mon Mar 16 15:11:13 2015 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Mon, 16 Mar 2015 16:11:13 +0100 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: Message-ID: Not much details to go by here, but I know I've had problems with SElinux blocking for getting filesystems mounted automatically at boot, while manually mounting afterwards worked fine. I fixed that by disabling automatic startup of gpfs from standard initscripts, and used this in rc.local instead: # make sure GPFS is not confined by selinux policies: runcon -t unconfined_t /usr/lpp/mmfs/bin/mmstartup -jf On Fri, Mar 13, 2015 at 7:20 PM, Richard Lefebvre < richard.lefebvre at calculquebec.ca> wrote: > Hi, > > I have a GPFS cluster (v3.5). Last month, one of the filesystem had a > problem with one of it's NSD. With the help tech support, the problem was > fix and everything is OK data wise. The thing is when the problem occurred > the filesystem was offline and once the filesystem was back to being OK, we > were able to remount it. The problem is that now, every time a node is > restarted, the filesystem is not mounted automatically as before. The other > filesystems mount automatically. I did a mmlsfs on the file system and -A > is at yes. Can someone tell my what I'm missing here. I'm sure it is > something simple, but I'm drawing a blank right now. > > Richard > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From YARD at il.ibm.com Mon Mar 16 15:14:25 2015 From: YARD at il.ibm.com (Yaron Daniel) Date: Mon, 16 Mar 2015 17:14:25 +0200 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: Message-ID: Hi In case of Linux - Check also : #chkconfig --list |grep gpfs Regards Yaron Daniel 94 Em Ha'Moshavot Rd Server, Storage and Data Services - Team Leader Petach Tiqva, 49527 Global Technology Services Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com IBM Israel From: Jan-Frode Myklebust To: gpfsug main discussion list Date: 03/16/2015 05:11 PM Subject: Re: [gpfsug-discuss] GPFS file system won't mount automatically anymore. Sent by: gpfsug-discuss-bounces at gpfsug.org Not much details to go by here, but I know I've had problems with SElinux blocking for getting filesystems mounted automatically at boot, while manually mounting afterwards worked fine. I fixed that by disabling automatic startup of gpfs from standard initscripts, and used this in rc.local instead: # make sure GPFS is not confined by selinux policies: runcon -t unconfined_t /usr/lpp/mmfs/bin/mmstartup -jf On Fri, Mar 13, 2015 at 7:20 PM, Richard Lefebvre < richard.lefebvre at calculquebec.ca> wrote: Hi, I have a GPFS cluster (v3.5). Last month, one of the filesystem had a problem with one of it's NSD. With the help tech support, the problem was fix and everything is OK data wise. The thing is when the problem occurred the filesystem was offline and once the filesystem was back to being OK, we were able to remount it. The problem is that now, every time a node is restarted, the filesystem is not mounted automatically as before. The other filesystems mount automatically. I did a mmlsfs on the file system and -A is at yes. Can someone tell my what I'm missing here. I'm sure it is something simple, but I'm drawing a blank right now. Richard -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1851 bytes Desc: not available URL: From luke.raimbach at oerc.ox.ac.uk Tue Mar 17 12:52:25 2015 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Tue, 17 Mar 2015 12:52:25 +0000 Subject: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans Message-ID: Hi Experts, I'm worrying about creating independent filesets which start to get in the way of one another: Let's pretend I create independent "fileset1" with a new inode space and preallocate say 1 million inodes. I start pushing files in to fileset1. I also start pushing files in to the root fileset. When fileset1 fills up, and because I've not specified a maximum number of inodes at creation time, presumably the file system manager gets on the case and preallocates more inodes as needed for fileset1. But while fileset1 has been filling up, I've also been filling up the root fileset (pretend I put 5 million files in there by the time fileset1 filled up). Now what happens? I'm guessing the FS manager starts chopping out sections of the root fileset beyond the 6 million region and marking them as independent and belonging to fileset1, in much the same way as it extends the allocated inode space when the file system starts running low. I hope this happens anyway. Let's say it does. So, now what about performance when running policy engine scans? I'm wanting to speed things up by restricting the policy scan to fileset1 (because I know that's where I put my special files). So I write the rules to select files and give them to mmapplypolicy. Does performance go down the drain as my once neatly arranged inode space is fragmented by not having enough inodes preallocated to begin with? Imagine fileset1 and the root fileset filling continuously at a similar rate and constantly "overtaking" one another in the inode space... does the performance advantage of independence start to go away rapidly? Cheers, Luke. -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 From makaplan at us.ibm.com Tue Mar 17 14:22:58 2015 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 17 Mar 2015 10:22:58 -0400 Subject: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans In-Reply-To: References: Message-ID: Luke, Thanks for your question. Independent filesets and their inodes are not implemented the way you might be imagining or guessing. Suppose you have two independent filesets "root" and "fset2" in the same GPFS file system. It is true that all the inode records (typically 512 bytes each - see mmcrfs) go into the same special file. BUT if you look at any given metadata allocation block --metadata-block-size (defaults to 256K) you'll only see inodes for either "root" or "fset2" not both in the same block. Moreover we try to "pre"-allocate several blocks of inodes for the same independent fileset contiguously - so that typically GPFS can do one seek and then read several blocks of inodes from the same independent fileset. So there can be performance advantages to using independent filesets and restricting your mmapplypolicy scans to just the fileset you need. To gain maximal advantage, use the following form of the command: mmapplypolicy /gpfs/path-to-the-directory-tree-I-want-to-scan --scope {fileset | inodespace } ... An inodespace is the set of all inodes in an independent fileset. An inodespace may contain several "dependent" filesets. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 21994 bytes Desc: not available URL: From zgiles at gmail.com Tue Mar 17 15:49:06 2015 From: zgiles at gmail.com (Zachary Giles) Date: Tue, 17 Mar 2015 11:49:06 -0400 Subject: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans In-Reply-To: References: Message-ID: I've always wondered also what the decision point is to make new inode tables per fileset. Sounds like the main benefit is scope of mmapplypolicy ( and other commands maybe ) and subblocks being grouped together for performance reasons. Is that right? Are there other benefits like portability or grouping a fileset's inode table into specific failure groups (in the system pool) etc? Any downsides? ( for example: now you have more overall metadata reads.. or only a limited number of inode tables.. or total number of inodes in the system is higher due to overallocation on N tables ? ) -Zach On Tue, Mar 17, 2015 at 10:22 AM, Marc A Kaplan wrote: > Luke, > > Thanks for your question. Independent filesets and their inodes are not > implemented the way you might be imagining or guessing. > > Suppose you have two independent filesets "root" and "fset2" in the same > GPFS file system. > It is true that all the inode records (typically 512 bytes each - see > mmcrfs) go into the same special file. BUT if you look at any given > metadata allocation block --metadata-block-size (defaults to 256K) you'll > only see inodes for either "root" or "fset2" not both in the same block. > > Moreover we try to "pre"-allocate several blocks of inodes for the same > independent fileset contiguously - so that typically GPFS can do one seek > and then read several blocks of inodes from the same independent fileset. > > So there can be performance advantages to using independent filesets and > restricting your mmapplypolicy scans to just the fileset you need. > To gain maximal advantage, use the following form of the command: > > mmapplypolicy /gpfs/path-to-the-directory-tree-I-want-to-scan --scope > {fileset | inodespace } ... > > > An inodespace is the set of all inodes in an independent fileset. An > inodespace may contain several "dependent" filesets. > > > > > [image: Marc A Kaplan] > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- Zach Giles zgiles at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 21994 bytes Desc: not available URL: From Paul.Sanchez at deshaw.com Tue Mar 17 16:01:21 2015 From: Paul.Sanchez at deshaw.com (Sanchez, Paul) Date: Tue, 17 Mar 2015 16:01:21 +0000 Subject: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans In-Reply-To: References: Message-ID: <201D6001C896B846A9CFC2E841986AC145360AA6@mailnycmb2a.winmail.deshaw.com> It?s also required for per-fileset snapshots, which are useful if you have differing snapshot requirements or want the ability to flush high-churn snapshots that are consuming an unexpected amount of space, but with better resolution than a whole filesystem. -Paul Sanchez From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Zachary Giles Sent: Tuesday, March 17, 2015 11:49 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans I've always wondered also what the decision point is to make new inode tables per fileset. Sounds like the main benefit is scope of mmapplypolicy ( and other commands maybe ) and subblocks being grouped together for performance reasons. Is that right? Are there other benefits like portability or grouping a fileset's inode table into specific failure groups (in the system pool) etc? Any downsides? ( for example: now you have more overall metadata reads.. or only a limited number of inode tables.. or total number of inodes in the system is higher due to overallocation on N tables ? ) -Zach On Tue, Mar 17, 2015 at 10:22 AM, Marc A Kaplan > wrote: Luke, Thanks for your question. Independent filesets and their inodes are not implemented the way you might be imagining or guessing. Suppose you have two independent filesets "root" and "fset2" in the same GPFS file system. It is true that all the inode records (typically 512 bytes each - see mmcrfs) go into the same special file. BUT if you look at any given metadata allocation block --metadata-block-size (defaults to 256K) you'll only see inodes for either "root" or "fset2" not both in the same block. Moreover we try to "pre"-allocate several blocks of inodes for the same independent fileset contiguously - so that typically GPFS can do one seek and then read several blocks of inodes from the same independent fileset. So there can be performance advantages to using independent filesets and restricting your mmapplypolicy scans to just the fileset you need. To gain maximal advantage, use the following form of the command: mmapplypolicy /gpfs/path-to-the-directory-tree-I-want-to-scan --scope {fileset | inodespace } ... An inodespace is the set of all inodes in an independent fileset. An inodespace may contain several "dependent" filesets. [Marc A Kaplan] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Zach Giles zgiles at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 21994 bytes Desc: image001.gif URL: From zgiles at gmail.com Tue Mar 17 16:04:25 2015 From: zgiles at gmail.com (Zachary Giles) Date: Tue, 17 Mar 2015 12:04:25 -0400 Subject: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans In-Reply-To: <201D6001C896B846A9CFC2E841986AC145360AA6@mailnycmb2a.winmail.deshaw.com> References: <201D6001C896B846A9CFC2E841986AC145360AA6@mailnycmb2a.winmail.deshaw.com> Message-ID: Ah. that's a good point about the snapshots. On Tue, Mar 17, 2015 at 12:01 PM, Sanchez, Paul wrote: > It?s also required for per-fileset snapshots, which are useful if you > have differing snapshot requirements or want the ability to flush > high-churn snapshots that are consuming an unexpected amount of space, but > with better resolution than a whole filesystem. > > > > -Paul Sanchez > > > > *From:* gpfsug-discuss-bounces at gpfsug.org [mailto: > gpfsug-discuss-bounces at gpfsug.org] *On Behalf Of *Zachary Giles > *Sent:* Tuesday, March 17, 2015 11:49 AM > *To:* gpfsug main discussion list > *Subject:* Re: [gpfsug-discuss] Fragmented Inode Space and Performance of > Policy Scans > > > > I've always wondered also what the decision point is to make new inode > tables per fileset. > > > > Sounds like the main benefit is scope of mmapplypolicy ( and other > commands maybe ) and subblocks being grouped together for performance > reasons. Is that right? > > Are there other benefits like portability or grouping a fileset's inode > table into specific failure groups (in the system pool) etc? > > Any downsides? ( for example: now you have more overall metadata reads.. > or only a limited number of inode tables.. or total number of inodes in the > system is higher due to overallocation on N tables ? ) > > > > -Zach > > > > > > On Tue, Mar 17, 2015 at 10:22 AM, Marc A Kaplan > wrote: > > Luke, > > Thanks for your question. Independent filesets and their inodes are not > implemented the way you might be imagining or guessing. > > Suppose you have two independent filesets "root" and "fset2" in the same > GPFS file system. > It is true that all the inode records (typically 512 bytes each - see > mmcrfs) go into the same special file. BUT if you look at any given > metadata allocation block --metadata-block-size (defaults to 256K) you'll > only see inodes for either "root" or "fset2" not both in the same block. > > Moreover we try to "pre"-allocate several blocks of inodes for the same > independent fileset contiguously - so that typically GPFS can do one seek > and then read several blocks of inodes from the same independent fileset. > > So there can be performance advantages to using independent filesets and > restricting your mmapplypolicy scans to just the fileset you need. > To gain maximal advantage, use the following form of the command: > > mmapplypolicy /gpfs/path-to-the-directory-tree-I-want-to-scan --scope > {fileset | inodespace } ... > > > An inodespace is the set of all inodes in an independent fileset. An > inodespace may contain several "dependent" filesets. > > > > > [image: Marc A Kaplan] > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > -- > > Zach Giles > zgiles at gmail.com > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- Zach Giles zgiles at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 21994 bytes Desc: not available URL: From chair at gpfsug.org Tue Mar 17 16:47:43 2015 From: chair at gpfsug.org (Jez Tucker (Chair)) Date: Tue, 17 Mar 2015 16:47:43 +0000 Subject: [gpfsug-discuss] GPFS User Group 2015 Meeting Date confirmed - RSVP Message-ID: <55085AAF.8050108@gpfsug.org> Hello all I can now confirm that 2015 GPFS User Group Meeting will be on May 20th, in tandem with SPXXL. For full details please see: http://www.gpfsug.org/gpfs-user-group-2015-invite/ RSVP to secretary at gpfsug.org See you there. Jez From richard.lefebvre at calculquebec.ca Thu Mar 19 22:32:58 2015 From: richard.lefebvre at calculquebec.ca (Richard Lefebvre) Date: Thu, 19 Mar 2015 18:32:58 -0400 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: Message-ID: Thanks for those who replied. One thing I didn't mention correctly is that I have several gpfs mount say: /gpfs1 /gpfs2 /gpfs3 Say it was gpfs2 that had the NSD problem. So now when I reboot a node, /gpfs1 and /gpfs3 mount, but /gpfs2 doesn't, I have to manually type mmmount gpfs2 on the node to get it mounted after the boot. So I don't think that it is a problem with mounting within linux boot sequence. But a flag than I'm missing in the gpfs setup. Or status of the filesystem that is not set OK that I cannot figure out. The output of mmlsfs of a system that mounts compared the the one that doesn't are the same except for mount point, NSDs and creation time. Is there another "mm..." command with which to check the status of a filesystem? Richard 2015-03-13 14:20 GMT-04:00 Richard Lefebvre < richard.lefebvre at calculquebec.ca>: > Hi, > > I have a GPFS cluster (v3.5). Last month, one of the filesystem had a > problem with one of it's NSD. With the help tech support, the problem was > fix and everything is OK data wise. The thing is when the problem occurred > the filesystem was offline and once the filesystem was back to being OK, we > were able to remount it. The problem is that now, every time a node is > restarted, the filesystem is not mounted automatically as before. The other > filesystems mount automatically. I did a mmlsfs on the file system and -A > is at yes. Can someone tell my what I'm missing here. I'm sure it is > something simple, but I'm drawing a blank right now. > > Richard > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Thu Mar 19 22:35:38 2015 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Thu, 19 Mar 2015 22:35:38 +0000 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: Message-ID: Check "mmlsdisk". Are all disks for the filesystem ok? -jf On tor. 19. mar. 2015 at 23.33 Richard Lefebvre < richard.lefebvre at calculquebec.ca> wrote: > Thanks for those who replied. One thing I didn't mention correctly is that > I have several gpfs mount say: > /gpfs1 > /gpfs2 > /gpfs3 > > Say it was gpfs2 that had the NSD problem. So now when I reboot a node, > /gpfs1 and /gpfs3 mount, but /gpfs2 doesn't, I have to manually type > mmmount gpfs2 on the node to get it mounted after the boot. > > So I don't think that it is a problem with mounting within linux boot > sequence. But a flag than I'm missing in the gpfs setup. Or status of the > filesystem that is not set OK that I cannot figure out. The output of > mmlsfs of a system that mounts compared the the one that doesn't are the > same except for mount point, NSDs and creation time. Is there another > "mm..." command with which to check the status of a filesystem? > > Richard > > > 2015-03-13 14:20 GMT-04:00 Richard Lefebvre < > richard.lefebvre at calculquebec.ca>: > > Hi, >> >> I have a GPFS cluster (v3.5). Last month, one of the filesystem had a >> problem with one of it's NSD. With the help tech support, the problem was >> fix and everything is OK data wise. The thing is when the problem occurred >> the filesystem was offline and once the filesystem was back to being OK, we >> were able to remount it. The problem is that now, every time a node is >> restarted, the filesystem is not mounted automatically as before. The other >> filesystems mount automatically. I did a mmlsfs on the file system and -A >> is at yes. Can someone tell my what I'm missing here. I'm sure it is >> something simple, but I'm drawing a blank right now. >> >> Richard >> >> -- >> Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" >> Richard.Lefebvre at calculquebec.ca -- THGTTG >> Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) >> > > > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.lefebvre at calculquebec.ca Thu Mar 19 22:38:43 2015 From: richard.lefebvre at calculquebec.ca (Richard Lefebvre) Date: Thu, 19 Mar 2015 18:38:43 -0400 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: Message-ID: Hi JF, There are all up and ready. Otherwise I wouldn't be able to mmmount it. Richard 2015-03-19 18:35 GMT-04:00 Jan-Frode Myklebust : > Check "mmlsdisk". Are all disks for the filesystem ok? > > > -jf > On tor. 19. mar. 2015 at 23.33 Richard Lefebvre < > richard.lefebvre at calculquebec.ca> wrote: > >> Thanks for those who replied. One thing I didn't mention correctly is >> that I have several gpfs mount say: >> /gpfs1 >> /gpfs2 >> /gpfs3 >> >> Say it was gpfs2 that had the NSD problem. So now when I reboot a node, >> /gpfs1 and /gpfs3 mount, but /gpfs2 doesn't, I have to manually type >> mmmount gpfs2 on the node to get it mounted after the boot. >> >> So I don't think that it is a problem with mounting within linux boot >> sequence. But a flag than I'm missing in the gpfs setup. Or status of the >> filesystem that is not set OK that I cannot figure out. The output of >> mmlsfs of a system that mounts compared the the one that doesn't are the >> same except for mount point, NSDs and creation time. Is there another >> "mm..." command with which to check the status of a filesystem? >> >> Richard >> >> >> 2015-03-13 14:20 GMT-04:00 Richard Lefebvre < >> richard.lefebvre at calculquebec.ca>: >> >> Hi, >>> >>> I have a GPFS cluster (v3.5). Last month, one of the filesystem had a >>> problem with one of it's NSD. With the help tech support, the problem was >>> fix and everything is OK data wise. The thing is when the problem occurred >>> the filesystem was offline and once the filesystem was back to being OK, we >>> were able to remount it. The problem is that now, every time a node is >>> restarted, the filesystem is not mounted automatically as before. The other >>> filesystems mount automatically. I did a mmlsfs on the file system and -A >>> is at yes. Can someone tell my what I'm missing here. I'm sure it is >>> something simple, but I'm drawing a blank right now. >>> >>> Richard >>> >>> -- >>> Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" >>> Richard.Lefebvre at calculquebec.ca -- THGTTG >>> Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) >>> >> >> >> >> -- >> Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" >> Richard.Lefebvre at calculquebec.ca -- THGTTG >> Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at gpfsug.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) -------------- next part -------------- An HTML attachment was scrubbed... URL: From red_steel at au1.ibm.com Fri Mar 20 01:48:52 2015 From: red_steel at au1.ibm.com (Tony Steel) Date: Fri, 20 Mar 2015 12:48:52 +1100 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: Message-ID: Hi You don't have a file (assuming gpfs2 is the problem) /var/mmfs/etc/ignoreStartupMount.gpfs2 ?? Cheers, - Sense Amid Madness, Wit Amidst Folly - ---------- Antony (Red) Steel, Advanced Technical Support -------------------- Senior IT Specialist Ph: +61 293978414 Mb: +61 409 971 083 Home Page: http://sydgsa.ibm.com/projects/t/team_web/index.html ----------------------------- red_steel at au.ibm.com ----------------------------------- From ckerner at ncsa.uiuc.edu Fri Mar 20 02:56:55 2015 From: ckerner at ncsa.uiuc.edu (Chad Kerner) Date: Thu, 19 Mar 2015 21:56:55 -0500 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: Message-ID: <20150320025655.GA2140@logos.ncsa.illinois.edu> Richard, what does "mmlsfs gpfs2 -A" return? Is it set to automount? --Chad Chad Kerner, Systems Engineer Storage Enabling Technologies National Center for Supercomputing Applications University of Illinois, Urbana-Champaign On Thu, Mar 19, 2015 at 06:38:43PM -0400, Richard Lefebvre wrote: > Hi JF, > > There are all up and ready. Otherwise I wouldn't be able to mmmount it. > > Richard > > > > 2015-03-19 18:35 GMT-04:00 Jan-Frode Myklebust : > > Check "mmlsdisk". Are all disks for the filesystem ok? > > > -jf > On tor. 19. mar. 2015 at 23.33 Richard Lefebvre < > richard.lefebvre at calculquebec.ca> wrote: > > Thanks for those who replied. One thing I didn't mention correctly is > that I have several gpfs mount say: > /gpfs1 > /gpfs2 > /gpfs3 > > Say it was gpfs2 that had the NSD problem. So now when I reboot a node, > /gpfs1 and /gpfs3 mount, but /gpfs2 doesn't, I have to manually type > mmmount gpfs2 on the node to get it mounted after the boot. > > So I don't think that it is a problem with mounting within linux boot > sequence. But a flag than I'm missing in the gpfs setup. Or status of > the filesystem that is not set OK that I cannot figure out. The output > of mmlsfs of a system that mounts compared the the one that doesn't are > the same except for mount point, NSDs and creation time. Is there > another "mm..." command with which to check the status of a filesystem? > > Richard > > > 2015-03-13 14:20 GMT-04:00 Richard Lefebvre < > richard.lefebvre at calculquebec.ca>: > > > Hi, > > I have a GPFS cluster (v3.5). Last month, one of the filesystem had > a problem with one of it's NSD. With the help tech support, the > problem was fix and everything is OK data wise. The thing is when > the problem occurred the filesystem was offline and once the > filesystem was back to being OK, we were able to remount it. The > problem is that now, every time a node is restarted, the filesystem > is not mounted automatically as before. The other filesystems mount > automatically. I did a mmlsfs on the file system and -A is at yes. > Can someone tell my what I'm missing here. I'm sure it is something > simple, but I'm drawing a blank right now. > > Richard > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't > Panic" > Richard.Lefebvre at calculquebec.ca -- > THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada ( > computecanada.ca) > > > > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't > Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From luke.raimbach at oerc.ox.ac.uk Fri Mar 20 08:37:02 2015 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Fri, 20 Mar 2015 08:37:02 +0000 Subject: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans Message-ID: Hi Marc, Thanks for this information. One thing I still can't extract from the manual (though I guess I could just do a small experiment to find out) is whether the number of allocated inodes is automatically extended when running low, up to the maximum for an independent fileset? Cheers, Luke. From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Marc A Kaplan Sent: 17 March 2015 14:23 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans Luke, Thanks for your question. Independent filesets and their inodes are not implemented the way you might be imagining or guessing. Suppose you have two independent filesets "root" and "fset2" in the same GPFS file system. It is true that all the inode records (typically 512 bytes each - see mmcrfs) go into the same special file. BUT if you look at any given metadata allocation block --metadata-block-size (defaults to 256K) you'll only see inodes for either "root" or "fset2" not both in the same block. Moreover we try to "pre"-allocate several blocks of inodes for the same independent fileset contiguously - so that typically GPFS can do one seek and then read several blocks of inodes from the same independent fileset. So there can be performance advantages to using independent filesets and restricting your mmapplypolicy scans to just the fileset you need. To gain maximal advantage, use the following form of the command: mmapplypolicy /gpfs/path-to-the-directory-tree-I-want-to-scan --scope {fileset | inodespace } ... An inodespace is the set of all inodes in an independent fileset. An inodespace may contain several "dependent" filesets. [Marc A Kaplan] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 21994 bytes Desc: image001.gif URL: From makaplan at us.ibm.com Fri Mar 20 12:10:34 2015 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 20 Mar 2015 08:10:34 -0400 Subject: [gpfsug-discuss] Fragmented Inode Space and Performance of Policy Scans In-Reply-To: References: Message-ID: Seems to me that's why this option has two parameters: mmchfileset ... [--inode-limit MaxNumInodes[:NumInodesToPreallocate]] Specifies the new inode limit for the inode space owned by the specified fileset. The FilesetName or JunctionPath must refer to an independent fileset. The NumInodesToPreallocate specifies an optional number of additional inodes to pre-allocate for the inode space. Use the mmchfs command to change inode limits for the root fileset. See also mmchfs! And RTFM! ( ReadTheFantasicManual ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.lefebvre at calculquebec.ca Fri Mar 20 17:22:43 2015 From: richard.lefebvre at calculquebec.ca (Richard Lefebvre) Date: Fri, 20 Mar 2015 13:22:43 -0400 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: <20150320025655.GA2140@logos.ncsa.illinois.edu> References: <20150320025655.GA2140@logos.ncsa.illinois.edu> Message-ID: Hi Tony, Chad, There is nothing in /var/mmfs/etc directory of the nodes. "-A" on the nodes return "yes" Richard 2015-03-19 22:56 GMT-04:00 Chad Kerner : > Richard, what does "mmlsfs gpfs2 -A" return? Is it set to automount? > > --Chad > Chad Kerner, Systems Engineer > Storage Enabling Technologies > National Center for Supercomputing Applications > University of Illinois, Urbana-Champaign > > On Thu, Mar 19, 2015 at 06:38:43PM -0400, Richard Lefebvre wrote: > > Hi JF, > > > > There are all up and ready. Otherwise I wouldn't be able to mmmount it. > > > > Richard > > > > > > > > 2015-03-19 18:35 GMT-04:00 Jan-Frode Myklebust : > > > > Check "mmlsdisk". Are all disks for the filesystem ok? > > > > > > -jf > > On tor. 19. mar. 2015 at 23.33 Richard Lefebvre < > > richard.lefebvre at calculquebec.ca> wrote: > > > > Thanks for those who replied. One thing I didn't mention > correctly is > > that I have several gpfs mount say: > > /gpfs1 > > /gpfs2 > > /gpfs3 > > > > Say it was gpfs2 that had the NSD problem. So now when I reboot > a node, > > /gpfs1 and /gpfs3 mount, but /gpfs2 doesn't, I have to manually > type > > mmmount gpfs2 on the node to get it mounted after the boot. > > > > So I don't think that it is a problem with mounting within linux > boot > > sequence. But a flag than I'm missing in the gpfs setup. Or > status of > > the filesystem that is not set OK that I cannot figure out. The > output > > of mmlsfs of a system that mounts compared the the one that > doesn't are > > the same except for mount point, NSDs and creation time. Is there > > another "mm..." command with which to check the status of a > filesystem? > > > > Richard > > > > > > 2015-03-13 14:20 GMT-04:00 Richard Lefebvre < > > richard.lefebvre at calculquebec.ca>: > > > > > > Hi, > > > > I have a GPFS cluster (v3.5). Last month, one of the > filesystem had > > a problem with one of it's NSD. With the help tech support, > the > > problem was fix and everything is OK data wise. The thing is > when > > the problem occurred the filesystem was offline and once the > > filesystem was back to being OK, we were able to remount it. > The > > problem is that now, every time a node is restarted, the > filesystem > > is not mounted automatically as before. The other > filesystems mount > > automatically. I did a mmlsfs on the file system and -A is > at yes. > > Can someone tell my what I'm missing here. I'm sure it is > something > > simple, but I'm drawing a blank right now. > > > > Richard > > > > -- > > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 > "Don't > > Panic" > > Richard.Lefebvre at calculquebec.ca > -- > > THGTTG > > Calcul Quebec (calculquebec.ca) ------ Calcul Canada ( > > computecanada.ca) > > > > > > > > > > -- > > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 > "Don't > > Panic" > > Richard.Lefebvre at calculquebec.ca > -- THGTTG > > Calcul Quebec (calculquebec.ca) ------ Calcul Canada ( > computecanada.ca) > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at gpfsug.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at gpfsug.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > > > > > > -- > > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" > > Richard.Lefebvre at calculquebec.ca -- THGTTG > > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at gpfsug.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) -------------- next part -------------- An HTML attachment was scrubbed... URL: From YARD at il.ibm.com Fri Mar 20 17:50:33 2015 From: YARD at il.ibm.com (Yaron Daniel) Date: Fri, 20 Mar 2015 19:50:33 +0200 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: <20150320025655.GA2140@logos.ncsa.illinois.edu> Message-ID: Hi Can u send the output of: #mmlsdisk gpfs2 #mmlsnsd |grep gpfs2 #getenforce #iptables -L #cat /etc/fstab Regards Yaron Daniel 94 Em Ha'Moshavot Rd Server, Storage and Data Services - Team Leader Petach Tiqva, 49527 Global Technology Services Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com IBM Israel From: Richard Lefebvre To: gpfsug main discussion list Date: 03/20/2015 07:22 PM Subject: Re: [gpfsug-discuss] GPFS file system won't mount automatically anymore. Sent by: gpfsug-discuss-bounces at gpfsug.org Hi Tony, Chad, There is nothing in /var/mmfs/etc directory of the nodes. "-A" on the nodes return "yes" Richard 2015-03-19 22:56 GMT-04:00 Chad Kerner : Richard, what does "mmlsfs gpfs2 -A" return? Is it set to automount? --Chad Chad Kerner, Systems Engineer Storage Enabling Technologies National Center for Supercomputing Applications University of Illinois, Urbana-Champaign On Thu, Mar 19, 2015 at 06:38:43PM -0400, Richard Lefebvre wrote: > Hi JF, > > There are all up and ready. Otherwise I wouldn't be able to mmmount it. > > Richard > > > > 2015-03-19 18:35 GMT-04:00 Jan-Frode Myklebust : > > Check "mmlsdisk". Are all disks for the filesystem ok? > > > -jf > On tor. 19. mar. 2015 at 23.33 Richard Lefebvre < > richard.lefebvre at calculquebec.ca> wrote: > > Thanks for those who replied. One thing I didn't mention correctly is > that I have several gpfs mount say: > /gpfs1 > /gpfs2 > /gpfs3 > > Say it was gpfs2 that had the NSD problem. So now when I reboot a node, > /gpfs1 and /gpfs3 mount, but /gpfs2 doesn't, I have to manually type > mmmount gpfs2 on the node to get it mounted after the boot. > > So I don't think that it is a problem with mounting within linux boot > sequence. But a flag than I'm missing in the gpfs setup. Or status of > the filesystem that is not set OK that I cannot figure out. The output > of mmlsfs of a system that mounts compared the the one that doesn't are > the same except for mount point, NSDs and creation time. Is there > another "mm..." command with which to check the status of a filesystem? > > Richard > > > 2015-03-13 14:20 GMT-04:00 Richard Lefebvre < > richard.lefebvre at calculquebec.ca>: > > > Hi, > > I have a GPFS cluster (v3.5). Last month, one of the filesystem had > a problem with one of it's NSD. With the help tech support, the > problem was fix and everything is OK data wise. The thing is when > the problem occurred the filesystem was offline and once the > filesystem was back to being OK, we were able to remount it. The > problem is that now, every time a node is restarted, the filesystem > is not mounted automatically as before. The other filesystems mount > automatically. I did a mmlsfs on the file system and -A is at yes. > Can someone tell my what I'm missing here. I'm sure it is something > simple, but I'm drawing a blank right now. > > Richard > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't > Panic" > Richard.Lefebvre at calculquebec.ca -- > THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada ( > computecanada.ca) > > > > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't > Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada ( computecanada.ca) > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1851 bytes Desc: not available URL: From makaplan at us.ibm.com Fri Mar 20 19:24:25 2015 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 20 Mar 2015 15:24:25 -0400 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: <20150320025655.GA2140@logos.ncsa.illinois.edu> Message-ID: Look in /var/adm/ras/mmfs.log.* -- on a node that should be mounting the filesystem -- find a file that covers the time when GPFS is starting up and should be mounting the filesystem. Look for clues and/or error messages. If you are not familiar, you may also want to look at what messages are issued when everything is working just right. -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.lefebvre at calculquebec.ca Fri Mar 20 20:32:46 2015 From: richard.lefebvre at calculquebec.ca (Richard Lefebvre) Date: Fri, 20 Mar 2015 16:32:46 -0400 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: <20150320025655.GA2140@logos.ncsa.illinois.edu> Message-ID: Hi Mark, Thank you for your answer. You are right, I have the following: Mon Mar 16 11:02:24 EDT 2015: mounting /dev/gpfs2 Mon Mar 16 11:02:24.296 2015: Command: mount gpfs2 Mon Mar 16 11:02:24.723 2015: Disk failure. Volume gpfs2. rc = 5. Physical volume gpfs2nsd04. Mon Mar 16 11:02:24.724 2015: Disk failure. Volume gpfs2. rc = 5. Physical volume gpfs2nsd12. Mon Mar 16 11:02:24.723 2015: File System gpfs2 unmounted by the system with return code 5 reason code 0 Mon Mar 16 11:02:24.724 2015: Input/output error Mon Mar 16 11:02:24.725 2015: Failed to open gpfs2. Mon Mar 16 11:02:24.724 2015: Input/output error Mon Mar 16 11:02:24.725 2015: Command: err 666: mount gpfs2 Mon Mar 16 11:02:24.724 2015: Input/output error mount: Stale NFS file handle Mon Mar 16 11:02:24 EDT 2015: finished mounting /dev/gpfs2 But it is not mounted. This is after issuing the "mmmount gpfs2" command Mon Mar 16 11:03:52.010 2015: Command: mount gpfs2 Mon Mar 16 11:03:52.234 2015: VERBS RDMA connecting to 172.25.101.2 (heca2-ib0) on mlx4_0 port 1 id 4 Mon Mar 16 11:03:52.235 2015: VERBS RDMA connected to 172.25.101.2 (heca2-ib0) on mlx4_0 port 1 sl 0 Mon Mar 16 11:03:52.381 2015: Command: err 0: mount gpfs2 Now, my next step it to figure out how the clear the failure since it is not failed anymore. The mmlsdisk reports all the nsd are up and ready. One command I'm also looking for is a command to list which nsd server is used for a particular nsd. I have a redundant config for which 2 servers can access the same nsd. But I think that for some of them, the default server is not used and the load of the servers is not distributed as wanted. Richard 2015-03-20 15:24 GMT-04:00 Marc A Kaplan : > Look in /var/adm/ras/mmfs.log.* -- on a node that should be mounting the > filesystem -- find a file that covers the time when GPFS is starting up > and should be mounting the filesystem. > > Look for clues and/or error messages. If you are not familiar, you may > also want to look at what messages are issued when everything is working > just right. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) -------------- next part -------------- An HTML attachment was scrubbed... URL: From Dan.Foster at bristol.ac.uk Mon Mar 23 08:43:33 2015 From: Dan.Foster at bristol.ac.uk (Dan Foster) Date: Mon, 23 Mar 2015 08:43:33 +0000 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: <20150320025655.GA2140@logos.ncsa.illinois.edu> Message-ID: Hi Richard, On 20 March 2015 at 20:32, Richard Lefebvre wrote: > One command I'm also looking for is a command to list which nsd server is > used for a particular nsd. I have a redundant config for which 2 servers can > access the same nsd. But I think that for some of them, the default server > is not used and the load of the servers is not distributed as wanted. Does the '-M' option to mmlsdisk provide the information you're looking for? i.e. from the man page: """ Displays whether I/O requests to the disk are satisfied on the local node, or using an NSD server. If the I/O is done using an NSD server, shows the NSD server name and the underlying disk name on that server node. """ It should display something similar to the following: Disk name IO performed on node Device Availability ------------ ----------------------- ----------------- ------------ nsd1 localhost /dev/disk1 up nsd2 node01.example.com /dev/disk2 up nsd3 localhost /dev/disk3 up HTH, Dan. > > 2015-03-20 15:24 GMT-04:00 Marc A Kaplan : >> >> Look in /var/adm/ras/mmfs.log.* -- on a node that should be mounting the >> filesystem -- find a file that covers the time when GPFS is starting up >> and should be mounting the filesystem. >> >> Look for clues and/or error messages. If you are not familiar, you may >> also want to look at what messages are issued when everything is working >> just right. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at gpfsug.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > > > -- > Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" > Richard.Lefebvre at calculquebec.ca -- THGTTG > Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Dan Foster | Senior Storage Systems Administrator | IT Services e: dan.foster at bristol.ac.uk | t: 0117 3941170 [x41170] m: Advanced Computing Research Centre, University of Bristol, 8-10 Berkeley Square, Bristol BS8 1HH From PEREZJOS at uk.ibm.com Mon Mar 23 14:13:08 2015 From: PEREZJOS at uk.ibm.com (Jose Perez) Date: Mon, 23 Mar 2015 14:13:08 +0000 Subject: [gpfsug-discuss] Jose Perez is OoO. Message-ID: I will be out of the office starting 23/03/2015 and will not return until 24/03/2015. I will respond to your message when I return. For urgent issues please contact Ross Keeping, Thanks. Thanks, From chair at gpfsug.org Mon Mar 23 14:34:20 2015 From: chair at gpfsug.org (Jez Tucker (Chair)) Date: Mon, 23 Mar 2015 14:34:20 +0000 Subject: [gpfsug-discuss] Jose Perez is OoO. In-Reply-To: References: Message-ID: <5510246C.6090009@gpfsug.org> Apols for that one which slipped through, we'll add 'OoO' to the list blocker. Jez On 23/03/15 14:13, Jose Perez wrote: > I will be out of the office starting 23/03/2015 and will not return until > 24/03/2015. > > I will respond to your message when I return. For urgent issues please > contact Ross Keeping, Thanks. > Thanks, > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From makaplan at us.ibm.com Mon Mar 23 14:33:29 2015 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 23 Mar 2015 10:33:29 -0400 Subject: [gpfsug-discuss] GPFS file system won't mount automatically anymore. In-Reply-To: References: <20150320025655.GA2140@logos.ncsa.illinois.edu> Message-ID: mmlsnsd has several options... RTFM... but this one will show you which nodes can directly access and/or serve each NSD: mmlsnsd -M -v -M Maps the NSD names to its disk device name on all nodes. This is a slow operation and its usage is suggested for problem determination only. -v Specifies that the output should contain error information, where available. -------------- next part -------------- An HTML attachment was scrubbed... URL: From secretary at gpfsug.org Wed Mar 25 14:42:52 2015 From: secretary at gpfsug.org (Secretary GPFS UG) Date: Wed, 25 Mar 2015 14:42:52 +0000 Subject: [gpfsug-discuss] 2nd 'Meet the Devs' Event 28/4/15 Message-ID: <6715b1493f52d222ab24b49195ab8fc2@webmail.gpfsug.org> Dear members, Our second Meet the GPFS Developers 'coffee shop' meet is going to be taking place on Tuesday 28th April. The day will give you the opportunity to meet the developers and get hands on beta exploration and test of unreleased GPFS functions. Date: 28th April 2015 Time: 10:30 - 15:30 Location: IBM, 3rd Floor, Maybrook House, 40 Blackfriars Street, Manchester M3 2EG Agenda: 10:30 Introduction discussion 11:30 Beta session 1 12:30 Break for pizza (social chats with wider GPFS dev team in Manchester) 13:30 Beta session 2 14:45 Group discussion and feedback 15:30 End Lunch and refreshments will be provided. To attend, please register with Claire Robson, GPFS UG Secretary secretary at gpfsug.org. 10 places are available. Full directions and instructions will be sent to confirmed attendees. We look forward to seeing you in April. Thanks, Claire -- GPFS User Group Secretary From ross.keeping at uk.ibm.com Thu Mar 26 14:59:31 2015 From: ross.keeping at uk.ibm.com (Ross Keeping3) Date: Thu, 26 Mar 2015 14:59:31 +0000 Subject: [gpfsug-discuss] Potential talks for GPFS UG May 20th Message-ID: Hi There are a couple of talks we could include in the one day GPFS UG on May 20th. We'd appreciate some feedback on whether there is any interest or objection to including these. These are: 1) Tivoli Storage Productivity Center Technical Demo: A tool for managing heterogeneous storage infrastructures and specifically GPFS. Can read more here http://www-03.ibm.com/software/products/en/tivostorprodcent 2) GPFS for life sciences: A member of NERSC would like to talk about their efforts to improve GPFS file systems for a life sciences workload - specifically genomic sequencing and analysis that ensues. Details about life science workflow and requirements will be provided so the listener may understand the aspects of GPFS that are important. Cheers, Ross Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From stijn.deweirdt at ugent.be Thu Mar 26 15:07:29 2015 From: stijn.deweirdt at ugent.be (Stijn De Weirdt) Date: Thu, 26 Mar 2015 16:07:29 +0100 Subject: [gpfsug-discuss] Potential talks for GPFS UG May 20th In-Reply-To: References: Message-ID: <551420B1.3080002@ugent.be> no objections for any of them, but i'd prefer talk 2 (if a choice has to be made) stijn On 03/26/2015 03:59 PM, Ross Keeping3 wrote: > Hi > > There are a couple of talks we could include in the one day GPFS UG on May > 20th. > > We'd appreciate some feedback on whether there is any interest or > objection to including these. > > These are: > 1) Tivoli Storage Productivity Center Technical Demo: A tool for managing > heterogeneous storage infrastructures and specifically GPFS. Can read more > here http://www-03.ibm.com/software/products/en/tivostorprodcent > > 2) GPFS for life sciences: A member of NERSC would like to talk about > their efforts to improve GPFS file systems for a life sciences workload - > specifically genomic sequencing and analysis that ensues. Details about > life science workflow and requirements will be provided so the listener > may understand the aspects of GPFS that are important. > > Cheers, > Ross > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From robert at strubi.ox.ac.uk Thu Mar 26 15:13:44 2015 From: robert at strubi.ox.ac.uk (Robert Esnouf) Date: Thu, 26 Mar 2015 15:13:44 +0000 (GMT) Subject: [gpfsug-discuss] Potential talks for GPFS UG May 20th In-Reply-To: References: Message-ID: <201503261513.062218@mail.strubi.ox.ac.uk> Yes, unsurprisingly I'd be interested in the second talk! Thanks, Robert -- Dr. Robert Esnouf, University Research Lecturer, Head of Research Computing Core, NDM Research Computing Strategy Officer Room 10/028, Wellcome Trust Centre for Human Genetics, Old Road Campus, Roosevelt Drive, Oxford OX3 7BN, UK Email: robert at strubi.ox.ac.uk / robert at well.ox.ac.uk Tel: (+44) - 1865 - 287783 ---- Original message ---- >Date: Thu, 26 Mar 2015 14:59:31 +0000 >From: gpfsug-discuss-bounces at gpfsug.org (on behalf of Ross Keeping3 ) >Subject: [gpfsug-discuss] Potential talks for GPFS UG May 20th >To: gpfsug-discuss at gpfsug.org > > Hi > > There are a couple of talks we could include in the > one day GPFS UG on May 20th. > > We'd appreciate some feedback on whether there is > any interest or objection to including these. > > These are: > 1) Tivoli Storage Productivity Center Technical > Demo: A tool for managing heterogeneous storage > infrastructures and specifically GPFS. Can read more > here > http://www-03.ibm.com/software/products/en/tivostorprodcent > > 2) GPFS for life sciences: A member of NERSC would > like to talk about their efforts to improve GPFS > file systems for a life sciences workload - > specifically genomic sequencing and analysis that > ensues. Details about life science workflow and > requirements will be provided so the listener may > understand the aspects of GPFS that are important. > > Cheers, > Ross > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England > and Wales with number 741598. > Registered office: PO Box 41, North Harbour, > Portsmouth, Hampshire PO6 3AU >________________ >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at gpfsug.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Mark.Roberts at awe.co.uk Thu Mar 26 15:24:17 2015 From: Mark.Roberts at awe.co.uk (Mark.Roberts at awe.co.uk) Date: Thu, 26 Mar 2015 15:24:17 +0000 Subject: [gpfsug-discuss] EXTERNAL: Potential talks for GPFS UG May 20th In-Reply-To: References: Message-ID: <201503261524.t2QFOQrO020615@msw1.awe.co.uk> As well as supporting the efforts of our friends at NERSC I think should be a good interesting talk. Mark Roberts ________________________________ From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Ross Keeping3 Sent: 26 March 2015 15:00 To: gpfsug-discuss at gpfsug.org Subject: EXTERNAL: [gpfsug-discuss] Potential talks for GPFS UG May 20th Hi There are a couple of talks we could include in the one day GPFS UG on May 20th. We'd appreciate some feedback on whether there is any interest or objection to including these. These are: 1) Tivoli Storage Productivity Center Technical Demo: A tool for managing heterogeneous storage infrastructures and specifically GPFS. Can read more here http://www-03.ibm.com/software/products/en/tivostorprodcent 2) GPFS for life sciences: A member of NERSC would like to talk about their efforts to improve GPFS file systems for a life sciences workload - specifically genomic sequencing and analysis that ensues. Details about life science workflow and requirements will be provided so the listener may understand the aspects of GPFS that are important. Cheers, Ross Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU ___________________________________________________ ____________________________ The information in this email and in any attachment(s) is commercial in confidence. If you are not the named addressee(s) or if you receive this email in error then any distribution, copying or use of this communication or the information in it is strictly prohibited. Please notify us immediately by email at admin.internet(at)awe.co.uk, and then delete this message from your computer. While attachments are virus checked, AWE plc does not accept any liability in respect of any virus which is not detected. AWE Plc Registered in England and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR -------------- next part -------------- An HTML attachment was scrubbed... URL: From Sandra.McLaughlin at astrazeneca.com Thu Mar 26 15:56:16 2015 From: Sandra.McLaughlin at astrazeneca.com (McLaughlin, Sandra M) Date: Thu, 26 Mar 2015 15:56:16 +0000 Subject: [gpfsug-discuss] EXTERNAL: Potential talks for GPFS UG May 20th In-Reply-To: <201503261524.t2QFOQrO020615@msw1.awe.co.uk> References: <201503261524.t2QFOQrO020615@msw1.awe.co.uk> Message-ID: Life Sciences talk would be spot on for me too, please. Sandra McLaughlin From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Mark.Roberts at awe.co.uk Sent: 26 March 2015 15:24 To: gpfsug-discuss at gpfsug.org Subject: Re: [gpfsug-discuss] EXTERNAL: Potential talks for GPFS UG May 20th As well as supporting the efforts of our friends at NERSC I think should be a good interesting talk. Mark Roberts ________________________________ From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Ross Keeping3 Sent: 26 March 2015 15:00 To: gpfsug-discuss at gpfsug.org Subject: EXTERNAL: [gpfsug-discuss] Potential talks for GPFS UG May 20th Hi There are a couple of talks we could include in the one day GPFS UG on May 20th. We'd appreciate some feedback on whether there is any interest or objection to including these. These are: 1) Tivoli Storage Productivity Center Technical Demo: A tool for managing heterogeneous storage infrastructures and specifically GPFS. Can read more here http://www-03.ibm.com/software/products/en/tivostorprodcent 2) GPFS for life sciences: A member of NERSC would like to talk about their efforts to improve GPFS file systems for a life sciences workload - specifically genomic sequencing and analysis that ensues. Details about life science workflow and requirements will be provided so the listener may understand the aspects of GPFS that are important. Cheers, Ross Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU ___________________________________________________ ____________________________ The information in this email and in any attachment(s) is commercial in confidence. If you are not the named addressee(s) or if you receive this email in error then any distribution, copying or use of this communication or the information in it is strictly prohibited. Please notify us immediately by email at admin.internet(at)awe.co.uk, and then delete this message from your computer. While attachments are virus checked, AWE plc does not accept any liability in respect of any virus which is not detected. AWE Plc Registered in England and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR ________________________________ AstraZeneca UK Limited is a company incorporated in England and Wales with registered number: 03674842 and a registered office at 2 Kingdom Street, London, W2 6BD. Confidentiality Notice: This message is private and may contain confidential, proprietary and legally privileged information. If you have received this message in error, please notify us and remove it from your system and note that you must not copy, distribute or take any action in reliance on it. Any unauthorised use or disclosure of the contents of this message is not permitted and may be unlawful. Disclaimer: Email messages may be subject to delays, interception, non-delivery and unauthorised alterations. Therefore, information expressed in this message is not given or endorsed by AstraZeneca UK Limited unless otherwise notified by an authorised representative independent of this message. No contractual relationship is created by this message by any person unless specifically indicated by agreement in writing other than email. Monitoring: AstraZeneca UK Limited may monitor email traffic data and content for the purposes of the prevention and detection of crime, ensuring the security of our computer systems and checking compliance with our Code of Conduct and policies. -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.kidger at uk.ibm.com Thu Mar 26 16:06:24 2015 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Thu, 26 Mar 2015 16:06:24 +0000 Subject: [gpfsug-discuss] Potential talks for GPFS UG May 20th In-Reply-To: <201503261513.062218@mail.strubi.ox.ac.uk> References: <201503261513.062218@mail.strubi.ox.ac.uk> Message-ID: I too would want to upvote the second. I am seeing lots of interest in the Life Sciences particularly for things like large Genomic datasets and NGS. Daniel Dr.Daniel Kidger No. 1 The Square, Technical Specialist SDI (formerly Platform Computing) Temple Quay, Bristol BS1 6DG Mobile: +44-07818 522 266 United Kingdom Landline: +44-02392 564 121 (Internal ITN 3726 9250) e-mail: daniel.kidger at uk.ibm.com From: Robert Esnouf To: "gpfsug main discussion list" Date: 26/03/2015 15:13 Subject: Re: [gpfsug-discuss] Potential talks for GPFS UG May 20th Sent by: gpfsug-discuss-bounces at gpfsug.org Yes, unsurprisingly I'd be interested in the second talk! Thanks, Robert -- Dr. Robert Esnouf, University Research Lecturer, Head of Research Computing Core, NDM Research Computing Strategy Officer Room 10/028, Wellcome Trust Centre for Human Genetics, Old Road Campus, Roosevelt Drive, Oxford OX3 7BN, UK Email: robert at strubi.ox.ac.uk / robert at well.ox.ac.uk Tel: (+44) - 1865 - 287783 ---- Original message ---- >Date: Thu, 26 Mar 2015 14:59:31 +0000 >From: gpfsug-discuss-bounces at gpfsug.org (on behalf of Ross Keeping3 ) >Subject: [gpfsug-discuss] Potential talks for GPFS UG May 20th >To: gpfsug-discuss at gpfsug.org > > Hi > > There are a couple of talks we could include in the > one day GPFS UG on May 20th. > > We'd appreciate some feedback on whether there is > any interest or objection to including these. > > These are: > 1) Tivoli Storage Productivity Center Technical > Demo: A tool for managing heterogeneous storage > infrastructures and specifically GPFS. Can read more > here > http://www-03.ibm.com/software/products/en/tivostorprodcent > > 2) GPFS for life sciences: A member of NERSC would > like to talk about their efforts to improve GPFS > file systems for a life sciences workload - > specifically genomic sequencing and analysis that > ensues. Details about life science workflow and > requirements will be provided so the listener may > understand the aspects of GPFS that are important. > > Cheers, > Ross > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England > and Wales with number 741598. > Registered office: PO Box 41, North Harbour, > Portsmouth, Hampshire PO6 3AU >________________ >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at gpfsug.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 360 bytes Desc: not available URL: From zgiles at gmail.com Thu Mar 26 16:14:36 2015 From: zgiles at gmail.com (Zachary Giles) Date: Thu, 26 Mar 2015 12:14:36 -0400 Subject: [gpfsug-discuss] EXTERNAL: Potential talks for GPFS UG May 20th In-Reply-To: References: <201503261524.t2QFOQrO020615@msw1.awe.co.uk> Message-ID: It's at the GPFSUG in the UK. Looks like the UK GPFSUG is much more active than the US one's. They've had a few meetups actually. I'd fly out there and attend them, but I figured probably it wouldn't count since it's not a real conf. This on is in May 20th. On Thu, Mar 26, 2015 at 11:56 AM, McLaughlin, Sandra M < Sandra.McLaughlin at astrazeneca.com> wrote: > Life Sciences talk would be spot on for me too, please. > > > > Sandra McLaughlin > > > > *From:* gpfsug-discuss-bounces at gpfsug.org [mailto: > gpfsug-discuss-bounces at gpfsug.org] *On Behalf Of *Mark.Roberts at awe.co.uk > *Sent:* 26 March 2015 15:24 > *To:* gpfsug-discuss at gpfsug.org > *Subject:* Re: [gpfsug-discuss] EXTERNAL: Potential talks for GPFS UG May > 20th > > > > As well as supporting the efforts of our friends at NERSC I think should > be a good interesting talk. > > > > Mark Roberts > > > ------------------------------ > > *From:* gpfsug-discuss-bounces at gpfsug.org [ > mailto:gpfsug-discuss-bounces at gpfsug.org > ] *On Behalf Of *Ross Keeping3 > *Sent:* 26 March 2015 15:00 > *To:* gpfsug-discuss at gpfsug.org > *Subject:* EXTERNAL: [gpfsug-discuss] Potential talks for GPFS UG May 20th > > > > Hi > > There are a couple of talks we could include in the one day GPFS UG on May > 20th. > > We'd appreciate some feedback on whether there is any interest or > objection to including these. > > These are: > 1) Tivoli Storage Productivity Center Technical Demo: A tool for managing > heterogeneous storage infrastructures and specifically GPFS. Can read more > here http://www-03.ibm.com/software/products/en/tivostorprodcent > > 2) GPFS for life sciences: A member of NERSC would like to talk about > their efforts to improve GPFS file systems for a life sciences workload - > specifically genomic sequencing and analysis that ensues. Details about > life science workflow and requirements will be provided so the listener may > understand the aspects of GPFS that are important. > > Cheers, > Ross > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > ___________________________________________________ > ____________________________ The information in this email and in any > attachment(s) is commercial in confidence. If you are not the named > addressee(s) or if you receive this email in error then any distribution, > copying or use of this communication or the information in it is strictly > prohibited. Please notify us immediately by email at admin.internet(at) > awe.co.uk, and then delete this message from your computer. While > attachments are virus checked, AWE plc does not accept any liability in > respect of any virus which is not detected. AWE Plc Registered in England > and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR > ------------------------------ > > AstraZeneca UK Limited is a company incorporated in England and Wales with > registered number: 03674842 and a registered office at 2 Kingdom Street, > London, W2 6BD. > > *Confidentiality Notice: *This message is private and may contain > confidential, proprietary and legally privileged information. If you have > received this message in error, please notify us and remove it from your > system and note that you must not copy, distribute or take any action in > reliance on it. Any unauthorised use or disclosure of the contents of this > message is not permitted and may be unlawful. > > *Disclaimer:* Email messages may be subject to delays, interception, > non-delivery and unauthorised alterations. Therefore, information expressed > in this message is not given or endorsed by AstraZeneca UK Limited unless > otherwise notified by an authorised representative independent of this > message. No contractual relationship is created by this message by any > person unless specifically indicated by agreement in writing other than > email. > > *Monitoring: *AstraZeneca UK Limited may monitor email traffic data and > content for the purposes of the prevention and detection of crime, ensuring > the security of our computer systems and checking compliance with our Code > of Conduct and policies. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- Zach Giles zgiles at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From luke.raimbach at oerc.ox.ac.uk Fri Mar 27 12:16:16 2015 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Fri, 27 Mar 2015 12:16:16 +0000 Subject: [gpfsug-discuss] mmunlinkfileset problems Message-ID: Hi All, I'm having a problem where unlinking a fileset is taking 7 or 8 minutes to complete. We are running 3.5.0.22 and the cluster is small (9 nodes: 8 Linux plus 1 Windows node), one file system remote mounted by a 3-node Linux cluster, and generally not very busy. I wouldn't expect mmunlinkfileset to require the file system to quiesce (and even if it did 8 minutes would seem like an awfully long time). Previous unlink commands (on version 3.5.0.19) have returned within milliseconds. This very long running command appeared to pause IO for a sufficiently long time that cNFS clients timed out too and various services elsewhere crashed. Is there any reason why unlinking a fileset would take this long? Also, I can't find anywhere a list of commands which might require the FS to quiesce - is this information available somewhere? Cheers, Luke. -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 From jonathan at buzzard.me.uk Fri Mar 27 12:49:38 2015 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Fri, 27 Mar 2015 12:49:38 +0000 Subject: [gpfsug-discuss] mmunlinkfileset problems In-Reply-To: References: Message-ID: <1427460578.3553.153.camel@buzzard.phy.strath.ac.uk> On Fri, 2015-03-27 at 12:16 +0000, Luke Raimbach wrote: > Hi All, > > I'm having a problem where unlinking a fileset is taking 7 or 8 minutes > to complete. We are running 3.5.0.22 and the cluster is small (9 nodes: > 8 Linux plus 1 Windows node), one file system remote mounted by a > 3-node Linux cluster, and generally not very busy. > > I wouldn't expect mmunlinkfileset to require the file system to quiesce > (and even if it did 8 minutes would seem like an awfully long time). > Previous unlink commands (on version 3.5.0.19) have returned within > milliseconds. If there is a waiter around mmunlinkfileset can block even with an empty fileset and turn your file system unresponsive. > > This very long running command appeared to pause IO for a sufficiently > long time that cNFS clients timed out too and various services > elsewhere crashed. > Yep, lesson for the day only unlink filesets when the file system is quiet, aka a maintenance window. > Is there any reason why unlinking a fileset would take this long? Also, > I can't find anywhere a list of commands which might require the FS to > quiesce - is this information available somewhere? > Deleting snapshots, at least in the past. It might be that guidance from IBM has changed but once you have been stung by this you kind of thing you don't like to try it again any time soon. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From luke.raimbach at oerc.ox.ac.uk Fri Mar 27 13:27:20 2015 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Fri, 27 Mar 2015 13:27:20 +0000 Subject: [gpfsug-discuss] mmunlinkfileset problems In-Reply-To: <1427460578.3553.153.camel@buzzard.phy.strath.ac.uk> References: <1427460578.3553.153.camel@buzzard.phy.strath.ac.uk> Message-ID: > > Is there any reason why unlinking a fileset would take this long? > > Also, I can't find anywhere a list of commands which might require the > > FS to quiesce - is this information available somewhere? > > > > Deleting snapshots, at least in the past. It might be that guidance from IBM > has changed but once you have been stung by this you kind of thing you > don't like to try it again any time soon. Precisely why I'm after a list of commands that might require quiescence! I know that snapshot operations (generally) and deleting filesets require a consistent view so will pause IO, but unlinking (in my mind) wouldn't need this... I'd probably want to generate lists of 'cleanup' commands that might interrupt IO and run these in maintenance windows as you say. Gather up all the filesets that want deleting and do them once a week / whenever. Cheers, Luke. From jonathan at buzzard.me.uk Fri Mar 27 13:39:19 2015 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Fri, 27 Mar 2015 13:39:19 +0000 Subject: [gpfsug-discuss] mmunlinkfileset problems In-Reply-To: References: <1427460578.3553.153.camel@buzzard.phy.strath.ac.uk> Message-ID: <1427463559.3553.156.camel@buzzard.phy.strath.ac.uk> On Fri, 2015-03-27 at 13:27 +0000, Luke Raimbach wrote: > > > Is there any reason why unlinking a fileset would take this long? > > > Also, I can't find anywhere a list of commands which might require the > > > FS to quiesce - is this information available somewhere? > > > > > > > Deleting snapshots, at least in the past. It might be that guidance from IBM > > has changed but once you have been stung by this you kind of thing you > > don't like to try it again any time soon. > > Precisely why I'm after a list of commands that might require > quiescence! I know that snapshot operations (generally) and deleting > filesets require a consistent view so will pause IO, but unlinking (in > my mind) wouldn't need this... > mmdeldisk is another. From what I can make out anything that takes away from the file system is a potential problem. Anything that adds to the file system is just peachy fine to be run at any time. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom.